# Package-Aware Approach for Repository-Level Code Completion in Pharo

Omar Abedelkader, Stéphane Ducasse, Oleksandr Zaitsev, Romain Robbes, Guillermo Polito

► To cite this version:

Omar Abedelkader, Stéphane Ducasse, Oleksandr Zaitsev, Romain Robbes, Guillermo Polito. Package-Aware Approach for Repository-Level Code Completion in Pharo. Proceedings of the International Workshop on Smalltalk Technologies 2025, IWST2025, 2025. hal-05446902

HAL Id: hal-05446902

https://hal.science/hal-05446902v1

Submitted on 8 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Package-Aware Approach for Repository-Level Code Completion in Pharo

Omar AbedelKader<fnref n="1" />, Stéphane Ducasse<fnref n="1" />, Oleksandr Zaitsev<fnref n="2" />, Romain Robbes<fnref n="3" /> and Guillermo Polito<fnref n="1" />

## Abstract

Pharo offers a sophisticated completion engine based on semantic heuristics, which coordinates specific fetchers within a lazy architecture. These heuristics can be recomposed to support various activities (e.g., live programming or history usage navigation). While this system is powerful, it does not account for the repository structure when suggesting global names such as class names, class variables, or global variables. As a result, it does not prioritize classes within the same package or project, treating all global names equally.

In this paper, we present a new heuristic that addresses this limitation. Our approach searches variable names in a structured manner: it begins with the package of the requesting class, then expands to other packages within the same repository, and finally considers the global namespace. We describe the logic behind this heuristic and evaluate it against the default semantic heuristic and one that directly queries the global namespace. Preliminary results indicate that the Mean Reciprocal Rank (MRR) improves, confirming that package-awareness completions deliver more accurate and relevant suggestions than the previous flat global approach.

## Keywords

Repository-Level Completion, Code Completion, Pharo, Smalltalk

## 1. Introduction

Modern code completion systems are increasingly expected to be intelligent, predicting not only syntactically valid tokens but also semantically relevant ones, thereby reducing the effort required to navigate large codebases [JH25]. In the Pharo programming environment [BDN⁺⁰⁹], Complishon is a sophisticated code completion engine designed with these goals. Developed by the last author, Complishon employs a modular and lazy architecture based on semantic heuristics that consider both the type of entity (e.g., variable, class, method) and its location in the program structure (e.g., prioritizing classes in the current class hierarchy before superclasses). Unlike traditional alphabetical or purely syntactic completion systems, such as those criticized by Bruch *et al.*, [BMM09] and Robbes *et al.*, [RL10], COMPLISHON is grounded in the idea that completion quality can be significantly improved by taking into account a context.

However, even with these heuristics, COMPLISHON, like many completion engines, originally treated the global namespace as flat, walking through global entities in a non-deterministic order other than matching prefixes. This becomes problematic in industrial-scale systems, where the number of possible completions can be overwhelming. For instance, companies using Pharo in production, such as Lifeware, maintain systems exceeding 30 million lines of code <fnref n="1" />. In these contexts, completion suggestions must

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<footnote n="1">¹Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 CRISTAL, Park Plaza, Parc scientifique de la Haute-Borne, 40 Av. Halley Bât A, 59650 Villeneuve-d'Ascq, France</footnote>

<footnote n="2">²CIRAD, UMR SENS, F-34000 Montpellier, France</footnote>

<footnote n="3">³CNRS, University of Bordeaux, Bordeaux INP, LaBRI, UMR5800, F-33400, Talence, France</footnote>

<footnote>IWST 2025: *International Workshop on Smalltalk Technologies, July 1-4, 2025, Gdańsk, Poland*</footnote>

<footnote>omar.abedelkader@inria.fr (O. AbedelKader); stephane.ducasse@inria.fr (S. Ducasse); oleksandr.zaitsev@cirad.fr</footnote>

<footnote>(O. Zaitsev); romain.robbes@labri.fr (R. Robbes); guillermo.polito@inria.fr (G. Polito)</footnote>

<footnote>https://omarabedelkader.github.io (O. AbedelKader); https://stephane.ducasse.free.fr/ (S. Ducasse);</footnote>

<footnote>https://umr-sens.fr/fr/-/zaitsev-oleksandr (O. Zaitsev); https://rrobbes.github.io/ (R. Robbes);</footnote>

<footnote>https://guillep.github.io (G. Polito)</footnote>

<footnote>0009-0005-1339-5683 (O. AbedelKader); 0000-0001-6070-6599 (S. Ducasse); 0000-0003-0267-2874 (O. Zaitsev);</footnote>

<footnote>0000-0003-4569-6868 (R. Robbes); 0000-0003-0813-8584 (G. Polito)</footnote>

<footnote>© 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).</footnote>

<footnote n="1">¹10th Anniversary of Pharo</footnote>
not only be semantically relevant but also efficiently scoped to avoid information overload. One
significant omission in the original design of COMPLISHON was awareness of package structure.

While it excelled in suggesting global names like classes or variables, it did not favor entities from the same package or related parts of the system. COMPLISHON does not prioritize or even adequately surface entities from the same package as the current editing context. As shown in Figure 1, when invoking code completion within the SpPresenter class (located in the Spec2-Core package), the suggestions are entirely drawn from the global namespace, ignoring nearby classes such as SpPresenterBuilder, SpTextPresenter, or SpApplication, which are defined in the same package. This leads to a degraded developer experience, especially in large codebases where numerous globally accessible entities can easily overshadow locally relevant ones.

Figure 1 shows the potential candidate completions from packages Spec2-core, highlighting COMPLISHON's failure to leverage local package context effectively. This is critical in Pharo, where projects are modularized into packages (e.g., Spec2-Core, Spec2-Dialogs, Spec2-Interaction) that group functionally related classes. Developers often work within a small subset of the system, typically their packages, its dependencies, and core libraries, and should not be distracted by completions from unrelated parts of the codebase. Specifically, the figure illustrates an error where the first ten completion suggestions (such as SpInteractionError, SpJobListPresenter, etc.) are not part of the Spec2-Core package, thus underscoring the importance of package-aware code completion that respects package boundaries.

<figure><img src="image_6.png" /><figcaption>Figure 1: Default code completion interface in Pharo (Flat Global Scope Suggestions Without Package Awareness).</figcaption></figure>

This article proposes and evaluates a repository-level completion strategy, a simple yet effective enhancement to the COMPLISHON architecture that ranks candidates from the current package highest, followed by suggestions from lateral packages, and only then from the rest global namespace. The goal is to improve both relevance and responsiveness by exploiting the modularity already present in Pharo subsystems. This approach resonates with trends in completion research that use structural or probabilistic models such as Bayesian strategies [PLM15] or JetBrain's' log-based rankings [BKL+22]
to make completions more context-aware. Our work is further motivated by principles from the
moldable development paradigm, as described by Chis et al., [CNG15], which argues that tools like
code completion should be extensible and adaptable to specific development contexts. In this sense,
COMPLISHON aligns with these ideals by offering a plugin architecture for heuristics, including our
proposed package-awareness logic.

This article is structured as follows: Section 2 gives an overview of the COMPLISHON engine and its modular heuristic-based architecture. Section 3 identifies the limitations of global-environment-based completion, presents our approach, and outlines our hypothesis for package awareness suggestions. Section 4 describes our evaluation across multiple projects and strategies. Section 5 discusses the findings and their implications for completion systems in dynamic, large-scale environments. Section 6 situates our work within the broader landscape of static code completion. Section 7 outlines the main limitations of our approach, including the reliance on static reference points, the use of truncated identifiers that may not reflect real-world usage, and the challenges posed by Pharo-specific naming and package structures. Finally, Section 8 outlines future directions for integrating adaptive and learned strategies into the COMPLISHON engine.

## 2. Background

COMPLISHON the Pharo completion engine (see Figure 2), consists primarily of three key components: Heuristics, Lazy Fetchers, and a lazily cached Result Set. At the core, heuristics provide semantic guidance for the completion process by analyzing the Abstract Syntax Tree (AST) node located at the cursor (editor caret) and selecting the appropriate fetchers for completion suggestions. These heuristics are structured in a chain of responsibility [GHJV95], allowing a heuristic to pass handling responsibilities down the chain if it cannot process the current AST node itself. Fetchers, implemented using combinators, lazily retrieve and filter potential code completion candidates based on the context and user input, significantly optimizing performance and memory usage. A decorator pattern further enhances fetchers by preventing duplicate suggestions, particularly crucial in scenarios involving method inheritance ensuring results remain relevant and unique. Fetchers utilize specialized filters (e.g., CoBeginsWithFilter), to match completion suggestions with the user's partially typed input.

<figure><img src="image_7.png" /><figcaption>Figure 2: COMPLISHON's architecture</figcaption></figure>

The Result Set component serves as a lazy, cached store that accumulates the suggestions provided by fetchers only as required, further enhancing efficiency. COMPLISHON's leverages AST-based analysis and
utilizes a double-dispatch mechanism to adapt dynamically to the surrounding context. It constructs a dedicated context for code completion, considering factors such as source text and caret position. The generation and presentation of suggestions are managed by the IDE, which also integrates strategies such as case sensitivity filtering and adaptive configuration based on the structure and semantics of the current code environment. This adaptability employs heuristic-based fetchers configured by visitor patterns, informed by parsing and typing processes, to refine output and dynamically eliminate redundant or irrelevant suggestions.

The heuristics are modular, specialized for different code elements such as messages, variables, and symbols, and systematically connected in sequences forming a robust and comprehensive filtering framework. This chaining process includes sophisticated program semantics-based strategies such as prioritizing instance variables before superclass variables, self message suggestions, inherited methods, and inferred initialization constructs. Consequently, COMPLISHON provides an efficient, contextually aware, and highly accurate completion experience tailored precisely to the user's current coding scenario.

# 3. Our approach: Repository level package structure

## 3.1. Repository Level Completion

Although COMPLISHON is effective in identifying global entities such as class names, class variables, or global variables, it currently does not leverage the package structure of a project effectively. As a result, it treats all global names uniformly, offering no preference to entities located in the same or related packages. This behavior contrasts with several insights from prior research, which emphasize the importance of contextual filtering and locality in improving the relevance of code completion results. For instance, Hou et al., [HP10] propose heuristic techniques to filter and sort API suggestions using type hierarchies and grouping, significantly reducing the visual clutter of irrelevant entries. Similarly, Bruch et al., [BMM09] introduced the Best Matching Neighbors (BMN) approach, which ranks suggestions based on similarity to local usage contexts, an idea analogous to prioritizing completions from the same or nearby packages. Robbes et al., [RL08a] further underscore the value of recent usage and lexical proximity, showing that local context often outperforms global frequency. More recently, Hellendoorn et al., [HPGB19] demonstrated that intra-project completions remain the hardest for current models, largely due to their inability to distinguish between local and global identifiers. In a complementary direction, Li et al., [LHL+21] proposed learned acceptance and ranking models to suppress noisy completions, optimizing not just for correctness but for developer effort.

Inspired by these works, we introduce a new heuristic in COMPLISHON that leverages package structure to improve completion prioritization (See Figure 3). For example, when performing auto-completion within a class in the P1-Core package, typing the letter A should ideally yield suggestions in the following order: first from P1-Core, then from related packages such as P1-Extension or P1-Test, and only afterward from the global namespace.

## 3.2. Leveraging Project Package Structure

To enable this behavior, we designed a package-aware completion heuristic that operates in three steps:

* **Identifying the current package:** The completion engine determines the active package context, such as P1-Core.

* **Collecting potential matches:** Typing A triggers a scan for all relevant matches in local and global names.

* **Prioritizing based on package proximity:**

1. First, suggest entities defined within the current package (e.g., P1-Core).
<figure><img src="image_8.png" /><figcaption>**Figure 3: Flat vs. Package Awareness.** In the flat model, all global variables are visible without a package context. In the package-aware model, name resolution follows a structured order: current package (P1-Core), then related packages (P1-Test, P1-Extension), and finally global namespace (AGlobal, Almage).</figcaption></figure>

2. Next, prefer suggestions from closely related packages (Lateral Dependencies) (e.g., P1-Extension or P1-Test). These are currently inferred from naming similarity rather than formal dependencies.

3. Finally, include entities from the remaining global scope, if further suggestions are needed.

This strategy mirrors user expectations by elevating locally relevant completions, thereby reducing cognitive load and supporting the findings of prior empirical and usability studies.

## 3.3. Implementation Details

This behavior is implemented through new extension points in COMPLISHON, enabling fine-grained control over completion fetchers and filtering logic. These enhancements support dynamic reordering of suggestions based on a package-locality heuristic, which prioritizes elements from the current package and its immediate relations. For instance, when auto-completing in a class within the package P1-Core, typing the letter A will first yield suggestions from P1-Core, followed by related packages like P1-Extension or P1-Test, and finally the global namespace. This structured prioritization reduces cognitive overhead and improves developer productivity. To evaluate the impact of this heuristic, we refined our benchmarks to focus on variable completions specifically. As shown in Figure 4, invoking completion in a Spec2-Core context now highlights relevant classes such as SpIconProvider, SpPresenterBuilder, and SpTextPresenter. These results confirm the heuristic's effectiveness in promoting package-local relevance and improving suggestion accuracy.

# 4. Evaluation

## 4.1. Benchmark Logic

In this work, we only focus on the completion of global variables such as class names <fnref n="2" />. To evaluate the effectiveness of variable name completion algorithms, we based our implementation on the benchmark methodology introduced by Robbes *et al.*, [RL08a, RL08b]. Although their original benchmark relied on a change-based repository of program history, our approach adapts the core idea for static analysis and variable-centric benchmarking, making it applicable in contexts where historical data is unavailable. The essence of this benchmark is to test whether a completion engine can correctly suggest the

<footnote n="2">²In Pharo, class names are global variables. In practice, most globals considered in this study are class names.</footnote>
<figure><img src="image_9.png" /><figcaption>Figure 4: Improved Suggestions with Package-Scoped Heuristic</figcaption></figure>

original names of variables when only partial prefixes are provided. The fundamental insight is that by systematically rewriting every variable access site with increasingly longer prefixes (ranging from 2 to 8 characters), we can invoke the completion engine and evaluate whether it ranks the correct name high in its suggestions. This method simulates realistic completion scenarios and allows us to measure not just correctness but also ranking performance and user effort reduction. Our system, implemented via the StaticBenchmarksVariables class, carries out the following steps for each method in a given package:

1. **AST Extraction:** The method's AST is retrieved to analyze variable usages within the code.

2. **Variable Filtering:** Each variable node is examined to confirm it is global (usually indicated by an uppercase-starting name).

3. **Name Masking:** The variable name is programmatically shortened to generate several prefixes of increasing lengths, from 2 up to 8 characters (or the full name length, if shorter). For example, the variable OrderedCollection yields prefixes such as Or, Ord, Orde, etc.

4. **Completion Invocation:** For each prefix, the completion engine is triggered as if the user were requesting suggestions after typing that fragment.

5. **Result Logging:** The engine's output is analyzed to determine if the original name appears among the top 10 suggestions. If found, we record the rank at which it was suggested. If not, it is considered a failure for that prefix length.

This process is repeated for all global variables in all methods of the targeted package or class scope. The benchmark is implemented in StaticBenchmarks class and uses a ResultSetBuilder to generate completions based on specified heuristics. We provide a modular API allowing execution across individual classes or entire packages. Several metrics are collected:
1. **Accuracy**: The proportion of cases where the original variable name appears within the top-k results (typically top-10).

2. **Rank Distribution**: The frequency at which the correct name appears at each specific rank position from 1st through 10th.

3. **Mean Reciprocal Rank (MRR)**: Measures the average reciprocal rank position of the first correct prediction. Formally, given a set of queries Q, the MRR is calculated as:

$$MRR = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{\text{rank}_i}$$

where rank$_i$ is the rank position of the first correct prediction for the i-th query. MRR emphasizes the importance of placing the first correct result as high as possible and is particularly suitable when only one relevant result is sufficient per query [MC18].

4. **Normalized Discounted Cumulative Gain (NDCG)**: Measures the usefulness of predictions based on their positions and graded relevance. The DCG is calculated as:

$$DCG_k = \sum_{i=1}^{k} \frac{2^{\text{rel}_i} - 1}{\log_2(i + 1)}$$

where rel$_i$ represents the graded relevance of the prediction at position i. NDCG normalizes this score by dividing by the ideal DCG (IDCG), yielding:

$$NDCG_k = \frac{DCG_k}{IDCG_k}$$

thus accounting logarithmically for position discounting [MC18].

5. **Timing and Memory Metrics**: Includes total and average completion times and memory usage, evaluated per prefix length.

The benchmark can be run in various configurations, from baseline alphabetical sorters to advanced heuristic-guided engines. It can also be configured with different heuristic templates, offering flexibility to evaluate a wide range of completion strategies. Our benchmark is designed not only to test whether a completion system finds the correct result but also how efficiently it does so, reflecting real-world usage where users expect high precision with minimal typing. By measuring performance across multiple prefix lengths and analyzing rank-based metrics, we can assess the trade-offs and effectiveness of various completion algorithms. We chose MRR as our primary evaluation metric because it effectively captures the quality of ranked suggestions in code completion tasks. MRR reflects how early the correct suggestion appears in the list, aligning closely with real user experience. This makes it particularly suitable for comparing ranking-based approaches, such as the STAN-based reranking models we evaluate, and is consistent with evaluation protocols in prior work on neural code completion [SLH+21].

## 4.2. Package Selection

Based on the statistical analysis of Pharo code by Zaitsev et al., [ZDA20] we selected packages that ensure a broad domain diversity, including web development, visualization, software analysis, and user-interface frameworks, while also focusing on projects that demonstrated active maintenance, extensive test suites, and representative structural attributes, such as method length distributions and language feature usage patterns.

We selected Pharo packages reflecting significant diversity in domain, size, and development activity. The chosen packages span essential areas such as visualization (Roassal), software analysis (Moose), web application development (Seaside), user interface framework (Spec2), and version control systems (Iceberg). These packages were carefully selected based on their substantial adoption, active maintenance, extensive test coverage, and their use of key language features including polymorphism, reflective capabilities, and Pharo-specific syntax. This selection strategy ensures our benchmarks effectively
capture common programming practices, typical complexity, and diverse usage patterns prevalent within the Pharo community. A comprehensive description of the selected packages is provided in Appendix A.

<table><caption>**Table 1**
Overview of Selected Pharo Frameworks for Benchmark</caption><thead><tr><th>Framework/Bib</th><th># Packages</th><th># Classes</th><th># Defined Classes</th><th># Methods</th><th>ρ<sub>int</sub></th><th>R<sub>int</sub></th><th>R<sub>ext</sub></th></tr></thead><tbody><tr><td>IceBerg</td><td>11</td><td>610</td><td>525</td><td>4882</td><td>0.35</td><td>229</td><td>292</td></tr><tr><td>Moose</td><td>72</td><td>910</td><td>751</td><td>7581</td><td>0.21</td><td>221</td><td>1035</td></tr><tr><td>Roassal</td><td>52</td><td>794</td><td>629</td><td>7230</td><td>0.14</td><td>158</td><td>1311</td></tr><tr><td>Seaside</td><td>44</td><td>884</td><td>758</td><td>6605</td><td>0.21</td><td>305</td><td>552</td></tr><tr><td>Spec</td><td>33</td><td>1115</td><td>939</td><td>8383</td><td>0.21</td><td>278</td><td>593</td></tr></tbody></table>

Table 1 provides a comprehensive overview of the selected Pharo frameworks evaluated in our benchmarking experiments. It includes the total number of packages, classes, defined classes, and methods analyzed in each framework. Additionally, the table presents key metrics such as the ratio of internal references ($ρ_{int}$), number of internal references ($R_{int}$), and the number of external references ($R_{ext}$). A higher $ρ_{int}$ indicates stronger intra-package cohesion, reflecting more frequent references to internal entities rather than external ones. For example, Iceberg shows a relatively high $ρ_{int}$ (0.35), suggesting significant internal cohesion, whereas Roassal exhibits the lowest $ρ_{int}$ (0.14), implying a greater reliance on external references. These metrics provide context for understanding how package structure and usage patterns influence the effectiveness of our package-aware heuristic.

# 5. Results and Discussion

## 5.1. Overall Results

Our evaluation demonstrates the advantages and limitations of introducing a package-aware heuristic into Pharo's completion engine. Overall, we observed (Table 2) a significant improvement in MRR, indicating that developers receive more contextually relevant completion suggestions when package structure is leveraged. The average MRR improvement across all evaluated frameworks was notable, especially pronounced in frameworks like Spec (7.59%) and Iceberg (6.09%), which exhibit well-defined package structures with strong intra-package cohesion. However, the results also reveal nuanced behavior depending on the type of package and the framework's architectural characteristics. For instance, the Moose framework showed modest overall gains (1.05%), which were more substantial in non-test packages (1.19%) but negligible or negative in test packages (0.31%).

This highlights a key insight completion accuracy gains are context-sensitive, often dependent on package dependencies and naming conventions. Test packages, for example, frequently access numerous external global variables and classes, reducing the effectiveness of a strictly package-local prioritization heuristic. The Roassal framework exhibited similar mixed outcomes. Non-test packages show improvements (2.14%), whereas test packages experienced performance degradation (-2.62%). This negative outcome indicates that the current heuristic, which infers package relationships from naming conventions alone, may inadvertently prioritize less relevant local completions in testing scenarios, where global or cross-package dependencies are prevalent.

Interestingly, Seaside and Spec demonstrated consistent improvements across both test and non-test packages, suggesting that in frameworks with strong internal modularization and delineated package structures, the heuristic significantly enhances completion relevance across diverse coding contexts. Another critical observation is that the heuristic's effectiveness diminishes as prefix lengths increase, with the most significant improvements consistently appearing at shorter prefix lengths (2-4 characters). This outcome aligns with real-world coding behavior, where developers rely heavily on early suggestions to reduce typing effort. However, the diminishing returns at longer prefixes suggest a reduced practical advantage once developers have provided extensive typing context. Our analysis also uncovered
<table><caption>**Table 2**</caption><thead><tr><th>Framework/Bib</th><th>Package Type</th><th>⟨Metric⟩</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td rowspan="3">Iceberg</td><td rowspan="3">Overall</td><td>Without</td><td>31.27</td><td>5.36</td><td>16.91</td><td>27.82</td><td>37.45</td><td>39.91</td><td>45.00</td><td>49.91</td></tr><tr><td>With</td><td>37.36</td><td>12.64</td><td>23.82</td><td>37.18</td><td>44.00</td><td>43.45</td><td>49.73</td><td>54.64</td></tr><tr><td>Δ</td><td>6.09</td><td>7.27</td><td>6.91</td><td>9.36</td><td>6.55</td><td>3.55</td><td>4.73</td><td>4.73</td></tr><tr><td rowspan="6">Moose</td><td rowspan="3">Overall</td><td>Without</td><td>40.53</td><td>17.32</td><td>19.98</td><td>31.92</td><td>37.95</td><td>44.31</td><td>49.92</td><td>58.95</td></tr><tr><td>With</td><td>41.58</td><td>23.19</td><td>23.47</td><td>31.44</td><td>37.15</td><td>44.37</td><td>49.32</td><td>58.44</td></tr><tr><td>Δ</td><td>1.05</td><td>5.86</td><td>3.49</td><td>-0.47</td><td>-0.80</td><td>0.07</td><td>-0.59</td><td>-0.51</td></tr><tr><td rowspan="3">Test</td><td>Without</td><td>22.09</td><td>3.79</td><td>10.29</td><td>17.93</td><td>21.57</td><td>26.43</td><td>32.36</td><td>47.71</td></tr><tr><td>With</td><td>22.39</td><td>6.79</td><td>11.57</td><td>17.21</td><td>20.29</td><td>25.64</td><td>32.00</td><td>48.64</td></tr><tr><td>Δ</td><td>0.31</td><td>3.00</td><td>1.29</td><td>-0.71</td><td>-1.29</td><td>-0.79</td><td>-0.36</td><td>0.93</td></tr><tr><td rowspan="3">Non-test</td><td>Without</td><td>33.40</td><td>7.03</td><td>19.48</td><td>31.38</td><td>37.79</td><td>43.90</td><td>48.10</td><td>56.86</td></tr><tr><td>With</td><td>34.59</td><td>12.97</td><td>21.52</td><td>32.10</td><td>38.21</td><td>44.52</td><td>47.28</td><td>55.59</td></tr><tr><td>Δ</td><td>1.19</td><td>5.93</td><td>2.03</td><td>0.72</td><td>0.41</td><td>0.62</td><td>-0.83</td><td>-1.28</td></tr><tr><td rowspan="6">Roassal</td><td rowspan="3">Overall</td><td>Without</td><td>34.74</td><td>3.16</td><td>15.40</td><td>29.94</td><td>43.68</td><td>51.04</td><td>56.86</td><td>65.10</td></tr><tr><td>With</td><td>35.64</td><td>6.00</td><td>18.54</td><td>31.48</td><td>45.34</td><td>50.98</td><td>54.02</td><td>63.10</td></tr><tr><td>Δ</td><td>0.90</td><td>2.84</td><td>3.14</td><td>1.54</td><td>1.66</td><td>-0.06</td><td>-2.84</td><td>-2.00</td></tr><tr><td rowspan="3">Test</td><td>Without</td><td>32.38</td><td>1.00</td><td>9.69</td><td>26.31</td><td>42.46</td><td>51.00</td><td>56.92</td><td>62.77</td></tr><tr><td>With</td><td>29.77</td><td>1.31</td><td>9.46</td><td>24.62</td><td>41.54</td><td>47.08</td><td>49.85</td><td>57.00</td></tr><tr><td>Δ</td><td>-2.62</td><td>0.31</td><td>-0.23</td><td>-1.69</td><td>-0.92</td><td>-3.92</td><td>-7.08</td><td>-5.77</td></tr><tr><td rowspan="3">Non-test</td><td>Without</td><td>35.57</td><td>3.92</td><td>17.41</td><td>31.22</td><td>44.11</td><td>51.05</td><td>56.84</td><td>65.92</td></tr><tr><td>With</td><td>37.71</td><td>7.65</td><td>21.73</td><td>33.89</td><td>46.68</td><td>52.35</td><td>55.49</td><td>65.24</td></tr><tr><td>Δ</td><td>2.14</td><td>3.73</td><td>4.32</td><td>2.68</td><td>2.57</td><td>1.30</td><td>-1.35</td><td>-0.68</td></tr><tr><td rowspan="6">Seaside</td><td rowspan="3">Overall</td><td>Without</td><td>37.72</td><td>6.28</td><td>16.75</td><td>32.38</td><td>43.53</td><td>51.28</td><td>62.25</td><td>66.94</td></tr><tr><td>With</td><td>42.44</td><td>9.41</td><td>24.38</td><td>37.66</td><td>49.16</td><td>55.03</td><td>65.59</td><td>69.94</td></tr><tr><td>Δ</td><td>4.72</td><td>3.13</td><td>7.62</td><td>5.28</td><td>5.63</td><td>3.75</td><td>3.34</td><td>3.00</td></tr><tr><td rowspan="3">Test</td><td>Without</td><td>34.55</td><td>7.27</td><td>17.27</td><td>31.45</td><td>41.18</td><td>47.09</td><td>56.82</td><td>58.09</td></tr><tr><td>With</td><td>38.09</td><td>9.55</td><td>20.64</td><td>35.91</td><td>45.91</td><td>49.91</td><td>58.73</td><td>60.45</td></tr><tr><td>Δ</td><td>3.55</td><td>2.27</td><td>3.36</td><td>4.45</td><td>4.73</td><td>2.82</td><td>1.91</td><td>2.36</td></tr><tr><td rowspan="3">Non-test</td><td>Without</td><td>39.38</td><td>5.76</td><td>16.48</td><td>32.86</td><td>44.76</td><td>53.48</td><td>65.10</td><td>71.57</td></tr><tr><td>With</td><td>44.71</td><td>9.33</td><td>26.33</td><td>38.57</td><td>50.86</td><td>57.71</td><td>69.19</td><td>74.90</td></tr><tr><td>Δ</td><td>5.33</td><td>3.57</td><td>9.86</td><td>5.71</td><td>6.10</td><td>4.24</td><td>4.10</td><td>3.33</td></tr><tr><td rowspan="6">Spec</td><td rowspan="3">Overall</td><td>Without</td><td>29.43</td><td>3.87</td><td>12.48</td><td>23.16</td><td>34.06</td><td>39.03</td><td>48.35</td><td>57.10</td></tr><tr><td>With</td><td>37.02</td><td>7.74</td><td>21.58</td><td>33.68</td><td>41.87</td><td>47.87</td><td>56.06</td><td>61.71</td></tr><tr><td>Δ</td><td>7.59</td><td>3.87</td><td>9.10</td><td>10.52</td><td>7.81</td><td>8.84</td><td>7.71</td><td>4.61</td></tr><tr><td rowspan="3">Test</td><td>Without</td><td>32.00</td><td>2.55</td><td>14.36</td><td>22.36</td><td>38.09</td><td>43.36</td><td>54.82</td><td>65.36</td></tr><tr><td>With</td><td>34.64</td><td>4.55</td><td>19.36</td><td>28.73</td><td>40.82</td><td>46.18</td><td>56.36</td><td>61.91</td></tr><tr><td>Δ</td><td>2.64</td><td>2.00</td><td>5.00</td><td>6.36</td><td>2.73</td><td>2.82</td><td>1.55</td><td>-3.45</td></tr><tr><td rowspan="3">Non-test</td><td>Without</td><td>28.02</td><td>4.60</td><td>11.45</td><td>23.60</td><td>31.85</td><td>36.65</td><td>44.80</td><td>52.55</td></tr><tr><td>With</td><td>38.32</td><td>9.50</td><td>22.80</td><td>36.40</td><td>42.45</td><td>48.80</td><td>55.90</td><td>61.60</td></tr><tr><td>Δ</td><td>10.31</td><td>4.90</td><td>11.35</td><td>12.80</td><td>10.60</td><td>12.15</td><td>11.10</td><td>9.05</td></tr></tbody></table>

MRR Performance Comparison between Default and Package-Aware across

edge cases where similarly named global variables (e.g., IceSBBrowserAbstractMethodCommand vs. IceSBBrowseFullMethodCommand) resulted in ambiguous completions. Addressing these cases may require additional heuristics or statistical models trained on historical usage data, highlighting an avenue for future work.

## 5.2. Challenges

In the appendix 8, detailed benchmarks for each evaluated framework are provided. It is important to note that, when computing averages and deltas presented in Table 2, zero values were omitted. A zero value typically represents cases where a package contains only one class with methods that are either trivial, purely for testing, or abstract (marked by self subclassResponsibility) with concrete implementations residing outside the package. Additionally, some packages might only contain extensions without complete implementations.

The occurrence of unchanged or marginal delta values (e.g., values close to 1) in MRR is often due to methods or variables sharing lengthy prefixes, such as IceSBB browserAbstractMethodCommand and IceSBBrowseFullMethodCommand. Given that our current benchmarking evaluates prefixes limited to 8 characters, differences beyond this length remain undetected. This represents a significant limitation of our evaluation method.

To overcome this limitation, future research should explore more sophisticated completion strategies. For instance, completing only the common prefix and then progressively refining the completion.
Consider an example from the Moose framework: typing the letter 'M' could immediately propose prefixes like 'Moose...', 'MooseMSE...', and 'MooseMSEImporter...'. Pressing 'Tab' after selecting a prefix would insert it directly without adding extra spaces, enabling seamless continued typing. Further keystrokes could then complete subsequent portions of the name (e.g., typing 'I' to complete 'Importer'). Such an approach could significantly reduce typing effort, allowing a long name like MooseMSEImporterTestEntity to be entered with just a few keystrokes ('M' - 'Tab' completes 'MooseMSE', then 'I' - 'Tab' completes 'ImporterTestEntity'). Integrating this prefix-driven completion strategy into future benchmarks will provide deeper insights into code completion effectiveness, especially in projects characterized by extensive naming conventions.

# 6. Limitations

A key limitation of our evaluation is its dependence on static references extracted from existing code. We simulate completion sites by truncating identifier names to 2-8 characters and then measuring whether our approach ranks the correct name near the top of the suggestion list. Although this technique is standard in code-completion research, it does not fully capture how developers behave in live sessions. Results could also differ when analyzing older repositories or external libraries that diverge from Pharo's usual package conventions.

Additionally, an important assumption underlying our heuristics is that global variables or classes are more likely to be referenced within the same package where they are defined, rather than in other packages within the same project. This assumption justifies our strategy of prioritizing classes first from the same package, followed by "friend" packages, and finally external packages. However, this assumption has not been empirically validated, and there are multiple scenarios where it may not hold true. For instance, test packages rarely reference classes from the same package but frequently reference classes from other packages within the same project. Similarly, classes defined in core packages or packages implementing design patterns such as visitors, exceptions, or commands often exhibit fewer references within their own packages but are extensively referenced by others.

To address this potential threat to validity, future work should investigate the actual degree of self-referencing within packages. If self-referencing is found to be low, prioritizing classes from the same package could negatively impact completion accuracy, as indicated by our results in Table 2 for test packages. Thus, it might be beneficial to prioritize classes from the same project rather than strictly from the same package, potentially improving completion suggestions, particularly in cases such as test packages.

Finally, very long prefixes, such as `IceSBBrowserAbstractMethodCommand` and `IceSBBrowseFull-MethodCommand` remain challenging because our current methodology considers only short prefixes. Future work should extend the analysis to longer prefixes and a broader range of code bases, thereby providing a more complete picture of completion behavior and ultimately yielding more accurate, context-aware suggestions for developers.

# 7. Related Works

Code completion has evolved from simple syntactic suggestions to sophisticated, context-sensitive systems powered by statistical, neural, and usability-aware methods. Early work such as Robbes et al. [RL08a] introduced context-sensitive filtering based on recent usage and program history. Shortly afterward, Bruch et al., [BMM09] and Hou et al., [HP10] emphasized syntactic similarity and ranking heuristics, relying on type hierarchies, usage popularity, and manual grouping. These heuristic and rule-based approaches established foundational techniques for example-based ranking and structural filtering still used in IDE plugins today.

**Statistical Approaches** Hindle et al., [HBS+12] demonstrated that software code exhibits significant regularities that can be effectively captured using statistical language models, specifically through
n-gram modeling. Statistical methods soon emerged, bringing greater accuracy through learning from large corpora. Nguyen et al., [NNNN13] introduced SLAMC, combining n-gram models with semantic roles and topic modeling. Raychev et al., contributed SLANG [RVY14], a statistical language model synthesizing code completions. Proksch et al., [PLM15] introduced Bayesian models offering compact and accurate recommendations via probabilistic reasoning over API usage patterns. Nguyen et al., [NHC+16] further advanced statistical methods with APIREC, learning fine-grained API usage patterns. Raychev et al., subsequently presented DEEP3 [RBV16], employing decision-tree-based generative models.

**Learning-Based Approaches** Deep neural architectures were increasingly adopted for code completion. Bielik et al., [BRV16] proposed PHOG, a grammar-aware generative model using probabilistic higher-order rules. Jin et al., [JS18] highlighted usability concerns by addressing the hidden costs of extensive suggestion lists. Hellendoorn et al., [HPGB19] provided a significant empirical analysis revealing the limitations of models in practical intra-project completions. Svyatkovskiy et al., [SLH+21] introduced a modular neural framework for code completion, leveraging static analysis and granular token encodings to design a memory-efficient reranking model with high predictive performance. Karampatsis et al., [KBR+20] introduced open-vocabulary neural models with Byte-Pair Encoding (BPE), effectively managing out-of-vocabulary issues. Matani et al., [Mat21] proposed an efficient segment-tree-based solution for prefix-based completion without the need for statistical training. Popov et al., [POL+21] delivered a practical, time-efficient GPT-2 variant for R code completion, achieving high accuracy within strict latency constraints. Li et al., [LHL+21] proposed benefit-cost-aware metrics to filter and reorder completions, significantly reducing irrelevant suggestions.

**Recent contributions** Recent contributions emphasize real-time usability, contextual awareness, and cross-language generalization. Bibaev et al., [BKL+22] advanced towards practical systems by training rankers using real IDE usage logs, and personalizing suggestions to reduce developer keystroke effort. Takerngsaksiri et al., [TTL24] introduced PyCoder, a syntax-aware transformer-based model predicting token types without explicit AST parsing. Wang et al., [WZL+25] proposed TIGER, a generate-then-rank approach using lightweight transformers for Python type inference, demonstrating state-of-the-art performance. Modern code completion systems now integrate structural analysis, statistical learning, and deep neural modeling, consistently targeting developer productivity through usability-driven metrics. Despite these advancements, significant challenges remain, especially in handling difficult intra-project completions and minimizing cognitive load. Future developments must balance sophistication, runtime efficiency, and developer experience, increasingly turning to hybrid models and log-informed personalization.

# 8. Conclusion and future works

COMPLISHON employs multiple semantic heuristics to produce context-aware completions in a live programming environment. However, it originally treated the system as a flat global space, limiting the precision of suggestions for large projects. In this paper, we proposed and evaluated a package-awareness completion heuristic to mitigate this issue. Our approach prioritizes local package entities, followed by similarly prefixed or related packages, and finally falls back to the global namespace. The results show that package-awareness completions indeed improve the ranking of relevant suggestions in some cases, especially when a package's references remain largely local. Conversely, packages with major cross-package dependencies, particularly testing packages, can perform worse with naive prefix-based ordering, highlighting the need to explicitly consider package dependencies and usage patterns.

We plan to integrate more advanced dependency-aware scoping, so that COMPLISHON understands not just one's immediate package but also any dependencies or related packages. We will also explore hybrid approaches that combine lightweight statistical frequency analysis with structural heuristics to
handle complex referencing patterns. By evolving in this direction, we aim to make COMPLISHON an increasingly effective code-completion system for large, modular software projects.

**Acknowledgments.** We thank Inria and the LLM4Code défi for the funding of the first author.

Declaration on Generative AI

The authors have not employed any Generative AI tools.

References

[AEH<sup>+</sup>20] Nicolas Anquetil, Anne Etien, Mahugnon Honoré Houekpetodji, Benoît Verhaeghe, Stéphane Ducasse, Clotilde Toullec, Fatija Djareddir, Jérôme Sudich, and Mustapha Derras. Modular Moose: A new generation of software reengineering platform. In *International Conference on Software and Systems Reuse (ICSR’20)*, number 12541 in LNCS, pages 119–134, December 2020.

[BDN<sup>+</sup>09] Andrew P. Black, Stéphane Ducasse, Oscar Nierstrasz, Damien Pollet, Damien Cassou, and Marcus Denker. *Pharo by Example*. Square Bracket Associates, Kehrsatz, Switzerland, 2009.

[BDR08] Alexandre Bergel, Stéphane Ducasse, and Lukas Renggli. Seaside – advanced composition and control flow for dynamic web applications. *ERCIM News*, 72, January 2008.

[Ber16] Alexandre Bergel. *Agile Visualization*. LULU Press, 2016.

[BKL<sup>+</sup>22] Vitaliy Bibaev, Alexey Kalina, Vadim Lomshakov, Yaroslav Golubev, Alexander Bezzubov, Nikita Povarov, and Timofey Bryksin. All you need is logs: Improving code completion by learning from anonymous IDE usage logs, 2022.

[BMM09] Marcel Bruch, Martin Monperrus, and Mira Mezini. Learning from examples to improve code completion systems. In *Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering*, pages 213–222, 2009.

[BRV16] Pavol Bielik, Veselin Raychev, and Martin Vechev. PHOG: Probabilistic model for code. In *International Conference on Machine Learning*, 2016.

[CNG15] Andrei Chiş, Oscar Nierstrasz, and Tudor Gîrba. Towards moldable development tools. In *Workshop on Evaluation and Usability of Programming Languages and Tools*, 2015.

[DHDJML24] Koen De Hondt, Séphane Ducasse, Sebastian Jordan Montano, and Esteban Lorenzano. *Application Building with Spec 2.0*. Book on Demand – Keepers of the lighthouse, 2024.

[DLR07] Stéphane Ducasse, Adrian Lienhard, and Lukas Renggli. Seaside: A flexible environment for building dynamic web applications. *IEEE Software*, 24(5):56–63, 2007.

[DRS<sup>+</sup>10] Stéphane Ducasse, Lukas Renggli, C. David Shaffer, Rick Zaccone, and Michael Davies. *Dynamic Web Development with Seaside*. Square Bracket Associates, 2010.

[GHJV95] Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. *Design Patterns: Elements of Reusable Object-Oriented Software*. Addison-Wesley, 1995.

[HBS<sup>+</sup>12] Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. On the naturalness of software. In *Software Engineering (ICSE), 2012 34th International Conference on*, pages 837–847. IEEE, 2012.

[HP10] Daqing Hou and David M. Pletcher. Towards a better code completion system by api grouping, filtering, and popularity-based ranking. In *International Workshop on Recommendation Systems for Software Engineering (RSSE)*, 2010.

[HPGB19] Vincent J Hellendoorn, Sebastian Proksch, Harald C Gall, and Alberto Bacchelli. When code completion fails: A case study on real-world completions. In *2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)*, pages 960–970. IEEE, 2019.
[JH25] Hangzhan Jin and Mohammad Hamdaqa. Ccci: Code completion with contextual information for complex data transfer tasks using large language models. arXiv, 2025.

[JS18] Xianhao Jin and Francisco Servant. The hidden cost of code completion: understanding the impact of the recommendation-list length on its efficiency. In *International Conference on Mining Software Repositories*, 2018.

[KBR<sup>+</sup>20] Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. Big code != big vocabulary: open-vocabulary models for source code. In *International Conference on Software Engineering (ICSE)*, 2020.

[LHL<sup>+</sup>21] Jingxuan Li, Rui Huang, Wei Li, Kai Yao, and Weiguo Tan. Toward Less Hidden Cost of Code Completion with Acceptance and Ranking Models. In *International Conference on Software Maintenance and Evolution (ICSME)*, 2021.

[Mat21] Dhruv Matani. An $o(k \log{n})$ algorithm for prefix based ranked autocomplete. arXiv, 2021.

[MC18] Bhaskar Miutra and Nick Craswell. An introduction to neural information retrieval. *Foundations and Trends in Information Retrieval*, 2018.

[NDG05] Oscar Nierstrasz, Stéphane Ducasse, and Tudor Gîrba. The story of Moose: an agile reengineering environment. In Michel Wermelinger and Harald Gall, editors, *Proceedings of the European Software Engineering Conference*, ESEC/FSE'05, pages 1-10, New York NY, 2005. ACM Press. Invited paper.

[NHC<sup>+</sup>16] Anh Tuan Nguyen, Michael Hilton, Mihai Codoban, Hoan Anh Nguyen, Lily Mast, Eli Rademacher, Tien N. Nguyen, and Danny Dig. Api code recommendation using statistical learning from fine-grained changes. In *International Symposium on Foundations of Software Engineering*, 2016.

[NNNN13] Tung Thanh Nguyen, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N Nguyen. A statistical semantic language model for source code. In *Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering*, pages 532-542, 2013.

[PDO20] Guillermo Polito, Stéphane Ducasse, and Allex Oliveira. Manage your code with git and iceberg, 2020.

[PLM15] Sebastian Proksch, Johannes Lerch, and Mira Mezini. Intelligent code completion with bayesian networks. *Transactions on Software Engineering and Methodology (TOSEM)*, 1(25), 2015.

[POL<sup>+</sup>21] Artem Popov, Dmitrii Orekhov, Denis Litvinov, Nikolay Korolev, and Gleb Morgachev. Time-efficient code completion model for the r programming language. In *Workshop on Natural Language Processing for Programming*, 2021.

[RBV16] Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. In *International Conference on Object-Oriented Programming, Systems, Languages, and Applications*, 2016.

[RL08a] Romain Robbes and Michele Lanza. How program history can improve code completion. In *Proceedings of ASE 2008 (23rd International Conference on Automated Software Engineering)*, pages 317-326, 2008.

[RL08b] Romain Robbes and Michele Lanza. SpyWare: a change-aware development toolset. In *Proceedings of the 30th International Conference on Software Engineering*, ICSE'08, pages 847-850, New York, NY, USA, 2008. ACM.

[RL10] Romain Robbes and Michele Lanza. Improving code completion with program history. *Journal of Automated Software Engineering*, 17(2):181-212, 2010.

[RVY14] Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In *Acm Sigplan Notices*, volume 49, pages 419-428. ACM, 2014.

[SLH<sup>+</sup>21] Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Franco, and Miltiadis Allamanis. Fast and memory-efficient neural code completion. In *International Conference on Mining Software Repositories (MSR)*, 2021.

[TTL24] Wannita Takerngsaksiri, Chakkrit Tantithamthavorn, and Yuan-Fang Li. Syntax-aware on-the-fly code completion. *Information and Software Technology*, 2024.
[WZL$^{+}$25] Chong Wang, Jian Zhang, Yiling Lou, Mingwei Liu, Weisong Sun, Yang Liu, and Xin Peng. TIGER: A Generating-Then-Ranking Framework for Practical Python Type Inference. In *International Conference on Software Engineering (ICSE)*, 2025.

[ZDA20] Oleksandr Zaitsev, Stéphane Ducasse, and Nicolas Anquetil. Characterizing pharo code: A technical report. Technical report, Inria Lille Nord Europe - Laboratoire CRISTAL - Université de Lille ; Arolla, January 2020.
## A. Project Descriptions

**Spec.** Spec is a modern user interface framework integrated into Pharo. It adopts a modular architecture centered around "presenters," which allows developers to efficiently compose, nest, and manage interactive UI elements [DHDJML24]. Spec2 facilitates dynamic interface layouts, enabling real-time modifications without the need for extensive interface rebuilds, thus enhancing adaptability and responsiveness. Additionally, it supports multiple rendering backends, including Morphic and GTK+3, providing flexibility for cross-platform application development. Spec2's design promotes streamlined communication between components, significantly simplifying interaction handling and improving maintainability.

**Roassal.** Roassal is a lightweight and extensible visualization engine developed in Pharo, designed to facilitate agile and interactive data visualization [Ber16]. It provides a rich set of graphical primitives and layout algorithms, enabling developers to craft domain-specific visualizations with minimal effort. Roassal supports interactive features such as zooming, dragging, and tooltips, allowing users to explore complex data structures dynamically. Its integration with the Pharo environment allows for seamless development and immediate feedback, making it an effective tool for both exploratory data analysis and the development of custom visualization tools.

**Seaside.** Seaside is a web application framework for Smalltalk, particularly well-integrated with Pharo, that enables the development of complex web applications through a component-based architecture [DLR07, DRS+10]. Unlike traditional web frameworks that rely on templates, Seaside allows developers to build web pages by composing stateful components, each encapsulating its rendering and behavior. It leverages continuations to manage control flow, facilitating the creation of sophisticated user interactions and workflows. Seaside's approach promotes code reuse and modularity, and its tight integration with the Pharo development environment allows for live debugging and real-time updates, enhancing developer productivity [BDR08].

**Iceberg.** Iceberg is the primary version control system (VCS) integration tool within the Pharo environment, providing a seamless interface to Git repositories. It enables developers to perform standard Git operations such as cloning, committing, branching, merging, and pushing directly from the Pharo image, eliminating the need for external command-line tools. Designed to bridge the gap between Pharo's live object model and Git's file-based architecture, Iceberg ensures that changes made within the Pharo environment are accurately reflected in the Git repository. This integration facilitates efficient management of code versions, supports collaborative development workflows, and simplifies the process of contributing to and maintaining Pharo-based projects [PDO20].

**Moose.** Moose is an open-source platform for software and data analysis, developed in Pharo, that enables analysts to construct custom analysis tools and workflows [AEH+20]. It offers services such as data importation, modeling, measurement, querying, mining, and the development of interactive visual analysis tools. Moose supports the creation of dedicated analysis tools and the customization of analysis processes. It provides mechanisms for importing and meta-modeling through a generic meta-described engine, parsing using various technologies, and visualization via graph and chart engines. The platform is primarily used for software analysis but is designed to handle various types of data. Moose is based on Pharo and is open-source under BSD/MIT licenses [NDG05].
## B. Seaside

<table><caption>Table 3: Performance of Seaside Framework</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td>Seaside-Ajaxifier-Core</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Ajaxifier-Core</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Canvas</td><td>without</td><td>0.48</td><td>0.04</td><td>0.17</td><td>0.44</td><td>0.6</td><td>0.67</td><td>0.72</td><td>0.8</td></tr><tr><td>Seaside-Canvas</td><td>with</td><td>0.55</td><td>0.05</td><td>0.31</td><td>0.57</td><td>0.68</td><td>0.72</td><td>0.77</td><td>0.85</td></tr><tr><td>Seaside-Component</td><td>without</td><td>0.40</td><td>0.02</td><td>0.12</td><td>0.39</td><td>0.48</td><td>0.58</td><td>0.64</td><td>0.74</td></tr><tr><td>Seaside-Component</td><td>with</td><td>0.60</td><td>0.16</td><td>0.5</td><td>0.62</td><td>0.69</td><td>0.77</td><td>0.77</td><td>0.85</td></tr><tr><td>Seaside-Continuation</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Continuation</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Core</td><td>without</td><td>0.31</td><td>0.03</td><td>0.12</td><td>0.29</td><td>0.4</td><td>0.43</td><td>0.49</td><td>0.53</td></tr><tr><td>Seaside-Core</td><td>with</td><td>0.42</td><td>0.04</td><td>0.24</td><td>0.43</td><td>0.53</td><td>0.55</td><td>0.63</td><td>0.67</td></tr><tr><td>Seaside-Development</td><td>without</td><td>0.36</td><td>0.02</td><td>0.09</td><td>0.27</td><td>0.41</td><td>0.45</td><td>0.67</td><td>0.75</td></tr><tr><td>Seaside-Development</td><td>with</td><td>0.45</td><td>0.08</td><td>0.28</td><td>0.41</td><td>0.51</td><td>0.53</td><td>0.72</td><td>0.79</td></tr><tr><td>Seaside-Email</td><td>without</td><td>0.29</td><td>0.05</td><td>0.15</td><td>0.26</td><td>0.29</td><td>0.38</td><td>0.46</td><td>0.52</td></tr><tr><td>Seaside-Email</td><td>with</td><td>0.37</td><td>0.08</td><td>0.3</td><td>0.41</td><td>0.42</td><td>0.42</td><td>0.5</td><td>0.54</td></tr><tr><td>Seaside-Environment</td><td>without</td><td>0.26</td><td>0.0</td><td>0.0</td><td>0.06</td><td>0.1</td><td>0.1</td><td>0.79</td><td>0.79</td></tr><tr><td>Seaside-Environment</td><td>with</td><td>0.34</td><td>0.0</td><td>0.07</td><td>0.22</td><td>0.22</td><td>0.22</td><td>0.85</td><td>0.85</td></tr><tr><td>Seaside-Examples</td><td>without</td><td>0.32</td><td>0.03</td><td>0.1</td><td>0.21</td><td>0.36</td><td>0.38</td><td>0.66</td><td>0.69</td></tr><tr><td>Seaside-Examples</td><td>with</td><td>0.38</td><td>0.07</td><td>0.29</td><td>0.33</td><td>0.36</td><td>0.37</td><td>0.67</td><td>0.71</td></tr><tr><td>Seaside-Flow</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Flow</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-JSON-Core</td><td>without</td><td>0.35</td><td>0.09</td><td>0.22</td><td>0.33</td><td>0.35</td><td>0.36</td><td>0.61</td><td>0.61</td></tr><tr><td>Seaside-JSON-Core</td><td>with</td><td>0.34</td><td>0.1</td><td>0.2</td><td>0.31</td><td>0.33</td><td>0.34</td><td>0.61</td><td>0.61</td></tr><tr><td>Seaside-Pharo-Canvas</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo-Canvas</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo-Continuation</td><td>without</td><td>0.40</td><td>0.07</td><td>0.32</td><td>0.38</td><td>0.45</td><td>0.58</td><td>0.58</td><td>0.58</td></tr><tr><td>Seaside-Pharo-Continuation</td><td>with</td><td>0.44</td><td>0.22</td><td>0.38</td><td>0.4</td><td>0.45</td><td>0.58</td><td>0.58</td><td>0.58</td></tr><tr><td>Seaside-Pharo-Core</td><td>without</td><td>0.53</td><td>0.14</td><td>0.14</td><td>0.14</td><td>0.33</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Seaside-Pharo-Core</td><td>with</td><td>0.53</td><td>0.14</td><td>0.14</td><td>0.14</td><td>0.33</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Seaside-Pharo-Development</td><td>without</td><td>0.43</td><td>0.08</td><td>0.13</td><td>0.18</td><td>0.59</td><td>0.64</td><td>0.74</td><td>0.83</td></tr><tr><td>Seaside-Pharo-Development</td><td>with</td><td>0.42</td><td>0.09</td><td>0.16</td><td>0.19</td><td>0.58</td><td>0.62</td><td>0.71</td><td>0.79</td></tr><tr><td>Seaside-Pharo-Email</td><td>without</td><td>0.28</td><td>0.0</td><td>0.0</td><td>0.13</td><td>0.21</td><td>0.43</td><td>0.57</td><td>1.0</td></tr><tr><td>Seaside-Pharo-Email</td><td>with</td><td>0.33</td><td>0.0</td><td>0.0</td><td>0.15</td><td>0.45</td><td>0.45</td><td>0.6</td><td>1.0</td></tr><tr><td>Seaside-Pharo-Environment</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo-Environment</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo-Flow</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo-Flow</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo-JSON-Core</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo-JSON-Core</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo-Tools-Web</td><td>without</td><td>0.52</td><td>0.22</td><td>0.41</td><td>0.49</td><td>0.57</td><td>0.64</td><td>0.69</td><td>0.7</td></tr><tr><td>Seaside-Pharo-Tools-Web</td><td>with</td><td>0.53</td><td>0.22</td><td>0.44</td><td>0.51</td><td>0.59</td><td>0.65</td><td>0.71</td><td>0.72</td></tr><tr><td>Seaside-Pharo-Welcome</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo-Welcome</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo100-Tools-Spec2</td><td>without</td><td>0.49</td><td>0.03</td><td>0.27</td><td>0.42</td><td>0.68</td><td>0.68</td><td>0.68</td><td>0.76</td></tr><tr><td>Seaside-Pharo100-Tools-Spec2</td><td>with</td><td>0.50</td><td>0.03</td><td>0.27</td><td>0.34</td><td>0.72</td><td>0.72</td><td>0.72</td><td>0.81</td></tr><tr><td>Seaside-Pharo90-REST-Core</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Pharo90-REST-Core</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-REST-Core</td><td>without</td><td>0.50</td><td>0.1</td><td>0.23</td><td>0.52</td><td>0.62</td><td>0.65</td><td>0.67</td><td>0.76</td></tr><tr><td>Seaside-REST-Core</td><td>with</td><td>0.60</td><td>0.19</td><td>0.53</td><td>0.68</td><td>0.68</td><td>0.69</td><td>0.72</td><td>0.79</td></tr><tr><td>Seaside-RenderLoop</td><td>without</td><td>0.43</td><td>0.01</td><td>0.14</td><td>0.43</td><td>0.54</td><td>0.62</td><td>0.68</td><td>0.7</td></tr><tr><td>Seaside-RenderLoop</td><td>with</td><td>0.44</td><td>0.06</td><td>0.2</td><td>0.36</td><td>0.49</td><td>0.57</td><td>0.72</td><td>0.75</td></tr><tr><td>Seaside-Session</td><td>without</td><td>0.39</td><td>0.01</td><td>0.14</td><td>0.4</td><td>0.55</td><td>0.56</td><td>0.56</td><td>0.6</td></tr><tr><td>Seaside-Session</td><td>with</td><td>0.45</td><td>0.02</td><td>0.17</td><td>0.44</td><td>0.6</td><td>0.63</td><td>0.67</td><td>0.7</td></tr><tr><td>Seaside-Tests-Canvas</td><td>without</td><td>0.25</td><td>0.02</td><td>0.07</td><td>0.13</td><td>0.22</td><td>0.23</td><td>0.64</td><td>0.68</td></tr><tr><td>Seaside-Tests-Canvas</td><td>with</td><td>0.30</td><td>0.03</td><td>0.14</td><td>0.2</td><td>0.29</td><td>0.29</td><td>0.64</td><td>0.69</td></tr><tr><td>Seaside-Tests-Component</td><td>without</td><td>0.38</td><td>0.03</td><td>0.15</td><td>0.32</td><td>0.43</td><td>0.62</td><td>0.69</td><td>0.73</td></tr><tr><td>Seaside-Tests-Component</td><td>with</td><td>0.38</td><td>0.1</td><td>0.23</td><td>0.33</td><td>0.43</td><td>0.52</td><td>0.61</td><td>0.66</td></tr><tr><td>Seaside-Tests-Core</td><td>without</td><td>0.27</td><td>0.02</td><td>0.1</td><td>0.24</td><td>0.37</td><td>0.38</td><td>0.45</td><td>0.46</td></tr><tr><td>Seaside-Tests-Core</td><td>with</td><td>0.27</td><td>0.02</td><td>0.09</td><td>0.26</td><td>0.35</td><td>0.38</td><td>0.45</td><td>0.46</td></tr><tr><td>Seaside-Tests-Environment</td><td>without</td><td>0.32</td><td>0.01</td><td>0.09</td><td>0.31</td><td>0.42</td><td>0.44</td><td>0.56</td><td>0.5</td></tr><tr><td>Seaside-Tests-Environment</td><td>with</td><td>0.38</td><td>0.01</td><td>0.15</td><td>0.35</td><td>0.48</td><td>0.55</td><td>0.67</td><td>0.62</td></tr><tr><td>Seaside-Tests-Flow</td><td>without</td><td>0.40</td><td>0.02</td><td>0.13</td><td>0.46</td><td>0.53</td><td>0.57</td><td>0.61</td><td>0.6</td></tr><tr><td>Seaside-Tests-Flow</td><td>with</td><td>0.39</td><td>0.04</td><td>0.13</td><td>0.35</td><td>0.54</td><td>0.58</td><td>0.61</td><td>0.61</td></tr><tr><td>Seaside-Tests-Functional</td><td>without</td><td>0.37</td><td>0.1</td><td>0.2</td><td>0.32</td><td>0.44</td><td>0.48</td><td>0.58</td><td>0.7</td></tr><tr><td>Seaside-Tests-Functional</td><td>with</td><td>0.39</td><td>0.1</td><td>0.24</td><td>0.33</td><td>0.44</td><td>0.48</td><td>0.6</td><td>0.7</td></tr><tr><td>Seaside-Tests-Pharo-Canvas</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr></tbody></table>

Continued on next page
<table><caption>Table 3 – continued from previous page</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td>Seaside-Tests-Pharo-Canvas</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Tests-Pharo-Continuation</td><td>without</td><td>0.46</td><td>0.14</td><td>0.42</td><td>0.44</td><td>0.45</td><td>0.71</td><td>0.75</td><td>0.75</td></tr><tr><td>Seaside-Tests-Pharo-Continuation</td><td>with</td><td>0.32</td><td>0.14</td><td>0.22</td><td>0.29</td><td>0.4</td><td>0.5</td><td>0.5</td><td>0.5</td></tr><tr><td>Seaside-Tests-Pharo-Core</td><td>without</td><td>0.32</td><td>0.12</td><td>0.19</td><td>0.29</td><td>0.37</td><td>0.38</td><td>0.48</td><td>0.55</td></tr><tr><td>Seaside-Tests-Pharo-Core</td><td>with</td><td>0.32</td><td>0.12</td><td>0.17</td><td>0.3</td><td>0.37</td><td>0.38</td><td>0.47</td><td>0.55</td></tr><tr><td>Seaside-Tests-Pharo-Functional</td><td>without</td><td>0.46</td><td>0.33</td><td>0.37</td><td>0.51</td><td>0.51</td><td>0.51</td><td>0.51</td><td>0.51</td></tr><tr><td>Seaside-Tests-Pharo-Functional</td><td>with</td><td>0.69</td><td>0.33</td><td>0.38</td><td>0.83</td><td>0.83</td><td>0.83</td><td>0.83</td><td>0.83</td></tr><tr><td>Seaside-Tests-RenderLoop</td><td>without</td><td>0.26</td><td>0.0</td><td>0.03</td><td>0.22</td><td>0.35</td><td>0.37</td><td>0.49</td><td>0.42</td></tr><tr><td>Seaside-Tests-RenderLoop</td><td>with</td><td>0.49</td><td>0.15</td><td>0.4</td><td>0.52</td><td>0.56</td><td>0.56</td><td>0.66</td><td>0.61</td></tr><tr><td>Seaside-Tests-Session</td><td>without</td><td>0.31</td><td>0.01</td><td>0.15</td><td>0.22</td><td>0.44</td><td>0.49</td><td>0.49</td><td>0.49</td></tr><tr><td>Seaside-Tests-Session</td><td>with</td><td>0.26</td><td>0.01</td><td>0.12</td><td>0.19</td><td>0.36</td><td>0.42</td><td>0.42</td><td>0.42</td></tr><tr><td>Seaside-Tools-Core</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Tools-Core</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Tools-Web</td><td>without</td><td>0.40</td><td>0.04</td><td>0.13</td><td>0.35</td><td>0.49</td><td>0.58</td><td>0.68</td><td>0.72</td></tr><tr><td>Seaside-Tools-Web</td><td>with</td><td>0.49</td><td>0.08</td><td>0.3</td><td>0.45</td><td>0.59</td><td>0.64</td><td>0.73</td><td>0.76</td></tr><tr><td>Seaside-Welcome</td><td>without</td><td>0.38</td><td>0.04</td><td>0.15</td><td>0.47</td><td>0.5</td><td>0.53</td><td>0.58</td><td>0.5</td></tr><tr><td>Seaside-Welcome</td><td>with</td><td>0.35</td><td>0.09</td><td>0.16</td><td>0.32</td><td>0.49</td><td>0.51</td><td>0.54</td><td>0.45</td></tr><tr><td>Seaside-Widgets</td><td>without</td><td>0.36</td><td>0.17</td><td>0.26</td><td>0.38</td><td>0.36</td><td>0.43</td><td>0.57</td><td>0.81</td></tr><tr><td>Seaside-Widgets</td><td>with</td><td>0.44</td><td>0.19</td><td>0.37</td><td>0.42</td><td>0.43</td><td>0.58</td><td>0.68</td><td>0.87</td></tr><tr><td>Seaside-Zinc-Core</td><td>without</td><td>0.39</td><td>0.02</td><td>0.17</td><td>0.36</td><td>0.52</td><td>0.54</td><td>0.63</td><td>0.64</td></tr><tr><td>Seaside-Zinc-Core</td><td>with</td><td>0.42</td><td>0.05</td><td>0.22</td><td>0.4</td><td>0.54</td><td>0.56</td><td>0.63</td><td>0.64</td></tr><tr><td>Seaside-Zinc-Pharo</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Seaside-Zinc-Pharo</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr></tbody></table>
C. Spec

<table><caption>Table 4: Performance of Spec Framework</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td>Spec2-Adapters-Morphic</td><td>without</td><td>0.38</td><td>0.04</td><td>0.18</td><td>0.31</td><td>0.43</td><td>0.59</td><td>0.62</td><td>0.66</td></tr><tr><td>Spec2-Adapters-Morphic</td><td>with</td><td>0.45</td><td>0.05</td><td>0.29</td><td>0.39</td><td>0.5</td><td>0.65</td><td>0.69</td><td>0.72</td></tr><tr><td>Spec2-Adapters-Morphic-LISTView</td><td>without</td><td>0.31</td><td>0.0</td><td>0.05</td><td>0.21</td><td>0.46</td><td>0.49</td><td>0.49</td><td>0.49</td></tr><tr><td>Spec2-Adapters-Morphic-LISTView</td><td>with</td><td>0.40</td><td>0.09</td><td>0.15</td><td>0.28</td><td>0.58</td><td>0.58</td><td>0.58</td><td>0.58</td></tr><tr><td>Spec2-Adapters-Morphic-Tests</td><td>without</td><td>0.29</td><td>0.0</td><td>0.06</td><td>0.2</td><td>0.36</td><td>0.4</td><td>0.41</td><td>0.81</td></tr><tr><td>Spec2-Adapters-Morphic-Tests</td><td>with</td><td>0.22</td><td>0.0</td><td>0.04</td><td>0.15</td><td>0.28</td><td>0.31</td><td>0.33</td><td>0.55</td></tr><tr><td>Spec2-Adapters-Stub</td><td>without</td><td>0.16</td><td>0.0</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.41</td><td>0.64</td></tr><tr><td>Spec2-Adapters-Stub</td><td>with</td><td>0.15</td><td>0.03</td><td>0.03</td><td>0.03</td><td>0.03</td><td>0.03</td><td>0.35</td><td>0.55</td></tr><tr><td>Spec2-Backend-Tests</td><td>without</td><td>0.26</td><td>0.0</td><td>0.04</td><td>0.22</td><td>0.3</td><td>0.36</td><td>0.44</td><td>0.52</td></tr><tr><td>Spec2-Backend-Tests</td><td>with</td><td>0.34</td><td>0.0</td><td>0.12</td><td>0.36</td><td>0.39</td><td>0.46</td><td>0.53</td><td>0.58</td></tr><tr><td>Spec2-Code</td><td>without</td><td>0.26</td><td>0.09</td><td>0.14</td><td>0.2</td><td>0.24</td><td>0.24</td><td>0.49</td><td>0.55</td></tr><tr><td>Spec2-Code</td><td>with</td><td>0.30</td><td>0.1</td><td>0.17</td><td>0.22</td><td>0.25</td><td>0.28</td><td>0.57</td><td>0.6</td></tr><tr><td>Spec2-Code-Backend-Tests</td><td>without</td><td>0.15</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.11</td><td>0.11</td><td>0.37</td><td>0.64</td></tr><tr><td>Spec2-Code-Backend-Tests</td><td>with</td><td>0.35</td><td>0.02</td><td>0.21</td><td>0.21</td><td>0.34</td><td>0.34</td><td>0.72</td><td>0.83</td></tr><tr><td>Spec2-Code-Commands</td><td>without</td><td>0.12</td><td>0.0</td><td>0.02</td><td>0.06</td><td>0.16</td><td>0.17</td><td>0.21</td><td>0.31</td></tr><tr><td>Spec2-Code-Commands</td><td>with</td><td>0.09</td><td>0.0</td><td>0.02</td><td>0.02</td><td>0.12</td><td>0.13</td><td>0.18</td><td>0.21</td></tr><tr><td>Spec2-Code-Diff</td><td>without</td><td>0.45</td><td>0.14</td><td>0.24</td><td>0.25</td><td>0.57</td><td>0.57</td><td>0.81</td><td>0.88</td></tr><tr><td>Spec2-Code-Diff</td><td>with</td><td>0.47</td><td>0.21</td><td>0.3</td><td>0.31</td><td>0.55</td><td>0.55</td><td>0.77</td><td>0.86</td></tr><tr><td>Spec2-Code-Diff-Morphic</td><td>without</td><td>0.31</td><td>0.02</td><td>0.06</td><td>0.18</td><td>0.23</td><td>0.53</td><td>0.65</td><td>0.68</td></tr><tr><td>Spec2-Code-Diff-Morphic</td><td>with</td><td>0.48</td><td>0.17</td><td>0.2</td><td>0.36</td><td>0.45</td><td>0.67</td><td>0.82</td><td>0.86</td></tr><tr><td>Spec2-Code-Diff-Tests</td><td>without</td><td>0.62</td><td>0.07</td><td>0.26</td><td>0.26</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Spec2-Code-Diff-Tests</td><td>with</td><td>0.54</td><td>0.15</td><td>0.29</td><td>0.29</td><td>0.84</td><td>0.8</td><td>0.8</td><td>0.8</td></tr><tr><td>Spec2-Code-Morphic</td><td>without</td><td>0.28</td><td>0.0</td><td>0.27</td><td>0.3</td><td>0.31</td><td>0.36</td><td>0.36</td><td>0.36</td></tr><tr><td>Spec2-Code-Morphic</td><td>with</td><td>0.45</td><td>0.37</td><td>0.42</td><td>0.45</td><td>0.45</td><td>0.51</td><td>0.51</td><td>0.51</td></tr><tr><td>Spec2-Code-Tests</td><td>without</td><td>0.27</td><td>0.03</td><td>0.06</td><td>0.14</td><td>0.21</td><td>0.21</td><td>0.59</td><td>0.78</td></tr><tr><td>Spec2-Code-Tests</td><td>with</td><td>0.37</td><td>0.1</td><td>0.18</td><td>0.22</td><td>0.35</td><td>0.35</td><td>0.71</td><td>0.8</td></tr><tr><td>Spec2-CommandLine</td><td>without</td><td>0.18</td><td>0.03</td><td>0.03</td><td>0.2</td><td>0.2</td><td>0.28</td><td>0.28</td><td>0.28</td></tr><tr><td>Spec2-CommandLine</td><td>with</td><td>0.30</td><td>0.03</td><td>0.03</td><td>0.37</td><td>0.37</td><td>0.44</td><td>0.44</td><td>0.44</td></tr><tr><td>Spec2-Commander2</td><td>without</td><td>0.33</td><td>0.03</td><td>0.15</td><td>0.41</td><td>0.43</td><td>0.42</td><td>0.5</td><td>0.5</td></tr><tr><td>Spec2-Commander2</td><td>with</td><td>0.46</td><td>0.14</td><td>0.44</td><td>0.5</td><td>0.52</td><td>0.52</td><td>0.58</td><td>0.58</td></tr><tr><td>Spec2-Commander2-Tests</td><td>without</td><td>0.60</td><td>0.01</td><td>0.64</td><td>0.68</td><td>0.71</td><td>0.72</td><td>0.75</td><td>0.76</td></tr><tr><td>Spec2-Commander2-Tests</td><td>with</td><td>0.59</td><td>0.0</td><td>0.64</td><td>0.67</td><td>0.69</td><td>0.69</td><td>0.75</td><td>0.76</td></tr><tr><td>Spec2-Commands</td><td>without</td><td>0.38</td><td>0.24</td><td>0.24</td><td>0.28</td><td>0.31</td><td>0.35</td><td>0.49</td><td>0.88</td></tr><tr><td>Spec2-Commands</td><td>with</td><td>0.51</td><td>0.24</td><td>0.32</td><td>0.51</td><td>0.53</td><td>0.62</td><td>0.62</td><td>0.88</td></tr><tr><td>Spec2-CommonWidgets</td><td>without</td><td>0.26</td><td>0.03</td><td>0.11</td><td>0.28</td><td>0.34</td><td>0.34</td><td>0.35</td><td>0.43</td></tr><tr><td>Spec2-CommonWidgets</td><td>with</td><td>0.37</td><td>0.04</td><td>0.22</td><td>0.36</td><td>0.4</td><td>0.49</td><td>0.51</td><td>0.61</td></tr><tr><td>Spec2-Core</td><td>without</td><td>0.37</td><td>0.08</td><td>0.17</td><td>0.35</td><td>0.43</td><td>0.47</td><td>0.52</td><td>0.63</td></tr><tr><td>Spec2-Core</td><td>with</td><td>0.495</td><td>0.08</td><td>0.3</td><td>0.5</td><td>0.58</td><td>0.63</td><td>0.65</td><td>0.76</td></tr><tr><td>Spec2-Dialogs</td><td>without</td><td>0.18</td><td>0.02</td><td>0.03</td><td>0.21</td><td>0.24</td><td>0.24</td><td>0.25</td><td>0.29</td></tr><tr><td>Spec2-Dialogs</td><td>with</td><td>0.33</td><td>0.02</td><td>0.18</td><td>0.33</td><td>0.35</td><td>0.48</td><td>0.48</td><td>0.53</td></tr><tr><td>Spec2-Dialogs-Tests</td><td>without</td><td>0.28</td><td>0.05</td><td>0.13</td><td>0.19</td><td>0.34</td><td>0.56</td><td>0.56</td><td>0.56</td></tr><tr><td>Spec2-Dialogs-Tests</td><td>with</td><td>0.27</td><td>0.05</td><td>0.12</td><td>0.19</td><td>0.31</td><td>0.56</td><td>0.56</td><td>0.56</td></tr><tr><td>Spec2-Examples</td><td>without</td><td>0.18</td><td>0.0</td><td>0.02</td><td>0.19</td><td>0.24</td><td>0.24</td><td>0.28</td><td>0.33</td></tr><tr><td>Spec2-Examples</td><td>with</td><td>0.23</td><td>0.01</td><td>0.13</td><td>0.23</td><td>0.27</td><td>0.3</td><td>0.34</td><td>0.37</td></tr><tr><td>Spec2-Interactions</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Spec2-Interactions</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Spec2-Layout</td><td>without</td><td>0.49</td><td>0.03</td><td>0.14</td><td>0.38</td><td>0.63</td><td>0.72</td><td>0.77</td><td>0.83</td></tr><tr><td>Spec2-Layout</td><td>with</td><td>0.42</td><td>0.04</td><td>0.42</td><td>0.45</td><td>0.45</td><td>0.49</td><td>0.53</td><td>0.58</td></tr><tr><td>Spec2-ListView</td><td>without</td><td>0.27</td><td>0.02</td><td>0.07</td><td>0.29</td><td>0.31</td><td>0.34</td><td>0.44</td><td>0.44</td></tr><tr><td>Spec2-ListView</td><td>with</td><td>0.37</td><td>0.03</td><td>0.18</td><td>0.38</td><td>0.39</td><td>0.46</td><td>0.57</td><td>0.57</td></tr><tr><td>Spec2-ListView-Tests</td><td>without</td><td>0.25</td><td>0.03</td><td>0.05</td><td>0.08</td><td>0.25</td><td>0.34</td><td>0.52</td><td>0.52</td></tr><tr><td>Spec2-ListView-Tests</td><td>with</td><td>0.24</td><td>0.03</td><td>0.03</td><td>0.21</td><td>0.24</td><td>0.32</td><td>0.44</td><td>0.44</td></tr><tr><td>Spec2-Microdown</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Spec2-Microdown</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Spec2-Morphic</td><td>without</td><td>0.40</td><td>0.13</td><td>0.3</td><td>0.37</td><td>0.44</td><td>0.49</td><td>0.54</td><td>0.73</td></tr><tr><td>Spec2-Morphic</td><td>with</td><td>0.47</td><td>0.18</td><td>0.35</td><td>0.46</td><td>0.55</td><td>0.6</td><td>0.64</td><td>0.76</td></tr><tr><td>Spec2-Morphic-Backend-Tests</td><td>without</td><td>0.19</td><td>0.01</td><td>0.05</td><td>0.14</td><td>0.21</td><td>0.25</td><td>0.32</td><td>0.41</td></tr><tr><td>Spec2-Morphic-Backend-Tests</td><td>with</td><td>0.29</td><td>0.04</td><td>0.15</td><td>0.26</td><td>0.34</td><td>0.4</td><td>0.44</td><td>0.51</td></tr><tr><td>Spec2-Morphic-Examples</td><td>without</td><td>0.11</td><td>0.0</td><td>0.03</td><td>0.11</td><td>0.13</td><td>0.21</td><td>0.24</td><td>0.24</td></tr><tr><td>Spec2-Morphic-Examples</td><td>with</td><td>0.17</td><td>0.0</td><td>0.08</td><td>0.15</td><td>0.17</td><td>0.35</td><td>0.37</td><td>0.37</td></tr><tr><td>Spec2-Morphic-Tests</td><td>without</td><td>0.38</td><td>0.05</td><td>0.21</td><td>0.34</td><td>0.42</td><td>0.53</td><td>0.71</td><td>0.74</td></tr><tr><td>Spec2-Morphic-Tests</td><td>with</td><td>0.39</td><td>0.1</td><td>0.28</td><td>0.4</td><td>0.45</td><td>0.57</td><td>0.58</td><td>0.6</td></tr><tr><td>Spec2-Tests</td><td>without</td><td>0.23</td><td>0.01</td><td>0.06</td><td>0.19</td><td>0.28</td><td>0.29</td><td>0.36</td><td>0.45</td></tr><tr><td>Spec2-Tests</td><td>with</td><td>0.21</td><td>0.01</td><td>0.07</td><td>0.2</td><td>0.26</td><td>0.28</td><td>0.34</td><td>0.38</td></tr><tr><td>Spec2-Transmission</td><td>without</td><td>0.183</td><td>0.02</td><td>0.02</td><td>0.12</td><td>0.25</td><td>0.26</td><td>0.26</td><td>0.36</td></tr></tbody></table>

Continued on next page
<table><caption>Table 4 – continued from previous page</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td>Spec2-Transmission</td><td>with</td><td>0.75</td><td>0.07</td><td>0.33</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.98</td><td>0.98</td></tr></tbody></table>
D. Roassal

<table><caption>Table 5: Performance of Roassal Framework</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td>Roassal</td><td>without</td><td>0.29</td><td>0.05</td><td>0.13</td><td>0.22</td><td>0.32</td><td>0.4</td><td>0.44</td><td>0.64</td></tr><tr><td>Roassal</td><td>with</td><td>0.31</td><td>0.08</td><td>0.18</td><td>0.27</td><td>0.32</td><td>0.39</td><td>0.42</td><td>0.64</td></tr><tr><td>Roassal-Animation</td><td>without</td><td>0.41</td><td>0.09</td><td>0.38</td><td>0.43</td><td>0.45</td><td>0.57</td><td>0.59</td><td>0.53</td></tr><tr><td>Roassal-Animation</td><td>with</td><td>0.54</td><td>0.15</td><td>0.55</td><td>0.55</td><td>0.57</td><td>0.74</td><td>0.74</td><td>0.73</td></tr><tr><td>Roassal-Animation-Tests</td><td>without</td><td>0.33</td><td>0.01</td><td>0.05</td><td>0.34</td><td>0.46</td><td>0.47</td><td>0.47</td><td>0.57</td></tr><tr><td>Roassal-Animation-Tests</td><td>with</td><td>0.27</td><td>0.01</td><td>0.03</td><td>0.21</td><td>0.39</td><td>0.41</td><td>0.41</td><td>0.47</td></tr><tr><td>Roassal-BaselineMap</td><td>without</td><td>0.31</td><td>0.02</td><td>0.08</td><td>0.19</td><td>0.36</td><td>0.42</td><td>0.59</td><td>0.64</td></tr><tr><td>Roassal-BaselineMap</td><td>with</td><td>0.33</td><td>0.06</td><td>0.14</td><td>0.27</td><td>0.41</td><td>0.45</td><td>0.5</td><td>0.55</td></tr><tr><td>Roassal-BaselineMap-Tests</td><td>without</td><td>0.30</td><td>0.0</td><td>0.07</td><td>0.36</td><td>0.39</td><td>0.39</td><td>0.44</td><td>0.5</td></tr><tr><td>Roassal-BaselineMap-Tests</td><td>with</td><td>0.25</td><td>0.0</td><td>0.13</td><td>0.33</td><td>0.33</td><td>0.33</td><td>0.33</td><td>0.33</td></tr><tr><td>Roassal-Builders</td><td>without</td><td>0.40</td><td>0.04</td><td>0.19</td><td>0.38</td><td>0.57</td><td>0.61</td><td>0.66</td><td>0.73</td></tr><tr><td>Roassal-Builders</td><td>with</td><td>0.29</td><td>0.04</td><td>0.15</td><td>0.3</td><td>0.43</td><td>0.42</td><td>0.45</td><td>0.51</td></tr><tr><td>Roassal-Chart</td><td>without</td><td>0.49</td><td>0.04</td><td>0.31</td><td>0.49</td><td>0.61</td><td>0.67</td><td>0.76</td><td>0.81</td></tr><tr><td>Roassal-Chart</td><td>with</td><td>0.47</td><td>0.05</td><td>0.31</td><td>0.45</td><td>0.59</td><td>0.65</td><td>0.69</td><td>0.78</td></tr><tr><td>Roassal-Chart-Examples</td><td>without</td><td>0.33</td><td>0.03</td><td>0.13</td><td>0.29</td><td>0.44</td><td>0.48</td><td>0.55</td><td>0.61</td></tr><tr><td>Roassal-Chart-Examples</td><td>with</td><td>0.29</td><td>0.03</td><td>0.13</td><td>0.26</td><td>0.39</td><td>0.42</td><td>0.47</td><td>0.51</td></tr><tr><td>Roassal-Chart-Tests</td><td>without</td><td>0.48</td><td>0.04</td><td>0.25</td><td>0.48</td><td>0.57</td><td>0.67</td><td>0.76</td><td>0.81</td></tr><tr><td>Roassal-Chart-Tests</td><td>with</td><td>0.30</td><td>0.04</td><td>0.12</td><td>0.26</td><td>0.38</td><td>0.43</td><td>0.48</td><td>0.5</td></tr><tr><td>Roassal-Class-Examples</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Roassal-Class-Examples</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Roassal-Colors</td><td>without</td><td>0.52</td><td>0.11</td><td>0.51</td><td>0.55</td><td>0.55</td><td>0.78</td><td>0.78</td><td>1.0</td></tr><tr><td>Roassal-Colors</td><td>with</td><td>0.52</td><td>0.11</td><td>0.51</td><td>0.55</td><td>0.55</td><td>0.78</td><td>0.78</td><td>1.0</td></tr><tr><td>Roassal-DSM</td><td>without</td><td>0.48</td><td>0.01</td><td>0.14</td><td>0.43</td><td>0.66</td><td>0.88</td><td>0.88</td><td>0.85</td></tr><tr><td>Roassal-DSM</td><td>with</td><td>0.32</td><td>0.01</td><td>0.09</td><td>0.34</td><td>0.48</td><td>0.51</td><td>0.51</td><td>0.53</td></tr><tr><td>Roassal-Event</td><td>without</td><td>0.21</td><td>0.13</td><td>0.16</td><td>0.22</td><td>0.22</td><td>0.27</td><td>0.27</td><td>0.27</td></tr><tr><td>Roassal-Event</td><td>with</td><td>0.21</td><td>0.13</td><td>0.16</td><td>0.22</td><td>0.22</td><td>0.27</td><td>0.27</td><td>0.27</td></tr><tr><td>Roassal-Examples</td><td>without</td><td>0.34</td><td>0.01</td><td>0.11</td><td>0.25</td><td>0.44</td><td>0.53</td><td>0.65</td><td>0.71</td></tr><tr><td>Roassal-Examples</td><td>with</td><td>0.36</td><td>0.01</td><td>0.11</td><td>0.29</td><td>0.5</td><td>0.57</td><td>0.63</td><td>0.7</td></tr><tr><td>Roassal-Experimental</td><td>without</td><td>0.37</td><td>0.06</td><td>0.26</td><td>0.34</td><td>0.41</td><td>0.48</td><td>0.51</td><td>0.62</td></tr><tr><td>Roassal-Experimental</td><td>with</td><td>0.37</td><td>0.07</td><td>0.27</td><td>0.35</td><td>0.43</td><td>0.49</td><td>0.51</td><td>0.62</td></tr><tr><td>Roassal-Exporters</td><td>without</td><td>0.29</td><td>0.01</td><td>0.3</td><td>0.33</td><td>0.34</td><td>0.37</td><td>0.37</td><td>0.37</td></tr><tr><td>Roassal-Exporters</td><td>with</td><td>0.31</td><td>0.02</td><td>0.33</td><td>0.34</td><td>0.36</td><td>0.38</td><td>0.38</td><td>0.39</td></tr><tr><td>Roassal-Exporters-Examples</td><td>without</td><td>0.36</td><td>0.01</td><td>0.08</td><td>0.22</td><td>0.53</td><td>0.6</td><td>0.6</td><td>0.71</td></tr><tr><td>Roassal-Exporters-Examples</td><td>with</td><td>0.38</td><td>0.01</td><td>0.09</td><td>0.26</td><td>0.54</td><td>0.61</td><td>0.61</td><td>0.75</td></tr><tr><td>Roassal-Exporters-Tests</td><td>without</td><td>0.28</td><td>0.01</td><td>0.05</td><td>0.19</td><td>0.4</td><td>0.46</td><td>0.55</td><td>0.64</td></tr><tr><td>Roassal-Exporters-Tests</td><td>with</td><td>0.30</td><td>0.01</td><td>0.06</td><td>0.25</td><td>0.45</td><td>0.49</td><td>0.57</td><td>0.66</td></tr><tr><td>Roassal-FlameGraph-Examples</td><td>without</td><td>0.41</td><td>0.0</td><td>0.1</td><td>0.38</td><td>0.58</td><td>0.62</td><td>0.67</td><td>0.68</td></tr><tr><td>Roassal-FlameGraph-Examples</td><td>with</td><td>0.29</td><td>0.0</td><td>0.06</td><td>0.32</td><td>0.4</td><td>0.42</td><td>0.46</td><td>0.48</td></tr><tr><td>Roassal-Global-Tests</td><td>without</td><td>0.28</td><td>0.02</td><td>0.09</td><td>0.24</td><td>0.36</td><td>0.48</td><td>0.55</td><td>0.66</td></tr><tr><td>Roassal-Global-Tests</td><td>with</td><td>0.28</td><td>0.01</td><td>0.07</td><td>0.21</td><td>0.43</td><td>0.49</td><td>0.52</td><td>0.68</td></tr><tr><td>Roassal-Inspector</td><td>without</td><td>0.21</td><td>0.0</td><td>0.05</td><td>0.2</td><td>0.28</td><td>0.29</td><td>0.32</td><td>0.39</td></tr><tr><td>Roassal-Inspector</td><td>with</td><td>0.19</td><td>0.04</td><td>0.07</td><td>0.16</td><td>0.25</td><td>0.25</td><td>0.27</td><td>0.37</td></tr><tr><td>Roassal-Inspector-Tests</td><td>without</td><td>0.25</td><td>0.0</td><td>0.08</td><td>0.2</td><td>0.35</td><td>0.43</td><td>0.43</td><td>0.5</td></tr><tr><td>Roassal-Inspector-Tests</td><td>with</td><td>0.31</td><td>0.0</td><td>0.11</td><td>0.22</td><td>0.49</td><td>0.49</td><td>0.49</td><td>0.64</td></tr><tr><td>Roassal-Interaction</td><td>without</td><td>0.35</td><td>0.01</td><td>0.13</td><td>0.3</td><td>0.44</td><td>0.53</td><td>0.57</td><td>0.7</td></tr><tr><td>Roassal-Interaction</td><td>with</td><td>0.35</td><td>0.02</td><td>0.11</td><td>0.28</td><td>0.47</td><td>0.52</td><td>0.55</td><td>0.71</td></tr><tr><td>Roassal-Interaction-Tests</td><td>without</td><td>0.26</td><td>0.0</td><td>0.06</td><td>0.18</td><td>0.33</td><td>0.4</td><td>0.48</td><td>0.59</td></tr><tr><td>Roassal-Interaction-Tests</td><td>with</td><td>0.28</td><td>0.0</td><td>0.08</td><td>0.22</td><td>0.38</td><td>0.42</td><td>0.44</td><td>0.58</td></tr><tr><td>Roassal-LayoutStudio</td><td>without</td><td>0.24</td><td>0.01</td><td>0.08</td><td>0.25</td><td>0.32</td><td>0.32</td><td>0.33</td><td>0.4</td></tr><tr><td>Roassal-LayoutStudio</td><td>with</td><td>0.26</td><td>0.05</td><td>0.17</td><td>0.29</td><td>0.31</td><td>0.32</td><td>0.33</td><td>0.4</td></tr><tr><td>Roassal-LayoutStudio-Tests</td><td>without</td><td>0.49</td><td>0.0</td><td>0.19</td><td>0.25</td><td>0.75</td><td>0.75</td><td>0.75</td><td>0.75</td></tr><tr><td>Roassal-LayoutStudio-Tests</td><td>with</td><td>0.27</td><td>0.0</td><td>0.19</td><td>0.25</td><td>0.38</td><td>0.38</td><td>0.38</td><td>0.38</td></tr><tr><td>Roassal-Layouts</td><td>without</td><td>0.42</td><td>0.11</td><td>0.2</td><td>0.35</td><td>0.49</td><td>0.57</td><td>0.61</td><td>0.78</td></tr><tr><td>Roassal-Layouts</td><td>with</td><td>0.52</td><td>0.12</td><td>0.37</td><td>0.51</td><td>0.59</td><td>0.66</td><td>0.7</td><td>0.87</td></tr><tr><td>Roassal-Layouts-Tests</td><td>without</td><td>0.34</td><td>0.01</td><td>0.13</td><td>0.3</td><td>0.47</td><td>0.48</td><td>0.54</td><td>0.54</td></tr><tr><td>Roassal-Layouts-Tests</td><td>with</td><td>0.20</td><td>0.01</td><td>0.08</td><td>0.24</td><td>0.28</td><td>0.26</td><td>0.29</td><td>0.29</td></tr><tr><td>Roassal-Layouts-Util</td><td>without</td><td>0.43</td><td>0.22</td><td>0.41</td><td>0.49</td><td>0.51</td><td>0.42</td><td>0.42</td><td>0.62</td></tr><tr><td>Roassal-Layouts-Util</td><td>with</td><td>0.44</td><td>0.22</td><td>0.44</td><td>0.5</td><td>0.52</td><td>0.44</td><td>0.44</td><td>0.63</td></tr><tr><td>Roassal-Legend</td><td>without</td><td>0.27</td><td>0.0</td><td>0.09</td><td>0.23</td><td>0.38</td><td>0.44</td><td>0.5</td><td>0.64</td></tr><tr><td>Roassal-Legend</td><td>with</td><td>0.22</td><td>0.0</td><td>0.05</td><td>0.19</td><td>0.35</td><td>0.36</td><td>0.38</td><td>0.55</td></tr><tr><td>Roassal-Legend-Examples</td><td>without</td><td>0.35</td><td>0.0</td><td>0.1</td><td>0.28</td><td>0.44</td><td>0.49</td><td>0.69</td><td>0.71</td></tr><tr><td>Roassal-Legend-Examples</td><td>with</td><td>0.28</td><td>0.0</td><td>0.1</td><td>0.28</td><td>0.4</td><td>0.41</td><td>0.47</td><td>0.47</td></tr><tr><td>Roassal-Menu</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Roassal-Menu</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Roassal-Mondrian</td><td>without</td><td>0.36</td><td>0.01</td><td>0.15</td><td>0.28</td><td>0.46</td><td>0.55</td><td>0.6</td><td>0.66</td></tr></tbody></table>

Continued on next page
<table><caption>Table 5 – continued from previous page</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td>Roassal-Mondrian</td><td>with</td><td>0.33</td><td>0.04</td><td>0.19</td><td>0.3</td><td>0.43</td><td>0.46</td><td>0.48</td><td>0.55</td></tr><tr><td>Roassal-Pie</td><td>without</td><td>0.49</td><td>0.03</td><td>0.26</td><td>0.51</td><td>0.58</td><td>0.74</td><td>0.74</td><td>0.93</td></tr><tr><td>Roassal-Pie</td><td>with</td><td>0.36</td><td>0.03</td><td>0.14</td><td>0.29</td><td>0.38</td><td>0.56</td><td>0.63</td><td>0.86</td></tr><tr><td>Roassal-Pie-Examples</td><td>without</td><td>0.36</td><td>0.03</td><td>0.19</td><td>0.32</td><td>0.45</td><td>0.58</td><td>0.62</td><td>0.69</td></tr><tr><td>Roassal-Pie-Examples</td><td>with</td><td>0.35</td><td>0.03</td><td>0.17</td><td>0.32</td><td>0.46</td><td>0.57</td><td>0.59</td><td>0.68</td></tr><tr><td>Roassal-SVG</td><td>without</td><td>0.40</td><td>0.13</td><td>0.24</td><td>0.32</td><td>0.35</td><td>0.4</td><td>0.6</td><td>0.87</td></tr><tr><td>Roassal-SVG</td><td>with</td><td>0.43</td><td>0.17</td><td>0.3</td><td>0.37</td><td>0.4</td><td>0.4</td><td>0.6</td><td>0.87</td></tr><tr><td>Roassal-SVG-Examples</td><td>without</td><td>0.42</td><td>0.02</td><td>0.12</td><td>0.26</td><td>0.45</td><td>0.71</td><td>0.77</td><td>0.86</td></tr><tr><td>Roassal-SVG-Examples</td><td>with</td><td>0.42</td><td>0.01</td><td>0.09</td><td>0.27</td><td>0.46</td><td>0.73</td><td>0.76</td><td>0.86</td></tr><tr><td>Roassal-SVG-Tests</td><td>without</td><td>0.33</td><td>0.0</td><td>0.02</td><td>0.23</td><td>0.37</td><td>0.7</td><td>0.73</td><td>0.73</td></tr><tr><td>Roassal-SVG-Tests</td><td>with</td><td>0.36</td><td>0.0</td><td>0.03</td><td>0.24</td><td>0.46</td><td>0.74</td><td>0.76</td><td>0.76</td></tr><tr><td>Roassal-Shapes</td><td>without</td><td>0.49</td><td>0.08</td><td>0.23</td><td>0.42</td><td>0.62</td><td>0.71</td><td>0.77</td><td>0.84</td></tr><tr><td>Roassal-Shapes</td><td>with</td><td>0.54</td><td>0.12</td><td>0.42</td><td>0.51</td><td>0.65</td><td>0.74</td><td>0.77</td><td>0.84</td></tr><tr><td>Roassal-Shapes-Tests</td><td>without</td><td>0.31</td><td>0.03</td><td>0.12</td><td>0.26</td><td>0.4</td><td>0.44</td><td>0.71</td><td>0.74</td></tr><tr><td>Roassal-Shapes-Tests</td><td>with</td><td>0.33</td><td>0.03</td><td>0.11</td><td>0.29</td><td>0.48</td><td>0.5</td><td>0.61</td><td>0.68</td></tr><tr><td>Roassal-Spec</td><td>without</td><td>0.45</td><td>0.0</td><td>0.25</td><td>0.5</td><td>0.6</td><td>0.6</td><td>0.6</td><td>0.6</td></tr><tr><td>Roassal-Spec</td><td>with</td><td>0.85</td><td>0.55</td><td>0.67</td><td>0.75</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Roassal-Spec-Examples</td><td>without</td><td>0.39</td><td>0.01</td><td>0.16</td><td>0.32</td><td>0.51</td><td>0.55</td><td>0.63</td><td>0.81</td></tr><tr><td>Roassal-Spec-Examples</td><td>with</td><td>0.34</td><td>0.01</td><td>0.09</td><td>0.27</td><td>0.48</td><td>0.5</td><td>0.55</td><td>0.71</td></tr><tr><td>Roassal-Spec-Morphic</td><td>without</td><td>0.11</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.2</td><td>0.2</td><td>0.2</td><td>0.2</td></tr><tr><td>Roassal-Spec-Morphic</td><td>with</td><td>0.70</td><td>0.1</td><td>0.33</td><td>0.5</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Roassal-Spec-Tests</td><td>without</td><td>0.31</td><td>0.01</td><td>0.12</td><td>0.18</td><td>0.32</td><td>0.56</td><td>0.56</td><td>0.56</td></tr><tr><td>Roassal-Spec-Tests</td><td>with</td><td>0.42</td><td>0.06</td><td>0.22</td><td>0.3</td><td>0.47</td><td>0.69</td><td>0.69</td><td>0.69</td></tr><tr><td>Roassal-Sunburst</td><td>without</td><td>0.32</td><td>0.06</td><td>0.21</td><td>0.35</td><td>0.36</td><td>0.4</td><td>0.4</td><td>0.51</td></tr><tr><td>Roassal-Sunburst</td><td>with</td><td>0.48</td><td>0.27</td><td>0.41</td><td>0.46</td><td>0.46</td><td>0.57</td><td>0.57</td><td>0.71</td></tr><tr><td>Roassal-Sunburst-Examples</td><td>without</td><td>0.36</td><td>0.0</td><td>0.08</td><td>0.24</td><td>0.47</td><td>0.52</td><td>0.68</td><td>0.77</td></tr><tr><td>Roassal-Sunburst-Examples</td><td>with</td><td>0.281</td><td>0.0</td><td>0.08</td><td>0.23</td><td>0.37</td><td>0.42</td><td>0.47</td><td>0.52</td></tr><tr><td>Roassal-TreeMap</td><td>without</td><td>0.35</td><td>0.08</td><td>0.23</td><td>0.34</td><td>0.43</td><td>0.46</td><td>0.52</td><td>0.61</td></tr><tr><td>Roassal-TreeMap</td><td>with</td><td>0.50</td><td>0.22</td><td>0.42</td><td>0.46</td><td>0.56</td><td>0.61</td><td>0.66</td><td>0.76</td></tr><tr><td>Roassal-TreeMap-Examples</td><td>without</td><td>0.28</td><td>0.01</td><td>0.1</td><td>0.22</td><td>0.4</td><td>0.46</td><td>0.5</td><td>0.57</td></tr><tr><td>Roassal-TreeMap-Examples</td><td>with</td><td>0.27</td><td>0.01</td><td>0.09</td><td>0.23</td><td>0.38</td><td>0.44</td><td>0.46</td><td>0.52</td></tr><tr><td>Roassal-UML</td><td>without</td><td>0.24</td><td>0.0</td><td>0.06</td><td>0.16</td><td>0.34</td><td>0.39</td><td>0.45</td><td>0.59</td></tr><tr><td>Roassal-UML</td><td>with</td><td>0.28</td><td>0.01</td><td>0.08</td><td>0.2</td><td>0.44</td><td>0.45</td><td>0.47</td><td>0.6</td></tr><tr><td>Roassal-UML-Calypso</td><td>without</td><td>0.35</td><td>0.03</td><td>0.15</td><td>0.27</td><td>0.43</td><td>0.51</td><td>0.59</td><td>0.72</td></tr><tr><td>Roassal-UML-Calypso</td><td>with</td><td>0.31</td><td>0.03</td><td>0.13</td><td>0.25</td><td>0.4</td><td>0.48</td><td>0.51</td><td>0.62</td></tr><tr><td>Roassal-UML-Examples</td><td>without</td><td>0.31</td><td>0.0</td><td>0.07</td><td>0.22</td><td>0.33</td><td>0.37</td><td>0.6</td><td>0.75</td></tr><tr><td>Roassal-UML-Examples</td><td>with</td><td>0.26</td><td>0.01</td><td>0.04</td><td>0.15</td><td>0.32</td><td>0.38</td><td>0.45</td><td>0.58</td></tr><tr><td>Roassal-UML-Tests</td><td>without</td><td>0.25</td><td>0.0</td><td>0.03</td><td>0.21</td><td>0.35</td><td>0.4</td><td>0.43</td><td>0.57</td></tr><tr><td>Roassal-UML-Tests</td><td>with</td><td>0.30</td><td>0.0</td><td>0.0</td><td>0.18</td><td>0.48</td><td>0.49</td><td>0.51</td><td>0.75</td></tr></tbody></table>
E. IceBerg

<table><caption>Table 6: Performance of Iceberg Library</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td rowspan="2">Iceberg</td><td>without</td><td>0.38</td><td>0.03</td><td>0.08</td><td>0.23</td><td>0.43</td><td>0.6</td><td>0.67</td><td>0.69</td></tr><tr><td>with</td><td>0.39</td><td>0.04</td><td>0.08</td><td>0.27</td><td>0.45</td><td>0.61</td><td>0.67</td><td>0.73</td></tr><tr><td rowspan="2">Iceberg-Libgit</td><td>without</td><td>0.30</td><td>0.02</td><td>0.05</td><td>0.15</td><td>0.32</td><td>0.43</td><td>0.55</td><td>0.64</td></tr><tr><td>with</td><td>0.32</td><td>0.04</td><td>0.06</td><td>0.21</td><td>0.34</td><td>0.45</td><td>0.57</td><td>0.66</td></tr><tr><td rowspan="2">Iceberg-Libgit-Filetree</td><td>without</td><td>0.26</td><td>0.08</td><td>0.1</td><td>0.21</td><td>0.3</td><td>0.33</td><td>0.34</td><td>0.5</td></tr><tr><td>with</td><td>0.40</td><td>0.29</td><td>0.31</td><td>0.38</td><td>0.41</td><td>0.41</td><td>0.43</td><td>0.59</td></tr><tr><td rowspan="2">Iceberg-Libgit-Tonel</td><td>without</td><td>0.16</td><td>0.04</td><td>0.06</td><td>0.1</td><td>0.2</td><td>0.22</td><td>0.23</td><td>0.31</td></tr><tr><td>with</td><td>0.32</td><td>0.21</td><td>0.24</td><td>0.28</td><td>0.35</td><td>0.37</td><td>0.39</td><td>0.47</td></tr><tr><td rowspan="2">Iceberg-Metacello-Integration</td><td>without</td><td>0.26</td><td>0.0</td><td>0.02</td><td>0.3</td><td>0.31</td><td>0.42</td><td>0.44</td><td>0.39</td></tr><tr><td>with</td><td>0.45</td><td>0.15</td><td>0.16</td><td>0.57</td><td>0.58</td><td>0.6</td><td>0.62</td><td>0.56</td></tr><tr><td rowspan="2">Iceberg-Plugin</td><td>without</td><td>0.57</td><td>0.22</td><td>0.67</td><td>0.71</td><td>0.78</td><td>0.5</td><td>0.5</td><td>0.5</td></tr><tr><td>with</td><td>0.63</td><td>0.29</td><td>0.73</td><td>0.83</td><td>0.83</td><td>0.5</td><td>0.5</td><td>0.5</td></tr><tr><td rowspan="2">Iceberg-Plugin-GitHub</td><td>without</td><td>0.26</td><td>0.02</td><td>0.08</td><td>0.18</td><td>0.36</td><td>0.36</td><td>0.43</td><td>0.45</td></tr><tr><td>with</td><td>0.28</td><td>0.06</td><td>0.12</td><td>0.21</td><td>0.38</td><td>0.38</td><td>0.45</td><td>0.47</td></tr><tr><td rowspan="2">Iceberg-Plugin-Migration</td><td>without</td><td>0.27</td><td>0.0</td><td>0.05</td><td>0.2</td><td>0.28</td><td>0.43</td><td>0.43</td><td>0.51</td></tr><tr><td>with</td><td>0.32</td><td>0.13</td><td>0.17</td><td>0.35</td><td>0.36</td><td>0.36</td><td>0.43</td><td>0.47</td></tr><tr><td rowspan="2">Iceberg-Plugin-Pharo</td><td>without</td><td>0.51</td><td>0.1</td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.75</td><td>0.75</td></tr><tr><td>with</td><td>0.51</td><td>0.1</td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.75</td><td>0.75</td></tr><tr><td rowspan="2">Iceberg-TipUI</td><td>without</td><td>0.21</td><td>0.01</td><td>0.04</td><td>0.15</td><td>0.28</td><td>0.29</td><td>0.34</td><td>0.41</td></tr><tr><td>with</td><td>0.23</td><td>0.01</td><td>0.04</td><td>0.16</td><td>0.28</td><td>0.29</td><td>0.39</td><td>0.47</td></tr><tr><td rowspan="2">Iceberg-TipUI-SnapshotBrowser</td><td>without</td><td>0.26</td><td>0.07</td><td>0.21</td><td>0.33</td><td>0.36</td><td>0.31</td><td>0.27</td><td>0.34</td></tr><tr><td>with</td><td>0.26</td><td>0.07</td><td>0.21</td><td>0.33</td><td>0.36</td><td>0.31</td><td>0.27</td><td>0.34</td></tr></tbody></table>
## F. Moose

<table><caption>Table 7: Performance of Moose Framework</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td>Moose-Blueprint-Invocations-Models</td><td>without</td><td>0.428</td><td>0.02</td><td>0.49</td><td>0.49</td><td>0.49</td><td>0.49</td><td>0.49</td><td>0.53</td></tr><tr><td>Moose-Blueprint-Invocations-Models</td><td>with</td><td>0.846</td><td>0.83</td><td>0.84</td><td>0.84</td><td>0.84</td><td>0.84</td><td>0.84</td><td>0.89</td></tr><tr><td>Moose-Blueprint-Models</td><td>without</td><td>0.373</td><td>0.03</td><td>0.19</td><td>0.39</td><td>0.49</td><td>0.52</td><td>0.52</td><td>0.62</td></tr><tr><td>Moose-Blueprint-Models</td><td>with</td><td>0.397</td><td>0.03</td><td>0.23</td><td>0.42</td><td>0.51</td><td>0.55</td><td>0.55</td><td>0.66</td></tr><tr><td>Moose-Blueprint-Models-Tests</td><td>without</td><td>0.089</td><td>0.0</td><td>0.04</td><td>0.06</td><td>0.06</td><td>0.08</td><td>0.08</td><td>0.31</td></tr><tr><td>Moose-Blueprint-Models-Tests</td><td>with</td><td>0.086</td><td>0.0</td><td>0.06</td><td>0.06</td><td>0.06</td><td>0.06</td><td>0.06</td><td>0.31</td></tr><tr><td>Moose-Blueprint-Visualization-Models</td><td>without</td><td>0.407</td><td>0.06</td><td>0.2</td><td>0.43</td><td>0.51</td><td>0.58</td><td>0.58</td><td>0.65</td></tr><tr><td>Moose-Blueprint-Visualization-Models</td><td>with</td><td>0.410</td><td>0.06</td><td>0.21</td><td>0.43</td><td>0.51</td><td>0.58</td><td>0.6</td><td>0.67</td></tr><tr><td>Moose-Configuration</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-Configuration</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-Core</td><td>without</td><td>0.338</td><td>0.12</td><td>0.2</td><td>0.24</td><td>0.27</td><td>0.45</td><td>0.59</td><td>0.63</td></tr><tr><td>Moose-Core</td><td>with</td><td>0.396</td><td>0.16</td><td>0.24</td><td>0.29</td><td>0.32</td><td>0.57</td><td>0.63</td><td>0.67</td></tr><tr><td>Moose-Core-Generator</td><td>without</td><td>0.158</td><td>0.13</td><td>0.17</td><td>0.17</td><td>0.17</td><td>0.17</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-Core-Generator</td><td>with</td><td>0.158</td><td>0.13</td><td>0.17</td><td>0.17</td><td>0.17</td><td>0.17</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-Core-Tests</td><td>without</td><td>0.146</td><td>0.03</td><td>0.05</td><td>0.07</td><td>0.07</td><td>0.24</td><td>0.29</td><td>0.32</td></tr><tr><td>Moose-Core-Tests</td><td>with</td><td>0.137</td><td>0.03</td><td>0.05</td><td>0.06</td><td>0.06</td><td>0.22</td><td>0.28</td><td>0.3</td></tr><tr><td>Moose-Core-Tests-Entities</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-Core-Tests-Entities</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-Importers</td><td>without</td><td>0.198</td><td>0.02</td><td>0.05</td><td>0.08</td><td>0.04</td><td>0.11</td><td>0.44</td><td>0.71</td></tr><tr><td>Moose-Importers</td><td>with</td><td>0.280</td><td>0.14</td><td>0.18</td><td>0.21</td><td>0.19</td><td>0.19</td><td>0.42</td><td>0.68</td></tr><tr><td>Moose-Importers-Tests</td><td>without</td><td>0.080</td><td>0.03</td><td>0.03</td><td>0.06</td><td>0.08</td><td>0.11</td><td>0.11</td><td>0.19</td></tr><tr><td>Moose-Importers-Tests</td><td>with</td><td>0.117</td><td>0.07</td><td>0.07</td><td>0.1</td><td>0.12</td><td>0.14</td><td>0.14</td><td>0.22</td></tr><tr><td>Moose-Query</td><td>without</td><td>0.293</td><td>0.1</td><td>0.25</td><td>0.25</td><td>0.25</td><td>0.27</td><td>0.27</td><td>0.67</td></tr><tr><td>Moose-Query</td><td>with</td><td>0.332</td><td>0.15</td><td>0.3</td><td>0.3</td><td>0.3</td><td>0.31</td><td>0.31</td><td>0.67</td></tr><tr><td>Moose-Query-Extensions</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-Query-Extensions</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-Query-Test</td><td>without</td><td>0.105</td><td>0.0</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.04</td><td>0.25</td><td>0.43</td></tr><tr><td>Moose-Query-Test</td><td>with</td><td>0.092</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.03</td><td>0.21</td><td>0.4</td></tr><tr><td>Moose-SmalltalkImporter</td><td>without</td><td>0.262</td><td>0.12</td><td>0.19</td><td>0.22</td><td>0.27</td><td>0.3</td><td>0.34</td><td>0.45</td></tr><tr><td>Moose-SmalltalkImporter</td><td>with</td><td>0.345</td><td>0.22</td><td>0.29</td><td>0.32</td><td>0.37</td><td>0.36</td><td>0.41</td><td>0.49</td></tr><tr><td>Moose-SmalltalkImporter-Core-Tests</td><td>without</td><td>0.326</td><td>0.04</td><td>0.11</td><td>0.33</td><td>0.33</td><td>0.38</td><td>0.5</td><td>0.72</td></tr><tr><td>Moose-SmalltalkImporter-Core-Tests</td><td>with</td><td>0.487</td><td>0.37</td><td>0.38</td><td>0.39</td><td>0.39</td><td>0.49</td><td>0.61</td><td>0.88</td></tr><tr><td>Moose-SmalltalkImporter-KGB-Tests</td><td>without</td><td>0.175</td><td>0.02</td><td>0.02</td><td>0.03</td><td>0.03</td><td>0.03</td><td>0.46</td><td>0.66</td></tr><tr><td>Moose-SmalltalkImporter-KGB-Tests</td><td>with</td><td>0.178</td><td>0.02</td><td>0.02</td><td>0.03</td><td>0.03</td><td>0.03</td><td>0.46</td><td>0.66</td></tr><tr><td>Moose-SmalltalkImporter-LAN-Tests</td><td>without</td><td>0.359</td><td>0.1</td><td>0.1</td><td>0.39</td><td>0.41</td><td>0.44</td><td>0.51</td><td>0.67</td></tr><tr><td>Moose-SmalltalkImporter-LAN-Tests</td><td>with</td><td>0.380</td><td>0.03</td><td>0.03</td><td>0.39</td><td>0.42</td><td>0.51</td><td>0.57</td><td>0.88</td></tr><tr><td>Moose-TestResources-KGB-P10InteractedReferee</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-KGB-P10InteractedReferee</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-KGB-P11FullReferee</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-KGB-P11FullReferee</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-KGB-P12FullReferencer</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-KGB-P12FullReferencer</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-KGB-P13FullReferencer</td><td>without</td><td>0.785</td><td>0.5</td><td>0.5</td><td>0.5</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P13FullReferencer</td><td>with</td><td>0.785</td><td>0.5</td><td>0.5</td><td>0.5</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P14FullReferee</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-KGB-P14FullReferee</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-KGB-P1FullReferencer</td><td>without</td><td>0.666</td><td>0.33</td><td>0.33</td><td>0.67</td><td>0.67</td><td>0.67</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P1FullReferencer</td><td>with</td><td>0.704</td><td>0.47</td><td>0.47</td><td>0.67</td><td>0.67</td><td>0.67</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P2InteractedReferencerReferee</td><td>without</td><td>0.768</td><td>0.56</td><td>0.56</td><td>0.63</td><td>0.63</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P2InteractedReferencerReferee</td><td>with</td><td>0.942</td><td>0.9</td><td>0.9</td><td>0.9</td><td>0.9</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P3InteractedReferencer</td><td>without</td><td>0.800</td><td>0.49</td><td>0.49</td><td>0.81</td><td>0.81</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P3InteractedReferencer</td><td>with</td><td>0.6875</td><td>0.43</td><td>0.43</td><td>0.48</td><td>0.48</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P4FullInteracted</td><td>without</td><td>0.628</td><td>0.2</td><td>0.2</td><td>0.5</td><td>0.5</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P4FullInteracted</td><td>with</td><td>0.714</td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.5</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P5FullReferee</td><td>without</td><td>0.857</td><td>0.5</td><td>0.5</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P5FullReferee</td><td>with</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P6InteractedReferee</td><td>without</td><td>0.571</td><td>0.25</td><td>0.25</td><td>0.5</td><td>0.5</td><td>0.5</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P6InteractedReferee</td><td>with</td><td>0.642</td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.5</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P7ReferencerReferee</td><td>without</td><td>0.803</td><td>0.55</td><td>0.57</td><td>0.88</td><td>0.88</td><td>0.88</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P7ReferencerReferee</td><td>with</td><td>0.627</td><td>0.29</td><td>0.31</td><td>0.54</td><td>0.54</td><td>0.88</td><td>1.0</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P8FullReferencer</td><td>without</td><td>0.698</td><td>0.44</td><td>0.44</td><td>0.5</td><td>0.83</td><td>0.83</td><td>0.83</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P8FullReferencer</td><td>with</td><td>0.714</td><td>0.5</td><td>0.5</td><td>0.5</td><td>0.83</td><td>0.83</td><td>0.83</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P9FullReferencer</td><td>without</td><td>0.654</td><td>0.42</td><td>0.42</td><td>0.5</td><td>0.75</td><td>0.75</td><td>0.75</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-P9FullReferencer</td><td>with</td><td>0.654</td><td>0.42</td><td>0.42</td><td>0.5</td><td>0.75</td><td>0.75</td><td>0.75</td><td>1.0</td></tr><tr><td>Moose-TestResources-KGB-PExtensions</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr></tbody></table>

Continued on next page
<table><caption>Table 7 – continued from previous page</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td>Moose-TestResources-KGB-PExtensions</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-LAN</td><td>without</td><td>0.337</td><td>0.08</td><td>0.08</td><td>0.31</td><td>0.38</td><td>0.51</td><td>0.53</td><td>0.5</td></tr><tr><td>Moose-TestResources-LAN</td><td>with</td><td>0.343</td><td>0.09</td><td>0.09</td><td>0.34</td><td>0.38</td><td>0.51</td><td>0.53</td><td>0.5</td></tr><tr><td>Moose-TestResources-LCOM</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-LCOM</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-PackageBlueprint-P1</td><td>without</td><td>1.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-PackageBlueprint-P1</td><td>with</td><td>1.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-PackageBlueprint-P2</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-PackageBlueprint-P2</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-PackageBlueprint-P3</td><td>without</td><td>1.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-PackageBlueprint-P3</td><td>with</td><td>1.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-PackageBlueprint-P4</td><td>without</td><td>1.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-PackageBlueprint-P4</td><td>with</td><td>1.0</td><td>1.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-Reference-Core</td><td>without</td><td>0.463</td><td>0.33</td><td>0.35</td><td>0.41</td><td>0.45</td><td>0.53</td><td>0.61</td><td>0.68</td></tr><tr><td>Moose-TestResources-Reference-Core</td><td>with</td><td>0.464</td><td>0.37</td><td>0.37</td><td>0.4</td><td>0.45</td><td>0.51</td><td>0.59</td><td>0.65</td></tr><tr><td>Moose-TestResources-Reference-External</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-Reference-External</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-Reference-PackageOne</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-Reference-PackageOne</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-Reference-PackageTwo</td><td>without</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-TestResources-Reference-PackageTwo</td><td>with</td><td>0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>Moose-WelcomeBrowser</td><td>without</td><td>0.410</td><td>0.0</td><td>0.22</td><td>0.33</td><td>0.36</td><td>0.63</td><td>0.65</td><td>0.69</td></tr><tr><td>Moose-WelcomeBrowser</td><td>with</td><td>0.284</td><td>0.0</td><td>0.1</td><td>0.17</td><td>0.18</td><td>0.49</td><td>0.51</td><td>0.53</td></tr><tr><td>MooseIDE-Analysis</td><td>without</td><td>0.193</td><td>0.01</td><td>0.03</td><td>0.16</td><td>0.21</td><td>0.25</td><td>0.3</td><td>0.4</td></tr><tr><td>MooseIDE-Analysis</td><td>with</td><td>0.380</td><td>0.08</td><td>0.31</td><td>0.36</td><td>0.41</td><td>0.47</td><td>0.48</td><td>0.56</td></tr><tr><td>MooseIDE-AttributedText</td><td>without</td><td>0.288</td><td>0.01</td><td>0.1</td><td>0.28</td><td>0.33</td><td>0.32</td><td>0.59</td><td>0.72</td></tr><tr><td>MooseIDE-AttributedText</td><td>with</td><td>0.348</td><td>0.06</td><td>0.17</td><td>0.33</td><td>0.38</td><td>0.5</td><td>0.59</td><td>0.72</td></tr><tr><td>MooseIDE-ButterflyMap</td><td>without</td><td>0.523</td><td>0.2</td><td>0.45</td><td>0.54</td><td>0.63</td><td>0.61</td><td>0.62</td><td>0.68</td></tr><tr><td>MooseIDE-ButterflyMap</td><td>with</td><td>0.435</td><td>0.25</td><td>0.32</td><td>0.44</td><td>0.53</td><td>0.49</td><td>0.5</td><td>0.55</td></tr><tr><td>MooseIDE-ButterflyMap-Tests</td><td>without</td><td>0.345</td><td>0.22</td><td>0.27</td><td>0.27</td><td>0.45</td><td>0.49</td><td>0.45</td><td>0.48</td></tr><tr><td>MooseIDE-ButterflyMap-Tests</td><td>with</td><td>0.336</td><td>0.22</td><td>0.26</td><td>0.26</td><td>0.43</td><td>0.47</td><td>0.44</td><td>0.47</td></tr><tr><td>MooseIDE-ClassBlueprint</td><td>without</td><td>0.219</td><td>0.03</td><td>0.06</td><td>0.2</td><td>0.23</td><td>0.83</td><td>0.83</td><td>1.0</td></tr><tr><td>MooseIDE-ClassBlueprint</td><td>with</td><td>0.244</td><td>0.09</td><td>0.12</td><td>0.2</td><td>0.23</td><td>0.83</td><td>0.83</td><td>1.0</td></tr><tr><td>MooseIDE-CoUsageMap</td><td>without</td><td>0.329</td><td>0.02</td><td>0.11</td><td>0.29</td><td>0.44</td><td>0.49</td><td>0.53</td><td>0.58</td></tr><tr><td>MooseIDE-CoUsageMap</td><td>with</td><td>0.328</td><td>0.02</td><td>0.1</td><td>0.27</td><td>0.44</td><td>0.49</td><td>0.53</td><td>0.59</td></tr><tr><td>MooseIDE-CoUsageMap-Tests</td><td>without</td><td>0.200</td><td>0.03</td><td>0.08</td><td>0.12</td><td>0.26</td><td>0.26</td><td>0.27</td><td>0.43</td></tr><tr><td>MooseIDE-CoUsageMap-Tests</td><td>with</td><td>0.149</td><td>0.03</td><td>0.05</td><td>0.07</td><td>0.19</td><td>0.19</td><td>0.19</td><td>0.35</td></tr><tr><td>MooseIDE-Core</td><td>without</td><td>0.264</td><td>0.03</td><td>0.09</td><td>0.26</td><td>0.29</td><td>0.36</td><td>0.4</td><td>0.45</td></tr><tr><td>MooseIDE-Core</td><td>with</td><td>0.283</td><td>0.05</td><td>0.15</td><td>0.28</td><td>0.31</td><td>0.37</td><td>0.41</td><td>0.45</td></tr><tr><td>MooseIDE-Core-Reporter</td><td>without</td><td>0.254</td><td>0.01</td><td>0.01</td><td>0.26</td><td>0.36</td><td>0.37</td><td>0.39</td><td>0.41</td></tr><tr><td>MooseIDE-Core-Reporter</td><td>with</td><td>0.287</td><td>0.03</td><td>0.12</td><td>0.28</td><td>0.38</td><td>0.39</td><td>0.41</td><td>0.43</td></tr><tr><td>MooseIDE-CriticBrowser</td><td>without</td><td>0.249</td><td>0.04</td><td>0.11</td><td>0.22</td><td>0.27</td><td>0.3</td><td>0.31</td><td>0.5</td></tr><tr><td>MooseIDE-CriticBrowser</td><td>with</td><td>0.267</td><td>0.06</td><td>0.12</td><td>0.22</td><td>0.3</td><td>0.32</td><td>0.33</td><td>0.52</td></tr><tr><td>MooseIDE-CriticBrowser-Tests</td><td>without</td><td>0.165</td><td>0.0</td><td>0.07</td><td>0.14</td><td>0.16</td><td>0.16</td><td>0.18</td><td>0.45</td></tr><tr><td>MooseIDE-CriticBrowser-Tests</td><td>with</td><td>0.156</td><td>0.0</td><td>0.07</td><td>0.13</td><td>0.15</td><td>0.15</td><td>0.16</td><td>0.43</td></tr><tr><td>MooseIDE-Dependency</td><td>without</td><td>0.481</td><td>0.12</td><td>0.34</td><td>0.52</td><td>0.6</td><td>0.6</td><td>0.64</td><td>0.66</td></tr><tr><td>MooseIDE-Dependency</td><td>with</td><td>0.384</td><td>0.12</td><td>0.23</td><td>0.42</td><td>0.49</td><td>0.47</td><td>0.5</td><td>0.53</td></tr><tr><td>MooseIDE-Duplication</td><td>without</td><td>0.420</td><td>0.01</td><td>0.39</td><td>0.43</td><td>0.49</td><td>0.52</td><td>0.53</td><td>0.6</td></tr><tr><td>MooseIDE-Duplication</td><td>with</td><td>0.161</td><td>0.02</td><td>0.05</td><td>0.16</td><td>0.2</td><td>0.22</td><td>0.22</td><td>0.29</td></tr><tr><td>MooseIDE-Export</td><td>without</td><td>0.343</td><td>0.06</td><td>0.26</td><td>0.35</td><td>0.4</td><td>0.43</td><td>0.44</td><td>0.46</td></tr><tr><td>MooseIDE-Export</td><td>with</td><td>0.288</td><td>0.1</td><td>0.19</td><td>0.28</td><td>0.33</td><td>0.36</td><td>0.37</td><td>0.39</td></tr><tr><td>MooseIDE-Famix</td><td>without</td><td>0.277</td><td>0.02</td><td>0.1</td><td>0.2</td><td>0.32</td><td>0.39</td><td>0.43</td><td>0.5</td></tr><tr><td>MooseIDE-Famix</td><td>with</td><td>0.309</td><td>0.07</td><td>0.21</td><td>0.26</td><td>0.34</td><td>0.39</td><td>0.43</td><td>0.49</td></tr><tr><td>MooseIDE-LayerVisualization</td><td>without</td><td>0.420</td><td>0.15</td><td>0.27</td><td>0.4</td><td>0.54</td><td>0.54</td><td>0.55</td><td>0.64</td></tr><tr><td>MooseIDE-LayerVisualization</td><td>with</td><td>0.420</td><td>0.16</td><td>0.26</td><td>0.4</td><td>0.54</td><td>0.54</td><td>0.55</td><td>0.64</td></tr><tr><td>MooseIDE-Meta</td><td>without</td><td>0.231</td><td>0.02</td><td>0.07</td><td>0.24</td><td>0.29</td><td>0.3</td><td>0.32</td><td>0.41</td></tr><tr><td>MooseIDE-Meta</td><td>with</td><td>0.246</td><td>0.04</td><td>0.09</td><td>0.25</td><td>0.3</td><td>0.31</td><td>0.34</td><td>0.42</td></tr><tr><td>MooseIDE-NewTools</td><td>without</td><td>0.241</td><td>0.02</td><td>0.11</td><td>0.22</td><td>0.31</td><td>0.32</td><td>0.34</td><td>0.38</td></tr><tr><td>MooseIDE-NewTools</td><td>with</td><td>0.254</td><td>0.05</td><td>0.15</td><td>0.23</td><td>0.31</td><td>0.33</td><td>0.34</td><td>0.38</td></tr><tr><td>MooseIDE-NewTools-Tests</td><td>without</td><td>0.203</td><td>0.0</td><td>0.06</td><td>0.17</td><td>0.17</td><td>0.24</td><td>0.26</td><td>0.53</td></tr><tr><td>MooseIDE-NewTools-Tests</td><td>with</td><td>0.123</td><td>0.0</td><td>0.0</td><td>0.04</td><td>0.04</td><td>0.12</td><td>0.2</td><td>0.47</td></tr><tr><td>MooseIDE-QueriesBrowser</td><td>without</td><td>0.329</td><td>0.06</td><td>0.16</td><td>0.36</td><td>0.38</td><td>0.42</td><td>0.45</td><td>0.49</td></tr><tr><td>MooseIDE-QueriesBrowser</td><td>with</td><td>0.349</td><td>0.08</td><td>0.19</td><td>0.37</td><td>0.4</td><td>0.45</td><td>0.47</td><td>0.51</td></tr><tr><td>MooseIDE-QueriesBrowser-Tests</td><td>without</td><td>0.463</td><td>0.02</td><td>0.34</td><td>0.52</td><td>0.56</td><td>0.57</td><td>0.59</td><td>0.64</td></tr><tr><td>MooseIDE-QueriesBrowser-Tests</td><td>with</td><td>0.449</td><td>0.06</td><td>0.33</td><td>0.51</td><td>0.54</td><td>0.55</td><td>0.56</td><td>0.61</td></tr><tr><td>MooseIDE-QueriesDashboard</td><td>without</td><td>0.694</td><td>0.53</td><td>0.56</td><td>0.66</td><td>0.75</td><td>0.78</td><td>0.79</td><td>0.83</td></tr><tr><td>MooseIDE-QueriesDashboard</td><td>with</td><td>0.694</td><td>0.53</td><td>0.56</td><td>0.66</td><td>0.74</td><td>0.78</td><td>0.78</td><td>0.82</td></tr><tr><td>MooseIDE-Spotter</td><td>without</td><td>0.310</td><td>0.0</td><td>0.03</td><td>0.09</td><td>0.38</td><td>0.41</td><td>0.56</td><td>0.71</td></tr><tr><td>MooseIDE-Spotter</td><td>with</td><td>0.3505</td><td>0.13</td><td>0.13</td><td>0.15</td><td>0.38</td><td>0.41</td><td>0.56</td><td>0.71</td></tr><tr><td>MooseIDE-Spotter-Tests</td><td>without</td><td>0.105</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.05</td><td>0.08</td><td>0.16</td><td>0.45</td></tr></tbody></table>

Continued on next page
<table><caption>Table 7 – continued from previous page</caption><thead><tr><th>Package</th><th>Metric</th><th>MRR</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th></tr></thead><tbody><tr><td>MooseIDE-Spotter-Tests</td><td>with</td><td>0.146</td><td>0.07</td><td>0.07</td><td>0.07</td><td>0.07</td><td>0.1</td><td>0.19</td><td>0.45</td></tr><tr><td>MooseIDE-Tagging</td><td>without</td><td>0.214</td><td>0.0</td><td>0.02</td><td>0.26</td><td>0.28</td><td>0.3</td><td>0.32</td><td>0.33</td></tr><tr><td>MooseIDE-Tagging</td><td>with</td><td>0.227</td><td>0.01</td><td>0.04</td><td>0.26</td><td>0.29</td><td>0.32</td><td>0.34</td><td>0.34</td></tr><tr><td>MooseIDE-Tagging-Tests</td><td>without</td><td>0.231</td><td>0.0</td><td>0.12</td><td>0.17</td><td>0.18</td><td>0.37</td><td>0.41</td><td>0.44</td></tr><tr><td>MooseIDE-Tagging-Tests</td><td>with</td><td>0.206</td><td>0.0</td><td>0.1</td><td>0.14</td><td>0.15</td><td>0.33</td><td>0.38</td><td>0.42</td></tr><tr><td>MooseIDE-Tests</td><td>without</td><td>0.205</td><td>0.04</td><td>0.15</td><td>0.18</td><td>0.21</td><td>0.25</td><td>0.26</td><td>0.39</td></tr><tr><td>MooseIDE-Tests</td><td>with</td><td>0.185</td><td>0.05</td><td>0.13</td><td>0.16</td><td>0.19</td><td>0.23</td><td>0.24</td><td>0.36</td></tr><tr><td>MooseIDE-Visualization</td><td>without</td><td>0.539</td><td>0.1</td><td>0.42</td><td>0.56</td><td>0.61</td><td>0.67</td><td>0.73</td><td>0.79</td></tr><tr><td>MooseIDE-Visualization</td><td>with</td><td>0.328</td><td>0.09</td><td>0.17</td><td>0.34</td><td>0.39</td><td>0.41</td><td>0.46</td><td>0.52</td></tr></tbody></table>