Refactoring Codebases Through Library Design

## Abstract

Abstract Maintainable and general software allows developers to build robust applications efficiently, yet achieving these qualities often requires refactoring specialized solutions into reusable components. This challenge becomes particularly relevant as code agents become used to solve isolated one-off programming problems. We investigate code agents’ capacity to refactor code in ways that support growth and reusability. We first investigate what makes a good refactoring, finding via simulation results and a human study that Minimum Description Length best correlates with preferable refactorings. We then present both a benchmark and a method for refactoring: MiniCode , a benchmark where multiple files must be refactored into a shared library, and Librarian , a sample-and-rerank method for generating reusable libraries. We compare Librarian to state-of-the-art library generation methods, and study it on real-world code bases.

## 1 Introduction

Writing code is mainly a matter of *re*writing code: debugging, refactoring, optimizing, and other activities within the software engineering lifecycle.
But poor rewrites incur technical debt, with such debt costing up to $2 *trillion* annually (Tews,).
This problem will likely worsen as language models become increasingly responsible for generating code, because they excel at solving isolated programming problems, but their context length demands a myopic view of the codebase.
Effective code refactoring at scale is a design problem whose concerns center on re-usability and maintainability. A classic example illustrates this design challenge: Human programmers often create overly-specialized, redundant solutions to similar problems and would benefit from redesigning specialized solutions into a shared library.

Here we focus on refactoring multiple files into a reusable software library, which raises two questions: (1) How should we quantify the quality of a library refactoring, and (2)
How can we use LLMs to refactor specialized code into reusable libraries?
To answer those question, we develop a new method and a benchmark.
This goes beyond past work in *library learning* (Wong et al., 2021; Stengel-Eskin et al., 2024; Ellis et al., 2021; Bowers et al., 2023; Dechter et al., 2013; Grand et al., 2023; Liang et al., 2010b) which synthesizes subroutines across small programs in i.e. $\lambda$-calculus, because instead we address the more naturalistic problem of redesigning code written in contemporary high-level languages, such as Python, producing classes, methods, and helper functions in the style of a human-written library.
We develop a method, Librarian ([Figure˜1](https://arxiv.org/html/2506.11058v3#S1.F1)), which samples possible code rewrites and then reranks those samples based on criteria designed to capture a good refactoring.
To generate potential rewrites, we develop methods for clustering pieces of code together that share common structure so that a succinct prompt can rewrite them jointly into their refactored form.
To find strong criteria for ranking potential rewrites, we study a variety of metrics across machine learning and software engineering, both on programming benchmarks and via a human study.

To evaluate our method and systematically assess the capability of current agents to generate libraries, we introduce a new benchmark, MiniCode, which addresses three key desiderata missing from existing benchmarks. First, open-ended design: unlike SWE-Bench (Jimenez et al., 2024), Commit0 (Zhao et al., 2025), and RefactorBench (Gautam et al., 2025) which primarily focus on functional correctness, MiniCode presents an unconstrained library design problem. Agents create a library that can be imported back into a repository, with complete freedom to design the interface and implementation from scratch—optimizing for software engineering objectives like reusability and maintainability. Second, verifiability: we ensure objective evaluation by retaining the unit tests from all repositories that will import the designed library, allowing us to verify that the solutions work correctly across multiple use cases. Third, large context: agents must understand and synthesize information from multiple code sources (files) simultaneously to design a unified library that consolidates specialized code sources into a general interface.

Figure: Figure 1: Overview of the refactoring problem. A refactoring task comprises a set of files. We refactor the files by designing a new library. Candidate refactorings are evaluated based on a refactoring metric, and are expected to maintain correctness of the original code sources (pass rate). We explore several refactoring metrics in this paper.
Refer to caption: x1.png

We contribute the following:

- 1.
Study of different metrics for library creation via a user study and program synthesis benchmarks, with the surprising finding that neural language models offer a stronger signal of refactoring quality than classic metrics from the software engineering community.
- 2.
A new benchmark, MiniCode, covering both competition programming and real-world machine learning programs from the Transformers and Diffusers libraries.
- 3.
A new algorithm, Librarian, which outperforms prior library learners on existing benchmarks and scales to complex codebases.
- 4.
Refactorings of real-world codebases widely used in the machine learning community. Our method successfully refactors the Transformers and Diffusers libraries from Huggingface to be shorter and more reusable.
To the best of our knowledge this is the first time library learning has have been successfully applied to real-world software projects.

## 2 Related work

#### Library Learning.

Systems which perform library learning research discover shared abstractions across a large number of small programs, which they use to automatically define new subroutines.
Systems such as DreamCoder (Ellis et al., 2021), Trove (Wang et al., 2024), LiLo (Grand et al., 2023), and REGAL (Stengel-Eskin et al., 2024) automatically construct such libraries with the goal of making future program synthesis tasks easier to solve, once the learned library is in hand.
Our work is closest to REGAL (Stengel-Eskin et al., 2024), which clusters related code and refactors using language models.
However, existing library learning approaches have primarily been demonstrated in small-scale, constrained domains, limiting their applicability to typical software engineering tasks, such as consolidating multiple repositories into cohesive libraries. By framing library learning within the context of realistic software development, we expand the relevance of library learning to the everyday practice of software engineering.

#### Repo-level coding benchmarks.

Recent work has explored the application of language models to repository-level software engineering tasks. Existing benchmarks include SWE-bench (Jimenez et al., 2024), which evaluates models on their ability to resolve real-life GitHub issues, and Commit-0 (Zhao et al., 2025), which requires agents to fill in function definitions. Such benchmarks primarily evaluate functional correctness via unit tests, without assessing the quality or maintainability of the resulting codebase.
RefactorBench (Gautam et al., 2025) takes a step in this direction by benchmarking the ability to follow specific refactoring instructions.
Our work differs by requiring models to perform a more open-ended task:
Redesigning code to be more modular and compact by discovering reused abstractions, while retaining verifiability by re-using downstream unit tests. Additionally, libraries must be created without any scaffolding limitations such as preexisting function definitions affording more design freedom than Commit-0.

#### Program optimization.

While our goal is to optimize the quality of libraries, other works focus on improving execution speed through correctness-preserving transformations
(Waghjale et al., 2024; Ouyang et al., 2025; Schkufza et al., 2013).
Both forms of program optimization, compression and speed, are more open-ended than optimizing only for correctness, as there does not exist a ground-truth answer.
Prior work on program optimization benchmarks study code at the file level. We propose a benchmark that transforms programs at a larger scale, across multiple code files.

## 3 Problem Statement

Given $N$ related files $\left\{\rho_{n}\right\}_{n=1}^{N}$, the goal is to create a library $\mathcal{L}$ that captures shared abstractions.
The original files $\left\{\rho_{n}\right\}_{n=1}^{N}$, which define the refactoring problem, we call a task.
The new library must
support all original use cases in the task by extracting latent shared abstractions.
This is accomplished by searching for refactorings that are both correct and ‘natural’. Correctness is straightforward to define via unit tests, but naturalness is more challenging to quantify.

Shorter code is potentially simpler and less redundant. One potential metric is to count the number of tokens, lines-of-code, or syntax tree nodes in the proposed library and refactored code  (Dechter et al., 2013; Polozov & Gulwani, 2015; Bowers et al., 2023; Cao et al., 2023).
But minimizing program size has obvious failure modes: code should also be understandable and extensible, which can be in tension with merely finding the shortest program.(^1^11[Perl Golf](https://wiki.c2.com/?PerlGolf) is a game where participants attempt to write the shortest Perl program accomplishing a given task. The resulting code is famously incomprehensible, even by the standards of Perl.)
Other work in program synthesis (Liang et al., 2010a; Solomonoff, 1964; Ellis et al., 2021) instead optimizes *Minimum Description Length* (MDL), or negative log probability under a reference distribution.
In the software engineering community, other metrics such as cyclomatic complexity and maintainability index have been defined for similar purposes:
These are more complex metrics that examine the syntax tree, call graph, and other statically-analyzable structures (McCabe, 1976).
What metric should we use? We revisit this question in Section [6](https://arxiv.org/html/2506.11058v3#S6), where we empirically compare candidate metrics and human preferences before fixing our choice for the rest of the paper.

For now assume a placeholder metric $M$ measuring refactoring quality; we seek to minimize $M$ while preserving correctness.
Given a task comprising files $\left\{\rho_{n}\right\}_{n=1}^{N}$, we output both a new library $\mathcal{L}$, as well as rewritten refactorings of the original files, $\left\{\rho^{\prime}_{n}\right\}_{n=1}^{N}$.
We define tests passed $\tau(\rho_{n})$ as the set of unit tests $\rho_{n}$ passes, and consider both refactoring several files ($N>1$) and also refactoring a single large file ($N=1$).
We optimize the following objective, which prefers passing at least the same tests as the original programs *and* minimizing the chosen metric $M$:

$$ $\ell\left(\mathcal{L},\{\rho_{n}^{\prime}\}\right)=\begin{cases}M(\mathcal{L},\{\rho_{n}^{\prime}\})&\forall\rho_{n},\tau(\rho_{n})\subseteq\tau(\rho_{n}^{\prime})\\ \infty&\text{otherwise}\end{cases}$ (1) $$

## 4 MiniCode —Library Design and Refactoring Benchmark

MiniCode presents systems with a task comprising a set of files, then asks them to refactor the files into a unified library alongside refactorings of the original files.
There are two key desiderata for benchmark tasks: They should have related files sharing latent abstractions, and should also be verifiable, to measure how well refactored files preserve functional correctness.
We source a variety of problems ([Table˜1](https://arxiv.org/html/2506.11058v3#S4.T1)).

**Table 1: MiniCode Statistics**
| Domain | Files | Tasks | Avg LoC | Avg Tests / file |
| --- | --- | --- | --- | --- |
| Code Contests (Li et al., 2022b) | 300 | 10 | 87 | 10 |
| Transformers (Wolf et al., 2020) | 10 | 1 | 538 | 181 |
| Diffusers (von Platen et al., 2022) | 11 | 2 | 685 | 75 |
| Logo (Wong et al., 2021) | 300 | 1 | 10 | 1 |
| Date (Srivastava et al., 2023) | 246 | 1 | 14 | 1 |

#### CodeContests.

Competition problems are crafted with specific variations of algorithmic approaches in mind, resulting in both shared latent concepts and the required test cases. As a result, competition coding is both verifiable, and ready to refactor.
We therefore take solutions, prompts, and tests from CodeContests (Li et al., 2022a), a competition programming dataset.

#### Huggingface Transformers Library.

We test refactoring across implementations of large language and vision–language models from the Huggingface transformers repository (modelling_<name>.py files, e.g., Qwen2, LLaMA, DeepSeek-V3). Unlike competition coding, these sources are production-scale and Huggingface requires that all changes pass an extensive suite of integration tests before merging into the main branch. A refactoring is only deemed correct if it passes the unmodified Transformers test suite, making this a high-stakes setting that requires correctness and compatibility.

#### Huggingface Diffusers Library.

We test refactoring across implementations of diffusion models from the Huggingface diffusers repository (unet_<name>.py and scheduler_<name>.py files, e.g., Stable Diffusion UNet, DDPMScheduler), yielding two distinct tasks. Like Transformers, Diffusers requires that all changes pass a comprehensive suite of integration tests before merging into the main branch.

#### Logo & Date.

The library learning literature already has existing benchmarks:
Typically they seek to learn a single library from a task comprising many sources, and then test that library on holdout program synthesis tasks.
Logo and Date were used in the recent related work REGAL (Stengel-Eskin et al., 2024), which we incorporate wholesale to understand how our new method compares to state-of-the-art library learning.
The associated programming problems were created by humans, but their solutions were generated by gpt-3.5-turbo.

## 5 Librarian : Refactoring Code to Create Libraries

Librarian generates a new library from a set of files, while migrating the files to use that new library (Figure [1](https://arxiv.org/html/2506.11058v3#S1.F1)), following a sample-and-rerank framework:
Prompting a backend LLM or agent to sample $K$ candidates, and picking the one minimizing the loss $\ell$.
Naively,

$$ $\displaystyle\mathcal{L},\left\{\rho^{\prime}_{n}\right\}=\operatorname*{arg\,min}_{\mathcal{L},\left\{\rho^{\prime}_{n}\right\}\in\textsc{Sample}_{K}\left(\left\{\rho_{n}\right\}\right)}\ell\left(\mathcal{L},\left\{\rho^{\prime}_{n}\right\}\right)$ (2) $$

for metric $M$ and sampling budget $K$.
But this cannot work for large tasks with many programs, which would not fit into the context of most LLMs.
Even long context models cannot process the entirety of e.g the Linux kernel, and even if they could, it is not clear that such a strategy is the most efficient way of focusing the language model’s attention.
To address this, we wrap sample-and-rerank with a clustering algorithm that decomposes the task into manageable chunks, described next.

#### Clustering.

Meaningful abstractions arise when programs share underlying functionality or structure. To surface these, we cluster the task’s files into small groups that are likely to share reusable structure,
and refactor each cluster separately from the rest.
This decomposition shrinks the prompt size, and gives independent searches for the best per-cluster refactoring, which may be more tractable.

Librarian’s clustering extends REGAL (Stengel-Eskin et al., 2024), which clusters programs by assuming each program is paired with a natural language description of the problem it solves, and clustering embeddings of those descriptions.
Since similar problems need not imply similar solution code, we instead prompt a model to summarize each file and cluster by these summaries.
Specifically, we define $\textsc{Cluster}_{S}\left(\left\{\rho_{n}\right\}\right)$ as performing agglomerative clustering (Ward Jr, 1963) on the task’s files $\left\{\rho_{n}\right\}$ to produce a set of set of files, each of which is a cluster of size $S$.
We use text-embedding-ada-002
to embed descriptions of code sources for clustering.

#### Combining clustering with sample-and-rerank.

The simplest approach is to refactor each cluster independently and take the union (concatenation) of each cluster’s library:

$$ $\displaystyle\mathcal{L}^{\star}$ $\displaystyle=\bigcup_{c\in\textsc{Cluster}_{S}(\left\{\rho_{n}\right\})}\mathcal{L}_{c}$ (3) $\displaystyle\left\{\rho^{\prime}_{n}\right\}$ $\displaystyle=\bigcup_{c\in\textsc{Cluster}_{S}(\left\{\rho_{n}\right\})}\left\{\rho^{\prime}_{c,i}\right\}_{i=1}^{|c|}$ (4) $\displaystyle\mathcal{L}_{c},\left\{\rho^{\prime}_{c,i}\right\}_{i=1}^{|c|}$ $\displaystyle=\operatorname*{arg\,min}_{\mathcal{L},\left\{\rho^{\prime}_{c,i}\right\}_{i=1}^{|c|}\in\textsc{Sample}_{K}\left(c\right)}\ell\left(\mathcal{L},\left\{\rho^{\prime}_{c,i}\right\}_{i=1}^{|c|}\right)\text{, for each }c\in\textsc{Cluster}_{S}(\left\{\rho_{n}\right\})$ (5) $$

The approach above ignores the fact that library abstractions discovered in one cluster might be useful in another cluster.
A more sophisticated approach accumulates a library across clusters, and when refactoring a cluster, adds the accumulated library to the prompt.
This lets abstractions discovered earlier carry forward across the collection.
[Appendix˜A](https://arxiv.org/html/2506.11058v3#A1) describes this extension.

## 6 What Makes a Good Refactoring?

We compare different metrics $M$ measuring the quality of a refactoring:

Tokens measures the total number of tokens in the refactored files *and* in the library.
It minimizes program size, but not at the expense of creating a bloated library:
Simply replacing every program with its own monolithic library function would not improve the tokens metric, because it measures library size as well.
Concretely, $M_{\text{tokens}}(\mathcal{L},\{\rho^{\prime}_{n}\})=\textsc{Tokens}(\mathcal{L})+\sum_{n}\textsc{Tokens}(\rho^{\prime}_{n})$.

Minimum Description Length (MDL)
evaluates the negative log probability under a reference distribution, taking into account both the library and refactored sources.
Concretely, $M_{\text{MDL}}(\mathcal{L},\{\rho^{\prime}_{n}\})=-\log p_{\text{LM}}(\mathcal{L})+\sum_{n}-\log p_{\text{LM}}(\rho^{\prime}_{n}\mid\mathcal{L})$, where $p_{LM}(\rho_{n}^{\prime}|\mathcal{L})$ is concatenating the library and the program into one prompt, but only counting the perplexity of the later program tokens.
This has a Bayesian justification: The MDL library is the maximum aposteriori estimate of $\mathcal{L}$ given conditionally-independent programs.
We use Qwen-2.5-3B as our reference language model, as it is modern, open, and has publicly-available endpoints for querying logits, which is required for scoring refactorings. To confirm that our MDL optimization results are not model specific, we computed MDL values for 15 Code Contests clusters, each of 50 valid refactoring candidates, using Qwen and Llama-3.2-3B models. We found a 94% agreement in the minimum MDL candidates from both models.

Cyclomatic Complexity (CC) is a longstanding metric from the software engineering community which measures the number of linearly independent paths through a program’s control flow graph (McCabe, 1976).
Smaller programs often have lower cyclomatic complexity.
It is equivalent to defining $M_{\text{CC}}(\mathcal{L},\{\rho^{\prime}_{n}\})=\textsc{cc}(\mathcal{L})+\sum_{n}\textsc{cc}(\rho^{\prime}_{n})$ and $\textsc{cc}(\rho)=E(\rho)-N(\rho)+2P(\rho)$, where $E$, $N$, and $P$ measure the number of control flow edges, nodes, and connected components, respectively.

Maintainability Index (MI) is a modern software engineering metric combining several other metrics, including lines-of-code, cyclomatic complexity, and Halstead volume into a single score. Higher MI values are intended to indicate easier-to-maintain code, so we define $M_{\text{MI}}(\mathcal{L},\{\rho^{\prime}_{n}\})=-\textsc{MI}(\mathcal{L})+\sum_{n}-\textsc{MI}(\rho^{\prime}_{n})$.

### 6.1 Asymptotic behavior of metrics in large-sample regime

Are these metrics equally effective at encouraging modular and reusable libraries?
To answer this question, we run Librarian on 15 CodeContests (each of three files) using MDL, tokens, maintainability index, and cyclomatic complexity, while varying the inference-time sample budget $K$ ([Figure˜2](https://arxiv.org/html/2506.11058v3#S6.F2),[7](https://arxiv.org/html/2506.11058v3#A4.F7)). We use Best@k estimator for the expected value of metrics for all $k\leq K$ (we describe the estimator and prove its correctness in Appendix [F](https://arxiv.org/html/2506.11058v3#A6)).
Tokens and MDL separate cleanly from classic software engineering metrics:
Optimizing tokens/MDL, both of which essentially compress the original programs, does *not* yield steady improvements in MI/CC, and vice-versa.
To understand whether these libraries expose shared abstractions, we examine the average number of times that each library routine is used, and the average number of library invocations per library function.
This teases apart tokens and MDL:
Optimizing MDL yields more more reusable libraries (used about $8\times$ per task), with each function called more often (called about $2.2\times$ per function)—exceeding the other metrics we consider.
See Appendix [Figure˜7](https://arxiv.org/html/2506.11058v3#A4.F7) for full results, including Cyclomatic Complexity.

Figure: Figure 2: (A) Asymptotic behavior of metrics for scoring libraries and refactorings (columns) varying refactoring budget (horizontal axes). (B) Comparing metrics via proxies of downstream library quality (total library usage and average calls per library function), for which MDL$>$Tokens$>$MI. All results are estimated using Best@k. See also Appendix [Figure˜7](https://arxiv.org/html/2506.11058v3#A4.F7).
Refer to caption: x2.png

Figure: Figure 3: Best@K MDL ratio. Increasing sample budget improves MDL on Transformers.
Refer to caption: x3.png

Studying these metrics at large $k$ allows understanding their inference-time scaling behavior.
While the underlying metric itself improves (with diminishing returns), this was not to be taken for granted:
The backend language model must produce sufficient diversity to steadily improve these metrics.
Prior state-of-the-art, such as Stengel-Eskin et al. (2024), instead take a single sample, but the results here suggest benefit from further test-time search, and indeed,
real-world repos benefit from steady improvement with increased samples (Figure [3](https://arxiv.org/html/2506.11058v3#S6.F3)).
But our proxies of library utility plateau much earlier, around $k=20$ samples, suggesting large $k$ is unnecessary in practice:
Effective library building benefits from test-time compute, but does not demand an exorbitant amount of it.

### 6.2 Human evaluation of refactoring metrics

Figure: Figure 4: Human evaluation of different refactoring objectives. Judges compare pairs of refactorings that both pass all test cases. MDL aligns best with human preferences.
Refer to caption: images/consensus_75_clean.png

We perform a human study to corroborate the findings of [Section˜6.1](https://arxiv.org/html/2506.11058v3#S6.SS1) using the exact same CodeContests clusters ([Appendix˜E](https://arxiv.org/html/2506.11058v3#A5)).
The human study
compares tokens, MDL, and Maintainability Index by (1) refactoring clusters into libraries, (2) presenting human participants with the original sources and their refactorings under pairs of metrics, and (3) eliciting pairwise preferences from human participants.

Humans prefer MDL-minimizing libraries ([Figure˜4](https://arxiv.org/html/2506.11058v3#S6.F4)), and although the preference is only statistically significant for MDL vs. MI, the data suggest a rank-order preference of MDL>Tokens>MI; Tokens vs. MI was slightly below the threshold of significance.
We ran $N=14$ participants (eliciting 129 judgments), and although more participants would make more of these comparisons statistically significant, already we see a general preference for compression-based metrics (MDL and Tokens) with only MDL crossing the threshold of statistical significance.
Indeed, minimizing MDL is not quite the same as minimizing tokens, as
[Figure˜5](https://arxiv.org/html/2506.11058v3#S6.F5) illustrates:
We believe basically every human coder would prefer the MDL-minimizing program in this example.

We therefore adopt $M_{MDL}$ as the primary objective in the remainder of this paper:
In addition to support from this human study, (1) Bayesian arguments support MDL; (2) corner cases in the style of ‘Perl golf’ provide existence proofs of the liability of merely minimizing tokens; and (3) reasonable proxies for library reuse favor MDL ([Section˜6.1](https://arxiv.org/html/2506.11058v3#S6.SS1)).

Figure: Figure 5: Example where tokens and MDL diverge: Obfuscating the original library definitions (left) by shortening variable names (right) reduces tokens but increases MDL.

## 7 What we learn from running Librarian on MiniCode

We empirically study Librarian on MiniCode with the goal of understanding (1) the degree to which library abstractions are reused across programs, (2) how our method compares to state-of-the-art library learning on existing datasets, and
(3) whether Librarian holds value for real-world repos.

#### Librarian discovers reusable functions for competition programming–but some functions are only called once.

We test on CodeContests with a cluster size of $S=3$ and a sample budget of $K=8$ draws from o4-mini, as reasoning models perform well on competition programming.(^2^22Code agents such as Codex, Claude Code, and others underperformed o4-mini ([Appendix I](https://arxiv.org/html/2506.11058v3#A9)))
[Table˜2](https://arxiv.org/html/2506.11058v3#S7.T2) shows that the resulting refactors and libraries approximately halve the MDL, which incidentally reduces program size as well (44% relative reduction in token count).
Pass rate modestly improves as an incidental consequence of sampling and filtering with test cases.
Libraries average 10 functions, each heavily reused:
Averaging 5 uses per function within tasks comprising only 10 programs.
But almost 40% of library functions are only used once.
Why is that?

A signature of the MDL objective is a preference for whatever a language model assigns high apriori probability to.
Although a single-use function does not reduce line count or tokens —the function could simply be inlined—it improves MDL if it yields a more natural decomposition of the target programs.
Indeed, human-written libraries sometimes include functions that are seldom used, provided they serve as a conceptually modular abstraction.
We therefore see single-use functions as a feature, not a bug. See Appendix [J](https://arxiv.org/html/2506.11058v3#A10) for an example refactoring candidate on CodeContests.

#### Are these libraries useful for solving new, unseen programming problems?

Library learning has long sought to learn libraries from training programs which then help solve new unseen program synthesis tasks.
The Logo and Date datasets fit within this paradigm.
Recently REGAL improved the state-of-the-art on these library learning datasets.
Because our clustering is heavily inspired by REGAL, for fair comparison, we keep exactly their clustering setup but add MDL-based reranking using $K=5$ samples.
Despite the simplicity of these datasets, we find value in our more complicated method.
[Table˜3](https://arxiv.org/html/2506.11058v3#S7.T3) shows that sampling and reranking by MDL yields up to a 41.8% relative improvement in solve rate on unseen programming problems, and that even when the gains are more modest, we still improve upon the state-of-the-art.
But these are relatively simple problems solvable with about ten lines of code—does this work in the real world?

**Table 2: Results for Librarian on 10 Code Contests tasks ($K=8,S=3$)**
| Metric | Value |
| --- | --- |
| Pass Rate | 90.67% ±1.88 |
| Pass Rate Improvement | 6.33% ±1.41 |
| MDL Ratio | 0.53 ±0.03 |
| Token Ratio | 0.66 ±0.04 |
| Library Functions | 10.30 ±1.41 |
| Avg Calls per Function | 5.17 ±1.08 |
| % Single Use Functions | 38.03% ±4.88 |

Real-World Refactoring.
The HuggingFace Transformers library is used by nearly 400k GitHub projects.
We deploy Librarian to 10 source files, using Claude Code to sample $K=15$ refactorings per cluster of size $S=5$, believing an agent such as Claude Code would excel at repo-level edits.
Librarian distilled repeated abstractions such as MLPs, Attention, Decoder classes, RoPE helper functions, etc., lowering MDL to 67.2% of its original value while still passing all integration tests. The top-3 refactorings based on MDL have an average of $18\pm 4.4$ abstractions (functions, classes) in the library, each of which is called on average $4.59\pm 0.39$ times in the refactored models.
For Diffusers, scheduler clusters yielded top-3 MDL refactorings with an average of $12.3\pm 1.6$ functions and $3.0\pm 0.4$ calls per function, while UNet refactorings produced richer abstractions with an average of $17.0\pm 5.6$ functions/classes and $3.43\pm 0.67$ calls each.

Refactoring at scale proved expensive:
Each refactoring took approximately 30 minutes to generate and test.
But this is a one-off cost, and in our view, the refactored Transformers and Diffusers sources are much cleaner, and the new library is transparently reusable ([Figure˜6](https://arxiv.org/html/2506.11058v3#S7.F6)).
To the best of our knowledge, this is the first time any library learning algorithm has been successfully applied to real-world software projects.

Figure: Figure 6: Representative result for refactoring HuggingFace Transformers using Librarian
Refer to caption: x4.png

Learned libraries from these real-world codebases are useful for unseen downstream refactoring tasks. When a library learned on one cluster of Transformer files (5 models) is applied to refactor a second cluster, Librarian reduces the unseen cluster’s MDL to 73% of the its original value, with an average of 3.0 calls per library function. This demonstrates that Librarian learned libraries that can be repurposed to more compactly rewrite unseen real-world code sources.

Human performance. HuggingFace is concurrently refactoring the Transformers library. On the same 10 source files used above, their refactoring achieves an MDL compression ratio of 66.5%. To estimate an independent ceiling, we the authors refactor the task ourselves and obtain 62%. This suggests room for a little improvement beyond Librarian’s 67.2%, but not by much (from inspection, specifically with better implementations of mixture-of-experts and vision models).

## 8 Conclusion

We introduce a new benchmark MiniCode and method Librarian for compressing files through reusable abstractions.
We highlight the challenges of producing modular and maintainable libraries, then present an effective method for using LLMs to do this task.
By framing refactoring as an optimization problem, our work opens new directions for building more general and scalable code understanding and generation systems.
In particular, the structure of MiniCode lends itself well to reinforcement learning, where training would entail synthesizing collections of repositories to refactor then computing rewards based on MDL or other metrics.

Limitations.
We evaluate on synthetic toy problems (Logo and Date), and on competition programming problems, neither of which are naturalistic, although this is partly counterbalanced by our study of real-world refactoring.
Compression, whether measured by tokens or MDL, may not always correlate with reuse, a limitation we sought to address through
our experiments on downstream programming problems, and on holdout Transformers files:
But investigating reuse on unseen app-building problems for real-world repo-level refactors remains open.

## Acknowledgements

KE and ZC were supported by NSF grant #2310350.

## References

- Anthropic (2025)
Anthropic.
Claude code: An agentic coding tool that lives in your terminal.
[https://github.com/anthropics/claude-code](https://github.com/anthropics/claude-code), 2025.
An agentic coding tool that lives in your terminal, understands your codebase, and helps you code faster by executing routine tasks, explaining complex code, and handling git workflows - all through natural language commands.
- Bowers et al. (2023)
Matthew Bowers, Theo X. Olausson, Lionel Wong, Gabriel Grand, Joshua B. Tenenbaum, Kevin Ellis, and Armando Solar-Lezama.
Top-down synthesis for library learning.
*Proc. ACM Program. Lang.*, 7(POPL), January 2023.
doi: 10.1145/3571234.
URL [https://doi.org/10.1145/3571234](https://doi.org/10.1145/3571234).
- Cao et al. (2023)
David Cao, Rose Kunkel, Chandrakana Nandi, Max Willsey, Zachary Tatlock, and Nadia Polikarpova.
Babble: Learning better abstractions with e-graphs and anti-unification.
*POPL*, 2023.
doi: 10.1145/3571207.
URL [https://doi.org/10.1145/3571207](https://doi.org/10.1145/3571207).
- Dechter et al. (2013)
Eyal Dechter, Jon Malmaud, Ryan P. Adams, and Joshua B. Tenenbaum.
Bootstrap learning via modular concept discovery.
In *IJCAI*, 2013.
- Ellis et al. (2021)
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum.
Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning.
In *PLDI*, 2021.
doi: 10.1145/3453483.3454080.
URL [https://doi.org/10.1145/3453483.3454080](https://doi.org/10.1145/3453483.3454080).
- Gautam et al. (2025)
Dhruv Gautam, Spandan Garg, Jinu Jang, Neel Sundaresan, and Roshanak Zilouchian Moghaddam.
Refactorbench: Evaluating stateful reasoning in language agents through code.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=NiNIthntx7](https://openreview.net/forum?id=NiNIthntx7).
- Grand et al. (2023)
Gabriel Grand, Li Siang Wong, Matthew Bowers, Theo X. Olausson, Muxin Liu, Joshua B. Tenenbaum, and Jacob Andreas.
Lilo: Learning interpretable libraries by compressing and documenting code.
*ArXiv*, abs/2310.19791, 2023.
URL [https://api.semanticscholar.org/CorpusID:264806856](https://api.semanticscholar.org/CorpusID:264806856).
- Jimenez et al. (2024)
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan.
SWE-bench: Can language models resolve real-world github issues?
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=VTF8yNQM66](https://openreview.net/forum?id=VTF8yNQM66).
- Li et al. (2022a)
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.
Competition-level code generation with alphacode.
*arXiv preprint arXiv:2203.07814*, 2022a.
- Li et al. (2022b)
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.
Competition-level code generation with alphacode.
*Science*, 378(6624):1092–1097, 2022b.
doi: 10.1126/science.abq1158.
URL [https://www.science.org/doi/abs/10.1126/science.abq1158](https://www.science.org/doi/abs/10.1126/science.abq1158).
- Liang et al. (2010a)
Percy Liang, Michael I. Jordan, and Dan Klein.
Learning programs: A hierarchical bayesian approach.
In *ICML*, 2010a.
- Liang et al. (2010b)
Percy Liang, Michael I Jordan, and Dan Klein.
Learning programs: A hierarchical bayesian approach.
In *ICML*, volume 10, pp. 639–646, 2010b.
- McCabe (1976)
T.J. McCabe.
A complexity measure.
*IEEE Transactions on Software Engineering*, SE-2(4):308–320, 1976.
doi: 10.1109/TSE.1976.233837.
- OpenAI (2024)
OpenAI.
Introducing o3 and o4 mini.
[https://openai.com/index/introducing-o3-and-o4-mini/](https://openai.com/index/introducing-o3-and-o4-mini/), 2024.
Accessed: 2025-05-13.
- Ouyang et al. (2025)
Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini.
Kernelbench: Can llms write efficient gpu kernels?, 2025.
URL [https://arxiv.org/abs/2502.10517](https://arxiv.org/abs/2502.10517).
- Polozov & Gulwani (2015)
Oleksandr Polozov and Sumit Gulwani.
Flashmeta: A framework for inductive program synthesis.
*ACM SIGPLAN Notices*, 50(10):107–126, 2015.
- Qwen et al. (2025)
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025.
URL [https://arxiv.org/abs/2412.15115](https://arxiv.org/abs/2412.15115).
- Schkufza et al. (2013)
Eric Schkufza, Rahul Sharma, and Alex Aiken.
Stochastic superoptimization.
In *Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems*, ASPLOS ’13, pp. 305–316, New York, NY, USA, 2013. Association for Computing Machinery.
ISBN 9781450318709.
doi: 10.1145/2451116.2451150.
URL [https://doi.org/10.1145/2451116.2451150](https://doi.org/10.1145/2451116.2451150).
- Solomonoff (1964)
Ray J Solomonoff.
A formal theory of inductive inference.
*Information and control*, 7(1):1–22, 1964.
- Srivastava et al. (2023)
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
*Transactions on machine learning research*, 2023.
- Stengel-Eskin et al. (2024)
Elias Stengel-Eskin, Archiki Prasad, and Mohit Bansal.
Regal: refactoring programs to discover generalizable abstractions.
In *Proceedings of the 41st International Conference on Machine Learning*, ICML’24. JMLR.org, 2024.
- (22)
Shane Tews.
Inside tech’s $2 trillion technical debt | american enterprise institute - aei.
URL [https://www.aei.org/technology-and-innovation/inside-techs-2-trillion-technical-debt/](https://www.aei.org/technology-and-innovation/inside-techs-2-trillion-technical-debt/).
- von Platen et al. (2022)
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf.
Diffusers: State-of-the-art diffusion models.
[https://github.com/huggingface/diffusers](https://github.com/huggingface/diffusers), 2022.
- Waghjale et al. (2024)
Siddhant Waghjale, Vishruth Veerendranath, Zora Zhiruo Wang, and Daniel Fried.
Ecco: Can we improve model-generated code efficiency without sacrificing functional correctness?
*arXiv preprint arXiv:2407.14044*, 2024.
- Wang et al. (2024)
Zhiruo Wang, Daniel Fried, and Graham Neubig.
Trove: Inducing verifiable and efficient toolboxes for solving programmatic tasks, 2024.
URL [https://arxiv.org/abs/2401.12869](https://arxiv.org/abs/2401.12869).
- Ward Jr (1963)
Joe H Ward Jr.
Hierarchical grouping to optimize an objective function.
*Journal of the American statistical association*, 58(301):236–244, 1963.
- Wolf et al. (2020)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.
Transformers: State-of-the-art natural language processing.
In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pp. 38–45, Online, October 2020. Association for Computational Linguistics.
URL [https://www.aclweb.org/anthology/2020.emnlp-demos.6](https://www.aclweb.org/anthology/2020.emnlp-demos.6).
- Wong et al. (2021)
Catherine Wong, Kevin Ellis, Joshua B. Tenenbaum, and Jacob Andreas.
Leveraging language to learn program abstractions and search heuristics.
In *International Conference on Machine Learning*, 2021.
- Zhao et al. (2025)
Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gallé, and Alexander M Rush.
Commit0: Library generation from scratch.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=MMwaQEVsAg](https://openreview.net/forum?id=MMwaQEVsAg).

## Appendix A Incremental Version of Librarian

We support sharing library components across clusters through the below incremental version of our algorithm.
It processes clusters sequentially, rather than in parallel, and its output can depend on the ordering imposed upon the clusters.

$$ $\displaystyle\mathcal{L}_{0}=\varnothing$ $\displaystyle\{\rho_{n}^{\prime}\}=\bigcup_{c\in\text{CLUSTER}(\{\rho_{n}\})}\{\rho_{c,i}^{\prime}\}_{i=1}^{|c|}$ $\displaystyle(\Delta\mathcal{L}_{c},\{\rho_{c,i}^{\prime}\}_{i=1}^{|c|})=\arg\min_{(\Delta\mathcal{L},\{\rho_{i}^{\prime}\})\in\text{SAMPLE}_{k}(c;\,\mathcal{L}_{t-1})}\ \ell(\mathcal{L}_{t-1}\cup\Delta\mathcal{L},\{\rho_{i}^{\prime}\}_{i=1}^{|c|})\quad\forall c\in\text{CLUSTER}(\{\rho_{n}\})$ $\displaystyle\mathcal{L}_{t}=\mathcal{L}_{t-1}\cup\Delta\mathcal{L}_{t},\qquad\mathcal{L}^{\star}=\mathcal{L}_{|\text{CLUSTER}(\{\rho_{n}\})|}$ $$

## Appendix B Algorithm

Figure: Algorithm 1 Refactoring Specialized Programs into a Joint Library

## Appendix C Experimental Setup

#### Grouping Programs into Collections

To facilitate parallel application of Librarian and manage the dataset scale, we assume that semantically distant files will have minimal overlap in their optimal library functions. Therefore, our overall approach partitions the dataset into disjoint collections through clustering.

For CodeContests, these collections are constructed from an initial corpus of $\sim$9k problems with Python solutions: We first filter these files, removing those whose selected canonical solution is under 10 lines (minimal refactoring potential). For the remaining 4596 solutions we use a language model to generate textual descriptions of canonical solutions—emphasizing reusable components—which are embedded using OpenAI’s text-embedding-ada-002.

Agglomerative Clustering (Ward Jr, 1963) is subsequently applied to these embeddings to partition the files into a predefined number of initial clusters, in our case 120. To create uniformly sized experimental units, we subsample each such cluster to form collections of 30 files. This collection size was empirically chosen because it balanced between the runtime of Librarian without limiting compression. We select 10 collections that we then use to evaluate our methods.

For Transformers, since the number of models is on the lower end, we manually chose a set of popular LLM / VLM models and passed them to the agent in collections of 5 code sources.

#### REGAL Baselines.

To evaluate the ability of our libraries to support reuse on new problems, we turn to the program synthesis tasks used in REGAL, where learned libraries are added to help the program synthesizer. We evaluate on the two domains published by the authors, Logo and Date.
Because our clustering is inspired by REGAL but adds additional complexity, for fair comparison, we keep their setup the same and only augment the training using sample + MDL rerank procedure described in [Section˜5](https://arxiv.org/html/2506.11058v3#S5).

#### Code Contests.

To evaluate Librarian on refactoring Code Contests we select 6 collections of 30 files (problems). In each collection we group the problems into tuples of size $3$. We set the sample budget to be $K=8$, since our ablations show that with larger $K$ we discover better libraries [2](https://arxiv.org/html/2506.11058v3#S6.F2). We use the MDL objective for rankings.
The model used for sampling is OpenAI’s o4-mini (OpenAI, 2024).
To obtain MDL scores we use Qwen 2.5 7B Instruct (Qwen et al., 2025)
as a balance between quality, speed, and cost.

#### Code Agents on Transformers and Diffusers Repositories.

To fairly evaluate performance on the task by state-of-the-art systems, we use coding agents that advertise long-context ability to reason about, write, and refactor code repositories. Specifically, we use Claude Code (Cl) (Anthropic, 2025) which uses the Opus 4.1.

We test whether code agents can refactor collections of code sources autonomously, without human intervention.
Refactoring repositories with code agents involves planning and iterative (re-)implementation and testing. Code agents are prompted to perform each of these steps, with feedback from the unit tests. Agents must run and repair unit tests autonomously.
We run coding agents multiple times per task, logging their progress in checklists stored in text files.

The instruction provided to human evaluators is as follows:

Figure: Listing 1: Human Evaluation Instruction

## Appendix D Further Results on Refactoring Metrics

Here we include the full graph of asymptotic behavior of all four scoring metrics (MDL, tokens, MI and CC), reporting the raw library metrics and the metric ratios. We can see that refactored programs have higher CC than the baseline and that optimizing CC as an objective does not decrese the other metrics.
MDL performs best on total raw library usage as well as on average times a library function is used in the refactorings. MI ends up optimizing the number of library functions the best, but their reusability is below the average for the refactorings. Optimizing for tokens produces smaller libraries with less usage per function compared to optimizing MDL.

Figure: Figure 7: Asymptotic behavior of metrics for scoring libraries and refactorings (columns) varying refactoring budget (horizontal axes).
Refer to caption: x5.png

## Appendix E Human Study Details

We ran a user study with 14 participants where each participant had to judge 10 refactoring pairs. Each pair was comparing two out of three metrics, MDL, tokens, and MI. We showed the participants the original programs, as well as both of the refactoring versions (including the programs and the learned libraries). The participants were able to choose either version or say that the two refactorings are almost the same.

To quantify pairwise preferences between refactoring metrics, we employed the Bradley-Terry model, a standard framework for analyzing paired comparison data. We fit the model using maximum likelihood estimation with mutual information (MI) as the reference category $(\pi_{M}I=1)$. To address potential noise in human judgments, we applied consensus-based filtering with a 75% threshold, retaining all responses for comparisons with low consensus (indicating genuine ambiguity) while excluding minority responses on high-consensus comparisons where the majority preference likely indicates the correct judgment. This conservative approach preserved 92.4% of responses while strengthening the statistical evidence for metric preferences, with MDL significantly outperforming MI ($p=0.67$, $95\%$ Confidence Interval: $[0.56,0.78]$).
Even without the filtering MDL preference over MI was statistically significant.

## Appendix F Best@k Compression is a U-Statistic

We wish to estimate the expected compression ratio achieved by our *sample + rerank* method, which samples $k$ candidate refactorings, discards any that do not pass the tests, and selects the one with the lowest score (total log-prob).

Background on U-Statistics. Let $Z_{1},\dots,Z_{n}\overset{\text{i.i.d.}}{\sim}F$. For a symmetric function $h:\mathcal{Z}^{k}\to\mathbb{R}$, the *U-statistic of order $k$* is defined as

$$ $U_{n}\;=\;\binom{n}{k}^{-1}\sum_{1\leq i_{1}<\cdots<i_{k}\leq n}h(Z_{i_{1}},\dots,Z_{i_{k}}).$ (6) $$

By construction,

$$ $\mathbb{E}[U_{n}]\;=\;\mathbb{E}[h(Z_{1},\dots,Z_{k})],$ (7) $$

so $U_{n}$ is an unbiased estimator of the population quantity $\theta=\mathbb{E}[h(Z_{1},\dots,Z_{k})]$.

Application to Best@k Compression. Let each valid refactoring be a pair $Z=(S,C)$, where $S$ is the score and $C$ is the compression ratio. Define the symmetric function

$$ $h_{k}(z_{1},\dots,z_{k})\;=\;C_{j^{*}},\qquad j^{*}=\arg\min_{1\leq j\leq k}S_{j},$ (8) $$

the compression ratio of the lowest-score refactoring among $k$ draws.
The population target is then

$$ $\theta_{k}=\mathbb{E}[h_{k}(Z_{1},\dots,Z_{k})].$ (9) $$

Given $n$ valid samples, our estimator is

$$ $\widehat{\theta}_{k}\;=\;\binom{n}{k}^{-1}\sum_{1\leq i_{1}<\cdots<i_{k}\leq n}h_{k}(Z_{i_{1}},\dots,Z_{i_{k}}).$ (10) $$

Proposition.
$\widehat{\theta}_{k}$ is a U-statistic of order $k$ with function $h_{k}$, and hence an unbiased estimator of $\theta_{k}$.

#### Proof.

(1) *Symmetry of the function $h_{k}$.*
$h_{k}$ selects the compression associated with the lowest score among its $k$ arguments. Permuting the inputs does not affect this outcome (ties can be resolved with a fixed, permutation-invariant rule). Thus $h_{k}$ is symmetric.

(2) *U-statistic form.*
By definition, a U-statistic of order $k$ with kernel $h_{k}$ is

$$ $U_{n}=\binom{n}{k}^{-1}\sum_{1\leq i_{1}<\cdots<i_{k}\leq n}h_{k}(Z_{i_{1}},\dots,Z_{i_{k}}),$ $$

which matches $\widehat{\theta}_{k}$ exactly.

Therefore, $\widehat{\theta}_{k}$ is a U-statistic of order $k$. By the unbiasedness property of U-statistics,

$$ $\mathbb{E}[\widehat{\theta}_{k}]=\theta_{k}.$ $$

$\square$

Thus, our reported *best@k compression curves* provide unbiased estimates of the expected performance of the *sample + rerank* method.

## Appendix G Clustering Analysis: CodeContests

We analyze the coherence of the clusters underlying collections in MiniCode-CodeContests.
In particular, we compare clustering based on o4-mini generated file descriptions against task descriptions.
Since task descriptions in competition coding problems are designed to hide the algorithmic approach needed to solve problem, we expect that clusters based on file descriptions are more coherent. We use Normalized Tag Instance Entropy and Herfindahl-Hirschman Index to evaluate clusterings. [Figure˜8](https://arxiv.org/html/2506.11058v3#A7.F8) shows our clustering approach yields more thematically coherent clusters, evidenced by achieving lower entropy and higher HHI values across the entire tested range of $N$. We provide definitions of our measures below.

Figure: Figure 8: Clustering analysis of 4,596 Code Contest problems, comparing the thematic coherence of clusters formed using our proposed method versus REGAL-style clustering.
Refer to caption: x6.png

### G.1 Collection Coherence Measures

We use two measures
to evaluate the thematic coherence of collections:
Good collections should group files with a (1) concentrated and (2) identifiable set of shared *conceptual tags*, which for CodeContests are provided as ground truth (trees, graphs, etc.).

We provide the full definitions of the collection coherence measures below.

#### Normalized Tag Instance Entropy:

This measures the concentration of tag instances within a collection $C$.
Let $p_{i}$ be the proportion of the $i$-th unique tag type among all tag instances in $C$, and $D_{C}$ be the number of distinct tag types in $C$. If $D_{C}>1$, the normalized entropy $H_{N}$ is defined as:

$$ $H_{N}=-\frac{\sum_{i=1}^{D_{C}}p_{i}\log_{2}p_{i}}{\log_{2}D_{C}}$ (11) $$

If $D_{C}\leq 1$, then $H_{N}=0$. Lower $H_{N}$ (closer to 0) indicates higher thematic purity, meaning fewer tag types dominate the bulk of tag mentions.

#### Herfindahl-Hirschman Index (HHI) for Problem Presence:

This measures tag concentration across distinct problems in a cluster $C$. Let $s_{t}$ be the proportion of problems in $C$ that include tag $t$ (a problem contributes to $s_{t}$ if $t$ is one of its unique tags). A higher HHI signifies that the problems are collectively characterized by a smaller, more focused set of tags.

$$ $\text{HHI}=\sum_{t\in\text{Tags}(C)}s_{t}^{2}$ (12) $$

where $\text{Tags}(C)$ represents the set of unique tags present in cluster $C$.

## Appendix H Benchmark Comparison

We compare our benchmark, MiniCode, to similar benchmarks in Table [4](https://arxiv.org/html/2506.11058v3#A8.T4).
We define creativity and design as the need to explore diverse solutions in order to find the best solution possible.
For example, optimizing for program correctness alone does not require exploring a large solutions space, whereas optimizing a program for speed would.
In the case of compressing large files, we must explore the large space of shared abstractions afforded by libraries in order to maximize compression.

**Table 4: Comparison of Code Benchmarks**
| Benchmark | Creativity/Design | Scale |
| --- | --- | --- |
| SWE-bench (Jimenez et al., 2024) | Low | Repository |
| Commit-0 (Zhao et al., 2025) | Medium | Repository |
| RefactorBench (Gautam et al., 2025) | Low | File |
| ECCO (Waghjale et al., 2024) | High | Function |
| KernelBench (Ouyang et al., 2025) | High | Function |
| MiniCode(Ours) | High | Repository |

## Appendix I Full MiniCode Codecontests Results

We present the full agent scores for the CodeContests split in Table [5](https://arxiv.org/html/2506.11058v3#A9.T5). The results are given both for each cluster of code sources, as well as averaged across clusters.

**Table 5: Comparison of the pass rate and compression metrics of the original files, Claude Sonnet 4 and codex-mini refactorings across CodeContests clusters.**
| Cluster | Agent | Tokens | CC | Pass % | MDL | MDL % |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | original | 9088 | 95 | 80.3 | 11745.85 | 100.0 |
| sonnet 3.7 | 18114 | 176 | 87.0 | 15005.18 | 127.7 |  |
| sonnet 4 | 11121 | 138 | 80.3 | 9901.53 | 84.3 |  |
| codex-mini | 9321 | 95 | 80.3 | 9990.74 | 85.1 |  |
| 1 | original | 12531 | 255 | 89.7 | 13431.86 | 100.0 |
| sonnet 3.7 | 10470 | 239 | 96.7 | 8933.65 | 66.5 |  |
| sonnet 4 | 11325 | 298 | 96.7 | 8214.42 | 61.2 |  |
| codex-mini | 12762 | 255 | 89.7 | 11798.73 | 87.8 |  |
| 2 | original | 14087 | 376 | 89.0 | 15012.77 | 100.0 |
| sonnet 3.7 | 17345 | 429 | 91.3 | 13145.02 | 87.6 |  |
| sonnet 4 | 14270 | 356 | 93.0 | 10522.66 | 70.1 |  |
| codex-mini | 14318 | 376 | 89.0 | 13273.81 | 88.4 |  |
| 3 | original | 14261 | 246 | 90.3 | 13348.82 | 100.0 |
| sonnet 3.7 | 20749 | 241 | 97.7 | 15859.02 | 118.8 |  |
| sonnet 4 | 13433 | 197 | 80.7 | 11937.04 | 89.4 |  |
| codex-mini | 14495 | 246 | 90.3 | 11616.41 | 87.0 |  |
| 4 | original | 17693 | 336 | 80.7 | 14665.16 | 100.0 |
| sonnet 3.7 | 29860 | 358 | 100.0 | 20666.52 | 141.0 |  |
| sonnet 4 | 18684 | 352 | 82.0 | 12801.21 | 87.3 |  |
| codex-mini | 17923 | 336 | 80.7 | 12902.09 | 88.0 |  |
| 5 | original | 12588 | 286 | 92.0 | 12790.11 | 100.0 |
| sonnet 3.7 | 10580 | 128 | 99.3 | 8435.12 | 65.9 |  |
| sonnet 4 | 10416 | 155 | 99.3 | 9167.85 | 71.7 |  |
| codex-mini | 12819 | 286 | 92.0 | 11086.19 | 86.7 |  |
| 6 | original | 11020 | 131 | 54.3 | 13540.41 | 100.0 |
| sonnet 3.7 | 21747 | 502 | 88.0 | 19446.07 | 143.6 |  |
| sonnet 4 | 11177 | 143 | 57.3 | 10492.00 | 77.5 |  |
| codex-mini | 11251 | 131 | 54.3 | 11651.65 | 86.1 |  |
| 7 | original | 12301 | 180 | 80.0 | 12393.73 | 100.0 |
| sonnet 3.7 | 16390 | 166 | 91.0 | 13371.59 | 107.9 |  |
| sonnet 4 | 11625 | 150 | 85.7 | 9304.25 | 75.1 |  |
| codex-mini | 12534 | 180 | 80.0 | 10549.04 | 85.1 |  |
| Avg | original | 12946 | 238 | 82.0 | 13366.09 | 100.0 |
| sonnet 3.7 | 18157 | 280 | 93.9 | 14357.77 | 107.4 |  |
| sonnet 4 | 12756 | 224 | 84.4 | 10292.62 | 77.1 |  |
| codex-mini | 13178 | 238 | 82.0 | 11608.58 | 86.8 |  |

## Appendix J Refactoring examples of Librarian on Code Contests

### J.1 Example 1

In code snippets LABEL:code:v1, LABEL:helper:v1, LABEL:code:v2, LABEL:helper:v2 one example of 2 refactoring versions. Specifically, the versions are both passing at least as many test cases as the original and they have the biggest difference in MDL among all the sample refactorings for that tuple. Sample + rerank filtering selected refactoring V2. You can observe that refactoring V1 introduces some problem specific functions like build_max_beauty_perm(), while refactoring V2 sticks to more generally useful functions.

Figure: Listing 2: Version 1, New Helpers

Figure: Listing 3: Version 1, Refactored Programs

Figure: Listing 4: Version 2, New Helpers

Figure: Listing 5: Version 2, Refactored Programs

### J.2 Example 2

In code snippets LABEL:code:v1_2, LABEL:helper:v1_2, LABEL:code:v2_2, LABEL:helper:v2_2 is another example of 2 refactorings where V1 was better according to Librarian. We can observe that V2 creates helper functions that are overly specific to the problem. You can see that refactoring V2 introduces overly specialized functions like dijkstra_special() or compute_min_moves_opposite_parity(). In comparison, refactoring V1 generates only general versions of these functions (e.g. dijkstra()).

Figure: Listing 6: Version 1, New Helpers

Figure: Listing 7: Version 1, Refactored Programs

Figure: Listing 8: Version 2, New Helpers

Figure: Listing 9: Version 2, Refactored Programs

## Appendix K Obfuscation example

We provide the full source code, both refactored and obfuscated, for the MDL and token comparison here.

Figure: Listing 10: Refactored code from modeling_llama.py from the refactored Transformers repository.

Figure: Listing 11: Obfuscated code from modeling_llama.py from the refactored Transformers repository.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - Could you provide concrete examples showing these improve code quality beyond just MDL score?
- How does LIBRARIAN's performance degrade with larger cluster sizes S? The paper fixes S=3 for CodeContests but S=5 for Transformers without justification.
soundness: 2
strengths: - This paper is well written and achieves impressive real-world validation by refactoring HuggingFace production code with 67% MDL reduction while maintaining correctness.
- This work provides systematic comparison of multiple metrics through asymptotic analysis and human studies, finding MDL superior to traditional software engineering metrics.
summary: This paper presents LIBRARIAN, a method for refactoring multiple code files into reusable libraries, and MINICODE, a benchmark for evaluating library design. The authors investigate what makes good refactoring metric through human study and asymptotic analysis, finding that MDL aligns best with developer preferences. They demonstrate their approach on competition programming and real-world repositories like HuggingFace Transformers and Diffusers.
weaknesses: - This evaluation covers only 10 Transformers files and 2 Diffusers tasks, which seems insufficient to support claims about general applicability to real software projects.
- This human study with only 12 participants lacks statistical power to distinguish between MDL and tokens metrics, yet the authors make strong claims about MDL superiority.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - In coding contest setting, the same problem can be solved using different algorithmic techniques. How well does the clustering work in such a setting? What is the quality of clustering if you were give entire code repositories (e.g., transformers) rather than hand-selected 10 files?
- As noted in the experiments, Librarian can create functions that are used only once. Curious if they are used within the refactored code itself (e.g., private methods) or outside?
- How does the performance vary if you use GPT models instead of o4-mini on CodeContents?
soundness: 3
strengths: - Code refactoring is an important software engineering activity. This paper demonstrates progress on this problem using a pipeline of clustering of code by natural language summary, cluster-specific library extraction and then rewriting the complete code corpus.
- It assembles a benchmark taking code contest solutions, previous refactoring benchmarks and small sets of related files from transformers and diffusers libraries. The resulting refactorings are ranked using MDL and evaluated for correctness using tests. It reports better refactoring accuracy than closely related work Regal on a subset of the benchmark.
- The comparative analysis between MDL, tokens and software engineering metrics is interesting and justifies the use of MDL in ranking.
summary: This paper addresses the problem of refactoring existing code to generate library code that is then used for rewriting the original code snippets. The paper assembles a benchmark called MiniCode and proposes a technique called Librarian. The paper compares different ranking methods and finds that minimum description length (MDL) to be a suitable metric to rank different refactoring suggestions. The evaluation on MiniCode shows effectiveness of Librarian.
weaknesses: - While the problem of refactoring is important, the proposed method is evaluated in limited setting. It does not present results at large scale where refactorings are most important and useful. Though the paper states that the proposed method is evaluated on "real-world code bases", the scope is restricted to a total of 3 tasks with 10 files each from 2 repositories.
- The paper's novelty over past work, Regal, is limited as both of them apply clustering based refactoring. 
- The study of different metrics showing MDL > tokens > MI and that test-time scaling can help are the interesting parts in the paper. As an aside, I could find a citation for the MI metric; please add.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: - Are there any cases where refactorings produced by LIBRARIAN in real projects have been successfully merged into the community codebase?
- Typos: line 218: “set of set of”; line 273: “more more.”
- Missing citation for the MI metric.
- What does “change from non-refactored” mean in Figures 2 and 3, and how is it calculated?
soundness: 2
strengths: - MINICODE offers a practical and effective benchmark for evaluating code refactoring
- The proposed metric MDL is reasonable and interesting.
summary: This paper studies code refactoring in software engineering, focusing on maintainability and reusability. The authors analyze different metrics for evaluating refactoring quality and find that Minimum Description Length (MDL) is the best metric. To evaluate and advance research on code refactoring, the paper introduces the MINICODE benchmark and the LIBRARIAN method.
weaknesses: - The proposed sample-and-rerank approach is relatively simple, and the methodological insights it provides are limited.
- The main risk of using MDL is that it can be heavily influenced by a single model. The paper only briefly discusses cross-model agreement for MDL in Section 6; a more detailed analysis would make the claim more convincing.
- Even if the refactored code passes all unit tests, there is still a risk of semantic inequivalence with the original code. The paper lacks an analysis of this risk.

---

confidence: 2
contribution: 3
initial_rating: 10
presentation: 3
questions: No question.
soundness: 4
strengths: + The investigated problem is clearly defined, novel and important to the community. MINICODE emphasizes open-ended library design, objective verifiability via unit tests, and large-context understanding across multiple files, addressing gaps in prior repo-level benchmarks.
+ LIBRARIAN combines sample-and-rerank with semantic clustering and a progressive, cross-cluster library accumulation strategy, which is practical for long-context constraints.
+ The paper compares Tokens/MDL vs. CC/MI, shows MDL better promotes reusable abstractions, and corroborates this with a human study.
+ Evaluations show strong empirical results. LIBRARIAN shows above 90% pass rate on CodeContests, which is a successful practical application of library learning to real software projects.
summary: The paper introduces MINICODE, an open-ended benchmark for refactoring multiple source files into a reusable library, and LIBRARIAN, a sample-and-rerank approach to synthesize such libraries. A comparative study finds that Minimum Description Length aligns best with developer preferences for refactoring quality; the method is further validated on real-world repositories, showing promising practical implications.
weaknesses: - In practice, a single file often mixes heterogeneous concerns (utility helpers, adapters, domain logic, classes), so the reviewer is not sure that file-level clusters could be too coarse.
- Figure 6 is a case study refactoring the code from the HuggingFace Transformers codebase. However, it is a little complex to understand. Authors are suggested to walk through the end-to-end pipeline that produced this figure.


Decision: Reject