Commit0: Library Generation from Scratch
Authors: Wenting Zhao
Nan Jiang
Celine Lee
Justin T Chiu
Claire Cardie
Matthias Gallé
Alexander M Rush
Venue: ICLR 2025 Poster (2025)
Keywords: code generation
language model
evaluation
feedback
Primary area: datasets and benchmarks
TL;DR: We introduce Commit0, a novel benchmark designed to challenge AI agents to write libraries from scratch.

Abstract: With the goal of benchmarking generative systems beyond expert software development ability, we introduce Commit0, a benchmark that challenges AI agents to write libraries from scratch. Agents are provided with a specification document outlining the library’s API as well as a suite of interactive unit tests, with the goal of producing an implementation of this API accordingly. The implementation is validated through running these unit tests. As a benchmark, Commit0 is designed to move beyond static one-shot code generation towards agents that must process long-form natural language specifications, adapt to multi-stage feedback, and generate code with complex dependencies. Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate. Our experiments demonstrate that while current agents can pass some unit tests, none can yet fully reproduce full libraries. Results also show that interactive feedback is quite useful for models to generate code that passes more unit tests, validating the benchmarks that facilitate its use. We publicly release the benchmark, the interactive environment, and the leaderboard.

Published as a conference paper at ICLR 2025
COMMITO: LIBRARY GENERATION FROM SCRATCH
Wenting Zhao'
Nan Jiang' Celine Lee"
Justin T Chiu
Claire Cardie
Matthias Gallé
Alexander M Rush
'Corell University wz.366ccornell
Cohere
ABSTRACT
With the goal of benchmarking generative systems beyond expert software devel- opment ability, we introduce COMMITO, a benchmark that challenges Al agents to write libraries from scratch. Agents are provided with specification document outlining the library's API as well as suite of interactive unit tests, with the goal producing an implementation of this API accordingly. The implementation is validated through running these unit tests. As a benchmark, COMMITO is designed to move beyond static one-shot code generation towards agents that must process long-form natural language specificationss adapt multi-stage feedback, and gen- erate code with complex dependencies. COMMITO also offers an interactive envi- ronment where models receive static analysis and execution feedback the code they generate. Our experiments demonstrate that while current agents can pass some unit tests, none can yet fully reproduce full libraries. Results also show that interactive feedback is quite useful for models to generate code that passes more unit tests, validating the benchmarks that facilitate use. We publicly release the benchmark the interactive environmen and the leaderboard
INTRODUCTION
AI agents have been increasing rapidly in ability, particularly in domains such as problem-solving, math, and coding. Tasks related to software development have been particularly promising areas due to both their clarity of evaluation and economic value. This has motivated the release of several coding benchmarks in recent years (Hendrycks et al. 2221aa] Chen et al., 2021 Zhuo et al.) 2024). A notable example is SWE-bench (Jimenez al., (024), which assesses the ability of agents to generate patches to resolve real-world GitHub issues. While critical, these tasks generally remain within the skill set of an experienced software engineer. If LLM systems continue to improve at current rates, these tasks will be completely solvable.
We interested in benchmarks that exist further beyond both the frontier of expert human ability as well current model ability. Specifically, tasks that experts struggle to solve but can still be fully specified and reliably verified. Software engineering is an appealing domain for this, as the process of developing actual implementations of functions is very complex. Nevertheless, humans can fully specify the desired behavior of functions and validate them through unit testing.
With this goal in mind, we introduce COMMITÓ, benchmark that tests agent's 's ability generate a software library from scratch. This task is especially challenging - large, real-world libraries are notoriously difficult to design, often requiring hundreds of engineers and years of development. Nonetheless, this task remains verifiable without requiring humans to solve it directly. Humans can provide specifications that outline the library's 's API and write unit tests to verify whether the API has been implemented correctly.
COMMITO extends beyond existing henchmarks in several ways. Central COMMITO is interactive feedback. Due to the complexity of generating a library, it is improbable, or likely impossible, that an agent could generate complete, working version in one shot. Instead, the benchmark is constructed
httee://ungiintise.o/aassstt http:///..../.ttaaee///////////e
httos://qithub.ccmm wwt.:///ccmmimimmmmmmmmm commii t
http:://comit--.giib3e

---

Published as a conference paper at ICLR 2025
An event that hapsen
commit 0
</>
def desecsee1)) > st:: stzing *"
0
/00
class Process (svens): speopezty
</>
def targetteel)) Event:
process...
agent: commit 1 /0/100 tests possed
</>
</>
"o*ke that
</>
def (deac(aeLE) ate:
"Retusn string +Event setumn fself., closs
Boroperty
:eocen((tvent):
commit N 57 / 100 tosts passed
def targotiselr] -> Event: "The event that the process...
PDF
taxpet
Figure 1: An overview of COMMITÓ. Given a starter repository with empty function body, a spec- ification, and suite of unit tests, agents are required produce an implementation of the library that passes all unit tests.
such that must adapt to multi-stage feedback such as unit test errors. Libraries also feature complex dependencies. Implementing one function in a library involves calling other functions, and therefore Agents need identify the right order to implement the functions. Finally COMMITO features long-context processing: agents must navigate specifications of hundreds of pages, and generate thousands of functions, both of which require processing texts in long context.
While our main focus is the benchmark itself, we also introduce a prototype agent SDE-I for com- pleting the benchmark. The agent introduces a basic method for traversing the complex library dependencies, uses best-in-class LLMs to process long contexts, and responds to the interactive feedback of the system. To perform code completion SDE-I uses a state-of-the-ane coding agent.
We empirically evaluate this system on COMMITO. Our experiments show that with a state-of-the- art LLM without feedback, it can pass 17% unit tests in the easier libraries but can only pass 6% in all libraries. We find that iterating on error messages from unit tests improves the pass rate of unit tests to 26% on the easier libraries, demonstrating the utility of leveraging execution feedback. Finally, conditioning on relevant files i.e., ensuring the agent considers related file dependencies and context - further enhances performance.
2 RELATED WORK
Evaluation of LMs. Recent benchmarks for evaluating agents focus on knowledge-intens exam-style questions in domains ranging from grade-school mathematics to quantum mechanics (Hendrycks al., 2021c; Srivastava et al. 2023 Hendrycks et al. 2021b Rein al. 2023). While these questions are challenging, with some requiring PhD level knowledge, they are often short and easy memorize, and they only require few steps of sequential reasoning. In contrast, Commito requires reasoning over a long horizon. Mastering the ability to develop full repositories requires considering many files, many unit tests, and complex static analysis feedback.
Software engineering benchmarks Existing benchmarks for code generation focus on specific aspects of the software engineering pipeline. Program synthesis benchmarks evaluate code gener- ation, for example, HumanEval (Chen et al. (0021). MBPP (Austin et al.] 2021), BigCodeBench (Zhuo et al., 2024), CodeBenchGen (Xie et al. 22024), and Classeval (Du al. 2023). Segmented benchmarks, such as DevBench (Li et al.] (024). separately evaluate different aspects such as code design, code generation, and unit test synthesis. R2F (Jain et al. 2024) introduces more chal-
2

---

Published as a conference paper at ICLR 2025
18
14-
12-
* libraries
librartes
12
10
1000
3000
4000
5000
6000
0.0
1.0
2.5
unit_tests
spec_tokens
libraries
10
librartte
0-
6-
4-
2-
files 250
functions
Figure 2: Basic statistics of COMMITO. The red dotted line denotes the median. Top left: the distribution of the number of unit tests in library; top right: the distribution of the number of tokens in specification; bottom left: the distribution of the number of source files in a library; bottom right: the distribution the number of public functions be implemented in a library.
lenging task by requiring function generation that involves dependencies within and across files. SWE-bench (Jimenez et al., 2024) provides a more holistic evaluation of a model's ability to re- solve pull requests, requiring the incorporation of repository-level context. However, the amount of context necessary to resolve a specific pull request varies greatly and is small on average. These previous benchmarks focus on generating one few functions and are thus manageable via static one-shot code generation. In contrast, COMMITO requires generating an entire codebase consisting of numerous interdependent functions. which necessitates series of refinements based on execution results to pass all unit tests.
Software engineering agents Recent work has made impressive progress in developing software engineering agents that operate on repositories (Yang et al.] 2024 Zhang al., 2023 Wang al., 2024a). Commito proposes a software agent that not only operates at the repository-level but also self-corrects given test feedback. Our method extends prior work on self-correction (Madaan al.)
2023
Shinn et al. 2023) to the larger-scale problem of repository generation.
3 THE COMMITO BENCHMARK
COMMITO benchmarks an agent's 's ability to generate functioning library from scratch. It consists of 54 Python libraries covering wide range of topics, including machine learning, networking, databases, and data visualization. Given, (1) a specification document that contains both texts and images, (2) starter repository with both unit tests and files to fill in, agents are tasked with complet- ing the implementation of the API described in the specification document. Libraries are prepared by removing the core source code from their repo in a systematized manner.
The agent is provided with specification in PDF format and starter repository containing source code directory and a test directory. The task to make edits to the repository. practice, the model generates modified versions of the source code files. We then replace the original files with the modified ones and perform a commit to save the repository state. The model allowed to run unit tests interactively along with code generation. For evaluation, clone this commit in clear repository and run unit tests. The model's performance is measured only by the pass rate these unit tests.
To prevent the model from copying source code, we restrict access to original GitHub repositories via web retrieval. However, the model is allowed the web for general knowledge lookup. For

---

Published as a conference paper at ICLR 2025
instance, if it needs to implement radar plot for visualization, it is allowed to search for relevant information online.
Figure 2 presents basic statistics of COMMITO. The top left of Figure 2 shows the distribution of unit tests across all libraries. Approximately 30 libraries have fewer than 1,000 unit tests, but the distribution shows long tail. The top right illustrates the distribution of tokens in the specifications, with even the smallest specification containing over 10,000 tokens. Most libraries have fewer than 50,000 tokens, though the longest specification reaches up 300,000 tokens. The bottom left displays the distribution of files, where the majority of libraries have fewer than 50 files, but some exceed 100. Finally, the bottom right shows the number of public functions to be implemented. While most libraries have fewer than 250 functions, this number can exceed 1,300 in some cases.
3.1 LIBRARY SELECTION
We focus on Python libraries due to their widespread use, abundant data resources, and strong ecosystem support. To select a set of high-quality libraries for models to implement, we design three sets of filtering criteria.
Library requirements. We restrict libraries Python-only. Specifically, the library needs to contain over 95% Python code. The library also needs to have native Python implementations instead of using Python wrapping libraries in other languages. Finally, the library must support testing with pytest.
Specification requirements. We identify libraries that have comprehensive specifications. The specification must have its own webpage rather than plain README page. The specification document must cover both a user guide which describes how the library is intended to be used and a comprehensive API reference that defines the input and the output of function. The specification should both describe in natural language what are the inputs and outputs and specify the types of inputs and outputs.
Unit test requirements. We include libraries with comprehensive unit tests to test the implemen- tation of library, while having understandable tests that are feasible to run in an interactive system. We limit the libraries to those with over 90% of code that can be covered by unit tests. We filter libraries whose unit tests take over than 30 minutes to run on single CPU and the libraries where significant number of unit tests can only be run on GPU.
To compile the list of libraries included in COMMITO, we consider both generally popular Python 5
ET libraries and PyPI packages with top download count We follow the annotation guideline to filter out the libraries that satisfy the criteria described above.
We create two dataset splits: lite, which includes libraries with fewer functions to implement, and all, which contains all libraries. Lite has total of 16 libraries. Due to the complexity of COMMITO and budget constraints, we focus most of our evaluation on COMMITO lite.
3.2 BENCHMARK CONSTRUCTION
Ensuring Replicability. A key aspect of COMMITO is replicable running of unit tests across all the libraries, which depends on the correct setup of development environments. To achieve this, we annotate setup commands for each library. We begin by annotating specific commit of the library repository, which used to extract installation requirements and generate the starter repository. The installation requirements typically include a compatible Python version, necessary pip packages, and an installation command. Some libraries may also require system-level dependencies, such as clang. Finally, we annotate the pytest command, the directory containing the unit tests, and the source code directory.
Preparing Libraries. We prepare library for COMMITO by removing its core code in sys- tematic way. We assume that a library contains public functions which are accessible to users, and private functions which are not supposed to be called. This is often not enforced explicitly by Python
fhttps: / /hwgovk. github. oo/top-ppoooooooooooooooo pypi -packags/ We include the annotation guideline Appendix.
4

---

Published as a conference paper at ICLR 2025
but is upheld by convention. To determine if function is a public function, check if it has an associated docstring. To prepare COMMITO, we replace the function body of all public functions to be empty (pass) and remove all private functions entirely. We perform these code modifications by first parsing each Python file into an abstract syntax tree, performing transforms on the syntax tree, and converting back source code
Preparing Specifications. Specifications exist in different forms. Some libraries have pure text descriptions while others have extensive figures to demonstrate how the libraries work. For example, seaborn is data visualization library; it uses figures to demonstrate the expected outcomes of API functions. To unify the format, we convert all specifications to the PDF format. Specifically, starting from the main documentation page, crawl the webpage well as all the internal links recursively and save them as PDF.
3.3 INTERACTIVE ENVIRONMENT
key feature of COMMITO is its interactivity. Generating an entire library a single attempt is chal- lenging for an agent; it may need to iteratively incorporate feedback to refine the implementations. COMMITO provides an interactive environment that allows agents multiple sources of feedback, including unit testing, static analysis, and coverage analysis.
Unit Test Feedback Unit tests are crucial for validating the specified behavior of functions. The results from unit tests provide valuable information about implementation issues, including error types and execution traces. Our interactive environment allows agents to execute an arbitrary number of unit tests for any library in parallel. The primary challenge in this process is the need to set up environments for each library to run the unit tests. To address this, we create a Docker image for each library and execute the tests these isolated environments. This setup allows agents to simultaneously develop and run unit tests across all libraries using the pre-built images.
Static Analysis Feedback Our interactive environment also offers comprehensive static analysis feedback, including linting and type checking, as an additional corrective signal. We apply stan- dardized linter and configuration file across all libraries to ensure consistency. Specifically, we use ruff as our linter" For reproducibility, we release both the Docker images and the linter configu- ration file alongside our benchmark.
Coverage Feedback Coverage analysis serves another valuable signal. For instance, if unit test passes one run but fails in another, the difference in coverage can help identify which lines of code are causing errors by comparing the differences in coverage. To provide this coverage information, we use pytest cov and leverage the pre-built isolated environments described above to run the coverage analysis in a reproducible way.
4 THE SDE-I AGENT
Challenges involving complex multi-file dependencies and interactive feedback are challenging for current agentic systems (Yang et 2024). To the difficulty level of COMMITO, we design prototype interactive agent. SDE-I performs a basic software engineering loop. It writes functions, runs unit tests, and iteratively edits the code based on error messages. The agent operates in three stages as shown in Figure 3
Stage I: Draft initial implementations. SDE-I drafts an initial implementation for each func- tion. The first challenge determining the appropriate unit for generation. Implementing all func- tions at once is impractical, as concatenating them often exceeds the model's maximum context length. However, implementing each function in isolation loses the broader context of the other functions. To balance this, we choose to treat all functions within single moduld as one genera- tion unit.
The second challenge is managing complex dependencies between modules since implementing a module often requires understanding which other modules depends on. SDE-I performs
'dnne with the ast library
igi tiithub.com com/ ast -sh/ ruff In Python, file is module.
5

---

Published as a conference paper at ICLR 2025
Stage Fill-inn
Stage 2: Revise based on linting
Stage 3: Revise based on testing
minitorch
autodi 29:22 error:
minitorch/ httpe/eeeerrrreeeeeeeeee __ppppyppppp :4:
operators- PY
tensor_og _ew.or__pppppp PP
(repor wineeaaaaeeee
in Anit
.id cnap
autodifE
autodiÉE, 36:36:45
(repoz ERe tyra
azrOF:
E AttrabuteEiroo::
ops. .nap (operatora.
Fill autodiff. py based on
autodLÉE p:: 49;17 wrror:
type object 'Sipllogga' attribute 'cnap'.
(ssportAtteibutekkcoo wtw.:tttteebbtttcceeaaaaaaa
operators. py
tansor_cpe wwwpppppppppppppp
SDE-I
SDE-I
SDE-I
Sanitorch/auttoiie
Il monittrch/autooe for parent node: for parent nodes:
f mini tortttttt///tdttt.. class Simple 00ps:
pass visited ()
self. cnap
Figure 3: Overview of SDE-I.
topological sort on imports of source code modules. It constructs directed acyclic graph (DAG) of the library, where each node represents a module. If module imports others, the imported modules set its parent nodes. In the case of conditional import cycles, random edge is removed to break the cycle. SDE-I then proceeds by filling in the modules in the order determined by the topological sort. When implementing module, it also includes the content of all modules that the current module imports. In this step, SDE-I entirely ignores whether the generated code is executable or not.
Stage 2: Refine based on static analysis. SDE-I improves the initial implementations by running static analysis to detect and correct issues related to code style, syntax, and type errors. This static analysis helps enforce coding standards and catch potential problems before running more resource- intensive tests. SDE-I appends these results the context and generates revised versions of where issues were located. This process is repeated until all errors are resolved or the maximum number of runs is reached.
Stage 3: Refine based on unit test results. SDE-I refines the implementation further by running unit tests ensure functional correctness. A challenge similar to the first stage arises: running all unit tests simultaneously may generate error messages that exceed the model's maximum context length. However, unit tests are naturally grouped by functionality, with tests for related features typically organized within the same test module. We leverage this structure by executing unit tests one module at time. The results are then incorporated into the context, allowing SDE-I revise the code based on the error messages. This process is iterative, with SDE-I continually revising the code until all tests pass or a predefined limit on test runs reached.
Implementation The SDE-I is implemented to be modular for the underlying coding system and language model. For code generation, we default the Aider framework Aider's interface allows us to define a prompt, a lint command, and test command. We construct a message that includes a prompt to fill in the missing function body, along with texts from specifications and any necessary import modules. For the LLM, we evaluate several model families known for their strong performance on coding benchmarks. Specifically, we consider GPT-4o-mini (Hurst et al. 2024), ol- preview (OpenAl 2024), Claude 3.5 Sonne [0 DeepSeek-V2 (Guo et al. 2024) Llama-3. 1-BB- (Dubey et al. 2024). and Codestra
Instruct, Llama-3. 7-B-----srrutt,, Llama-3.
5 RESULTS
To assess the effectiveness of each stage in the SDE-I agent, we evaluate ablated versions of the method where we apply a fixed number of stages. We summarize the results on COMMITO lite in Table D (Note that we skip stage 2 for OpenAl ol-preview due to its high costs.) Among the three models, Claude 3.5 Sonnet consistently delivers the best performance across all three stages.
aiider.chat
10 htwppppp..oooeeeeeelaadd------o-
mimstral hiissrallaiieews/nneesrrall

---

Published as a conference paper at ICLR 2025
Stage
Stage
Stage 3
OpenAl ol ppeeeiwww
10.34
21.46913.35
Claude 3.5 Sonnet
17.80
1.55
18.79
29.30 99.39
DeepSeek-V22 V2.55
16.55
11.61
25.43
1.43
Llama-3. Hlama-3.1-8B-Imm.
1.--8--Instruc
6.03
1.47
0.23
1.78
0.37
2.77
Llama-3, 1ottna-..-77B----
7.10
10.85
1.83
2,49
Llama-3.1-4000
8.08
7.94
1.7612.20
4,95
29.10
Codestral
6.34
0.30
6.34
0.36
7,41
1.99
Table I: Unit test pass rates across three stages of SDE-I on COMMITO lite. Subscripts are corre-
sponding costs in US dollars.
Library
Total
Stage 1
Stage 2
Stage
babel
5663
0
cachetools
215
173
179
179
chardet
376
3
25
cookiecutter
367
108
102
16
deprecated
171
73
80
151
imapelient
267
0
0
31
jinja
851
marshmallow
1229
456
338
456
minitorch
230
0
parsel
206
10
10
portalocker
36
15
15
pyjwt
259
11
11
128
simpy
140
20
17
94
tinydb
201
27
38
64
voluptuous
149
0
0
0
wcwidth
38
6
Table 2: Pass rate on COMMITO lite across three stages of SDE-I.
Surprisingly progressing from Stage I Stage results in decline in performance with the open- weights models. As discussed in the qualitative analysis in Section 6 although the static analysis feedback provides useful guidance for fixing bugs, the model struggles to apply it effectively, often introducing additional errors. This issue particularly affects the less capable models. However, moving from Stage 2 Stage 3 consistently improves the average pass rate, demonstrating value of utilizing unit test feedback. In the cost-constrained setting, Codestral in stage I has the best performance (6.34%) under S1, and Claude 3.5 Sonnet in stage has the best performance (%29.3) under $100.
Table showws the pass rate for each library using Claude 3.5 Sonnet at each stage. At the individual library level, the results are mixed. In many cases, when models attempt address static analysis issues and unit test feedback, they inadvertently introduce new errors. However, for libraries like deprecated, parsel, and tinydb, Claude 3.5 Sonnet shows the greatest improvement from execution feedback.
Stage 1
Stage
Stage
GPT-4o-mini
2.87 8.73
1.4227.22
4.24
Claude 3.5 Sonnet
6.12
DeepSeek- V2.5
2.33
2.95
4.93 84.11
Table Unit pass rates at the first stage of SDE-I on COMMITO all. Subscripts are correspond- ing costs in US dollars.
Results COMMITO all. We summarize the results in Table Due to the high API cost of Claude 3.5 Sonnet, we only run it for the first stage of the SDE-I agent on COMMITO all. We note that running just the first stage with Claude 3.5 Sonnet already outperforms running all stages with

---

Published as a conference paper at ICLR 2025
Avg. Pass
15
20
stage
spec (first)
spec (bm)c)
+ test Info
orrdr
Import
Figure 4: Ablations on different configurations of what context is provided to SDE-1.
1.0
30.0
10 times
27.5
1 times
times
10-gram Overlap
25.0
0.6
22.5
20.0
10 times
Independent Gen
0.2
times times
Iterative Test
100
150
100
200
300
400
500
600
700
Cost ($)
Length (tokens)
Figure 5: Left: Cost-constrained test-time approaches. The blue line shows test-time scaling in stage
1, where SDE-I generates copies of a module and picks the copy that passes the most unit tests. The green line shows test-time scaling in stage 3, where SDE-I iterates unit test feedback for iterations. Right: 10-gram overlaps between Claude-generated functions and reference functions. Each dot function. We sort the 10- g0-gram overlap by function lengths.
GPT-4o-mini or DeepSeck-V2.5. the the best performance (2.87%) under $1, and Claude 3.5 Sonnet in stage has the best performance (%6.12) under $100.
setting, GPT-4o-mini stage has
Results on statc-of-the-art software development agent: OPENHANDS. We test OPENHANDS [Wang et al., 2024b), the best-performing agent on SWE-bench. To have OPENHANDS take full advantage the specifics COMMITO, we share the list of unit test IDs, static analysis commands, and test commands with the agent. With this deep integration, OPENHANDS passes 42.95% of unit tests on COMMITO lite (a 13.65% improvement compared to SDE-I) and 15.25% on COMMITO all (a 9% improvement). To understand this improvement, OPENHANDS improved from 64 passed tests to 174. We attribute the success of OPENHANDS to its better debugging capabilities. Unlike SDE-I, which often repetitively generates the same fix for a bug, OPENHANDS is able to explore different solutions.
analyze the TinyD8 example, where
6 ANALYSIS
Ablations. We conduct ablation studies based on the first stage of SDE-I in Figure First, we investigate whether including information from the specifications and tests can help LLM agents pass more unit tests. Since the length of the specification often exceeds the maximum context length of the LMs, we feed only the first 10,000 tokens from the specification. For test, we append the prompt to include test modules. Surprisingly. both additions reduce performance. We hypothesize that much the specification and tests are irrelevant to implementing specific modules, which may distract the model. To better leverage the specification, the agent will likely need to first filter out only the relevant information. To verify this hypothesis, we perform retrieval to obtain 10,000 break the specification into chunks of 1,000 tokens and retrieve the top 10 tokens
tokens. Specifically, chunks to include in the context. With the same number tokens, using yields higher unit test pass rate, suggesting that agents benefit from relevant context.
8

---

Published as a conference paper at ICLR 2025
Next, explore whether filling in functions following the order found by topological sort is help- ful. Interestingly, we find that a random-ordering approach leads to more passed unit tests (22% compared to 17%). Upon further inspection, found conditioning incorrect implementations of previous modules was the issue. Relying on incorrect implementations appears to be more harm- ful than relying empty ones. Lastly, assess the impact of excluding relevant modules from the context. As expected, excluding these relevant imports results in fewer passed unit tests.
Table 4: Qualitative example for using static analysis feedback to revise function implementations.
Test Results Before
PASSED cest://eet_uudib..yy::tt
PASSED testss/estt
Lint Feedback
cooklecutter//ttll..y:..33
ANN201 Missing
return type annotation for public function 'workiin'
37 @contextlib.cootexxtaaat
38
def work_in (diamme-oni))
6*00060 ANN201
39
"""Context manager version of os.chdir. help: Add return type annotation
Revised Implementation
@context contextmanagerr def work_in (dirname-Nene))
def work_in
dirname: Optional [Union (str, "os. Pas.PathLike(str] -> Any:
None
"""Context manager version of os. chdir..
@0 -4,,11 +50, 12 33 def work in (der
try:
if dirname is not None: os. chdir (dirnane)
yield
yield None
finally: os. chdir (curdir)
Test Results After
Failed :t
TypeError: 'NoneType' object is not an iterator
Failed wtt.essstutisssytttee.tt wookrtwortttttttteoott in without path
TypeError: 'NoneType' object is not iterator
Test-time Scaling. An interesting question is how unit test pass rates scale with more test-time compute. We address this question with two experiments. First, we sample module 1, 3, and 10 times, picking the best implementation based on pass rates before proceeding to the next module. Additionally, we test whether continuous iterations on unit test feedback will eventually enable agents to pass all unit tests. We conducted an experiment where we applied unit test feedback over different numbers of iterations: 1, 3, and 10. We summarize the results on the left of Figure We observed that, in both cases, unit test pass rates improve with more test- best-time compute.
Library Memorization. COMMITO tests models on existing libraries, versions of which are may be part of their training data. This raises the possibility that LLMs may be simply recalling these libraries from memory, but reasoning about the specifications and unit tests. (We not that similar question exists for many other coding benchmarks.) To explore whether current models are primarily memorizing the libraries, we calculate the 10-gram overlap between the generated and reference libraries. We create mapping from function IDs to their corresponding function bodies for public functions. We then compute the 10-gram overlap between the generated implementation and the reference implementation for each function. The overlaps sorted by function length. The results, presented on the right of Figure 5 indicate that LMs either fully memorize the functions produce implementations that significantly different. Shorter functions more frequently

---

Published as a conference paper at ICLR 2025
memorized. Given the low total test accuracy, the observed degree of memorization is not major factor for the benchmark.
Table 5: Qualitative example for using unit test feedback to revise function implementations.
Test Results Before
FAILED
_tattth_or__oddd cond
FAILED
Unit Test Feedback
event. event._value = event._calliee ceabbbcccelbbcceeeeeeeee (event)
AttributeBrror: 'Initialize' object attribute '_callback', Did you mean: 'callbacks'?
Revised Implementation
event. _ook True
event. event._value event._ catets___abbbceeeeee event pprntt_processed True
try:
if hasattr (event, '_callback') event__value event eventt__allbacht (event)
event._ok True
elif hasattr (event, 'callbacks'); for callback in event event.callbacks: callback(event) event._ok True
elif isinstance (event, Process):
if event._tärges is None:
raise iuntimekrror Invalid yield value "None"") eventt_resumeedeee resume (event. (teveet._targgt)
else:
event. ok True except Exception
event False ovent._value if not getattr ' '_defused', False) raise
finally:
event. processe True
Test Results After
PASSED hett://eee_coodddooooorrrttttttt witr_wtth_ooom cond
PASSED tests/test
wtttloo_iihhhh_ddh__eeee and_
Qualitative Analysis. In the first example, we show how static analysis feedback might hurt per- formance. Presented in Table 4 the function implemented in stage successfully passes the unit tests. However, the agent misinterpreted the static analysis feedback, which requested type annota- tions. In response, the agent adds Any as the return type and modifies the function include yield None to match the type. Unfortunately, since generator cannot return NoneTType this introduces a new error. In the second example, we show test feedback can improve performance. Presented in Table [5] the unit test feedback points out that the attribute "_callback" is missing. The agent thus revised the implementation by adding an statement to check for relevant attributes, hence passing the unit tests.
7 CONCLUSION
We introduce COMMITO, a challenging task that requires LMs to generate libraries from scratch. Our task provides signals through unit test feedback, including both enror messages and execution traces. It also offers comprehensive static analysis feedback including type checking. This task is meant to be beyond the level of most human experts, and currently seems beyond what state- of-the-art LLMs are capable of. We hope that it can serve as both as progress benchmark for AI development well as spurring agent architectures and methods.
10

---

Published as a conference paper at ICLR 2025
LIMITATIONS
One limitation of COMMITO is that we exclusively include Python libraries in our evaluation, which may limit the generalizability of our findings to other programming languages. Additionally, our experimental environment requires a well-specified document and full suite of unit tests for each library, which may not reflect the typical development process where documentation and tests are often incomplete or evolving. While these assumptions may not be entirely realistic in every real- world scenario, they are closely aligned with the principles of test-driven development (TDD), where rigorous testing and clear documentation are integral parts the development process.
Furthermore, reliance unit tests the primary means of evaluating the correctness of the generated code introduces additional limitations. Unit tests may not cover all possible use cases or edge cases, potentially allowing erroneous suboptimal code to pass unnoticed. This overreliance on testing may mask underlying issues that would be evident a thorough code review or through integration testing. As result, the agent's ability to generate code that passes all tests does not necessarily equate to producing robust, high-quality software.
Another concern is the risk associated with having agents generate code that humans may find dif- ficult to read or maintain. Automatically generated code might be syntactically correct and func- tionally adequate pass tests, but may lack clarity, proper documentation, standards and best practices. This can lead to maintainability issues, as future developers may strug- gle understand or modify the codebase, increasing the risk of introducing bugs or vulnerabilities in subsequent updates. The opaqueness of aechine-generated code also raises questions about ac- countability and traceability software development.
adherence coding
ETHICS STATEMENT
COMMITO consists of forked public repositories whose licenses permit our use. Our study does involve human participants, and does not rely on human task workers for data collection. We do not gather any data, including personal data, from GitHub.
Additionally, code generation without human oversight may raise potential ethical issues. For ex- ample, agents could introduce security vulnerabilities or violate licensing terms. Ensuring that the generated code is safe, compliant, and ethically sound may require human intervention, which our current benchmark does not account for.
Finally, automating software engineering is challenge that has both potential benefits and harms. Our release COMMITO serves to measure progress towards this challenge.
REPRODUCIBILIITY
We release the COMMITO benchmark in its entirety, along with all methods and results. We also provide the code for reproducing the dataset, so that it may be used for synthesizing data.
ACKNOWLEDGEMENTS
We thank Ofir Press, Xingyao Wang, Graham Neubig, Tanya Goyal, and Sanjiban Choudhury for the helpful discussions. We thank Modal for providing with credits to unit tests using their ser- vice. We thank annotators from Cohere for providing annotations for additional libraries. AMR, CL, and NJ supported by NSF CA355R-2037519, NSF-2242302, and NSF Grant DRL-2229873.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski. David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. htt.ttss:: htt.t:p::///xxiv..rrrg
07732

---

Published as a conference paper at ICLR 2025
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually-crafted benchmark for eval- uating Iims on class-level code generation. ArXiv, abs/2308.01861, 2023. URL https:
7/api. www.eaatttccccolll..aeeoooeerroe
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint 2xXiv:247.227780 2024.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseck-coder: When the large language model meets programming the rise of code intelligence, 2024. URL https:
//rxivv.oq/ abs 2211.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurlPS, 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021b. URL httts:://ppenrreeeee d7KB Jm 3GmQ.
net/ fww.mrrrmiooooo
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021c.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21272 2024.
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica, R2e: Turn- ing any github repository into programming agent environment. In Forty-first International Conference on Machine Learning, 2024. URL KXHqEYPyE3
net forummiiiii did=
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. https:::::/ ptttps:://opnrevviee
net/foo www.rooiuouu....... ByNQM66
Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang. Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du Ping Yang, Dahua Lin, Chao Peng, and Kai Chen. DevBench: A comprehensive benchmark for software develop- ment. ArXiv, abs/2403,08604, 2024. URL htttp:://appp segaiissommtitcccoosss.. orgg/ CorpusID:268379443
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffee Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirth-seventh Conference on Neural Information Pro- cessing Systems, 2023.
OpenAl. Openai ol system card. Technical report, September 2024. URL https: //ppnai....... com/ researchh www.ooeeniiiiieee waw.aaaiillssssss--eeeeeeeeeeeee Introduction to the ol model series, focusing on safety and robustness through advanced reasoning with chain of thought and risk management protocols.
12

---

Published as a conference paper at ICLR 2025
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-prood q&a bench- mark. arXiv preprint arXiv:2311.1202 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Re- flexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. / net/ forum?
cKW6
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma. Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Opendevin: An open platform for software developers generalist agents, 2024a. URL https: / vv. org/ abs/ / 2407 16741
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software 2024b.
developers generalist agents. arXiv preprint arXiv:2407.1677
Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, and Carolyn Rose. CodeBench- Gen: Creating scalable execution-based code generation benchmarks. ArXiv, abs/2404.0055 000666, 2024. htt.tttt::////t
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. URL
15793
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung. Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2471 1471-4,,, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v Ttt5533/1/2223emm////. 12022..emmlmmmmai. 151. URL https://aaaaaattooooo..... org 2023. wwwwpeennaa..eeeee mwa..ain..... 151
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari. Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiao ke Hong, Wen-Ding Li, Jean Kaddour, Minglian Xu, Zhi- han Zhang, Prateek Yadav, Naman Jain, Alex Gu. Zhoujun Cheng, Jiawei Liu, Qian Liu, Zi- jian Wang. David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiao-Nan Du, Harm de Vries, and Leandro Werra. BigCodeBench: Benchmarking code generation with di- verse function calls and complex instructions. ArXiv, 15872406.158777, 2024. URL https:
13

---

Published as a conference paper at ICLR 2025
A
DATA ANNOTATION
Table lists what annotated for each library in order execute the unit
Annotation
Description
Repository Name
Owner and repository name
Commit or Tag
Annotate either specific commit or version tag (recommended), Python version that is compatible with the specified code state Path to 'requirements.oxt that contains packages to be installed Additional pip packages to install
Python Version
Packages
Pip packages
Install
Installation command (must be in editable mode and include test dependencies) System-less dependencies (e.g.,
Pre-install
clang, etc.)
Specification
URL link to the project specification, preferably PDF link
Test Command
pytest command for running unit tests
Test Directory
Directory where unit tests are located
Source Directory
Directory where the source code located (e.g., web3/ for web3 ..y py library)
Table 6: Annotations for setting executing unit tests
B IMPLEMENTATION DETAILS
Here is your task:
You need to complete the implementations for all functions (1.c., those with pass statements) and pass the unit tests. Do not change the names existing functions or classes, as they may be referenced from other code like unit tests, etc.
IMPORIANT: When you generate code, you must maintain the original formatting of the function stubs (such waitespaces), otherwise will not able search/replace blocks for code modifications, and therefore you will receive a score of 0 for your generated code.
Figure 6: The prompt provided to SDE-I at stage 1.
Prompt.
We present the stage-1 SDE-I prompt in Figure 6
C LIBRARY ANNOTATION GUIDELINES
We share the library annotation guidelines that we provide to annotators. To incorporate new libraries into COMMITO, one can follow these guidelines to produce the necessary annotations for the libraries.
12
D ABLATIONS
Fill in by Module
Fill in by Pass Statement
Avg. pass
17.80
12.42
Table 7: Comparison of average pass rates between filling in by module and by function.
htttps://tiiynnn - guidell ines
14

---

Published as a conference paper at ICLR 2025
Generation units. Here analyze the generation unit for filling in libraries from scratch. We compare filling in by module and by pass statement and present the results in Table 7 We find that filling in by pass statement leads to significantly worse results than filling in at the module level. When generating code for one pass statement, models tend to focus solely on local contexts and often overlook interactions with other parts in the modele. In contrast, generating code the module level enables models to develop a holistic understanding of a file. For example, in the following porta cortalocker: py file, when filling for the only pass statement, the agent terminates without any modification.
import import typing from import constants, exceptions
LockFlags = constants. LockFlags
class HasFileno (ropiongProoooo)) pass LOCKER: lrpiin.optional(tyy/ee [typing.Callaabble. [typing. Union [int, HasFileno], int], (yping.Any]] None
if os. name = 'nt';
import msvert.
import pywintypes
import win32con
import: win32file
import winerror
overlapped www.ntypes..vERL....
elif os.name == 'posix':
import errno
import fentl
LOCKER fenti.
else:
raise RuntimeError (' Portalocker only defined for nt and posix platforms')
As result, all the unit tests failed in the collection stage:
ImportError while loading conftest hetpssere_essts/oooottsstttsee from portalocker import utils in <module>
in <module>
Bortalocker__init____.
lock = portalocker. lock
AttributeError: module 'portalocker portakkoce...tttcce.....k has attribute 'lock'
However, when the agent is asked to fill in the whole module, returns the following message:
Thank you for providing the task details and the content of the relevant files. I'll focus on completing the implementations for the functions with 'pass' file. Since there are no explicit
statements in the "portalocter/pportaoocee... 'pass' statements in the provided code, I'll assume that we need to implement the locking and unlocking functionality for both Windows (nt) and POSIX systems.
Let's start by implementing the 'lock' and 'unlock' functions for both platforms.
Following this message, the agent produces the code:
if os. name = '
[...]
def lock (file, flags)
if flags & constants. LockFlags........ BXw.EUIIIS..EEIISIVEE....
lock_type win32con.
15

---

Published as a conference paper at ICLR 2025
else:
lock_type = 0
if flags
NON BLOCKING:
lock_type I= win32con.
IATELY
hfile (f (ileo ())
try:
lock_type, 0, -0x10000, overlapped)
except pywintypes. as exc_value:
exc_value se_vall..winerror winerror. ERROR ERROR_LCCK LOCK_VIOLATION:
raise exceptions LockException (fh=file)
else:
raise
def unloc(ffiee:
hfile msscrt..get.offaaddee.c.. (file. ())
try:
Minlffilee. (hfill,, 0, -0x10000, overlapped)
except prwintypes.error as exc_value:
if exc_value.winerne
winerror, EORTO
# File was not locked.
pass
else:
raise
elif os. name == 'posix':
[...]
def lock (file, flags):
locking_flags = fcnt1.LOcK EX
if flags & constants constants. wow.attsss..occcca........o
else fent1.Lock_sH
if flags . NOi_NNN_____AA_
NON_BLOCKTT,,,, locking_flags = f
BLOCKING:
try:
fontl.floct
(), locking_flags)
except I0Error as exc_value:
if exc_value. errno == arrno.EACCo or exc_value.errng == errno. EACAIN:
raise exceptions LockException (fh=file)
else:
raise
def (fice)
fent1.flock (file. fileno (),, fent1. (oCK1.,OCK_UN)
else:
raise RuntimeError (' ('Portalocker only defined for nt and posix platforms')
16


Reviews:
[{"review_id": "MGkPEexYDN", "reviewer": "Reviewer_FDu7", "summary": "This paper introduces commit0, a benchmark for evaluating programming agents on generating software libraries from scratch given comprehensive specifications and unit tests. The authors then present SDE-1, a prototype agent that demonstrates different aspects of designing agents for this task such as drafting an implementation, feedback via linters, refining with execution feedback.", "questions": "Que: In Figure4, it's unclear what the columns \"stage 1\" and \"+ spec\" suggests. It seems that the only prompt for a commit0 task is the specification of the library. However, the \"stage 1\" column suggests a setting where you don't pass the spec at all? What is the prompt in this case?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Ambitious benchmark even if not for current gen models.** I particularly like that the benchmark pushes beyond current codegen evaluation setups by targeting full library implementations. Although this might seem too difficult a task currently, having such a benchmark could produce interesting solutions in the space (a la swebench), esp. for such long-horizon tasks.\n\n2. **Multiple sources of feedback.** It's nice to see the benchmark itself integrate several sources of feedback such as linting, coverage, and unit tests. This is usually moved to the agent/system being evaluated, but I think having some standard ones such as linting within the benchmark makes comparing solutions more standardized.", "weaknesses": "**Missing actionable insights.** While commit0 tackles a newer task compared to related work like HumanEval, SWEBench, and R2E, I'm not convinced the insights gathered are interesting or new. While the community can attempt to extract more, it would be nice for the paper to suggest a few directions even with primitive experiments. For instance, the diminishing returns on iterating with execution feedback has already been shown in prior work. I'm wondering if there's any evidence that this task will bring new insights for designing code generation systems/agents or training better models.\n\n**SDE-1 Design** The authors mention that to balance the model's broader context of other functions, they use an entire module as one unit of generation with module-level dependency analysis. I am not convinced that this is a good choice. A module can be quite large in popular repositories and libraries (such as the ones in commit0) spanning a few 100 lines making it quite complicated to generate. My concern is that the result \"random-ordering leads to more passed tests than topological sort\" is a result of the unit being too large to generate, leading to more incorrect implementations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "stGFinuhfD", "reviewer": "Reviewer_ffwo", "summary": "This paper introduces a new benchmark commit0, evaluating agents abilities of writing python libraries from scratch. In this benchmark, agents are provided with starter repos, specifications, and unit tests to start with. Then, the agents' final commits with the implementation of the libraries would be tested by the unit tests. They also designed a prototype agent SDE-I that operates in three stages.", "questions": "- It would be nice if the authors could provide further explanation on how they prevent agents from accessing existing online implementations of the python functions via web browsing.\n- It would be nice if they authors could provide more example instances from commit0 and provide example trajectories by SDE-I.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The authors curated a new benchmark commit0 that could assess agents' abilities of implementing various python functions. This benchmark could help with evaluating and thus further improving agents' code and repo generation abilities.\n- The authors introduced a new framework SDE-I which could assist the repo generation process for agents.\n- The authors perform some ablation studies that might provide insights into how agents utilize additional information of specifications/tests", "weaknesses": "- The authors did not provide comprehensive evaluation of their proposed agent SDE-I on the full benchmark commit0 that they curated. They only provide full results of SDE-I of stage 1 on the full benchmark. Thus, their results might not fully reflect the actual abilities of SDE-I on commit0. It would be good to have the results of all stages of SDE-I on commit0. \n- The authors did not evaluate other existing agents on commit0, which makes it less clear how current agents perform on such tasks. It would be interesting to have results of agents based on frameworks like ReACT or CodeAct on commit0.\n- The authors do not provide example instances from the commit0 benchmark, nor example trajectories by SDE-I on commit 0. It would also be nice to have code for synthesizing new data and example benchmark evaluation code. With example instances provided, it will be more clear what are tasks like in commit0. It would be good to have 1 or 2 examples in the main text of the paper and include more in the appendix.\n- The authors did not provide the average cost of SDE-I on commit0. For example, it would be good to have a table containing the average cost of the proprietary models.\n- The authors did not make clear how they prevent agents from accessing existing online implementations of the python functions via web browsing. It would be good to have a more detailed explanation maybe in the appendix.\n- When performing ablation study, the authors found a decrease in performance of the agent when additionally provided with the specifications or tests, and hypothesized that this might be due to the specifications/tests not relevant to implementation. However, the authors only feed the first 10000 tokens of the full specification / test to the agent. The prompts being incomplete might affect the performance. It would be more interesting to see if including relevant details of the specification / test would help improve agent performances.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uStNvcstNN", "reviewer": "Reviewer_xSyA", "summary": "Summary: The Commit0 paper proposes a new benchmark for the coding proficiency of large language models. Simple existing coding benchmarks are typically done on smaller well defined problems with a simpler interaction model of providing a problem description to the LLM and getting back the LLM’s generated code solution. Commit0 tries to create problems to solve with the complexity a human software engineer would have to solve in a real job, like implementing a whole set of related functions from the spec for the library. Commit0 also includes the iterative nature of software development where the LLM’s code solutions are run through lint checks, and test case checks and the test results and LLM generated code are passed back to the LLM to allow it to iterate and attempt to fix any issues found.\n\nCommit0 also includes an example LLM agent SDE-I to implement the functionality needed to solve the benchmark proposed. The SDE-I agent calls leading LLM models from places like Anthropic and OpenAI to provide baseline scores for the Commit-0 benchmark. The SDE-I agent defines the basic dataflow, prompts and interfaces needed to complete the Commit0 benchmark tasks. \n\nThe goal of this benchmark is to more accurately assess how well LLMs can do solving coding problems that are a lot more like what real human SDE have to do at work. Improvements in the benchmark will come from better LLMs being developed, and improvements to SDE-I to create better prompts and work-flows to pass to the LLMs the best context in terms of related code and task descriptions of the functionality required in the context window passed to the LLM.", "questions": "Q0: What does it cost to run the Commit0 tests to get the numbers presented in the paper? It would be nice to see more baselines like from Llama 8,70,405 or Mistral or the leading coding LLMs, even if just for the lite/small testset.\n\nQ1: I think defining a smaller subset of the benchmark was smart in the paper, given the cost to solve the whole thing. Would it make sense or be useful to take current simple benchmarks (humaneval, etc) and plug them in as tiny benchmark sets in Commit0, so the power of being able to iterate on the lint and test results can be checked out by smaller labs. Also to debug and tune future SDE-I agents, it might be nice to have proof of concept small Commit0 test sets to test against.  I suggest repackaging common coding datasets in Commit0 format because some effort has been made to exclude (humaneval, etc) from LLM training datasets, so as a fast easy Commit0 dataset they could serve some use.\n\nQ2: Would a benchmark for Commit0 leaderboard of best results for $20 compute, $200 compute make sense?  Like should I use OpenAI’s most expensive model for fewer calls/iterations or use OpenAI’s smaller less expensive models for more calls/iterations? Basically level the playing field so it’s the best agent for a fixed cost – use any LLM you want – but you only have $20 or $200 to work with. Design a test subset and pick a dollar $$$ amount that is reasonable for labs to participate.  In the SDE-I agent basically you run through all the problems in the benchmark in 1 pass and then you just keep iterating on your results till you run out of money.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Commit0 is a much more realistic measure of how well an LLM will do software development on a task that matches what real human software developers have to do in their jobs.\n\nIncluding the iterative nature of software development by passing back lint and test errors to the LLM to allow it to attempt further edits and improvements to the code is great.\n\nHaving a whole set of functions to implement to support a scenario is great to force the LLM to handle the complexity of tasks real human software engineers have to deal with.", "weaknesses": "W0: The computational burden to solve the full Commit0 test set is immense, so the cost to run this benchmark might be prohibitive for many labs to participate. What does it cost currently?\n\nW1: Real developers have to produce specs for the high-level design and shared data structures, and also unit tests and docs for all the private functions they are writing. That sort of thing I guess could be all done inside the SDE-I agent, but the ability for the agent to ask itself, what is the spec for this internal function, what are the unit tests for to verify this internal function is correctly implemented and allowing the agent to iterate on the errors it finds by running it’s own unit tests on it’s own internal (and public) functions and being able to ask itself if the function it wrote completely implements all aspects of the spec in a “reflexion” sort of way I think will be important to enable surpassing human levels of coding in the SDE-I agent.\n\nW2: With training LLMs there are these scaling laws about if you use a bigger model, or train for longer on more data – the results just get better.  Similarly for the SDE-I agent if it’s allowed to independently generate each function multiple times and then pick the best implementation in a “reflexional” sort of manner, or allowed to iterate more times to fix errors, and to restart from scratch if it is still failing a function’s tests after N calls – the error rate can be pushed lower with more CPU/GPU power applied. So comparing different SDE-I agents – it seems like there has to be budget of how many tokens-in/tokens-out are allowed to compare different approaches. If you are running your LLM local, then maybe it’s wall clock time.  If you are calling models in the cloud, maybe it’s total cost in $$$, but it just feels like to find the best SDE-I design the total compute between different approaches has to be held constant, or the total cost of calling models in the cloud has to be constant, for fair comparisons between SDE-I agents and the LLMs they use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)