Repo2Run: Automated Building Executable Environment for Code Repository at Scale
Authors: Ruida Hu
Chao Peng
XinchenWang
Junjielong Xu
Cuiyun Gao
Venue: NeurIPS 2025 spotlight ()
Keywords: Code data scaling
Large language model
Executable environment building
Primary area: infrastructure
TL;DR: An LLM-based framework that automates executable environment building to enable scaling up executable code data.

Abstract: Scaling up executable code data is significant for improving language models’ software engineering capability. The intricate nature of the process makes it labor-intensive, time-consuming and expert-knowledge-dependent to build a large number of executable code repositories, limiting the scalability of existing work based on running tests. The primary bottleneck lies in the automated building of test environments for different repositories, which is an essential yet underexplored task. To mitigate the gap, we introduce Repo2Run, the first LLM-based agent aiming at automating the building of executable test environments for any repositories at scale. Specifically, given a code repository, Repo2Run iteratively builds the Docker image, runs unit tests based on the feedback of the building, and synthesizes the Dockerfile until the entire pipeline is executed successfully. The resulting Dockerfile can then be used to create Docker container environments for running code and tests. We created a benchmark containing 420 Python repositories with unit tests for evaluation. The results illustrate that Repo2Run achieves an 86.0% success rate, outperforming SWE-agent by 77.0%. The resources of Repo2Run are available at https://github.com/bytedance/Repo2Run.

Repo2Run: Automated Building Executable Environment for Code Repository at Scale
Ruida Hulx
Chao Peng21
Xinchen Wang
Junjiclong Xu²
Cuiyun Gao
Harbin Institute of Technology, Shenzhen
ByteDance, Beijing {200111107, 200111115) hit.edu. en
{pengchao... x, xujunjielong @byteddnceeee......ceeeeerr..ee com gaocuiyun@hit.. edu. cn
Abstract
Scaling up executable code data is significant for improving language models' software engineering capability. The intricate nature of the process makes it labor-intensive, time-consuming. and number of executable code repositories, limiting the scalability of existing work based on running tests. The primary bottleneck lies in the automated building of test environments for different repositories, which is essential yet underexplored task. To mitigate the gap, we introduce Repo2Run, the first LLMM-asedd agent aiming at automating the building of executable test environments for any repositories scale. Specifically, given a code repository, Repo2Run iteratively builds the Docker image, runs unit tests based the Dockerfile until the entire pipeline is executed successfully. The resulting Dockerfile can then be used to create Docker container environments for running code and tests. We created a benchmark containing 420 Python repositories with unit tests for evaluation. The results illustrate that Repo2Run achieves an 86.0% success rate, outperforming SWE-agent by 77.0% The resources of Repo2Run are available httpt::://///....t / thub. com/ bwtteeaaeeeeeeeeeee.....
Sanowledge-depenet
to build a large
the feedback of the building, and synthesizes
Introduction
Code Repo Crawling
Environment Building
Executable Instance
crawiig
Source code
manually wriling
test running
tests/test 1.py
tests/test
GitHub
Repo
Unit tests
Developer
Dockerfile
Repo
X tests/test_n..t
2. Dependencies
Base
Installation
3. Test running and error handling.
4. Dockerfile
selection.
variables export.
written manually.
Figure 1: The pipeline of code repository mining and manual environment building. Developers manually write Dockerfiles through iterative steps including base environment selection, dependency installation, test running, error handling, validating the environment by running unit tests.
Large language models (LLMs) have recently illustrated significant progress on solving software engineering issues [[] driving the advent numerous coding LLM agents like MetaGPT (2), SWE- agent (3). OpenHands [4). Copilot (5), and Cursor (6). However, the lack of training environment
Work done during an internship at ByteDance.
i Corresponning authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Dependency missing or conflict
Base image Incompatible
Environment variable missing
65
Source code syntax error
Povvle
Sockar
ceceet last
Creape Fnnnckee
HRUIE Demuu wel seceei Fooe cord cony Alpeniirr
tete CLEDA sockee
X
Reposhory
nerw Dherr from fowum
meaul rout
Buldd
aggnt
LLM
2 LL55
Se uoposting
LLM
pibbull Co onialatiio
putft
Heampping rolemaa
Figure 2: Four error types (highlighted in different colors) that SWE-agent fails to resolve during environment building.
3.10
to solve:
doctor
FiJsA
divs
COCKY
Filss
Eire
change
d1d complete successfuliy: exit coda: 1
ginal envronment
execute tailed
"oo knnd envronment
Docierbe
Emorlo
(a) A failed execution of "commarda" leads to "pollution" the environment.
(b) Building fuilure caused by adding "commandA" the Dockeffile.
Figure 3: The illustration of command executing failed and "polluting" the environment. (a) Failed commands like "commandA" can irreversibly "pollute" the environment by altering packages, files, or directories, making subsequent builds unstable. (b) To reproduce the changes, such failed command "RUN commandA" needs
needs to be added to the Dockerfile. However, adding it will lead to building failure.
remains a significant challenge to advancing LLMs in software engineering As shown in Figure 1 while static code is readily accessible through the GitHub API, building executable environments for code testing and patch validation demands extensive manual effort even for skilled developers F This
forces researchers to choose between (1) relying on unreliable static code metrics for rewarding (e.g..
SWE-Fixer 77) and SWE-RL 8 and (2) engaging developers manually build limited repository environments (c.g., SwE-Gym O and SWE-Smith (10)) without scalability. Thus, an automated environment building infrastructure is urgently needed but remains underexplored.
In this paper, we introduce Repo2Run, an LLM -Las- agent designed to build executable environ- ments. The key idea is that if the agent can successfully navigate the process of building repository's dependencies and running its unit tests, its actions can be recorded and replayed to programmatically synthesize a runnable Dockerfile. This enables automated creation of isolated and consistent environ- ments across different platforms via only one line command "docker build", eliminating "il works on my machine" problem П.I.
Repo2Run addresses two key challenges in automating executable environment building:
1. Hard to explore valid trajectory that can successfully build the environment. We found that without effective actions, the LLM agent would struggle to handle such a complex process. As shown in Figure 2 SWE-agent unresolved issues during environment building due to its lack of specialized actions. To address this challenge, Repo2Run uses an internal environment for building and an external environment for assistance, with nine actions designed to resolve these issues.
3 a well-known LLM--bsse agent, often struggles with
2. Failing to synthesize a runnable Dockerfile without execution failure. As shown in Figure 3 we observed that some incorrect commands in the trajectory can "pollute" the environment, leading to errors during synthesis. To tackle this challenge, we design an adaptive rollback action. When encountering failed commands like "commandd"" that may cause "pollution" to the environment, Repo2Run restores the internal environment to its pre-execution state. This ensures the building process remains consistent and repeatable using the Dockerfile, avoiding unexpected errors.
To demonstrate Repo2Run's 's effectiveness, we first created a benchmark of 420 latest Python reposi- tories with unit tests from GitHub in 2024, as existing datasets include only a very small number of repositories [I. We choose Python for two key reasons: (1) the most popular software engineering benchmarks (e.g., SWE-bench ( focus on Python repositories, and (2) Python is one of the most widely used programming languages today [12]. We then evaluated Repo2Run this benchmark, finding it successfully built the environments for 361 repositories, achieving an 86.0% success rate,
We conducted a study show how industry developers struggle in building environments. See Appendix F

---

which is 63.9% higher than the leading baseline. Notably, Repo2Run outperforms the SWE-agent by 77.0%, demonstrating the effectiveness of the proposed agent.
We believe Repo2Run will serve as the foundational infrastructure, driving future research at the intersection of Al and Software Engineering and facilitating the community efficiently scale up their executable code data and environments.
2
Formulation
In this part, we define the task of executable environment building. Given repository R, executable environment building aims determine suitable base image and building process P such that the resulting environment state S satisfies the verification E, defined in Equation (T)
e eRR, S) = 0, where S 8(B, P)
(1)
Here, 6 is the state transition function, and =RR, S) 0 indicates that the verification is satisfied.
2.1 State transition
Environment state (S): The environment state E S represents the current state of the computer system, which encompasses all variables, files, cache, etc.
Command sequence (C): The command sequence C E C represents a set of individual commands. Each individual command C C refers to instruction action that be executed in the environment via interfaces like bash, thereby changing the system state.
State transition (ō): The state transition ò is function that defines the process through which system transitions from the start state Sstart E to the end state Send E after executing of command sequence C, as defined in Equation (2):
8:8x C S, 5(Sstart, C) Send
2.2 Base image
Empty state (Sp): The empty state Sg E S represents completely bare operating system or purely hypothetical state without any builds.
Base image (B): The base image B E S is a special type of environment state, typically managed by professional teams for user convenience, such "python:3. 10". Starting from the empty state Sg. the base image B can be created by executing a predefined command sequence Cn E C, i.e., B 8(S. CB). Users can utilize these base images by simply adding their names to the Dockerfiles.
Building process and state verification
Building process (P): The building process P C is the command sequence designed build the environment from the base image B. We denote the resulting state as Sy E S, where Sy = 6(B, P).
State verification (e): The state verification is a Boolean function used to determine whether the resulting state S successfully runs all tests in the repository R. e R, Sy) indicates that all tests in the repository R can be successfully run. Otherwise, at least one test fails.
3 Repo2Run
In this part, introduce the design of Repo2Run. Given code repository, Repo2Run synthesizes runnable Dockerfile, enabling the building of an executable environment. As shown in Figure this process is divided into phases: the build phase and the record phase.
In build phase, Repo2Run employs a dual-environment architecture consisting an internal environment and an external environment. The internal environment is a Docker container where the agent can execute actions restricted to a container, e-g., test running. The external environment terminal where the agent can execute actions prohibited within container, e.g., base image change.

---

External Environment
G Finish running all tests the Docker container.
Preset actions the envtronmen
Internal Environment
Dockerfile synthesizer
testuztert J.dr
Repo
Image Python:3.10.
e commands
resuit processor
testertenrt-l
Invoke the actions the environment
docker
dependency installation
Dockerfile
the trajectory.
base image change
Nultiple inviroment-ggem Son.
Build executable environment run the tests.
test running
Executable Dockerfile
monitoring
editing
observations,
Build phase
Record phase
Figure 4: The workflow of Repo2Run, involving two phases: the build phase and the record phase. The build phase utilizes a dual-environment architecture: the internal environment with five actions for environment building, while the external environment with three actions assists the internal environment. The record phase converts the validated command sequence into runnable Dockerfile for reconstructing the executable environment. See Appendix A for more examples of these actions.
Once all tests in the internal environment successfully executed, Repo2Run transitions to the record phase.
In record phase, Repo2Run synthesizes runnable Dockerfile based on the command sequence executed during the build phase. This Dockerfile serves as precise record the build process, and its execution can be viewed as replay of the original build phase.
Next, we introduce the external environment, internal enviromment, and Dockerfile synthesizer.
3.1 External environment
The external environment serves as bridge between the LLM and internal environment. It transmits actions from the LLM the internal environment and relaying observations back. Following with the ReAct framework [3). the external environment maintains history of thoughts, actions (i.e., successfully executed commands), and observations. This iterative process ensures that the LLM stays informed about the system's 's state and makes accurate decisions while effectively interacting with the internal environment.
As shown in Figure commands like "commandA" " may "pollute" the environment, making it difficult to reproduce using Dockerfile. When command fails (i.e., returns non-zero return code), the environment transitions to an uncertain state. Therefore, we introduce a rollback action ensure the building process remains consistent. Specifically, before executing command, we use "docker commit" to create a reserve snapshot of the current state (i.e., save the image at that point). the return code is not 0, indicating failure, the environment rolled back the most recent reserve snapshot. However, certain commands, referred to as safe commands (e.g.. "cat"), generally do not alter the state of the environment and are exempt from rollback. For complete list of safe commands, see Appendix B.2
Rollback:
Base image change: If the LLM agent determines that the current base image is unsuitable during the environment building process, it can reselect and switch to a new base image. This change invalidates the previous building process, clearing all executed commands and requiring restart.
Result processor: Command execution during interactions can produce extensive output (e.g., error logs), potentially overwhelming the LLM agent. To mitigate this, long outputs are truncated by this action, retaining only the initial and final sections up to specified length.
'We collected some commands that "pollute" the environment, Appendix

---

Begin
FROM
Begin
ENV
Begin
COPY
Begin
RUN
Defouit
export tame 322
python coft_elic.pp
instoLL RUN pip InstaLL A tEKE
FROM python:3.11
3.
ENV Coken 123
2ython:3.12
change Jove. Smage
#xport Come 324
COPY cade_ pectr-yy
/code, edit.py
(fer conttoring)
FROM 33thon:3.21
ENV token 124
COPY patch /patch
RUN Pup instalL Aww1.5
Figure 5: Rules for Dockerfile synthesis, illustrating how the Dockerfile synthesizer maps executed commands into Dockerfile statements using four keywords: "FROM", "ENV", "COPY", and "RUN". Black arrows represent the creation of statements, while green arrows indicate their transformations. Red text next the arrows specifies the commands executed during each step.
3.2 Internal environment
The internal environment is Docker container. Based on the latest data [4] from 2025, select Python 3. 10 as the default Docker base image due to its broadest adoption among Python versions.
Environment monitoring: It serves as the eyes of the LLM agent in the internal environment, allowing it to observe the current environment state. These commands typically do not change the state of the environment. Basic commands like "Is" and "cat" are used to inspect directories and files, "find" used to locate files, while more advanced commands like "pip list" and "pipdeptree"
[15] retrieve the versions of installed libraries and dependency relationships
serves as installing third-party packages required for running tests, using tools like pip for Python and apt-get for system-level packages. To resolve potential conflicts (e.g., version constraints). a dependency management is implemented assist the installation, where packages to be installed are added to waiting list and resolved as needed. If installation succeeds, the environment updated; otherwise, the rollback is performed. For detailed designs, see Appendix G
Dependency installation:
Test running: serves as both a compass and a checkpoint, guiding the building process and verifying whether the environment runs all tests successfully. By executing "pytest" for unit tests, determines whether the Docker container has been correctly built. If all tests run successfully, the process concludes; otherwise, error logs are sent to the LLM agent for further adjustments.
Code editing: It enables the LLM agent to modify the code within the internal environment, including both inside and outside the repository. Direct code editing is rare, but sometimes needed for issues like syntax errors. To prevent the LLM agent from bypassing tests by directly altering or deleting test files, it is restricted from modifying or deleting the original test files within the repository.
Bash commands: Like many other LLM- agents 244, Repo2Run allowed to invoke bash commands, allowing it fully operate within the Docker container. It enables the agent to execute dynamically defined actions that are not pre-specified, ensuring flexibility handling various errors.
3.3 Dockerfile synthesizer
Once the LLM agent successfully runs all tests, the Dockerfile synthesizer converts the command se- quence from the building process into runnable Dockerfile. It processes each command sequentially, following the rules in Figure synthesize Dockerfile statements. The synthesis uses four Dockerfile keywords: "FROM", "ENV". "COPY", and "RUN". For more details and examples, see Appendix B
FROM: The FROM statement defines the base image and is typically the first line a Dockerfile. the LLM agent changed the base image, the FROM statement must be updated, and all subsequents are cleared to ensure consistency.
ENV: The ENV statement persistently sets environment variables in the Docker container. When command with "export" is detected, it is converted into an ENV statement. If the vaule of an environment variable is overwritten, this statement is updated accordingly.
COPY: The COPY statement copies local files directories into the Docker container. As shown in Figure 5 it is often used for code editing to import editing scripts (i.e., code_edit, edlit.yyt and patches.
RUN: The RUN statement executes commands in the container, with each line creating new bash session. cannot used for persistent environment variables. Specially, for every installed packages, versions are recorded to ensure reproducibility, and the RUN statement is updated accordingly.
5

---

Experiment
We evaluate the effectiveness of Repo2Run on 420 Python code repositories. As the popular option, we select gpt-4o-2024-05- 20-1-- for all experiments, with the temperature uniformly set to 0.2.
4.1 Benchmark
To the best of our knowledge, there is no prior work similar to Repo2Run that automates executable environment building. Existing manually constructed datasets are limited to very few repositories [ To validate the capability of Repo2Run, repositories from GitHub based on the following criteria:
create new benchmark consisting of filtered Python
Creation date: To avoid the data leakage, we select repositories created in 2024, ensuring they are not part of mainstream LLM training data.
Star count: To maintain quality, we only include repositories with more than 100 stars.
Test directory: We focus on repositories likely to contain unit tests, identified using pytest, leading Python testing framework compatible with tools like unittest. Pytest detects test files with "test_" prefix " "_test" suffix, typically located in "test" or "tests" directories. We only retain repositories that have these directorics.
Using these criteria, we crawled 449 repositories in December 2024 and filtered 420 containing at least one unit test to form our benchmark. For statistic of their scale, see Appendix
4.2 Evaluation metries
Dockerfile Generation Success Rate (DGSR): It indicates the percentage of attempts where the method successfully generates runnable Dockerfile. To be considered successful, the generated Dockerfile must be able to build without errors. If the Dockerfile for a code repository successfully builds, it regarded as a successful generation. Generating runnable Dockerfile is fundamental for successfully building the executable environment.
Environment Building Success Rate (EBSR) It represents the percentage of attempts where the method successfully builds executable environments. For a successful building, the generated Dockerfile must not only build successfully but also allow tests container. We are only concerned with whether tests can be executed, regardless of whether they pass or fail, as outcomes of tests may inherently vary within the repository.
by "pytest" in the Docker
4.3 Baselines
is an automated tool that generates "requirements. txt" file by analyzing the import statements in the Python scripts and identifying the necessary dependencies without LLM. Using the requite txt file generated by pipreqs, we create a Dockerfile. The detail provided in Appendix I.I
pipreqs [6]:
The "README" file in code repository usually contains environment building instructions. Therefore, we directly drive the LLM to read the "README" file and generate an executable Dockerfile accordingly.
LLM generator:
SWE-agent establishes a custom agent-computer interface (ACI) that uses the LLM agent's interaction with the repository environment by allowing actions such reading files, editing files, and executing bash commands. Initially intended as an LLM agent for bug fixing, we preserve its framework and default settings, adjust its prompts, as shown in Appendix [.2
SWE-agent 3:
4.4 Experimental Results
The results of different baselines are presented in Table [ We observe that Repo2Run consistently outperforms other basclines on both DGSR and EBSR. Repo2Run ultimately completed environment building for 361 code repositories, achieving an EBSR 86.0%. is 63.9% higher than the highest rate achieved by other methods, demonstrating great advantages. Due to the design of
We have also conducted direct evaluation based on test pass rates in Appendix E

---

Table 1: The results of different baselines, including the percentage and number of successfully generated Dockerfiles and successfully built environments.
# Successfully Generated Dockerfiles
# Successfully Built Environments
Metric
DGSR
EBSR
pipreqs 29.8%
125
6.0%
25
LLM generator
47.6%
200
22.1%
93
SWE-agent
26.9%
113
9.0%
38
Repo2Run 100%
420
86.0%
361
Table Repository distribution across different domains and their EBSR, where SE means Software Engineering, NLP means Natural Language Processing, and DV means Data Visualization.
Type
AT/MI
SE
NLP
DV
Security
Others
All
Success
221
70
41
10
7
12
361
Total
267
73
47
11
14
420
EBSR
82.8%
95.9%
87.2%
90.9%
87.5%
85.7%
86.0%
rollback, Repo2Run successfully generates Dockerfiles that can be built successfully for all 420 code repositories. which other actions cannot guarantee. For details of each repostories, see Appendix H
For pipreqs, the main failures come from two reasons. First, generating the requirements.te txt fails when there are issues within the repository, such as encoding errors or syntax errors in the files. This happens in 30 repositories (7.1%). Second, even when requirements.ng is generated, it might not be downloaded properly due to package version conflicts. This occurs in 265 repositories (63. 16). Besides, both the LLM generator and SWE-agent fail to ensure that the generated Dockerfile can be successfully built due to the lack of an ensuring mechanism. Surprisingly, the ability of SWE-agent, general agent framework, generate Dockerfiles even weaker than simply letting the LLM read the "README" file. This indicates that a general agent framework cannot guarantee the generation of runnable Dockerfiles. Ensuring mechanisms like rollback is necessary effectively use the interactive information from the agent to generate runnable Dockerfiles.
As shown in Table 2 manually categorized the 420 repositories into six domains. EBSR across these domains all exceed 80%, outperforming all baselines and demonstrating Repo2Run's 's consistent performance across different domains. Prior research [17 [8] shows that the real-world benchmarks differs from synthetic ones and better reflect a model's true capabilities. Our benchmark is built entirely from real-world GitHub data, filtered only by repository creation date, star count, and test, with no additional filtering. Thus, it accurately represents the current distribution of Python repositories. Moreover, previous real-world benchmark (18) includes 59 59.6% of AI/ML repositories, closely matching the 63.6% proportion in our benchmark. This aligmment further confirms that our benchmark reflects the real-world scenarios.
4.5 Ablabtion of Repo2Run
To investigate the impacts of the dual-environment architecture and Dockerfile synthesizer separately, as two parts of Repo2Run, we separately remove each component them. For the experiment without the dual-environment architecture, we retain only the internal environment's bash commands as the most basic interface and remove all other actions. For the experiment without the Dockerfile synthesizer, we directly instruct the LLM to synthesize runnable Dockerfile.
Experimental result of the ablation study is shown in Table We observe that removing the dual- environment architecture and retaining only bash commands results in a 7.6% decrease in DGSR. The main reason for this drop is the removal of rollback and other designs, making the system more prone to entering uncertain states and subsequently failing to reproduce. In addition, EBSR shows a 44.3% decrease, primarily because the simplification of design makes more difficult for the LLM agent to execute all tests in the internal environment. Besides, removing the Dockerfile synthesizer directly leads to an 80.5% drop in DGSR. This indicates that having the L.LM directly generate Dockerfiles is unlikely to fully follow the event history, resulting in Dockerfiles that fail to build successfully.
7

---

Table The results of ablation experiments, including the percentage and number of successfully generated Dockerfiles and successfully built environments.
Successfully Generated Dockerfiles
Successfully Built Environments
Metric
DGSR
EBSR
w/o drra-ennir
92.4%
L 7.6
388
41.7%
4 44.3
175
- wo rollback
96.9%
3.1
407
83.6%
1 2.4
351
w/o Dockerfile generator
19.5%
4. 80.5
82
13.8%
4 72.2
58
Repo2Run
100%
420
86.0%
361
This also directly causes sharp decline in EBSR. Specially, also perform ablation study for the rollback. The results show that removing the rollback causes 3.1% % of generated Dockerfiles to become unrunnable, demonstrating the effectiveness of the rollback mechanism.
is also observed that Repo2Run without the Dockerfile synthesizer is outperformed by the LLM generator. This is because the LLM generator leverages the "README" file, which provides clear, simple and high-level overview of the executable environment building, allowing for more accurate Dockerfile synthesizer. In contrast, the event history- based approach lacks this context, making harder for the LLM to fully understand the building goals. However, the Dockerfile synthesizer effec- tively utilizes the detailed event history, highlighting the complementary roles of both components of Repo2Run in generating reliable runnable Dockerfile.
Discussion
5.1 Time consumption
Figure 6 shows the distribution of time spent successfully building 361 code repositories. The average time for successful building using Repo2Run is 29.03 minutes. 111 (30.7%) of the repositories are successfully built in less than 10 minutes. Additionally, our empirical study through sampling indicates an average manual building time of 21.33 minutes (Appendix FF) Considering network differences and random- Figure 6: Time distribution of successful building. ness, Repo2Run achieves a time consumption comparable to manual building. Additionally, for complex issues, Repo2Run shows greater advantages over manual building. Repo2Rum success- fully builds all the cases that were manually successful in our empirical study.
100-
111
Frequenn aare 25
48
30
30
40
80
100
Time (minl
Table 4: Analysis of failed cases in Repo2Run.
Table 5:
Comparison of Dockerfile qual- ity. "Warning" counts in each group show Repo2Run vs. baseline.
Category
# Case (%)
Hardware Insufficiency
21 (35.6%)
Baseline
"Warning"
Missing Token
1 (1.7%)
Repository Defects
5 (8.5%)
pipreqs
5.17 /.0
Dependency Installation Timeout
(13.5%)
vs. LLM generator
5.42 10.16
Runtest Timeout
24 (40.7%)
SWE-agent
5.43 6.49
5.2
Failure case study
For the code repositories that fail to build, we manually inspect the reasons for failures and find that most are due to issues within the repositories themselves. Table demonstrates the analysis of 59 failed cases Repo2Run. The first three categories (45.8%) primarily fail due to external factors, while the latter two categories (54.2%) are unable to be configured successfully within the allotted time due excessive dependencies and extensive tests inherent the repositories. We also provide detailed case in Appendix
8

---

5.3 Dockerfile Quality
We utilize hadolint [99] to analyze the quality of the Dockerfile generated by Repo2Run, comparing with Dockerfiles successfully generated by three baselines and counting the number of warnings. Fewer warnings indicate higher quality in Dockerfile generation. As shown in Ta Dockerfiles generated by Repo2Run's 's consistently have fewer warnings than those generated by baselines. Specif- ically, compared to pipreqs, LLM generator, and SWE-agent, Repo2Run reduces warnings from 6.0 5.17. 10.16 5.42, and 6.49 to 5.43, respectively. The results indicate that Repo2Run produces higher-quality Dockerfiles with fewer potential issues.
5.4 Limitation
Repo2Run can automate building executable enviromments for many repositories but may still encounter challenges in some edge cases. For complex environments that remain unresolved. manual verification and intervention are necessary. Nonetheless, as important step toward full automation, we believe this work makes significant contribution to the community's s infrastructure.
Related Work
6.1 L.M--aseed agent
LLM-based agents
typically consist of four key components: planning, memory, perception, and action 22 Planning is crucial for agent systems, as it schedules agents to ensure a smooth
20 21
process. LLM-based agents employ various planning strategies, including single (23 or multiple
planners 24 single [25] or multi-turn planning [26]. and single [27] or multi-path planning (28).
The memory component LLM-based agents stores historical data to support coherent reasoning and complex tasks. Implementations vary in terms of memory duration (short-term [29 or long-
term memory 300), memory ownership (specific [31 or shared memory (32 For perception,
LLM-based agents primarily utilize textual input 133 30) (natural language and programming
language) and visual input [34] 35 (images and diagrams) to perceive and process information. To extend capabilities beyond interactive dialogue, the action component employs various external tools [36 37,, such as searching tools, file operations, and GUI operations. Nowadays, LLM-bbsee agents have demonstrated superior performance compared to standalone LLMs in various software engineering tasks (38 Many complex software engineering problems, especially repository- level challenges (48H51). can thereby be effectively addressed. However, LLM-base agents are specifically designed for executable environment building currently. To fill this gap, this paper employs novel approach for automated executable coding environment building and runnable Dockerfile generation.
6.2 Environment building and Dockerfile generation
There have already been many efforts devoted to environment building. Oss-Fuzz-Gen [52 relies on predefined build instructions (such " ht/bbbtsttr....... "./configure" "make") build projects for fuzzing but lacks of flexibility for diverse projects when specified files are absent. Existing solutions that help developers write Dockerfiles broadly fall into three categories: (1) Template-based generators that create Dockerfiles based on project context [53] [54, (2) Task-specific tools such as pipreqs [16), DockerizeMe (55) which supports environment dependency inference for Python projects, and DockerGen (50] for dependency recommendations based on knowledge graphs built from existing Dockerfiles, whose overall effectiveness still has substantial room for improvement [57], and (3) Code completion tools including GitHub Copilot [5] and HumpBack [58] that generate suggestions for developers while writing Dockerfiles. The use of deep learning models to generate Dockerfiles based on natural language specifications of software requirements is also investigated [59]. While these approaches provide valuable assistance, they either require significant manual input from developers are limited to specific use cases.

---

7 Conclusion
In this paper, we propose Repo2Run, the first LLM-based agent for automated coding executable environment building and Dockerfile generation for Python repositories. With dual-environment architecture and a Dockerfile synthesizer, Repo2Run is able to select and change the base image, manage and install dependencies based on action observation and rollback mechanism, and utilize bash commands and existing test suites. Our evaluation of 420 popular Python repositories hosted on GitHub demonstrates the effectiveness of Repo2Run with an 86.0% success rate. We believe Repo2Run will serve as the foundational infrastructure, enabling the community to efficiently scale up executable environments.
8 Acknowledgement
This research is supported by the National Natural Science Foundation of China under project (No. 62472126), Natural Science Foundation Guangdong Province (Project No. 2023A 151001 1959), and Shenzhen- Hong Kong Jointly Funded Project (Category A, No. SGDX202301 16 091 246007).
References
[1] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
[2] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang. Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024.
[3] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15799 2024.
[4] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Opendevin: An open platform for software developers generalist agents. arXiv preprint arXiv:2407.16741, 2024.
[5] GitHub. Github copilot your ai pair programmer.
https://gitee http:///gtthub..ommffttauuutt
copilot
[6] Anysphere Cursor. Cursor.
cursor.
[7] Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fixer: Training open-source Ilms for effective and efficient github issue resolution. arXiv preprint 20X5v:2501.05040, 2025.
[8] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing Ilm reasoning via reinforcement learning on open software evolution. arXiv preprint 11Xi:25502.1844 2025.
[9] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint 24X.::241..2113, 2024.
[10] John Yang, Kilian Leret, Carlos Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21798, 2025.
[11] Sander Valstar, William G Griswold, and Leo Porter. Using devcontainers to standardize student development environments: experience report. Proceedings of the 2020 ACM conference on innovation and technology in computer science education, pages 377-383, 2020.
10

---

[12] github. Octoverse-20e octoverse- 200o0/
[13] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and
Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,
2023. OpenReview.net, 2023. URL URL.httpp::/oppereviie htt.pppawww...eeeeooooouoraaaaaa
net/forum?id-Ne vite__llYUL-XXX
[14] w w3techs. Historical trends in the usage statistics of Python version for websites.
https
// hrttes/s..././ooooooogggg////aaa
[15] dev. A command line utility display dependency tree of the installed Python packages. https://gi thub. www/tooo fllo--de/ppiddpptreee
[16] bndr. Generate pip requirements.Ng file based on imports of any project. looking for maintainers move this project forward. https://githun
[17] Vincent J. Hellendoorn, Sebastian Proksch, Harald C. Gall, and Alberto Bacchelli. When code completion fails: a case study on real-world completions. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle, editors, Proceedings of the s1 International Conference ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pages 960-970. IEEE/ ACM, 2019. doi: URL http::ttp///// //doi org/ 10. 1110/ICS9E.... 201.://...E.............. 2019.00111
Software Engineering,
10.100//CSE.2219.0000
[18] Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. CoRR, abs/2404.00599 2024. doi: (044550/ARXXI.2444.0555 URI.attpss//Bo...og//0....
2404
00599
[19] hadolint. Haskell dockerfile linter. http:////ggittub.cco/o//oooooiio
[20] Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. Marscode agent: Ai-native automated bug fixing, arXiv preprint urXiv:2409000899 2024.
[21] Pengfei Gao, Zhao Tian, Xiangxin Meng. Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, al. Trae agent: An Ilm-based agent for software engineering with test-time scaling. arXiv preprint 2rXiv:2507.23333 2025.
[22] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang. Changhao Jiang. Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui. The rise and potential of large language model based agents: survey. CoRR, abs/2309.07866 2023. doi: 10.48500ARXXV2230.777888 https://///d org/10. 1.11.4855//aaii11 2309. 07864
[23] Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. ACM Trans. Softw: Eng. Methodol,, (3(7): 18:1):199 2024. doi: 10.1145/3672459, URL
https:///oo org/10. 10. 1145/3672459
[24] Martin Josifoski, Lars Henning Klein, Maxime Peyrard, Yifei Li, Saibo Geng, Julian Paul Schnitzler, Yuxing Yao, Jiheng Wei, Debjit Paul, and Robert West. Flows: Building blocks of reasoning and collaborating AI. CoRR, abs/2308.01288 001885 2023. doi: 10.48550/ARXIV223088
01285. URL httph:ttt//dttt /ttp://o...rr/11..88555/////////
48550/5550// arwiv. 2308 012885
[25] Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D. Goodman, and Nick Haber. Parsel: Algorithmic reasoning with language models by composing decompositions. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neurul Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URLhttps: https: // d/ll acm. org /oi/ 5555 /5w.//55662122266666656 3667448

---

[26] Andy Zhou, Kai Yan, Michal Shlapentokh Shlapentokh-Rothman Haohan Wang, and Yu-Xi Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Forty- first International Conference on Machine Learning. ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. Utt.tttss: htatttpps///peneeveeww net wett//oouumii-njwwyBBes..
[27] Xinyi He, Jiaru Zou, Yun Lin, Mengyu Zhou, Shi Han, Zejian Yuan, and Dongmei Zhang. CONLINE: complex code generation and refinement with online searching and correctness testing. CoRR, abs/2403. 135883, 2024. doi: (0.4855/ARRXV.24005.. Aoboi.org/ln 48550/ arXiv. 2403. 13583
URL https
[28] Kechi Zhang. Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Jui-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 13643-13658, Association for Computational Linguistics, 2024. doi: 10.18653/V AA0-02444ALLLLL...1590 URL inttps://dat http:///oo...r////// 18653/v1/ 2024. acl- a long. 737
[29] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer ferd-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5484-5495, Association for Computational Linguistics, 2021. doi: 10.18653/V /1W.//222.ELL--AAIAAAM 446. URL
emn mamnlppmain...m 446
[30] Juyeon Yoon, Rohert Feldt, and Shin Yoo. Autonomous large language model agents enabling intent-driven mobile GUI testing. CoRR, abs/2331..08644 2023. doi: 10.48550/ARXIV.23111 08649, URL https://doii
11w.././//////.....222 08649
[31] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Communicative agents for software development. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024, pages 15174-15186 Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024 ACL-LONG.810, http:htts:://a org/ / 10. 11863//v//200/
acl-long 810
[32] Sirui Hong, Mingehen Zhuge, Jonathan Chen, Xiawu Zheng. Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:://ppenreviees net.
[33] Juan Altmayer Pizzorno and Emery D. Berger. Coverup: Coverage- Ilm-based test generation. CoRR, 1b2/2403,116218, 2024. doi: 10.4850/ARXXI.2403..6222 URL https / //doi. 100/10.0. 48550 arxvv 2403 16218
[34] Zhitao Wang, Wei Wang. Zirao Li, Long Wang, Can Yi, Xinjie Xu, Luyang Cao, Hanjing Su, Shouzhi Chen, and Jun Zhou. Xuat-copilot: Multi-agent collaborative system for automated user acceptance testing with large language model. CoRR, abs/2401 02705, 2024. doi: 10. 48550/ARXX1..401.0277 URL BtLLhttps: //stts:///oi... oralii..rr111.1. 48550/arXiv 2401 2401.02705
[35] Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng. Yue Jiang, and Jeffrey Nichols. Axnav: Replaying accessibility tests from natural language. In Florian 'Floyd' Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, editors, Proceedings of the CHI Conference on Humun Factors in Computing Systems, CHI 2024, Honolulu, HI, USA, May 11-16, 2024, pages 962: 9622 16. ACM, 2024. doi: 10. 145/36 022///3333994430 390443642777.. URL https: / //oii org 10. / 3613904 3642777
12

---

[36] Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. Toolcoder: Teach code generation models to use API search tools. CoRR, abs/2305. 2023. doi: 10.48550/ARXXI.2305.00000 URL
https://doi... /ttp::://doooor/1..... 48550/8550///0 arxiv 2305 04032
[37] Ziyang Li. Saikat Dutta, and Mayur Naik. LIm-assisted static analysis for detecting security vulncrabilitics. CoRR, abs/2405. 122228, 2024. doi: 10.8550/ARXX..206517235. URL
attps://doi.org// 48550/ arXiv 2405 17238
[38] Dong Huang, Qingwen Bu, Jie M. Zhang, Michael Luck, and Heming Cui. Agentcoder: Multi- agent-based code generation with iterative testing and optimisation. CoRR, abs/2312.130100 2023, doi: 1044550/ARXIV.312...000
URLhttps://ddo org/ 0tte//10.4555/////// 2312
13010
[39] Xinchen Wang, Pengfei Gao, Xiangxin Meng, Chao Peng, Ruida Hu, Yun Lin, and Cuiyun Gao. Aegis: An agent-based framework for bug reproduction from issue descriptions. FSE Companion '25, page 331-342, New York, NY, USA, 2025. Association for Com- puting Machinery. ISBN 9798400712760.
doi: 10. 10.1145//99663337788557 URL https
//doi.org/10 00445/366630..0 3728557
[40] Gang Fan, Xiaoheng Xie, Xunjin Zheng, Yinan Liang, and Peng Di. Static code analysis in the AT era: An in-depth exploration of the concept, function, and potential of intelligent code analysis agents. CoRR, abs/2310.08833 2023. doi: 10.45500ARXXV.2110.0585 https://doi. ottp://///oooo//1..... 48550/ 205540/55aaiii arriv. 2310. 08837
URL
[41] Islem Bouzenia, Premkumar T.. Devanbu, and Michael Pradel. Repairagent: An autonomous, llm-based agent for program repair. CoRR, abs/2403.17134 2024. doi: 10.48550/ARXIV.2400.
17134. jttpps://///o org/10. wtwp://...855///// arXiv. 2403. 17134
[42] Huacan Wang, Ziyi Ni, Shuo Zhang, Shuo Lu, Sen Hu, Ziyang He, Chen Hu, Jiaye Lin, Yifu Guo, Ronghao Chen, et al. Repomaster: Autonomous exploration and understanding of github repositories for complex task solving. arXiv preprint urXiv:2505.2/577 2025.
[43] Xinchen Wang, Pengfei Gao, Chao Peng, Ruida Hu, and Cuiyun Gao. Codevisionary: An agent-based framework for evaluating large language models in code generation. arXiv preprint arXiv:2504. 13422, 2025.
[44] Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. Openrca: Can large language models locate the root cause of software failures? In The Thirteenth International Conference on Learning Representations, 2025. URL htttp::/ http:///wwweeeeeeeeeeeererrrraa/
[45] Xin-Cheng Wen, Zirui Lin, Yijun Yang, Cuiyun Gao, and Deheng Ye, Vul-r2: A reasoning l1m for automated vulnerability repair. arXiv preprint arXiv:2510000548. 2025.
[46] Ziyi Ni, Huacan Wang, Shuo Zhang, Shuo Lu, Ziyang He, Wang You, Zhenheng Tang, Yuntao Du, Bill Sun, Hongzhang Liu, al. Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging. arXiv preprint arXiv:2508. 18993, 2025.
[47] Junjielong Xu, Ying Fu, Shin Hwei Tan, and Pinjia He. Aligning the objective of Ilm-based program repair. arXiv preprint arXhv:240..08877, 2024.
[48] Xin-Cheng Wen, Yijun Yang, Cuiyun Gao, Yang Xiao, and Deheng Ye. Boosting vulnerability detection of Ilms via curriculum preference optimization with synthetic reasoning data. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 August 1, 2025, pages 8935 8949, Association for Computational Linguistics, 2025.
[49] Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. Reposvul: A repository-level high-quality vulnerability dataset. In Proceedings of the 2024 IEEF/ACM 46th International Conference on Software Engineering: Companion Proceedings, pages 472-483, 2024.
13

---

[50] Qinyun Wu, Chao Peng, Pengfei Gao, Ruida Hu, Haoyu Gan, Bo Jiang, Jinhe Tang, Zhiwen Deng, Zhamming Guan, Cuiyun Gao, et al. Repomastereval: Evaluating code completion via real-world repositories. arXiv preprint 235.::2000003//99 2024.
[51] Xin-Cheng Wen, Xinchen Wang. Yujia Chen, Ruida Hu, David Lo. and Cuiyun Gao. Vuleval: Towards repository-level evaluation of software vulnerability detection. CoRR, abs/2404.1559, 2024. URL /ttpsst::////ooorr 10. 48550/ arXiv. 2404. 15596
[52] Dongge Liu, Oliver Chang, Jonathan metzman, Martin Sablotny, and Mihai Maruseac. OSS- Fuzz-Gen: Automated Fuzz Target Generation, May 2024. URL
com/
google/oss-fuzz gen
[53] cloud66 Starter: Helping you get started with containerized apps. https://github..co
cloud66- oss/starter
[54] Microsoft. Yeoman
generator for docker.
httppss//ggthubb.coo/mmcooooos
generator-docker
[55] Eric Horton and Chris Parnin. Dockerizeme: Automatic inference of environment dependencies for python code snippets. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 328-338 IEEE, 2019.
[56] Hongjie Ye, Jiahong Zhou, Wei Chen, Jiaxin Zhu, Guoquan Wu, and Jun Wei. Dockergen: A knowledge graph based approach for software containerization. In 202/ IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC), pages 986-991. IEEE, 2021.
[57] Yun Peng, Ruida Hu, Ruoke Wang, Cuiyun Gao, Shuqing Li, and Michael R Lyu. Less is more? an empirical study on configuration issues python pypi ecosystem. Proceedings of the IEEF/ACM 46th international conference software engineering, pages 1-12, 2024.
[58] K Hanayama, S Matsumoto, and S Kusumoto. Humpback: Code completion system for dockerfiles based on language models. Proc. Workshop on Natural Language Processing Advancements for Software Engineering, pages 1-4, 2020.
[59] Giovanni Rosa, Antonio Mastropaolo, Simone Scalabrino, Gabriele Bavota, and Rocco Oliveto. Automatically generating dockerfiles via deep learning: Challenges and promises. 2023 IEEE/ACM International Conference on Software and System Processes (ICSSP), pages 1-12. IEEE, 2023.
[60] EvanLi. Top 100 stars in python. nasterr/Top100/Python...
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main claims, contributions, and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes dedicated "L "Limitations" section in Section 5.4 We also reflect on the impact of these limitations on the results and their generalizability.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
The paper does not include any theoretical results, so this question is not applicable.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the complete code along with instructions build an executable environment, ensuring that the main experimental results can be fully reproduced. This allows others to verify and replicate our findings reliably.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the code and data in anonymized repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/ gublic/guidess htllai//giidee//eeeebbbmmseee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips cc/ httpppullllplppiee http://oooeeuuaassssooooooooo1
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the experimental design in the main paper and provide the LLM prompts in an anonymized repository to ensure full transparency and reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We have not repeated experiments many times to get error bars since experi- ments of multiple times is cime-consuming and expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the LLM used in the experiments and report the temperature settings applied during training and evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research fully complies with the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The research presented in this paper remains at the experimental levels and has not yet resulted in any societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose any risks related to misuse or dual-use, so no special safeguards necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited all original creators and clearly stated the licenses and terms of use for all assets used in our paper. In the implementation of Repo2Run, dependencies are clearly cited in the "requirements. txt". The usage of open-source code complies to its license and the copyright notice and source are placed in the our source code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsiiiie thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Our benchmark is constructed in accordance with the relevant license require- ments, ensuring proper compliance and documentation. The usage of the implementation of Repo2Run is provided in its README. md file.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codd/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We provided the full instructions and details for the study on how industry developers struggle in building environments in Section F The human study was approved by the team leader and legal business partner of these developers, and was conducted during their working hours as paid work. Their salary was above the minimum wage in the country where this study was conducted. No compensation details are involved as participants were not compensated.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer: [Yes]
Justification; Our study on how industry developers struggle in building environments in Section F involved only survey participation and was approved by the IRB from their organization. We have complied with all relevant ethical guidelines.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We provide thorough description of the usage of LLMs as core component of our research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

Appendix
Repo2Run details
A.1 Example of Repo2Run
Tigure example process of Repo2Run, including the event stream and Repo2Run workflow. Below, we introduce the executed action based on the example:
In the tenth part of Event Stream, when the LLM agent discovers that the "StrEnum" class cannot be imported through testing, which is only available in the Python standard library starting from version 3.11, it decides to reselect the base imgge to python:3.11.
Base Image Change:
Environment monitoring: In the fourth part, the LLM agent calls the "find" command to search for requirements. txt within the environment to determine which dependencies to install.
Dependency installation: the sixth part, the action "waitinglist addfile" adds all elements from requirementa txt the waiting list, while executing the "download" command. Third- party libraries are taken one by one from the waiting list and installed in the internal environment. The installation commands (e.g., "pip install" and "apt-ges get install") are executed.
Code editing: The eighteenth part demonstrates code editing to correct a syntax error resulting from the improper use of double quotes in a Python f-string.
Bash commands: The sixteenth part shows that the LLM agent uses the command "export WYTHONPATH=/IepO
Test running: The eighth part shows that the LLM agent calls the "runtest" command to execute unit tests.
A.2 Action usage frequency
As shown in Figure we analyze the invocation times of various action types in 361 successfully built projects. including the five actions within the internal environment and the action of base image change. Bash commands are the most frequently used action, as they encompass the majority of instructions. Additionally, we observe that the LLM agent tends to call dependency installation quite frequently, averaging about 18 times per building, which means roughly 18 dependencies are installed per building on average. Moreover, the LLM agent calls test running approximately 3.5 times per building on average, which typically helps the agent better identify issues. Among successful buildings, we find 48 instances of changing the base image, accounting for 13.3% of success cases, indicating that the initial selection of the base image is often incorrect and requires subsequent adjustments.
B Dockerfile synthesizer details
B.1
Example
If the LLM agent successfully runs all tests in the repository in the internal environment, the building process needs be recorded in Dockerfile. Figure@sllustrates the Dockerfile synthesizer's operation. Figure (a) lists the internal commands actually executed in the internal environment, their return codes (e.g., indicates success, otherwise failure), and the current directory (dir). Figure (b)
shows the process of generating the Dockerfile.
The principles for the conversion process include the following essential steps: scanning all internal commands sequentially, and the line numbers mentioned below precisely match the line number of Figure 9 (a):
By default, use python: 3. as the base image.
2. For commands that run successfully (i.e., have return code of 0), prepend the command with "RUN" to form a Dockerfile statement (e.g., line 3). Commands that fail (i.e.. have a return code other than 0, such as line 4) are rolled back and not included in the Dockerfile. An exception is that some commands typically not change the current state (such as "cat" on line 2). They are not added to the Dockerfile. We show the complete list in Appendix B.2
22

---

Figure 7: An example process of Repo2Run, which illustrates two main parts: Event Stream and Repo2Run Workflow. The Event Stream tracks the entire aetion-observati process, where the green box represents the abbreviated system prompt, yellow boxes represent responses from the LLM agent including the actions, and blue boxes represent observations from the Runtime Environment. The shaded actions indicate buildings abandoned after base image change. In the Event Stream, the blue text indicates that the command starts running, the green text indicates that the command runs successfully, and the red text indicates that the command fails. The Repo2Run Workflow consists of internal environment and the external environment. The internal environment serves as actual buildings Docker-basse sandbox, which builds an actual testing runtime environment. The process and assists in the building process Event Stream
mudie Kode nater' rasten ten
Hele intrecini scretcr disttry
EP00 crlaiin Cilve tanayy
[] anveronment mon Booing
observation
pil mam
Ph bar seaiiig w1w.aaa------- PyBlor
[|] enveronment
Bu
[T] observation
ediing
155
Sovelcal Taxs coondeddee
dualing
to
ungor goceec
brr
observation
oythondrarrs
cocaanmeaaiioo 2016 eifiheinnn againn Ai
sainal priduar sucied Aonbiode
anstronment
enpprcciilln Seeerroees parggnt PNTTOCNAAT ande
Mrticaeei soll rngit dedti
absarvation
[1]] absorvation
Repo2Rum Workftow
External Enviromment
Internal Environment
X
Event Natory Adle Prnne areaa
D
doctor
+
roRbach
change
nuwving
COSB nillngg
menttring
Reponioryy
Jonticii deparcercy monagemert
iranciice
Dockerli
Actinn
75:0 RUR
Ago Senbot
external environment executes the within the internal environment.
17212
15000
Frequency
10000
6373
5000
1489
1267
882
0
Environment
Dependency installation
Test running
Code editing
Bash commands
Change base image
monitoring
Figure 8: Action usage frequency of the success building.
23

---

Internal command
return code
dir
FROM pystrer-3 -3. 20
We bedh welt defiun inace pythort 16.
2042 commandd posts instll felld, Saee
the socuuiionn Rovn (P.0 1yaon
Li /raps
3/4. conamande monedue Tol out are vehroseneeeee it wont
FROOS surthan
Cnasall
The cormomennn
lubuo soroiiing buck.
RUN oF /rapo de sytton Koadcantic.u wn /pototre.sstcc
partee
Ssccall frace
RUN feacaLL
13: The sxpofiltaammtt convated EnE command
of /raps
5. The bace image succeshiily chaneed, ell previcue suldingg
cscandos Kaaing only now buse kmags pyllict
RUN of /ress pie truteet
FROM portuc--s 9.27
/code, /betch/R.potch
RUN pte textatt deet <2.
fexcall
34-28 passling soss, car "apdeetorr noard and
PotRes cale wele. by kyjatton
0
Agpo
scold ris the Decurlis. traeedd oo save the communnd' "ec pipintill
rop aoa ron-upec coroiuins the Decket win thoe adual "oodo weil soipt and Saaju Dec conkiieer
werions the
FROM pytten-33 22 RUN /raps 85 plp tratact
11
PIe Ssccall Rivd.0, 12.4
FROM pytoen:22222 22
Sucoeeridl adtve file Aaply paith ecverna
COPT rede, ,etr,W /rode, sett.y
RUN ca /rope a8 si inuteds
14 PNSTECE
RUN fress 64 aie testact nt
pytthe- 2.22
teutect $.x2.5..
/rode, edit.ay /netvt/R.,atoo
(a) Commands running in inturnal environment
(b) process synthesize the Dockurfile by Dockerfile synthesizer.
Figure 9: An example of Dockerfile synthesizer to transfer the commands into a runnable Dockerfile.
dir is not the root directory, use cd to change to that directory before running the command, as each Dockerfile statement runs in its own directory session.
4. If a base image change statement is encountered (e.g., line 5), discard all previous buildings and switch to the new base image.
5. If code editing command is encountered (e.g., line 9), copy the patch and editing script into the Docker container before executing the editing command.
an export statement for adding environment variables encountered (e.g., line 13), convert it to persistent ENV statement.
7. After all statements have been scanned, check for dependency installation commands and replace any unspecified versions (e.g.. "B>=1.0, O, <2. O" in line 11) with the actual versions downloaded in the Docker container (e.g., "B==1.5 5. in the final Dockerfile of Figure (b)).
B.2 Safe command
If Repo2Run executes the following commands without using ">" or ">>" for output redirection, they are regarded to be safe commands that typically do not affect the system. Therefore, rollback is not necessary, and they are not added the generated Dockerfile.
["cd", "Is", "cat", "echo", "pwd", "whoami", "who", "date", "cal", "df", "free,, "uname", "uptime", "w", "ps", "pgrep", "top". "dmesg". "tail", "head", "grep". "find", "locate", "which", "file", "stat", "cmp". "diff", "xz", "unxz", "sort", "wc", "tr", "cut", "paste", "tee", "awk", "env", "printenv", "hostname", "ping", "traceroute", "ssh"]
C Pollution commands
Figure [0 two practical examples. As shown in Figure I0 (a). when we run "pip install cupy", the command fails to execute because the CUDA environment invalid. However, it still introduces the "fastrlock" and "numpy" packages which is not present in the original environment. In ( when we run the "rm -rf /path/to/logs command delete the directory, some files fail to be deleted due to lack of permission. However, the files that have already been deleted (e.g., log1. txt and log2. txt) are restored. Such "pollution" caused by failed commands may make the environment enter the uncertain state.
We also identified 30 Python packages that may introduce "pollution" when installing via pip, as listed in Table This process can be easily reproduced in a clean Docker environment (such as the initial environment shown in Figure I0 (a)). For example, simply running "pip install adb" even without specifying version, which defaults to the latest can cause such "pollution". This demonstrates that such scenarios are quite common in real-world development and also underscores the importance of the rollback action in Repo2Run.
24

---

usenshostnane:/8 pip list Version
uerrhoostaare//pat//////gggg /path/to/logt
Package
/path/to/logs
loga txt
23.0.1
log2. txt
setuptools 5.1
secure
wheel 0.44.8
secure_logt.
usenshostnane/S pip cupy error: subp racess exited-with tith--rroo
log2.txt
-rf /path/to//lgg
re: cannot renove
//stthtto//ogg//eeur////sseuu txg
ERROR: Failed building
cupy
denied
Exception:
Your CUDA environment is Invalid.
rn:
cannot renove
usenshostnanee/S pip list
Pernission denied
Package
Version
rn: cannet renove //ath/toolog//secrre Pemrission
denied
fastrlock
8.8.3
/var/logs/app
numpy 2.2:5
ppath/to/logs
pip
23.8.1
secure
setuptools
logure_oog.txt
wheel
0.44.
secure_ 1og2.,xt
(a) Failed "pip install" example.
(b) Failed "mn -rf"" example.
Figure 10: Examples of failed command causing "pollution" the environment.
Table 6: 30 examples of "pollution" during pip installation. "Package name" represents the name of the package that failed to download, "pollution" represents other packages introduced by the failed download, and "pollution" count represents the number of these introduced packages.
package
"pollution"
"pollution" count
zbarlight
pillow
mxnet-cu91
chardet, idna, urllib3
texthero
blis, catalogue, certifi, charset- normalizer, click,
41
url
robotframework-ride
certifi, charset-normalizer, idna, numpy, packaging, libusbl, typing
10
adb
onegov-core
fastcache, mailthon, passlib, polib, pytz,
12
postal
changes
requests
mxnet-cu75mkl
chardet, idna, urllib3
winpdb
numpy, six
slybot
attrs, Automat, certifi, chardet, charset-normalizer....
30
fbprophet
aiohappeyeyeballs aiohttp, aiosig- nal, appdirs, asyne-timeout.
44
mxnet-cu75
chardet, idna, urllib3
libarchive
nose
atari-py py
numpy, six
2
reppy
cachetools. certifi, charset- normalizer, idna, python-dateutil,
Continued next page
25

---

Table 6 continued from previous page
package name
"pollution"
"pollution" count
sovrin
leveldb, libnacl, msgpack-python, orderedset, Pympler,
15
scrapely
numpy, six, w3lib
kevinsr
python-version
pysurvive
colored, numpy, pillow, psutil, pygtrie,
horovod
cffi, cloudpickle, packaging, psutil, PyYAML,..
cupy
fastrlock, numpy
face-recognition
face-recognition-moodel
nsot
gunicorn, idna, ipaddress, ipython, itypes....
15
gooey
colored, numpy, pillow, psutil, pygtrie.... ...
wxpython
numpy, six
neuralcoref
annotated-types, blis, boto3, boto- core, catalogue,
47
hatphe--airflo-bbckpooo providers-apache-hise
apispec, argcomplete, attrs, babel, cached-property....
58
bcolz
numpy
1
D
Benchmark statistic
As demonstrated in Table 7 benchmark.
we list the statistics on lines of code for repositories in our constructed
Table Statistics on Lines of Code (LOC) for repositories in our benchmark. The table below illustrates the statistics regarding the lines of code (LOC) of all repositories in the benchmark. It separately counts the number of repositories with LOC greater than 10,000 and greater than 100,000. The last two rows provide reference LOC for well- industry-level repositories.
Configuration state
#LOC> 10,000
#LOC 100,000
Medium
Max
Success(361)
220 (60.9%)
27 (7.5%)
14,578
6,431,084
Fail(59)
50 (84.7%)
12 (30.6%)
43,001
3,584,542
All(420)
270 (64.3%)
39 (9.3%)
15,854
6,431,084
requests (reference)
Yes
No
12,782
12,782
pandas (reference)
Yes
Yes
571,849
571,849
E
Test pass rate experiment
We randomly sample 50 repositories from our benchmark, successfully run their entire test suites, and manually record the pass rates for each method. Table shows the percentage of repositories for which each method achieved a test pass rate exceeding various thresholds (from >=10% to 100%).
26

---

Table 8: Pass rates across tools at varying thresholds. Each cell shows the percentage and the number
repositories in parentheses.
Pass Rate Threshold
Repo2Run
SWE-agent
LLM generator
pipreqs
10%
70% (35)
26% (13)
32% (16)
10% (5)
20%
68% (34)
26% (13)
32% (16)
10% (5)
230%
64% (32)
26% (13)
32% (16)
10% (5)
240%
62% (31)
26% (13)
32% (16)
8% (4)
250%
62 % (31)
24% (12)
32% (16)
8% (4)
60%
58 (29)
20% (10)
30% (15)
8% 8% (4)
2 70%
54% (27)
18% (9)
28% (14)
8% (4)
80%
52% (26)
18% (9)
26% (13)
4% (2)
900%
48 % (24)
18% (9)
26% (13)
4% (2)
100%
34% (17)
10% (5)
14% (7)
4% (2)
The results are clear: regardless of the pass rate threshold chosen, Repo2Runccnsistently and substantially outperforms all baselines. This experiment demonstrates that even when evaluated by the metric you suggested, our method's superiority remains clear.
We conduct this experiment directly show that RepoRRunexcels not just in creating executable environments, but in creating high-quality, correct ones. We wish to briefly emphasize why we firmly believe that EBSR (Environment Building Success Rate) an appropriate metric for this task.
Our goal is to automate the crucial first step of software usability: creating an executable environment. Test pass rates, while informative, are "noisy" signal for environment quality because failures can arise from code bugs or test bugs, not just environment building errors. As industry practice (e.g.. CI/CD) shows, distinguishing between build/execution success and test pass success is standard and necessary distinction.
F
Empirical study
F1
Manual Experiment
F111
Settings
To ensure fair comparison among participants, we conducted training and demonstrated examples before the experiment to ensure everyone understood the procedure. Additionally, minimize discrepancies in time consumption due to network factors, all participants conducted the experiment in the same network environment.
E.1.2 Survey
We selected eight technical staff from internet companies to participate in the experiment and con- ducted survey regarding their backgrounds prior the experiment. Their development experience ranges from 4 to 11 years, with an average of years software development and 3.8 years Python development. Seven participants have experience in complex development projects, while has experience multiple small-scale projects.
Regarding environment building, three participants indicated that they spend significant amount of time building the environment when faced with an unfamiliar code repository; another three stated that, although they spend a long time, it is generally manageable; two participants reported spending minimal time.
In terms of successful environment building in their regular work, three participants mentioned they only fail with extremely complex environments, while five indicated they can build environments for most medium-scale repositories. As for their confidence in successfully building unfamiliar environments, six participants expressed that they are usually successful, and two said they are sometimes successful.
27

---

When it comes to the amount of time they are willing to wait to build an unfamiliar repository, four participants are willing to wait for over 90 minutes, two are willing wait 40-60 minutes, one is willing to wait 20-40 minutes, and one is only willing to wait 10-20 minutes.
Participants' overall evaluation of building code running environments is diverse: finds it very troublesome with many issues, four find it somewhat troublesome, two indicate moderate difficulty, and one finds relatively simple.
Additionally, all participants expressed high or very high willingness to use a tool that could automatically build the environment for an unfamiliar repository.
E.1.3 Experiment guideline
We request all participants to conduct the experiment following the guidelines below:
1. Environment Setup
Build the Docker environment. Verify the installation is successful: installed correctly, you should be able to use the following command (python:3.10 10 is just example; select the base image according to your requirements):
docker run -it python:3... 3. 10 bash
2. Overall Procedure
Our objective to install the given package in Docker container, build its environment, and be able to run its internal tests. During the process, record the time developers spend building the environment and eventually save the logs using docker logs.
2.1 Review the Repository to be Built (Optional)
Review the GitHub repository that needs to be built.
2.2 Determine the Docker Base Image (Generally start with python:3.10)
Select the Docker base image based on the repository (all repositories use Python as the main language). Common base images include: Note, in this experiment, it is generally sufficient official Python series images. The main concern the version; if uncertain, you can start with newer versions like 3.10, 3.11, 3.12, or select the recommended Python version based on the repository's README.
the
Python series (the number after python indicates the version): python:3.10 10., python:3.11 1.2, python:3.9, python:3.6, python:3.9... Py Torch series: pytorch/pytorch, pytorch/pytorch:1.99 1.w.ch::19900000 9.0-cuda100 1iduda10.22. 1-nn-cun runtime... Anaconda series: continuumio/anacondas.... Note: If you find the selected version is incorrect later, you can exit and reselect.
2.3 Create and Enter the Container
Using the determined base image name (e.g., python:3.10), enter the container with the following command:
docker run -it -name mytest python: 3. 10 bash
Here, mytest is the container name, recorded for log export later. It can be freely named, just keep track of it to avoid losing it later. Note: If any issues arise here, check if Docker is correctly installed and if the image name is valid, and troubleshoot accordingly.
2.4 Install Relevant Tools
APT tool downloads:
apt-get update apt-get install -y curl Download pytest:
pip install pytest
2.5 Download the Repository
Select the repository to built and download its GitHub repository a location (generally directly in the root directory):
28

---

git clone http::/ggttooooco///lllaaaaaaaa .git
2.6 Build the Environment
Now, use your skills build: First, enter the downloaded file directory, for example: cd wddbfs
Switch to the specified branch SHA (refer to the corresponding SHA of the repository), for example:
git checkout 5c68aa
Our goal is to successfully run pytest (not necessarily to pass all tests, just to run them). simple
criterion is successfully
pytest -collect-only -q
At this point, you can use your experience and information from the repository documentation and debugging eтror messages to build. However, there are few restrictions:
Do not directly edit the test files! (Files starting with test_ or ending with _test.py). Do not directly delete test files! Editing the original repository files is not recommended.
During this process, you may perform various operations, including but not limited to pip. apt-get, and other tool downloads, as well as searching online or using GPT for debugging help.
Additionally, if there are long download times requiring waiting, you may decide according your situation whether to leave this running and do other things (just don't forget about this task).
2.7 Completion and Logging
task can conclude in two scenarios:
If "pytest collecc-oonl -q" runs without issues, you can then execute pytest. If pytest completes successfully, the task is done. Scenario Two: If you feel the package is extremely difficult to build, for example exceeding your patience threshold (refer to your usual development habits), you may also terminate.
Scenario One:
Once finished, input exit to exit. Make sure save your output logs with the following command (replace container name with the container name you recorded earlier, if you forget, you can use docker ps to check):
docker logs container_naae _amme -t > wddbfs.1o .ogg
2.8 Fill Out the Form and Record Information
You need fill out the form according to your feelings.
Here, Table 9 presents the form that developers are required to complete after the environment building process, including items such "Is it successful?". "Final base image used", "Reason for failure", and so on. Besides, Tables 10 and I1 show two examples filled out by developers.
Table 9: The form that developers should fill in after the environment building.
Question
Description
Is it successful?
Yes or No whether the task successfully passed "pytest -collect-only -q" without emror and eventually ran "pytest".
Final base image used
R.g., python:3t110
Continued on next page
29

---

Table 9 continued from previous page
Question
Description
Reason for failure
Summarize the main reasons for failure (if failed), including:
Long download time
Difficulty handling repository dependencies
Unresolvable bug
Errors in repository tests
Lengthy test durations
Other reasons (please specify)
Waiting time
Approximate value, indicating the time spent waiting for dependen- cies to download. Does not include time spent on soccisio-mmakngg and research. Provide an approximate range:
<3 minutes
3-5 minutes
6- 10 minutes
1-20 minutes
20-40 minutes
40-60 minutes
60-90 minutes
90+ minutes
Longest time-consuming process
Describe the most time- coneeconsuming building process, such as:
Downloading specific dependency
Resolving specific error
Incorrect Python version selection
Tolerance level
Based your subjective feeling and development experience. Rate the process (1-5):
1: Extremely unbearable
2: Somewhat unbearable, but manageable
3: Neutral, tolerable
4: Comfortable, no significant discomfort
5: Very comfortable, highly satisfactory building experience
Building difficulty
Based on the complexity of the building process (ignoring time spent); Rate the process (1-5):
1: Very simple, completed with intuition and experience, refer-
ence materials needed
2: Fairly simple, referred to basic materials (e.g., README).
simple overall steps
3: Moderate difficulty, encountered some issues, but manageable
4: Difficult, required extensive debugging and building
5: Very difficult, needed numerous references and encountered unresolved or time-consuming issues
Materials referenced
List the materials used reference (e.g., README, internal build- ing files, online searches, GPT). If none, mention "Directory".
Biggest challenge during the process
Describe the most troublesome aspect of the building process, such as long wait times, unclear error messages, dependency version conflicts, inability to find required software versions, etc.
30

---

Example 1:
Table 10: sample form filled by a developer after environment building, illustrating successful environment building.
Question
Answer
successful?
Yes
Final base image used
python:3.10 10
Reason for failure
Successful
Waiting time
3-5 minutes
Longest time-consuming process
Downloading poetry
Tolerance level
Building difficulty
Materials referenced
Directory
Biggest challenge during the process
Waiting for building and installation
Example
Table 11: A sample form filled by a developer after environment building, illustrating a failed environment building.
Question
Answer
Is it successful?
No
Final base image used
python:3.11
Reason for failure
Unresolvable bug, provide bug image
Waiting time
40-60 minutes
Longest time-consuming process
Resolving a series of FileNottFundEErrr, ImportError, incorrect Python version selection
Tolerance level
Building difficulty
Materials referenced
README, GPT, StackOverflow
Biggest challenge during the process
Dependency version conflicts, long wait times
F11.4 Repository assignment
We randomly assigned each participant four unique code repositories that were successfully built by Repo2Run. Additionally, each participant was assigned two code repositories that Repo2Rum failed to build. To avoid chance occurrences, each failed repository was assigned to two different participants. Below the list selections:
Successfully built:
falexwlchan/safari wetarrriivebaarrereeeeee. ManiMozaffar/aioclock. sitpebbradd-ibbbacchhee mobiusm- /ggmlitee circlemind-aiffast-rraapph Knowsuulagenns/prroppiic, mbbodia//emboddied-ggenss
modelscoppelggnnsscpee,, AdilvaafaCCodonTTrassffooo
kennethreizz/iipplemnnn, Imstudio-
ai/venvstacks, milcaauchoismmiccngnaadcuuua I a MetaGiL./zzhipaiiss.
python-v4, openai/mle-bench RealCCrangeOneddjanng-ttass bast/MolPipeline. miiimoooo//ootrraakk
iioiddiinn/allhaatod---yooooo mistralaa//mittrllcoommm B.Mixel/moffce. DataformerAl/-
dataformer, jahwaa/ClaudeSyne. volfpeter/htmy, Genentech//ReLLü OpenNLPLLablighttin attention, paradigmxyz/spice, reagento/dishka, arcee-ai/fastmlx. KyanChen/RSMambe neuralmag- ic/guidellm, wwwpooowwwwles---oommmooo
Failed to build:
[zhuginfeng| 1999/Samba, dangguuyuelpeee-Reppeeeeeeiieei LazyAGG/LazyLLLM jialuechen/deep-
folio, ooSSss//h...ouuuuutttooommssssss AARG-FAN/Yolo, for. Wukong, plinder-
org/plinder, expectedparrot/cds]]]
31

---

F11.5 Results
Based on the times shown in the logs, calculated that the average building time for each repository 21.33 minutes. Furthermore, none of the repositories that Repo2Run failed build were success- fully built manually. Additionally, in the manual experiment, five environments that were successfully built by Repo2Run were not successfully built, representing 15.6% the total successfully built assignments.
The table information and logs are too lengthy, so we place them on httphttt:///eet http:////noommooss.uuomu science/r/Repo2Hun
F.2 Industry-level repository building validation
As shown in Table 12
present the evaluation of Repo2Run 59 popular industry grade 1 shows the successful configuration of additional industry-grade
repositories. Additionally, Table 13 repositories.
Table 12: Evaluation of Repo2Run on 59 popular industry-grade repositories. The repositories selected from GitHub Ranking [60]. filtering out those without tests and with #LOC < 10,000.
Configuration state
#LOC 10,000
#LOC > 100,000
Medium
Max
Success(30) F2i)
30 (100%)
14 (46.7%)
96, 96,157
2,389,065
29 (100%)
24 (82.6%)
333,464
2,669,973
All(59)
59 (100%)
40 (9.3%)
124,352
2,669,973
Table 13: Successful additional configuration of 30 industry-gradt repositories.
Repository
#LOC
Repository
#LOC
Repository
#LOC
comfyanonymous/ ComfyUI
737,543
OpenBB-
2,389,065
scrapy/scrapy
77,278
finance/OpenBB
yt-dlp/yt-dlp
155,662
keras-team/keras
204,537
AUTOMATICI111 stable-diffusion- webui
40,448
pallets/flask
25,190
geekan/MetaGPT
82,305
psf/black
118,551
deepfakes/faceswap
74,690
streamlit/streamlit
201,894
psf/requests
12,782
Illyasviel/Fooocus
328,199
labmlai/annotated_ _pa-
510,851
RVC-Boss/GPT- SoVITS
34,658
per per_implementations
mingrammer/ dia- grams
11,994
OpenInterppeteer/ppnn- interpreter
15,818
hiyouga/LLaMMA Factory
76,000
nvbn/thefuck
10,267
pandas- dev/pandas
571,849
ytdl-org/youtube- dl
96,157
TlgAlggtss Python
419,917
fastapi/fastapi
177,120
Textualize/rich
45,744
All-Hands-
160,514
Stability-AI/
13,698
ultralyties/ultaaltiiss
94,201
Al/OpenHands
stablediffusion
appdio-app/gradio
184,722
QuiviHQ/quivr
10,264
freqtradeefreeqtrade
188,889
G
Repo2Run tools
Showing in Table 14 we design the following actions for Repo2Run facilitate its invocation.
32

---

Table 14: Command list and their functions.
Command
Function
waitinglist add -p package_nas _ame
Add item into waiting list. If no "ver- sion_constir "oan__osttrints" are specified, the latest version will be downloaded by default.
[-v Carsionn_contraiinn tool
waitinglist addfile file_path
Add all entries from file similar require- ments.txt format to the waiting list. Format should be package name [version constraints].
waitinglist clear conflictlist solve -V " [[version_.comsraaitta
Clear all items in the waiting list.
Resolve the conflict for the first element in the conflict list, and update the version constraints for the corresponding package_nis gaggee__mmmeaa and tool to ver- sion_ constraints. If "version_ "version_constralaa "onsttaaint""" specified, the latest version will be downloaded by default. The package_nan @ome and tool in the origi- nal waiting list must match one of the elements in the conflictlist. Here, the version constraints are specified.
conflictlist solve -u
Keep the original version constraint that exists in the waiting list, and discard the other version con- straints with the same name and tool the conflict list.
conflictlist clear conflictlist show
Clear all items in the conffict list.
Show all items the conflict list.
waitinglist show download
Show all items in the waiting list.
Download all pending items in the waiting list at once, and the conflict list must be empty before executing.
runtest
Check the built environment is correct us- ing"pytest".
poetryruntest
Check if the built environment is correct in the poetry environment. If you want to run tests in the poetry environment, run
runpipreqs
Generate and
requirements. wttpprreennts_pippesseeee ppppress..... txt
appregs_output... txt pipreqs_error. txt.
and
change_python_ version oython_version
Switching the Python version in the Docker con- tainer will forgo any installations made prior to the switch. The Python version number should be rep- resented directly with numbers and dots, without any quotation marks.
clear_configuration
Reset all the buildings to the initial setting of 3 10.
33

---

H
Repository building success status
As shown in Table 15 we list the success status of all the packages our constructed benchmark.
Table 15: Success status of each package in the benchmark.
full name
sha
successfull name
sha
success
271374667/Videouuson
9ba7b8
Yes
6abd/horus
c1d093
Yes
wrtr-oo-w/covvvieox factory
80d115
Yes tes-g93/noo4j-runye
16b441
Yes
Maditya-Praad//connssseeeoy policy
eed0c4
No AARG-FAN/Yolo for Wukong
07161a
No
adamoheng/wddhffs
5c68an
Yes Adibvafa/Codon Transformer
2842ef
Yes
AditaaNG/kan-gpt
0c6c4c
Yes
dc332e
Yes
AgentOpssAA/AgeenSttee
ff9c6a
Yes bidaatookkollmms-ccchmaat
c6a5fd
Yes
AIR-Bench/AARR-ennccs
4b27b8
Yes airbyteho/Py
7e65ab
Yes
airtai/fastagency
If1503
Yes Akkudoktor-EOOS/EO
fT1685
Yes
alexmolas/miccrosaccch
632f12
Yes alexwichan/ssfarr-webbbhhh
0e4974
Yes
AlibabaPA//lummixx
b319b2
Yes Al-HHads-Al/Openhhall
246107
Yes
alvin-r/databonsai
3f2b7c
Yes amchi//t signer
926dbb
Yes
andrewyng/aisuite
763996
Yes undrewyng/rranslation agent
c0fc60
Yes
AnswerDorAA/byalle
4583c0
Yes AmswerDooA//rerankerss
ecd1
Yes
amigroup/agentUniiv Universe
ed8155
Yes apapiu/raassormmer latent difffusio
84a75e
Yes
apify/crawlee -ethon
267063
Yes apple/ToolSandbox
Ialde8
Yes
apple/ml- htae/mmllloosooooeeeeeeoooo cross-entropy
1f3ebd
Yes apple/ml- malm
9a5632
Yes
arcee- ai/fastmlx
Id37be
Yes argmaxinc/whisge whsperrkossssssettttooolsss
03898f
Yes
arvindrajan92/DDToon
al0aa0
Yes
Sstramind-ai/Aurann
c357al
Yes
atonderskinneuronnaa
ecdef2
Yes
aurclio-abs/eemnniiccuukkos
04acc2
Yes
AuvaLab//itext/ Azure/co-opp-rrassltoor
941ald
Yes
awslabwvagent-cvaluutioo
3df695
Yes
a4709c
Yes
61bde7
Yes
python
bananaml/fructoss
5124ec
Yes bast/MolPipeline
2f9bae
Yes
basf/mamba-tabulke
aflea0
Yes beatzxbb/min-toolboxx
728e35
Yes
DellingcatShaddwFinnde
578d5a
Yes Benexl/FastAnime
677f46
Yes
betaacid/FastAPP Reference- App
8cacca
Yes bhavnicksm/chonkic
990493
Yes
bigcode-prog project/bigcodebnnch
aa634d
Yes
B13f/yato
4906b0
Yes
Blealtan/efficient-ka
7b6ce1
Yes block/goose
c497a5
Yes
BMPixel/moffee
0c643d
Yes boheumd/MA-LMMM
ffe9fa
Yes
bytewiz3/TraveIGPT
b19b43
Yes Causall.caming/ReAct
7d3665
No
Gaahlgrenl/bbsereern
d46fdb
Yes chaidiscoverr/ccaiilBe
b6e7fa
Yes
pharacter-ai/promp--ooe
466432
Yes chealijs/palworlddseeeeeee
7dc2c7
Yes
chernyadew/bigym
724305
Yes chrschy/fact-fing finder
ca57d1
Yes
447511
Yes cloudfarreclloufffrr--ryo
228479
Yes
c93924
No
f88922
Yes
codematrixe//mmdriver2
c0d075
Yes
codeskyblne/tidevviee
d83c34
Yes
codeskyblue/uiaatoddv
eb8577
Yes Codium-ai/AlphaaCodin
eb7577
Yes
madlium-ai/cooer-ageet
Sc4b88
Yes DOLLA-LaorratrryTraane
de8bf3
No
7711db
Yes ConpEppigen/iigeno
14b904
Yes
computeerageentssagen---tt
d7f6cb
Yes
cosmic-cortex/mlffz
Sbf8d2
Yes
cremebrule/digital cousiins
49400b
Yes
crowAllnndcrewwAllooo
873935 Yes
ovg/nerf-oe on -net-----theee
3659e7
Yes
D D-Star-Al/dsRAC
2d5431
Yes
Vincc//Scrapling dai- tai-mmotoki/zoltraakk
012820
Yes DAGWorks Inc/burr
79137e
Yes
4dce44
Yes darrenburns/postingg
94feah
Yes
Dataformer I/dataformer
Ocf88c
Yes daxa-ai/pebhlo
c67h01
Yes
dayaa5576hhaverhaaiis
c01257
Yes
d6c6ac
Yes
26033f
Yes sandrrte-systems/deddeeee pythoee--ththhhtht
27c9da
Yes
denssrr-orgddeessr--tttiiee
76256e
No Lingo-actual/iniiii-raaftff tranrrrrriiireormeerrrr
08d0al
Yes
discord/access
19e9b1
Yes Dleemiller/WordLlamm
e38d47
Yes
dongxuyue/Open- Satpuuuueeepppeeeepeeeetttiiiii
83f0ac
No dottxt-ai/outines core
31ab9f
Yes
dottxt-ai/ ai/prompts
3d2689
Yes dreadnode/riging
82ac80
Yes
droid-datee doodddddasst/droodd. httpi////ddrridoooooo_o_ee policy_lcarning
205ff6
Yes DS4SD/docling
acc9c0
Yes
dynamiq-aiddynamig
6ccale
Yes lakmanng/sqlframe
61 Yes
EleutherAl/sae
0483b5
Yes emefithepipe
02c397
Yes
Continued on next page
34

---

Table - - from previous page
full name
sha
successfull name
sha
success
Emerging-A/ENNOWE AV/ENOVA
b3661d
Yes
EnhancedJax/Bagels
d72d7f
Yes
cnoch3711/Extract/
4872a7
No ppil-open-sourcc/scssmmmees
b3c812
Yes
spistoteles/Tensorluse
1564fa
Yes epogrebnyak/jusqpn
Oaca51
Yes
erezsh/reladiff
d8683b
Yes etianen/logot
54e5ef
Yes
expectedparroo//dsll
aa7a2d
No explosion/spacy-layoon
64c6f4
Yes
faceboorrsearrchaadiioel
cal0r5
Yes facebookressaarchiiighhnaaaa
34fe6c
Yes
facebookresearcchSpprrtt
52fb2f
Yes FalkorDB/GraphRAAG
250ebe
Yes
Pangi-Lin/Daate Scaling- Laws
bd6941
No fastapi/fastapi-ch
bc0840
Yes
fedirz/faster-ww
fedir//faster-whiserrssssss whisper-server
cbb6c9
Yes felafax/felafax
34a475
Yes
fllpsttrandmmflux
627398
Yes FlagOpen/FlagGiems
ca13b7
No
00fef7
Yes soandaaioonmmdddl-ttcc//msss
408c75
Yes
ipgmass/cookiecuteree.
90de47
Yes
frdel/agent-zerro
3cefal
Yes
frostming/tetos
106ea5
Yes
74753f
No
gauge-sh/bridge
8b3430
Yes Genentech/gReLU
efd308
Yes
ennomnnclogyy/uuz/yyy
d96243
Yes getludic/ludic
a6db96
Yes
getzep/graphiti
9f3dd5
Yes GigaxGames/gigax
c3c209
Yes
gojasper/tlash-diieee diffusion
48c3be
Yes gomate-communityyTuusi..G
1334c4
Yes
gomale-communniyyrgeeaaa
0le258
Yes google-deppminn/nanndoo
10aefd
Yes
songle-deepmind/penzai
fda6ed
Yes google-depmmin//reecopo
dac 18b
Yes
gongle-research/imessff
02be2f Yes
goombalab/hydra
b6b9h7
Yes
/pustack/gpustack
4f0c67
Yes
gregptt7/browser-usk
5e545d
Yes
grog/groq python
fa2e13
Yes gusyel234/nano-gnnn graphrag
18fa3a
Yes
mailo-ai/haile exaamleess
82cfc8
Yes
Haiyang- -/GGT
ef2b64
No
Hanaoka Auzia/Gmmini-Pl
e8a2d2
Yes HATTTER-L.ON//Verbiveer
821988 Yes
hinthonw/trusscall HKUDS/UrhanGPT
caaaad
Yes
HKUDS/HiGPT
260793
No
bc3515
No
hngprojects/
bc9740
Yes
mmgg_hoilerplate_pyvee
hpeaitech/Opeen-Son
38de63 Yes
lppaatech/Swiftlneer
239rd3
No
hrmoh24/stream-vvc
faa629
Yes Iuchenlei/ComffUU omost
7ef00d
Yes
huggingfacc/lerooot
4c4
Yes huggingfaac//ighteeal
6ad727
Yes
HZAI--JNU/Mamba- YOLO
ca97fc
No
IAAR- SuuR-SaanhaaiGGrmooi
3fc89d
Yes
bb-apirccooaded/bb __ayyyy _asyne
38cf54
Yes IBM/fastfit
396611
Yes
IBM/trrratrrc
16e5af
Yes
IEIT Yuan/ Yaaan//uan..00 2atnn....MMM..
b403a2 No
wwrhenav/SSLLModdlbbolllll
2ead04
Yes igorbenav/fastrrud
dc831b
Yes
1grek
0d6079
Yes illuin-tech/colpaii
e45c4c
Yes
469665
Yes Indoxer/I.KAN
16c48e No
lnfini-Al-Lab/Sequeel
688079
No instanseg/instanseg
0d18b2
Yes
instructlab/nntttuullabb
c978b2
Yes Integuru-A//Inteuunn
928e82
Yes
5ad2eb
Yes awvariatlabs-aiiinnaan
81547a
Yes
IST-DASLab/PaazaMMan
b1807c
Yes itcrativoddatachiin
h67d59
Yes
JyanDrokinnkorch-com--
7a0c83
Yes jahwag/llaudeSync
000633
Yes
jgravelle/pocketgroq
e995c4
Yes jhj05177AAdvaneeediveee
a7975c
Yes
jialuechen/deeepoliio
15d247
No jin-aa/late-chunkingg
db558c
Yes
jlowin/fastmcp
baa300
Yes jmschrei/tangeermenee
a96897
Yes
jonbarron/camp zipnerf
8e6d57
Yes JosepIJABBBIIDDARAA//yyyaaaaat
826930
Yes
c72f48
Yes jshuadvd/LongRRPP
eb9aba
Yes
salln-levels-of-rae
2ce110
Yes karpathy/minbpe
lacefe
Yes
kennettreitz/simpleennd
39b5a5
Yes kevinzakka/mink
cfla30
Yes
knowsuchaggency//roppii kohjingyu/eaarc--agents
a1930c
Yes koaning/uvtrick
2d7f27
Yes
7c35ac
No
KOSAASSHHpinnxxs--aatnoooo
7feff4 No
hanking-networr
kotaro- kinoshita/yomitoku
71c85b
Yes KruxAA/rabbuilder
db3d3d
No
jjjrraandd/keasygui-pppr
python
b1f293
No
KyanChen/RSMamba
3fa305
Yes
58db40
No landing-ai/viionn-ae...
63eab8
Yes
064e5b
Yes lavaguc-ai/La Vague
b3557f
Yes
Lazy LLL
c0dd38
No lenML/Speech-A Forge
0b31b2 Yes
lcopincy/eeralnoisee
c0313f
Yes Bachao-snnMMore
7a030c No
Baatminng-A//ltdaall
0a97de
Yes lightonai/pylan
8de 184
No
LilianHolllrd//eeYY..
872841
Yes line/ligthousee
ba9da7 Yes
LimKirraffas--lngdddeee
5728ba
Yes
LMCache/LMMaahe
7d3443 Yes
Continued next page
35

---

Table c from previous page
full name
sha successfull name
sha
success
imstuiioaa/mllx-one
dacb7a
Yes
Imstudio-ai/venvstaek
235ce3
Yes
uccadeeimanoguurrraPPyoooo
ed391c
Yes Lacas/aaudaannapkseenn
3b3c62
Yes
bucidrains/alphaffld--yy
49[7c9
Yes
5774bb
Yes
vucidrainsspiizerrrppyoree
8ad66f
Yes
lucidrains/tiiok-yyooe
219525
Yes
aucdraais/trannstsiiooppyyolll
16f73e
Yes
M/el
36caSe
Yes
ManiMozaffaraaioolock
3d196b
Yes
YaoXiaoYYZZ/Loogg -ovee--oelG-iii
e952ac
No
Marker Inc-
aaObtb
Yes
martius- lawihithhiiiiiiinnnnigii h rotations
45b49f
Yes
abodiii/emboodie--ae
-ggents
871516
Yes
McGill-NLL -NLl/weblxxww
6/2014
Yes
McGill-NN.P/welllan
696a7c
Yes
Menghuan 991 8/pdfdeal
e08199
Yes
meta-llama/llama-staee -aap--a---
f14a73
Yes
MetaGI M/shippaii-sttt-ppyooooo
7ff4de
Yes
metavoiceinlmetavviiessr
dc3fa2
Yes
microsof//MInferencee
7a3e5a
No
microsoft/Ting Troupe
9b8d4e
Yes
microsoft
826c15
Yes
microsoft/aurora
8b1165
Yes microsoft/graphragg
de1252
Yes
microsof//semanticliin link--lbbs
8e37ef
Yes mikekelly/AgentK
e9cc89
Yes
MindinventoryMinnd@Q...
3d0ff0
Yes
Minishl Lbboodeeaemmoodd///ee
4c3fba
Yes
miquido/draive
270f0c
Yes mistralai/mistralccoom
5cac5c
Yes
eestralaimmistrl-finneee
656dfl
Yes mixedbread-ai/bagueee
a6c915
Yes
mixedbread-ai//mee ai/batched
Ia1797
Yes mkji2/lockbox
58430d
Yes
mlccauchoissmmiccrgrraaaann
ablca0
Yes MLLT-OSS/pen--asiitatttsa
44eeaf
Yes
milx-graphs/mlx--rapp
4619d9
No mobiusml/gemlinn
Sehcca
Yes
ModelCCooud/PPiQMMModll
aSaefc
No modélconeetprroccoolyytton
aaf32b
Yes
Modeesscop/MMemorySae
330b76
Yes modelscoope/ngentssoppe
ceaf89
Yes
modern
tython/hhaat-deeendd
65e656
Yes muchadogesec//hssooryfffeee
614182
Yes
mudithargaaad66PPXLSSMMM
f3c9hb
Yes narwhals-dew/narwhalk
a2088f
Yes
nasa-jpl/rosa
5471dc
Yes newd//heeli-graahrra--yyyyy python
Oac06b
Yes
neuralmagic/AuuoFPS
c94461
Yes neuralmagie/guiielllm
ecf298
Yes
NewT1223-WM/mlemm
50cc75
Yes NexaAl/nexa-sch
33f6ba
No
nicobrennnerccommannjool
4c7264 Yes Nike-Inc/kohessse
9bd29e
Yes
almatiies/nlmmingestor
c72542
Yes NLPSCL/RAAG-Retrrie..
d73057
No
nomic-ai/contrastors
496a05
No Noussesearrchfinetttnn-----
e2f5eb
Yes
6c92ac
No NVIDIA/Megatron Enneggon
28aa3b
Yes
NVIDIA/NeMM-SSN Skills
5591f3
Yes NVIDIA/kvpress
715f8a Yes
NVIDIAA/ogits-pront http:oogstttrocsso-zoooooo
db179a
Yes NVIDIAAnv-ingest
ccc9fa No
NVIabs/Sana
41dcbe
Yes NVlabs/VILA
ec7fb2 No
NVlabs/nyTorchhe Cor
cc27be
Yes oggaab2/ccoiic-trassaaaee
1933d1
Yes
Open-Winee Wone-Component/lumer
b0c0d4
Yes
openai/mle-bench
51ec2b
Yes
launcher
openai/swarm
9db581 Yes opendatalab/MMinetU
391a99
Yes
ogenfoundry-ai/modde manager
3419If
Yes opengeos/HlyerrCoss
c1604c
Yes
OpenInterpreterfaifs
3f74c6
Yes OpenNN.PLabblighhning
d74395
Yes
openpsi- projeis/-ooocc/RaaLL..H
62d9ed
Yes openrecalllopenrcaal
225a27
Yes
OpenSPG/KAG
68b2c6
No orbitul-mattriialsort-mmeet
251573
Yes
049b40
Yes
OwIAIProjec//Owl
919226
Yes
ec6717
Yes
paradigmxyz/spicc
e962a9
Yes
Handbook
c9b02b
Yes paulrobellopparlamma
421238
Yes
PeiJieSun/NESCL
365d20
Yes plinder-org/plinde
9658cc
No
pomonam/Kronftuence
884255
Yes TrefectHQ/ComtrolFFoo
f259fa
Yes
Prinehitellactaa//ppennilloooo
71f5c2
Yes Primelntellect-ai/prine
a974cf
Yes
wrinceton-nlp/SS--agee
8h3571
Yes proger/acceleraacddsccmm
db7145
Yes
pydantic/logíire
3d7924
Yes pymupdf/R AG
625718 No
pytorch-labs/Leank
a416e6
Yes raphaelmaansuylcodeeponn
3b377b
No
RapidAl/RapidDoe
SeSfef
Yes RapidAl/RapidLayour
8e9677
Yes
reagento/dishka
2ed985
Yes
real- real-stanforduuniversa
298776
No
cammipulltionnintrffie
Realiscrad/fish-at
f32c7f
Yes
RealOrangeOnnddjjagottt
c6d26c
Yes
ciijs/text-schedull
8bb7d6
Yes
reka-ai/reka wawi/reka-vibb-eeee vibe-eval
93ecd9
Yes
remigenet/TKAN
8alde0
Yes rio-labs/rio
eda40a
Yes
robocasa/robocasa
271992
Yes
RobotecAl/rai
d15910
No
robusta-dev/holmesspt
c4743a
Yes
royreznik/exii
fidea8
Yes
Continued on next page
36

---

Table 15 - from previous page
full name
sha
successfull name
sha
success
run-
47cfff'
Yes run-lama/lllaaa _xttract
89438f Yes
run
f78186
Yes SamKhorc/ComfyLJJ-Der Deeeeee
edd7fe
No
ScrapGrraphAAlSSrraeegpppaa
bae92b
Yes seanchatmanggp/dspyyen
691305 No
erverfes-ccueeerrfoom-wwwww
2da837
No ServeriesLLL//SeererreeLLLL
8fle6b
Yes
ServiceNNow/BrooweeCCiyy
12aa5e
Yes ServiceNow/TTaeeAgenns
3eca5c Yes
ServiceNow/WorkAreen.
0ab9cb
No
ShaShekhar/aziclaa
4c8d6a No
shawntannssattermooe
63b76a
Yes
Shoggoth.A/mmotleyeeew
19837e
Yes
showlab/computer. _usc_oott
419d9d
Yes
shun mhun-liang/yt2doc
201ec2
Yes
siliconflow/Bizy Air
cdb3bb
Yes
simonw/filss-t-prroppp
f9a4d8
Yes
3imonw/lm-clauude33
c62bf2
Yes
simonw/llm-emd
74fb98
Yes
simonw/llm-jq
beaada
Yes
simular-ai/Agent-S
ca83he
Yes
sirocco- rrageuee
99dfe5
Yes
souzatharsis/podcattt
804a61
Yes
SpecterOps/cred ppy steinathan/reelsnnkke
432191
Yes
StacklokLabs/promppori
42f69b
Yes
75369c
Yes
stepheugpopeiio-cod--aadid architeets--
ffcla8
Yes
toolkit
Stony Brroooo.P//appooo.tlt
be9c47
Yes
Storia-Al/sage
f47fa4
No
superlincar-airraglie
b02c5a
Yes swarmzzro/swarmzer
6fcd7a
Yes
lahnok/colmi r02_ client
84d3a6
Yes laketwo/lIm--olamaa
dd616e
Yes
taobojienddjango-zzal
232987
Yes TencemtARC/BrussNeett
101dc3
No
TheASSinguariiyyrrpprrg--lea
beb98d
Yes
thousandbrrinspppjssttt....mm.
a39a26
Yes
ollama
THU-MIG/yolov10
6fbaf4
Yes
thu nics/MoA
da034c
No
Gimlabs/AgentRun
1997dd
Yes
tobiasfshr/map4d
0b8bed
Yes
Toloka/dbt-af
e7f436
Yes
TorchJD/torchjd
1caafe
Yes
tox-dev/to--uv
d7405a
Yes
Jaragal.ab//fllee
2c1088
Yes
uubb--lmm/urbo-alignnnes
009574
Yes Ny-Cheng/orchhiiecopuil
c3a477
Yes
ucbepic/docetl
00a761
Yes ultrasev/lmproxyy
1a1100
Yes
uname-n/deltabase
Seafb9
Yes unifyai/unify
ea2088
No
Vashkatsi/deply
6d6875
Yes tratt/fiaancill-attae
ffe6df
Yes
vittasotwware/jjjano---aaaiiai
5b26c7
Yes
985664
Yes
vllm
606aab
Yes
volcengine/verl
ed2caf
Yes
volfpeter/fasthx
c850b9
Yes
volfpeter/htmy
0322a3
Yes
rysakhO/dravid
25b03b
Yes warmshaooFasterLiveeoooo
6aa810
Yes
weareprestateechhhotpdff
55ab97
Yes webaareneex/isssalwwhaann
8915af
No
Weixiang-Sun/Bora
c08bb6
Yes whyhow-ai//nowllde---app grapssssggggaaaa-s-dtioooo
c41043
Yes
hyghow-ii/uuee-bsse--rrttee
91701f
Yes whyhow-ai/whyhow
63a3e6
Yes
YongKin Yuu/YOLL
b96c8e
Yes WU-CVGL/BAD-Gaune Gausssins
bdd8b3
Yes
wy-z/cootainer-vm
07d402
Yes xdit-projectxxDiT
a7bd51
Yes
Xhlucaabmm25s
c4fe12
Yes
e567c6
Yes
yihong
f5elca
Yes
yinjunbo/lS-Fusse
86c882
No
Detection
YUCHEN005/GenTTanssa
62c59d
No YUCHENOWS/RbbISEER
ad4e37 No
yurnjajalpanggaa--bnnch
eld12e
Yes
Alffastt-lll-secuuity-
6a867c
Yes
guardrails
zeroasiccorp/logik
ca4bbl
Yes
zhuqinfeng 1999/aamba
229687 No
zipnn/zipnn
007319
Yes gou-group/textgraa
b2dc68 Yes
37

---

Baseline settings
1.1
Pipreqs settings
Figure shows the template of a Dockerfile generated using "requirements_pprree txt" created by pipreqs.
FROM python:3.14
WORKDIR /
RUN apt-get update apt-get instaLL -y curl & \\
curL -SLL https///install..yyoooopooo org | python
ENV
PATH="/root/. Locp:///ooco///n:sPPPPAT
RUN
pip instaLL pytest pytest-xdist && 11 pip install pipdeptree & II git clone https: http:://gtthubb. com/ (uuuthoornnmme/(ppcceeee mkdir /repo && \\ git config --gLobal --add ciffeddrreytry /repo && \1 cp -r (package_ name)// /repo && rm -rf [package_name)) && 11 rm -rf (package_nane) name]
&&
COPY requirements_piirees
txt /requirements_ pettsssppppeeettxe txt bttprremmntt_piprprpppeee txt
RUN pip install -r RUN cd /repo && pytest -collecttonly -q
Figure 11: Dockerfile prompt using the "requirements.piprege txt" generated by pipreqs.
1.2
SWE-agent settings
Below the settings SWF-agent for building environment:
agent:
templates:
gystee_ttmplatte 1-
SETTING:
You are an expert skilled in environment configuration, operating directly in the command line with specialized interface.
The specialized interface consists of ([WINDOW)) lines of context at a time. In addition to typical bash commands, also the following commands to help you with environment setup
COMMANDS:
{[command_docs]}
Please note that THE EDIT COMMAND REQUIRES PROPER INDENTATION. Ensure correct indentation when modifying code.
RESPONSE FORMAT:
Your shell prompt is formatted as follows:
(Current task: <task name>) <cud> $
You need to format your output using two fields: discuesion and command.
Example:
DISCUSSION
I'll start by using is to the files in the root directory.
/repo
u""
You should only include a SINGLE command in the command section
and then wait for response. Everything in the DISCUSSION section will be saved for future reference.
You can use any other bash commands and the special commands listed above, but interactive session commands (e.g., python vim) are not supported. instance_templato:
38

---

We're currently setting up the environment for the following task. Here the details:
TASK:
([task,details))
INSTRUCTIONS:
Now, you'll carry out this task your Your terminal session has begun the repository
directory. Use the provided commands and any bash commands you need. Edit and check files as needed.
The goal is to generate a Dockerfile that can successfully build and the tests in the repository using the command pytest c-coleect -only -q".
NOTE:
1. The repository is cloned into /repo.
2. The Dockerfile should start with the following lines (if the base image is python:3.11 and repository adamobeng /eabbenn//ddbess sddbfs
FROM python:3.11
RUN pip install pytest
RUN git clone https://github. com wwwaaodngnndnabbaa...e
RUN mkdir /repo
RUN git config --global add safe. directory /repo
RUN cp -r /uddbfs/. /repo & rm -rf /uddbfe/
RUN -ff /uddbfs
un "
Your task includes:
0.
**Generate Dockerfile*s Create a file named "Dockerfile" in the root path (e.g, the absolute path is /Dockerfile).
1. **Read Directory Structure**; Check the folder structure the root directory.
2. ** Check the Configuration Files*:: Inspect files like requirements txt", peetup.py". "setup cfg", "Pipfile*", etc.
3. Determine Package Dependencies Handle dependencies and manage conflicting dependency versions.
4. **Testing**: Ensure "pytest /repo --collect- only -q" runs without errors
5. **Gerrate Dockerfile*:: Write necessary installation setup steps determined from the inspection. If you finish run testing successfully, you should modify the Dockerfile in /Dockerfile.
IMPORTANT TIPS:
Check the directory and files carefully.
Make sure Dockerfile commands are correct. Use proper Dockerfile syntax and indentation. Test the final Dockerfile by running "pytest /repo collect. only -q".
(Current taak: Generate Dockerfile) (Current directory: (frorking bash-$
next_step_templatee |-
[[observation])
(Current task: Generate Dockerfile)
(Current directory: (frorking_dir))
bash-$
text_step_no_onte wot_oooputtteeppateeeeeee 1-
Your command successfully and did produce any output. (Current task: Generate Dockerfile)
(Current directory: ((working.dir)))
bash-$
tools:
anv_variables:
WINDDW: 100
OVERLAP: 2
bundles:
39

---

path: tools/registry
path: tools/defaults
path: tools/search
path: tools/edit.lintingg
path: tools/submit
path: tools/env_setup
parse_function.
type: thought_action
history.processors
type: last_ Last_n_observations
n: 5
J Failure Case
Figure 12 illustrates "module not found" error when building the repository "jialuucchen/teepfolli""" In this the issuc arises from the absence of updating the unit tests. The test file "test_stats" attempts import modules from "deepf
stats". but "stats" does exist "deepfolio", Consequently, no matter how the LLM agent operates, it cannot directly run this test. This highlights the importance for developers to continuously update existing unit tests as the code repository evolves.
pests/testutiissteess stats.p
deepfolin/utils
_init_.py
deepfolio. utils.stats import (
hyperparamater_tunnng
assert 15 distance, assert is__s______ sss_uqu assert Eisert_is_symeerlle
metrics.py
risk management.pp
visualization.pi
Error: utils.stat module not found
Figure 12: An example of "module not found" error due to the absence of updating the unit tests in the repository.
40


Reviews:
[{"review_id": "EREeIGKr8j", "reviewer": "Reviewer_EWgw", "summary": "The paper presents a new coding agent framework, Repo2Run, which aims to build executable test environments (e.g., Docker + unit tests) for large-scale code repositories.\n\nSpecifically, it involves two phases: the build phase initializes a docker container and runs built-in tests with iterative agent-environment interactions. This phase involves both external and internal environments. The record phase aims to build a dockerfile based on the commands.", "questions": "See \"weakness\" for details.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. Automatic executable environment construction is an important task.\n2. The paper presents a pretty decent formulation of the environment construction task, where the goal is to generate a dockerfile for a given repo, and the main evaluation metric is the test success rate.\n\nWeakness:\n1. The dockerfile is directly generated by an LLM and would potentially contain malicious code (e.g., commands that take up too much disk space and break the system).  However, when evaluating the dockerfile, it will be directly executed without further examination. \n2. The evaluation is mainly on the SWE-Bench dataset with newly introduced metrics. While I agree that both the evaluation setting and the metrics are reasonable, the authors neglect a more relevant benchmark called SUPER: https://arxiv.org/abs/2409.07440. This paper presents the SUPER benchmark that directly evaluates the capability of LLMs in setting up and executing tasks from research repositories, which is more relevant than SWE-Bench.\n3. I'm not fully convinced that the authors are comparing to the strongest baselines. For instance, the authors mention OpenHands in the introduction section, but never compare to it, without any further explanations. Since OpenHands is also integrated with a set of execution tools, I'm interested in its performance on environment setup.", "was_revised": true, "final_justification": "Thanks for your clarification on the benchmark used and the detailed implementation of dockerfile generation.\n\nJust one clarification, the SUPER dataset has a \"dependencies\" split (~19.7% of the examples), which requires the agent to install or fix dependencies issues (please see Table 3 in the paper: https://arxiv.org/pdf/2409.07440 for details). I still believe it is directly relevant to your task of \"build executable environments\".", "rebuttal": "Thank you for your review. We provide the following clarifications in response to the weaknesses identified.\n\n**Regarding Weakness #1: Dockerfile generation and security**\n\nWe would like to begin by correcting a factual misunderstanding. The final Dockerfile is **not** directly generated by the LLM. Our framework's design prevents this:\n\n1. Our process is two-phased. In the build phase, the LLM's role is to explore and find a valid command sequence within a secure container. In the subsequent record phase, a deterministic **Dockerfile synthesizer** translates this proven success command sequence into the final Dockerfile. By design, commands that take too much disk space and break the system are not included in the synthesizer. The synthesis rule can be found in **Figure 5** and **Appendix B** in our paper. The LLM does not participate in this final synthesis step.\n\n2. To further validate the quality and safety of our output Dockerfile, we use `hadolint`, a widely-used linter that checks Dockerfiles for syntax errors, security vulnerabilities, and performance issues. As shown in **Table 5** of our paper, Dockerfiles from Repo2Run achieve the lowest warning count, demonstrating they are less prone to common security and performance pitfalls. We are also considering integrating `hadolint` directly into the Repo2Run framework as a quality gate in future work.\n\n**Regarding Weakness #2: The evaluation benchmark and SUPER**\n\nWe would like to clarify another factual point: **our evaluation is not conducted on the SWE-Bench dataset.**\n\n1. We construct our benchmark of 420 repositories to ensure scale and relevance. We are encouraged that our methodology was recognized by other reviewers as \"rigorous\" (Reviewer-S6iE) and \"sensible\" (Reviewer-q3cZ).\n\n2. Furthermore, the SUPER benchmark addresses a fundamentally different task. SUPER's goal is to **reproduce tasks from research papers with curated instructions**, whereas our goal is to **build executable environments for any arbitrary repository**. Our task is designed to solve a more foundational challenge in software engineering and AI for code, and thus involves a more general and complex problem space.\n\n**Regarding Weakness #3: Comparison with OpenHands**\n\nWe choose SWE-agent as our primary agent-based baseline for several well-considered reasons.\n\n1. We choose SWE-agent because it has an established technical precedent for interacting with Docker environments. Through its use in subsequent high-profile work like SWE-smith [1], a clear path for using SWE-agent in Docker-based build and test loops has been demonstrated. This makes it the most direct and scientifically sound baseline to compare against.\n\n2. In contrast, to the best of our knowledge, OpenHands does not have a readily available, established version that is adapted for this specific `docker build` interaction model. Our contribution is the Repo2Run framework itself, not the separate and significant research effort of adapting a generalist agent to this new domain.\n\n3. Therefore, comparing against SWE-agent allows for a fair evaluation of our specialized framework against the most relevant existing approach. Forcing a comparison with OpenHands would have required us to first design and build an adaptation for it, which is outside the scope of a baseline comparison.\n\nWe hope these clarifications, supported by new data and experiments, have fully addressed your concerns. Thank you again for your time and effort.\n\n[1] Yang J, Leret K, Jimenez C E, et al. Swe-smith: Scaling data for software engineering agents[J]. arXiv preprint arXiv:2504.21798, 2025."}, {"review_id": "vZEzLr2oER", "reviewer": "Reviewer_eMpx", "summary": "This paper introduces Repo2Run, an LLM-based agent designed to automate the construction of executable test environments for code repositories and generate runnable Dockerfiles. The approach adopts a dual-environment architecture (internal and external environments) and a Dockerfile synthesizer, and leverages operations such as rollback mechanisms, dependency installation, test running, and code editing. The authors claim that Repo2Run achieves an 86.0% success rate and generates 100% buildable Dockerfiles through evaluation on 420 Python repositories, significantly outperforming existing baselines such as SWE-agent and pipreqs.", "questions": "Can the authors provide more detailed arguments for the innovation of the “rollback mechanism” and “dual environment architecture” compared to Docker’s standard practices and the general agent-host control model? Renaming existing technologies does not constitute a true original contribution.\n\nThe EBSR metric ignores the actual success or failure of test cases, which makes the definition of “successful build” too loose. Why not use a stricter metric, such as requiring at least a certain percentage of test cases to pass? Among the 361 “successful” build cases, how many cases actually have the majority of tests failing?\n\nCan the authors elaborate on the details of the prompt engineering done when adapting SWE-agent to new tasks? Is it possible that the poor performance of the baseline is caused by poor prompts? \n\nWhat is ECSR in Tables 1 and 3? Is it EBSR?", "limitations": "No discussion of the generalizability of this method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper defines the EBSR as \"only caring whether the tests can be executed, regardless of whether they pass or fail\". This is an extremely loose success criterion. An \"executable environment\" that fails all tests due to configuration errors (such as missing environment variables, dependency version conflicts, test data files not found, etc.) cannot be considered a successful build in any practical application scenario. This definition seriously exaggerates the true capabilities of Repo2Run. \n\nThe core contribution of this research is over-packaged. The main components of Repo2Run, such as LLM-based agent, Docker container, rollback mechanism, and Dockerfile generator, are not new technical concepts. It combines the application of existing technologies.\n\nThe paper admits that its experiments \"have not repeated experiments many times to get error bars\" because they were time-consuming and expensive, which makes it impossible to judge the stability and contingency of the experimental results, weakening the scientific rigor of the conclusions.\n\nThe paper claims that \" for complex issues, Repo2Run shows greater advantages over manual building\", but does not provide any quantitative data to support this argument. \n\nThe experiments rely entirely on the closed-source model gpt-4o-2024-05-13 and are only tested on Python repositories, limiting the generalizability of their conclusions.", "was_revised": true, "final_justification": "The author has not yet addressed all my concerns, so I tend to maintain my original score.\n\nUpdate:\n\nAlthough I have raised my score, I wish to articulate my remaining concerns. In my view, the EBSR is not a sufficiently robust metric on its own. A success criterion that only considers whether tests can be executed—regardless of whether they pass or fail—is too permissive and can exaggerate the practical capabilities of the system.\n\nThe authors argued that “test pass rates can be a \"noisy\" signal for environment quality, as failures might stem from bugs in the source code or the tests themselves, not just from build errors”. However, I believe this issue can be mitigated by using human-verified benchmarks where the tests are known to pass in a correctly configured environment.\n\nNevertheless, I still see some value in this article, so after some hesitation, I chose to slightly lean towards weak accept this article.", "rebuttal": "We appreciate your detailed review and value the opportunity to clarify our design choices and contributions.\n\n# Response to Questions:\n\n**Regarding Question #1: Innovation of \"rollback\" and \"dual environment\"**\n\nOur work does not simply rename existing technologies. Our contribution is a novel, specialized architecture for the complex, stateful problem of automated executable environment building, a task for which general-purpose agent models are ill-equipped.\nOur framework is specifically designed for this task, and its core 'build-then-record' paradigm sets it apart from general agent-host models.\n\nOur *dual-environment architecture* is fundamentally different from a general agent-host model. It provides essential meta-level control, most notably through the `base_image_change` action, which allows the agent to switch the entire underlying OS and language version (e.g., from Python 3.10 to 3.6). An agent operating purely inside a container cannot perform such a critical recovery action.\n\nThe *rollback mechanism* is more than just `docker commit`; it is a strategic element of the agent's reasoning loop. The agent receives the full history of the failed attempt, allowing it to learn from the error and attempt a more effective sequence on a safe state, rather than getting stuck trying to patch the \"polluted\" environment.\n\n**Regarding Question #2: EBSR metric**\n\nWe chose the EBSR metric after careful consideration of the problem's practical realities.\n\n1. For any arbitrary \"in-the-wild\" repository, it is impossible to know the ground-truth test pass rate beforehand. Many repositories have legitimately failing tests as part of their development cycle. Manually establishing a baseline for all 420 repositories would be an immense and subjective undertaking, contrary to our goal of full automation.\n\n2. A failed test may have three possible root causes: (1) an incorrectly configured environment, (2) a bug in the test code itself, or (3) a bug in the source code that the test is correctly identifying. Simply counting failed tests does not distinguish between these cases and is therefore a noisy signal for environment quality.\n\n3. Given these factors, we chose EBSR as a practical metric, as it directly measures the agent's ability to create an environment where the developer can begin debugging—the primary goal.\n\n4. Due to time constraints, we ran the full test suites for 50 successfully built repositories, which took over 30 hours to execute the tests and check the pass number manually. The analysis shows that in 87.2% (8016/9190) of these tests, the majority of tests passed, and 36 repositories (72%) passed over 60% tests. It confirms that EBSR is a strong proxy for a useful environment.\n\n**Regarding Question #3: Prompt engineering for SWE-agent**\n\nWe ensure a fair comparison by following the standard practices for SWE-agent:\n\n1. We use the official SWE-agent framework, which has an established technical precedent for interacting with Docker environments, and its widely adopted prompt template. We only replace the task description from \"code fixing\" to \"executable environment building\", providing a clear workflow and examples. We use the `gpt-4o-2024-05-13` model and keep all other settings at default. The exact prompt is available in **Appendix H.2** for full transparency.\n\n2. We also ensure that the prompts for Repo2Run and SWE-agent are structurally consistent. The performance difference does not stem from \"poor prompts\" but from fundamental architectural limitations of the baseline. SWE-agent lacks the crucial two-phase build process and specialized actions (like `base image change`) that Repo2Run provides, making it incapable of solving many of the complex setup challenges in our benchmark.\n\n**Regarding Question #4: Typo**\n\nYou are correct. This was a typo and should have been EBSR. We apologize for the error and will correct it in the revised manuscript. Thank you for your careful reading.\n\n# Response to Weaknesses:\n\n**On the contribution being \"over-packaged\"**\n\nWe respectfully disagree with this characterization. Our innovation is a complete, specialized framework for a notoriously difficult problem. Simply combining existing technologies would not solve it.\n\nOur framework is purpose-built, with components like the rollback mechanism and Dockerfile synthesizer designed from the ground up for this task. The dual-environment architecture provides essential capabilities, like `base image change`, that are absent in general-purpose agents.\n\nWe are encouraged that this view is shared by other reviewers, who described our work as \"novel and clever\" (R-r9jR) and featuring an \"innovative agent architecture\" (R-S6iE).\n\n**On the lack of error bars/stability**\n\nTo quantitatively address this, we conduct a new experiment on stability. We randomly sample 50 repositories (40 built successfully, 80% EBSR) and rerun the build process two additional times. Repo2Run achieves the same success rate in these reruns. This confirms that once a valid environment path is found, our method is reliable and can deterministically reproduce the result.\n\n**On the lack of quantitative data for advantages over manual building**\n\nOur manual study is detailed in **Appendix E.1**. In that study, human experts successfully configured 26 of 40 repositories (65%), whereas Repo2Run succeeded on 32 of 40 (80%). Notably, Repo2Run solves all cases that the human experts could, plus six more that they could not. The raw data is also available via the anonymous link in our paper.\n\n**On generalizability (LLM and Language)**\n\n1. **LLM generalizability**: To investigate our framework's generalizability, we conduct a new experiment with Claude 3.7 Sonnet. Due to time and computational constraints, we randomly sample 50 repositories from our benchmark and replace the base LLM for all methods (Repo2Run, SWE-agent, and LLM generator) with `Claude 3.7 Sonnet`. The results for both Environment Building Success Rate (EBSR) and Dockerfile Generation Success Rate (DGSR) are presented below:\n\n| Base LLM | Repo2Run | SWE-agent | LLM generator |\n| --- | --- | --- | --- |\n| **EBSR** (GPT-4o) | 80% (40) | 8% (4) | 18% (9) |\n| **EBSR** (Claude 3.7 Sonnet) | 84% (42) | 32% (16) | 32% (16) |\n| **DGSR** (GPT-4o)| 100% (50) | 22% (11) | 46% (28) |\n| **DGSR** (Claude 3.7 Sonnet) | 100% (50) | 38% (19) | 64% (32) |\n\nOur analysis of these results is as follows:\nAs expected, the stronger reasoning capabilities of Claude 3.7 Sonnet lead to a performance increase across all methods. Repo2Run's EBSR improved from 80% to 84%, while SWE-agent and LLM generator see more substantial gains, rising from 8% to 32% and 18% to 32%, respectively.\n\nHowever, a great performance gap between Repo2Run and the baselines remains. The primary reason for this is architectural. The baselines lack a mechanism to guarantee that the generated Dockerfile is runnable. This is illustrated by the Dockerfile Generation Success Rate (DGSR). While Claude 3.7 Sonnet's improved understanding boosts the DGSR for the baselines, many of their generated Dockerfiles still fail to build. In contrast, Repo2Run's `build-then-record` design ensures a 100% DGSR by definition, as it only synthesizes a Dockerfile from a proven, successful command history.\n\nThis confirms that while a more powerful LLM benefits all approaches, the fundamental architectural innovations of Repo2Run are the key drivers of its superior reliability and success.\n\n2. **Language generalizability**: This is an excellent point, and we agree that demonstrating generalizability is a critical next step. While we focus on Python due to its prevalence, our framework's design is language-agnostic. It can be adapted to other languages by changing the Docker base image and installation tools, as shown below. The core structure does not need to change. We have already implemented a prototype for TypeScript/JavaScript and plan to expand support for more programming languages.\n| Language | Docker base image | Installation tool |\n| --- | --- | --- |\n| Python | python:[version] | pip |\n| JavaScript/TypeScript | node:[version] | npm |\n| Java | OpenJDK:[version] | maven |\n| Rust | rust:[version] | cargo |\n| Ruby | ruby:[version] | bundler |\n| R | r-base:[version] | install.packages |\n| Go | golang:[version] | go get |\n| PHP | php:[version] | composer |\n\nWe will also add a paragraph discussing this challenge in the revised version.\n\nWe hope these clarifications, supported by new data and experiments, have fully addressed your concerns. Thank you again for your time and effort."}, {"review_id": "c9jdUDEn8d", "reviewer": "Reviewer_S6iE", "summary": "This paper addresses the challenging and practical problem of automating environment configuration for code repositories. It proposes **Repo2Run**, an LLM-driven agent that uses a dual-environment design and rollback mechanism to generate Dockerfiles and build reproducible environments at scale. The authors evaluate their approach on a benchmark of Python repositories, demonstrating strong performance when compared to baselines like SWE‑agent and pipreqs, and report high success in both environment builds (86%) and Dockerfile generation (100%).", "questions": "Please respond to the above weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- **Clear motivation & practical significance**: Automating environment setup is a longstanding pain point in software development; framing it as an LLM-based agent problem is timely and relevant to the ML+systems community.\n- **Innovative agent architecture**: The dual-environment sandbox plus rollback strategy is a clever solution to mutable build environments and dependency conflicts.\n- **Rigorous empirical evaluation**: Experiments span real-world Python repos, include strong baselines, and feature ablation studies—this lends credible support to the core claims.\n\n## Weaknesses\n\n1. **Limited LLM comparisons**: Experiments only use GPT‑4o. It remains unclear whether the method generalizes to other LLMs (e.g., Gemini-2.5, Claude-3.7). Additionally, the LLM‑generator baseline could benefit from swapping in different reasoning models.\n2. **Lack of cost/resource analysis**: Although build times (~29 min) are reported, there is no discussion of financial cost.\n3. **Reliance on unit tests**: The system depends heavily on existing unit tests for feedback. Many real-world repos lack tests, limiting applicability. No fallback mechanisms are explored.", "was_revised": true, "final_justification": "I want to keep my original score.", "rebuttal": "We sincerely thank you for your positive feedback on our work's motivation, innovative architecture, and rigorous evaluation. We appreciate the opportunity to address your constructive questions.\n\n**Regarding Weakness #1: Limited LLM comparisons**\n\nThis is an excellent point. To investigate our framework's generalizability, we conduct a new experiment with `Claude 3.7 Sonnet`. Due to time and computational constraints, we randomly sample 50 repositories from our benchmark and replace the base LLM for all LLM-driven methods (Repo2Run, SWE-agent, and LLM generator) with Claude 3.7 Sonnet. The results for both Environment Building Success Rate (EBSR) and Dockerfile Generation Success Rate (DGSR) are presented below:\n\n| Base LLM | Repo2Run | SWE-agent | LLM generator |\n| --- | --- | --- | --- |\n| **EBSR** (GPT-4o) | 80% (40) | 8% (4) | 18% (9) |\n| **EBSR** (Claude 3.7 Sonnet) | 84% (42) | 32% (16) | 32% (16) |\n| **DGSR** (GPT-4o)| 100% (50) | 22.0% (11) | 46% (28) |\n| **DGSR** (Claude 3.7 Sonnet) | 100% (50) | 38% (19) | 64% (32) |\n\nOur analysis of these results is as follows:\n\nAs expected, the stronger reasoning capabilities of Claude 3.7 Sonnet lead to a performance increase across all methods. Repo2Run's EBSR improved from 80% to 84%, while SWE-agent and LLM generator see more substantial gains, rising from 8% to 32% and 18% to 32%, respectively.\n\nHowever, a great performance gap between Repo2Run and the baselines remains. The primary reason for this is architectural. The baselines lack a mechanism to guarantee that the generated Dockerfile is runnable. This is illustrated by the Dockerfile Generation Success Rate (DGSR). While Claude 3.7 Sonnet's improved understanding boosts the DGSR for the baselines, many of their generated Dockerfiles still fail to build. In contrast, Repo2Run's `build-then-record` design ensures a 100% DGSR by definition, as it only synthesizes a Dockerfile from a proven, successful command history.\n\nThis confirms that while a more powerful LLM benefits all approaches, the fundamental architectural innovations of Repo2Run are the key drivers of its superior reliability and success.\n\n**Regarding Weakness #2: Lack of cost/resource analysis**\n\nThank you for this practical suggestion. We follow the precedent set by prior work, like SWE-agent [1], and set a maximum budget of 4 USD per repository. It is widely accepted by the research community. Across all 420 repositories in our benchmark (including both successful and failed attempts), the average cost per repository is approximately 3.56 USD. We believe this is a reasonable and acceptable cost for fully automating such a complex engineering task, especially when compared to the cost of manual human effort. We will add this analysis to the revised version.\n\n**Regarding Weakness #3: Reliance on unit tests**\n\nThis is an insightful observation. We choose unit tests as the primary feedback signal for two main reasons:\n\n1. Unit tests provide the most direct and reliable signal for verifying whether an environment is correctly configured to run the project's code as the original developers intended.\n\n2. A key application for our work is to enable large-scale training of reinforcement learning models for software engineering (e.g., SWE-RL [2] and SWE-Fixer [3]). In this domain, unit test outcomes are always considered the standard and most trusted form of feedback signal.\n\nWhile our current focus is on repositories with existing tests, we agree that handling projects without them is also an important future direction. For such cases, one could envision a fallback mechanism where the agent uses static analysis checks as a proxy for success. For example, the agent's goal could be to create an environment where a linter (like `pylint`) runs without import errors, or where a simple script can successfully import the main application modules. Exploring these alternative feedback signals is an avenue for future work, but it is beyond the scope of our paper.\n\nWe hope these clarifications, supported by new data and experiments, have fully addressed your concerns. Thank you again for your time and effort.\n\n[1] Yang J, Jimenez C E, Wettig A, et al. SWE-agent: agent-computer interfaces enable automated software engineering[C]//Proceedings of the 38th International Conference on Neural Information Processing Systems. 2024: 50528-50652.\n\n[2] Wei Y, Duchenne O, Copet J, et al. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution[J]. arXiv preprint arXiv:2502.18449, 2025.\n\n[3] Xie C, Li B, Gao C, et al. Swe-fixer: Training open-source llms for effective and efficient github issue resolution[J]. arXiv preprint arXiv:2501.05040, 2025."}, {"review_id": "INgJnFlVHh", "reviewer": "Reviewer_r9jR", "summary": "The authors propose Repo2Run, an LLM-based agent designed for automatically building and generating Dockerfiles for Python repositories. Their approach entails a dual-environment architecture, with an internal environment corresponding to a Docker container, and an external environment which serves as the bridge between the LLM and internal environment. The external environment is equipped with a \"rollback\" feature to discard erroneous commands that would otherwise \"pollute\" the environment. The agent can do many different things like select and change the base image, manage and install dependencies, utilize bash commands, edit files, and execute test suites. Once the agent successfully runs all tests, there is a Dockerfile synthesizer component which converts the command sequence from the building process into a runnable Dockerfile. For evaluation, the authors build a benchmark consisting of 420 Python repositories from GitHub (2024).", "questions": "- The underlying LLM for Repo2Run is gpt-4o-2024-05-13. Is the same underlying model used for the \"LLM generator\" and \"SWE-agent\" baselines? If it is not, then it seems that much of the difference in performance could be explained due to differences in the base model.\n- Due to the rollback mechanism, it seems like the LLM is not given enough information to do self-repair. As a result, do you observe the same erroneous commands being repeatedly generated in subsequent steps, following a rollback?\n\n\nSuggestions:\n\n- L2: engineerin -> engineering\n- The ablation without rollback is not clear. In Table 3, it seems that the only difference between the top 2 rows is the removal of rollback in row #2. In that case, it seems that the performance is actually better without rollback as the numbers in row #2 are higher than row #1. However, I assume the \"w/o rollback\" includes everything in the full Repo2Run setting including the dual-environment, except for rollback. \n- L213-217: Is rollback enabled for the LLM generator? This is a bit confusing.", "limitations": "This work only considers Python repositories. There are no empirical results to suggest that this will generalize to other programming languages.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The authors tackle a very impactful problem — scaling up executable environments for training and evaluating code LLMs is critical for the research community, and the authors have proposed a very novel and clever approach for doing this in an automated manner.\n- The findings related to the dual-architecture and rollback mechanism are quite interesting and may be relevant to other tasks for LLM agents. \n- The authors have done an excellent job with their analysis. They have extensively covered ablation results, error patterns, performance across different domains, and latency.\n\nWeaknesses:\n- The authors have chosen to focus on Python only. It is not clear whether the results will be as high for other languages, as building environments for other languages may be more difficult.", "was_revised": true, "final_justification": "The authors have addressed the points I raised. I still think this is a good paper that should be accepted. I will keep my score the same.", "rebuttal": "We are incredibly grateful for your enthusiastic and insightful review. Thank you for recognizing the impact of our work and the novelty of our approach.\n\n**Response to Questions**\n\n1. **On the underlying LLM for baselines:**\n\nThis is a crucial point for ensuring a fair comparison, and we thank you for raising it. Yes, you are absolutely correct. All LLM-based methods in our paper, including our full Repo2Run agent, the SWE-agent baseline, and the LLM generator baseline, use the exact same underlying model: `gpt-4o-2024-05-13`. We make this deliberate choice to ensure that the significant performance gains demonstrated by Repo2Run stem directly from its superior architectural design and not from an unfair advantage in the base model.\n\n2. **On the rollback mechanism and self-repair:**\n\nThis is a very insightful question that gets to the core of our agent's learning process. Our design of the rollback mechanism is carefully considered. While the mechanism reverts the environment state to a clean checkpoint, **it does not erase the agent's memory**. The full context of the failed attempt, including the command that leads to the error and the resulting error message, is explicitly preserved and fed back into the agent's history.\n\nThis ensures the LLM is fully aware of what is wrong, prompting it to learn from the failure and attempt a different, often more effective, strategy (e.g., switching from `pip` to `apt-get`, or changing a dependency version). We do not observe the agent repeatedly making the same mistakes in our experiments, which validates this strategic choice.\n\n**Response to Suggestions**\n\nWe sincerely thank you for your meticulous reading and for providing such helpful suggestions.\n\n1. On the \"engineerin\" typo:\n\nThank you for catching this! We will correct it in the revised manuscript.\n\n2. On the clarity of the ablation study:\n\nYour understanding is correct: the \"w/o rollback\" ablation removes only the rollback feature while keeping the rest of the dual-environment architecture. We will revise the caption and the corresponding text for Table 3 to make the structure and results of our ablation studies clear and unambiguous.\n\n3. On rollback for the \"LLM generator\":\n\nThank you for seeking this clarification. It is not enabled for any of the baselines, including the \"LLM generator\". We will make this distinction more explicit in the paper.\n\n**Response to Limitations (Extending to other languages)**\n\nThis is an excellent point, and we agree that demonstrating generalizability is a critical next step. While we focus on Python due to its prevalence, our framework's design is language-agnostic. It can be adapted to other languages by changing the Docker base image and installation tools, as shown below. The core structure does not need to change. We have already implemented a prototype for TypeScript/JavaScript and plan to expand support for more programming languages.\n| Language | Docker base image | Installation tool |\n| --- | --- | --- |\n| Python | python:[version] | pip |\n| JavaScript/TypeScript | node:[version] | npm |\n| Java | OpenJDK:[version] | maven |\n| Rust | rust:[version] | cargo |\n| Ruby | ruby:[version] | bundler |\n| R | r-base:[version] | install.packages |\n| Go | golang:[version] | go get |\n| PHP | php:[version] | composer |\n\nWe will also add a paragraph discussing this challenge in the revised version.\n\nThank you once again for your strong support and for your detailed feedback, which has been invaluable to us."}, {"review_id": "UwTWJY0V2g", "reviewer": "Reviewer_q3cZ", "summary": "The paper proposes an agentic solution to automate Dockerfile generation for building executable environment for code repositories at scale. It consists of two phases. In the build phase, the agent interacts with containerized environments to reach a state where all tests are runnable. During this process, the agent can not only execute commands inside the container, but also roll back to a previous container state (checkpointed through *docker commit*), or start over with another base image. In the second record phase, the agent synthesizes a Dockerfile by referencing the command history in the first phase, mapping each command to its corresponding Dockerfile statement step by step. Experiment results show that the proposed Repo2Run method successfully built Dockerfiles of executable environments for 361 out of 420 repositories, significantly outperforming baselines. Further analysis demonstrates the importance of using an external environment for rollback and base image change, and the effectiveness of employing the step-by-step conversion of command history to write the Dockerfile.", "questions": "* The two sentences in L265-266 seem duplicated. \n* The first sentence in section 6.1 looks abrupt.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Building executable environments has been a key challenge for scalable RLEF (Reinforcement Learning with Execution Feedback) due to the intense human effort required. Repo2Run is a very timely work that addresses the problem by automating it through LLMs. The paper presents very promising results: 86% of the repos can be successfully built and the time cost is comparable to human. \n* While the task is relatively new, the experiment setup, including benchmark construction and baseline choices, is sensible.\n* The paper is written clearly and easy to follow.\n\nWeaknesses:\n* The evaluation only assesses whether all tests are runnable but does not take pass/fail into consideration. The validity of such an approach heavily depends on how the tests are written. For example, if the test files wrap all local imports inside try clauses and handle exceptions as test failures, the agent can then have a shortcut by building an environment only for the test files but not for the source files. To mitigate this, did you consider leveraging human-verified environments (e.g. from SWE-Smith) for evaluation, where the targeted tests are guaranteed to succeed in the groundtruth environment. \n* The paper only focuses on Python. While extending the implementation to other programming languages requires a considerable amount of effort and might be out of scope, it is worth having a paragraph discussing the potential challenges for doing so. For example, in the absence of pip, can LLMs figure out the correct package manager based on its own knowledge?", "was_revised": true, "final_justification": "I do not have major concerns with the paper and will keep my score.", "rebuttal": "We are sincerely grateful for your incredibly supportive and insightful review.\n\n# Response to Weaknesses\n\n## 1. On the evaluation metric and leveraging human-verified environments.\nThis is a very insightful point, and we thank you for raising it. We would like to address the two parts of your suggestion separately.\n\n**On considering test pass/fail rates**\n\nThis is an excellent suggestion, and we do consider it deeply during our experimental design. We choose EBSR as our primary metric for several practical reasons:\n\n  (1) For an arbitrary repository, the ground-truth pass rate is unknown. Many have legitimately failing tests, and establishing a reliable baseline for 420 diverse projects would require a massive, subjective manual effort. We lack the personnel to undertake this task. Moreover, it runs counter to our automation goal.\n\n  (2) A failed test can stem from multiple causes, like environment, test bug, source code bug, or wrapping imports in `try...except` clauses, etc. It makes the pass rate a potentially noisy signal for environment quality alone.\n\n  (3) To validate our choice, we conduct a new experiment. We randomly sample 50 successfully built repositories and run the full test suites. This process alone took over 30 hours. The analysis reveals that in these successfully built environments, a majority of tests indeed pass. Specifically, 87.2% of all tests (8,016 out of 9,190) pass, and 36 out of 50 repositories (72%) have a pass rate exceeding 60%. It confirms that EBSR is indeed a proxy for a highly functional and useful environment.\n  \n**On leveraging human-verified environments like SWE-Smith**\n\nThis is a fantastic idea for future validation. At the time we initiated this work, benchmarks like SWE-Smith were not yet available. Our initial manual experiments on a smaller scale confirm that just getting tests to run is an incredibly time-consuming and difficult task for humans, which motivates our large-scale automated approach. We now see a great opportunity to use curated benchmarks like SWE-Smith as a \"gold standard\" for evaluating the correctness of our generated environments in future work. Thank you for this suggestion.\n\n## 2. On discussing generalizability to other languages\n\nThis is an excellent point, and we agree that demonstrating generalizability is a critical next step. While we focus on Python due to its prevalence, our framework's design is language-agnostic. It can be adapted to other languages by changing the base image and installation tools, as shown below. We have already implemented a prototype for TypeScript/JavaScript and plan to expand support for more programming languages.\n\n| Language | Docker base image | Installation tool |\n| --- | --- | --- |\n| Python | python:[version] | pip |\n| JavaScript/TypeScript | node:[version] | npm |\n| Java | OpenJDK:[version] | maven |\n| Rust | rust:[version] | cargo |\n| Ruby | ruby:[version] | bundler |\n| R | r-base:[version] | install.packages |\n| Go | golang:[version] | go get |\n| PHP | php:[version] | composer |\n\nWe will add a paragraph discussing these potential challenges in the revised version.\n\n# Response to Questions\n\n1. On the duplicated sentences in L265-266:\nThank you for your meticulous reading! You are absolutely right; the sentences are redundant. We will correct this in our revised manuscript.\n\n2. On the abrupt first sentence in Section 6.1:\nThank you for the helpful suggestion on our writing. We will revise the opening of this section to ensure a smoother transition.\n\nThank you once again for your positive feedback and the constructive suggestions that have helped us strengthen our paper."}]

Author rebuttal:

Meta-review: - Summary: This paper proposes Repo2Run, an agentic approach to automatically build test environments for code repositories. The approach involves iteratively building the Docker image of a repository until the agent successfully runs all the tests in the repository in an internal environment as a Docker container. The commands during this phase are then synthesized to an executable Dockerfile for reconstructing the executable environment. The paper also introduces a benchmark composed of repositories from Github with unit tests. 
- Strengths:
  - The paper explores a significant problem for training code LLMs enabling further research facilitating training with RLEF. The proposed approach is novel.
  - The paper is well written.
  - Experiments are rigorous and designed well.
- Weaknesses:
  - The generalizability of the method beyond Python remains unknown. 
  - The reviewers raised good points around leveraging pass/fail metrics for unit tests, leveraging different models with reasoning capabilities, which were addressed during rebuttal.
  - The proposed approach has heavy dependencies on the existence of unit tests in repositories, which might be limiting.  
- Recommendation:
  - All of the five reviewers recommended acceptance of the paper, three of whom gave borderline acceptance scores.
  - I believe this paper will be a good contribution to Neurips 2025. The paper tackles an important research problem enabling further research for training code LLMs and will be of interest to the Neurips community. I recommend its acceptance.
Decision: Accept (spotlight)