Title: Repo2Run: Automated Building Executable Environment for Code Repository at Scale
Venue: NeurIPS 2025
Accepted: yes
Keywords: Code data scaling

Large language model

Executable environment building
Abstract: Scaling up executable code data is significant for improving language models’ software engineering capability. The intricate nature of the process makes it labor-intensive, time-consuming and expert-knowledge-dependent to build a large number of executable code repositories, limiting the scalability of existing work based on running tests. The primary bottleneck lies in the automated building of test environments for different repositories, which is an essential yet underexplored task. To mitigate the gap, we introduce Repo2Run, the first LLM-based agent aiming at automating the building of executable test environments for any repositories at scale. Specifically, given a code repository, Repo2Run iteratively builds the Docker image, runs unit tests based on the feedback of the building, and synthesizes the Dockerfile until the entire pipeline is executed successfully. The resulting Dockerfile can then be used to create Docker container environments for running code and tests. We created a benchmark containing 420 Python repositories with unit tests for evaluation. The results illustrate that Repo2Run achieves an 86.0% success rate, outperforming SWE-agent by 77.0%. The resources of Repo2Run are available at https://github.com/bytedance/Repo2Run.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an agentic solution to automate Dockerfile generation for building executable environment for code repositories at scale. It consists of two phases. In the build phase, the agent interacts with containerized environments to reach a state where all tests are runnable. During this process, the agent can not only execute commands inside the container, but also roll back to a previous container state (checkpointed through *docker commit*), or start over with another base image. In the second record phase, the agent synthesizes a Dockerfile by referencing the command history in the first phase, mapping each command to its corresponding Dockerfile statement step by step. Experiment results show that the proposed Repo2Run method successfully built Dockerfiles of executable environments for 361 out of 420 repositories, significantly outperforming baselines. Further analysis demonstrates the importance of using an external environment for rollback and base image change, and the effectiveness of employing the step-by-step conversion of command history to write the Dockerfile.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Building executable environments has been a key challenge for scalable RLEF (Reinforcement Learning with Execution Feedback) due to the intense human effort required. Repo2Run is a very timely work that addresses the problem by automating it through LLMs. The paper presents very promising results: 86% of the repos can be successfully built and the time cost is comparable to human. \n* While the task is relatively new, the experiment setup, including benchmark construction and baseline choices, is sensible.\n* The paper is written clearly and easy to follow.\n\nWeaknesses:\n* The evaluation only assesses whether all tests are runnable but does not take pass/fail into consideration. The validity of such an approach heavily depends on how the tests are written. For example, if the test files wrap all local imports inside try clauses and handle exceptions as test failures, the agent can then have a shortcut by building an environment only for the test files but not for the source files. To mitigate this, did you consider leveraging human-verified environments (e.g. from SWE-Smith) for evaluation, where the targeted tests are guaranteed to succeed in the groundtruth environment. \n* The paper only focuses on Python. While extending the implementation to other programming languages requires a considerable amount of effort and might be out of scope, it is worth having a paragraph discussing the potential challenges for doing so. For example, in the absence of pip, can LLMs figure out the correct package manager based on its own knowledge?\n\nQUESTIONS:\n* The two sentences in L265-266 seem duplicated. \n* The first sentence in section 6.1 looks abrupt.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Repo2Run, an LLM-based agent designed for automatically building and generating Dockerfiles for Python repositories. Their approach entails a dual-environment architecture, with an internal environment corresponding to a Docker container, and an external environment which serves as the bridge between the LLM and internal environment. The external environment is equipped with a \"rollback\" feature to discard erroneous commands that would otherwise \"pollute\" the environment. The agent can do many different things like select and change the base image, manage and install dependencies, utilize bash commands, edit files, and execute test suites. Once the agent successfully runs all tests, there is a Dockerfile synthesizer component which converts the command sequence from the building process into a runnable Dockerfile. For evaluation, the authors build a benchmark consisting of 420 Python repositories from GitHub (2024).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors tackle a very impactful problem — scaling up executable environments for training and evaluating code LLMs is critical for the research community, and the authors have proposed a very novel and clever approach for doing this in an automated manner.\n- The findings related to the dual-architecture and rollback mechanism are quite interesting and may be relevant to other tasks for LLM agents. \n- The authors have done an excellent job with their analysis. They have extensively covered ablation results, error patterns, performance across different domains, and latency.\n\nWeaknesses:\n- The authors have chosen to focus on Python only. It is not clear whether the results will be as high for other languages, as building environments for other languages may be more difficult.\n\nQUESTIONS:\n- The underlying LLM for Repo2Run is gpt-4o-2024-05-13. Is the same underlying model used for the \"LLM generator\" and \"SWE-agent\" baselines? If it is not, then it seems that much of the difference in performance could be explained due to differences in the base model.\n- Due to the rollback mechanism, it seems like the LLM is not given enough information to do self-repair. As a result, do you observe the same erroneous commands being repeatedly generated in subsequent steps, following a rollback?\n\n\nSuggestions:\n\n- L2: engineerin -> engineering\n- The ablation without rollback is not clear. In Table 3, it seems that the only difference between the top 2 rows is the removal of rollback in row #2. In that case, it seems that the performance is actually better without rollback as the numbers in row #2 are higher than row #1. However, I assume the \"w/o rollback\" includes everything in the full Repo2Run setting including the dual-environment, except for rollback. \n- L213-217: Is rollback enabled for the LLM generator? This is a bit confusing.\n\nLIMITATIONS:\nThis work only considers Python repositories. There are no empirical results to suggest that this will generalize to other programming languages.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenging and practical problem of automating environment configuration for code repositories. It proposes **Repo2Run**, an LLM-driven agent that uses a dual-environment design and rollback mechanism to generate Dockerfiles and build reproducible environments at scale. The authors evaluate their approach on a benchmark of Python repositories, demonstrating strong performance when compared to baselines like SWE‑agent and pipreqs, and report high success in both environment builds (86%) and Dockerfile generation (100%).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- **Clear motivation & practical significance**: Automating environment setup is a longstanding pain point in software development; framing it as an LLM-based agent problem is timely and relevant to the ML+systems community.\n- **Innovative agent architecture**: The dual-environment sandbox plus rollback strategy is a clever solution to mutable build environments and dependency conflicts.\n- **Rigorous empirical evaluation**: Experiments span real-world Python repos, include strong baselines, and feature ablation studies—this lends credible support to the core claims.\n\n## Weaknesses\n\n1. **Limited LLM comparisons**: Experiments only use GPT‑4o. It remains unclear whether the method generalizes to other LLMs (e.g., Gemini-2.5, Claude-3.7). Additionally, the LLM‑generator baseline could benefit from swapping in different reasoning models.\n2. **Lack of cost/resource analysis**: Although build times (~29 min) are reported, there is no discussion of financial cost.\n3. **Reliance on unit tests**: The system depends heavily on existing unit tests for feedback. Many real-world repos lack tests, limiting applicability. No fallback mechanisms are explored.\n\nQUESTIONS:\nPlease respond to the above weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Repo2Run, an LLM-based agent designed to automate the construction of executable test environments for code repositories and generate runnable Dockerfiles. The approach adopts a dual-environment architecture (internal and external environments) and a Dockerfile synthesizer, and leverages operations such as rollback mechanisms, dependency installation, test running, and code editing. The authors claim that Repo2Run achieves an 86.0% success rate and generates 100% buildable Dockerfiles through evaluation on 420 Python repositories, significantly outperforming existing baselines such as SWE-agent and pipreqs.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper defines the EBSR as \"only caring whether the tests can be executed, regardless of whether they pass or fail\". This is an extremely loose success criterion. An \"executable environment\" that fails all tests due to configuration errors (such as missing environment variables, dependency version conflicts, test data files not found, etc.) cannot be considered a successful build in any practical application scenario. This definition seriously exaggerates the true capabilities of Repo2Run. \n\nThe core contribution of this research is over-packaged. The main components of Repo2Run, such as LLM-based agent, Docker container, rollback mechanism, and Dockerfile generator, are not new technical concepts. It combines the application of existing technologies.\n\nThe paper admits that its experiments \"have not repeated experiments many times to get error bars\" because they were time-consuming and expensive, which makes it impossible to judge the stability and contingency of the experimental results, weakening the scientific rigor of the conclusions.\n\nThe paper claims that \" for complex issues, Repo2Run shows greater advantages over manual building\", but does not provide any quantitative data to support this argument. \n\nThe experiments rely entirely on the closed-source model gpt-4o-2024-05-13 and are only tested on Python repositories, limiting the generalizability of their conclusions.\n\nQUESTIONS:\nCan the authors provide more detailed arguments for the innovation of the “rollback mechanism” and “dual environment architecture” compared to Docker’s standard practices and the general agent-host control model? Renaming existing technologies does not constitute a true original contribution.\n\nThe EBSR metric ignores the actual success or failure of test cases, which makes the definition of “successful build” too loose. Why not use a stricter metric, such as requiring at least a certain percentage of test cases to pass? Among the 361 “successful” build cases, how many cases actually have the majority of tests failing?\n\nCan the authors elaborate on the details of the prompt engineering done when adapting SWE-agent to new tasks? Is it possible that the poor performance of the baseline is caused by poor prompts? \n\nWhat is ECSR in Tables 1 and 3? Is it EBSR?\n\nLIMITATIONS:\nNo discussion of the generalizability of this method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new coding agent framework, Repo2Run, which aims to build executable test environments (e.g., Docker + unit tests) for large-scale code repositories.\n\nSpecifically, it involves two phases: the build phase initializes a docker container and runs built-in tests with iterative agent-environment interactions. This phase involves both external and internal environments. The record phase aims to build a dockerfile based on the commands.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Automatic executable environment construction is an important task.\n2. The paper presents a pretty decent formulation of the environment construction task, where the goal is to generate a dockerfile for a given repo, and the main evaluation metric is the test success rate.\n\nWeakness:\n1. The dockerfile is directly generated by an LLM and would potentially contain malicious code (e.g., commands that take up too much disk space and break the system).  However, when evaluating the dockerfile, it will be directly executed without further examination. \n2. The evaluation is mainly on the SWE-Bench dataset with newly introduced metrics. While I agree that both the evaluation setting and the metrics are reasonable, the authors neglect a more relevant benchmark called SUPER: https://arxiv.org/abs/2409.07440. This paper presents the SUPER benchmark that directly evaluates the capability of LLMs in setting up and executing tasks from research repositories, which is more relevant than SWE-Bench.\n3. I'm not fully convinced that the authors are comparing to the strongest baselines. For instance, the authors mention OpenHands in the introduction section, but never compare to it, without any further explanations. Since OpenHands is also integrated with a set of execution tools, I'm interested in its performance on environment setup.\n\nQUESTIONS:\nSee \"weakness\" for details.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}