RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing
Authors: Yiqing Xie
Alex Xie
Divyanshu Sheth
Pengfei Liu
Daniel Fried
Carolyn Rose
Venue: COLM 2025 (2025)
Keywords: code generation training
repo-level code generation
TL;DR: Automatically constructing repo-level execution-based environments for training and evaluation.

Abstract: We present RepoST, a scalable method to construct environments that provide execution feedback for repository-level code generation for both training and evaluation. Unlike existing works that aim to build entire repositories for execution, which is challenging for both human and LLMs, we provide execution feedback with sandbox testing, which isolates a given target function and its dependencies to a separate script for testing. Sandbox testing reduces the complexity of external dependencies and enables constructing environments at a large scale. We use our method to construct RepoST-Train, a large-scale train set with 7,415 functions from 832 repositories. Training with the execution feedback provided by RepoST-Train leads to a performance gain of 5.5% Pass@1 on HumanEval and 3.5% Pass@1 on RepoEval. We also build an evaluation dataset, RepoST-Eval, and benchmark 12 code generation models. Code and datasets available at https://repost-code-gen.github.io/

Published as a conference paper at COLM 2025
REPOST: Scalable Repository-Level
Coding Environment Construction with Sandbox Testing
Yiqing Xie
Alex Xie
Divyanshu Sheth
Pengfei Liu? Daniel Fried
Carolyn Rosé
*Carnegie Mellon University
2Shanghai Jiao Tong University
yiqingxi@andrew. www.drewwc........ edu
Abstract
We present REPOST, a scalable method to construct repository-level code generation environments that provide execution feedback, which can be used for both training and evaluation. Unlike existing works that require building the entire repository for execution, which is challenging for both human and LLMs, we provide execution feedback with sandbox testing, which isolates the target function and its dependencies to a separate script for testing. Sandbox testing reduces the complexity of external depen- dencies and enables constructing environments at a large scale. We use method to construct REPOST-TRAIN, large-scale train set with 7,415 functions from 824 repositories. Training with the execution feedback pro- vided by REPOST-TRAIN leads to performance gain of 5.5% Pass@1 on HumanEval and 3.5% Pass@1 on RepoEval. We also build an evaluation dataset, REPOST-EVAL, to showcase the potential of REPOST for live bench- mark construction."
1 Introduction
Code generation a special NLP task that can ben- efit from execution-based feedback (Simon, 1963; Feng et al., 2020; Chen et al., 2021). With online cod- ing platforms scale coding datasets with test cases, existing works demonstrate the effectiveness of execution-based signals in training data construction (Ni et al., 2024; Zhang al., 2024; Liu et al., 2023a) and live evalua- tion (Jain et al., 2024a), which is updated over time to prevent contamination. However, models solely trained on algorithm problems cannot well gener- alize to repository-level (repo-level) code genera- tion (Zhang et al., 2023; Jimenez et al., 2024), which aims to generate code for real, naturally occurring repositories and is well aligned with real-world software engineering practice.
Repo-level Code-Gen
(Training)
def ..()s # TED
natural resource to build large-
LLM
if Passed
Training Data
Solution
Eval Script
(Evaluation)
Passed / Failed
Pass@k
Figure 1: We can use the coding envi- ronments built by REPOST for train- ing and evaluation. We first apply the code generation model to gener- ate candidate solutions with the orig- inal repository as context. Then evaluate the solutions by executing the evaluation script built by REPOST. For evaluation, we directly compute Pass@k scores. For training, we add all successful solutions to the train set and further finetune the model.
It is non-trivial to build execution-based datasets for repo-level code generation at large scale. One major challenge is to set up executable environ- ments. Existing repo-level datasets typically con- duct integration testing that executes test files inside the repositories, which requires building the entire repository (Jimenez et al., 2024; Jain al., 2024b). As shown in prior research, this setup process is ex- tremely challenging (Bogin et al., 2024), which may include installing external packages, determining execution context (e.g., relative import),
1 Code and datasets available https: http://///pssooeeeeeeeeee www.eeeee.gittub..o../o///

---

Published as a conference paper at COLM 2025
debugging installation errois runtime errors, etc. Hence, existing methods for coding environment construction either require huge manual effort (Zhang et al., 2023; Jimenez et al., 2024; Pan et al., 2024) or, when automated, suffer from low success rate (Jain et al., 2024b), limiting the scale of resulting datasets (see Table for details).
In this work, instead of integration testing, we present REPOST, automated framework to construct Repo-level coding environments with Sandbox Testing. Specifically, given function in a GitHub repository, we sandbox the function and its local dependencies separate script and generate tests with an LLM. To control the quality of the evaluation script, we iteratively resolve environment or runtime errors and improve test coverage. We also conduct execution-based, AST-based, and LLM-based quality checks and only keep examples where the functionality of the sandboxed function does not alter and the tests are valid, reasonable, and have high coverage. The resulting dataset allows us to obtain execution feedback in training and evaluation. As shown in Figure 1, the models generate the target function with the entire GitHub repository as context. We then use the evaluation script to obtain execution feedback.
Compared to integration testing used by previous datasets, we highlight the benefits of sandbox testing in constructing scalable coding environments: (1) In general, the external dependencies of function are typically much simpler than repository. By isolating the function and its local dependencies, we can execute the function by only installing the necessary packages. (2) If any execution error occurs, we can debug the separate scripts without modifying the original repository. This ensures that the datasets remain naturalistic: code generation models can still access the original, real-world repository.
With the high scalability of our framework, we construct coding environment datasets for both training and evaluation: REPOST-TRAIN and REPOST-EVAL. As shown in Table 1, to our knowledge, REPOST-TRAIN is currently the largest repo-level code generation dataset with execution support, with 7,415 functions sampled from 824 repositories. The large scale enables training on REPOST-TRAIN and evaluating on other benchmarks such as RepoEval (Zhang et al., 2023) or HumanEval (Chen et al., 2021). Due its fully automated nature, REPOST also enables constructing live benchmarks to avoid contamination issues. We build REPOST-EVAL as an example, which contains 296 functions from 99 repositories.
Experiments show that by training on REPOST-TRAIN, we achieve performance gain on both algorithm problems in HumanEval (Chen et al., 2021) and repo-level tasks in RepoE- val (Zhang et al., 2023) and REPOST-EVAL. For instance, we improve Qwen2.5-Coder by 5.5% Pass@1 on HumanEval and 3.5% Pass@1 on RepoEval. We also benchmark 12 code generation models REPOST-EVAL. The best model (GPT-40) only achieves 39.5 Pass@1, suggesting REPOST can construct challenging code generation benchmarks.
Note that REPOST-TRAIN provides coding tasks based on real-world repositories that have execution feedback, we would like to highlight that REPOST can also be applied to coding agent (Yang et al., 2024; Wang et al., 2025) training and evaluation in future works.
2 The REPOST Framework
The framework of REPOST is illustrated in Figure 2. After sampling GitHub repositories and extracting functions ($2.1), we sandbox each function and its local dependencies to separate script ($2.2), generate tests (62.3), and conduct quality control on the executability of the scripts, the functionality of the sandboxed function, and the test quality (52.4). We provide all the prompts in SA.1. When we use our environments to train (54) and evaluate ($5) code generation models, the models can still access the original GitHub repository, and we provide execution feedback by executing the evaluation scripts. To create user-friende datasets, we keep shared Docker environment for all the evaluation scripts.
2.1 Repository and Function Curation
We first randomly sample non-forked, MIT-licensed, Python GitHub Repositories with file sizes smaller than 10M. With our sandbox testing method, we are able to set up environments
2

---

Published as a conference paper at COLM 2025
Repo Function Curation
2
Sandboxing
Test Generation
implement fsperi Fundion in wro/ ptyp
Eval Script
test farmard
Repo
src/Artent-ar-py
srcleark.py
docs
Lervard(self, ..ds Sgeryy key, value for 4
...
Test
Giit Repe
Eval Scripl
stse aost the
Correctuess Check
py
Functionaliity Equivalence Chock
Outcome:
wer atte senuttmon
Execution Debug. Coverage
Coding Environments
in py
I11 all time prgintiin
for to
Quality Control Improvement
@
Kiltering
( Can be used for
Train Evall)
Figure 2: The REPOST coding environment construction framework. We sandbox the target function and its dependencies to a separate evaluation script for execution, which avoids building the entire repository. We design careful quality control strategies with iterative quality improvement and post-filtering. The outcome of REPOST is a set of executable repo-level coding environments, which can be used for training and evaluation.
for individual functions and do not need to build the entire repository. Hence, unlike previous works (Jain et al., 2024b), we do not need to filter out repositories without setup files (e.g, setup.py) The detailed repository statistics are provided in $3.
Then we extract functions from the repositories. To build the datasets in docker with no access to GPUs or external services, we follow R2E (Jain et al., 2024b) and filter out functions associated with GPUs, cloud tasks, etc. by keywords. To balance the distribution of examples, we keep at most 30 functions for each repository. After this step, for the train set, we started from 1,000 repositories and obtained 17,448 functions from 851 repositories. For the evaluation set, we started from 200 repositories and obtained 1,043 functions from 139 repositories. In theory, we can further scale up the dataset with more starting repositories.
2.2 Sandboxing: Key to Environment Setup
is non-trivial to create coding environments that provide correct execution feedback for the
Orig Function
def API (pronpt str) -> str:
implementation of GitHub functions. Existing methods typically provide execution feedback with integration testing, which creates test files that import the target function from the code- base. Executing such tests requires building the entire repository, which generally requires complicated environment and is challenging for both human and LLMs.
return API client.generatee model=API_MODELL nessages=[...)
Eval Script
class (Fbbjett:)
def generate (model, messages):
return "nock_" nessages (0] ['content")
APT MODEL = "nodel nane" API [PI_client def API iPI_calt(prompe) str) -> str:
With the observation that the external depen- dencies of a function are typically much simpler than repository, we tackle this challenge with sandbox testing, where we create a separate script containing the target function with the ex- act same functionality as the original one and the context that supports its executability. By isolating the function and its local dependencies, can execute the function by only installing the necessary packages.
return API client.generate L=API_ nessages=(...)
Figure 3: An example where the LLM suc- cessfully creates mock class, Mock_APT to replace real external API calls. In this way, while the function body of the tar- get function API-call remains exactly the same as in the original codebase, it can be executed without making real API calls.
Sandboxing based on Local Dependencies. The major challenge of sandboxing is to ensure the executability of the function. While a standalone function can be directly copied to separate script and executed, a typical GitHub function may depend on other modules, classes, or global variables in the same repository. Thus, leverage the call graph to extract such
3

---

Published as a conference paper at COLM 2025
local dependencies that the target function directly indirectly calls. Then we combine all the code fragments into the context and prompt an LLM (e.g,, GPT-4o) to aggregate all the code fragments into single script, with as little editing possible.
Sandboxing External APIs and Files. Even if all the dependencies are presented, it is still nontrivial to execute the sandboxed script if it requires external API, databases, files, etc. We explicitly prompt the LLM to create mock connections for any external API and create strings or write example files to a specific directory for file reading. Figure 3 presents successful case of sandboxing with mock APIs. We provide examples of the generated sandboxed scripts in Figures 5 to 10.
Functionality Equivalence Control. In addition to executability, another challenge of sandboxing is to ensure that the target function's functionality does not alter. We first conduct list of sanity checks on the function name, length of scripts, etc. (see Table 11 for details), and re- re-generate examples that do not satisfy the requirements. We also have final quality control step (in $2.4) that compares the functionality of the sandboxed and original functions. The precision of quality control is further verified in our human study in $3.1.
2.3 Test Generation
Equivalence Testing. We generate synthetic tests in the sandboxed script to evaluate the correctness of the generated code. Specifically, we prompt the LLM (1) generate a set of test inputs to the target function and (2) conduct equivalence testing that checks whether the function generated in evaluation has the same behavior the sandboxed function (i.e., the "'ground-ruun "utttuhh""" implementation) Compared to traditional methods that specify the I/O examples in the test cases (Chen et al., 2021), equivalence lesting does not need to predict the expected outputs and is more feasible for LLMs (Jain et al., 2024b).
Test Generation with Mock APIs and Files. We observe that the LLM is able to generate tests with the mock classes created in the sandboxing step as context. As shown in Figures to 10, we create mock class instances as the test inputs and still ensure that the function body of the sandboxed function remains the same the original function.
Test Quality Control. Similarly to the sandboxing step, for quality control purposes, conduct a series of sanity checks, such as requiring the test function to call the target function and have at least 3 assertions (see Table 12 for the full list). We further check the coverage and correctness of the tests in the final quality control step (62.4),
2.4 Quality Control and Filtering
We conduct quality control for the executability of the evaluation script, the functionality of the sandboxed function, and the test quality.
Iterative Execution and Debugging. In principle, the model-generated function is exactly the same the ground truth, the evaluation scripts should be successfully executed, with all the tests passed. Hence, we execute the examples sequentially in a docker. If there are any execution errors or if the ground truth implementation cannot pass any test cases, we provide the error message as the context and prompt an LLM to debug the evaluation script. Examples that still have errors after execution-debugging iterations filtered out.
We also dynamically install external packages during execution. During execution, there are any ModuleNotFol pip install command, and execute the script again. In case the package name differs from the import name or the code requires a specific version of packages, we also allow the LLM to output package installation commands during debugging. In this way, we are able to install external packages for functions extracted from repositories without setup files.
errors, we extract the package names from the error message, run
Iterative Test Coverage Improvement. To ensure that the test functions cover the major functionality of the target functions, we further compute the branch coverage rate all the evaluation scripts. If the branch coverage rate is lower than some threshold (we set 80% for
4

---

Published as a conference paper at COLM 2025
Dataset
#Ex
#Repo
Repo?
AutoTest?
Check
Func
Test
HumanEval
164
X
X
Human
Yes
No
Yes
No
DS1000
1,000
X
ElassEval
100
K
GPT-4o: Yes
13
0
1
RepoEval-Fumm
455
GPT--4:: No
1
6
6
SWE-Bench
2,294
CoderBval
230
X
Table 2: Agreement between human and
DevEval
1,874
117
X
GPT-4o on checking (1) the functionality equivalence between the sandboxed and original function, and (2) the test correct- ness. When GPT-4o predicts "Yes" for
EvoCodeBench
275
25
X
SWE-Gym
2,438
X
R2E-Eval1
246
137
R2E (Ours)
744
123
V
REPOST-TRAIN
7,415
824
both quality checks, it has high agree- ment with human.
REPOST-EVAL
296
Table 1: Statistics of REPOST-TRAIN and REPOST-EVAL compared to existing execution- based code generation datasets. R2E (Our Input) applies the R2E method to the same set of in- put repositories as REPOST-TRAID but results in smaller number of repositories and examples. "Repo?" and " "Auto Test?" refer to the repo-level setting and automatically generated tests.
Solved 81.5%
% Use-Tool
Easy / Med / Hard 29.6 51.9 / 18.5
59.3%
Table 3: Human study results. We ask the participants to complete the function with the same context we use to evaluate code generation models in $5.
the train set and 100% for the evaluation set), we provide the LLM with the missing lines and prompt it improve the test function by incorporating more tests.
Final-Step Quality Check & Filtering. As final step, we conduct two quality checks: func- tionality equivalence check and test correctness check, and filter out unqualified examples.
To ensure the validity of sandbox testing, examine the functionality equivalence be- tween the sandboxed and original function. We first compare the AST of the function bodies, which is sufficient condition of functionality equivalence. In our resulting dataset, 81.7% of the examples have the same AST. The remaining ones are filtered out from the evaluation set. To include more examples in the train set, since it is also possible that code with the same functionality has different ASTs (e.g,, HTML tags can be parsed with LexborHTMLParser and Beauti f ful we prompt an LLM 2 to compare the functionality equivalence, which is shown to have high agreement with human (see our human study in $3.1 for details). We also include the examples that pass the LLM check for the train set.
In principle, a test that calls the target function without any assertion checks still has a 100% coverage rate. We hence conduct test correctness check and apply LLM to check whether the tests are correct, reasonable, and are completing the verification of the functionality. Human study in $3.1 demonstrates the high precision of the LLM test correctness checker.
Resulting Datasets: REPOST-TRAIN and REPOST-EVAL.
Statistics. We build train set, REPOST-TRAIN, and an evaluation set, REPOST-EVAL. To re- duce contamination, we build REPOST-TRAIN from repositories created between 2023-01-31 and 2024-08-31, and build REPOST-EVAL from repositories created after 2024-09-01.
We compare the statistics of our datasets and existing execution- datasets in Table 1. To our knowledge, REPOST-TRAI is currently the largest repo-level dataset with execution feedback. Prior works such as RepoEval (Zhang et al., 2023) or SWE-Gym (Pan et al., 2024) require huge human effort to set up the environments. Automated frameworks such as R2E (Jain et al., 2024b) apply LLMs to the complicated task of building the entire repositories and suffer from low success rates. In comparison, REPOST only requires
2The original code could be inexecutable due to package installation errors (e.g., Beauti fulSoup in this case), so we do not execute the original code.

---

Published as a conference paper at COLM 2025
necessary dependencies for each function, which is more feasible for LLMs and benefits scalability.
Table 4 shows detailed statistics of our datasets. We achieve high test num- bers and branch coverage rates with it- erative test coverage improvement and quality filtering. Results also show that REPOST can create relatively complex examples in terms of length and local and external dependencies. The per- centage of standalone functions (i.e., functions without local dependencies) are 28. 1 % and 26.4%, our datasets. Both are very close to 27%, the percentage of standalone functions among all GitHub code estimated by Li et al. (2024b).
Avg Stats.
TRAIN
EVAL
Target # Tokens (Lines)
112.4 (12.8)
102.7 (9.9) 1217.5 (122.3)
Eval Script @ Tokens (Lines)
842.5 (75.7)
lest Cases
5.7
Test Branch Coverage
97.8%
100%
% Standalone Functions
28.1%
26.4%
# External Libraries
894
106
Table 4: Detailed statistics of our datasets.
3.1 Quality Verification with Human Study
Agreement between LLM Checker and Human. We conduct human study to verify the precision of our LLM-based quality check strategies introduced in $2.4. We ask 3 computer science graduate students to conduct functionality equivalence and test correctness checks for 20 examples sampled from REPOST-EVAL (before the final filtering), with the same instruction we use prompt the LLM checkers. The Kappa agreement scores among human annotators are 0.9179 for the functionality check and 0.7750 for the test check.
Results in Table show that all 13/20 examples that pass GPT-4o's functionality check are also predicted as "same functionality" by human. Among 10 examples that pass GPT-4o's test quality check, 9 of them are predicted as "high-quality tests" by human. It demonstrates that after applying our filtering strategies for quality control, the remaining examples have high quality. In principle, one can further enhance dataset quality by manually inspecting and selecting the examples. We provide more details about the experiments in SA.2.
Solvability Check. We conduct another human study to check whether the examples are reasonable and can be solved by human. We assigned 27 examples constructed by REPOST to 9 computer science students, with no overlaps, and asked them to complete the function and answer questions about the difficulties of the examples (see SA.2 for details).
Results show that 81.5% of the examples were solved by human, indicating that most examples are reasonable and not too complicated. The remaining examples were not solved for various reasons, such as the participant is not familiar with the task (e.g., reading html data) or related libraries (e.g., Beauti Soup4) the intent of the function cannot be fully entailed from the context, etc. Towards the unclear intent issue, we designed an experiment in $5, where we generated additional instructions to improve the clarity. We also observe that the generated examples have varying complexity levels based the usage of external tools and the difficulty ratings. Furthermore, 33.3% of the examples are solved in the first submission and 37. 00% require more than 5 submissions to solve.
Code Generation Training Experiments
Training with REPOST-TRAIN, In standard supervised fine-tuning (SFT), we can train the model with the code context c as the input and the ground truth target function f as the output. The execution feedback provided by our REPOST evaluation scripts further allows us to employ the rejection sampling fine-tuning (RFT) algorithm to generate additional valid training targets. Specifically, we apply the model itself to our dataset, generating n candidate solutions for each function based on its code context: (c, (c, The method is denoted RFT (Self). We can also apply other stronger models to generate candidates (denoted as RFT (DistilD).. Only solutions that pass our test cases are retained.
We then finetune the model on the successful functions ff....... (c, and the ground truth (c) using the standard negative log- ligelihod loss.

---

Published as a conference paper at COLM 2025
HvaanEval
RepoEval-Func
REPOST-EVAL
Model
Pass@1
Pass@1
A
Pass@1
StarCoder2-7B (Lozhkov et al., 2024)
34.76
32.98
26.35
-STT
37.20
12.44
33.78
10.80
27.70
*1.35
RFT (Self)
39.63
14.87
34.58
11.61
28.38
*2.03
RFT (Distill)
40.24
15.49
35.12
12.14
29.05
22.70
Qwen2.5-Coder-77 (Hui et al., 2024)
79.27
38.01
29.39
+ SFT
80.48
f1.21
39.94
11.88
30.74
11.35
+ RFT (Self)
84.76
15.49
40.75
12.69
31.76
22.36
+ RFT (Distill)
84.76
15.49
41.55
13.49
32.43
"3.04
Table 5: Code generation training results. We evaluate Pass@1 for all experiments. For RepoEval, we use the "Oracle" repo-level context as used in their original paper.
To obtain more training pairs for both the "Self" and "Distill" settings, we further prompt the model itself or stronger model to debug the failed solutions, with the error message in the context. We also train the model on successfully debugged solutions (c, (c, (h)
4.1 Experimental Setup
Datasets and Evaluation Metrics. We train two models: StarCoder2-7B (Lozhkov et al., 2024) and Qwen2.5-Coders (Hui et al., 2024) with REPOST-TRAIN and evaluate on two public benchmarks: HumanEval (Chen et al., 2021), an algorithm problem dataset, and RepoEval (Zhang et al., 2023), repo-level code generation dataset. We also evaluate REPOST-EVAL. For RepoEval, we only evaluate on the "function" split, which supports execution. We use the "Oracle" context to mitigate the bias of context retrieval methods. We report the P scores on all datasets. More evaluation details are provided in $A.3.
Training Details. We compare three training methods: SFT, RFT (Self), and RFT (Distill). For the "Distill" method, we apply GPT-4o and Claude-3.5-Sonnel to generate and debug candidate solutions, separately, and combine their successful candidates for training. We provide the number of examples where we obtained at least one successful solution in Table 9. Other details are shown in subsection A.3.
4.2 Experimental Results
Main Results. Table 5 demonstrates that models trained with REPOST-TRAIN can generalize well to other public benchmarks. Specifically, we improve Qwen2.5-Coder by 5.5% Pass@1 on HumanEval, 3.5% RepoEval-Func, and 3.0% on REPOST-EVAL. Furthermore, in all experiments, training with RFT, even with self-training only, achieves better performance than finetuning the original GitHub function only. For instance, RFT (Distill) outperforms SFT by 4.3% Pass@1 1 on HumanEval. This shows the benefit of training with environments that provide execution feedback. We also observe that RFT with self-training in general has lower performance than distilling from stronger models. As shown in Table 9, we can only obtain 1573 additional training targets with StarCoder2, but obtained 3606 by combining GPT-4o and Claude-3.5, We hypothesize that one can obtain more examples and hence achieve better performance by sampling with more candidate solutions, generating from different types of context, etc., and we leave that to future work.
Scaling Law Analysis. In Figure 4a, we investigate how the scale of training data affects model performance. Specifically, from REPOST-TRAIN to train the model. We can see that the performance of RFT (Distill) increases as scale up the training data, which suggests the advantage of training data with high scalability. Furthermore, with different scales of data, RFT consistently achieves better performance than SFT, which further demonstrates the effectiveness of training environments with execution feedback.
randomly sample different numbers of examples
7

---

Published as a conference paper at COLM 2025
Model
Pass@1
Repair Iter
2
Codel.lama-77 (Rozière al, 2024)
25.68
GPI-4o-mini
35.81
43.58
17.77
46.62
13.04
47.97
11.35
StarCoder2-7B (Lozhkov et al., 2024)
26.35
GPT-4o
48.31
18.78
52.36
14.05
Qwee2.5-Coder77 Coder-BB (Hui ct al., 2024)
29.39
MagiCoder---DS66.. (Wei et al., 2023)
33.78
Table 7: Performance on REPOST-EVAL. with
CodeLJama-34B (Rozière al., 2024)
32.43
self-repairing. We show the Pass@1 gain com-
Owen2.5-Coder-322 (Hui al., 2024)
33.11
pared to the previous repairing iteration.
Rl-----Qen-32 (DeepSeek-AA, 2025)
34.46
CodeLlama-70B (Rozière et al., 2024)
32,43
DS- -0-Ra---lmm--70B (DeepSeek-Al, 2025)
33.45
-PT--o-mini (OpenAl, 2024)
35,81
Docstring (DocS)
w/o Dod5
w/ Orig DocS
Gen-DocS
Claude-3.5-Sonal (Anthropic, 2024)
37.16
GPT-4o-mini GPT-4o
37.28
GPT-4o (OpenA1, 2023)
39.53
39.55
42.37
61.02
Table 6: Code generation results on REPOST-EVAL. The model model/open-source model with the best performance is high- lighted in bold/ underlined.
Table Performance on the subset of REPOST- EVAL examples. The performances of both models are largely improved with model- generated docstrings.
Repository Diversity Experiment. Fig- ure 4b examines whether broader reposi- tory coverage leads to better performance, given a fixed budget of training exam- ples. We fix the number of examples to 2,000 and experiment with two exam- ple sampling methods: (1) Sample-by- Repo, where we keep sampling reposito- ries and adding all the functions in the repository to the training set, until the data size reaches 2,000; (2) Sample-by- Example, which is the same setting as Figure 4a, where we randomly sample functions from REPOST-TRAIN. We ob- serve that Sample-by-Example, covering 678 repositories, outperforms Sample-by- Repo, which only covers 221. The perfor- mance is further improved by RFT. Recall that our method only needs to set up in- dividual functions, compared to existing methods, such as R2E, that need to build the entire repository, it is much easier to build datasets with broad repository coverage with our method, which benefits model training.
Scaling Law Analysis
Repo Diversity Analysis
42
41.55
42
By-Ropo
Rej Sampling
By-Example
41
SFT
41
Rej Sampl.
39.94
39.94
40.48
40
39.
39.14
39
39
38.33
38.87
38.06
38
0 1k
4k
All (7k)
Data Sampling Method
(a) Scaling law analysis.
(b) Repo diversity.
Figure (a) Pass@1 scores on RepoEval with different numbers of training examples. (b) Pass@1 scores on RepoEval with different meth- ods to sample 2,000 training examples. Sample- by-Example has a broader repository coverage and achieves better Pass@1. The performance is
further enhanced with RFT (Distill).
5 Benchmarking with REPOST-EVAL
Experimental Setup. We benchmark LLMs on REPOST-EVAL to evaluate their abilities to generate code in real GitHub repositories. Specifically, we prompt the LLMs to generate the target functions based on code context retrieved from the repository. To ensure that we cover all the relevant dependencies in the context, we follow R2E (Jain et al., 2024b) and build the context by extracting modules that the target function depends on. Then we copy model solutions to the evaluation scripts for execution and compute the Pass@1 scores (Chen al., 2021): the fraction of generated solutions that pass all test cases.
Main Results. Table 6 presents the code generation results of open-source and proprietary models. The best model (GPT-4o) only achieves 39.53 Pass@1, which shows that our benchmark is challenging and has large room for improvement. There is still gap of 5.07 Pass@1 between the best open-source which calls for future work to improve open-source models by training.
and proprietary model (GPT-4o),
8

---

Published as a conference paper at COLM 2025
Self- belf-Repairing Experiments. In addition to static code generation, we also study the effect of self-repairing (Chen et al., 2024), which allows the models to iteratively repair solutions based on the error message and stack trace and has been shown to be effective for code generation. As shown in Table 7, both models consistently benefit from self-repairing, but the performance gain becomes saturated after 3 debugging iterations. We can also see that the performance gap between the two models becomes larger when we allow more iterations, indicating that GPTT-oo is stronger in both code generation and debugging.
Generation with Model-Generated Docstrings. In our human study (53.1), observe that in some examples, the intent of the target function cannot be fully inferred from the context. The same issue is reported in existing repo-level benchmarks such as R2E (Jain et al., 2024b). In this experiment, we investigate whether model-generated docstrings can provide better evaluate specifications. On the subset of REPOST-EVAL examples that have docstrings, code generation (1) with the docstring removed, (2) with the original docstring, and (3) with GPT--o-genee -----rreeed docstrings, with the function and its dependencies as the context.
We observe that both models achieve much higher performance when provided with gener- ated docstrings. One possible explanation that the quality of human-written docstrings varies widely. Some docstrings may only contain limited information, while the gener- ated docstrings generally contain more details. On average, the original docstrings have 36.82/3.99 tokens/ling /lines, while the UPPT-oo-gneertee docstrings have 130.84/9.99 Further- more, with more detailed docstrings, the performance gap between the two models becomes larger. When the specifications are not clear, it could be impossible to complete the function as intended, causing both weak and strong models to fail.
Related Work
Code Generation Training. Existing works have shown the effectiveness of pretrain- ing (Rozière et al., 2024; Lozhkov et al., 2024; Guo et al., 2024) or instruction tuning (Wei et al., 2023; Luo et al., 2023; DeepSeek-Al, AI, 2025) on real-world code. To further finetune models for code generation, existing works have built large-scale training datasets with test cases by leveraging large-scale online algorithm problems. The execution feedback from test cases is used in constructing training targets (Ni et al., 2024; Zhang et al., 2024) or reward signals (Liu et al., 2023a; Jiang et al., 2024). As repo-level code generation, existing works such SWE-Gym (Pan et al., 2024) build training environments by manually setting up dependencies and configurations for the entire GitHub repositories. The repository setup process is complicated and challenging automate, resulting in limited dataset scales. In comparison, we design a sandbox testing method that only requires setting up the necessary dependencies for individual GitHub functions, which reduces the difficulty of environment setup and leads to better scalability.
Code Generation Benchmarks. Execution-based benchmarks have been widely adopted for code generation, which provide test cases to evaluate the generated code (Chen al., 2021; Hendrycks et al., 2021; Austin et al., 2021). Researchers have built benchmarks with large test coverage (Liu et al., 2023b), multiple languages (Cassano et al., 2022), diverse domains (Lai et al., 2023; Du et al., 2023), etc. LiveCodeBench (Jain et al., 2024a) periodically extracts newly released algorithm problems, which enables contamination-fae free evaluation.
To assess the models' ability on repo-level coding tasks, recent works leverage repositories with test cases to build benchmarks on code patch generation (Zhang et al., 2023; Xie et al., 2024), issue solving (Jimenez et al., 2024), test generation (Jain et al., 2025), test execution (Bouzenia & Pradel, 2024), environment setup (Bogin et al., 2024), etc. Recent works aim to provide live evaluation manually (Li et al., 2024a) or leverage an LLM to set up the repository environment and generate test cases. However, both methods require building the entire repository, which is challenging for both human and LLMs and limit the scale of the resulting datasets. REPOST improves the scalability with sandbox testing and enables the construction of live benchmarks from naturally occurring repositories.
9

---

Published as a conference paper at COLM 2025
7 Conclusion and Future Works
We present REPOST, a scalable method to construct environments for code generation in real-world repositories that support sandbox testing. REPOST is fully automatic and enables the construction of scalable execution-based training environments and live benchmarks. Experiments demonstrate that training with the resulting train set, REPOST-TRAIN, leads to performance gain on other public benchmarks. For instance, we improve Qwen2.5Coder by 5.49% Pass@1 on HumanEvall /eeoovll We also build an evaluation set, REPOST- EVAL, to showcase the potential of REPOST to construct live benchmarks.
Future works may include: (1) further scaling up the datasets with more input repositories, (2) exploring the utility of different types of context in training and evaluation, (3) adapting REPOST to other repo-level coding tasks such as issue-solving (Pan et al., 2024), code translation (Xie et al., 2023), code refactoring (Gautam et al., 2025), environment setup (Bogin et al., 2024), etc., and (4) using REPOST-TRAIN to train and evaluate coding agents (Yang et al., 2024; Wang et al., 2025). This is possible because to the original GitHub repositories and execution feedback. Specifically, one can set the instruction as "generate the target function", and the coding agent will need to explore and interact with the entire repository by itself to obtain relevant information. We can then use the evaluation scripts to select successful trajectories and use them for model training.
datasets provide both access
Reproducibility Statement
We provide the following ways reproduce our results: (1) We release the code for the entire REPOST pipeline, including repository and function curation, sandboxing, test generation, execution, and final-stage quality verification. (2) We release the code for training data construction, including both the SFT and RFT settings. (3) We release the REPOST-TRAIN and REPOST-EVAL datasets, including the repository commit ids and the evaluation scripts generated by REPOST. We also release the RFT (Distill) data constructed based on REPOST- TRAIN. (4) We release the docker images of REPOST-TRAIE and REPOST-EVAL, which already install all the external packages required for executing all the evaluation scripts.
All the above resources can be found at https: / /ttpss:////ossteeeeeeeeeeeese code code- -eneeegen.... won.gittuu.iii.
8
Acknowledgement
We thank Ofir Press, Saujas Vaduguru, Atharva Naik, Yuning Mao, and Xuhui Zhou for their helpful feedback on this work. We thank all participants in our human study. This work was supported in part by NSF grant DSES 2222762. Yiqing Xie is supported by the Carnegie Mellon University Presidential Fellowship in the Language Technologies Institute.
References
Anthropic. Claude-3.5, 2024. URL
http://ppp...com/e//////aaaueeee -eweccaauudee----ooeeeeaa sonnet.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. SUPER: Evaluating agents on setting up and Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12622-12645, Miami, Florida, USA, November 2024. Association for Compulational Linguistics. doi: //aclanthology. .rgee...rrrrrr org/202// wtgg/222..mm.mmmaaall emnlp-main. 1tlppmmai..72.//
executing tasks from research repositories.
URL https:
Islem Bouzenia and Michael Pradel. You name it, run it: An Ilm agent to execute tests of 10133.
arbitrary projects, 2024. URL htthtt/::
10

---

Published as a conference paper at COLM 2025
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feld- man, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. The Twelfth International Conference on Learning Representations, 2024. URL https: /ttttre///peerevieew wttpe/ewwwwwwwweeeeeeeeeeeeeeeee mett/forrumtdd-uuiiillpp.
DeepSeek-AL.
Incentiiinng reasoning capability in Ilms via reinforcement learning, 2025. URL https://arriv..rr/abb//5/31 12990
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually-crafted benchmark for evaluating iims class-level code generation, 2023.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, 2020. URL https: //aclanthology. www.thhoooooooooo2222222 findings-nnnen empsp. 139.
Dhruv Gautam, Spandan Garg, Jinu Jang, Neel Sundaresan, and Roshanak Zilouchian Moghaddam. Refactorbench: Evaluating stateful reasoning in language agents through code. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrev/ net/forum? NNNNIIIttttti
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence, 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps, 2021.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. 12186.
URL http://///axxivoog/aba//////////
Kush Jain, Gabriel Synnaeve, and Baptiste Roziere. Testgeneval: A real world unit test generation and test completion benchmark. In The Thirteenth International Conference on Learning Representations, 2025. URL https:///ppenrevieew wttpeeewwwee.oeeeeeeeeeeeeee meee/fooumm?i--75565gee...
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Lolar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024a. URL https: //arxiv. strxxvvorrg/abs/4403.0 07974.
11

---

Published as a conference paper at COLM 2025
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2E: Turning any github repository into a programming agent environment. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 21192-1112222 212224. PMLR, 21-27 Jul 2024b. URL https: http:///ooocedeeneee
mlr. www.mm.r....//..mmmmmmmmmmm wwtpps//ee/eaaeeeeeeeeeeee jain24c...2c.... html.
Nan Jiang, Xiaopeng Li, Shiqi Wang, Qiang Zhou, Soneya Binta Hossain, Baishakhi Ray, Varun Kumar, Xiaofei Ma, and Anoop Deoras. Ledex: Training LLMs to better self-debug and explain code. In The Thirty-eighth Annual Conferennee on Neural Information Processing Systems, 2024. URL https:// wttpppeewwew...eeooooreeeraa.a.. weet/forum?iddd11/22EE
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Represenlations, 2024. URL https://openreviis net//orrmiid--vFayyaeeeee 8YNM66.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: a natural and reliable benchmark for data science code generation. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. Evocodebench: An evolving code generation benchmark with domain- specific evaluations. In The eight Conference on Neural Information Processing Sys- fems Datasets and Benchmarks Track, 2024a. URL http:////pperreeeeeeeeeffrree kvjbFVHpny.
Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Zhi Jin, Hao Zhu, Huanyu Liu, Kaibo Liu, Lecheng Wang, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yihong Dong, Yuqi Zhu, Bin Gu, and Mengfei Yang. Deveval: Evaluating code generation in practical software projects, 2024b.
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, QIANG FU, Xiao Han, Yang Wei, and Deheng Ye. RLTF: Reinforcement learning from unit test feedback. Transactions on Machine Learning Research, 2023a. ISSN 2835-8856. URL https: http:///ppeeeeviiee evee. wwt/ wet/forrmmid--hyymsWlini
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code gener- ated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023b.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024. https://///rr
www.orgg//////.oooo/.ooo... 19173.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023. URL
Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. Next: teaching large language models to reason about code execution. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
12

---

Published as a conference paper at COLM 2025
OpenAl. Gpt-4 technical report, 2023.
OpenAL.
Gpt4o-mini, 2024.
URL
https:///ppeaii.cm///nin
Jiayi Pan, Xingyao Wang. Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym, 2024. URL
http://aaxiii.ooora//////1111/
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024.
Herbert A. Simon. Experiments with a heuristic compiler. J. ACM, 1963. URL https: ///61..rr/11..115//211111 1115/3211886.... 321192.
Xingyao Wang, Boxuan Li, Yufan Song, Frank Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for AI software developers as generalist agents. In The Thirteenth International Conference Learning Representations, 2025. URL https://openreview. forum??ddOOd3aa00Do..
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code all you need. Xiv preprint arXfv:2312.02120, 2023.
Yiqing Xie, Atharva Naik, Daniel Fried, and Carolyn Rose. Data augmentation for code translation with comparable corpora and multiple references. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2023, pp. 13725-13739, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/vi v01/3//vv2223...// //aclanthology. org/2023. mwwdinngs-enl.9999 917//
URL https:
Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, and Carolyn Rose. Codebenchgen: Creating scalable execution-based code generation benchmarks, 2024.
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-comppter interfaces enable automated software engineering, 2024. URL
15793.
Dylan Zhang, Shizhe Diao, Xueyan Zou, and Hao Peng. PLUM: Improving code Ims with execution-guided on-policy preference learning driven by synthetic test cases, 2024. URL https://arxiv. www.:////////....o.oooooo.s 2406/ 06887.
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian- Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2023, URL https://aclanthoolggy org/2023. emnlp-main. 151.
13

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Data Construction Details
Sandboxing and Test Generation Details. Table 14 shows the prompt we to sandbox the target function and its local dependencies to a separate evaluation script. Table 15 shows the prompt template we use to generate tests in the evaluation scripts.
Quality Control Details. Table 16 shows the prompt we use to debug the evaluation scripts there are any errors when we copy the ground truth solution as the new implementation and execute the scripts. Table 17 shows the prompt use to improve the coverage of the test function if there are any missing branches. Table 18 shows the prompt we use in the quality check stage, where we check whether the sandboxed and original functions have exactly the same functionality. Table 19 shows the prompt we use in the quality check stage, where check whether the generated test function is correct, reasonable, and are completing the verification of the functionalities of the ground truth function and the new implementation.
A.2 Human Study Details
Quality Check Agreement Check. For functionality equivalence check, we randomly sample 20 examples from REPOST-EVAL. The instruction we present to the participants exactly the same as the prompt for the LLM functionality checker, as shown in Table 18. Specifically, we show them the original and sandboxed functions and ask them whether their functionalities are the same. We allow minor differences including additional sanity checks or different print information.
As for the test correctness check, we randomly select 10 examples where the LLM predicts as "Yes" and 10 examples where the LLM predicts "No". The instruction is the same as the prompt for the LLM test correctness check, as shown in Table 19. We additionally prevent checking the values of printed logged information because it is typically difficult to match the exact same information in code generation.
Solvability Check. Similar to the setting we use to benchmark coding models in $6, we show the participants the direct or indirect dependencies of the target function and ask them to complete it. After submitting an answer, the participants will see the execution results of our evaluation scripts and can choose to revise their answers accordingly or to give up. Finally, we asked them whether they used external tools (e.g., search engines) in completing the function and asked them to rate the difficulty of each example. Note that do not directly show the evaluation script to the participants and explicitly ask them not to use any AI models.
A.3 Code Generation Training Details
Evaluation Details. The performance of repo-level code generation also depends on the quality of the retrieved context. To mitigate the bias of retrieval models, we evaluate the models with the "Oracle" context for the two repo-level datasets. For RepoEval-Func, we follow the setting in their original paper that retrieves in-repo code fragments with the target function as the query. For REPOST-EVAL, to ensure that we cover all the relevant dependencies in the context, we follow R2E (Jain et al., 2024b) and use dependency context, which includes modules that the target function depends on with the call graph.
Training Details. We train the models with a learning rate of 2e 6, a batch size of 32, and a warm-up ratio of 0.1 for 1 epoch. For QwenCoder, we add a linebreak after the prompt to prevent the first token of the target output from being linebreak.
Model Behavior Analysis. Table 10 shows that the model learns to generate more comments after RFT compared to the base model or SFT. An explanation is that when generating candidate implementations, GPT-4o and Claude-3.5 generate more comments than the
14

---

Published as a conference paper at COLM 2025
Docstring (DocS)
Initial Generation
w/ Debugging
StarCoder2-7B
1327
7415
1573
7415
Qwen225-Coder-76
1428
7415
1708
7415
GPT-4o
2438
7415
2861
7415
Claude-3.5
2503
7415
3092
7415
GPT-4o & Claude-3.5
3110
7415
3606
7415
Table 9: The number examples where we obtained least one successful solution RFT (Self) and RFT (Distill). We report the number (1) after the initial round of generation, and (2) after debugging.
Avg of Comments
Method
Train Data
Output on RepoEval
Qwen2.5-Coder-75
0.48
SFT
0.69
0.56
RFT (Distill)
1.06
0.61
Table 10: Number of comments in (1) the training targets and (2) generations the REPOST- EVAL dataset.
Sanity Checks for Sandboxing
The target function should exist in the evaluation script.
The number tokens in the sandboxed function should NOT be more than 20 fewer than that in the original function.
3. The number of tokens in the entire evaluation script should NOT be more than 50 fewer than the number of tokens in the combination of local de- pendencies.
Table 11: The sanity checks conduct for the sandboxing step.
human-written function. This indicates that training with RFT enables the model to output more interpretable and well-documented code.
A.4 Error Analysis on RepoEval
As shown in Table 13, we randomly sampled 10 failure examples from RepoEval and manually checked the causes of error.
Among the 10 failure cases, 6/10 have unclear instructions unclear package versions, which means they cannot even be solved by human. This is a known issue with the RepoEval dataset, where the intention of the functions cannot be fully entailed from the context.
Since performance is calculated over the total set of examples, including many that are inherently unsolvable or ambiguous, we believe adjusting for this by considering only clearly solvable cases would highlight more substantial gain from our method.
We observe similar issues with our RepoST-Eval datasets and present a potential solution: using model-generated docstrings to improve clarity (see section 5 for details).
15

---

Published as a conference paper at COLM 2025
Sanity Checks for Test Generation
The target function should exist in the evaluation script.
2. The number of tokens in the sandboxed function should NOT be more than 20 fewer than that in the original function.
3. The number of tokens in the entire evaluation script should NOT be fewer than the number of to- kens in the evaluation script obtained from the sand- boxing step.
4. A test function named test. (func name () should exist.
The function should call the target function.
The test function should call the new implementa- tion (new implementation (func name)
7. There should be at least assertions in the test function.
8. There should be main function. The main function should call the test function. the main function calls the test function, the function call should not be a try-except block.
Table 12: The sanity checks we conduct for the test generation step.
Issue
Frequency
Instruction not clear
2/10
Package version mismatch (e.g., transformers)
1/10
Problem with boundary cases
4/10
Correct functionality but wrong implementation
1/10
Wrong functionality
1/10
Format error (e.g., does not generate function)
1/10
Table 13: Summary of common error types on RepoEval. The examples are randomly sampled.
16

---

Published as a conference paper at COLM 2025
Sandboxing Prompt
Instructions:
You're given a piece of PYTHON CODE containing a function called (func name). We also provide you the CONTEXT of the
PYTHON CODE. Your goal is aggregale the PYTHON CODE and the CONTEXT one script, that directly call the (func name; function WITHOUT ANY MODIFICTIONS.
You should edit the original PYTHON CODE little possible and you can add code only if necessary.
NOT any external API, datahase, etc. Instead, create mock interface.
Make sure that your code can be direcily execuled without any modification. For example, statements like "token
"your auth token here" # You need replace this with real token' NOT allowed.
1f you need to write files
disk, use {docker CACHE DIRY as the directory.
Provide your reasoning
revised PYTHON CODE below SOLUTION.
PYTHON CODE:
"python
[code]
CONTEXT:
[contexi)
Your should follow format below:
Reasoning:
"python
# Your Code.
Do NOT include other formatting, Oulput every token of the content with no omission or abbreviation. For example, abbreviation like code keeps unchanged' NOT allowed.
SOLUTION:
The prompt we use to sandbox the target function and its local dependencies to separate evaluation script.
Table 14:
Test Gemeration Prompt
Instructions:
You're given piece of PYTHON CODE containing function called [func_name). Assume will later have another implentation
of the (func.name) function called (func.name) maann.aamee_new...e new www.immleementttionn
Your goal is to add (I) a test function called [test_func_namy, to check whether (func wwwee/ammmeeeeeee... pew.implementation has the same (2) __main__ function
functionality the (func name) function,
calls test function.
the PYTHON CODE already contains __main_. function, remove it and write new ,_main_.. function.
The function (test fune_nameeen should contain at least assert statements, [func_name)
implementation has different
functionality as (func_name), ), an Assertion Error should be triggered.
The test function hest_func_nat
should cover all the major branches of the (func_name) function
on error handling
on print information
function.
The __main_. function should NOT contain a try except structure. If the implementation incorrect, the program should have
exit code.
original PYTHON CODE If you need to write files to the disk, use [docker CACHE DIRY' as the directory.
should
as possible.
Provide your reasoning and the new PYTHON CODE containing your test function feat_ and the __main__ function below SOUUTION.
PYTHON CODE:
"python
(code)
Your answer should follow the format below:
Reasoning:
"'python
The PYTHON CODE containing your test function fuet. name) and the main.... function.
NOT include other formatting, Output every token your edited PYTHON CODE omission abbreviation.
SOLUTION:
Table 15: The prompt we use to generate tests in the evaluation scripts.
17

---

Published as a conference paper at COLM 2025
Debugging Prompt data construction)
Instructions:
called (fune name) its function called (test func name). Assume we will later add another function called (func.name) www.implementaionn the test function aims to check whether [func_ name)
ro given piece PYTHON contaiming
implementation has the same functionality (func_name).
In our experiments, we implemented [func_name] nnew nnw implementation exactly the same as (fune.name), but the PYTHON CODE cannot be sucessfully executed.
debug PYTHON You should modify the code as little as possible, especially the test. / name] function and the name) function.
Your
based
MESSAGE.
Make that after debugging the function test. (func name] still have at least three statements and cover all the major branches (func_name) function.
DO NOT test the logging information of crnor handling and DO NOT test on the print information in the function.
you need to write files disk, (docker.CACHE..D. CCCCHEDDRR the directory.
Providde reasoning and the debugged PYTHON CODE below SOLUTION. necessary, output the bash scripts for Linux
another code block the of
PYTHON CODE:
"python
(code)
FRROR MESSAGE:
(err.nsg)
Your answer should follow the format below:
Reasoning:
"python
# The debugged PYTHON CODE in one piece.
"bash
@ the bash script, if necessary
Do NOT include other formatting. Oulputevery token of your debugged PYTHON CODE with no omission or abbreviation.
SOLUTION:
Table 16: The prompt we use to debug the evaluation scripts if there are any errors when we copy the ground truth solution the new implementation and execute the scripts.
Test Coverage Improvement Prompt
Instructions:
You'de re given a piece of PYTHON CODE containing function called (func_name) and its test function called (test check whether
func func name),
Assume will later add another function called (func name) new implementation, the test
namee) new implementation has the same functionality as name))
You're also given the MISSING LINES of the (func name) new implementation function that are NOT covered by (test_func: func.uccenc nane)name).
Your task is to improve the branch coverage rate of the (test func name) function. should only
the fusc__fun__aaee function. DO NOT modify other parts code. DOO NOT test the logging information of eruor handling and DO NOT test on the print information function.
need to write files the disk, 'docker.. CACHEE..CCHEDIII
the directory.
Provide your reasoning and your revised (test.func.name) function below SOLUTION.
PYTHON CODE:
"python
(code)
MISSING LINES:
(missing.codee code)
Your answer should follow the format below:
Reasoning:
"'yyhhnn
# Your revised [fest_func_name] function
NOT include other formatting. Output every token ffsst_func_mame) function abbreviation.
SOLUTION:
Table 17: The prompt we use to improve the coverage of the test function if there are any missing branches.
18

---

Published as a conference paper at COLM 2025
Functionality Equivslence Check Prompt
Instructions:
We revised python function called (func name) so it can be directly executed in an isolated environment.
given the ORIGINAL FUNCTION and CODE containing REVISED FUNCTION,
Your task is to check whether the functionality the REVISED FUNCTION the same as the ORIGINAL FUNCTION.
If the REVISED FUNCTION is exactly the same the ORINIGAL FUNCTION, output "same" as your answer. same ORIGINAL FUNCTION, output "yes" as your
Otherwise, functionality the REVISED FUNCTION
answer.
(he functionality of the REVISED FUNCTION different, oulput "no". "no".
Provide your reasoning and the answer under "SOLUTION".
ORIGINAL FUNCTION:
(orig. func)
CODE containing the REVISED FUNCTION: (new code)
Your answer should follow the format below:
REASONING: Your reasoning, ANSWER: "same", "yes" "no".
NOT include formatting.
SOLUTION:
Table 18: The prompt we use in the quality check stage, where we check whether the sandboxed and original functions have exactly the same functionality.
Test Correctuess Check Prompt
Instructions:
You given piece PYTHON CODE containing function called (func.name), new implementation
(func name) new implementation (now hidden) and its test function called [test func name).
Your lask is judge whether the lest function satisfies all the CONDITIONS:
The [func name) function should either have return values or modifies global variables or input arguments (such as a list, dictionary, a class, etc.).
<e CONDITION
CONDITION2** ** The lest should only check contents.
values variable states. shoull NOT check printed logged
CONDITION 3 [func_name) new implementation can pass all the test cases IF AND ONLY IF it has the EXACTLY same functionality (func name).
CONDITION 4 The lest cases and assert statements are reasonable. For example, (func name) does not have return values, you should NOT 'assert (func name]0 (func name) new www.iimpemennttonn///c CONDITION The test cases are non-trivial.
the implementation.
the test function satisfies all the CONDITIONS, answer "yes". Provide your reasoning and the answer under "SOLUTION".
PYTHON CODE: (code)
Your answer should follow the format below:
REASONING: Your reasoning.
ANSWER: "yes" or "no".
Do NOT include other formatting.
SOLUTION:
Table 19: The prompt we use in the quality check stage, where we check whether the generated test function is correct, reasonable, and actually comparing the functionalities of the ground truth function and the new implementation.
19

---

Published as a conference paper at COLM 2025
[Case Origifinal Function GitHub Context)
score_ ciareeeeplccittguestiiie
python
quan_uucstions.py
inport prorpts
helpers isport peppenat_ansserr chunker, clean_and, aasiaand_dahh hashh clean_htetl, query_to_ seeery_o__earc_uuu
def shoor_explicit_geestttee question(sttring str) list:
Uses LLMs to score question based on whether it considered explicit or inplicit. OpenAI.
prompt prompts. IS EXPLICIT answor get_ www.ppppnnn..ccmmmm answer htt.ppnss_ansarrprrrr..rooo
string))
results (sonsoods(annseer) ("essttts"
return results
* pyttoon #t helpers.py
inport openai
inport config
oppesss_nnswerpprrppt response_ sespons__temat""janoojjjece rodel-Nonee:
initiate
c1 cene Openai.peenAT/at haall__kyycooooo ig.oPENTT_KKK
model conFiq.MOOEL
Get response
response Llaent.chet.commlettios.c.cc
Lndel-r
tomperature-corrig..eeeE. TEMPERATUUE
nax max_tokens www_tokknssccnnigg wgw.xxonnnn__N___ooo LOO.OUTUTTTTOT TOKENS,
response_Fornatt("vve response_Tornat),
aessages-[<
"role": "user",
"content": prorpt
)1
return respanse. choicos(0) (espanse.choiccs(()).. hts.:e/ccccccceoooeesssssecooooo .ontene.coone
Figure 5: Case study 1. The original score score.expl function and its context extracted from the original GitHub repository. The function calls the text completion function from the OpenAl API.
20

---

Published as a conference paper at COLM 2025
[Case 1, Evaluation Script Sandboxing]
score_explicit_guesseee
inport 1sn
Nock iouratioon
class config:
OPENAIKKY "nock_oopens__ke""
RODEL "ndek_
TEMPERATURE
MVX_OUTPUT_TOKEN
Nock prorpts
proupts:
T5_EXPLICIT following question explicit inplicit? Lingut]"
Nock OpenAT API response
get_ get_openai_ Ipsuettproupt:: str, response respons__fomat="Jse format=""sonoooooioooo"" pojjctt'""" model=Mone):
response sinulating nock_response
"results":
("question": prompt, "scare": "explicit" "explicit" in proppt else "irplicit")
return daan.dunssnnock. responsee
original function from than_questions.p score_esse explicit_questionne
Uses LLMs to score question based on whether it is considered explicit inplicit. Uses OpenAI.
proepts.
answer get get_openai. retaaaaa (AAnpt"""" string))
Laadssa ""eeultss)
return results
Figure Case study 1. The sandboxed score wewpiiciipuuesseeeeeeeeepeepeeee function in the evaluation script. The LLM generates a mock function called get_openai_answer to replace the real API call. With the mock class, the wwtpreeepppiiiiuuueeeeeee as the original function, but does not make real OpenAl API calls.
has the exactly same functionality
21

---

Published as a conference paper at COLM 2025
[Case 1, Evaluation Script Tests]
score_explicit_questiee
cest_score_oo wwt.sscoorriiiccce question(]:
Test Explicit
input_strl "Is this an explicit question?"
assert soore_espiciit..esttimo inpiittt__tt input_stri] = sww.eeeeeeetti....oooooooeeeeeee
Teplicit question
input_str2 "What is the neaning Life?"
assert pespplicctt.uestionnnnnnne
= score_exp
case question
explicit nor implicit)
input_strá "How (tsputssstt__ score_ese ixplicit_qiestion http://owww..olleaattttttototpoo assert cearo_epplicit_uussteee
Very question
Input_stré "?" assert ssore_explicit_quess Lnnp:ppppeeiiionnnnnittc
Test case 5: Very long question
input_strs peciiiitppeiiinniipppuuee
question?"
Test case 6: tase sensitivity
input_stes stri "is this an explicit question?"
Test case 7: different punctuation
input_str7 "Is this an explicit question!"
Test case Bt Special characters
input_stra "Is this an explicit question? #5s"
assert wwwpppccct_uuussstottt.. (nnprtonnintt_strr strr] score_explicit_quctii http:/uwwoooooo.ooeoeooeoaaaaaar
Test case 9: Nurbers in question
input_stra "Is 42 an explicit number?"
assert pepe_eexticiit_uesiioo input_str9) exore_explicit_quees wwwossooon_.e..ieeeeeeeettttttoo
Test 10: Expty string input_str10 assert score_explicit questtppestiiiiiiiuee (inpot_str0)) =
Leading and trailing whitespace
input_stril Is this explicit question?
assert score scorn_explicte wwwppeiptttteeeeeee wwppiicit_uussttnnttttto input_strii strit) == score_ecree score_explictt wwwpeeepccccceeeee wwtpe_/cctt_uuutttioe questiinnneeee htgaro/oooooo.oooeeeeooooooooooo
Test input_strik str12 "Is this explicit question?"
internal whitespace
assert
== score_expticie ettee_epppiicttqesseee itt.ssoooo...e.ooeeeeeetettoiooo
if nane "__nii__:: test_ wtt__ecoo_coolttittuustteee
Figure 7: Case study 1. The tests in the evaluation scripts.
22

---

Published as a conference paper at COLM 2025
[Case 2, Origifinal Function & GitHub Context)
QuizBrain. next wtet_neeeeeeeeeeeeeee certt_quesioo(
python
Python Boetcanp/Day Gare/qutz_brain.pp
class QuizBrain:
def = self-question_Ls5 questions self.scare
questions]:
def stillhhss Saill_ha_qquees ques Lionsself:::: return suuueeeeiiionnuberrr lentself..ueesioon_Lug.
def =eeeseestiio())])
user_answer
current question self.question, List[self.
seff.qustion__is((sef... question, number)
user answwer.towrr)) not ("true", "false"):
user_ansver input ("Qself sonoolffguusttionnnbbbree 1): (current_question..uuete question) (True/False)")
salf-question number
www.cccceeecccreeieeeeee _nnseer, current_question.
def bheck_answerlselff, user_ansver, serrect_ansver):
user_answer. lower() = correct_ansver.. lover(1: (rtnr("(aree"E!")
self. +
else:
prcone"Incorreet))
(T" correct answer is: booreett_nnserr)ynYoo current score (se11.coon//sel...uuttto.... "
Figure 8: Case study 2. The original next eext.question function. The function reads from system inputs, which are not available when testing in docker.
23

---

Published as a conference paper at COLM 2025
[Case Evaluation Script Sandboxing]
QuizBrain next_ (extt_quettio()
MockInput:
def Lini__selff responsesl:
relf.responses responses sedf.
def [sell[sself, prompt):
1f self.index len/sself.respanssss).
response self.. self.respnsestbettt.eee
self. print(prarpt respanse) Sinulate
prompt typing response
return response
ralse Exception("No nore rock Inputs available.") ")
def
answer):
self-question question
self.ansver answer
class QuizBrain:
def _init__self,, questions]:
=
self-question_Liss questiion self.scare
def stilthass heu__shss_uuu_____________ ques LiontseLf);
return self.questionnnureee lentself-question__Lu
def next_queee (salt:
user ___avaewer
current question self.question, ceff.qustion_.is(ssef.... qusetself.uustion nunber)
user, user_answert ansswer.towwr)) not ("true", "false"):
user_ansver input (opptt""Q(selfeeeee wew.iiiggusssttnnuntttieee 1): (curreet_question.uueeee questtionn (True/False)")
deff-quetiio_namber
celf.check, beteeck_sswerruusrrrr.. (ussesrsse_e_rrr,,,, current_qqestion answer)
def check theck_ansveriself, user_ansver, correct_answer user aww.ansse........ lover() correct_answer lover(1:
("Corr"CCe")""
self.score +m
else:
("inoo"Tac)reec") (rint(("The
(correct_ansver)vvie
Figure 9: Case study 2. The sandboxed next_question function the evaluation script. The LLM generates a mock class called MockInput to replace the real system input. With the mock class, the next question has the exactly same functionality as the original function, but does not read system inputs.
24

---

Published as a conference paper at COLM 2025
[Case 2, Evaluation Script Tests]
QuizBrain. next_ [ext_question]
test_next_ ():
Question("Is blue?", "True"), Oisstionn"Is the grass red7", "False"),
wiz_originet QuizBrain(qusstionss quiz_new quizBrain(questions:
responess ["True", "False"]
sock_input Nockingutt
Replace the built-in function with input for testing global input
origina input
input nock_input
Run original inplementation
while wtw..orriiinet.sil....... Stg.inttssill__heeeeeetttee tisneesstion wtwp_orriiiat.nett.ueesstt
Reset input for inplementation
tock_input tockknpuutincckkesspnssss))
input nock_input
while ixplesentation
quiz_new. jo__nd.nne, next wtw..eeneetguuusttoonnoooo new, [ev_implceentation!
quiz_or lriginel. score,
Implerentations"
quiz_originel. quesstion_nubber wiiz.ne..uusttoon.nmbe. "Question numbers differ between
imp lenentatioon""
assert quiz suiz_origins originai....... staii wat.anllsssstthhaaeeeetttteee quiz_new.stii wtiilllaasquusttieee "Question completion state differs
imp tonentatioon"
1f none main test_nest cett_nex__quetioon
Figure 10: Case study The tests in the evaluation scripts, which call the MockInput class to mock the system input.
25


Reviews:
[{"review_id": "qaREE0pSCW", "reviewer": "Reviewer_qkFa", "summary": "REPOST addresses the challenge of repository-level code generation by automatically constructing executable coding environments using sandbox testing. Instead of attempting full integration testing (running original repository test suites, which requires building the entire repo environment), RepoST isolates each target function along with its dependencies into a self-contained script for unit testing. This sandboxing approach dramatically simplifies environment setup by avoiding complex external dependencies and install issues. Using this method, the authors curate RepoST-Train, a training set of 7,415 functions from 824 GitHub repositories, each paired with an auto-generated test suite. They also release RepoST-Eval (296 functions from 99 repos) for benchmarking models on realistic, multi-file coding tasks. Training a code language model on RepoST-Train with execution feedback yields performance gains (+5.5% Pass@1 on HumanEval, +3.5% on a prior repo-level eval). The paper evaluates 12 code generation models on RepoST-Eval where GPT-4 achieves only ~39.5%, indicating the difficulty of these tasks and room for improvement. Overall, RepoST’s novel sandbox testing paradigm enables scalable creation of repository-level code generation benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a critical gap by moving to realistic repository-level code generation tasks, which is highly relevant for advancing code AI. The new REPOST-Eval benchmark is challenging (GPT-4 Pass@1 ~ 39%) and will drive research on more capable coding models.\n\n2. It proposes a novel solution (sandbox testing) to automate environment construction. This approach is innovative compared to prior methods (r2e) and enables an order-of-magnitude more tasks (7.4k vs a few hundred) to be collected. The idea of isolating functions with mocks is a clever way to obtain execution feedback at scale.\n\n3. The pipeline is well-designed with call graph dependency extraction, LLM-generated tests, and rigorous quality control (including LLM and human checks) to ensure each example is correct. High test coverage (~ 98%) indicates the generated tests are thorough. The paper provides strong evidence that training on execution feedback data improves models  (+5.5% Pass@1 on HumanEval) after fine-tuning with REPOST-Train. \n\n4. Overall the paper is clearly written and structured, with detailed examples and an open-source release of code/data. The process is well-documented (prompts in appendix, etc.); the contribution is easy to understand.", "reasons_to_reject": "1. RepoST focuses only on functions that can be tested in isolation. This excludes more complex “integration” tasks where multiple components must work together. Thus, the benchmark may not cover certain real-world scenarios (e.g. adding a feature touching multiple files). The claim of handling repository-level generation is somewhat restricted and limits the impact or generality of the approach. Also it looks like it is focused on python which makes the impact further limited\n\n2. The pipeline heavily depends on GPT-4 for creating the dataset (merging code, writing mocks/tests). The reliance on closed-source AI for data generation might complicate replication.\n\n3. Quality. The dataset’s automatic nature might still hide some incorrect assumptions or trivialized tasks. For example, some tests might not fully specify the problem (e.g., only one possible solution fits the tests but others are logically valid). Such issues could limit the dataset’s effectiveness or the correctness of evaluation. The model then is trained to reproduce possibly flawed logic.\n\n4. Limited eval: it is well known that humaneval is saturated and contaminated with the latest model. The authors must present results on other eval datasets like livecodebench and bigcodebench.", "questions_to_authors": "1. How well would the REPOST sandbox strategy generalize to tasks requiring modifications across multiple functions or files (beyond a single target function)? In other words, can this approach be extended to handle a multi-function feature implementation or bug fix, or is it inherently limited to one function at a time?\n\n2. Did you observe any cases where the LLM-generated tests were incomplete or too lenient, allowing incorrect implementations to pass? More broadly, what’s the confidence that the generated tests fully capture the specification of the target function, and have you considered techniques (or metrics) to measure and improve the specification completeness of these tests?", "extra_scores": {}}, {"review_id": "3jTdavtmr4", "reviewer": "Reviewer_GgBH", "summary": "This paper presents REPOST, a scalable framework for constructing repository-level coding environments using sandbox-based testing to generate execution feedback for code generation models. Unlike traditional integration testing, REPOST isolates functions and their local dependencies, simplifying environment setup and greatly improving scalability.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Proposes a sandbox-based framework that effectively addresses the scalability challenges of constructing executable environments, improving over prior integration testing approaches.\n2. Fully automates environment setup, significantly reducing manual effort and enhancing practical scalability.\n3. Introduces thorough quality control measures, including iterative debugging and test coverage improvements, to ensure the dataset’s reliability and quality.", "reasons_to_reject": "1. Assumes relatively simple local dependencies and API mocking; the impact of this assumption in complex real-world settings is under-explored.\n2. The human study is based on a small sample size (20 examples for quality checks, 27 examples for solvability), limiting the strength of conclusions about dataset quality and difficulty.", "questions_to_authors": "1. It would strengthen the paper by clearly differentiating your method's novelty from closely related recent works, especially R2E and SWE-Gym, emphasizing distinct conceptual or practical improvements.\n\n2. How robust is the sandboxing method when dealing with complex external APIs or highly interactive repository structures? Could you provide more detailed analyses or examples of failure cases?\n\n3. I appreciate the contribution of this work and took time to examine details of the dataset. Line 115 mentions that \"by isolating the function and its local dependencies, we can execute the function by only installing the necessary packages.\" However, the dataset seems to lack a critical field specifying the exact \"necessary packages\" (and their versions) needed to run the sandbox (eval_script). Considering the problem of third-party API evolution, preserving this requirement information could be crucial for future reproducibility and usability.", "extra_scores": {}}, {"review_id": "XLXcdL8LlF", "reviewer": "Reviewer_dCkp", "summary": "The paper addresses the challenge of constructing large-scale, executable repository-level coding datasets for code generation and evaluation. Existing approaches rely on full repository setups for integration testing, which are complex and error-prone. RepoST introduces a scalable alternative by isolating individual functions and their dependencies in sandboxed environments, enabling targeted execution feedback without extensive repository setup.\n\nThe RepoST framework comprises four main steps: repository and function curation, sandboxing with mock APIs for external dependencies, synthetic test generation, and rigorous quality control through iterative debugging and coverage checks.\n\nThe paper also introduces **RepoST-Eval**, a benchmark containing 296 functions from 99 repositories. The best model (GPT-4o) achieves only 39.53 Pass\\@1, underscoring the benchmark’s complexity and potential for further research.\n\nAdditionally, the authors leverage **Rejection Sampling Fine-Tuning (RFT)** using execution feedback from RepoST, achieving notable improvements in Pass\\@1 scores across benchmarks, including a 5.5% gain on HumanEval and a 3.5% gain on RepoEval for the Qwen2.5-Coder model.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Quality and Clarity:**\n\n* The paper is well-structured and clearly written, effectively presenting its motivation, dataset construction pipeline, and methodology with adequate visual aids and illustrative examples.\n* The explanations of the proposed RepoST framework, from sandboxing to test generation and quality control, are clearly articulated, making the methodology easy to follow.\n\n**Novelty and Soundness:**\n\n* The paper addresses a pressing need in the research community for scalable, executable repository-level datasets, and the proposed RepoST method shows strong potential for continued scalability to meet future research demands.\n* The dataset construction and quality control pipeline are meticulously designed, resulting in a high-quality dataset with rigorous filtering and debugging processes.\n* Comprehensive experiments, including RFT analysis, human performance benchmarking, and docstring analysis, enhance the robustness and soundness of the proposed framework.", "reasons_to_reject": "**Performance:**\n\n* While the inclusion of RepoST-Train leads to some improvements, the overall gains remain relatively modest. The limited improvement on HumanEval is understandable given its algorithmic focus, but the marginal gains on RepoST-Eval raise questions about the dataset’s efficacy in training repo-level models. A more granular error analysis could provide valuable insights into the underlying challenges.\n\n**Repo-Level Context:**\n\n* The current implementation primarily focuses on local dependencies for context generation. However, generating code for repo-level functions may benefit from broader contextual information, such as similar or related functions across other files in the repository. Discussing how RepoST handles such contextual retrieval could further strengthen the framework.\n\n**Cost Considerations:**\n\n* The paper does not provide sufficient information regarding the cost of constructing RepoST, including API calls and manual verification efforts. A rough cost breakdown or discussion of resource allocation would be useful to assess the practical feasibility and scalability of the proposed pipeline.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents RepoST, a novel framework for repository-level code generation that focuses on sandbox testing instead of traditional integration testing approaches. This work addresses the important challenge of scalable execution environment construction for code generation models by isolating target functions and their dependencies into self-contained scripts. The contribution is significant and timely given the growing interest in repository-level code generation capabilities both from evaluation as well as training with execution feedback perspective. The authors have developed both a training dataset (RepoST-Train with 7,415 functions) and an evaluation benchmark (RepoST-Eval with 296 functions), showing that training with execution feedback from their system yields meaningful performance improvements across benchmarks. While the approach has limitations in handling complex multi-component tasks, the paper is clearly written, technically sound, and provides a valuable contribution to the field that enables more realistic evaluation and training on repository-level code tasks. While authors don't explore it in the paper, I believe the proposed sandbox testing env can be used for reinforcement learning with verified reward, a key direction for community. 

Pros
- Addresses a critical gap in code generation by focusing on realistic repository-level code tasks beyond simple function-level generation
Introduces an innovative sandboxing approach that overcomes scalability issues of previous integration testing methods
- Creates substantially larger datasets (7,415 functions) compared to prior work through automation
- Demonstrates concrete improvements when fine-tuning models with execution feedback (+5.5% on HumanEval, +3.5% on RepoEval)
- Well-designed pipeline with thorough quality control measures and high test coverage (~98%)
- Open-source release of code and data enhances reproducibility and community impact

Cons
- Limited to functions that can be tested in isolation, excluding more complex real world scenarios. 
- Relatively small human evaluation studies (20-27 examples) to validate dataset quality
- Currently focused primarily on Python, limiting language coverage
Decision: Accept