Can Language Models Replace Programmers? REPOCOD Says ‘Not Yet’
Authors: Shanchao Liang, Nan Jiang, Yiran Hu, Lin Tan
Accepted at: acl25

Can Language Models Replace Programmers for Coding?
REPOCOD Says ‘Not Yet’

Shanchao Liang
Purdue University
liang422@purdue.edu

Nan Jiang
Purdue University
jiang719@purdue.edu

Yiran Hu
Purdue University
hu954@purdue.edu

Lin Tan
Purdue University
lintan@purdue.edu

Abstract
Recently, a number of repository-level code generation benchmarks—such as CoderEval, DevEval, RepoEval, RepoBench, and Long-Code-Arena—have emerged to evaluate the capabilities of large language models (LLMs) beyond standalone benchmarks like HumanEval and MBPP. Thus, a natural question is, would LLMs have similar performance in real world coding tasks as their performance in these benchmarks? Unfortunately, one cannot answer this question, since these benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks.

To address these challenges, we create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects and appropriate metrics for evaluating source code. It includes 980 whole-function generation tasks from 11 popular projects, 50.8 % of which require repository-level context. REPOCOD includes 314 developer-written test cases per instance for better evaluation. We evaluate ten LLMs on REPOCOD and find that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development. In addition, we found that retrieval-augmented generation achieves better results than using target function dependencies as context.

1 Introduction
One critical application of large language models (Bian et al., 2023; Zheng et al., 2023) is code generation (Li et al., 2023a; Ouyang et al., 2023), where models generate executable code snippets given natural language descriptions (Li et al., 2023b; Lozhkov et al., 2024; Touvron et al., 2023; Dubey et al., 2024; Achiam et al., 2023). Such code generation tasks are an integral part of software development.

Research of LLMs for code generation requires a high-quality dataset that evaluates LLMs’ ability to code in real-world development scenarios. Existing LLMs achieve high accuracies, i.e., over 90% pass@1, in solving Python coding tasks in self-contained benchmarks. Would these LLMs achieve similar accuracies in real-world code-generation tasks? To answer this question, we need benchmarks that represent real-world code generation tasks. Specifically, benchmarks need to meet the following four criteria:

Firstly, tasks should be real-world coding tasks such as implementing complete functions based on specifications. While manually-crafted benchmarks such as HumanEval and MBPP are useful at the earlier development of LLMs for code, they fail to represent realistic software development, where real project requirements drive tasks.

Secondly, both tasks and source repositories need to be complex, as prior work (Kiela et al., 2021; Ott et al., 2022) highlights that benchmarks quickly become saturated as models evolve and outperform them. To evaluate the true capabilities of current and future LLMs, it is crucial to construct benchmarks with challenging tasks (Jimenez et al., 2024; bench authors, 2023). Benchmarks such as CoderEval (Zhang et al., 2024), DevEval (Li et al., 2024b), RepoEval (Zhang et al., 2023), RepoBench (Liu et al., 2024), and Long-Code-Arena (Bogomolov et al., 2024), focus on single-line completion or short-function generation, failing to capture the complexity of multi-hundred-line functions commonly found in real-world projects. In addition, the repositories need to be complex to evaluate LLMs’ ability to understand and navigate complex context. However, existing code generation benchmarks overlook this aspect, with the average number of files in their source repositories fewer than 200. Without a stronger benchmark that incorporates complex tasks and large-scale projects, we cannot reliably assess the boundaries of LLMs’
code generation capabilities.

Thirdly, tasks should have repository-level dependencies. Real-world code rarely exists in isolation—only 27% of functions in 500 repositories are self-contained, meaning they do not invoke other functions or modules within the repository and only use standard or public libraries (Li et al., 2024b). Most functions in real-world repositories depend on other components in the repository. Benchmarks such as CoderEval (Zhang et al., 2024) and DevEval (Li et al., 2024b) have the majority of tasks as self-contained or with simple dependencies, such as those dependent on the current file only. Only 10–31% of their tasks require dependencies beyond their current file, failing to represent coding tasks that require repository-level dependencies.

Finally, tasks should be evaluated with appropriate evaluation metrics. Similarity- or matching-based evaluation metrics, such as CodeBLEU (Ren et al., 2020) and BLEU (Papineni et al., 2002) measure textual similarities, and fail to determine whether two code snippets are functionally equivalent. Previous work has shown CodeBLEU and BLEU are unreliable metrics for code generation due to their high mismatch rate with human assessment (Evtikhiev et al., 2023). Execution-based evaluation (e.g., unit tests), although imperfect, avoids these pitfalls by directly verifying functionality. Appendix A.2 uses examples to show how CodeBLEU makes mistakes.

This paper designs and builds REPOCOD, a benchmark for assessing LLMs’ ability in real-world coding tasks, i.e., generating complex functions that require repository-level dependencies in real-world projects, using developer test cases as the validation method.

This paper makes the following contributions:

• A dataset of 980 challenging coding tasks from 11 popular Python projects, characterized by:
  – Complex canonical solutions (Avg. 331.6 tokens per instance).
  – Extensive context from large repositories (Avg. 2,610 files per repository).
  – A focus on repository-level tasks—over half (498, 50.8 %) of tasks require context from other files in the repository.
  – Rigorous evaluation (Avg. 314 developer-written test cases per instance).

• A novel test collection pipeline that reduces the evaluation time to 10.4% of the original

• A thorough study of ten LLMs’ performance on REPOCOD, and the key findings include:
  – LLMs are ineffective on REPOCOD (only 28.6% pass@1 under oracle-retrieval).
  – Higher recall in retrieving a target function’s dependencies as context improves LLMs’ performance.
  – Providing dependencies is suboptimal for repository-level code generation tasks.
  – For self-contained functions, additional context can still help generate better solutions.
  – LLMs exhibit diverse strengths, as each model has uniquely solved tasks.

2 REPOCOD Benchmark

This section introduces REPOCOD’s automated data collection process (Section 2.1), data structure (Section 2.2), and the statistics (Section 2.3).

2.1 Data Collection Pipeline

Figure 1 (a) shows the data collection pipeline. REPOCOD utilizes GitHub as the source of our data. To filter noisy data such as functions with missing descriptions and test cases, we employ a three-stage data collection pipeline to efficiently collect target functions with good documentation and the corresponding test cases for validation.

Step I: Repository Selection. The selection criteria include open-sourced repositories where Python is the primary language (>70%) and those with no less than 2k stars, as popular repositories tend to be well-maintained (Jimenez et al., 2024). We clone the latest version (as of October 2024) of these repositories for analysis in Step II and Step III.

Step II: Target Function Selection. For each collected repository, we perform both static and dynamic analysis to accurately identify the functions defined in the repository that are invoked by the developer-provided test cases, with detailed docstrings as function specifications.

We first collect all the test functions in a repository using PyTest’s test discovery rules1. Then, we identify the functions invoked within these test functions using static and dynamic analysis techniques. For static analysis, we use treesitter (Brunsfeld et al., 2024) to parse the test functions and collect the functions that are invoked. For dynamic analysis, Python’s trace module is used

1https://docs.pytest.org/en/stable/announce/release-8.3.0.html
to examine the execution of each test case. This approach also identifies the indirect function calls invoked by test functions, which are challenging to detect through static analysis alone.

Finally, we filter the target functions, retaining only those with more than ten lines of docstrings and at least two lines of function body.

Step III: Relevant Test Cases Collection. To validate the correctness of LLM-generated code efficiently, we avoid running the entire test suite and instead execute only the relevant test cases—those that specifically invoke the target functions. This targeted approach significantly reduces verification time in large repositories with extensive test suites.

While the previous step provides a mapping between target functions and relevant test cases, certain mappings may still be missed. For instance, test functions might use subprocess.run(command) to invoke repository functions indirectly, which is not collected by static or simple dynamic analysis. Therefore, we employ a two-step collection procedure to capture all relevant test cases. First, we execute all test cases in the repository to establish a reference result. Then, for each target function, we replace it with an assertion failure, rerun all test cases, and compare the results to the reference. If a test result changes from pass to fail, it indicates that the test case is relevant to the target function.

2.2 Benchmark Structure

Figure 1 (b) illustrates the components of REPOCOD’s instances: the target function description, repository snapshot, relevant test cases, and canonical solution. REPOCOD uses the developer-provided docstring as the target function description. The repository snapshot, is the source repository with the target function body removed.

To use REPOCOD as a code generation benchmark, the LLM is provided with the target function description and the repository snapshot, then it is expected to generate a functionally equivalent code snippet as the canonical solution. The LLM-generated solution is considered correct if it passes all relevant tests. Finally, the canonical solutions are the developer-written function bodies.

<table>
  <tr>
    <th rowspan="2">Benchmarks</th>
    <th colspan="4">Instance Statistics</th>
    <th colspan="2">Repository Scale</th>
  </tr>
  <tr>
    <th>#Instances</th>
    <th>#Tokens</th>
    <th>Cyclo.</th>
    <th>#Tests</th>
    <th>#Lines</th>
    <th>#Files</th>
  </tr>
  <tr>
    <td>CrossCodeEval</td>
    <td>2,665</td>
    <td>13.2</td>
    <td>1.0</td>
    <td>0</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>RepoBench</td>
    <td>23,561</td>
    <td>13.7</td>
    <td>1.0</td>
    <td>0</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LCA</td>
    <td>934</td>
    <td>12.0</td>
    <td>1.0</td>
    <td>0</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>CoderEval<sup>1</sup></td>
    <td>230</td>
    <td>108.2</td>
    <td>4.7</td>
    <td>-</td>
    <td>48,821</td>
    <td>152</td>
  </tr>
  <tr>
    <td>DevEval<sup>2</sup></td>
    <td>1,825</td>
    <td>86.3</td>
    <td>3.5</td>
    <td>2.1</td>
    <td>36,640</td>
    <td>164</td>
  </tr>
  <tr>
    <td>R2E-Eval<sup>3</sup></td>
    <td>246</td>
    <td>127.2</td>
    <td>-</td>
    <td>11.5</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>RepoEval</td>
    <td>373</td>
    <td>84.1</td>
    <td>2.7</td>
    <td>274.5</td>
    <td>7,387</td>
    <td>119</td>
  </tr>
  <tr>
    <td>REPOCOD</td>
    <td>980</td>
    <td>331.6</td>
    <td>9.0</td>
    <td>313.5</td>
    <td>290,110</td>
    <td>2,610</td>
  </tr>
</table>

Table 1: Benchmarks Comparison. ‘-’ indicates the data is not publicly available. #Instances: number of instances in each benchmark; #Tokens: average number of tokens of the canonical solution; Cyclo. average cyclomatic complexity of the canonical solution; #Tests: average number of test cases per instance; #Lines: average number of lines per repositories;

are the developer-written function bodies.

2.3 Benchmark Statistics

Table 1 compares REPOCOD with existing code generation benchmarks that use repository-level context, including CrossCodeEval (Ding et al., 2023), RepoBench (Liu et al., 2024), Long-Code-Arena (LCA) (Bogomolov et al., 2024), CoderEval (Zhang et al., 2024), DevEval (Li et al., 2024b), R2E-Eval1 (Jain et al., 2024), and RepoEval (Zhang et al., 2023).

CrossCodeEval, RepoBench, and LCA contain only single-line code generation tasks, resulting in shorter canonical solutions compared to REPOCOD. In addition, they rely on similarity-based metrics, which are insufficient to evaluate code quality. Therefore, we mainly compare REPOCOD against datasets that require full function generation and include test cases for evaluation, such as CoderEval, RepoEval and etc. Among all function generation datasets, REPOCOD outperforms in nearly every aspect, with the exception of having

1CoderEval only releases the task instances along with ground truths but omits the test cases.
2For data statistics not available in DevEval’s paper, we recompute the data from their released dataset available in their GitHub repository.
3R2E-Eval1 has not released its dataset, and the data is derived from the published paper.
<table>
  <tr>
    <th>Dataset</th>
    <th>Repository-Level</th>
    <th>File-Level</th>
    <th>Self-Contained</th>
  </tr>
  <tr>
    <td>CoderEval</td>
    <td>10.0</td>
    <td>53.5</td>
    <td>36.5</td>
  </tr>
  <tr>
    <td>DevEval</td>
    <td>31.3</td>
    <td>41.2</td>
    <td>27.5</td>
  </tr>
  <tr>
    <td>REPOCOD</td>
    <td>50.8</td>
    <td>18.1</td>
    <td>31.1</td>
  </tr>
</table>

Table 2: Data context complexity distribution (%) of code generation benchmarks.

fewer instances compared to DevEval. However, REPOCOD leverages an automated annotation process for annotating the task descriptions for each instance, whereas DevEval relies on human effort, making REPOCOD more scalable for expansion.

Repository Scale. The instances in REPOCOD are collected from repositories with an average of 2,610 files and 290,110 lines of code per repository. This significantly exceeds the scale of source repositories from other benchmarks, underscoring the challenge of efficiently utilizing repository-level context for the evaluated models.

Test Scale. REPOCOD utilizes an average of 313.5 tests per instance, significantly surpassing other benchmarks, highlighting REPOCOD’s robustness in evaluating models. RepoEval’s #Tests is computed as the average number of test cases across all available repositories, regardless of the tests’ relevance to the task. If measured the same way, REPOCOD’s average #Test would be 17,974.

Length Complexity. REPOCOD presents the most challenging tasks among all datasets. With the largest average token count for canonical solutions at 331.6 (Table 1), LLMs must generate solutions that may require more than twice as much code as those in other benchmarks to complete the task.

Cyclomatic Complexity. REPOCOD also includes a cyclomatic complexity (McCabe, 1976) score of 9.0, indicating that the canonical solutions in REPOCOD have a higher structural complexity regarding control flow.

Context Complexity. Table 2 shows REPOCOD, DevEval and CoderEval’s distribution of tasks by three types of context complexity: Self-Contained, File-Level, and Repository-Level. Self-Contained functions use only standard or public libraries; File-Level functions reference functions or classes from the same file but not others; Repository-Level functions may invoke functions or classes from the same file or other files in the repository. Among these benchmarks, REPOCOD has the highest ratio of repository-level dependencies (498).

These statistics show that the tasks in REPOCOD are the most challenging, and contain more repository-level context than other benchmarks, making it particularly suitable for evaluating current and future models on repository-level tasks.

Optimized Test Execution. With our test-collection pipeline, we reduce the number of required test cases per instance from an average of 17,974 (all available test cases from the source repository) to just 313, cutting execution time from 216.9 hours to 22.6 hours, demonstrating a substantial improvement in efficiency.

3 Experiment Setup

Due to the large repository sizes, most LLMs face context window limitations that prevent them from processing all the context in REPOCOD. To address this, we employ Retrieval-Augmented Generation (RAG). We evaluate three popular retrieval settings: \( RAG_{BM25} \), \( RAG_{Dense} \), and current file. In addition to a Baseline (no context) setting, we use two unrealistic “oracle” settings, Callees, and \( RAG_{Dense-oracle} \), to study the potential best scenarios.

3.1 Retrieval Settings

We first define our retrieval corpus, which is used across all RAG methods. The retrieval corpus contains complete function definitions, each including the function signature and full body, sourced from all functions in the repositories. For \( RAG_{BM25} \) and \( RAG_{Dense} \), since the target function body is unavailable during inference, we query using its signature and docstring to retrieve similar functions.

\( RAG_{BM25} \). We use BM25 (Robertson and Zaragoza, 2009) to extract relevant functions from the corpus as context for generation. The signatures, docstrings, file paths, and bodies of the retrieved functions are provided as context.

\( RAG_{Dense} \). We encode all functions in the corpus using the text-embedding-3-small model. The target function’s signature and docstring are encoded into an embedding to be compared against the function embeddings in the corpus. The top-ranked functions based on cosine similarity are provided as context, in the same format as sparse retrieval.

Current File. In this setting, the context is limited to the file containing the target function, with the entire file provided as context, excluding the target function’s body.

3.2 Additional Settings

Baseline. In this setting, the models are provided only with the function signature and the docstring
<table>
  <tr>
    <th>Models</th>
    <th>RAG<sub>BM25</sub></th>
    <th>RAG<sub>Dense</sub></th>
    <th>Current-File</th>
  </tr>
  <tr>
    <td>CodeLlama-7B</td>
    <td>10.7</td>
    <td>10.4</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td>CodeLlama-34B</td>
    <td>12.4</td>
    <td>12.8</td>
    <td>9.6</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-6.7B</td>
    <td>14.0</td>
    <td>14.1</td>
    <td>10.9</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-33B</td>
    <td>16.7</td>
    <td>17.1</td>
    <td>14.9</td>
  </tr>
  <tr>
    <td>OpenCodeInterpreter-6.7B</td>
    <td>12.1</td>
    <td>12.5</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td>OpenCodeInterpreter-33B</td>
    <td>15.3</td>
    <td>16.3</td>
    <td>18.3</td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>14.4</td>
    <td>17.5</td>
    <td>19.8</td>
  </tr>
  <tr>
    <td>DeepSeek-V2.5</td>
    <td>18.5</td>
    <td>20.7</td>
    <td>27.0</td>
  </tr>
  <tr>
    <td>GPT-4o-Mini</td>
    <td>15.1</td>
    <td>15.0</td>
    <td>18.7</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>27.4</td>
    <td>27.0</td>
    <td>26.8</td>
  </tr>
</table>

Table 3: Pass@1(%) of SOTA LLMs on REPOCOD.

to generate the target function. This serves as a minimal context setting, evaluating the model’s ability to generate functions without context information.

Callees. Functions invoked by the canonical solution are considered as oracle context in several benchmarks (Zhang et al., 2024; Li et al., 2024b). We extract these invoked functions and include them along with the last 1,024 tokens from the target function’s prefix. This setup evaluates whether explicitly providing referenced functions, with the target function’s most recent context, improves generation quality.

RAG<sub>Dense-oracle</sub>. We include the canonical solutions along with the task descriptions as queries to perform RAG<sub>Dense</sub>. This setting establishes an upper bound on retrieval effectiveness for RAG<sub>Dense</sub>, showing the potential of an ideal retrieval system.

The Callees and RAG<sub>Dense-oracle</sub> settings are unrealistic in practice, as developers cannot obtain the exact function dependencies beforehand nor have access to the correct solutions.

3.3 Task Formulation

Each data instance of REPOCOD consists of a repository snapshot, target function signature, and docstring (from Section 2.2). The task is to generate the target function that passes all relevant test cases. In practice, LLMs are provided with a prompt consisting of the retrieved context and asked to generate the solution.

Once the model generates the function, the synthesized code is inserted back into the repository to perform test execution. The solution is considered to pass if all relevant test cases pass. We provide the details of prompt construction in Appendix A.3.

We report the models’ performance using Pass@1 (Chen et al., 2021) that measures the probability that the first generated code sample is correct.

![Bar chart comparing intersection sizes for Current-File, RAG_Dense, and RAG_BM25](page_1012_180_370_246.png)

Figure 2: GPT-4o’s passed tasks in different retrievals.

3.4 Model Setup

Given the complexity of REPOCOD, we evaluate only LLMs that meet the following criteria: state-of-the-art (SOTA) performance on existing benchmarks, and a context length of at least 16K.

Thus, we evaluate GPT-4o, GPT-4o-mini, DeepSeek-V2.5, and Claude 3.5 Sonnet, representing commercial LLMs. We also evaluate open-source LLMs such as CodeLlama, DeepSeek-Coder, and OpenCodeInterpreter series, ranging from 6.7B to 34B parameters. For commercial models, we use their official API, and for the open-source LLMs, we use the implementations provided by HuggingFace and vLLM (Kwon et al., 2023). Under each experimental setting (retrieval approach), we let each LLM generate one output per instance in REPOCOD using greedy decoding.

4 Result

4.1 SOTA LLMs’ Pass@1 on REPOCOD

Table 3 shows ten LLMs’ performance on REPOCOD, under three retrieval approaches. On all retrieval approaches, commercial LLMs perform better. Specifically, GPT-4o has the best result, reaching up to 27.4% pass@1. On the other hand, none of the open-sourced LLMs has over 20% pass@1.

This result shows that SOTA LLMs still struggle with repository-level code generation. Compared to their pass@1 on HumanEval (about 90% (Guo et al., 2024)), SOTA LLMs are still far away from writing real-world programs requiring repository-level information. We provide a detailed discussion of the results below.

Impact of Model Size. Table 3 demonstrates the consistent advantage of larger models in solving complex repository-level tasks compared to smaller models within the same architecture.

Impact of Retrieval Approach. The pass@1 results are higher for commercial models when using contexts from the current-file setting, while RAG<sub>BM25</sub> and RAG<sub>Dense</sub> yield similar but lower
outcomes. This trend is also observed in the Open-CodeInterpreter series of LLMs. In contrast, other open-source LLMs perform better with RAG_{BM25} and RAG_{Dense} compared to the current-file setting. These results suggest that commercial models are more likely to benefit from contexts drawn from the current-file setting.

To further investigate the overlap and uniqueness of tasks solved using different retrieval approaches, Figure 2 is an UpSet plot (Lex et al., 2014) that illustrates the overlap and uniqueness of model results using GPT-4o’s result. Though most solutions are solvable by all approaches, each approach has uniquely solved tasks. This suggests that retrieval approaches differ in overall effectiveness and may capture complementary contextual information.

Model-Specific Solution Diversity. Beyond retrieval approaches, we also analyze the distribution of uniquely solved tasks by each model using the same context. Figure 3 shows the number of passed tasks for each commercial LLM and their intersections in REPOCOD under the RAG_{Dense}. Notably, each model solves a unique set of tasks, highlighting the specialized capabilities of individual models. Particularly, Claude 3.5 Sonnet and GPT-4o each have a high number of uniquely solved tasks, indicating that they each excel at certain tasks.

4.2 Impact of Complexity on Performance

We show the performance of SOTA LLMs negatively correlates with the level of complexities. We compare three types of complexities: Context Complexity, Canonical Solution Length, and Cyclomatic Complexity levels. We demonstrate the result in the RAG_{Dense} setting.

Context Complexity. Figure 4a details the comparison of the evaluation results of SOTA LLMs for tasks with different context complexities. All models have the lowest pass@1 when generating functions with repository-level context compared to functions with less complex dependencies. The overall pass rate decreases as the complexity of the context level increases.

Cyclomatic Complexity. Figure 4b shows that LLM performance declines as cyclomatic complexity increases. In the most challenging setting, the best LLM achieves only a 7.0% pass@1 rate.

Token Length. Figure 4b presents the performance of LLMs on tasks that require generating functions of varying token lengths. The pass@1 of the LLMs gradually decreases as the length complexity of the functions increases. In the more challenging scenarios (length > 232), even the top-performing model, GPT-4o, achieves less than 10% pass@1.

With the highest level of complexities in all three categories, REPOCOD establishes a new standard for LLMs in code generation.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="3">Recall Range</th>
  </tr>
  <tr>
    <th>0</th>
    <th>(0, 0.50]</th>
    <th>(0.50, 1]</th>
  </tr>
  <tr>
    <td rowspan="2">DeepSeek-v2.5</td>
    <td>RAG_{BM25}</td>
    <td>10.4</td>
    <td>10.9</td>
    <td>28.8</td>
  </tr>
  <tr>
    <td>RAG_{Dense}</td>
    <td>12.3</td>
    <td>11.0</td>
    <td>27.2</td>
  </tr>
  <tr>
    <td rowspan="2">Claude-3.5</td>
    <td>RAG_{BM25}</td>
    <td>7.1</td>
    <td>4.5</td>
    <td>24.7</td>
  </tr>
  <tr>
    <td>RAG_{Dense}</td>
    <td>7.2</td>
    <td>8.7</td>
    <td>28.3</td>
  </tr>
  <tr>
    <td rowspan="2">GPT-4o</td>
    <td>RAG_{BM25}</td>
    <td>20.3</td>
    <td>16.4</td>
    <td>41.1</td>
  </tr>
  <tr>
    <td>RAG_{Dense}</td>
    <td>16.7</td>
    <td>14.2</td>
    <td>38.0</td>
  </tr>
  <tr>
    <td rowspan="2">GPT-4o-Mini</td>
    <td>RAG_{BM25}</td>
    <td>10.8</td>
    <td>7.3</td>
    <td>24.7</td>
  </tr>
  <tr>
    <td>RAG_{Dense}</td>
    <td>7.7</td>
    <td>9.4</td>
    <td>28.3</td>
  </tr>
</table>

Table 4: Pass@1 by Recall for RAG_{BM25} and RAG_{Dense}-Distribution: (BM25: 492/110/73, Dense: 456/127/92).

Interestingly, When recall falls within (0, 0.5], model performance is comparable to that with a 0 recall rate. In contrast, high recall (in range (0.5,1]) in retrieved content consistently leads to better model performance, indicating the importance of retrieving target function dependencies.

4.4 Impact of Context Type on Performance

We compare the three retrieval approaches with three additional settings using GPT-4o and GPT-4o-mini in Table 5.

Finding 1: SOTA LLMs underperform on REPOCOD even with RAG_{Dense-oracle}. Despite using canonical solutions as queries, RAG_{Dense-oracle} only marginally outperforms RAG_{Dense}. Further analysis reveals that, on average, 22.4 of the top 30 retrieved functions (75%) overlap between RAG_{Dense} and RAG_{Dense-oracle}, suggesting that
(a) Pass@1(%) of LLMs across different context complexities.
(b) Pass@1(%) of LLMs across different cyclomatic complexities.
(c) Pass@1(%) of LLMs across different length complexities.

Figure 4: Pass@1 under context complexity and length complexity. Let n represent the number of instances in each bin. Length complexities distribution:: [0, 66): n=201, [66, 128) : n=194, [128, 232): n=194, [232, 435]: n=196, [435, 4308]: n=195. Cyclomatic complexity distribution (M): Simple (\( M \leq 2 \)): n=269, Moderate (3 \( \leq M \leq 5 \)): n=211 , Complex (6 \( \leq M \leq 10 \)): n=215, and Very Complex (\( M \geq 11 \)): n=285.

<table>
  <tr>
    <th>Model</th>
    <th>Method</th>
    <th>Self</th>
    <th>File</th>
    <th>Repo.</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td rowspan="4">GPT-4o-Mini</td>
    <td>Baseline</td>
    <td>11.5</td>
    <td>7.3</td>
    <td>2.6</td>
    <td>6.2</td>
  </tr>
  <tr>
    <td>RAG<sub>BM25</sub></td>
    <td>22.6</td>
    <td>22.0</td>
    <td>8.0</td>
    <td>15.1</td>
  </tr>
  <tr>
    <td>RAG<sub>Dense</sub></td>
    <td>24.2</td>
    <td>23.7</td>
    <td>6.2</td>
    <td>15.0</td>
  </tr>
  <tr>
    <td>Current-File</td>
    <td>30.5</td>
    <td>23.7</td>
    <td>9.6</td>
    <td>18.7</td>
  </tr>
  <tr>
    <td rowspan="2">GPT-4o</td>
    <td>Callees</td>
    <td>27.9</td>
    <td>23.7</td>
    <td>7.0</td>
    <td>16.5</td>
  </tr>
  <tr>
    <td>RAG<sub>Dense-stable</sub></td>
    <td>28.2</td>
    <td>24.3</td>
    <td>5.2</td>
    <td>15.8</td>
  </tr>
  <tr>
    <td rowspan="4">GPT-4o</td>
    <td>Baseline</td>
    <td>23.6</td>
    <td>11.3</td>
    <td>3.8</td>
    <td>11.3</td>
  </tr>
  <tr>
    <td>RAG<sub>BM25</sub></td>
    <td>39.3</td>
    <td>31.1</td>
    <td>18.7</td>
    <td>27.3</td>
  </tr>
  <tr>
    <td>RAG<sub>Dense</sub></td>
    <td>44.6</td>
    <td>36.7</td>
    <td>12.9</td>
    <td>27.0</td>
  </tr>
  <tr>
    <td>Current-File</td>
    <td>39.3</td>
    <td>35.0</td>
    <td>16.3</td>
    <td>26.8</td>
  </tr>
  <tr>
    <td rowspan="2">GPT-4o</td>
    <td>Callees</td>
    <td>35.1</td>
    <td>31.1</td>
    <td>12.2</td>
    <td>22.8</td>
  </tr>
  <tr>
    <td>RAG<sub>Dense-stable</sub></td>
    <td>45.2</td>
    <td>34.5</td>
    <td>16.3</td>
    <td>28.6</td>
  </tr>
</table>

Table 5: Pass@1 (%) of GPT-4o across context complexities. Self, File, and Repo. refer to Self-Contained, File-Level and Repository-Level.

task descriptions alone enable effective retrieval in REPOCOD. This result underscores that even with an ideal retrieval setup, SOTA LLMs struggle on REPOCOD tasks, highlighting the limitations of existing LLMs for real-world coding tasks.

Finding 2: Additional context significantly improves performance for all types of tasks in REPOCOD. All settings are better than the Baseline setting. Even for self-contained tasks, the inclusion of 1,024 prefix tokens (Callees setting) improves the pass@1 result, suggesting that additional context—beyond just the task description—improves generation performance.

Finding 3: Dependency-based context is sub-optimal. The Callees setting is comparable to RAG<sub>BM25</sub> and RAG<sub>Dense</sub> but does not consistently outperform them. Across all complexity levels, it lags behind the Current-File setting. This indicates that solely using dependencies is not ideal for repository-level code generation. Combined with the findings from Section 4.3, future work should aim for retrieving contents with a high recall of the target function’s dependencies while maintaining high similarity to the task description.

Figure 5: Models’ pass@1 against syntax error rate in different retrievals. Each pair of model and retrieval approach is represented as a dot.

4.5 Failure Reasons

We compute the ratio of syntax errors as the proportion of tasks with syntax errors among all failed samples and present the results of all combinations of models (10 models) and retrieval approaches in Figure 5. Results show that the Current-File setting leads to a higher syntax error ratio, and models with a higher syntax error ratio tend to achieve lower Pass@1 performance. For RAG<sub>BM25</sub> and RAG<sub>Dense</sub> methods, the syntax error ratio remains below 25%, suggesting that LLMs are less likely to produce syntax errors when leveraging these retrieval strategies.

We study GPT-4o’s failures on 30 randomly sampled tasks and find two primary failure reasons. First, LLM-generated functions frequently lack sufficient input validation and parameter handling, such as ignoring or improperly transforming input values, leading to errors when processing unexpected inputs or misusing function arguments. Second, incorrect or incomplete implementation of core logic leads to deviations from the intended functionality, including selecting suboptimal computational methods, mismanaging object states, or misapplying parallel processing strategies. We provide case studies in the Appendix A.7.1.
5 Related Work

5.1 LLMs for Code
LLMs have been widely used for code-related tasks, among which code generation is the most important and common task (Achiam et al., 2023; Li et al., 2023b; Lozhkov et al., 2024; Nijkamp et al., 2023; Rozière et al., 2024; Guo et al., 2024; Luo et al., 2024). Existing work has shown an impressive ability to generate self-contained programs.

However, the performance of LLMs for code generation within the context of repositories remains underexplored. Despite the use of retrieval-augmented generation (RAG) to retrieve relevant context from the repository (Ding et al., 2023; Zhang et al., 2024; Wu et al., 2024a), generating code within the context of the repositories remains more challenging and yields lower accuracy compared to generating self-contained code.

5.2 Code Generation Benchmarks
Code generation benchmarks are important for evaluating LLMs’ ability to generate code. Self-contained benchmarks (Hendrycks et al., 2021; Li et al., 2022; Xia et al., 2024; Liu et al., 2023) are unable to evaluate LLMs’ capability to generate code for real-world projects.

A few repository-level code generation benchmarks have been developed. Concode (Iyer et al., 2018) represents an early approach in this domain, focusing on repository-level code generation evaluated through similarity-based metrics rather than the test-based metrics used in REPOCOD. Cross-CodeEval (Ding et al., 2023), RepoBench (Liu et al., 2024), and Long-Code-Arena (Bogomolov et al., 2024) are benchmarks collected from GitHub projects. They consist of tasks generating a single line of code given the incomplete code snippets and repository context. As discussed in the introduction, they share two limitations: (1) their target code is limited to single-line outputs, and (2) they rely on similarity-based evaluation metrics.

Other benchmarks, such as CoderEval (Zhang et al., 2024), RepoEval (Zhang et al., 2023), R2E-Eval1, EvoCodeBench (Li et al., 2024a) and DevEval(Li et al., 2024b) consists of function generation tasks in Python and/or Java tasks collected from GitHub repositories. However, they fall short in terms of task complexity and scale. For example, they utilize repositories with limited size, fewer than 243 files on average, significantly smaller than widely-used repositories such as scikit-learn (1,640 files by REPOCOD’s collected date).

There is also contemporary work such as SWE-Gym (Pan et al., 2024), which also engages with code at the repository level. However, its primary focus is on training agents for issue fixing, distinguishing it from the code generation emphasis of the benchmarks previously discussed and the objective of REPOCOD.

In contrast, REPOCOD combines complex real-world tasks (analyzed in Section 2.3) from the most popular Python repositories with rigorous test-based assessments and consists of 980 code generation tasks that require LLMs to write large functions instead of single-line snippets, aligning model evaluation with the expectations of modern software development.

Version-Aware Code Generation Benchmarks.
Recognizing the dynamic nature of software, another line of benchmarks specifically addresses API version evolution, a factor that can significantly influence code generation performance. LibEvolutionEval (Kuhar et al., 2025) focuses on version-specific in-line code completion and documentation retrieval across library versions. GitChameleon (Islah et al., 2024) provides manually curated Python problems with executable unit tests, conditioned on specific library versions to assess functional accuracy. CodeUpdateArena (Liu et al., 2025) uses synthetic API updates to evaluate knowledge editing in LLMs, testing if models can apply updates without in-context documentation. VersiCode (Wu et al., 2024b) introduces tasks where models must complete code for a specific library version or update code to a different version. It uses a large, diverse dataset and its novel 'Critical Diff Check' metric, designed to specifically assess correct API usage across versions.

While the evolving nature of library versions and its impact on code generation is an important consideration, REPOCOD and the aforementioned version-focused benchmarks evaluate different facets of this challenge. They primarily target the effect of package versions on LLMs’ code generation ability, often through simpler tasks such as token/line level completions, specific API updates, or less complex function generation. In contrast, REPOCOD evaluates LLMs’ ability to solve complex tasks requiring a deep understanding of repository-level context.
5.3 Automated Test Collection Pipelines

Various approaches have been proposed for collecting benchmarks with repository-level task instances (Zhang et al., 2024, 2023; Jain et al., 2024; Li et al., 2024b). RepoEval executes all tests in the repositories but is impractical for popular repositories due to the sheer volume of test cases. R2E-Eval1, on the other hand, uses LLMs to generate equivalence tests. However, this strategy lacks verification of the correctness of the test cases, limiting its reliability for trustworthy evaluations.

The most comparable approach to ours is proposed in CoderEval, which employs static analysis to identify test cases that reach the target function and supplements them with manually curated test cases. While this approach works for small repositories, it is neither scalable—since it requires human effort—nor does it fully utilize the test cases provided by developers, as static analysis cannot capture all dependencies.

In contrast, our approach uses dynamic analysis to automatically identify test cases from existing developer tests. This execution-based approach is more scalable and is better suited for leveraging existing developer-provided test cases in popular repositories. Additional details on our data collection pipeline are provided in Section 2.1.

6 Conclusion

We present REPOCOD, a real-world, complex dataset designed for code generation tasks with repository-level context. REPOCOD comprises 980 instances from 11 popular Python repositories, including 498 that require repository-level context, with canonical solutions averaging 331 tokens in length—highlighting the benchmark’s complexity and comprehensiveness. In addition, We introduce a scalable automatic extraction method for collecting repository-level code generation tasks. Our evaluation of SOTA LLMs on REPOCOD reveals a maximum pass@1 of 28.6%, with even lower scores for functions requiring repository-level context, showing that existing LLMs fall short of generating realistic repository-level code. This work underscores the need for further research in repository-level code generation.

7 Limitation

Our work has limitations. First, we collect data from only 11 repositories, covering a small subset of those that could be included in this benchmark; future versions of REPOCOD will expand to other popular Python repositories. Second, we evaluate only ten models, representing a subset of popular LLMs. With more time and resources, we could test a broader range, providing a more comprehensive view of LLM capabilities in repository-level code generation. We will publish REPOCOD so the community can evaluate additional models, broadening the scope of tested LLMs.

8 Acknowledgements

We thank the anonymous reviewers for their feedback on this work. This research was supported in part by NSF 1901242 and 2006688 and a CFI fund.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.

Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, Ben He, Shanshan Jiang, and Bin Dong. 2023. Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. arXiv preprint arXiv:2303.16421.

Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, and Timofey Bryksin. 2024. Long code arena: a set of benchmarks for long-context code models. Preprint, arXiv:2406.11612.

Max Brunsfeld, Andrew Hlynkyi, Amaan Qureshi, Patrick Thomson, Josh Vera, Phil Turnbull, dundargoc, Timothy Clem, ObserverOfTime, Douglas Creager, Andrew Helwer, Rob Rix, Daumantas Kavolis, Hendrik van Antwerpen, Michael Davis, Ika, Tuân-Anh Nguyễn, Amin Yahyaabadi, Stafford Brunk, Matt Massicotte, Niranjan Hasabnis, bfredl, Mingkai Dong, Samuel Moelius, Steven Kalt, Will Lillis, Kolja, Vladimir Panteleev, and Jonathan Arnett. 2024. tree-sitter/tree-sitter: v0.22.6.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidi Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz
Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code.

Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Haitian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Preprint, arXiv:2310.11248.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2023. Out of the blue: How should we assess quality of the code generation models? Journal of Systems and Software, 203:111741.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. Preprint, arXiv:2401.14196.

Dan Hendrycks, Steven Basart, Saurav Kadavath, Manasas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with apps. NeurIPS.

Nizar Islah, Justine Gehring, Diganta Misra, Eilif Muller, Irina Rish, Terry Yue Zhuo, and Massimo Caccia. 2024. Gitchameleon: Unmasking the version-switching capabilities of code generation models. Preprint, arXiv:2411.05830.

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643–1652, Brussels, Belgium. Association for Computational Linguistics.

Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. 2024. R2e: Turning any github repository into a programming agent environment. In ICML.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110–4124, Online. Association for Computational Linguistics.

Sachit Kuhar, Wasi Uddin Ahmad, Zijian Wang, Nihal Jain, Haifeng Qian, Baishakhi Ray, Murali Krishna Ramathan, Xiaofei Ma, and Anoop Deoras. 2025. LibEvolutionEval: A benchmark and study for version-specific code generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6826–6840, Albuquerque, New Mexico. Association for Computational Linguistics.

Woosuk Kwon, Zhuchan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattentive. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Alexander Lex, Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, and Hanspeter Pfister. 2014. Upset: Visualization of intersecting sets. IEEE Transactions on Visualization and Computer Graphics, 20:1983–1992.

Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2023a. Enabling programming thinking in large language models toward code generation. arXiv preprint arXiv:2305.06599.

Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024a. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. Preprint, arXiv:2404.00599.

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, and Mengfei Yang. 2024b. DevEval: A manually-annotated code generation benchmark aligned with real-world code repositories. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3603–3614, Bangkok, Thailand. Association for Computational Linguistics.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zhetonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj,
Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebeze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhai Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023b. Starcoder: may the source be with you! Preprint, arXiv:2305.06161.

Yuji Li, David Choi, Junyoung Chung, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, volume 36, pages 21558–21572. Curran Associates, Inc.

Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Representations.

Zeyu Leo Liu, Shrey Pandit, Xi Ye, Eunsol Choi, and Greg Durrett. 2025. Codeupdatearena: Benchmarking knowledge editing on api updates. Preprint, arXiv:2407.06249.

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wending Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhai Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Cheng-hao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebeze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Motawey, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2024. Starcoder 2 and the stack v2: The next generation. Preprint, arXiv:2402.19173.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations.

T.J. McCabe. 1976. A complexity measure. IEEE Transactions on Software Engineering, SE-2(4):308–320.

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. Preprint, arXiv:2203.13474.

Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Janina Brauner, and Matthias Samwald. 2022. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13.

Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2023. Llm is like a box of chocolates: the non-determinism of chatgpt in code generation. arXiv preprint arXiv:2308.02828.

Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training software engineering agents and verifiers with swe-gym. Preprint, arXiv:2412.21139.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. Preprint, arXiv:2009.10297.

Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattarola, Wenhan Xiong, Alexandre Défossé, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024. Code llama: Open foundation models for code. Preprint, arXiv:2308.12950.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwijal Bhargava, Shruti
Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. 2024a. Repoformer: Selective retrieval for repository-level code completion. Preprint, arXiv:2403.10059.

Tongtong Wu, Weigang Wu, Xingyu Wang, Kang Xu, Suyu Ma, Bo Jiang, Ping Yang, Zhenchang Xing, Yuan-Fang Li, and Gholamreza Haffari. 2024b. Versi-code: Towards version-controllable code generation. CoRR, abs/2406.07411.

Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang. 2024. Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm. arXiv preprint arXiv:2403.19114.

Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2471–2484. Singapore. Association for Computational Linguistics.

Yakun Zhang, Wenjie Zhang, Dezhi Ran, Qihao Zhu, Chengfeng Dou, Dan Hao, Tao Xie, and Lu Zhang. 2024. Learning-based widget matching for migrating gui test cases. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, volume 66 of ICSE '24, page 1–13. ACM.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
A Appendix

A.1 Basic Statistics of REPOCOD

REPOCOD contains 980 code generation tasks from 11 widely-used repositories, covering a wide range of functionalities, including data science, scientific computing, web development, and software development tools.

Table 6 presents detailed statistics for instances from each repository, categorized by context complexity types: repository-level, file-level, and self-contained. For each category, it shows #NL (number of tokens in target function descriptions), #GT (number of tokens in canonical solutions), Cyclo (average cyclomatic complexity of the canonical solution) (McCabe, 1976)), and #Funcs. (number of target functions). Additionally, we report repository statistics: #Line (average lines in Python files per repository) and #Files (average Python files per repository).

We define three types of context complexities. Repository-level involves canonical solutions that call functions from other files in the repository; file-level involves calls to functions within the same file (excluding the target function) but not from other files; and self-contained includes functions that only use commonly used libraries (e.g., numpy) or Python built-in modules (e.g., os).

Among the three settings, the repository-level functions have the longest token length for the canonical solutions (441.2), compared to file-level functions (192.0) and self-contained functions (233.8). Additionally, the repository-level functions have the highest cyclomatic complexity (10.8) compare to other two categories. The highest length and cyclomatic complexity with the additional repository-level context makes repository-level the most challenging category.

A.2 Motivating Example

Figure 6 (a) shows two RepoBench examples where LLM-generated code is semantically correct but wrongly penalized by CodeBLEU, edit similarity, and exact match. In contrast, Figure 6 (b) demonstrates how these metrics assign high scores to incorrect solutions, underscoring their limitations in accurately evaluating generated code.

A.3 Prompting Format

As described in Section 2.2, the data instance of REPOCOD consists of the repository snapshot, target function signature, and docstring. Figure 7 demonstrates an example of the prompt construction process, the format of the prompt, and the evaluation pipeline used for all our experiments.

The section highlighted in green represents the data instance of REPOCOD (step ①). The format of the prompt is detailed in the section highlighted in blue (step ②). Our prompt consists of the system prompt, the file path to the target function (relative to the repository root), the retrieval context, and the target function description (the function signature and docstring of the target function). The retrieval context contains the relative file path, as well as the signature, the docstring, and the body of the retrieved functions. If the context exceeds LLM’s context window, we truncate it from the beginning.

Once the LLM generates the solution (step ③), the code snippet is inserted into the repository for testing (step ④, modified file highlighted in blue). Then REPOCOD executes the relevant test cases (step ⑤) and determines the correctness of the solution (step ⑤). As one of the test cases fails, this generated code snippet is considered incorrect.

<table>
  <tr>
    <th>CodeBLEU: 58.06</th>
    <th>Edit Similarity: 85</th>
    <th>Exact Match: False</th>
  </tr>
  <tr>
    <td colspan="3">GT:<br>image_paths = [for p in image_paths<br>if is_valid_image(p)]</td>
  </tr>
  <tr>
    <td colspan="3">LLM:<br>image_paths = [image_path for image_path in image_paths<br>if is_valid_image(image_path)]</td>
  </tr>
</table>

(a): Semantic equivalent code snippet with low score

<table>
  <tr>
    <th>CodeBLEU: 76.86</th>
    <th>Edit Similarity: 98</th>
    <th>Exact Match: False</th>
  </tr>
  <tr>
    <td colspan="3">GT:<br>self.up2 + UpsampleBN(skip_input = 1024 + 64,<br>output_features = 512)</td>
  </tr>
  <tr>
    <td colspan="3">LLM:<br>self.up2 + UpsampleBN(skip_input = 1024 + 64,<br>output_features = 512)</td>
  </tr>
</table>

(b): Incorrect code snippet with high score

Figure 6: Two examples from RepoBench showing misleading metrics results. The yellow and blue highlights indicate the difference between ground truth (GT) and the LLM-generated code.

A.4 Evaluation Result By Repository

Table 7 presents detailed LLM performance on REPOCOD, revealing several key insights. First, repositories differ significantly in difficulty. Models perform well on flask, with pass@1 often above 40%, indicating an alignment with LLM capabilities. Second, GPT-4o is the strongest LLM on REPOCOD, consistently outperforming or matching other models across all repositories. Finally, LLMs exhibit specialization across repositories. For instance, while Claude 3.5 Sonnet and DeepSeekCoder-33B perform similarly overall, Claude excels on pylint, whereas DeepSeekCoder performs better on plotly.py.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="4">Repository-level</th>
    <th colspan="4">File-level</th>
    <th colspan="4">Self-contained</th>
    <th colspan="4">Total</th>
  </tr>
  <tr>
    <th>#NL</th><th>#GT</th><th>Cyclo.</th><th>#Funcs.</th>
    <th>#NL</th><th>#GT</th><th>Cyclo.</th><th>#Funcs.</th>
    <th>#NL</th><th>#GT</th><th>Cyclo.</th><th>#Funcs.</th>
    <th>#NL</th><th>#GT</th><th>Cyclo.</th><th>#Funcs.</th>
  </tr>
  <tr>
    <td>astropy</td><td>360.6</td><td>447.9</td><td>10.5</td><td>51</td><td>404.1</td><td>195.7</td><td>4.9</td><td>14</td><td>227.6</td><td>238.2</td><td>5.9</td><td>20</td><td>336.5</td><td>357.0</td><td>8.5</td><td>85</td>
  </tr>
  <tr>
    <td>datasets</td><td>499.7</td><td>348.7</td><td>9.6</td><td>19</td><td>292.3</td><td>168.8</td><td>5.1</td><td>18</td><td>312.5</td><td>128.9</td><td>3.1</td><td>22</td><td>366.6</td><td>211.9</td><td>5.8</td><td>59</td>
  </tr>
  <tr>
    <td>flask</td><td>277.9</td><td>171.9</td><td>5.2</td><td>8</td><td>356.1</td><td>160.4</td><td>5.2</td><td>10</td><td>233.0</td><td>88.6</td><td>3.5</td><td>25</td><td>270.0</td><td>120.8</td><td>4.2</td><td>43</td>
  </tr>
  <tr>
    <td>more-itertools</td><td>136.6</td><td>172.2</td><td>2.6</td><td>8</td><td>293.1</td><td>127.0</td><td>4.0</td><td>18</td><td>243.2</td><td>105.2</td><td>5.1</td><td>60</td><td>252.2</td><td>106.5</td><td>4.6</td><td>86</td>
  </tr>
  <tr>
    <td>plotly.py</td><td>1,357.4</td><td>1,373.9</td><td>37.0</td><td>15</td><td>1,153.3</td><td>1,091.9</td><td>9.9</td><td>7</td><td>1,169.5</td><td>1,055.5</td><td>24.3</td><td>54</td><td>1,249.0</td><td>1,153.5</td><td>25.3</td><td>76</td>
  </tr>
  <tr>
    <td>pandas</td><td>180.9</td><td>386.2</td><td>14.0</td><td>9</td><td>163.9</td><td>272.3</td><td>7.1</td><td>7</td><td>169.8</td><td>178.9</td><td>8.0</td><td>10</td><td>172.0</td><td>275.8</td><td>9.8</td><td>26</td>
  </tr>
  <tr>
    <td>scikit-learn</td><td>217.9</td><td>372.1</td><td>7.2</td><td>24</td><td>299.7</td><td>294.9</td><td>5.6</td><td>27</td><td>212.8</td><td>237.1</td><td>5.5</td><td>50</td><td>224.1</td><td>344.0</td><td>6.8</td><td>314</td>
  </tr>
  <tr>
    <td>seaborn</td><td>230.8</td><td>538.8</td><td>15.0</td><td>13</td><td>272.7</td><td>168.0</td><td>4.8</td><td>38</td><td>163.3</td><td>201.1</td><td>5.7</td><td>27</td><td>227.9</td><td>241.3</td><td>6.8</td><td>78</td>
  </tr>
  <tr>
    <td>sphinx</td><td>201.1</td><td>548.4</td><td>15.4</td><td>14</td><td>228.0</td><td>240.8</td><td>5.5</td><td>4</td><td>274.1</td><td>88.1</td><td>3.9</td><td>15</td><td>237.5</td><td>301.9</td><td>8.9</td><td>33</td>
  </tr>
  <tr>
    <td>sympy</td><td>874.7</td><td>603.4</td><td>17.8</td><td>67</td><td>821.3</td><td>182.7</td><td>6.5</td><td>14</td><td>962.4</td><td>138.7</td><td>4.6</td><td>16</td><td>881.5</td><td>466.0</td><td>14.0</td><td>97</td>
  </tr>
  <tr>
    <td>xarray</td><td>791.5</td><td>366.9</td><td>10.7</td><td>57</td><td>731.6</td><td>102.1</td><td>2.8</td><td>20</td><td>306.2</td><td>114.3</td><td>2.3</td><td>6</td><td>742.0</td><td>284.8</td><td>8.2</td><td>83</td>
  </tr>
  <tr>
    <td>Average</td><td>431.9</td><td>441.2</td><td>10.8</td><td>498</td><td>428.1</td><td>192.0</td><td>5.0</td><td>177</td><td>528.0</td><td>233.8</td><td>8.3</td><td>305</td><td>461.1</td><td>331.6</td><td>9.0</td><td>980</td>
  </tr>
</table>

Table 6: Basic statistics of REPOCOD, with details broken down by each collected repository.

![Illustrative example of prompting LLMs and evaluating the LLM-generated code.](page_101_624_1247_312.png)

Figure 7: Illustrative example of prompting LLMs and evaluating the LLM-generated code.

A.5 Recall of Retrieval approach

Table 8 presents the recall rates of two different retrieval settings—RAG_{BM25} and RAG_{Dense}—applied across all selected repositories. These repositories include pylint, sphinx, seaborn, flask, sympy, more-itertools, scikit-learn, xarray, datasets, plotly.py, and astropy. The recall rates reflect performance at both the repository-level and file-level contexts, providing insights into how each setting performs for individual repositories as well as overall.

The recall rates for RAG_{Dense} retrieval are generally higher than those for RAG_{BM25} retrieval across most repositories, highlighting the effectiveness of the RAG_{Dense} method in retrieving the invoked functions relevant to the target function.

There are notable variations between repositories. For instance, more-itertools and plotly.py show significantly higher recall rates with the RAG_{Dense} setting, while scikit-learn demonstrates relatively low recall rates in both settings.

The total recall rate is computed using micro-averaging, and the result shows that neither retrieval method achieves a high recall for the oracle contexts.

A.6 Inference Length V.S. GT length

We analyze the relationship between inference length and the length of canonical solutions across all retrieval settings. Table 9 reports the average inference-to-canonical solution length ratio using micro-averaging. On average, LLM-generated results are much longer than canonical solutions.

We further investigate the causes of excessive generation length and summarize the following contributing factors:

Firstly, it is common for those underperforming LLMs to generate repetitive code snippets until they reach the token limit.

Secondly, despite being instructed to generate only the target function, LLMs sometimes produce additional functions, resulting in longer outputs. One contributing factor is that when models reference functions not retrieved in the context, they generate the missing helper function definitions, further inflating the output length.

To address this, we apply a filtering method that retains only the first unnested function in the generated output. Specifically, we discard all generated tokens following the first complete function
<table>
  <tr>
    <th>Setting</th>
    <th>pylint</th>
    <th>sympy</th>
    <th>sphinx</th>
    <th>seaborn</th>
    <th>flask</th>
    <th>more-itertools</th>
    <th>scikit-learn</th>
    <th>xarray</th>
    <th>datasets</th>
    <th>plotly.py</th>
    <th>astropy</th>
    <th>Total</th>
  </tr>
  <tr>
    <td>CodeLlama-7B</td>
    <td>7.69</td>
    <td>7.22</td>
    <td>9.09</td>
    <td>23.08</td>
    <td>37.21</td>
    <td>13.95</td>
    <td>5.10</td>
    <td>6.02</td>
    <td>22.03</td>
    <td>6.58</td>
    <td>5.88</td>
    <td>104.11</td>
  </tr>
  <tr>
    <td>CodeLlama-34B</td>
    <td>0.00</td>
    <td>4.12</td>
    <td>18.18</td>
    <td>21.79</td>
    <td>32.56</td>
    <td>24.42</td>
    <td>5.73</td>
    <td>16.87</td>
    <td>18.64</td>
    <td>14.47</td>
    <td>10.59</td>
    <td>127.76</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-6.7B</td>
    <td>7.69</td>
    <td>6.19</td>
    <td>15.15</td>
    <td>26.92</td>
    <td>41.86</td>
    <td>12.79</td>
    <td>6.37</td>
    <td>16.87</td>
    <td>16.95</td>
    <td>25.00</td>
    <td>14.12</td>
    <td>140.88</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-33B</td>
    <td>3.85</td>
    <td>7.22</td>
    <td>27.27</td>
    <td>28.21</td>
    <td>37.21</td>
    <td>11.63</td>
    <td>7.32</td>
    <td>15.66</td>
    <td>33.90</td>
    <td>43.42</td>
    <td>16.47</td>
    <td>171.44</td>
  </tr>
  <tr>
    <td>OpenCodeInterpreter-6.7B</td>
    <td>3.85</td>
    <td>4.12</td>
    <td>12.12</td>
    <td>16.67</td>
    <td>41.86</td>
    <td>18.60</td>
    <td>6.37</td>
    <td>10.84</td>
    <td>16.95</td>
    <td>21.05</td>
    <td>12.94</td>
    <td>124.54</td>
  </tr>
  <tr>
    <td>OpenCodeInterpreter-33B</td>
    <td>11.54</td>
    <td>7.22</td>
    <td>21.21</td>
    <td>20.51</td>
    <td>41.86</td>
    <td>29.07</td>
    <td>8.28</td>
    <td>13.25</td>
    <td>25.42</td>
    <td>23.68</td>
    <td>16.47</td>
    <td>163.33</td>
  </tr>
  <tr>
    <td>Claude-3.5 Sonnet</td>
    <td>19.23</td>
    <td>11.34</td>
    <td>15.15</td>
    <td>25.64</td>
    <td>39.53</td>
    <td>38.37</td>
    <td>6.05</td>
    <td>9.64</td>
    <td>27.12</td>
    <td>36.84</td>
    <td>10.59</td>
    <td>174.55</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>15.38</td>
    <td>13.40</td>
    <td>27.27</td>
    <td>30.77</td>
    <td>41.86</td>
    <td>36.05</td>
    <td>9.87</td>
    <td>16.87</td>
    <td>30.51</td>
    <td>35.53</td>
    <td>16.47</td>
    <td>207.71</td>
  </tr>
  <tr>
    <td>GPT-4o-Mini</td>
    <td>11.54</td>
    <td>11.34</td>
    <td>15.15</td>
    <td>25.64</td>
    <td>39.53</td>
    <td>18.60</td>
    <td>6.05</td>
    <td>10.84</td>
    <td>33.90</td>
    <td>25.00</td>
    <td>9.41</td>
    <td>150.00</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>19.23</td>
    <td>15.46</td>
    <td>27.27</td>
    <td>37.18</td>
    <td>58.14</td>
    <td>43.02</td>
    <td>13.69</td>
    <td>24.10</td>
    <td>47.46</td>
    <td>44.74</td>
    <td>23.53</td>
    <td>270.04</td>
  </tr>
</table>

Table 7: Pass@1(%) of LLMs on REPOCOD under RAG_{Dense} retrieval, with details shown for each repository.

<table>
  <tr>
    <th>Repository</th>
    <th>RAG_{BM25}</th>
    <th>RAG_{Dense}</th>
  </tr>
  <tr>
    <td>pylint</td>
    <td>0.14</td>
    <td>0.18</td>
  </tr>
  <tr>
    <td>sympy</td>
    <td>0.16</td>
    <td>0.14</td>
  </tr>
  <tr>
    <td>sphinx</td>
    <td>0.09</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>seaborn</td>
    <td>0.17</td>
    <td>0.21</td>
  </tr>
  <tr>
    <td>flask</td>
    <td>0.11</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>more-itertools</td>
    <td>0.05</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>scikit-learn</td>
    <td>0.09</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>xarray</td>
    <td>0.07</td>
    <td>0.15</td>
  </tr>
  <tr>
    <td>datasets</td>
    <td>0.21</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>plotly.py</td>
    <td>0.02</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>astropy</td>
    <td>0.17</td>
    <td>0.15</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>0.12</th>
    <th>0.15</th>
  </tr>
</table>

Table 8: Recall rate of the top three and top ten retrieved contents across retrieval settings for repository-level and file-level contexts.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">RAG_{BM25}</th>
    <th colspan="2">RAG_{Dense}</th>
    <th colspan="2">Current-File</th>
  </tr>
  <tr>
    <th>Pass</th>
    <th>Fail</th>
    <th>Pass</th>
    <th>Fail</th>
    <th>Pass</th>
    <th>Fail</th>
  </tr>
  <tr>
    <td>CodeLlama-7B</td>
    <td>18.2</td>
    <td>11.6</td>
    <td>9.1</td>
    <td>11.9</td>
    <td>30.4</td>
    <td>22.3</td>
  </tr>
  <tr>
    <td>CodeLlama-34B</td>
    <td>1.8</td>
    <td>15.2</td>
    <td>1.9</td>
    <td>11.5</td>
    <td>22.5</td>
    <td>34.5</td>
  </tr>
  <tr>
    <td>DeepSeek-6.7B</td>
    <td>2.2</td>
    <td>9.6</td>
    <td>7.0</td>
    <td>5.7</td>
    <td>20.0</td>
    <td>13.1</td>
  </tr>
  <tr>
    <td>DeepSeek-33B</td>
    <td>8.2</td>
    <td>7.4</td>
    <td>8.4</td>
    <td>3.9</td>
    <td>17.9</td>
    <td>26.5</td>
  </tr>
  <tr>
    <td>OCL-6.7B</td>
    <td>2.3</td>
    <td>2.5</td>
    <td>3.5</td>
    <td>2.8</td>
    <td>2.5</td>
    <td>1.8</td>
  </tr>
  <tr>
    <td>OCL-33B</td>
    <td>2.4</td>
    <td>1.7</td>
    <td>2.1</td>
    <td>1.5</td>
    <td>2.3</td>
    <td>1.3</td>
  </tr>
  <tr>
    <td>Claude-3.5</td>
    <td>19.2</td>
    <td>1.7</td>
    <td>25.5</td>
    <td>1.7</td>
    <td>9.8</td>
    <td>7.0</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>7.3</td>
    <td>6.4</td>
    <td>8.0</td>
    <td>7.6</td>
    <td>3.6</td>
    <td>1.7</td>
  </tr>
  <tr>
    <td>GPT-4-Mini</td>
    <td>4.0</td>
    <td>1.9</td>
    <td>4.9</td>
    <td>1.8</td>
    <td>12.1</td>
    <td>1.5</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>1.7</td>
    <td>1.5</td>
    <td>1.7</td>
    <td>1.5</td>
    <td>2.5</td>
    <td>1.2</td>
  </tr>
</table>

Table 9: Ratio of generation length over canonical solution length

that matches the target function name. Table 10 presents the average filtered inference-to-canonical solution length ratio using micro-averaging. We make two key observations: (1) Failed generations consistently exhibit longer relative lengths after filtering, particularly in Current-File retrieval settings, suggesting that verbosity may correlate with incorrect solutions. (2) LLMs consistently generate longer code snippets than human developers, even when only the first unnested function is considered, indicating a systematic tendency toward verbosity.

Model-Specific Insights.

GPT-4o exhibits similar generation ratios before and after filtering, suggesting it adheres more closely to the task instructions and avoids unnecessary verbosity.

<table>
  <tr>
    <th>Models</th>
    <th>RAG_{BM25}</th>
    <th>RAG_{Dense}</th>
    <th>Current-File</th>
    <th># of Dependencies</th>
  </tr>
  <tr>
    <td>CodeLlama-7B</td>
    <td>31.8</td>
    <td>22.1</td>
    <td>14.2</td>
    <td>7.1</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td>CodeLlama-34B</td>
    <td>35.4</td>
    <td>28.4</td>
    <td>14.2</td>
    <td>6.0</td>
    <td>6.2</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>42.6</td>
    <td>31.1</td>
    <td>22.0</td>
    <td>17.3</td>
    <td>9.1</td>
  </tr>
  <tr>
    <td>Claude-3.5</td>
    <td>27.2</td>
    <td>18.9</td>
    <td>9.9</td>
    <td>2.4</td>
    <td>2.3</td>
  </tr>
  <tr>
    <td>GPT-4-Mini</td>
    <td>33.1</td>
    <td>22.6</td>
    <td>12.8</td>
    <td>4.8</td>
    <td>0.6</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>22.6</td>
    <td>20.5</td>
    <td>14.9</td>
    <td>6.0</td>
    <td>5.1</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>24.3</td>
    <td>21.1</td>
    <td>13.5</td>
    <td>5.4</td>
    <td>2.8</td>
  </tr>
  <tr>
    <td>GPT-4-Mini</td>
    <td>30.5</td>
    <td>23.2</td>
    <td>14.9</td>
    <td>8.9</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>39.3</td>
    <td>32.6</td>
    <td>22.7</td>
    <td>16.7</td>
    <td>14.8</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>44.6</td>
    <td>36.3</td>
    <td>16.3</td>
    <td>12.5</td>
    <td>9.1</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>39.3</td>
    <td>33.7</td>
    <td>24.1</td>
    <td>16.1</td>
    <td>10.2</td>
  </tr>
</table>

Table 10: Ratio of filtered generation length over canonical solution length.

<table>
  <tr>
    <th>Model</th>
    <th>Retrieval</th>
    <th colspan="5"># of Dependencies</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>0</td>
    <td>1</td>
    <td>2</td>
    <td>[3,4]</td>
    <td>[5,35]</td>
  </tr>
  <tr>
    <td>RAG_{BM25}</td>
    <td></td>
    <td>31.8</td>
    <td>22.1</td>
    <td>14.2</td>
    <td>7.1</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td>RAG_{Dense}</td>
    <td></td>
    <td>35.4</td>
    <td>28.4</td>
    <td>14.2</td>
    <td>6.0</td>
    <td>6.2</td>
  </tr>
  <tr>
    <td>Current-File</td>
    <td></td>
    <td>42.6</td>
    <td>31.1</td>
    <td>22.0</td>
    <td>17.3</td>
    <td>9.1</td>
  </tr>
  <tr>
    <td>RAG_{BM25}</td>
    <td></td>
    <td>27.2</td>
    <td>18.9</td>
    <td>9.9</td>
    <td>2.4</td>
    <td>2.3</td>
  </tr>
  <tr>
    <td>RAG_{Dense}</td>
    <td></td>
    <td>33.1</td>
    <td>22.6</td>
    <td>12.8</td>
    <td>4.8</td>
    <td>0.6</td>
  </tr>
  <tr>
    <td>Current-File</td>
    <td></td>
    <td>25.2</td>
    <td>28.4</td>
    <td>16.3</td>
    <td>14.9</td>
    <td>8.5</td>
  </tr>
  <tr>
    <td>RAG_{BM25}</td>
    <td></td>
    <td>22.6</td>
    <td>20.5</td>
    <td>14.9</td>
    <td>6.0</td>
    <td>5.1</td>
  </tr>
  <tr>
    <td>RAG_{Dense}</td>
    <td></td>
    <td>24.3</td>
    <td>21.1</td>
    <td>13.5</td>
    <td>5.4</td>
    <td>2.8</td>
  </tr>
  <tr>
    <td>Current-File</td>
    <td></td>
    <td>30.5</td>
    <td>23.2</td>
    <td>14.9</td>
    <td>8.9</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td>RAG_{BM25}</td>
    <td></td>
    <td>39.3</td>
    <td>32.6</td>
    <td>22.7</td>
    <td>16.7</td>
    <td>14.8</td>
  </tr>
  <tr>
    <td>RAG_{Dense}</td>
    <td></td>
    <td>44.6</td>
    <td>36.3</td>
    <td>16.3</td>
    <td>12.5</td>
    <td>9.1</td>
  </tr>
  <tr>
    <td>Current-File</td>
    <td></td>
    <td>39.3</td>
    <td>33.7</td>
    <td>24.1</td>
    <td>16.1</td>
    <td>10.2</td>
  </tr>
</table>

Table 11: Pass Rate by Number of dependencies, binned by the number of unique dependencies. The bins are distributed as follows: 0 (305 instances), 1 (190 instances), 2 (141 instances), [3,4] (168 instances), and [5,35] (176 instances).

Claude-3.5 shows an inverse trend, where successful generations tend to be longer than failed ones (e.g., RAG_{Dense}: Pass = 25.5 vs. Fail = 1.7). This suggests that Claude-3.5 generates and utilizes helper functions effectively, which may contribute to its success.

A.6.1 Detailed context complexity’s impact on Performance

Table 11 demonstrates the relationship between pass@1 and the detailed context complexity (number of unique dependencies) present in the context. Across all models, performance declines as the
You are an exceptionally intelligent coding assistant that understands detailed code examples and follows user instructions. You will be provided a function signature and the documentation, and your task is to complete the function body. The target function is in a large project, so it may use the classes or functions defined in the project to complete the target function.

These are a list of files that contain the related classes and functions for you to refer to.
## File path: sklearn/svm/_classes.py
python
def fit(self, X, y=None, sample_weight=None):
...
...

## File path: sklearn/cluster/_kmeans.py
python
class MiniBatchKMeans:
    def fit(self, X, y=None, sample_weight=None):
        ...
        # Compute the centroids on X.
        X = validate_data(self, ...)
        return ...

class _BaseKMeans:
    def score(self, X, y=None, sample_weight=None):
    ...
    def fit_transform(self, X, y=None, sample_weight=None):
    ...
    def fit_predict(self, X, y=None, sample_weight=None):
    ...
...

This is the file that contains the target function.
## File path: sklearn/cluster/_kmeans.py
### Target function to complete
python
def fit(self, X, y=None, sample_weight=None):
    """Compute k-means clustering.
    Parameters
    ----------
    X : {array-like, sparse matrix} of shape (n_samples, n_features)
        Training instances to cluster. It must be noted that the data will be converted to C ordering, which will cause a memory copy if the given data is not C-contiguous. If a sparse matrix is passed, a copy will be made if it's not in CSR format.
    Returns
    -------
    self : object
        Fitted estimator.
    """
You must only complete the target function 'KMeans.fit' and do not generate any comments or explanation or other functions.
You must not leave the target function as 'not implemented' or pass.
...

Figure 8: Prompt Example for KMeans.fit in the file "sklearn/cluster/_kmeans.py" from REPOCOD.

number of dependencies increases, highlighting the growing challenge posed by more complex contextual requirements. For instance, DeepSeek-v2.5 with the BM25 retriever achieves a 31.8% pass rate when no dependencies are involved, but this drops to 5.7% when the number of dependencies reaches [5,35].

Notably, retrieval strategies significantly impact performance; models using the Current-File retrieval generally perform better for moderate dependency counts (e.g., DeepSeek-v2.5 achieving 17.3% at [3,4]), suggesting that partial in-file context remains beneficial before complexity outweighs retrieval effectiveness. However, even strong retrieval strategies cannot fully mitigate the difficulty of handling a high number of dependencies, reinforcing the intuition that increasing context complexity directly correlates with reduced task success.

A.7 Case Studies

Prompt Example. Figure 8 shows an example of a REPOCOD prompt, consisting of the system prompt, context information retrieved by RAG_{BM25}, and the target function details, with specific content replaced by ‘…’.

Passed Example. Figure 9 shows an example of a correctly generated solution from REPOCOD generated by GPT-4o under a RAG_{Dense} retrieval setting. This example passes all test cases and is considered correct. Upon inspection, the generated code snippet is identical to the canonical solution except for the text content in the exception message.

A.7.1 Failed Examples

Figure 10 shows an example of an incorrect solution generated by GPT-4o under RAG_{BM25} retrieval setting. The target function _set_order is designed to change the order of training data (x) and target values (y). The canonical solution validates order is one of [None, "C", "F"], raising a ValueError if an invalid value is provided. However, for the generated code, it does not validate order, which might cause unexpected behavior when order is incorrectly specified.

In addition, the canonical solution maintains proper sparse matrix handling by explicitly checking x and y before conversion, while the generated code can lead to errors if y is sparse, since np.ascontiguousarray() and np.asfortranarray() do not support sparse matrices.

Figure 11 shows another example of an incorrectly generated solution by GPT-4o under RAG_{BM25} retrieval setting. The canonical solution incorporates repository-level context; however, the solution generated by GPT-4o fails to match the functionality of the canonical solution. Specifically, the incorrect solution miscomputes the posterior mean by using an incorrect formulation for the covariance of the latent variables.

A.8 Uniqueness and Overlap of Correct Generations

Commercial LLMs with RAG_{BM25}. Figure 12a presents the UpSet plot comparing the performance of four commercial models using contexts from RAG_{BM25} retrieval, the largest overlap (more than 40) is shared by all models, indicating a significant
def false_alarm_probability(
    self,
    power,
    method='baluev',
    samples_per_peak=5,
    nyquist_factor=5,
    minimum_frequency=None,
    maximum_frequency=None,
    method_kwds=None):
    """
    False alarm probability of periodogram maxima under the null hypothesis.

    This gives an estimate of the false alarm probability given the height of the largest peak in the periodogram, based on the null hypothesis of non-varying data with Gaussian noise.

    Parameters
    ----------
    power : array-like
        The periodogram value.
    method : {'baluev', 'davies', 'naive', 'bootstrap'}, optional
        The estimation method to use.
    maximum_frequency : float
        The maximum frequency of the periodogram.
    method_kwds : dict, optional
        Additional method-specific keywords.

    Returns
    -------
    false_alarm_probability : np.ndarray
        The false alarm probability

    Notes
    -----
    The true probability distribution for the largest peak cannot be determined analytically, so each method here provides an approximation to the distribution. The available methods are:
    - "baluev" (default): the upper-limit to the alias-free probability using the approach of Baluev (2008) [1].
    - "davies": the Davies upper bound from Baluev (2008) [1].
    - "naive": the approximate probability based on an estimated effective number of independent frequencies.
    - "bootstrap": the approximate probability based on bootstrap resamplings of the input data.
    Note also that for normalization='psd', the distribution can only be computed for periodograms constructed with errors specified.

    See Also
    --------
    distribution
    false_alarm_level

    References
    ----------
    [1] Baluev, R.V. MNRAS 385, 1279 (2008)
    """
    if self.nitems != 1:
        raise NotImplementedError(
            "false alarm probability is not implemented for "
            "multitem periodograms."
        )
    if not (self.fit_mean or self.center_data):
        raise NotImplementedError(
            "false alarm probability is implemented "
            "only for periodograms of centered data."
        )
    fmin, fmax = self.autofrequency(
        samples_per_peak=samples_per_peak,
        nyquist_factor=nyquist_factor,
        minimum_frequency=minimum_frequency,
        maximum_frequency=maximum_frequency,
        return_freq_limits=True,
    )
    return_statistics.false_alarm_probability(
        power=power,
        fmax=fmax,
        t=self.trel,
        y=self.y,
        dy=self.dy,
        normalization=self.normalization,
        method=method,
        method_kwds=method_kwds,
    )

if self.nitems != 1:
    raise NotImplementedError(
        "false alarm level is not implemented for "
        "multitem periodograms."
    )
if not (self.fit_mean or self.center_data):
    raise NotImplementedError(
        "false alarm level is implemented "
        "only for periodograms of centered data."
    )
fmin, fmax = self.autofrequency(
    samples_per_peak=samples_per_peak,
    nyquist_factor=nyquist_factor,
    minimum_frequency=minimum_frequency,
    maximum_frequency=maximum_frequency,
    return_freq_limits=True,
)
return_statistics.false_alarm_probability(
    power=power,
    fmax=fmax,
    t=self.trel,
    y=self.y,
    dy=self.dy,
    normalization=self.normalization,
    method=method,
    method_kwds=method_kwds,
)

LLM Solution

Figure 9: An example of the correct output generated by LLMs, for function “false_alarm_probability” in the file “astropy/timeseries/periodograms/lombscargle/core.py” from REPOCOD.

degree of commonality. GPT-4o has the highest number of unique cases (over 60), suggesting its ability to capture distinct results in this scenario.

Commercial LLMs with Current-File. Figure 12b shows the Current File retrieval setting result. The overlap between all models increases greatly compared to the RAGBM25 retrieval (to over 75 cases), demonstrating stronger alignment across models with this retrieval setting. In addition, the unique solvable problem by GPT-4o reduces greatly, to a similar level compared to Claude 3.5 Sonnet and DeepSeek-V2.5.

A.9 Impact of Retrieval Methods on Pass@1 across Varying Context Complexities

Figure 13a and Figure 13b illustrate the pass@1 for instances across various context complexities under different retrieval settings: RAGBM25 and current file. In both settings, LLM performance tends to decrease as context complexity rises, progressing from self-contained to file-level and eventually to repository-level contexts. This trend aligns with the insights discussed in Section 4.2.

An interesting finding is that GPT-4-o achieves a higher pass@1 rate for instances under repository-level context complexity compared to file-level instances. This observation suggests two potential insights: (1) specific retrieval methods, like RAGBM25, may enhance performance on repository-level task instances; (2) higher context complexity does not necessarily result in reduced LLM performance.

A.10 REPOCOD-LITE

Given the large number of instances and the high evaluation cost of REPOCOD, we introduce REPOCOD-LITE—a 200 sample subset of the most challenging problems in REPOCOD—enabling faster and more targeted evaluation. We define difficulty based on the product of the prompt
Target Function Description:

def _set_order
    """Change the order of X and y if necessary.
    Parameters
    ----------
    X : {array-like, sparse matrix} of shape (n_samples, n_features)
        Training data.
    y : ndarray of shape (n_samples,)
        Target values.
    order : {None, 'C', 'F'}
        If 'C' dense arrays are returned as C-ordered,
        sparse matrices in csr format.
        If 'F', dense arrays are return as F-ordered,
        sparse matrices in csc format.
    Returns
    -------
    X : {array-like, sparse matrix} of shape (n_samples, n_features)
        Training data with guaranteed order.
    y : ndarray of shape (n_samples,)
        Target values with guaranteed order.
    """
Instance path:
'sklearn/linear_model/_coordinate_descent.py'

Canonical Solution

if order not in [None, 'C', 'F']:
    raise ValueError('Unknown value for order.')
    Got {} instead of None, 'C' or 'F'.format(order))
sparse_X = sparse.issparse(X)
sparse_y = sparse.issparse(y)
if order is not None:
    sparse_format = "csc" if order == "F" else "csr"
    if sparse_X:
        X = X.asformat(sparse_format, copy=False)
    else:
        X = np.asarray(X, order=order)
    if sparse_y:
        y = y.asformat(sparse_format)
    else:
        y = np.asarray(y, order=order)
return X, y

LLM Solution

def _set_order(self):
    if order == "C":
        X = X.tocsr()
    elif order == "F":
        X = X.tocsc()
    else:
        X = np.ascontiguousarray(X, dtype=X.dtype) if order == "C"
        X = np.asfortranarray(X, dtype=X.dtype) if order == "F"
    y = np.ascontiguousarray(y, dtype=y.dtype) if order == "C" else
    y = np.asfortranarray(y, dtype=y.dtype)
return X, y

Figure 10: A failed case study, for function '_set_order' in the file 'sklearn/linear_model/_coordinate_descent.py' from REPOCOD.

Target Function Description:

def transform(self, X):
    """Apply dimensionality reduction to X using the model.
    Compute the expected mean of the latent variables.
    See Barber, 21.2.33 (or Bishop, 12.66).
    Parameters
    ----------
    X : array-like of shape (n_samples, n_features) Training data.
    Returns
    -------
    X_new : ndarray of shape (n_samples, n_components).
        The latent variables of X.
    """
Repository-Level Contexts:
check_is_fitted sklearn/utils/validation.py
validate_data sklearn/utils/validation.py

Canonical Solution

def transform(self, X):
    check_is_fitted(self)
    X = validate_data(self, X, reset=False)
    # Compute the expected mean of the latent variables
    precision = np.linalg.inv(self.noise_variance_
        * np.eye(self.components_.shape[0]) + self.components_.T @ self.components_)
    X_transformed = X - self.mean_
    Ws1 = self.components_ @ self.noise_variance_
    cov_z = np.dot(Ws1, Ws1.T)
    tmp = np.dot(X_transformed, Ws1.T)
    X_transformed = np.dot(tmp, cov_z)
    return X_transformed

LLM Solution

def transform(self):
    check_is_fitted(self)
    X = validate_data(self, X, dtype=[np.float64, np.float32], reset=False)
    # Compute the expected mean of the latent variables
    precision = np.linalg.inv(self.noise_variance_
        * np.eye(self.components_.shape[0]) + self.components_.T @ self.components_)
    X_new = (X - self.mean_) @ self.components_.T @ precision
    return X_new

Figure 11: An example of the wrong output generated by LLMs, for function “transform” in the file “sklearn/decomposition/_factor_analysis.py” from REPOCOD.

<table>
  <tr>
    <th>Models</th>
    <th>RAG<sub>RMSE</sub></th>
    <th>RAG<sub>Dense</sub></th>
    <th>Current-File</th>
  </tr>
  <tr>
    <td>CodeLlama-7B</td>
    <td>1.0</td>
    <td>1.5</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>CodeLlama-34B</td>
    <td>2.5</td>
    <td>2.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>DeepSeek-6.7B</td>
    <td>2.5</td>
    <td>2.0</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>DeepSeek-33B</td>
    <td>4.0</td>
    <td>3.5</td>
    <td>2.5</td>
  </tr>
  <tr>
    <td>OpenCodeInterpreter-6.7B</td>
    <td>1.0</td>
    <td>5.0</td>
    <td>1.5</td>
  </tr>
  <tr>
    <td>OpenCodeInterpreter-33B</td>
    <td>4.0</td>
    <td>6.0</td>
    <td>5.5</td>
  </tr>
  <tr>
    <td>Claude-3.5</td>
    <td>2.5</td>
    <td>3.0</td>
    <td>2.5</td>
  </tr>
  <tr>
    <td>DeepSeek-v2.5</td>
    <td>4.0</td>
    <td>7.5</td>
    <td>5.5</td>
  </tr>
  <tr>
    <td>GPT-4o-Mini</td>
    <td>8.0</td>
    <td>6.5</td>
    <td>2.5</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>9.0</td>
    <td>8.5</td>
    <td>4.5</td>
  </tr>
</table>

Table 12: Pass@1(%) of SOTA LLMs on REPOCOD-LITE.

A.11 OpenHands’s Result on REPOCOD-LITE

To further explore the performance of advanced code generation approaches on REPOCOD and address the applicability of our benchmark to code length and canonical solution length (measured in lines of code), selecting samples with the highest scores. REPOCOD-LITE maintains the original benchmark’s diversity by including problems from all three categories: self-contained, file-level, and repo-level, with 66, 67, and 67 samples, respectively. This subset enables efficient model evaluation while preserving the complexity of the coding tasks found in the full benchmark. We show the details of REPOCOD-LITE in Table 13. The performance of models on REPOCOD-LITE is shown in Table 12. The SOTA models’ performance is consistently lower on REPOCOD-LITE compared to that on REPOCOD.
(a) Result for RAGBM25.

(b) Result for Current-File.

Figure 12: Correct generation result’s relationship under different retrieval settings.

(a) Pass@1(%) for LLMs based on RAGBM25 retrieval.

(b) Pass@1(%) for LLMs based on current file retrieval.

Figure 13: Pass@1 under different context complexity.

agents, we conducted additional experiments using OpenHands, a leading open-source agent framework. For this evaluation, we utilized REPOCOD-LITE

In our experiments, OpenHands, when paired with Claude-3.5 Sonnet as its underlying model, successfully solved 11 out of 200 tasks (5.5% Pass@1) in a single generation round. This result represents an improvement over the performance of Claude-3.5 Sonnet with standard Retrieval-Augmented Generation (RAG) approaches on REPOCOD-LITE.

However, despite this improvement, OpenHands with Claude-3.5 Sonnet still underperformed the best RAG-based result (GPT-4o achieving 9.0% Pass@1 with RAGBM25).

These preliminary findings suggest that while current agentic approaches like OpenHands can offer incremental performance gains on complex, repository-level code generation tasks, they still face significant challenges on REPOCOD. The difficulty of the REPOCOD benchmark, underscores the substantial room for improvement in both LLM capabilities and agentic framework strategies for real-world code generation. We believe these evaluations provide a solid foundation and demonstrate REPOCOD’s utility for benchmarking future advancements in code agents.

A.12 Potential Risk and Impact

This work aims to construct an evaluation benchmark for code generation using real-world, repository-level context. We do not foresee any significant risks associated with the misuse of this approach.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="4">Repository-level</th>
    <th colspan="4">File-level</th>
    <th colspan="4">Self-contained</th>
    <th colspan="4">Total</th>
  </tr>
  <tr>
    <th>#NL</th>
    <th>#GT Cyclo.</th>
    <th>#Funcs.</th>
    <th>#NL</th>
    <th>#GT Cyclo.</th>
    <th>#Funcs.</th>
    <th>#NL</th>
    <th>#GT Cyclo.</th>
    <th>#Funcs.</th>
    <th>#NL</th>
    <th>#GT Cyclo.</th>
    <th>#Funcs.</th>
  </tr>
  <tr>
    <td>astropy</td>
    <td>321.7</td>
    <td>1,104.0</td>
    <td>25.3</td>
    <td>3</td>
    <td>678.8</td>
    <td>1,107.2</td>
    <td>20.0</td>
    <td>4</td>
    <td>485.3</td>
    <td>618.8</td>
    <td>14.0</td>
    <td>9</td>
    <td>503.0</td>
    <td>831.9</td>
    <td>17.6</td>
    <td>16</td>
  </tr>
  <tr>
    <td>datasets</td>
    <td>1,432.8</td>
    <td>832.8</td>
    <td>24.0</td>
    <td>4</td>
    <td>1,341.0</td>
    <td>526.0</td>
    <td>14.0</td>
    <td>1</td>
    <td>1,211.0</td>
    <td>318.0</td>
    <td>8.0</td>
    <td>1</td>
    <td>1,378.8</td>
    <td>695.8</td>
    <td>20.7</td>
    <td>6</td>
  </tr>
  <tr>
    <td>flask</td>
    <td>661.0</td>
    <td>424.0</td>
    <td>12.0</td>
    <td>3</td>
    <td>595.0</td>
    <td>315.0</td>
    <td>12.0</td>
    <td>1</td>
    <td>452.0</td>
    <td>691.0</td>
    <td>16.0</td>
    <td>1</td>
    <td>569.3</td>
    <td>693.3</td>
    <td>13.3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>more-itertools</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0</td>
    <td>503.0</td>
    <td>815.0</td>
    <td>11.0</td>
    <td>1</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0</td>
    <td>503.0</td>
    <td>815.0</td>
    <td>11.0</td>
    <td>1</td>
  </tr>
  <tr>
    <td>plotly.py</td>
    <td>1,806.0</td>
    <td>3,393.0</td>
    <td>132.0</td>
    <td>1</td>
    <td>1,574.4</td>
    <td>1,191.8</td>
    <td>30.2</td>
    <td>14</td>
    <td>1,933.0</td>
    <td>1,464.2</td>
    <td>56.9</td>
    <td>23</td>
    <td>1,797.5</td>
    <td>1,414.6</td>
    <td>49.0</td>
    <td>38</td>
  </tr>
  <tr>
    <td>pylint</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0</td>
    <td>385.0</td>
    <td>762.0</td>
    <td>16.0</td>
    <td>1</td>
    <td>385.0</td>
    <td>762.0</td>
    <td>16.0</td>
    <td>1</td>
  </tr>
  <tr>
    <td>scikit-learn</td>
    <td>420.3</td>
    <td>886.3</td>
    <td>15.5</td>
    <td>51</td>
    <td>559.8</td>
    <td>727.4</td>
    <td>8.0</td>
    <td>5</td>
    <td>493.6</td>
    <td>713.0</td>
    <td>15.2</td>
    <td>5</td>
    <td>437.8</td>
    <td>859.1</td>
    <td>14.8</td>
    <td>61</td>
  </tr>
  <tr>
    <td>seaborn</td>
    <td>404.3</td>
    <td>750.0</td>
    <td>18.3</td>
    <td>3</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0</td>
    <td>363.8</td>
    <td>678.2</td>
    <td>22.5</td>
    <td>4</td>
    <td>381.1</td>
    <td>709.0</td>
    <td>20.7</td>
    <td>7</td>
  </tr>
  <tr>
    <td>sphinx</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0</td>
    <td>418.0</td>
    <td>946.0</td>
    <td>17.0</td>
    <td>1</td>
    <td>418.0</td>
    <td>946.0</td>
    <td>17.0</td>
    <td>1</td>
  </tr>
  <tr>
    <td>sympy</td>
    <td>978.0</td>
    <td>424.8</td>
    <td>10.8</td>
    <td>4</td>
    <td>1,135.2</td>
    <td>1,155.1</td>
    <td>34.5</td>
    <td>24</td>
    <td>810.2</td>
    <td>496.1</td>
    <td>14.4</td>
    <td>16</td>
    <td>1,002.7</td>
    <td>849.1</td>
    <td>25.0</td>
    <td>44</td>
  </tr>
  <tr>
    <td>xarray</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0</td>
    <td>1,562.5</td>
    <td>695.1</td>
    <td>18.5</td>
    <td>17</td>
    <td>1,158.8</td>
    <td>367.6</td>
    <td>13.0</td>
    <td>5</td>
    <td>1,470.7</td>
    <td>620.6</td>
    <td>17.2</td>
    <td>22</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>533.2</th>
    <th>889.7</th>
    <th>17.9</th>
    <th>67</th>
    <th>1,250.6</th>
    <th>987.3</th>
    <th>25.7</th>
    <th>67</th>
    <th>1,120.8</th>
    <th>879.0</th>
    <th>29.6</th>
    <th>66</th>
    <th>967.4</th>
    <th>918.9</th>
    <th>24.39</th>
    <th>200</th>
  </tr>
</table>

Table 13: Basic statistics of REPOCOD_LITE, with details broken down by each collected repository.

Metareview score: 4
Metareview: This paper introduces REPOCOD, a benchmark comprising 980 whole-function Python generation tasks derived from real-world repositories. It aims to evaluate large language models (LLMs) on code generation tasks requiring repository-level understanding. The benchmark features a novel test collection pipeline and a rigorous evaluation methodology using both retrieval-augmented generation (RAG) and traditional code generation techniques. Three reviewers commend the dataset's complexity, thoughtful experimental design, and clarity of writing. Reviewers agree that REPOCOD fills a gap by focusing on function implementation within realistic, interdependent codebases.

Reviewer zN2p highlights the benchmark’s strengths, including its complexity and thorough evaluation across 10 models, while raising concerns about the small model sample size, lack of human evaluation, and underdeveloped discussion of REPOCOD-LITE. The authors effectively address these in the rebuttal by adding Maintainability Index (MI) scores, expanding the evaluation to additional models, and reporting preliminary human evaluation results. Reviewer YfcA praises the paper’s clarity, data collection pipeline, and analysis, though notes that the number of repositories is modest and recommends discussing version-aware code generation. The authors accept this and commit to adding discussion of relevant related work. Reviewer QdQL questions the motivation for increasing benchmark complexity and the realism of single-function generation. In response, the authors argue that function generation is a core software engineering task distinct from issue fixing, and they provide supporting evidence via new experiments, including on OpenHands, to show agents also struggle with REPOCOD-LITE.

The paper significantly contributes to benchmarking LLMs in realistic software engineering tasks. The authors have been highly responsive in addressing weaknesses through clarifications, empirical extensions, and plans for revision. I recommend acceptance to main conference.
Summary of reasons to publish: - Substantial Benchmark Contribution: REPOCOD fills a gap in current evaluation datasets by targeting complex, repository-level function implementation tasks with realistic dependencies and long outputs, offering a challenging benchmark for current and future LLMs.

- Rigorous Evaluation and Analysis: The paper provides a comprehensive analysis across multiple state-of-the-art models, includes both functional correctness and maintainability metrics, and introduces an efficient test collection pipeline to reduce evaluation overhead.

- Responsive Revisions and Empirical Depth: Authors addressed reviewer concerns with new experiments (e.g., human evaluations, agentic baselines via OpenHands), enhanced documentation of REPOCOD-LITE, and expanded related work discussion, significantly strengthening the submission.