FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
Authors: Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, Scarlett Li
Accepted at: acl25

FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation

Wei Li*†, Xin Zhang*‡‡, Zhongxin Guo*, Shaoguang Mao*
Wen Luo*, Guangyue Peng*, Yangyu Huang*, Houfeng Wang*†, Scarlett Li*
* State Key Laboratory of Multimedia Information Processing,
School of Computer Science, Peking University
* Microsoft Research Asia
weili22@stu.pku.edu.cn
xinzhang3@microsoft.com, wanghf@pku.edu.cn

Abstract

Implementing new features in repository-level codebases is a crucial application of code generation models. However, current benchmarks lack a dedicated evaluation framework for this capability. To fill this gap, we introduce FEA-Bench, a benchmark designed to assess the ability of large language models (LLMs) to perform incremental development within code repositories. We collect pull requests from 83 GitHub repositories and use rule-based and intent-based filtering to construct task instances focused on new feature development. Each task instance containing code changes is paired with relevant unit test files to ensure that the solution can be verified. The feature implementation requires LLMs to simultaneously possess code completion capabilities for new components and code editing abilities for other relevant parts in the code repository, providing a more comprehensive evaluation method of LLMs’ automated software engineering capabilities. Experimental results show that LLMs perform significantly worse in the FEA-Bench, highlighting considerable challenges in such repository-level incremental code development. Our code will soon be publicly available at https://github.com/microsoft/FEA-Bench.

1 Introduction

The remarkable text generation capabilities of large language models (LLMs) (Achiam et al., 2023) have extended their impact into the domain of code generation (Xu et al., 2022), which has led to the emergence of developer assistants such as Copilot, Cursor, Devin, etc. An important research topic is evaluating the effectiveness of LLMs in generating code across diverse scenarios. Many of the existing benchmarks focus on evaluating standalone programming problems, such as HumanEval, MBPP, and LiveCodeBench (Chen et al., 2021; Austin et al., 2021; Jain et al., 2024). These benchmarks offer little insight into the challenges developers face in real-world projects, where codebases are composed of multiple interconnected files. In such projects, modifications in one part of the code often necessitate corresponding edits elsewhere. This type of collaborative, large-scale development is referred to as repository-level code development.

In the realm of repository-level code generation, much of the current evaluation effort is centered around code completion (Li et al., 2024b; Yang et al., 2024a). Code completion refers to generating correct code snippets at specified locations within a given code context. However, this task is inherently limited—it typically targets localized generation and does not account for broader implications beyond the scope of completion.

Recent advancements in the capabilities of LLMs have expanded their potential role from merely suggesting code snippets to managing the full lifecycle of repository development. A prominent benchmark in this domain is SWE-bench (Jimenez et al., 2024), which evaluates LLMs on resolving issues, primarily focusing on bug fixes within repositories. In practice, as shown in Figure 1, a more critical aspect of software engineering is the launch of new features, which often entails introducing new functions or even entire files

![A diagram showing the proposed FEA-Bench workflow: Initialize, Demand, Launch New Feature, Software (Repo), Fix Issues, SWE-bench, FEA-bench](page_1012_370_393_246.png)

Figure 1: The proposed FEA-Bench aims to evaluate incremental repository development, while SWE-bench (Jimenez et al., 2024) focuses on repairing issues.
into the repository. The continuous implementation of new features drives software growth and is a key focus of automated software engineering. We define such tasks as repository-level incremental code development. In this work, to bridge the gap in benchmarks for this domain, we construct a dataset derived from pull requests in GitHub repositories that specifically focus on adding new components, with the overarching goal of implementing new features. Each task instance in our dataset is paired with its corresponding unit test files, culminating in the creation of the Feature Implementation Benchmark (FEA-Bench), which comprises 1,401 task instances sourced from 83 diverse GitHub repositories.

Statistically, our dataset exhibits characteristics that significantly differ from the bug-fix-oriented SWE-bench. Task instances in FEA-Bench require the implementation of new functions and classes in Python and involve substantially longer code generation compared to SWE-Bench. Experimental results demonstrate that current LLMs perform poorly on the proposed benchmark. According to our execution-based metrics, the best-performing LLM, DeepSeek-R1, successfully resolves only about 10% of the task instances. The key contributions of this paper are as follows:

• We introduce the task of repository-level incremental code development, addressing a critical challenge in real-world software engineering where the continuous implementation of new features is essential for sustaining software growth.

• We construct the first benchmark to evaluate repository-level incremental code development. By employing parsing and other filtering methods, we constructed a dataset composed of feature implementation tasks, offering execution-based evaluation.

• Using an automated pipeline, we scale the test data to include 83 diverse code repositories, ensuring high diversity. We will publicly release our data collection and evaluation codebase, allowing FEA-Bench to be continuously updated and expanded.

2 Related Work

2.1 Code Large Language Models

Large language models (LLMs) have revolutionized software engineering by enabling code generation, debugging, and translation capabilities (Pan et al., 2024; Li et al., 2023a; Joshi et al., 2023; Shi et al., 2024). Large-scale pre-trained LLMs such as GPT-4 (Achiam et al., 2023), CodeLlama (Roziere et al., 2023), DeepSeek-Coder (Guo et al., 2024; Zhu et al., 2024), and Qwen2.5-Coder (Hui et al., 2024) have demonstrated proficiency in generating functional code across multiple programming languages. Recent advancements also include instruction-tuned models like Starcoder (Li et al., 2023b), WizardCoder (Luo et al., 2023), WaveCoder (Yu et al., 2024b), Magocoder (Wei et al., 2024) and EpiCoder (Wang et al., 2025). With these advancements, code LLMs are poised to further revolutionize how developers interact with code, promising increased efficiency in software creation. Additionally, the integration of agents (Luo et al., 2025) has further enhanced the performance of LLMs in software engineering tasks (Yang et al., 2024b; Xia et al., 2024; Zhang et al., 2024). In this paper, we further evaluate the performance of current LLMs in the incremental code development scenarios at the repository level. This investigation aims to drive the research on code LLMs toward addressing more intricate software engineering challenges, thereby advancing the capabilities of these models in handling sophisticated development tasks.

2.2 Code Generation Benchmarks

Recent code generation benchmarks, such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), have primarily focused on synthesizing standalone functions or scripts from natural language, while subsequent efforts like APPS (Hendrycks et al., 2021), EvalPlus (Liu et al., 2023a), CoderEval (Yu et al., 2024a), ClassEval (Du et al., 2023), BigCodeBench (Zhuo et al., 2024), and FullStackBench (Liu et al., 2024b) have expanded evaluation to more complex scenarios. However, these benchmarks largely overlook the repository-level challenges of real-world software development, a gap addressed by recent works (Bairi et al., 2024; Zhang et al., 2023). Because of the wild applications of auto code completion tools like Github Copilot (Dakhel et al., 2023), most repository-level code generation benchmarks aim to evaluate the code completion capabilities of LLMs (Liu et al., 2023b). Repository-level code completion aims to generate code for incomplete code snippets within a repository (Wang et al., 2024). Benchmarks such as DevEval (Li et al.,
2024b), EvoCodeBench (Li et al., 2024a), Codev-Bench (Pan et al., 2024), and ExecRepoBench (Yang et al., 2024a) evaluate this capability. However, code completion data constructed by removing a single line or function body can suffer from future context leakage issues (Zheng et al., 2024).

In practice, autonomous development of a code repository should include all edits to the code, rather than simply completing the specified code under perfect context. SWE-bench (Jimenez et al., 2024) focuses on repairing repositories’ issues by revising existing programs. This highlights the need for models to effectively modify and integrate changes within the repository. Our work fills another gap of incremental repository-level development by introducing a benchmark that evaluates LLMs on the implementation of new features, further bridging the divide between code completion and real-world software engineering.

3 Benchmark Construction

3.1 Overview
The task instances in FEA-Bench are constructed based on existing pull request (PR) data from GitHub. As illustrated in Figure 2, each task instance contains the following elements:

① Feature Request Content of the pull request and corresponding issues (if any) provide essential information regarding the new feature or functionality to be developed.

② Definition of New Components This includes the signatures and documentation of newly added functions and classes. The name of the new component must be consistent with that in PR, which is the prerequisite for completing the task, because unit tests are written based on the specified name.

③ Environment Setup This includes the relevant information for the repository and specifies the base commit of the code repository for each task instance. Besides, the configurations of the building execution environment are also included.

④ Patch It describes the changes made to the code in the repository and can be processed by the unidiff standard library1. Additionally, the changes can be applied to the repository using the git apply tool. A patch can be divided into a test patch and a gold patch; the former pertains to changes in test codes, while the latter involves the other changes affecting the software itself.

<table>
  <tr>
    <th>File</th>
    <th>Namespace</th>
    <th>Signature</th>
  </tr>
  <tr>
    <td>hf_file_system.py</td>
    <td>HFFileSystem.url</td>
    <td>def url(self, path: str)</td>
  </tr>
  <tr>
    <td>hf_file_system.py</td>
    <td>HFFileSystemFile.url</td>
    <td>def url(self)</td>
  </tr>
  <tr>
    <td>hf_file_system.py</td>
    <td>HFFileSystemStreamFile.url</td>
    <td>def url(self)</td>
  </tr>
</table>

Figure 2: An example of the task instances from the FEA-Bench. During the inference of LLMs, the first two items: feature request and new components are considered as known information. The environment setup serves as a prerequisite for creating the testbed and environment. Python file patches and unit tests are used as labels and evaluation metrics and should not be leaked during the inference of LLMs.

⑤ Unit Test The correctness of the code changes is verified based on the result of running these tests. We get the ground truth status by actually running pytest before and after applying the gold patch.

Our data collection pipeline is developed based on SWE-bench. As shown in Figure 3, the proposed collection pipeline ensures data diversity and comprehensiveness, enabling a robust evaluation of LLMs’ capabilities in implementing new features at the repository level. By making our data collection and evaluation codes publicly available, we aim to facilitate continuous updates and the creation of new versions of FEA-Bench. In the rest of this section, we will discuss the construction and the characteristics of FEA-Bench.

3.2 Repository Collection
To determine the scope of GitHub repositories for data collection, we initially focus on the GitHub repositories corresponding to Python packages listed on the Top PyPI website2. Packages appearing on this list are generally influential Python soft-

1https://github.com/matiasb/python-unidiff
2https://hugovk.github.io/top-pypi-packages/
Figure 3: The data collection pipeline for the FEA-Bench. First, determine the scope of GitHub repositories from which task instances will be collected. Next, gather pull requests as task instances and apply filtering criteria to select instances that meet the purpose of adding new features. Finally, use the included test files to execute unit tests, ensuring that only task instances with reproducible test results are included in FEA-Bench. For a more detailed construction process, please refer to Appendix A.2.

ware with high data quality, and most repositories use a unified pytest format for testing, which facilitates later execution. This leaderboard contains 8,000 Python packages. We obtain approximately 600 repositories that meet the criteria of having a license and more than 1,000 pull requests by applying a filtering process.

Fast Validation Except for repositories already included in SWE-bench, the remaining repositories do not have customized installation procedures or testing methods. While environment configurations vary among different Python packages, pip offers a unified installation approach3. And we adopt pytest as the default method for unit testing. Based on these settings, for each repository, we extract the first 20 pull requests that include changes to test files and observe the unit test status. Repositories that have at least one task instance where the unit tests passed with the default configuration are retained, as they possess the potential to generate task instances that meet our criteria.

After this fast validation process, in addition to the 18 Python packages included in the SWE-bench dataset, another 101 Python packages are identified as sources for extracting repository-level data for our benchmark.

3.3 Task Collection and Filtering

Based on existing repository data, we crawl all pull requests and consider those that include changes to test files as possible task instances. Given our focus on the incremental feature development task, we introduce the following steps to obtain the final task instances for FEA-Bench through filtering:

Extraction of new components For each task instance candidate, we perform parsing on all Python scripts involved in the gold patch. We compare the state before and after applying the patch to identify newly added components, including classes and functions, and extract their signatures and doc-strings as metadata.

Filtering based on new components We retain only those task instances that contain at least one new component. To ensure that implementing new features is the primary purpose of the pull request, we further restrict the new components to occupy more than 25% of all edited lines in gold patch. This threshold is set relatively low because code changes often need to include modifications to other related code in addition to the new components themselves.

Intent-based filtering The pull requests filtered using the above rule-based approach still include some that are not primarily aimed at feature implementation. Therefore, we use GPT-4o to classify the intent based on the pull request description. Only pull requests classified as "new feature" are retained.

Verification by running unit tests For each instance, we first set up the environment and testbed. We then apply the test patch and run the unit test files involved in the test patch. Theoretically, some unit tests should fail at this stage. After applying the gold patch, we rerun the same unit tests. If the configuration is correct, all unit tests should pass. Unlike SWE-bench, we do not impose restrictions on whether ImportError or AttributeError occurs before applying the gold patch, as these errors are almost inevitable before new components are implemented. Instead, to ensure data quality, we exclude samples where tests remain in the failed status after applying the gold patch. The status of each test function before and after applying the gold patch is recorded.

After task collection and filtering, 83 out of 119 repositories collected by the method in Section 3.2 produce 1,401 task instances for new feature implementation.

3pip install -e .
<table>
  <tr>
    <th></th>
    <th colspan="2">FEA-Bench</th>
    <th colspan="2">SWE-bench</th>
  </tr>
  <tr>
    <th></th>
    <th>Full</th>
    <th>Lite</th>
    <th>Full</th>
    <th>Verified</th>
  </tr>
  <tr>
    <td>[Repositories]</td>
    <td>83</td>
    <td>48</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>[Tasks]</td>
    <td>1401</td>
    <td>200</td>
    <td>2294</td>
    <td>500</td>
  </tr>
  <tr>
    <td># Lines of oracle files</td>
    <td>2115.5</td>
    <td>1366.8</td>
    <td>1961.3</td>
    <td>1488.2</td>
  </tr>
  <tr>
    <td># Files edited</td>
    <td>2.62</td>
    <td>1.54</td>
    <td>1.66</td>
    <td>1.24</td>
  </tr>
  <tr>
    <td># Lines edited</td>
    <td>128.5</td>
    <td>68.1</td>
    <td>37.71</td>
    <td>14.32</td>
  </tr>
  <tr>
    <td># Lines of added components</td>
    <td>87.1</td>
    <td>47.2</td>
    <td>10.9</td>
    <td>2.1</td>
  </tr>
  <tr>
    <td>% Added components</td>
    <td>67.8</td>
    <td>69.3</td>
    <td>28.9</td>
    <td>14.7</td>
  </tr>
  <tr>
    <td># Functions added</td>
    <td>4.49</td>
    <td>2.02</td>
    <td>0.73</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td># Classes added</td>
    <td>0.78</td>
    <td>0</td>
    <td>0.064</td>
    <td>0.006</td>
  </tr>
</table>

Table 1: Statistics for FEA-Bench and its lite subset, as well as for SWE-bench and its verified subset. The metrics include: 1) the number of task instances and involved repositories; 2) the average total number of lines in all Python files involved in code changes (i.e., the Oracle setting described in Section 4.2), and the average number of edited files and lines; 3) the average number of lines of new components, the average percentage of new component lines relative to all edited lines, and the average number of added functions and classes.

3.4 Benchmark Characteristics

Semi-guided software engineering task Although the signatures for new components are provided, the primary objective of the FEA-Bench is to evaluate the whole solution of new feature implementation. This is a comprehensive real-world software engineering task, distinct from code completion. As shown in Table 1, each instance in FEA-Bench involves an average of 128.5 modified lines, with 87.1 lines attributed to the new components themselves. This indicates that approximately 41.4 lines of changes are made elsewhere in the repository. Implementing new features not only requires the ability to generate code for specified new components but also necessitates making complementary changes within the existing repository.

Lite version We have also curated a subset to serve as a lite version of our dataset. This subset is filtered based on criteria including higher quality and lower difficulty. This lite version is particularly useful for evaluating systems that are computationally intensive and time-consuming. Detailed information can be found in Section A.3.

New components driven generation task From Table 1, we can observe that, on average, the number of lines for new components in each FEA-Bench task instance is more than 8× that of SWE-bench. Furthermore, the new components account for approximately 67.8% of all edited lines in FEA-Bench, compared to just 28.9% in SWE-bench. The difference in this metric between FEA-Bench lite and SWE-bench verified is even more pronounced. These statistics indicate that the task instances in FEA-Bench are primarily aimed at implementing new features. In SWE-bench, the average number of new functions is 0.73, indicating that its task instances mainly involve editing existing code rather than incremental development.

Complex solutions While SWE-bench focuses on fixing issues, which generally involve simpler problems, the task instances in FEA-Bench exhibit greater complexity. From Table 1, whether measured by the number of edited lines or edited files, the solutions of task instances in FEA-Bench are notably more complex than those in SWE-bench.

4 Experimental Design

4.1 Models

Due to its repository-level code generation characteristics, FEA-Bench requires models to have a long context window. We evaluate representative code LLMs and general-purpose LLMs with strong foundational capabilities on the FEA-Bench. The code LLMs used in our evaluation include CodeLlama, Codestral4, Qwen2.5-Coder, and DeepSeek-Coder-V2 (Roziere et al., 2023; Hui et al., 2024; Zhu et al., 2024). For general-purpose LLMs, we evaluate the performance of GPT-4, GPT-4o (Achiam et al., 2023; Hurst et al., 2024) and DeepSeek-V3 (Liu et al., 2024a), as well as models with long chain-of-thought (CoT) capabilities such as o1 and DeepSeek-R1 (Jaeck et al., 2024; Guo et al., 2025).

4.2 Context

To explore the capabilities and potential limits of LLMs in implementing new features within code repositories, we construct different prompts from several perspectives based on our collected data. Each task instance is evaluated using various context settings to provide a comprehensive understanding of model performance.

New component hints In FEA-Bench, information about new components can be derived from two sources: 1) signatures and documentation of newly extracted functions and classes, and 2) changes in non-Python files within the patch, which often contain relevant information about the new components. Based on this, we have two settings: Brief. Only provides the signatures of new compo-

4https://mistral.ai/en/news/codestral
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th rowspan="2">Window</th>
    <th colspan="4">FEA-Bench</th>
    <th colspan="4">FEA-Bench lite</th>
  </tr>
  <tr>
    <th>Oracle<br>Detailed</th>
    <th>Oracle<br>Brief</th>
    <th>BM25 (27K)<br>Detailed</th>
    <th>BM25 (27K)<br>Brief</th>
    <th>Oracle<br>Detailed</th>
    <th>Oracle<br>Brief</th>
    <th>BM25 (27K)<br>Detailed</th>
    <th>BM25 (27K)<br>Brief</th>
  </tr>
  <tr>
    <td>CodeLlama</td>
    <td>13B</td>
    <td>16K</td>
    <td>0.14</td>
    <td>0.43</td>
    <td>&times;</td>
    <td>&times;</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>&times;</td>
    <td>&times;</td>
  </tr>
  <tr>
    <td></td>
    <td>34B</td>
    <td>16K</td>
    <td>0.57</td>
    <td>0.57</td>
    <td>&times;</td>
    <td>&times;</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>&times;</td>
    <td>&times;</td>
  </tr>
  <tr>
    <td>Qwen2.5-Coder</td>
    <td>14B</td>
    <td>32K</td>
    <td>3.57</td>
    <td>3.57</td>
    <td>3.71</td>
    <td>2.93</td>
    <td>3.5</td>
    <td>4.5</td>
    <td>3.5</td>
    <td>2.5</td>
  </tr>
  <tr>
    <td></td>
    <td>32B</td>
    <td>32K</td>
    <td>4.43</td>
    <td>3.64</td>
    <td>3.85</td>
    <td>2.78</td>
    <td>6.0</td>
    <td>5.5</td>
    <td>2.0</td>
    <td>2.5</td>
  </tr>
  <tr>
    <td>Codestral-22B</td>
    <td>22B</td>
    <td>32K</td>
    <td>0.86</td>
    <td>0.93</td>
    <td>1.43</td>
    <td>1.36</td>
    <td>0.5</td>
    <td>1.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>DeepSeek-Coder-V2</td>
    <td>16B</td>
    <td>128K</td>
    <td>0.21</td>
    <td>0.29</td>
    <td>0.57</td>
    <td>0.36</td>
    <td>0.0</td>
    <td>0.5</td>
    <td>0.0</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>DeepSeek-R1-Distill</td>
    <td>32B</td>
    <td>128K</td>
    <td>3.78</td>
    <td>4.07</td>
    <td>4.78</td>
    <td>4.21</td>
    <td>5.5</td>
    <td>7.5</td>
    <td>7.0</td>
    <td>5.5</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>671B</td>
    <td>64K</td>
    <td>8.14</td>
    <td>6.92</td>
    <td>8.21</td>
    <td>7.64</td>
    <td>14.5</td>
    <td>10.5</td>
    <td>13.0</td>
    <td>12.0</td>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>671B</td>
    <td>64K</td>
    <td>9.92</td>
    <td>8.35</td>
    <td>10.49</td>
    <td>9.85</td>
    <td>14.5</td>
    <td>14.5</td>
    <td>12.0</td>
    <td>13.5</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td></td>
    <td>128K</td>
    <td>4.71</td>
    <td>4.21</td>
    <td>3.14</td>
    <td>2.86</td>
    <td>6.0</td>
    <td>6.5</td>
    <td>2.0</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td></td>
    <td>128K</td>
    <td>6.14</td>
    <td>5.57</td>
    <td>5.28</td>
    <td>4.50</td>
    <td>5.0</td>
    <td>5.0</td>
    <td>4.0</td>
    <td>3.5</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td></td>
    <td>128K</td>
    <td>1.93</td>
    <td>1.86</td>
    <td>2.28</td>
    <td>2.57</td>
    <td>2.0</td>
    <td>3.5</td>
    <td>1.0</td>
    <td>2.0</td>
  </tr>
  <tr>
    <td>o1</td>
    <td></td>
    <td>200K</td>
    <td>7.28</td>
    <td>6.57</td>
    <td>6.78</td>
    <td>6.64</td>
    <td>10.0</td>
    <td>12.5</td>
    <td>5.0</td>
    <td>7.0</td>
  </tr>
</table>

Table 2: The resolved ratios on FEA-Bench (lite) task instances. The evaluation is conducted on single-round generation outputs by each model and a task instance is considered resolved only if all unit tests are passed. The prompt using BM25 retrieved files is limited to length of 27K tokens. This ensures that, with a maximum generation of 4K tokens, the total length will not exceed 32K tokens, which is the context window limits of most tested models. "Detailed" and "Brief" refer to the levels of hints regarding new components in prompt, as mentioned in Section 4.2.

nents. Detailed. Includes all the aforementioned information.

Retrieval method Given the extensive amount of code across multiple files in a repository, selective inclusion of file contents as context is necessary. Firstly, the README file and files containing new components are always included in the context. For other files, similar to SWE-bench, we divide retrieval methods into: Oracle. Includes all files involved in the patch in the context. BM25. Retrieves relevant files across the entire repository by BM25 algorithm (Robertson et al., 2009) based on the content of the pull request, and ranks them by relevance and filling the context until reaching a specified length.

Output format Generating an entire file can lead to interruptions due to generation limits and is costly. Therefore, our experiments offer two edit-based generation settings: Natural. Generates code edits in a natural format as pairs of before-and-after snippets, which can be converted into patches applicable to the code repository through post-processing. Patch. Directly generates edits in patch format. Since patches use line numbers for fragment location, this setting includes line numbers in the context’s code content.

The details of prompt and experimental settings are shown in Appendix B.

5 Evaluation Results

The performance of LLMs The evaluation results of FEA-Bench are presented in Table 2. It is observed that in Oracle and Detailed prompt settings, the best resolved ratio of task instances is 9.92%, indicating the poor performance of LLMs in the incremental development task at the repository level. Generally, the models with larger parameter sizes demonstrate better results. Among code LLMs, Qwen2.5-Coder exhibited performance comparable to that of GPT-4, highlighting its superiority in the domain of code generation. Despite this, general-purpose LLMs with stronger foundational capabilities can approach the performance of specialized code LLMs. For example, R1-Distill, which shares the same underlying architecture as Qwen2.5-Coder, showed competitive performance. Among the models evaluated, the latest DeepSeek-V3 and R1 models achieve the best performance, significantly outperforming OpenAI’s GPT-4 and o1 series. This underscores the importance of foundational capabilities in LLMs for repository-level development tasks. Additionally, the performance on the lite version is slightly higher, but the relative trends remain largely consistent. When computational resources are limited, the metrics from FEA-Bench lite can be used to reflect the performance on the full benchmark.

The performance under different contexts As
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">BM25 (27K)</th>
    <th colspan="2">BM25 (40K)</th>
  </tr>
  <tr>
    <th>Retrieval Metrics (%)</th>
    <th></th>
    <th>Retrieval Metrics (%)</th>
    <th></th>
  </tr>
  <tr>
    <td>Precision Avg.</td>
    <td>40.26</td>
    <td></td>
    <td>31.85</td>
    <td></td>
  </tr>
  <tr>
    <td>Recall Avg.</td>
    <td>76.04</td>
    <td></td>
    <td>77.14</td>
    <td></td>
  </tr>
  <tr>
    <td>Recall All</td>
    <td>51.61</td>
    <td></td>
    <td>53.03</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="2">Resolved Ratio (%)</th>
    <th>Detailed</th>
    <th>Brief</th>
    <th>Detailed</th>
    <th>Brief</th>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td></td>
    <td>3.14</td>
    <td>2.86</td>
    <td>3.14</td>
    <td>2.78</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td></td>
    <td>5.28</td>
    <td>4.50</td>
    <td>4.78</td>
    <td>4.64</td>
  </tr>
</table>

Table 3: The retrieval metrics and the instance resolved ratios under 27K and 40K token length limits of the prompt in *BM25* retrieval mode.

shown in Table 2, we evaluate the performance of LLMs under different settings of new component hints and retrieval methods. Overall, detailed new component hints lead to better model performance. However, in the results from FEA-Bench lite, brief hints that only provide signatures performed better. This discrepancy could be attributed to the lack of structured presentation of new components and their documentation within the prompt, as illustrated in Figure 6. Regarding retrieval methods, the *Oracle* setting generally outperforms the *BM25* setting, although the difference is not substantial. This may be because code files containing new components, which are regarded as known conditions in the FEA-Bench, ensure a certain baseline performance. Considering the simplest instances in the dataset: those involving modifications to only one code file. Since instances in FEA-Bench always include new components, the unique code file is known information. In this case, *BM25* retrieved files supplement additional information, which can lead to better model performance compared to the *Oracle* setting.

6 Discussion

6.1 Retrieved Files in Context

We aim to investigate whether providing more code context helps LLMs improve new feature development. Therefore, we conducted experiments by increasing the prompt limit from 27K to 40K tokens, which surpasses the original context window limit of Qwen2.5-Coder but can contain more retrieved files. The results are presented in Table 3. The evaluated models include GPT-4 and GPT-4o, and we also report the average precision and recall of Python files in the context relative to those in the gold patch, as well as the proportion of instances where all involved files are recalled.

Although the recalls slightly improve at the limit of 40K tokens, model performance decreases. This indicates that current LLMs still struggle to extract useful information from long contexts in repository-level code development tasks. To improve performance on FEA-Bench, enhancing the precision of retrieval may be an effective approach than simply increasing the context length.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">%Apply</th>
    <th colspan="2">%Resolved</th>
  </tr>
  <tr>
    <th>Natural</th>
    <th>Patch</th>
    <th>Natural</th>
    <th>Patch</th>
  </tr>
  <tr>
    <td>Qwen2.5-Coder(32B)</td>
    <td>44.82</td>
    <td>12.92</td>
    <td>4.43</td>
    <td>1.71</td>
  </tr>
  <tr>
    <td>R1-Distill(32B)</td>
    <td>55.75</td>
    <td>19.06</td>
    <td>3.78</td>
    <td>1.71</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>59.10</td>
    <td>33.26</td>
    <td>4.71</td>
    <td>3.07</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>66.38</td>
    <td>19.49</td>
    <td>6.14</td>
    <td>1.86</td>
  </tr>
  <tr>
    <td>o1</td>
    <td>57.03</td>
    <td>-</td>
    <td>7.28</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DeepSeek-V3</td>
    <td>69.09</td>
    <td>-</td>
    <td>8.14</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DeepSeek-R1</td>
    <td>73.16</td>
    <td>-</td>
    <td>9.92</td>
    <td>-</td>
  </tr>
</table>

Table 4: The impact of output formats *Natural* and *Patch*. We show the success rates of applying code edits to the code repository and the final resolved ratios. The experiments on directly generating patches are conducted only on the models shown in the first four columns.

6.2 Output Format of Edits

Direct repository-level code development outputs edits rather than new code itself. How LLMs can better generate edits remains an open question. Therefore, we analyze the impact of output formats using the two configurations described in Section 4.2. The two rightmost columns of Table 4 illustrate a comparison between the *Natural* and the *Patch* generation method, with the former demonstrating significantly higher performance. The possible reason is that generating patches imposes stricter formatting requirements, which current LLMs find challenging to adhere to accurately. Therefore, the main results presented in Table 2 adopt the performance by *Natural* generation method.

Whether using patches converted by converting before-and-after code snippets in *Natural* mode or directly generating patches in *Patch* mode, during evaluation, these patches must be applied to the repository. The success rates of the git apply are shown in the middle two columns of Table 4. It can be observed that the success rate in the *Natural* prompt mode is significantly higher, contributing to the superior performance of LLMs in *Natural* mode. Further observation reveals a significant positive correlation between the success rate of applying patches and the resolved ratio, regardless of whether *Natural* or *Patch* mode is used. DeepSeek-
<table>
  <tr>
    <th></th>
    <th colspan="2">Agentless</th>
    <th colspan="2">Agentless-Lite</th>
    <th colspan="2">Oracle</th>
    <th colspan="2">BM25 (27K)</th>
  </tr>
  <tr>
    <th></th>
    <th>%Reso.</th>
    <th>%Apply</th>
    <th>%Reso.</th>
    <th>%Apply</th>
    <th>%Reso.</th>
    <th>%Apply</th>
    <th>%Reso.</th>
    <th>%Apply</th>
  </tr>
  <tr>
    <th>DeepSeek-V3</th>
    <td>-</td>
    <td>-</td>
    <td>11.0</td>
    <td>71.0</td>
    <td>14.5</td>
    <td>70.5</td>
    <td>13.0</td>
    <td>69.0</td>
  </tr>
  <tr>
    <th>GPT-4</th>
    <td>-</td>
    <td>-</td>
    <td>3.5</td>
    <td>94.5</td>
    <td>6.0</td>
    <td>54.0</td>
    <td>2.0</td>
    <td>37.0</td>
  </tr>
  <tr>
    <th>GPT-4o</th>
    <td>9.0</td>
    <td>87.5</td>
    <td>9.5</td>
    <td>96.0</td>
    <td>5.0</td>
    <td>60.0</td>
    <td>4.0</td>
    <td>53.5</td>
  </tr>
  <tr>
    <th>o1-mini</th>
    <td>4.5</td>
    <td>69.5</td>
    <td>4.5</td>
    <td>81.5</td>
    <td>2.0</td>
    <td>32.5</td>
    <td>1.0</td>
    <td>23.5</td>
  </tr>
  <tr>
    <th>o1</th>
    <td>14.0</td>
    <td>90.5</td>
    <td>10.0</td>
    <td>89.5</td>
    <td>10.0</td>
    <td>49.5</td>
    <td>5.0</td>
    <td>33.5</td>
  </tr>
</table>

Table 5: The performance of Agentless on FEA-Bench lite, compared to the direct retrieval by Oracle and BM25. "%Reso." refers to the resolved ratio of task instances, while "%Apply" indicates the success rate of applying the generated code edits to the repository.

![Bar chart showing resolved ratios grouped by categories of repositories](page_1012_370_410_246.png)

Figure 4: The resolved ratios grouped by the categories of the repositories.

R1, which performs the best on FEA-Bench, has the highest values for both metrics. This indicates that the format of code edits is a critical factor limiting the performance of LLMs on such tasks.

6.3 Evaluation of Agent Frameworks
To further evaluate the performance of current state-of-the-art methods on FEA-Bench, we conduct experiments on the lite subset using the cost-efficient Agentless and Agentless-lite frameworks (Xia et al., 2024). We compare the final performance of using these agents with that of directly constructing the context using Oracle and BM25 retrieval methods, as shown in Table 5. Notably, Agentless improves the resolved ratio of task instances over BM25 for most models, with the largest gains observed for GPT-4o, o1-mini, and o1. This improvement strongly correlates with the increased success rate of applying code edits, highlighting the challenge of adhering to the required code editing format, as discussed in Section 6.2. Overall, Agentless and Agentless-lite perform similarly, with no significant advantage over the results under the Oracle setting. This indicates that current methods still have substantial room for improvement in repository-level new feature implementation.

6.4 Performances across Repositories
To further analyze LLMs’ performance at a finer granularity, we examine resolved ratios across different categories of repositories. We classify repositories into several categories, as shown in Appendix A.1. The performance of DeepSeek-R1, o1, and GPT-4o across different categories is illustrated in Figure 4. The resolved ratios of different models vary across categories; task instances in the Testing category (from the repo: joke2k/faker) have the highest resolved ratio, followed by the Internet category. For the remaining categories, the pass ratio of the three models are at a similar level.

Among the three LLMs, GPT-4o shows slightly weaker performance compared to the other models, but its trend is largely consistent with o1. Notably, DeepSeek-R1 exhibits weaker performance in the Testing category but significantly outperforms both GPT-4o and o1 in all other categories. This suggests that integrating different models might further enhance overall performance on FEA-Bench.

6.5 Performances under Different Complexity of New Components
To further investigate whether the implementation of new components adds pressure on LLMs in completing feature implementation tasks, we examine the relationship between the number of new functions and the number of resolved instances. The number of added functions is more indicative of the complexity of new components, as functions are more atomic compared to classes. Figure 5 illustrates the distribution of task instances and those solved by R1 with respect to the number of added functions. Excluding instances where the number
Figure 5: Histogram of the number of added functions both in all task instances and resolved task instances by DeepSeek-R1 (under Natural and Detailed prompt settings).

of added functions is zero (i.e., adding only classes for storing variables), it is evident that the resolved ratio decreases as the number of added functions increases. The resolved ratio is 18.96% when the number of added functions is 1, 8.24% when it is 2, and 5.47% when it is greater than or equal to 3. This indicates that implementing new features is a more challenging task than fixing bugs, which constitute the majority of instances in SWE-bench. Moreover, the more complex the new components, the higher the difficulty of successful implementation.

7 Conclusion

In this paper, we introduce FEA-Bench, a novel benchmark for evaluating the repository-level incremental code development capabilities of large language models (LLMs). Our benchmark focuses on the critical task of implementing new features by adding new components to existing code repositories. Through our comprehensive dataset and rigorous evaluation, we demonstrate that current LLMs face significant challenges in this domain. We also analyzed that the retrieval method of files, the output format, the repository itself, and the complexity of new components all impact the implementation of new features. Our work highlights the need for further advancements in LLMs’ reasoning and generation capabilities to better address real-world software engineering tasks. We hope that FEA-Bench will serve as a valuable resource for the research community, driving progress in this important area.

Limitations

Our constructed data and experiments have certain limitations. First, our benchmark includes only Python repositories, as Python projects are easier to execute and generally follow consistent testing frameworks. Second, the quantity of high-quality data suitable for repository-level incremental development is limited. High-quality and usable pull requests for new feature development are relatively scarce. Many repository-level code developments for implementing new functionalities were committed during the early stages of repositories, without going through the rigorous code review process typical of the open-source community, resulting in lower data quality that cannot be utilized. Furthermore, the software’s early-stage developments might not even have been conducted using the GitHub platform, posing a challenge for data collection and utilization. Consequently, FEA-Bench, which is built on publicly available data and subjected to stringent filtering, may exhibit certain scenario limitations.

Due to the long context involved in repository-level code development, the cost of conducting experiments using LLMs is relatively high. Therefore, the experimental results are based on a single round generation, akin to Pass@1, which may introduce a certain level of bias into the results. Additionally, given the scarcity of API resources for models like DeepSeek-V3 and R1, some results in the main experiments presented in Table 2 are missing. We hope that more affordable models similar to DeepSeek can be further developed to facilitate research and applications in the field of repository-level code development.

Ethics Statement

We collected the data from publicly available Github repositories only for research purposes. All the repositories have licenses that allow free software use. LLMs are used only for classification during the construction of the FEA-Bench dataset, so no harmful information can be created in the dataset. The dataset and code for our proposed method will be made publicly available for academic research. However, we should note that the inference results of the task instances from the benchmark may contain code that is harmful to computer systems. Evaluation by docker is recommended, just like SWE-bench.

Additionally, the ChatGPT platform was used as
an AI assistant for refining the paper writing.

Acknowledgments

This work was supported by National Science and Technology Major Project (No. 2022ZD0116308) and National Natural Science Foundation of China (62036001). The corresponding author is Xin Zhang and Houfeng Wang.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. ArXiv preprint, abs/2303.08774.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. ArXiv preprint, abs/2108.07732.

Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B Ashok, and Shashank Shet. 2024. Codeplan: Repository-level coding using llms and planning. Proceedings of the ACM on Software Engineering, 1(FSE):675–698.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374.

Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023. Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software, 203:111734.

Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeeval: A manually-crafted benchmark for evaluating llms on class-level code generation. ArXiv preprint, abs/2308.01861.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. ArXiv preprint, abs/2501.12948.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming—the rise of code intelligence. ArXiv preprint, abs/2401.14196.

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring coding challenge competence with apps. ArXiv preprint, abs/2105.09938.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. ArXiv preprint, abs/2409.12186.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os-trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. ArXiv preprint, abs/2410.21276.

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. ArXiv preprint, abs/2412.16720.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjin Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. ArXiv preprint, abs/2403.07974.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.

Harshit Joshi, José Pablo Cambronero Sánchez, Sumit Gulwani, Vu Le, Gust Verbruggen, and Ivan Radicek. 2023. Repair is nearly generation: Multilingual program repair with llms. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 5131–5140. AAAI Press.

Jia Li, Ge Li, Zhuo Li, Zhi Jin, Xing Hu, Kechi Zhang, and Zhiyi Fu. 2023a. Codeeditor: Learning to edit source code with pre-trained models. ACM Transactions on Software Engineering and Methodology, 32(6):1–22.

Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024a. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. ArXiv preprint, abs/2404.00599.

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, et al. 2024b. Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories. ArXiv preprint, abs/2405.19856.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023b. Starcoder: may the source be with you! ArXiv preprint, abs/2305.06161.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. ArXiv preprint, abs/2412.19437.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023a. Is your code generated by chat-gpt really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Siyao Liu, He Zhu, Jerry Liu, Shulin Xin, Aoyan Li, Rui Long, Li Chen, Jack Yang, Jinxiang Xia, ZY Peng, et al. 2024b. Fullstack bench: Evaluating llms as full stack coder. ArXiv preprint, abs/2412.00535.

Tianyang Liu, Canwen Xu, and Julian McAuley. 2023b. Repobench: Benchmarking repository-level code auto-completion systems. ArXiv preprint, abs/2306.03091.

Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. 2025. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardocoder: Empowering code large language models with evol-instruct. ArXiv preprint, abs/2306.08568.

Zhenyu Pan, Rongyu Cao, Yongchang Cao, Yingwei Ma, Binhua Li, Fei Huang, Han Liu, and Yongbin Li. 2024. Codev-bench: How do llms understand developer-centric code completion? ArXiv preprint, abs/2410.01353.

Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaogang Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. ArXiv preprint, abs/2308.12950.

Yuling Shi, Songsong Wang, Chengcheng Wan, and Xiaodong Gu. 2024. From code to correctness: Closing the last mile of code generation with hierarchical debugging. ArXiv preprint, abs/2410.01215.

Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. 2024. Rlcoder: Reinforcement learning for repository-level code completion. ArXiv preprint, abs/2407.19487.

Yaoxiang Wang, Haoling Li, Xin Zhang, Jie Wu, Xiao Liu, Wenxiang Hu, Zhongxin Guo, Yangyu Huang, Ying Xin, Yujiu Yang, et al. 2025. Epicoder: Encompassing diversity and complexity in code generation. arXiv preprint arXiv:2501.04694.

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning.

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. ArXiv preprint, abs/2407.01489.

Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Joshua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, MAPS 2022, page 1–10, New York, NY, USA. Association for Computing Machinery.

Jian Yang, Jiajun Zhang, Jiaxi Yang, Ke Jin, Lei Zhang, Qiya Peng, Ken Deng, Yibo Miao, Tianyu Liu, Zeyu Cui, et al. 2024a. Excrepobench: Multi-level executable code completion evaluation. ArXiv preprint, abs/2412.11990.

John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024b. Swe-agent: Agent-computer interfaces enable automated software engineering. ArXiv preprint, abs/2405.15793.

Hao Yu, Bo Shen, Dezhil Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024a. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1–12.

Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. 2024b. WaveCoder: Widespread and versatile enhancement for code large language models by instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5140–5153, Bangkok, Thailand. Association for Computational Linguistics.

Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2471–2484, Singapore. Association for Computational Linguistics.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, page 1592–1604, New York, NY, USA. Association for Computing Machinery.

Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. 2024. Towards more realistic evaluation of llm-based code generation: an experimental study and beyond. ArXiv preprint, abs/2406.06918.

Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazhuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. ArXiv preprint, abs/2406.11931.

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratmadira Widyasari, Imam Nur Bani Yusuf, Haoalan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. ArXiv preprint, abs/2406.15877.

A Dataset Details

A.1 Repositories

The task instances in FEA-Bench are derived from 83 Python packages corresponding to GitHub repositories. For each package, we obtained its license information and topic classification from the PyPI website. For packages missing the topic attribute, we utilized their README files and employed GPT-4o to classify them into topics available on the PyPI website5. Finally, considering the distribution of the topics of task instances in FEA-Bench, we further simplify the topics to several categories to facilitate data visualization and analysis, as presented in Section 6.4.

The relevant information regarding the involved code repositories is summarized in Table 6.

A.2 Construction Details

Table 7 shows the number of remaining task instances for each stage during the data collection process. In Section 3.2, we mentioned that after the fast validation step, including the 18 repositories from SWE-bench, a total of 119 repositories (18 + 101) are available for further data collection. On this basis, we crawl the pull requests from GitHub to obtain code changes. Files with names containing words like test are identified as unit test files executable by pytest.

Initially, we filter pull requests (PRs) based on whether they were merged, resulting in the number of PRs shown in the "# PR" column of Table 7. Next, we excluded PRs without any test files, leaving only those PRs with the necessary conditions to be considered valid task instances, as indicated by the "# All tasks" column. Further filtering was conducted according to the third and fourth stages illustrated in Figure 3, retaining only those PRs that introduce at least one new component and are classified as "new feature" types by GPT-4o based on the PR content. Additionally, instances with patch lengths exceeding 8K (8192) tokens are excluded to remove long-tail distributions and noise. The remaining candidate task instances are listed under the "# Candidates" column. This filtering step does not exist in SWE-bench, leading to a smaller number of candidate task instances in our benchmark dataset.

Finally, for each instance, we apply the test patch to the repository in the base commit state to verify that unit tests could accurately evaluate the corresponding code edits. Similar to SWE-bench, our pipeline annotates each instance with the corresponding repository version and provides environment and installation configurations based on versions. For repositories included in SWE-bench, we directly utilize their environment setup and test configurations grouped by versions. For other repositories, minimal installation instructions (pip install -e .) and basic pytest testing configurations are used. Initially, we run the unit tests directly and observe the pass status. Subsequently, we apply the gold patch from the PR and execute the unit tests again. We allowed errors such as AttributeError and ImportError in the first test, which are common when new components are not yet implemented, but these errors are not permitted in SWE-bench data collection pipeline. For the second test after applying the gold patch, any FAILED status is unacceptable to ensure data quality. After this execution-based filtering process, the final task instances constitute the FEA-Bench task instances, as shown in the "# in Full" column of Table 7.

We only present information for repositories that have at least one instance in FEA-Bench. Specifically, out of the 119 repositories identified during the repository collection phase, only 83 repositories contain at least one task instance.

5https://pypi.org/classifiers/
<table>
  <tr>
    <th>Premise</th>
    <td>You will be provided with a partial code base and an feature request which requires a new feature to add in the code repository.</td>
  </tr>
  <tr>
    <th>Request Content</th>
    <td>
      <request>
        Pull Request Title + Pull Request Body<br>
        Comments before first commit
      </request>
    </td>
  </tr>
  <tr>
    <th>Issue Content (If any)</th>
    <td>
      Here is the discussion in the issues of the pull request.<br>
      <issues>
        Issue Title + Issue Body<br>
        Comments before first commit
      </issues>
    </td>
  </tr>
  <tr>
    <th>Doc Changes (If any)</th>
    <td>
      To implement the new features mentioned above, some design in this repository need to be modified, as the modification in document files.<br>
      <description changes>
        Gold patches that are not from .py files
      </description changes>
    </td>
  </tr>
  <tr>
    <th>New Components</th>
    <td>
      There are several new functions or classes that need to be implemented, using the definitions below:<br>
      <definitions>
        [start of new definitions in file 1]<br>
        (definition of namespace of new component 1)<br>
        signature of new component 1 in file 1<br>
        (definition of namespace of new component 2)<br>
        signature of new component 2 in file 1<br>
        [end of new definitions in file 1]<br>
        [start of new definitions in file 2]<br>
        ...
      </definitions>
    </td>
  </tr>
  <tr>
    <th>Code Files</th>
    <td>
      All the involved readme files and code files are listed below with their contents. If a file’s content is indicated as empty, it means that the file does not yet exist in the current repository and needs to be created with new content.<br>
      <code>
        [start of file 1]<br>
        All content in file 1: In "Patch" mode, line number is added<br>
        [end of file 1]<br>
        [start of file 2]<br>
        ...
      </code>
    </td>
  </tr>
  <tr>
    <th>Output Instruction</th>
    <td>
      "Natural" Format Output Instructions / "Patch" Format Instructions
    </td>
  </tr>
</table>

![A flowchart showing the de-duplication process from README files, Files with new components, and Files Retrieved ("Oracle" or BM25) to the final output instructions](page_1047_682_349_246.png)

Figure 6: The prompt for the inference of the task instances.

A.3 FEA-Bench lite

The feature implementation task proposed in this study, as one primary type of repository-level incremental code development tasks, require LLMs with long context capabilities to perform reasoning over extensive file contents. Such inference is computationally expensive. Considering the possible evaluation of multi-round code generation systems, it is necessary to select a high-quality subset for more efficient evaluation. Therefore, we establish stricter criteria to curate a higher-quality, lower-difficulty FEA-Bench lite subset.

Instances meeting any of the following low-quality criteria are excluded:

• The feature request descriptions contain fewer than 40 words.
• The instance involves cascading issues or commit SHA-256 references.
• The descriptions contain images.

Additionally, to limit task difficulty, instances meeting any of the following criteria are also excluded:

• Involve deleting code files.
• Involve more than three code files.
• The gold patch contains More than 10 code change hunks.
• Natural-formatted code change content exceeding 4K(4096) tokens.

• Contain new class(es).

• Contain more than ten added functions.

Beyond the lite subset, we also expect to collaborate with professional software engineers to annotate a verified subset, similar to SWE-bench Verified\footnote{https://openai.com/index/introducing-swe-bench-verified/}.

B Inference

B.1 Prompt

In Section 4.2, we present different prompt settings for the context of inference. A more detailed prompt structure is illustrated in Figure 6. The prompt is constructed using the first two items shown in Figure 2 as known input information. The feature request includes both pull request content and issue content. Information about new components and related code files is listed straightforwardly within the prompt.

In Figure 6, the italicized text indicates placeholders that need to be filled with specific data of each task instance, while the other text represents standard prompt content. Red text highlights additional information for detailed hints of new components, and blue text indicates parts that need to be modified when directly generating patches as results.

In this study, we provide two formats for generating code edits: *Natural* and *Patch*. The corresponding output instructions are illustrated in Figure 7. To ensure that the models produce outputs in the correct format, both modes are accompanied by detailed instructions and examples.

We include several different prompts mentioned above for each task instance in the dataset files.

B.2 Generation Configurations

For models with fewer than 32 billion parameters, we utilized the vLLM framework\footnote{https://github.com/vllm-project/vllm} on an 8-GPU NVIDIA A100 workstation, employing tensor parallelism for inference. The maximum number of generated tokens is limited to 4096, matching the generation length limits of used OpenAI GPT-4 and GPT-4o in our experiments.

For larger open-source and all closed-source models, specifically DeepSeek-V3, R1, and OpenAI series models, we invoke their APIs for inference. The versions of the OpenAI models used are as follows:

• GPT-4: *gpt-4-turbo-2024-04-09*

• GPT-4o: *gpt-4o-2024-05-13*

• o1: *o1-2024-12-17*

• o1-mini: *o1-mini-2024-09-12*

When possible, the temperature and top-p settings are fixed at 0.2 and 0.95, respectively. For DeepSeek-V3 and R1, the max output tokens are 8K (8192). For o1 and o1-mini, the max output tokens is 100,000 and 65,536 (64K), respectively.

During the inference process, LLMs perform a single generation for each task instance in FEA-Bench, and the output is converted into a patch for evaluation. The evaluation tools are adapted from the SWE-bench evaluation scripts which are based on docker, ensuring safety and easy use for the evaluation process.
"Natural" Format Output Instructions

Please solve the feature request with adding the functions or classes between the <definitions> and </definitions>. You do not need to output any changes or edits for descriptions or other files in .rst files.
I need you to make multiple edits across one or more files in a repository to implement a specific feature or improvement mentioned above.
For each edit, output the changes in the following format:

...
<edit>
[start of the snippet before editing in <file_path>]
<code_before_edit>
[end of the snippet before editing in <file_path>]

[start of the snippet after editing in <file_path>]
<code_after_edit>
[end of the snippet after editing in <file_path>]
</edit>

Notes:
- The <file_path> is the relative path of the file being edited.
- The <code_before_edit> snippet should include several lines before and after the modified region, unless the file was originally empty.
- If a file was originally empty, let <code_before_edit> blank but ensure <code_after_edit> includes the new code.
- Ensure the edits are sequential and address all necessary changes to achieve the requested feature.

Here is an example of the output format:
<edit>
[start of the snippet before editing in src/code.py]
def factorial(a):
    res = 1
    while a >= 0:
        res *= a
        a -= 1
    return res
[end of the snippet before editing in src/code.py]

[start of the snippet after editing in src/code.py]
def factorial(a):
    assert type(a) == int and a >= 0
    res = 1
    while a >= 2:
        res *= a
        a -= 1
    return res
[end of the snippet after editing in src/code.py]
</edit>

<edit>
[start of the snippet before editing in src/code.py]
def exact_division(x, y):
    return x % y == 0
[end of the snippet before editing in src/code.py]

[start of the snippet after editing in src/code.py]
def exact_division(x, y):
    assert type(x) == type(y) == int and x > 0 and y > 0
    return x % y == 0
[end of the snippet after editing in src/code.py]
</edit>

<edit>
[start of the snippet before editing in src/demo.py]
from code import factorial
print(factorial(5))
[end of the snippet before editing in src/demo.py]

[start of the snippet after editing in src/demo.py]
from code import factorial
print(factorial(5))
[end of the snippet after editing in src/demo.py]
</edit>

"I need you to solve the feature request with a series of edits in the format shown above. Respond below."

"Patch" Format Instructions

Please solve the feature request with adding the functions or classes between the <definitions> and </definitions>. You do not need to output any changes or edits for descriptions or other files in .rst files.
I need you to make multiple edits across one or more files in a repository to implement a specific feature or improvement mentioned above.
The edits should be output in patch format.

Here is an example of a patch file. It consists of changes to the code base. It specifies the file names, the line numbers of each change, and the removed and added lines.
A single patch file can contain changes to multiple files.

<patch>
diff --git a/src/code.py b/src/code.py
--- a/src/code.py
+++ b/src/code.py
@@ -1,14 +1,14 @@
 def factorial(a):
     assert type(a) == int and a >= 0
     res = 1
     while a >= 0:
         res *= a
         a -= 1
     return res
 #
 #
 #
 def exact_division(x, y):
     assert type(x) == type(y) == int and x > 0 and y > 0
     return x % y == 0
diff --git a/src/demo.py b/src/demo.py
new file mode 100644
--- /dev/null
+++ b/src/demo.py
@@ -0,0 +1,2 @@
+from code import factorial
+print(factorial(5))
</patch>

I need you to solve the provided feature request by generating a single patch file that I can apply directly to this repository using git apply. Please respond with a single patch file in the format shown above.
Respond below:
<table>
  <tr>
    <th>Repo Name</th>
    <th>License</th>
    <th>Topic</th>
    <th>Category</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>astropy/astropy</td>
    <td>BSD-3-Clause</td>
    <td>Scientific/Engineering::Astronomy</td>
    <td>Physics</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>django/django</td>
    <td>BSD-3-Clause</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>matplotlib/matplotlib</td>
    <td>Other</td>
    <td>Scientific/Engineering::Visualization</td>
    <td>Other</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>moxie/moxiebench</td>
    <td>BSD-3-Clause</td>
    <td>Scientific/Engineering::Visualization</td>
    <td>Other</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>pulset/pulse</td>
    <td>BSD-3-Clause</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>pylibv/lib-python</td>
    <td>BSD-3-Clause</td>
    <td>Scientific/Engineering::Physics</td>
    <td>Physics</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>pydata/xarray</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Information Analysis</td>
    <td>Other</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>pydicom/pydicom</td>
    <td>Others</td>
    <td>Scientific/Engineering::Medical Science Apps.</td>
    <td>Medical</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>pylint-devastard</td>
    <td>LGPL-2.1</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>pylint-devpylint</td>
    <td>GPL-2.0</td>
    <td>Software Development::Quality Assurance</td>
    <td>Other</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>pyyaml</td>
    <td>MIT</td>
    <td>Scientific/Engineering::Information Analysis</td>
    <td>Other</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>scikit-learn/scikit-learn</td>
    <td>BSD-3-Clause</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>sphinx-doc/sphinx</td>
    <td>BSD-2-Clause</td>
    <td>Text Processing::Markup</td>
    <td>Other</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>sqllift/sqllift</td>
    <td>MIT</td>
    <td>Software Development::Quality Assurance</td>
    <td>Other</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>sympy/sympy</td>
    <td>Others</td>
    <td>Scientific/Engineering::Mathematics</td>
    <td>Mathematics</td>
    <td>SWE-Bench</td>
  </tr>
  <tr>
    <td>Auter/Auter</td>
    <td>Apache-2.0</td>
    <td>Software Development::Code Generators</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>Cog-Cog/Red-DiscordBot</td>
    <td>GPL-3.0</td>
    <td>Communications::Chat</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>DLR-RM/stable-baselines</td>
    <td>MIT</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>EleutherAI/lm-evaluation-harness</td>
    <td>MIT</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>Project-MONA/MONAI</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Medical Science Apps.</td>
    <td>Medical</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>PyThaiNLP/Pythainlp</td>
    <td>Apache-2.0</td>
    <td>Text Processing::Linguistic</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>RDFLib/rdflib</td>
    <td>BSD-3-Clause</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>Textualize/textualize</td>
    <td>MIT</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>Textualize/textualize</td>
    <td>MIT</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>TitleDB-Inc/TitleDB-Py</td>
    <td>MIT</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>astronomer/astromer-cosmos</td>
    <td>Apache-2.0</td>
    <td>Software Development::Build Tools</td>
    <td>Build Tools</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>atlassian-app/atlassian-python-api</td>
    <td>Apache-2.0</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>aws-cloudformation/cfn-lint</td>
    <td>MIT-0</td>
    <td>Software Development::Quality Assurance</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>aws-powertools/powertools-lambda-python</td>
    <td>MIT-0</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>aws-sagemaker-python-sdk</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>bioinformatics/bioregistry</td>
    <td>MIT</td>
    <td>Scientific/Engineering::Bio-Informatics</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>botocore/botocore</td>
    <td>Apache-2.0</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>cocotb/cocotb</td>
    <td>BSD-3-Clause</td>
    <td>Scientific/Engineering::Electronic Design Automation (EDA)</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>conan-io/conan</td>
    <td>MIT</td>
    <td>Software Development::Build Tools</td>
    <td>Build Tools</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>deepest-air/haystack</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>docker/docker-py</td>
    <td>Apache-2.0</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>dpdk/dpdk</td>
    <td>Apache-2.0</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>embeddings-benchmark/mteb</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>facebookresearch/hydra</td>
    <td>MIT</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>fairlearn/fairlearn</td>
    <td>MIT</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>falconry/falcon</td>
    <td>Apache-2.0</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>google-deepmind/optax</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>googleps/python-onplatform</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>googleps/python-bugquery</td>
    <td>Apache-2.0</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>gradle-appgradolib</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Human Machine Interfaces</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>graphql-python/graphene</td>
    <td>MIT</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>huggingface/accelerate</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>huggingface/datasets</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>huggingface/huggingface_hub</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>huggingface/transformers</td>
    <td>Apache-2.0</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>jobjb/jobjb</td>
    <td>BSD-3-Clause</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>joke2k/joker</td>
    <td>MIT</td>
    <td>Software Development::Testing</td>
    <td>Testing</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>lark-parser/lark</td>
    <td>MIT</td>
    <td>Text Processing::Linguistic</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>minio/minio-py</td>
    <td>Apache-2.0</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>open-mmlab/mmengine</td>
    <td>Apache-2.0</td>
    <td>Utilities</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>openmmlab/datumaro</td>
    <td>MIT</td>
    <td>Scientific/Engineering::Image Processing</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>pgmpy/pgmpy</td>
    <td>MIT</td>
    <td>Scientific/Engineering::Artificial Intelligence</td>
    <td>AI</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>pre-commit/pre-commit</td>
    <td>MIT</td>
    <td>Software Development::Quality Assurance</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>prometheus/client_python</td>
    <td>Apache-2.0</td>
    <td>System::Monitoring</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>prompt-toolkit/python-prompt-toolkit</td>
    <td>BSD-3-Clause</td>
    <td>Software Development::User Interfaces</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>pygments/pygments</td>
    <td>Apache-2.0</td>
    <td>Software Development::Documentation</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>pycryptodome/pycryptodome</td>
    <td>MIT</td>
    <td>Software Development::Debugging</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>pyro-ppl/pyro</td>
    <td>Apache-2.0</td>
    <td>Software Development::Build Tools</td>
    <td>Build Tools</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>python-hyper/h2</td>
    <td>MIT</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>robolflow/supervision</td>
    <td>MIT</td>
    <td>Scientific/Engineering::Image Processing</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>rttilah/rttilah-mio</td>
    <td>GPL-3.0</td>
    <td>Home Automation</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>saleweaver/pytest-amazon-sp-api</td>
    <td>MIT</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>scrapy/scrapy</td>
    <td>BSD-3-Clause</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>scrapy/scrapy</td>
    <td>BSD-3-Clause</td>
    <td>Scientific/Engineering::Bio-Informatics</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>slackapi/bolt-python</td>
    <td>MIT</td>
    <td>Communications::Chat</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>slackapi/python-slack-sdk</td>
    <td>Apache-2.0</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>softlayer/softlayer-python</td>
    <td>MIT</td>
    <td>Software Development::Libraries</td>
    <td>Libraries</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>spec-first/connexion</td>
    <td>Apache-2.0</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>statecharts/statecharts</td>
    <td>BSD-3-Clause</td>
    <td>Scientific/Engineering::Information Analysis</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>tfrauzel/drft-specialcar</td>
    <td>BSD-3-Clause</td>
    <td>Software Development::Documentation</td>
    <td>Other</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>tobymason/giglot</td>
    <td>MIT</td>
    <td>Database::Database Engines/Servers</td>
    <td>Database</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>tornado/tornado-tornado</td>
    <td>Apache-2.0</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>tortoise/tortoise-orm</td>
    <td>Apache-2.0</td>
    <td>Database::Front-Ends</td>
    <td>Database</td>
    <td>Fast-Validation</td>
  </tr>
  <tr>
    <td>wagtail/wagtail</td>
    <td>BSD-3-Clause</td>
    <td>Internet::WWW/HTTP</td>
    <td>Internet</td>
    <td>Fast-Validation</td>
  </tr>
</table>

Table 6: The information of the repositories involved in FEABench.
<table>
  <tr>
    <th>Repository Name</th>
    <th>Class (Short)</th>
    <th>Max PR No.</th>
    <th># PR</th>
    <th># All tasks</th>
    <th># Candidates</th>
    <th># in Full</th>
    <th># in Lite</th>
  </tr>
  <tr><td>sympy/sympy</td><td>Mathematics</td><td>27454</td><td>12837</td><td>6034</td><td>819</td><td>239</td><td>46</td></tr>
  <tr><td>joke23/joker</td><td>Testing</td><td>2144</td><td>1386</td><td>518</td><td>186</td><td>126</td><td>8</td></tr>
  <tr><td>conan-io/conan</td><td>Build Tools</td><td>17534</td><td>6185</td><td>3715</td><td>473</td><td>124</td><td>23</td></tr>
  <tr><td>tobymao/sqllot</td><td>Database</td><td>4597</td><td>2463</td><td>1996</td><td>185</td><td>116</td><td>14</td></tr>
  <tr><td>scikit-learn/scikit-learn</td><td>AI</td><td>30289</td><td>17489</td><td>4174</td><td>186</td><td>83</td><td>9</td></tr>
  <tr><td>pvl/pvl/pvl-python</td><td>Physics</td><td>2336</td><td>1072</td><td>444</td><td>102</td><td>52</td><td>10</td></tr>
  <tr><td>deeppet/daihystack</td><td>AI</td><td>8669</td><td>4132</td><td>1438</td><td>317</td><td>49</td><td>8</td></tr>
  <tr><td>Project-MOON/MONAI</td><td>Medical</td><td>825</td><td>346</td><td>869</td><td>165</td><td>97</td><td>1</td></tr>
  <tr><td>mapdltools/mapdltools</td><td>Other</td><td>29140</td><td>18163</td><td>4057</td><td>164</td><td>34</td><td>7</td></tr>
  <tr><td>sphinx-doc/sphinx</td><td>Other</td><td>13196</td><td>5798</td><td>1597</td><td>126</td><td>30</td><td>4</td></tr>
  <tr><td>googleapis/python-apiplatform</td><td>AI</td><td>4830</td><td>2927</td><td>826</td><td>241</td><td>29</td><td>1</td></tr>
  <tr><td>astropy/astropy</td><td>Physics</td><td>17525</td><td>11205</td><td>4292</td><td>405</td><td>27</td><td>8</td></tr>
  <tr><td>Textualize/textual</td><td>Other</td><td>5444</td><td>2205</td><td>819</td><td>139</td><td>27</td><td>1</td></tr>
  <tr><td>falconry/falcon</td><td>Internet</td><td>2425</td><td>1279</td><td>499</td><td>67</td><td>27</td><td>3</td></tr>
  <tr><td>softlayer/softlayer-python</td><td>Libraries</td><td>224</td><td>127</td><td>127</td><td>1</td><td>0</td><td>0</td></tr>
  <tr><td>Textualize/textual</td><td>Libraries</td><td>3548</td><td>1142</td><td>283</td><td>41</td><td>24</td><td>4</td></tr>
  <tr><td>rytilah/tl/rytilah-mio</td><td>Other</td><td>1993</td><td>979</td><td>249</td><td>46</td><td>23</td><td>1</td></tr>
  <tr><td>sqfluff/sqfluff</td><td>Other</td><td>6534</td><td>3448</td><td>2079</td><td>296</td><td>19</td><td>2</td></tr>
  <tr><td>google-deepmind/optax</td><td>AI</td><td>1164</td><td>805</td><td>174</td><td>54</td><td>19</td><td>3</td></tr>
  <tr><td>pydata/xarray</td><td>Other</td><td>9879</td><td>4310</td><td>1838</td><td>234</td><td>17</td><td>2</td></tr>
  <tr><td>boto/boto3</td><td>Libraries</td><td>4371</td><td>757</td><td>130</td><td>38</td><td>17</td><td>2</td></tr>
  <tr><td>robotframework/supervision</td><td>Other</td><td>1737</td><td>1021</td><td>94</td><td>30</td><td>14</td><td>7</td></tr>
  <tr><td>RDFLib/RDFLib</td><td>Libraries</td><td>3025</td><td>1562</td><td>730</td><td>21</td><td>13</td><td>2</td></tr>
  <tr><td>huggingface/datasets</td><td>AI</td><td>7342</td><td>4118</td><td>792</td><td>99</td><td>42</td><td>2</td></tr>
  <tr><td>aws-cloudformation/cfn-lint</td><td>Other</td><td>3898</td><td>2583</td><td>790</td><td>92</td><td>11</td><td>0</td></tr>
  <tr><td>boto/botocore</td><td>Libraries</td><td>3331</td><td>2145</td><td>777</td><td>83</td><td>11</td><td>0</td></tr>
  <tr><td>pgmpy/pgmpy</td><td>AI</td><td>1887</td><td>902</td><td>368</td><td>76</td><td>11</td><td>2</td></tr>
  <tr><td>huggingface/huggingface_hub</td><td>AI</td><td>2683</td><td>1559</td><td>586</td><td>130</td><td>10</td><td>1</td></tr>
  <tr><td>prometheus/client_python</td><td>Other</td><td>1080</td><td>429</td><td>134</td><td>14</td><td>10</td><td>1</td></tr>
  <tr><td>pyyaml/pyyaml</td><td>Build Tools</td><td>1866</td><td>921</td><td>313</td><td>54</td><td>9</td><td>1</td></tr>
  <tr><td>scrapy/scrapy</td><td>Libraries</td><td>6598</td><td>3253</td><td>846</td><td>53</td><td>9</td><td>2</td></tr>
  <tr><td>slackap/python-slack-sdk</td><td>Other</td><td>1627</td><td>764</td><td>266</td><td>29</td><td>9</td><td>0</td></tr>
  <tr><td>djangos/django</td><td>Internet</td><td>18807</td><td>18482</td><td>725</td><td>32</td><td>7</td><td>1</td></tr>
  <tr><td>pydicom/pydicom</td><td>Medical</td><td>2195</td><td>979</td><td>494</td><td>42</td><td>7</td><td>1</td></tr>
  <tr><td>pylint-dev/pylint</td><td>Other</td><td>10168</td><td>4431</td><td>1859</td><td>79</td><td>7</td><td>1</td></tr>
  <tr><td>embeddings-benchmark/mteb</td><td>AI</td><td>1730</td><td>1097</td><td>337</td><td>15</td><td>7</td><td>1</td></tr>
  <tr><td>python-apt/python-apt</td><td>Internet</td><td>1290</td><td>1072</td><td>144</td><td>11</td><td>6</td><td>0</td></tr>
  <tr><td>mwasok/mwasok</td><td>Other</td><td>3798</td><td>1093</td><td>469</td><td>40</td><td>5</td><td>0</td></tr>
  <tr><td>pyvista/pyvista</td><td>Other</td><td>7045</td><td>3837</td><td>1149</td><td>292</td><td>5</td><td>2</td></tr>
  <tr><td>dpkp/kafka-python</td><td>Libraries</td><td>2442</td><td>937</td><td>212</td><td>17</td><td>5</td><td>0</td></tr>
  <tr><td>lark-parser/lark</td><td>Other</td><td>1503</td><td>467</td><td>135</td><td>10</td><td>5</td><td>2</td></tr>
  <tr><td>astronomer/astronomer-cosmos</td><td>Build Tools</td><td>1439</td><td>754</td><td>270</td><td>39</td><td>4</td><td>1</td></tr>
  <tr><td>fairlearn/fairlearn</td><td>AI</td><td>1472</td><td>937</td><td>229</td><td>29</td><td>4</td><td>1</td></tr>
  <tr><td>huggingface/accelerate</td><td>AI</td><td>3292</td><td>1697</td><td>627</td><td>67</td><td>4</td><td>2</td></tr>
  <tr><td>deskew/deskew.py</td><td>Libraries</td><td>3297</td><td>1554</td><td>538</td><td>24</td><td>3</td><td>1</td></tr>
  <tr><td>huggingface/trl</td><td>AI</td><td>2550</td><td>1181</td><td>248</td><td>29</td><td>3</td><td>0</td></tr>
  <tr><td>joblib/joblib</td><td>Libraries</td><td>1641</td><td>701</td><td>250</td><td>11</td><td>3</td><td>0</td></tr>
  <tr><td>open-mmlab/mmengine</td><td>Other</td><td>1620</td><td>1013</td><td>350</td><td>59</td><td>3</td><td>1</td></tr>
  <tr><td>openvinotoolkit/datamaro</td><td>Other</td><td>1689</td><td>1388</td><td>521</td><td>98</td><td>3</td><td>0</td></tr>
  <tr><td>pigment/pigments</td><td>Other</td><td>2837</td><td>886</td><td>346</td><td>79</td><td>3</td><td>0</td></tr>
  <tr><td>pyocd/pyOCD</td><td>Other</td><td>1774</td><td>954</td><td>173</td><td>11</td><td>3</td><td>0</td></tr>
  <tr><td>pyro-pyro</td><td>AI</td><td>3413</td><td>2302</td><td>1038</td><td>369</td><td>3</td><td>0</td></tr>
  <tr><td>tortoise/tortoise-orm</td><td>Database</td><td>1840</td><td>607</td><td>283</td><td>46</td><td>3</td><td>0</td></tr>
  <tr><td>DLR-RM/Unstable-baselines3</td><td>AI</td><td>2069</td><td>534</td><td>199</td><td>23</td><td>2</td><td>0</td></tr>
  <tr><td>EleutherAI/eval-evaluation-harness</td><td>AI</td><td>2609</td><td>961</td><td>52</td><td>9</td><td>2</td><td>1</td></tr>
  <tr><td>PyTorchNLP/pytestnlp</td><td>Other</td><td>1056</td><td>667</td><td>269</td><td>39</td><td>2</td><td>1</td></tr>
  <tr><td>TileDB-Inc/TileDB-Py</td><td>Libraries</td><td>2128</td><td>1184</td><td>488</td><td>34</td><td>2</td><td>0</td></tr>
  <tr><td>atlassian-apiflat/aslan-python-api</td><td>Internet</td><td>1476</td><td>849</td><td>149</td><td>14</td><td>2</td><td>0</td></tr>
  <tr><td>aws-sam/sam-python-sdk</td><td>AI</td><td>496</td><td>3219</td><td>715</td><td>116</td><td>22</td><td>2</td></tr>
  <tr><td>googleapis/python-bigquery</td><td>Internet</td><td>2102</td><td>1372</td><td>358</td><td>65</td><td>22</td><td>2</td></tr>
  <tr><td>gradio-app/gradio</td><td>Other</td><td>10271</td><td>4707</td><td>1210</td><td>81</td><td>20</td><td>2</td></tr>
  <tr><td>graphql-python/graphene</td><td>Libraries</td><td>1583</td><td>68</td><td>19</td><td>2</td><td>1</td><td>0</td></tr>
  <tr><td>prompt-toolkit/python-prompt-toolkit</td><td>Other</td><td>1949</td><td>713</td><td>49</td><td>3</td><td>2</td><td>0</td></tr>
  <tr><td>snowflake/snowflake-connector-python</td><td>Libraries</td><td>2127</td><td>1311</td><td>466</td><td>22</td><td>2</td><td>0</td></tr>
  <tr><td>spec-first/connection</td><td>Internet</td><td>2011</td><td>879</td><td>327</td><td>31</td><td>2</td><td>0</td></tr>
  <tr><td>statamodels/statamodels</td><td>Other</td><td>948</td><td>579</td><td>374</td><td>121</td><td>22</td><td>2</td></tr>
  <tr><td>tornado/tornado</td><td>Other</td><td>3452</td><td>1573</td><td>341</td><td>16</td><td>2</td><td>0</td></tr>
  <tr><td>pallets/flask</td><td>Internet</td><td>5640</td><td>2604</td><td>404</td><td>17</td><td>1</td><td>0</td></tr>
  <tr><td>pylint-dev/astroid</td><td>Libraries</td><td>2669</td><td>1810</td><td>301</td><td>8</td><td>1</td><td>0</td></tr>
  <tr><td>Ader-AFader</td><td>Other</td><td>2767</td><td>346</td><td>31</td><td>1</td><td>1</td><td>0</td></tr>
  <tr><td>Cog-Creators/Red-DiscordBot</td><td>Other</td><td>6499</td><td>4060</td><td>157</td><td>18</td><td>1</td><td>0</td></tr>
  <tr><td>aws-powertools/power-tools-lambda-python</td><td>Libraries</td><td>5814</td><td>4398</td><td>517</td><td>115</td><td>11</td><td>1</td></tr>
  <tr><td>biopandas/biopandas</td><td>Other</td><td>1543</td><td>794</td><td>286</td><td>28</td><td>1</td><td>0</td></tr>
  <tr><td>coocho/coocho</td><td>Other</td><td>4345</td><td>2137</td><td>493</td><td>34</td><td>1</td><td>0</td></tr>
  <tr><td>facebookresearch/hydra</td><td>Libraries</td><td>3005</td><td>1301</td><td>442</td><td>35</td><td>1</td><td>0</td></tr>
  <tr><td>huggingface/pytorch-image-models</td><td>Libraries</td><td>2398</td><td>551</td><td>85</td><td>17</td><td>1</td><td>1</td></tr>
  <tr><td>mono/mono-py</td><td>Libraries</td><td>1472</td><td>890</td><td>242</td><td>14</td><td>1</td><td>0</td></tr>
  <tr><td>pre-commit/pre-commit</td><td>Other</td><td>3382</td><td>1222</td><td>527</td><td>23</td><td>1</td><td>0</td></tr>
  <tr><td>saleweaver/python-amazon-sp-api</td><td>Internet</td><td>1638</td><td>1027</td><td>21</td><td>9</td><td>1</td><td>0</td></tr>
  <tr><td>severes/severes</td><td>Other</td><td>340</td><td>235</td><td>156</td><td>35</td><td>1</td><td>0</td></tr>
  <tr><td>slackapi/bolt-python</td><td>Other</td><td>1234</td><td>446</td><td>159</td><td>9</td><td>1</td><td>0</td></tr>
  <tr><td>tfranzel/lr-fspectacular</td><td>Other</td><td>1362</td><td>345</td><td>135</td><td>12</td><td>1</td><td>1</td></tr>
  <tr><td>wagtail/wagtail</td><td>Internet</td><td>12732</td><td>7070</td><td>1531</td><td>213</td><td>1</td><td>0</td></tr>
</table>

Table 7: Statistics of how many PRs (task instances) are left during the data collection procedures.