FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation
  for Feature Implementation
Authors: name: Wei Li, name: Xin Zhang, name: Zhongxin Guo, name: Shaoguang Mao, name: Wen Luo, name: Guangyue Peng, name: Yangyu Huang, name: Houfeng Wang, name: Scarlett Li

Implementing new features in repository-level codebases is a crucial
application of code generation models. However, current benchmarks lack a
dedicated evaluation framework for this capability. To fill this gap, we
introduce FEA-Bench, a benchmark designed to assess the ability of large
language models (LLMs) to perform incremental development within code
repositories. We collect pull requests from 83 GitHub repositories and use
rule-based and intent-based filtering to construct task instances focused on
new feature development. Each task instance containing code changes is paired
with relevant unit test files to ensure that the solution can be verified. The
feature implementation requires LLMs to simultaneously possess code completion
capabilities for new components and code editing abilities for other relevant
parts in the code repository, providing a more comprehensive evaluation method
of LLMs' automated software engineering capabilities. Experimental results show
that LLMs perform significantly worse in the FEA-Bench, highlighting
considerable challenges in such repository-level incremental code development.