DevEval: A Manually-Annotated Code Generation Benchmark Aligned with
  Real-World Code Repositories
Authors: name: Jia Li, name: Ge Li, name: Yunfei Zhao, name: Yongmin Li, name: Huanyu Liu, name: Hao Zhu, name: Lecheng Wang, name: Kaibo Liu, name: Zheng Fang, name: Lanshen Wang, name: Jiazheng Ding, name: Xuanming Zhang, name: Yuqi Zhu, name: Yihong Dong, name: Zhi Jin, name: Binhua Li, name: Fei Huang, name: Yongbin Li

How to evaluate the coding abilities of Large Language Models (LLMs) remains
an open question. We find that existing benchmarks are poorly aligned with
real-world code repositories and are insufficient to evaluate the coding
abilities of LLMs.
  To address the knowledge gap, we propose a new benchmark named DevEval, which
has three advances. (1) DevEval aligns with real-world repositories in multiple
dimensions, e.g., code distributions and dependency distributions. (2) DevEval
is annotated by 13 developers and contains comprehensive annotations (e.g.,
requirements, original repositories, reference code, and reference
dependencies). (3) DevEval comprises 1,874 testing samples from 117
repositories, covering 10 popular domains (e.g., Internet, Database). Based on
DevEval, we propose repository-level code generation and evaluate 8 popular
LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa).
Our experiments reveal these LLMs' coding abilities in real-world code
repositories. For example, in our experiments, the highest Pass@1 of
gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize
their shortcomings. We hope DevEval can facilitate the development of LLMs in
real code repositories. DevEval, prompts, and LLMs' predictions have been
released.