Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'
Authors: name: Shanchao Liang, name: Yiran Hu, name: Nan Jiang, name: Lin Tan

Large language models (LLMs) have shown remarkable ability in code generation
with more than 90 pass@1 in solving Python coding problems in HumanEval and
MBPP. Such high accuracy leads to the question: can LLMs replace human
programmers? Existing manual crafted, simple, or single-line code generation
benchmarks cannot answer this question due to their gap with real-world
software development. To answer this question, we propose REPOCOD, a code
generation benchmark with 980 problems collected from 11 popular real-world
projects, with more than 58% of them requiring file-level or repository-level
context information. In addition, REPOCOD has the longest average canonical
solution length (331.6 tokens) and the highest average cyclomatic complexity
(9.00) compared to existing benchmarks. In our evaluations on ten LLMs, none of
the models can achieve more than 30 pass@1 on REPOCOD, disclosing the necessity
of building stronger LLMs that can help developers in real-world software
development.