DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories (2024)

Abstract: How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. ❶ DevEval aligns with realworld repositories in multiple dimensions, e.g., code and dependency distributions. ❷ DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). ❸ DevEval comprises 1,825 testing samples from 115 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs’ coding abilities in real-world code repositories. For example, the highest Pass@1 of gpt-4 only is 53.04% in our experiments. We also analyze LLMs’ failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs’ predictions have been released1.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 3603–3614 August 11-16, 2024 ©2024 Association for Computational Linguistics
To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. ❶ DevEval aligns with realworld repositories in multiple dimensions, e.g., code and dependency distributions. ❷ DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). ❸ DevEval comprises 1,825 testing samples from 115 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs’ coding abilities in real-world code repositories. For example, the highest Pass@1 of gpt-4 only is 53.04% in our experiments. We also analyze LLMs’ failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs’ predictions have been released1.

heading: 1 Introduction
text: Code generation with Large Language Models (LLMs) has attracted lots of researchers’ attention (Guo et al., 2024; Rozière et al., 2023; Lozhkov et al., 2024), and some commercial products have been produced, e.g., GitHub Copilot (GitHub, 2023). With more and more LLMs emerging, how to evaluate LLMs on code generation remains an open question.
*Corresponding authors 1https://github.com/seketeam/DevEval
Existing benchmarks are mainly composed of hand-crafted programming problems and are poorly aligned with real-world code repositories. LLMs’ performance on these benchmarks is inconsistent with developers’ actual experiences in realworld software development. Thus, a benchmark aligned with real-world repositories is necessary. We analyze over 1 million functions from 500 realworld repositories (see Section 3) and think a good benchmark should satisfy the following features.
• Real-world Repository. The benchmark should be collected from real-world code repositories (Yu et al., 2023).
• Real Code Distribution. Real-world repositories comprise two types of code, i.e., standalone and non-standalone code. As shown in Figure 1, a standalone function solely uses built-in or public libraries, while a non-standalone one contains
3603
context-aware dependencies (i.e., invocations of code elements defined in current repositories). The benchmark should cover both types of code and ensure their ratios are realistic. The number of dependencies should also be consistent with real-world repositories.
• Comprehensive Annotations. The benchmark can offer comprehensive annotations, including natural language requirements, original repositories, and ground truths (code and dependencies).
• Robust Evaluation Metrics. The benchmark should provide execution-based metrics (e.g., Pass@k) evaluate functional correctness of programs and metrics to assess the accuracy of dependencies in programs.
However, as shown in Table 1, none of the existing benchmarks satisfies all aforementioned features. The problem hinders the evaluation and development of LLMs in the real development process.
To address the above problem, we propose a new code generation benchmark named DevEval, which aligns with real-world code repositories. As shown in Table 1, DevEval satisfies the above features. ❶ DevEval comprises 1,825 testing samples from 115 real-world repositories, which cover 10 popular domains (e.g., Internet, Database). ❷ DevEval is constructed through a rigorous pipeline and aligns with real-world repositories. Specifically, the distributions of code and dependencies in DevEval are consistent with the ones in 500 real-world repositories. Detailed statistics are in Section 2.4. ❸ DevEval is annotated by 13 developers and contains comprehensive annotations, e.g., detailed requirements, original repositories, reference code, and reference dependencies.
❹ DevEval leverages test cases to check models’ predictions and report Pass@k. It also proposes Recall@k to evaluate the dependencies in predictions.
Based on DevEval, we propose repository-level code generation, which simulates the developers’ coding process in a working repository. The task asks models to write the code based on requirements and a complete repository.
We evaluate 8 popular LLMs (i.e., gpt-4 (OpenAI, 2023b), gpt-3.5 (OpenAI, 2023a), DeepSeek Coder (Guo et al., 2024), StarCoder 2 (Lozhkov et al., 2024), CodeLLaMa (Rozière et al., 2023)). These LLMs exhibit low performance on DevEval, especially compared to their performance on previous benchmarks. For example, gpt-4-turbo1106 achieves a Pass@1 score of 80% on HumanEval, while its highest Pass@1 on DevEval is only 53.04%. Our results reveal the coding abilities of these LLMs in real-world repositories. We further analyze failed cases and summarize the shortcomings of existing LLMs in DevEval.
In summary, our contributions are as follows:
• We summarize four features (see Table 1) that a code generation benchmark for real-world repositories should satisfy.
• We propose a new code generation benchmark - DevEval, satisfying the above features. The benchmark has been released.
• We propose repository-level code generation, which provides a challenging and realistic evaluation scenario.
• We evaluate 8 popular LLMs on DevEval, analyzing their strengths and shortcomings in repository-level code generation.
We hope DevEval can align with the actual experiences of developers during the practical development process. By DevEval, practitioners can pick up superior LLMs and facilitate the application of code generation techniques in real-world repositories.

heading: 2 Benchmark - DevEval

heading: 2.1 Overview
text: DevEval contains 1,825 samples derived from 115 real-world code repositories. As shown in Figure 2, each sample consists of six components. ❶ Function Signature: The signature of the target code. ❷ Requirement: An English description detailing the functionality of the target code. ❸ Repository Contexts: Code contexts (e.g., classes, functions, variables) defined outside the target code in the current repository. ❹ Reference Code: A developerwritten implementation of the target code. This code may invoke dependencies defined in the current repository. ❺ Reference Dependency: The dependencies invoked in the reference code include intra-class, intra-file, and cross-file dependencies. ❻ Test Cases: Test cases are used to check the functional correctness of the code.

heading: 2.2 Task Definition
text: Based on DevEval, we propose repository-level code generation task. A model is given a function signature, a requirement, and a complete repository. The model is asked to output a function to satisfy the requirement. We then insert the function into its repository and check its correctness.

heading: 2.3 Evaluation Metrics
text: Pass@k (Functional Correctness). Following previous studies (Chen et al., 2021; Austin et al., 2021; Yu et al., 2023), we assess the functional correctness of programs by executing test cases and compute the unbiased Pass@k. Specifically, we generate n ≥ k programs per requirement, count the number of correct programs c ≤ n that pass test cases, and calculate the Pass@k:
Pass@k := E Requirements
 1− ( n− c k )
( n k )
  (1)
Recall@k (Recall of Reference Dependency). Besides the functional correctness, we expect LLMs to invoke relevant dependencies defined in contexts. Hence, we propose Recall@k, which
gauges the recall of reference dependencies in generated programs.
Specifically, LLMs generate k programs per requirement. For the i-th program, we employ a parser2 to extract its dependencies as Pi. Subsequently, we compare Pi with reference dependencies R and compute the Recall@k:
Recall@k := E Requirements [ max i∈[1,k] |R ∩ Pi| |R| ] (2)
where | · | means the number of elements of a set.

heading: 2.4 Features of DevEval
text: Compared to existing benchmarks, DevEval shows three unique advances, which we discuss below.
❶ Alignment with real-world code repositories. Table 2 shows the data distributions of existing benchmarks and DevEval. We also show the data distributions of 500 real-world repositories and consider them as the oracle. We can see that DevEval aligns with 500 real repositories in multiple aspects, i.e., code distributions, the number of dependencies, and the scale of repositories. Table 3 further shows the distribution of dependency types, i.e., intra-class, intra-file, and cross-file dependencies. DevEval outperforms previous benchmarks in all types, showing a distribution that is close to the distribution in 500 real repositories.
2We develop the parser based on an open-source static analysis tool - Pyan (Pyan, 2023).
❷ Comprehensive Annotations. As shown in Figure 2, DevEval provides comprehensive annotations that are labeled by 13 human developers. Particularly, DevEval has advantages in requirements and reference dependencies.
Requirements. Original code comments in repositories are often vague and are different from requirements in practice (GitHub, 2023). We engaged 13 developers to write requirements, costing approximately 674 person-hours manually. As depicted in Figure 2, each requirement encapsulates the code’s functionality and input-output parameters. The average length of requirements in DevEval (101.6 tokens) more than doubles that of CoderEval (41.5 tokens). Reference Dependencies. Previous benchmarks (i.e., CoderEval, ClassEval) only provide dependencies’ names (e.g., close). Because many functions have the same name in practice, it is hard to identify whether generated dependencies are correct by relying on names. DevEval annotates dependencies with paths (e.g., A.py::ClassB::close), addressing ambiguity and biases. These annotations offer a broad arena to explore repository-level code generation and evaluation.
❸ A realistic task and evaluation metrics. Traditional benchmarks fall into a simple requirementto-code task. In contrast, DevEval proposes a more realistic task - repository-level code generation. This task simulates the coding process of developers in a working repository. Besides, we design two metrics to comprehensively assess the correctness of generated programs in functionality and dependencies.
❹ Wide scope for research communities. The DevEval and repository-level code generation can serve as an arena to compare approaches ranging from retrieval and long-context models to decision-
making agents. DevEval also allows creative freedom, as models can generate diverse programs to meet requirements.

heading: 3 Benchmark Construction
text: As shown in Figure 3, the collection of DevEval consists of five stages.
Stage ❶: Repository Selection. PyPI (PyPI) is a rich data source for real code repositories. We identify the top 10 popular programming topics in PyPI and select the top 50 repositories under each topic. The selection follows three criteria: open-source licenses, non-fork and non-malicious repositories, and explicit unit tests. We download the latest released versions in November 2023 and finally obtain 500 practical projects (10 topics * 50 projects).
Stage ❷: Function Parse. We extract functions from 500 repositories and exclude trivial functions (i.e., empty or initialization functions). We extract each function’s signature and function body (i.e., reference code). The other programs with current repositories are considered as repository contexts. Finally, this stage obtain 590,365 candidate functions.
Stage ❸: Tests Construction. We extract test cases from repositories invoking specific candidate functions. We use a public framework - setuptools to automatically install running environments and run test cases with a popular testing framework - Pytest. Candidate functions without executable test cases are excluded. Meanwhile, we ensure our test cases succeed in the reference code and fail in the wrong programs. Finally, we retain 3,764 candidate functions.
Stage ❹: Human Annotation. We engaged 13 developers to manually annotate requirements and reference dependencies for each candidate function. Given their countries of residence, all annotators obtain adequate payments.
Through discussions with annotators, we establish two criteria for requirements. Naturalness– ensuring the requirement reads like a natural description from the perspective of a real-world developer. Functionality–demanding clear descriptions of the code’s purposes and input-output parameters. Each requirement undergoes a dual-annotation process, with one annotator assigned to its initial drafting and another responsible for a meticulous double-check. Trivial functions (e.g., shortcut functions) and functions violating the ethical code (e.g., malware) are excluded. Subsequently, the same 13 annotators review the reference code and label its reference dependencies. Finally, we retain 2,846 functions with high-quality requirements and reference dependencies.
Stage ❺: Benchmark Construction. We select candidate functions to construct based on two criteria: consistent with the data distribution of 500 realworld repositories and including as many functions as possible. Finally, we select 1,323 (73%) nonstandalone functions and 502 (27%) standalone functions to construct DevEval.

heading: 4 Experiments

heading: 4.1 Studied LLMs
text: As shown in Table 4, we evaluate 8 popular LLMs, including two closed-source models and six opensource models. We use official interfaces or implementations to reproduce these LLMs.

heading: 4.2 Experimental Settings
text: Repository-level code generation takes a requirement and a repository as inputs. Typically, a repository consists of hundreds of code files and is very
long. For example, the average length of 500 realworld repositories is 1.1 million tokens, surpassing the context windows of existing LLMs (e.g., gpt-4: 128k tokens). Inspired by related work (Shrivastava et al., 2023), we try to extract parts of code contexts from the repository as inputs and design the following experimental settings.
❶ Without context. In this setting, we ignore contexts and directly generate the code based on requirements and signatures.
❷ Local File (Completion). The local file denotes the code file where the reference code is in. This setting simulates the scenario where developers continue to write code at the end of a file. Thus, we consider code snippets above the reference code in the local file as contexts. Then, LLMs generate code in an autoregressive manner based on requirements, signatures, and contexts.
❸ Local File (Infilling). Different from the Local File (Completion) setting, this setting simulates the scenario where developers infill code in the middle of a file. Thus, we use the code snippets above and below the reference code in the local file as contexts. We evaluate LLMs that support code infilling and construct input sequences using official formats.

heading: 4.3 Evaluation
text: We use Pass@k and Recall@k (see Section 2.3) to assess generated programs. In this paper, k ∈ [1, 3, 5, 10]. When k = 1, we use the greedy search and generate a single program per requirement. When k > 1, we use the nucleus sampling with a temperature 0.4 and sample 20 programs per requirement. We set the top-p to 0.95 and the max generation length to 500.

heading: 4.4 Main Results
text: The Pass@k and Recall@k of different LLMs in three experimental settings are shown in Table 5. Without Context. gpt-4 and DeepSeek Coder achieve the highest Pass@1 and Recall@1 among all LLMs, respectively. However, all LLMs exhibit relatively low Pass@k and Recall@k values compared to their performance on previous benchmarks. For instance, gpt-4 achieves a Pass@1 score of 88.4 on HumanEval, whereas it scores 17.00 on Pass@1 in this setting. The decreases validate our motivation that existing benchmarks can not comprehensively assess the coding abilities of LLMs in real-world repositories. Furthermore, the results emphasize the importance of contexts. Local File (Completion) and (Infilling). After
introducing the contexts within local files, the Pass@k and Recall@k of all LLMs obviously increase. For example, the Pass@1 of gpt-4 is improved by 217% and 183% in two settings, respectively.
Successful Case Analyses. We further inspect successful cases of gpt-4 and attribute the improvements to the synergy of contexts and requirements. On the one hand, the contexts provide lots of domain knowledge. For example, the local file contains essential local environments (e.g., current classes, imported libraries) and a majority of dependencies (e.g., intra-class and intra-file: 70% in DevEval). Recent work (Zhang et al., 2023; Ding et al., 2023) in code completion also proved the importance of contexts. On the other hand, our manually written requirements elaborate on the code’s purposes and the repositories’ background knowledge. Thus, the requirements help LLMs understand long contexts and locate relevant dependencies.
Error Case Analyses. Although promising, LLMs’ performance in repository-level code generation is not satisfying. A manual inspection of failed cases reveals LLMs struggle with understanding contexts. Figure 4 illustrates a failed case. LLMs invoke a non-existent function - create_connection, even though a valid function connect is present in the contexts. We think two reasons cause this problem.
First, the contexts are too long. The complete repositories are lengthy, approximately 9 times the context window of the state-of-the-art LLM - gpt4-1106. Even when partial contexts are considered, their lengths match or exceed most current LLMs’ context windows. Recent work (Liu et al., 2023a) has found that LLMs often ignore relevant information in the middle of long contexts. This finding is consistent with our results. Second, the contexts are heterogeneous. In other words, the contexts are composed of discrete code snippets from different files rather than a continuous file. As shown in Figure 4, the programs within contexts come from multiple files, e.g., boto.regioninfo.py and boto.swf.layer1.py. However, LLMs are typically trained to predict the next tokens based on the continuous contexts. The gap between training and inference objectives leads to a poor understanding of LLMs in contexts. Recent work (Shi et al., 2023) also found similar gaps in reading comprehension and question answering.
We also obtain some interesting findings from Table 5.
❶ LLMs successfully generate some dependencies without context. Theoretically, LLMs do not see the contexts and cannot generate dependencies defined in the contexts. According to Table 5, we are surprised to find that LLMs are able to generate some dependencies without context. We manually inspect successful cases and summarize two reasons. First, LLMs can reason about some easy dependencies from requirements, e.g., initialization functions of returned objects. Second, LLMs can “guess“ dependencies from their functionalities. In practice, dependencies’ names come from their functional descriptions, e.g., send_request() - send a request to the server. LLMs are trained with a large code corpus and can learn the naming conventions. Thus, LLMs may successfully guess some dependencies from their functionalities.
❷ More contexts benefit code generation.
Based on Table 5, we compare the performance of an LLM (e.g., gpt-4) under different settings. Obviously, the more input contexts, the better the performance of the LLM. It inspires practitioners to extend the context windows of LLMs and input more contexts.
❸ In the without context setting, gpt family models have higher Pass@k and lower Recall@k, while other models are the opposite. We speculate the reason is that gpt family models are instruction-tuned models and focus on performing tasks based on given instructions. With limited contexts, gpt family models are conservative and tend to generate code independently. Other LLMs are fundamental language models trained with real code files containing dependencies. They are aggressive and generate dependencies that may exist. The comparisons show the importance of instruction tuning in practical applications.

heading: 4.5 Empirical Lessons
text: Based on the above experiments, we summarize the empirical lessons we learned as
❶ DevEval poses new challenges, i.e., repository-level code generation. The performance of existing LLMs on DevEval drops dramatically compared to their performance on previous benchmarks.
❷ LLMs benefit from code contexts in current repositories. With limited context windows, the contexts from local files can improve gpt-4 by 217% in Pass@1.
❸ Detailed and accurate requirements help LLMs understand programs’ purposes and long contexts.
❹ LLMs struggle with understanding long and heterogeneous contexts. It causes LLMs to disregard the knowledge in contexts and even generate hallucinations (e.g., non-existent functions).

heading: 5 Discussion
text: Results on different program types. Figure 5 shows Pass@1 of gpt-4 on different program types (i.e., standalone and non-standalone). We have two observations from the results. ❶ Contexts are crucial to generating non-standalone functions. For example, adding local files improves the Pass@1 on non-standalone functions from 11.26 to 54.5. ❷ Contexts also benefit standalone functions. This is attributed to the domain knowledge within contexts, aiding LLMs in understanding requirements. ❸
There is considerable room for improving LLMs on both programs. How to effectively retrieve relevant contexts is a key problem. Results on different dependency types. Figure 6 shows the Recall@1 of gpt-4 on different dependency types (i.e., intra-class, intra-file, and crossfile). The results yield two insights. ❶ Without context, LLMs can reason about some simple dependencies from requirements (e.g., initialization functions of returned objects), but still exhibit low Recall@1 values across three dependency types. ❷ With contexts, LLMs exhibit an improvement in generating dependencies. Nevertheless, LLMs have yet to grapple with generating dependencies, especially cross-file dependencies. As illustrated in Figure 4, LLMs often ignore available dependencies defined in contexts. Data leakage. Theoretically, all open-source code repositories may be included in the training data for LLMs. Consequently, there is a risk of data leakage where several repositories used to build DevEval appear in the training data. We think this risk has only a slight impact on DevEval due to three reasons. ❶ DevEval contains new data, i.e., manually written requirements. These requirements are never included in the training data. ❷ Existing LLMs do not show overfitting tendencies to DevEval. Based on the release dates of 8 LLMs (see Section 4.1), we divide DevEval into two groups: unseen repositories released later than LLMs and potentially seen repositories released earlier than LLMs. The average difference of Pass@1 between the two groups is around 0.36. Compared to the average variations between LLMs (e.g., 5.15 in Table 5), 0.36 is slight. ❸ DevEval is geared toward evaluating future LLMs. We release the links to our selected repositories and encourage practitioners to omit these repositories when collecting the training data for future LLMs.
The bias of Recall@k. As stated in Section 2.3, we develop a static analysis-based parser to extract dependencies in generated programs automatically. Because Python is a dynamically typed language, certain dependencies are only determined at runtime and may elude our parser. It may lead to lower Recall@k than actual values.
To gauge the above bias, we randomly select 50 programs generated by gpt-4 and annotate dependencies with them by our parser and two human developers, respectively. Based on the humanannotated and auto-extracted dependencies, we compute two Recall@1 values. The bias of two Recall@1 values is 0.16. Compared to the average variations between LLMs (2.16 in Table 5), 0.16 is slight. Consequently, we believe that Recall@k can effectively rank different LLMs, notwithstanding its slight bias.

heading: 6 Related Work
text: Large Language Models for Code Generation. The rise of pre-training technology has brought new impetus to the field of code generation, both in academia and industry (Li et al., 2022; Shen et al., 2022; Nijkamp et al., 2023; Fried et al., 2023). In this context, more and more LLMs have emerged, achieving significant advancements in code generation, such as Codex (Chen et al., 2021), ChatGPT (OpenAI, 2023a), CodeLlama (Rozière et al., 2023), DeepSeek Coder (Guo et al., 2024), and StarCoder2 (Lozhkov et al., 2024).
To effectively steer LLMs in various code generation scenarios, some works focus on improving the prompt technologies by introducing specific patterns, e.g., Structured Chain-of-Thought (Li et al., 2023a), Self-planning (Jiang et al., 2023), Selfdebug (Chen et al., 2023), Self-collaboration (Dong et al., 2023), and AceCoder (Li et al., 2023c). Code Generation Benchmarks. Early code generation benchmarks (Yin et al., 2018; Chen et al., 2021; Austin et al., 2021; Zan et al., 2022) evaluate code generation on relatively Python functions, such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). APPS (Hendrycks et al., 2021) evaluates code generation on more difficult competition-style problems. ClassEval (Du et al., 2023) evaluates LLMs on class-level code generation and contains 100 human-crafted selfcontained Python classes. Concode (Iyer et al., 2018) and CoderEval (Yu et al., 2023) further introduce non-standalone programs.
Compared to existing benchmarks, DevEval aligns with real-world code repositories (e.g., the distributions of code and dependency) and contains more comprehensive annotations (e.g., reference dependencies).
We have also noticed that some benchmarks have recently been proposed for repository-level tasks. CrossCodeEval (Ding et al., 2023), RepoBench (Liu et al., 2023b), and RepoEval (Zhang et al., 2023) are code completion benchmarks. They lack the necessary annotations (e.g., natural language requirements) for code generation. SWE-bench (Jimenez et al., 2023) focuses on repairing repositories’ issues by revising existing programs. In contrast, DevEval is collected for code generation and aims to generate new programs based on requirements for a repository. DevEval offers comprehensive annotations (e.g., natural language requirements, original repositories, reference code, and reference dependencies).

heading: 7 Conclusion and Future Work
text: This paper proposes a new code generation benchmark named DevEval. Collected through a meticulous pipeline, DevEval aligns with real-world code repositories in multiple dimensions, e.g., real code distributions, sufficient dependencies, and realscale repositories. We evaluate 8 popular LLMs in DevEval. The results reveal the strengths and weaknesses of LLMs in real repositories. Compared to previous benchmarks, DevEval offers a more challenging and practical evaluation scenario. We hope DevEval can facilitate the applications of LLMs in practical repositories.
In the future, we will continue to update DevEval, e.g., multilingual testing samples, more projects, and more test cases. Besides, we will explore how to improve the performance of LLMs in context-based code generation, e.g., retrievalaugmented and tool-augmented generation.

heading: 8 Acknowledgments
text: This research is supported by the National Key R&D Program under Grant No. 2023YFB4503801, the National Natural Science Foundation of China under Grant No. 62192731, 62152730, 62072007, 62192733, 61832009, 62192730, and the Major Program (JD) of Hubei Province (No.2023BAA024). Ge Li and Yongbin Li are the corresponding authors.

heading: 9 Limitations
text: This paper proposes a new code generation benchmark - DevEval, which aligns with real-world code repositories. Based on DevEval, we evaluate 8 popular LLMs and analyze their strengths and shortcomings. We think that DevEval has three limitations.
❶ DevEval is a monolingual benchmark (i.e., requirements in English and code in Python) and ignores other languages. In practice, LLMs require understanding requirements in different natural languages (e.g., Chinese, Spanish) and generating programs in various programming languages (e.g., Java, C). Thus, we plan to build a multilingual DevEval in future work.
❷ As stated in Section 5, Recall@k values in DevEval may have slight biases, i.e., they may be slightly less than actual values. Because Python is a dynamically typed language, certain dependencies can only be identified at runtime and may elude our parser. To gauge the bias introduced by our parser, we manually annotate dependencies within 100 programs generated by gpt-4. Simultaneously, we employ the parser to extract dependencies in the same 50 programs. Based on the human-annotated and auto-extracted dependencies, we compute two Recall@1 values. The bias of two Recall@1 is 0.16. Compared to the average variations between LLMs (2.16 in Table 5), 0.16 is slight. Consequently, the Recall@k can effectively rank different LLMs, notwithstanding its slight bias.
❸ In our experiments, we only consider the code contexts from local files. In the future, we will explore how to utilize broader contexts (e.g., imported files, sibling files).

heading: 10 Ethics Consideration
text: DevEval is collected from real-world code repositories. We manually check all samples in DevEval. We ensure all samples do not contain private information or offensive content. We ensure all programs in DevEval are behaving normally and exclude any malicious programs.

heading: Future Work
text: Thus, we plan to build a multilingual DevEval in future work. ❷ As stated in Section 5, Recall@k values in DevEval may have slight biases, i. e. , they may be slightly less than actual values. Because Python is a dynamically typed language, certain dependencies can only be identified at runtime and may elude our parser. To gauge the bias introduced by our parser, we manually annotate dependencies within 100 programs generated by gpt-4. Simultaneously, we employ the parser to extract dependencies in the same 50 programs. Based on the human-annotated and auto-extracted dependencies, we compute two Recall@1 values. The bias of two Recall@1 is 0. 16. Compared to the average variations between LLMs (2. 16 in Table 5), 0. 16 is slight. Consequently, the Recall@k can effectively rank different LLMs, notwithstanding its slight bias. ❸ In our experiments, we only consider the code contexts from local files. In the future, we will explore how to utilize broader contexts (e. g. , imported files, sibling files)