SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CROSSCODEEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
Yangruibo Ding $^{1*}$  Zijian Wang $^{2,*}$  Wasi Uddin Ahmad $^{2,*}$  Hantian Ding $^{2}$  Ming Tan $^{2}$  Nihal Jain $^{2}$  Murali Krishna Ramanathan $^{2}$  Ramesh Nallapati $^{2}$  Parminder Bhatia $^{2}$  Dan Roth $^{2}$  Bing Xiang $^{2}$
<sup>1</sup>Columbia University <sup>2</sup>AWS AI Labs  
yrbding@cs.columbia.edu {zijwan,wuahmad}@amazon.com  
https://crosscodeeval.github.io
Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus on code completion tasks within a single file. This over-simplified setting falls short of representing the real-world software development scenario where repositories span multiple files with numerous cross-file dependencies, and accessing and understanding cross-file context is often required to complete the code correctly.
To fill in this gap, we propose CROSSCODEEval, a diverse and multilingual code completion benchmark that necessitates an in-depth cross-file contextual understanding to complete the code accurately. CROSSCODEEval is built on a diverse set of real-world, open-sourced, permissively-licensed repositories in four popular programming languages: Python, Java, TypeScript, and C#. To create examples that strictly require cross-file context for accurate completion, we propose a straightforward yet efficient static-analysis-based approach to pinpoint the use of cross-file context within the current file.
Extensive experiments on state-of-the-art code language models like CodeGen and StarCoder demonstrate that CROSSCODEEval is extremely challenging when the relevant cross-file context is absent, and we see clear improvements when adding these context into the prompt. However, despite such improvements, the pinnacle of performance remains notably unattained even with the highest-performing model, indicating that CROSSCODEEval is also capable of assessing model's capability in leveraging extensive context to make better code completion. Finally, we benchmarked various methods in retrieving cross-file context, and show that CROSSCODEEval can also be used to measure the capability of code retrievers.
Language models for code (code LMs), such as Codex (Chen et al., 2021), CodeGen (Nijkamp et al., 2023b,a), and StarCoder (Li et al., 2023), have demonstrated their power to enhance developer productivity through their promising results in code completion tasks. To evaluate these models, researchers propose multiple code completion evaluation benchmarks, (e.g., Chen et al., 2021; Lu et al., 2021; Athiwaratkun et al., 2023; Austin et al., 2021), where the model is asked to complete the code given the context in the current file. However, such an evaluation setting is over-simplified, and it is not able to reflect the model's capability in code completion accurately. Specifically, in the realm of modern software development, repositories consist of multiple files, each interwoven with
* Equal Contribution. Work done while Yangruibo Ding was an intern at AWS AI Labs.
Figure 1: Code LM fails to complete a Python test case since the in-file context (left figure) does not provide sufficient information. The function name from the current file indicates that the completing function is a test case for convert_camel_to(snake, so with only such context, the model hallucinates wrong completion as convert_camel_to(snake. However, the failure is not due to the model's capacity, but the necessary cross-file context is not present (right figure). When the class CaseConverter is present in the prompt, the model generates camel_to(snake correctly.
extensive cross-file dependencies, i.e., contextual information from other source code files within the same repository. A significant drawback of most existing benchmarks is their tendency to overlook these complex dependencies. As a result, they fall short of providing a comprehensive evaluation of code completion models within realistic real-world scenarios.
Figure 1 illustrates the limitation of common code completion evaluation sets with a real example. The developer is writing a test case for a class, CaseConverter, implemented within the current repository. CodeGen-2B-mono (Nijkamp et al., 2023b), a large Python code LM, fails to complete the API call if only the current file context is present.
Motivated by such examples and to fill in the need of evaluating code completion in realistic software development with numerous intervening cross-file context dependencies, we propose CROSSCODEEVAL, a diverse and multilingual benchmark to evaluate code language models' ability to use cross-file context for code completion. This new dataset is composed of 10k examples from 1k repositories in 4 languages. Unlike existing datasets where the correct answer could be predicted with only context from the current file, CROSSCODEEVAL strictly requires cross-file context to correctly complete the missing code ( $\S 2.2$ ). CROSSCODEEVAL's examples are carefully curated from existing open-sourced repositories with a series of quality filters, and we ensure CROSSCODEEVAL has minimal overlap with the training data from existing code LMs, eliminating the confounder of data leakage and memorization in result interpretation ( $\S 2.1$  &  $\S 2.3$ ).
We conducted a comprehensive evaluation of popular public and proprietary code LMs: CodeGen (Nijkamp et al., 2023b,a) and StarCoder (Li et al., 2023) in various sizes from 350M to 16B parameters, and OpenAI's GPT-3.5-Turbo in §3. Empirical results reveal that when given only the current-file context, these models yield suboptimal results. Remarkably, incorporating cross-file context into the prompt significantly enhances the performance of these code LMs, even in a zero-shot setting. This underscores that CROSSCODEEval effectively serves its goal as a benchmark aimed at evaluating cross-file code completion. Moreover, even when providing cross-file context in the prompt, the performance of the most powerful models remains notably imperfect, highlighting that CROSSCODEEval is also instrumental in assessing a model's ability for leveraging extensive context in code completion. Lastly, we benchmarked various retrieval methods from sparse to dense, demonstrating that CROSSCODEEval can additionally serve as a benchmark for code retrieval.
2 CROSSCODEVAL: A Benchmark for Cross-File Code Completion
CROSSCODEEval is a diverse and multilingual scope completion dataset in four popular languages: Python, Java, TypeScript, and C# where examples include code prompts ending in an imagined cursor position and references include the code token sequences from the cursor position to the end of the statement. CROSSCODEEval examples have a key property - the statement to be completed must
Figure 2: We replace the third import statement, which is from the same repository, with an empty class. Consequently, camel_to(snake in the last line becomes an undefined name in the modified file. Thereby, we know this method in the original file is defined only in the cross-file context.
have at least one use of local API (classes, variables, and methods defined in the software repository). Next, we briefly describe how we collect the software repositories ( $\S 2.1$ ), select a subset of them for CROSSCODEEVAL construction ( $\S 2.2$ ), post-processing and quality control process ( $\S 2.3$ ), and CROSSCODEEVAL statistics and future scope for extending the dataset ( $\S 2.4$ ).
2.1 Dataset Collection
We collect permissively licensed repositories from GitHub. To mitigate potential data leakage issues, we focus on repos that were created recently and not forks. Specifically, we collected repos created between 2023-03-05 to 2023-06-15 on 2023-09-01. The time span ensures sufficient data collected with no overlap with the training data of many existing code LMs released before mid-2023, no matter whether the data is publicly available or not. We limit repos to contain the four languages we study and we keep only repos with zipped file size  $< 1\mathrm{MB}$  and number of stars  $>= 3$ . Then we filter out repos that have fewer than 10 or more than 50 source code files. Finally, we remove the repos with at least one source code file that exactly matches one of the code files in the commonly used Stack (Kocetkov et al., 2022) dataset. As a result, we ended up with 471, 239, 193, and 99 repos, respectively.
2.2 Dataset Generation
We propose a static-analysis-based method to identify code fragments that require cross-file context automatically. Our approach is illustrated in Fig 2. First, we find all intra-project imports in the original file. Next, an empty class is created for each imported name to replace the import statement. Since the imported name now refers to an empty class, any subsequent call to its member function or attribute will raise an undefined name error. We leverage static analysis to catch such errors in the modified file, which precisely correspond to the names in the original file that can only be resolved by cross-file context. We map the location of undefined names back to the original file to determine the split point of the prompt and the reference.
To increase the variety in the dataset, we randomly select a tree-sitter<sup>3</sup> token in the same line before the cross-file entity to be the cursor location, splitting the code to a prompt and a reference. It is often the case that the same cross-file API is called multiple times in a file, and there is a chance for models to infer the API name from previous calls even without cross-file context. Therefore, if the same undefined name is reported at multiple places in a file, we keep only the first occurrence. In this work, we focus on instantiating our approach in the four popular programming languages, while the idea can be generalized to other languages in principle. Specifically, for Python, we use PyInt<sup>4</sup> to detect undefined names; for Java, we use javac compiler; for Scala, we use tsc compiler;
${}^{2}$  Overlapping between CROSSCODEEval and data used to pretrain code LMs.
<sup>3</sup>https://tree-sitter.github.io/tree-sitter/
4Pylint is a static code analyzer for Python: https://pylint.readthedocs.io/en/latest/
for C#, we use csc compiler from the mono $^5$  image. We use tree-sitter to identify full statements to construct reference completions in Python. For Java, TypeScript, and C#, we consider statements ending with either “;”, “{”, or “}”. See Appendix A for more details.
2.3 Post-processing and Quality Control
We designed a series of rule-based and model-based post-processing filters to ensure the quality of the dataset. We filter examples if (1) fewer than  $N$  lines of code (lines not including import statements, where  $N = 10,20,30,5$  for Python, Java, TypeScript, and C# in the prompt, (2) too short ( $< 3$  tokens), or long ( $>30$  tokens) reference. We exclude examples if the references are found verbatim in any other source code file within the repository (i.e., cross-file). We further discard examples with duplicate references. The filtering steps collectively remove  $15\% -20\%$  of the examples.
Moreover, to ensure that the reference isn't predictably inferred solely from the current file (possibly owing to strong clues in function names and comments), we feed the examples (input prompts) to starcoderbase-1B model (Li et al., 2023) to complete the statement and remove the exact matches. This step results in the removal of  $< 10\%$  of the generated examples. As an ancillary benefit, this further safeguards that the examples are not seen by publicly available code LMs while CROSSCODEEval is constructed based on repositories that do not overlap with the Stack and possibly other private pre-training datasets created prior to 2023. Finally, we perform human annotations on a subsample of the resulting CROSSCODEEval and found that the dataset has a satisfactory quality to serve the goal of cross-file code completion. See Appendix B for more details.
2.4 Dataset Statistics, Scope, and Future Extensions
Statistics We present the statistics of CROSSCODEEval in Table 1. We use the StarCoder tokenizer (Li et al., 2023) to compute the number of tokens.
Scope In addition to prompts and references, we include the code lines that follow the references from the original source code files in CROSSCODEEVAL examples. Given the source code
Feature	Python	Java	JavaScript	C#
# Repositories	471	239	193	99
# Files	1368	745	779	642
# Examples	2665	2139	3356	1768
Avg. # lines in prompt	90.6	106.7	116.5	71.1
Avg. # tokens in prompt	938.9	995.3	944.9	584.1
Avg. # lines in reference	1.0	1.1	1.7	1.7
Avg. # tokens in reference	13.2	14.5	17.4	12.5
Table 1: CROSSCODEEval statistics.
lines to the left (prompt or prefix) and right (suffix) of the references, CROSSCODEEval can be used to evaluate code LMs for their fill-in-the-middle (FIM) capabilities (Bavarian et al., 2022).
Future Extensions CROSSCODEEval currently supports four popular languages. As our method is generalizable, CROSSCODEEval can potentially be extended to other languages. Additionally, we advise future code LM pre-training datasets should explicitly exclude CROSSCODEEval to minimize the effect of memorization.
We benchmark CROSSCODEEval with popular public and proprietary large language models.
CodeGen (Nijkamp et al., 2023b,a) is a series of generative code LMs. CodeGen supports left-only context. CodeGen2.5 notably supports fill-in-the-middle and further improves the performance via multi-epoch training. We benchmarked CodeGen models at various sizes from 350M to 16B.
StarCoder (Li et al., 2023) is a generative multi-query-based code LM with 15.5B model parameters trained on The Stack dataset (Kocetkov et al., 2022). It supports up to 8k tokens. We also benchmarked its base version with at varied sizes: 1B, 3B, and 7B.
Mono is an open-source implementation of the .NET Framework: https://www mono-project.com/
Figure 3: An illustrative example to showcase the use of in-file context, retrieved cross-file context, and retrieved context using reference in prompts. While the baseline prompting uses in-file context only, "Retrieval" and "Retrieval w/ Ref." prompting uses retrieved contexts by prepending them to the in-file context.
GPT-3.5-turbo (Ouyang et al., 2022) is one of the most powerful models developed by OpenAI. It was trained with comprehensive text and code data and supports up to  $4\mathrm{k}$  max sequence length. Its model weight remains proprietary and is accessible exclusively via APIs.
In evaluating the performance of code language models, we report performance in two main categories: code match and identifier match (Ding et al., 2022).
Code Match The code match metric directly compares the generated code with the reference and is measured using exact match (EM) and edit similarity (ES). These metrics help to assess the overall accuracy of the code completion process, taking into account elements such as identifiers, keywords, operators, delimiters, and literals.
Identifier Match This metric evaluates the model's ability to predict the correct application programming interfaces (APIs). To perform this evaluation, we first parse the code and extract the identifiers from the model prediction and reference, resulting in two ordered lists of identifiers. We then compare the predicted identifiers with the reference and report the results in EM and F1 score.
Our evaluation framework is based on the Transformers (Wolf et al., 2020) library. All the experiments are conducted with the zero-shot setting, and no training is involved. We use the same set of hyperparameters for code generation across all models. We set the maximum sequence length to 2,048 for the CodeGen family, 4096 for GPT-3.5-turbo, and 8,192 for the StarCoder family. We use a maximum generation length of 50 and the rest as the prompt.
We explore greedy search and nucleus sampling (Holtzman et al., 2020) with reranking (Hossain et al., 2020). We found there is no significant difference between the two, and we present the greedy search results in the main paper and refer readers to Appendix D.2 for nucleus sampling.
We post-process model predictions to extract statements. For Python, we iteratively parse the concatenation of prompt and  $n$  completion tokens (e.g.,  $n = 1,2,\ldots ,{50}$  ) until the sequence becomes
<sup>6</sup>We apply the same post-processing on the references before calculating the evaluation metrics.
parseable (no syntax error) and the  $(n + 1)$ -th completion token is a newline character. $^7$  For Java, TypeScript, and C#, we consider statements ending with “;”, “{” and “}”, instead of a new line.
Only In-File Context In standard practice, pre-trained language models are utilized to perform code completion in a zero-shot manner by taking into account the provided code context. Following the practice, we conduct experiments using the code LMs (3.1), where they are provided code context from the current file. As shown in Figure 3, the baseline prompt includes only in-file context.
Retrieved Cross-file Context Inspired by the effectiveness of the recently proposed retrieve-and-generate (RG) framework for repository-level code completion (Zhang et al., 2023), we adopt it for cross-file context retrieval. In the RG framework, the retrieval database is constructed by iteratively scanning the files in the repository and extracting contiguous  $M$  lines (in all our experiments,  $M = 10$ ) of non-overlapping code fragments, which are the candidates for cross-file context retrieval. The query for the retrieval is built using the last  $N$  lines (we set  $N = 10$ ) of the in-file context. We use BM25 (Robertson et al., 2009) to calculate the similarity between the query and the candidates (cross-file context chunks), and use the top-5 similar code snippets as the cross-file context, see "Retrieval Context" in Figure 3. We consider a maximum of 512 BPE tokens for such context, and the rest of the tokens will be truncated. Figure 3 illustrates the retrieved context and the corresponding prompt for the model to complete. Given the in-file context as a query, the RG framework successfully retrieves the class definition of CaseConverter that is in another file for utilities. We further wrap the class definition into a template as code comment and use it as the cross-file context. To build the retrieval prompt, we prepend the retrieved context to the in-file context.
Retrieval with Reference To quantify the upper bound impacts of cross-file context retrieved by the RG framework, we devise "retrieval with reference" for comparison. In this setting, we make use of not only the in-file context (as in standard retrieval setting) but also the reference to retrieve the cross-file context. Specifically, the query is constructed by using the last  $N$  lines of the concatenation of the in-file context and the reference completion, instead of the in-file context only in the standard retrieval setting. We prepend the retrieved context (i.e., "Retrieval w/ Ref. Context") to in-file context to construct the prompt for this setting.
Note that the Retrieval w/ Ref. context could not be applied to the realistic code completion, as the reference completion is unknown. We use it as an estimation of the upper bound model performance with the RG framework. Also, the model's performance in this setting is not optimal, as it can still be limited by imperfect retrieval and the model's capability in making use of retrieved code, and we perform additional benchmarking and analysis on retrieval methods later in §3.5.
3.4 Results
We present results in Table 2 and additional results in Table 7. We see that all models perform poorly when the prompt includes only the in-file context. For example, the best-performing StarCoder model at 15.5B size only reports  $8.82\%$  code exact match in Python. Even a large code LM struggles to achieve promising performance in completing CROSSCODEEval samples with only in-file context since it could not provide sufficient clues for code completion. This shows the design of CROSSCODEEval that cross-file context is necessary to complete the code correctly.
The performance improves dramatically when the cross-file context is added to the prompts across all models and sizes. Figure 4 shows the significant improvements resulting from the inclusion of cross-file context in CodeGen and StarCoder models. Looking at Table 2, we see that the StarCoder model reports up to  $3.0 \times$  and  $4.5 \times$  better exact code match when including retrieved and retrieved with reference context respectively. The results underline the limitation of existing datasets that only consider the in-file context to evaluate code LMs, making these datasets insufficient in reflecting models' best capacity in real-world scenarios. In contrast, CROSSCODEVAL maintains the cross-file
<sup>7</sup>We manually verified that this is required to extract full statements in Python.
<sup>8</sup>We tried to use the code made publicly available by the authors of Zhang et al. (2023) but failed to execute them with the provided instructions. As a result, we implemented the RG approach by ourselves. Note that, unlike Zhang et al. (2023), we perform only 1-step retrieval augmented generation.
For the convenience of illustration, we remove the irrelevant code snippets from the retrieved context to avoid confusion. In practice, the retrieved context spans across a fixed length of lines, which includes useful information as well as its surrounding lines. More implementation details are in Appendix C.
contexts for code completion samples, providing resources to both identify the model's best capacity and analyze the model's behavior when seeing a more comprehensive context.
Model	Code Match
Python	Java	TypeScript	C#
EM	ES	EM	ES	EM	ES	EM	ES
CodeGen25-7B	7.73	59.34	10.43	62.05	7.81	57.56	4.36	58.99
+ Retrieval	14.52	64.40	16.88	64.35	12.57	60.08	13.01	63.86
+ Retrieval w/ Ref.	19.17	67.46	20.20	66.17	15.35	62.73	17.87	66.14
StarCoder-15.5B	8.82	61.08	9.96	63.25	6.35	51.22	4.47	59.80
+ Retrieval	15.72	66.28	17.48	66.10	8.31	44.87	13.57	65.00
+ Retrieval w/ Ref.	21.01	68.66	19.92	67.75	11.02	46.67	20.08	67.97
GPT-3.5-turbo	4.88	52.58	12.30	63.52	6.38	53.78	3.56	56.48
+ Retrieval	10.77	54.92	19.12	65.61	10.94	55.83	11.82	62.40
+ Retrieval w/ Ref.	15.72	58.88	22.72	68.50	14.15	58.40	17.65	66.07
Model	Identifier Match
Python	Java	TypeScript	C#
EM	F1	EM	F1	EM	F1	EM	F1
CodeGen25-7B	14.26	46.02	16.60	51.43	12.46	47.75	7.69	33.81
+ Retrieval	22.96	53.68	24.03	55.48	17.85	51.27	17.36	43.56
+ Retrieval w/ Ref.	28.33	57.95	27.91	57.87	21.51	55.38	21.78	47.63
StarCoder-15.5B	15.72	48.16	18.28	53.23	11.86	43.53	8.54	34.33
+ Retrieval	24.77	55.57	25.95	57.74	14.09	39.50	18.04	44.38
+ Retrieval w/ Ref.	30.24	59.46	29.73	60.47	17.55	42.18	24.38	49.09
GPT-3.5-turbo	10.09	39.18	18.93	52.52	10.76	44.78	5.77	30.25
+ Retrieval	17.37	44.43	26.74	56.57	16.69	48.15	15.44	41.24
+ Retrieval w/ Ref.	23.49	50.14	31.79	60.52	20.65	51.54	21.72	47.21
Table 2: Performance of various code LMs on CROSSCODEVAL. "Retrieval" and "Retrieval w/ Ref." mean we construct the prompt by prepending the retrieved cross-file context retrieved with the prompt and the prompt + reference (see §3.3 for details). The performance with no cross-file context (first row in each section) is generally poor. When prompts are augmented with cross-file context (middle row in each section), the performance increases significantly. The use of reference completion in formulating the query for cross-file context retrieval (last row in each section) shows the upper bound of the retrieve-and-generate (RG) approach. Results of other models are in Table 7.
3.5 Analysis and Discussions