CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
Authors: Yangruibo Ding
Zijian Wang
Wasi Uddin Ahmad
Hantian Ding
Ming Tan
Nihal Jain
Murali Krishna Ramanathan
Ramesh Nallapati
Parminder Bhatia
Dan Roth
Bing Xiang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Code Benchmark
Code Evaluation
Code Completion
Cross-file Context
Project-level Context
Code Retrieval
TL;DR: CrossCodeEval is a multilingual benchmark built on diverse real-world repositories in Python, Java, TypeScript, and C#, evaluating code language models' capability in leveraging cross-file context for code completion.

Abstract: Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus on code completion tasks within a single file. This over-simplified setting falls short of representing the real-world software development scenario where repositories span multiple files with numerous cross-file dependencies, and accessing and understanding cross-file context is often required to complete the code correctly. 

To fill in this gap, we propose CrossCodeEval, a diverse and multilingual code completion benchmark that necessitates an in-depth cross-file contextual understanding to complete the code accurately. CrossCodeEval is built on a diverse set of real-world, open-sourced, permissively-licensed repositories in four popular programming languages: Python, Java, TypeScript, and C#. To create examples that strictly require cross-file context for accurate completion, we propose a straightforward yet efficient static-analysis-based approach to pinpoint the use of cross-file context within the current file. 

Extensive experiments on state-of-the-art code language models like CodeGen and StarCoder demonstrate that CrossCodeEval is extremely challenging when the relevant cross-file context is absent, and we see clear improvements when adding these context into the prompt. However, despite such improvements,  the pinnacle of performance remains notably unattained even with the highest-performing model,  indicating that CrossCodeEval is also capable of assessing model's capability in leveraging extensive context to make better code completion. Finally, we benchmarked various methods in retrieving cross-file context, and show that CrossCodeEval can also be used to measure the capability of code retrievers.

CROSS CROSSCODER CODEEVAL: A Diverse and Multilingual Benchmark for Cross-File Code Completion
Yangruibo Ding' Zijian Wang2.* Wasi Uddin 2
Hantian Ding Ming Tan Nihal Jain Murali Krishna Ramanathan" Ramesh Nallapati Parminder Bhatia² Dan Roth Bing Xiang
Columbia University AAWS AI Labs
yrbdingêes columbia. {zijwan, wuahmad) @amazon. mamazoon com
https://rrosscoleeval.gggt
Abstract
Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus code completion tasks within single file. This over-simplified setting falls short of representing the real-world software development scenario where repositories span multiple files with numerous cross-file dependencies, and accessing and understanding cross-file context is often required to complete the code correctly.
To fill in this gap, we propose CROSSCODEEVAL, diverse and multilingual code completion benchmark that necessitates an in-depth cross-file contextual understanding to complete the code accurately. CROSSCODEEVAL is built on a diverse set of real-world, open-sourced, pernissively-licensee repositories four popular programming languages: Python, Java, TypeScript, and C#. To create examples that strictly require cross-file context for accurate completion, we propose straightforward yet efficient static-analssis-basee approach to pinpoint the use of cross-file context within the current file.
Extensive experiments on state-of tiite-of-thee- the-art code language models like CodeGen and StarCoder demonstrate that CROSSCODEEVA is extremely challenging when the relevant cross-file context is absent, and we see clear improvements when adding these context into the prompt. However, despite such improvements, the pinnacle performance remains notably unattained even with the highest-performing model, indicating that CROssCODEEVAL is also capable of assessing model's s capability in leveraging extensive context to make better code completion. Finally, we benchmarked various methods in retrieving cross-file context, and show that CROssCODEEval VALL can also be used to measure the capability of code retrievers.
Introduction
Language models for code (code LMs), such as Codex (Chen al., 2021), CodeGen (Nijkamp et al., 2023b,a), and StarCoder (Li et al., 2023), have demonstrated their power to enhance developer productivity through their promising results in code completion tasks. To evaluate these models, researchers propose multiple code completion evaluation benchmarks, (e.g., Chen et al., 2021; 1; Lu al., 2021; Athiwaratkun et al., 2023; Austin al., 2021), where the model is asked complete the code given the context in the current file. However, such an evaluation setting is over-simplified, and it is not able to reflect the model's capability in code completion accurately. Specifically, in the realm of modern software development, repositories consist of multiple files, each interwoven with
Equal Contribution. Work done while Yangruibo Ding was an intern AWS Labs.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

File to be completed
Cross-ille Context
naigret/utils..
naigret/test utils.py
claas CaseConverter:
""""aigret utils test "unctions"""
@staticnethod
import itertools
def ccanel to (o_sinke((amllcoseedstttt str) str:
import
froo naigret.utils inport CaseConverter
és estaticnethod
def snake to com.amt(smakecsseed sttringddddsttgg-ee-e str) >> str:
def test cass convert canel to snake()
"Snake(asedString CaseConverter. [CURSOR_POSITION
estaticnethod
def snake to www.lee(ssneecceed str) str:
convert_carel carell toreee_tt__soelleee
casel to snake(a)
Code-LM fails to predict with in-file context
Ground truth requires cross-file context
Figure 1: Code LM fails to complete a Python test case since the in-file context (left figure) does not provide sufficient information. The function completing function is a test case for convert convert_camel_tee ____t___s___ snake, so with only such context, the model hallucinates wrong completion due to the model's capacity, but the necessary cross-file context is not present (right figure). When the class CaseConverter is present in the prompt, the model generates famel_to tel__tt_____a____eee__ snake correctly.
from the current file indicates that the
convert, camel_ to_ snake. However, the failure is not
extensive cross-file dependencies, i.e., contextual information from other source code files within the same repository. A significant drawback of most existing benchmarks is their tendency to overlook these complex dependencies. As a result, they fall short of providing a comprehensive evaluation of code completion models within realistic real-world scenarios.
Figure illustrates the limitation of common code completion evaluation sets with a real example. The developer is writing test case for class, CaseConverter, implemented within the current repository. CodeGen-2B-mon
(Nijkamp et al., 2023b), large Python code LM, fails to complete the API call if only the current file context is present.
Motivated by such examples and to fill in the need of evaluating code completion in realistic software development with numerous intervening cross-fil file context dependencies, we propose CROSSCODEE. VAL, a diverse and multilingual benchmark evaluate code language models' ability to use frles-file context for code completion. This new dataset is composed of 10k examples from 1k repositories in 4 languages. Unlike existing datasets where the correct answer could be predicted with only context from the current file, CROSSCODEEVAL strictly requires cross-file context to correctly complete the missing code ($2.2). CROSSCODEEVALE examples are carefully curated from existing open-sourced repositories with series of quality filters, and we ensure CROSSCODEEVAL has minimal overlap with the training data from existing code LMs, climinating the confounder data leakage and memorization in result interpretation ($2.1 & 2.3).
We conducted comprehensive evaluation of popular public and proprietary code LMs: CodeGen (Nijkamp et al., 2023b,a) and StarCoder (Li et al., 2023) in various sizes from 350M to 16B parameters, and OpenAl's GPT-3.5-Turbo in $3. Empirical results reveal that when given only the firrent-file context, these models yield suboptimal results. Remarkably, incorporating cross-file context into the prompt significantly enhances the performance of these code I.Ms, even in a zero- shot setting. This underscores that CROSSCODEEWS aimed at evaluating cross-file code completion. Moreover, even when providing cross-file context in the prompt, the performance of the most powerful models remains notably imperfect, highlighting that CROSSCODEES VAL L is also instrumental assessing a model's ability for leveraging extensive context in code completion. Lastly, we benchmarked various retrieval methods from sparse to dense, demonstrating that CROSsCODEEvAL can additionally serve as a benchmark for code retrieval.
effectively serves its goal benchmark
CROssCODEEVAL: Benchmark for Cross-File Code Completion
CROSSCODEEVAL is diverse and multilingual scope completion dataset four popular languages: Python, Java, TypeScript, and C# where examples include code prompts ending in an imagined cursor position and references include the code token sequences from the cursor position to the end of the statement. CROSSCODEEVAR examples have key property the statement to be completed must

---

Original file
Modifled file
naigret/test utils.py """Maigret utils test functions""" import itertools
"m"Raiggre utils test functions""" inport itertools
naigret/test, ut__uutssppp
Step 1: Expty class substitution
inport
re fron maigret.utils insort CaseConverter
fron noigret.utits inport-CaseConvennn CaseConverter
class (asetonverter: pass
def test_cas htt___cceccccortt..eeee convert_canel canel_ to_snake():
"SnakeCasedString"
def test
test_case convert canee_ so_snake()) "SnakeCasedString
CaseConverter. Cannl_ Saeee_tt___aaaappaee (n
CaseConverter canel, to moo_snaaee)
Step analysis
cross-file usage point
Backward napping
undefined
Figure 2: replace the third import statement, which is from the same repository, with an empty class. Consequently, camel_to snake the last line becomes an undefined name in the modified file. Thereby, we know this method in the original file is defined only in the cross-file context.
have least of local API (classes, variables, and methods defined in the software repository). Next, we briefly describe how we collect the software repositories ($2.1), select subset of them for CROSsCODEEVAL. construction ($2.2), post-processing and quality control process ($2.3), and CROSSCODEEVAL statistics and future scope for extending the dataset ($2.4).
2.1 Dataset Collection
We collect permissively licensed repositories from GitHub. To mitigate potential data leakage issues, we focus on repos that were created recently and not forks. Specifically, we collected repos created between 2023-03-05 to 2023-06-15 on 2023-09-01. The time span ensures sufficient data collected with no overlap with the training data of many existing code LMs released before mid-2023, no matter whether the data is publicly available not. We limit repos to contain the four languages study and we keep only repos with zipped file size < 1MB and number of stars >= Then we filter repos that have fewer than 10 more than 50 source code files. Finally, with at least one source code file that exactly matches one of the code files in the commonly used Stack (Kocetkov al., 2022) dataset. As result, ended up with 471, 239, 193, and 99 repos, respectively.
remove the repos
2.2 Dataset Generation
propose a static-analysis-basee method to identify code fragments that require cross-file context automatically. Our approach is illustrated in Fig 2. First, we find all intra-project imports in the original file. Next, an empty class created for each imported name to replace the import statement. Since the imported name now refers to empty class, any subsequent call to its member function or attribute will raise an undefined name error. We leverage static analysis to catch such errors in the modified file, which precisely correspond to the names in the original file that can only be resolved by cross-file context. We map the location of undefined names back to the original file determine the split point of the prompt and the reference.
To increase the variety in the dataset, we randomly select tree-sitter token in the same line before the cross-file entity to be the cursor location, splitting the code to prompt and reference. It is often the case that the same cross-file API called multiple times in a file, and there is chance for models to infer the API name from previous calls even without cross-file context. Therefore, if the same undefined name reported at multiple places in a file, we keep only the first occurrence. this work, focus instantiating
approach in the four popular programming languages, while the idea can be generalized other languages in principle. Specifically, for Python, we use Pylint* to detect undefined names; for Java, we use javac compiler; for TypeScript.
tsc compiler;
"Overlapping between CROssCODEEVAL. and data used to pretrain code LMs. Phttpa://tat httt//eeesittee gi thub. io/ siteeesitter// 'Pylint is static code analyzer for Python: https: :ttpps://pp0/ http:///l.....raateeooo.o..o.
3

---

for C#t, we use esc compiler from the mono" image. We use tree-sitter to identify full statements to construct reference completions Python. For Java, TypeScript, and C#,, we consider statements ending with either ";", "{", or "}". See Appendix A for more details.
2.3 Post-processing and Quality Control
We designed series of rule-based and model-based post-processing filters to ensure the quality of the dataset. We filter examples if (1) fewer than N lines of code (lines not including import statements, where N 10, 20, 30, 5 for Python, Java, TypeScript, and C#) in the prompt, (2) too short (< 3 tokens), or long (> 30 tokens) reference. We exclude examples the references are found verbatim in any other source code file within the repository (i.e., cross-file). We further discard examples with duplicate references. The filtering steps collectively remove 15%-20% of the examples.
Moreover, ensure that the reference isn't predictably inferred solely from the current file (possibly owing to strong clues function names and comments), we feed the examples (input prompts) to starcoderbase matches. This step results in the removal of < 10% of the generated examples. As an ancillary benefit, this further safeguards that the examples are not seen by publicly available code LMs while CROSSCODEE VAL constructed based on repositories that do not overlap with the Stack and possibly other private pre-training datasets created prior to 2023. Finally, subsample of the resulting CROSSCODEEVALL and found that the dataset has a satisfactory quality to serve the goal of cross-file code completion. See Appendix B for more details.
model (Li et al., 2023) to complete the statement and remove the exact
perform human annotations
2.4 Dataset Statistics, Scope, and Future Extensions
Statisties We present the statis- ties of CROSSCODEEVAL in Ta- ble We use the StarCoder tok- enizer (Li et al., 2023) to compute the number of tokens.
Feature
Python
Java
TypeScript
C##
# Repositories
471
239
193
# Files
1368
745
779
642
# Examples
2665
2139
3356
1768
Avg. lines in prompt
90.6
106.7
116.5
71.1
Scope In addition to prompts and references, we include the code lines that follow the refer- ences from the original source code files in CROSSCODEEVAL. examples. Given the source code lines to the left (prompt or prefix) and right (suffix) of the references, CROSSCODEEVAL can be used to evaluate code LMs for their fil-in-the-miidle (FIM) capabilities (Bavarian et al., 2022).
Avg. tokens in prompt
938.9
995.3
944.9
584.1
Avg. lines in reference
1.0
1.1
1.7
1.7
Avg. tokens in reference
13.2
14.5
17.4
12.5
Table 1: CROSSCODEEVAL,
stassCoo statistics.
Future Extensions CROSSCODEEVAL currently supports four popular languages. As method generalizable, CROSSCODEEVAL can potentially be extended to other languages. Additionally, we advise future code LM pre-training datasets should explicitly exclude CROSsCODEEVAL minimize the effect of memorization.
3 Experiments
3.1 Models
We benchmark CROSSCODEEVAL with popular public and proprietary large language models.
CodeGen (Nijkamp et al., 2023b,a) is a series of generative code LMs. CodeGen supports left-only context. 5odeGen2.5 notably supports fil-in-the-middle and further improves the performance via multi-epoch training. We benchmarked CodeGen models at various sizes from 350M to 16B.
StarCoder (Li et al., 2023) is generative multi-query based code LM with 15.5B model parameters trained on The Stack dataset (Kocetkov et al., 2022). supports up to 8k tokens. We also benchmarked its base version with at varied sizes: IB, 3B, and 7B.
Mono is open-source implementation of the NET Framework: https: mono project.co com/

---

"MMaggret utils test functions
import itertools
inport
isport
In-file Context
pest_case. convert_ canol_ caanal_t_ssnaallle snake(()
"Snake(asedStrinng
Prompt
Casetonverter.
In-file Context
Here relevant fragnents other files: fragnent be raigret/utils.py
class CaseConverter:
@staticrethod
Retrieval Context
def
str) -> stri
@staticrethod
Context
def snake to ca string: str) +> str:
estaticnethod
Retrieval Prompt
snake titletsnakecaadd string:
Retrieval Conteat
fragnents
below code fragient can be found in:
raigret/utils-pp
Retrieval w/ Ref. Context
@staticnethod
def canel_te to to_snake(canelccased string: str] ~> str: return re. ((te("( "_", carelcased string) Lower()
In-file Context
Retrieval w/ Ref. Context
carel_ to (aakeaa)
Completion
Retrieval Ref. Prompt
Figure 3: An illustrative example showcase the of in-file context, retrieved cross-file context, and retrieved context using reference in prompts. While the baseline prompting uses in-file context only, "Retrieval" and "Retrieval w/ Ref." prompting retrieved contexts by prepending them to the in-file context.
GPT-3.5-turbo (Ouyang et al., 2022) one of the most powerful models developed by OpenAL. trained with comprehensive text and code data and supports up to 4k sequence length. Its model weight remains proprietary and accessible exclusively via APIs.
3.2 Evaluation Metrics
In evaluating the performance of code language models, we report performance in two main categories: code match and identifier match (Ding et al., 2022).
Code Match The code match metric directly compares the generated code with the reference and measured using exact match (EM) and edit similarity (ES). These metrics help to assess the overall accuracy of the code completion process, taking into account elements such identifiers, keywords, operators, delimiters, and literals.
Identifier Match This metric evaluates the model's ability to predict the correct application programming interfaces (APIs). To perform this evaluation, we first parse the code and extract the identifiers from the model prediction and reference, resulting in two ordered lists of identifiers. We then compare the predicted identifiers with the reference and report the results in EM and F1 score.
3.3 Experimental Setup
Our evaluation framework is based on the Transformers (Wolf et al., 2020) library. All the experiments are conducted with the zero-shot setting, and no training is involved. We use the same set of hyperparameters for code generation across all models. We set the maximum sequence length to 2,048 for the CodeGen family, 4096 for GPT-3.5-turb,, and 8,192 for the StarCoder family. We use maximum generation length of 50 and the rest as the prompt.
We explore greedy search and nucleus sampling (Holtzman et al., 2020) with reranking (Hossain al., 2020). We found there is significant difference between the two, and present the greedy search results in the main paper and refer readers to Appendix D.2 for nucleus sampling.
We post-process model predictions to extract statements. 6 For Python, we iteratively parse the concatenation of prompt and n completion tokens (e.g., n 1,2, 1.2,....5000 50) until the sequence becomes
We apply the post- processing the references before calculating the evaluation metrics.

---

parseable (no syntax error) and the (n + 1)-th completion token is a newline character." ? For Java, TypeScript, and C#, we consider statements ending with ";", "(" and "J", instead of a new line.
Only In-File Context In standard practice, pre-trained language models are utilized to perform code completion in zero-shot manner by taking into account the provided code context. Following the practice, we conduct experiments using the code LMs (3.1), where they are provided code context from the current file. As shown in Figure 3, the baseline prompt includes only in-file context.
Retreived Cross-file Context Inspired by the effectiveness of the recently proposed retrieve-and- generate (RG) framework for repository-level code completion (Zhang et al., 2023), we adopt it for cross-file context retrieval. 8 In the RG framework, the retrieval database is constructed by iteratively scanning the files the repository and extracting contiguous M lines (in all our experiments, M = 10) of non-overlapping code fragments, which the candidates for cross- file context retrieval. The query for the retrieval is built using the last N lines (we set N = 10) of the in-file context. We use BM25 (Robertson et al., 2009) to calculate the similarity between the query and the candidates (cross-file context chunks), and use the top-5 similar code snippets as the cross-file context, see "Retrieval Context" in Figure 3. We consider maximum of 512 BPE tokens for such context, and the rest of the tokens will be truncated. Figure 3 illustrates" the retrieved context and the corresponding prompt for the model to complete. Given the in-file context as a query, the RG framework successfully retrieves the class definition of CaseConverter that is in another file for utilities. We further wrap the class definition into a template as code comment and use the frlss-file context. To build the retrieval prompt, we prepend the retrieved context to the in-file context.
Retrieval with Reference To quantify the upper bound impacts of cross-file context retrieved by the RG framework, we devise "retrieval with reference" for comparison. In this setting, we make use of not only the in-file context (as in standard retrieval setting) but also the reference to retrieve the cross-file file context. Specifically, the query constructed by using the last N lines of the concatenation of the in-file context and the reference completion, instead of the inlfle context only in the standard retrieval setting. We prepend the retrieved context (i.e. "Retrieval w/ Ref. Context") to in-file context to construct the prompt for this setting.
Note that the Retrieval w/ Ref. context could not be applied to the realistic code completion, as the reference completion is unknown. We use it as an estimation of the upper bound model performance with the RG framework. Also, the model's performance in this setting is not optimal, as it can still be limited by imperfect retrieval and the model's capability in making use of retrieved code, and we perform additional benchmarking and analysis on retrieval methods later in $3.5.
3.4 Results
We present results in Table 2 and additional results in Table 7. We see that all models perform poorly when the prompt includes only the in-file context. For example, the hest-performing StarCoder model at 15.5B size only reports 8.82% code exact match in Python. Even large code LM struggles to achieve promising performance in completing CROSSCODEEVAL samples with only in-file context since it could not provide sufficient clues for code completion. This shows the design of CROSSCODEEVAL that cross-file context is necessary to complete the code correctly.
The performance improves dramatically when the cross-file context is added to the prompts across all models and sizes. Figure 4 shows the significant improvements resulting from the inclusion of cross-file context in CodeGen and StarCoder models. Looking at Table 2, see that the StarCoder model reports up to 3.0x and 4.5x better exact code match when including retrieved and retrieved with reference context respectively. The results underline the limitation of existing datasets that only consider the in-file context to evaluate code LMs, making these datasets insufficient in reflecting models' best capacity in real-world scenarios. In contrast, CROSSCODEEVAL maintains the cross-file
We manually verified that this is required extract full statements in Python.
*We tried to use the code made publicly available by the authors of Zhang et al. (2023) but failed to execute them with the provided instructions. As result, we implemented the RG approach by ourselves. Note that, unlike Zhang al. (2023), we perform only 1-step retrieval augmented generation.
FFor the convenience of illustration, we remove the irrelevant code snippets from the retrieved context to avoid confusion. In practice, the retrieved context spans across a fixed length of lines, which includes useful information as well as its surrounding lines. More implementation details are in Appendix C.

---

contexts for code completion samples, providing resources to both identify the model's best capacity and analyze the model's behavior when seeing a more comprehensive context.
Code Match
Model
Python
Java
TypeScript
C#
EM
ES
EM
ES
EM
ES
EM
ES
CodeGen25-7B
7.73
59.34
10.43
62.05
7.81
57.56
4.36
58.99
+ Retrieval
14.52
64.40
16.88
64.35
12.57
60.08
13.01
63.86
+ Retrieval w/ Ref.
19.17
67.46
20.20
66.17
15.35
62.73
17.87
66.
StarCoder- 15.5B wow
8.82
61.08
9.96
63.25
6.35
51.22
4.47
59.80
+ Retrieval
15.72
66.28
17.48
66.10
8.31
44.87
13.57
65.00
+ Rettiieal w/ Ref.
21.01
68.66
19.92
67.75
11.02
46.67
20.08
67.97
GPT-3.5-turbo
4.88
52.58
12.30
63.52
6.38
53.78
3.56
56.48
+ Retrieval
10.77
54.92
19.12
65.61
10.94
55.83
11.82
62.40
+ Retrieval w/ Ref.
15.72
58.88
22.72
68.50
14.15
58.40
17.65
66.07
Identifier Match
Model
Python
Java
TypeScript
C#
EM
F1
EM
F1
FM
F1
EM
F1
CodeGen25-7B
14.26
46.02
16.60
51.43
12.46
47.75
7.69
33.81
+ Retrieval MAw.IAAee
22.96
53.68
24.03
55.48
17.85
51.27
17.36
43.56
+ Retrieval w/ Ref.
28.33
57.95
27.91
57.87
21.51
55.38
................................
21.78
47.63
StarCoder-Ie 15,5B
15.72
48.16
18.28
53.23
11.86
43.53
8.54
34.33
+ Retrieval
24.77
55.57
25.95
57.74
14.09
39.50
18.04
44.38
+ Retrieval w/ Ref.
30.24
59.46
29.73
60.47
17.55
42.18
24.38
49.09
GPT-3.5-turbo coceenonnn
10.09
39.18
18.93
52.52
10.76
44.78
5.77
30.25
+ Retrieval
17.37
44.43
26.74
56.57
16.69
48.15
15.44
41.24
+ Retrieval w/ Ref.
23.49
50.14
31.79
60.52
20.65
51.54
21.72
47.21
Table 2: Performance of various code LMs on CROSsCODEEVAL. "Reettrre and "Retrieval w/ Ref." mean we construct the prompt by prepending the retrieved cross-file file context retrieved with the prompt and the prompt + reference (see $3.3 for details). The performance with no cross-file context (first row in each section) is generally poor. When prompts are augmented with cross-file file context (middle row in each section), the performance increases significantly. The use of reference completion in formulating the query for cross-file file context retrieval (last row in each section) shows the upper bound of the retrieve-and -generate (RG) approach. Results of other models are in Table 7.
3.5 Analysis and Discussions
Improved vs. Degraded Code Completions Table 3 presents changes in the number of correct completions (based on exact match the references) across different prompt settings. The results suggest that all models follow a trend that the performance improves with better cross-file context (In-file- + Retrieval + Retrieval w/ Ref.). However, the variation of correct/incorrect generation is significant; for example, when changing from Retrieval to Retrieval w/ Ref. with StarCoder in CROSsCODEEVAL. Python, we see 327 correct generations changed to incorrect, and 468 generations changed the other way around. Upon manual inspections, we see that the retrieval of the correct cross-file context plays a huge role, as the quality of the retrieval directly correlates with whether the model is able to generate correctly. This effect is further enhanced by the fact that the retrieval happens in fixed lines of code that do not often follow code structure, making it difficult for the model to digest, especially at shot settings, echoing results from Zhang et al. (2023). This highlights that the current best models are still imperfect in leveraging extensive context to make better code completion. Further, it calls for additional studies in optimizing the retrieval methods for code: we show benchmarking with various retrieval methods later in the section.
7

---

Python
Java
Model
Ret. w/ Ref
Ret.
In-file
Ret.
In-file
Ret.
w/ Ref
CodleGen3350M
+72
+186
+257
+75
+149
135
+172
+182
+237
+145
+158
CodeGen- 16.1B
+183
1622
+332
-29
+444
+150
-133
+250
-209
+292
1311
1371
1233
1
CodeGen25-7B
+206
-172
+387
-906
+511
+223
-179
+361
-277
+432
+353
1440
+317
+:348
StarCoder
+235
192
+419
-327
+560
+213
-167
+374
-300
+426
+376
+468
+328
+-352
TypeSeript
C#
Model
Ret. w/ Ref
Ret.
In-file
Ret.
In-file
Ret.
w/ Ref
CodeeGen-350M
+93
+135
+175
+16
+58
+115
+58
+108
CodeGen- 16
+151
+237
+308
+32
498
+151
+226
+289
+95
+138
ColeGen25-7B
+262
226
+4222
341
+515
+77
62
+230
189
+316
+386
+134
+215
+275
StarCoder
+213
198
+279
241
+376
+79
68
+240
195
+-55
+264
+-338
+2227
+-110
Table 3: The numbers of correct code completions using different code generation models on the CROSSCODEEVAL. benchmark. "In-file" refers to the prompts being constructed only with in-file context, and "Retrieval" and "Ret. w/ Ref" refer to the prompts being constructed with the retrieved contexts described in $3.3.
Scalability of Model Performance Figure 4 visualizes how the performance of CodeGen and StarCoder scales w.r.t. model sizes. We see the
CodeGen StarCoder
CG-Retrieval
* CG EC--Retieeval Ref. Retrieval
Model Performance (Exact Match)
performance increases following the power law in all settings as expected (Kaplan et al., 2020).
20
However, again the performance far from per- fect even with the best performing model with the best context retrieved.
16
12
Locations of Retrieved Cross-file Context To identify relevant cross-file context, retrieve code snippets from other files of the repository. To understand which files contribute to the cross- file context, we further conduct a study on the retrieved code snippets. To analyze the code snip-
8
4
0
100
10a
pets retrieved for each prompt, we examine the files to determine whether they meet the follow- ing criteria: (1) they are imported by the target file, (2) they are located in the same directory as the target file, (3) they have a similar name to the target file (with filename sharing least one token, assuming snake-case or CamelCase style filenames), and (4) they include at least one API import within the project, similar to the target file. Our analysis shows that most the code snippets are sourced from files that are either from the same directory with the target file (Python: 49.0%, Java: 37.8%, TypeScript: 51.3%, C#: 51.7%), or have similar names (Python:33.44% Java:44.5% TypeScript: 24.9%, C#: 39%). We also observed that target files and cross cross-files often share at least one intra nntr--rroject API import statement. This result aligns with the findings of Zhang et (2023).
Model Parameter Size (Billion)
Figure 4: Performance of models in various sizes.
Identifier Overlap with Retrieved Cross-file Context Identifiers are significant part of pro- gramming language constructs that cover API mentions in source code. Therefore, we examine the distribution the retrieved cross-file contexts for examples in CROSSCODEEVAL that include mentions of identifiers that are also present in the references. In Figure 5, we show the distribution and the identifier exact match performance achieved by the best performing code LM, StarCoder. In general, it is evident that an increased ratio of identifier overlap results in higher performance,

---

0.5
identifee Execct Metch Score
500
0.5
Number Test Samples
500
0.4
400
0.4
400
300
300
0.3
200
0.2
Kdenfier
100
100
0.2
0.1
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
0.1
0.2 0.3 044 0,5 0.6 0.7 0.8 0.9 1.0
Ratio of Identifier Overiap
Ratio of Identifier Overlap
(a) Python
(b) Java
Number Samples
Ideettfler Exect Metch Score
Numbeer of "estt Samples
400
0.5
ldentiffer Exect Score
600
10.3
0.4
300
0.3
200
200
0.1
100
0.1
0.2
0.6
Ratio of Identifler Overiap
Ratio Identifler Overlap
(c) Tyrest
(d) C#
Figure 5: Distribution of the examples according to identifier overlap between the retrieved cross-file
context and the reference completion. We also show the corresponding identifier exact match scores.
Code Match
Model
Retriever
Python
Java
TypeScript
C#
EM
ES
EM
ES
EM
ES
EM
ES
CodeGen25-7B
7.73
59.34
10.43
62.05
7.81
57.56
4.36
58.99
Retrieval
BM25
14.52
64.40
16.88
64.35
12.57
60.08
13.01
63.86
Retrieval Ref.
BM25
19.17
67.46
20.20
66.17
15.35
62.73
17.87
66.14
Retrieval
UniXCoder
13.73
64.21
15.61
63.67
12.10
59.82
12.39
63.82
Retrieval w/ Ref.
UniXCoder
18.01
66.46
18.19
65.23
14.84
61.66
16.46
65.20
+ Retrieval
OpenAl ada
14.82
65.00
17.77
64.48
12.75
60.02
14.71
65.35
+ Retrieval Ref.
OpenAl ada
18.39
66.80
20.94
66.27
15.58
62.65
20.43
68.65
StarCoder- 15.5BB
8.82
61.08
9.96
63.25
6.35
51.22
4.47
59.80
Retrieval
BM25
15.72
66.28
17.48
66.10
8.31
44.87
13.57
65.00
+ Retrieval w/ Ref.
BM25
21.01
68.66
19.92
67.75
11.02
46.67
20.08
67.97
Retrieval Retrieval w/ Ref.
UniXCoder
15.87
66.07
16.83
66.09
7.87
44.67
11.93
63.90
UniXCoder
19.32
68.33
19.45
67.51
10.28
46.85
16.63
66.30
Retrieval
OpenAl ada
16.47
66.72
17.53
65.98
8.43
45.08
15.39
66.21
Retrieval w/ Ref.
OpenAl ada
20.53
68.50
21.69
68.11
11.83
47.31
23.49
70.56
Table 4: Evaluation results of various sparse and neural methods in retrieving cross-file context. Identifier Match results are in Table
demonstrating positive correlation. This calls for an investigation into retrieval techniques, with particular emphasis on key terms like identifiers for cross-file context retrieval.
CROSSCODEEVAL as Code Retrieval Benchmark The observations above (e.g., imperfect upper bound performance and identifier overlap) underscore the critical role of the code retrieval method. Given the strong dependency that the correct prediction requires an accurate retrieval of relevant CROSSCODEEWs
cross-file context, propose experiments with different retrievers from sparse (BM25 as we used the rest of the experiments) to neural (UniXCoder (Guo al., 2022) and OpenAl embedding" 1)). For UniXCoder,
code retrieval benchmark. We perform
We use text fext-enbedding-ada 000-000.

---

sequence length of 256 per 10-line chunk and for OpenAl embedding we use 8,000. We use the cosine similarity of the embedding of the prompt and the chunks to retrieve top 5 chunks.
Table shows the results with these retrieval methods. On one hand, we see BM25 provides a strong baseline and, in most of the cases, can outperform UniXCoder-basee retriever. On the other hand, retrieving with OpenAl's ada embedding is generally better than both BM25 and UniXCoder, especially for Java and C#. Nonetheless, the performance with the best performing retriever still suboptimal (< 20 EM in all languages), calling for future development of better code retriever.
4 Related Works
The advent of code language models (LMs) (Feng al., 2020; Ahmad et al., 2021; Wang et al., 2021; Guo et al., 2022) have bolstered the automation of software engineering applications. Among them, code completion has got the most attention, and as a result, generative Al powered by large language models for code (Chen et al., 2021; Xu et al., 2022; Wang and Komatsuzaki, 2021; Black et al., 2021, 2022: Nijkamp et al., 2023b; Fried et al., 2023; Li et al., 2022: CodeGeeX, 2022: Allal al., 2023; Li et al., 2023; Nijkamp et al., 2023a) has become a reality. Benchmark datasets have been playing a pivotal role in advancing the field of generative AI for code. A large pool of recent works (Chen et al., 2021; CodeGeeX, 2022; Austin et al., 2021; Athiwaratkun et al., 2023; Cassano et al., 2023;
Hendrycks et al., 2021: Raychev et al., 2016; Lu et al., 2021; Allamanis and Sutton, 2013; Puri et al., 2021; Husain et al., 2019; Clement et al., 2021; Ding et al., 2023; Wang et al., 2023; Lu et al., 2022) developed benchmarks to facilitate the evaluation of code LMs. These benchmarks typically assess code completion ability given in-file context code prompts containing code snippets from current files (where the user is writing code). Therefore, the capability of these code LMs to generate code that requires software repository-Iever context has been left unexplored until recently.
few recent works proposed repository-level code generation frameworks and benchmarks (Shri- vastava et al., 2023; Ding et al., 2022: Pei et al., 2023; Zhang et al., 2023). While these works share high-level insights with CROSSCODEEVAL, highlighting the importance of cross-file context, their focus is mainly on proposing new approach to incorporate such contexts, and datasets are collected to evaluate their own approaches rather than being carefully crafted as a benchmark to evaluate the code LMs in general. For example, Shrivastava et al. (2023) and Ding et al. (2022) only collect data for one single programming language, and Pei et al. (2023) narrows the completion scope to only function arguments. As comparison. CROSSCODEE EAL comprehensively includes four different programming languages (Python, Java, Typescript, and C#) and targets evaluating the general code completion capacity of code LMs rather than a specific type of application. REPOEVAL (Zhang et al., 2023) is a concurrent work building repository-leve code completion benchmark in Python, constructed from 16 GitHub repositories. These repositories are limited in domain (mainly academia/research work), some of them overlap with popular code pre-training datasets (such The Stack (Kocetkov et al., 2022)), and some with permiissie licenses. In contrast, CROSSCODEEVAL is derived from a diverse pool of permissively licensed GitHub repositories in 4 popular languages ($2.4). Furthermore, CROSSCODEEVAL does not overlap with The Stack avoid data leakage, minimizing potential memorization issues during evaluations.
5 Conclusion
We introduce CROSSCODREVAL. a diverse and multilingual benchmark for cross-file code completion. CRossCodEEVal necessitates cross-file contextual understanding to complete the code accurately. We use cattic-naalysi--ar
method to identify cross-file context usages in code, and take steps to ensure the dataset is of high quality and has minimal data leakage with the pre-training dataset of popular code LMs. We experiment with popular code language models and results show that the inclusion of cross- file context significantly improves their accuracy in code completion, demonstrating that CROSSCODEE VAL is an effective benchmark assessing cross-file code completion capabilities. Moreover, even the por-frrfonn model with the best retrieval method still exhibits great room for improvement, highlighting the need for further advancements in leveraging extensive context for code completion and better code retriever. In both directions, CROSSCODEEVAL stands as a pivotal benchmark. We envision CROSSCODEEVAL could fill in the gap of evaluating code completion that requires cross-file context and promote future research in all dimensions in this direction.
10

---

Acknowledgments
We would like to thank Ramana Keerthi and Akhilesh Bontala for their help in data collection, and Bryan McWhorter for various legal consultations.
References
Ahmad, W., Chakraborty, S., Ray. B., Chang, K. K.W., 2021. Unified pre-training for program under- standing and generation, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguisties, Online. pp. 2655-2668. URL: https::t//// httpp://aaaaatthllg.o.oo.ooooe/e org//021. naacl maacl-main... 211, 1 11865/v1/2021 maacl-main 211.
Allal, L.B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C.M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al., 2023. Santacoder: don't reach for the stars! arXiv preprint
arXiv:2301.03986 URL: https:ttp/... /tt////axivv org/abs/////oooeooooooggoooossrr 230.203011.......
Allamanis, M., Sutton, C., 2013. Mining source code repositories at massive scale using language modeling, in: Proceedings of the 10th Working Conference Mining Software Repositories, IEEE Press. 207-216. URL: httph::t////// //dl. acm. org/ doi //ff//df/////// /10.//1....0 00..5555/22400 2487085 2487248822 127.
Athiwaratkun, B., Gouda, S.K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W.U., Wang, S., Sun, Q. Shang. M., Gonugondla, S.K., Ding, H., Kumar, V., Fulton, N., Farahani, A., Jain, S., Giaquinto, R., Qian, H., Ramanathan, M.K., Nallapati, R., Ray, B., Bhatia, P., Sengupta, S., Roth, D., Xiang, B., 2023. Multi-lingual evaluation of code generation models, in: The Eleventh International Conference on Learning Representations, URL: https: //tprr/eee/pereeee. BeteruumdddBBoeeeeeeeeeeeeeeee
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le. Q., et al., 2021. Program synthesis with large language models. ArXiv preprint abs/2 108.07732. URL: htt::htps::4 / arxiv. org/ /tabb//108.. 07732
Barr, E.T., Brun, Y., Devanbu, P., Harman, M., Sarro, F., 2014. The plastic surgery hypothesis, in: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, Association for Computing Machinery, New York, NY, USA. p. 306-317. URL:
https://doi. org/10 1145/2635863 2635898, doi:10. 1145/2635868 2635898.
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., Chen, M.. 2022. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255 URL: https:
//arxiv. org/ abs/ /207 14255.
Black, S., Biderman, S., Hallahan, E., Anthony, Q.. Gao, L., Golding, L., He, H., Leahy, C., McDonell,
K., Phang. J., Pieler, M., Prashanth, U.S., Purohit, S., Reynolds, L., Tow, J., Wang, B., Weinbach, S., 2022. GPT-NeoX-205B An open-source autoregressive language model, in: Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Language Models, Association for Computational Linguisties, virtual+Dublin, pp. 95 136. URL: https: // //aclanthology org/ 2022. bigacience- 9, doi:10. 18653/ bi bigscience-1.. 1.99
Black, S., Gao, L., Wang. P., Leahy, C., Biderman, S., 2021. GPT-Neo: Large Scale Autoregres- sive Language Modeling with Mesh-Tensorfie URL: https: btttps://do...e /rg/://///oooo///.//o/////// 521.////02222//..... 5297715, doi:10. 5281 2081//zznooo 5297715.
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.H., Zi, Y., Anderson, C.J., Feldman, M.Q., Guha, A., Greenberg, M., Jangda, A., 2023. Multipl-e: scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering 49, 3695-3691. doi:10. 1109/TSE. 200..222223244666
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d..,, Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al., 2021. Evaluating large language models trained on code. ArXiv preprint abs/2107.03374 URL: https::///// // arxiv. org/ aa/aab//222 /2107 03374.
11

---

Clement, C., Lu, S., Liu, X., Tufano, M., Drain, D., Duan. N.. Sundaresan, N., Svyatkovskiy. A., 2021. Long-range modeling source code files with eWASH: Extended window access by syntax hierarchy, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguisties, Online and Punta Cana, Domini- can Republic. pp. 4713-4722. URL: hetps://aclanthoolgg. org/ 2221/221.. amnlp-main.360 doi:10. Noi:10.18653// http://222.mmmmmm...m mma.:mai.mmmmmmm33mm8111 main.387.
CodeGeeX, 2022. http://////..aa///////aaaaaao//. http://hwwb.ccc//tt/////Ceeeeee
Ding, H., Kumar, V., Tian, Y. Wang, Z., Kwiatkowski, R., Li, X., Ramanathan, M.K., Ray, B., Bhatia, P., Sengupta, S., et al., 2023. A static evaluation of code completion by large language models. arXiv preprint wrXiv:2306.03203
Ding, Y., Wang, Z., Ahmad, W.U., Ramanathan, M.K., Nallapati, R., Bhatia, P., Roth, D.. Xiang, B.. 2022. Cocomic: Code completion by jointly modeling in-file and cross- file context. arXiv preprint arXiv:2212.10000 URL: aattps://arxxt org/ abs/ abs/2212. 2212. 10007.
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M.. Shou, L., Qin, B., Liu, T., Jiang, D., Zhou, M., 2020. CodeBERT: A pre-trained model for programming and natural languages, in: Findings of the Association for Computational Linguisties: EMNLP 2020, Association for Computational Linguistics, Online. pp. 1536-1547. URL:
org/2020.
findings- emnlp. 139, doi: 10. 18653/v1/2022/ findings emnlp 139.
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F,, Zhong, R., Yih, S., Zettlemoyer, I., Lewis, M., 2023. Incoder: A generative model for code infilling and synthesis, in: The Eleventh International Conference on Learning Representations, URL: https://openrewt hitt/formmiddhhw---.
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., Yin, J., 2022. Unixcoder: Unified cross-modal pre-training for code representation, in: Proceedings the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7212-7225
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., Steinhardt, J., 2021. Measuring coding challenge competence with APPS, in: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). URL: https: http::ttttss//eeeeeeeeeee http:////epeeeeeeeeee/ee wttp://oooooomddddd333000...
Holtzman, A., Buys, J., Du, I.., Forbes, M., Choi, Y., 2020. The curious of neural text degeneration, in: International Conference on Learning Representations. URL: https: //openreview Cetpenrrvvee..ne//trumm//
Hossain, N., Ghazvininejad, M., Zettlemoyer, L., 2020. Simple and effective sttreve--dditrenne text generation, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online. pp. 2532-25338 URL: https:
//aclanthology .arey..rrgg//e 20000/222..022 acl - main.. 228, doi:10. 20053/v1/2020 acl aal-main..22
Husain, H., Wu, H.H., Gazit, T., Allamanis, M., Brockschmidt, M., 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint WwX.::19990099333 URL: https:
//arxiv. //aaaxivvrrg///ee/.e/../... abs/ hbbbb99..99936......
Kaplan, J., M.CandlCish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D., 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Kocetkov, D., Li, R., Allal, L.B., Li, J., Mou, C.,, Ferrandis, C.M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al., 2022. The stack: of permissively licensed source code. arXiv preprint 1.Xiv2221..15533 URL: https:://////.t //arxiv...ive///aa hargg/ab//2111 15533.
Le Goues, C., Nguyen, T., Forrest, S., Weimer, W, 2012. Genprog: A generic method for automatic software repair. IEEE Trans. Softw. Eng. 38, 54-72. URL: httph::tt///t :/tppp::://...i/ org/ 100 1109/TTT 2011. 104, doi:10. 1109/TSE TSS.TSEE..11111 2041.104.
12

---

Lee, K., Ippolito, D.. Nystrom, A.. Zhang, C., Eck, D., Callison- Burch, C., Carlini, N., 2022. Deduplicating training data makes language models better, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics. Dublin, Ireland. pp. 8424-8445. URL:
org/2022. acl- -oogg....... 577, doi:10. 20153/v112022 acl-long.
Li, R., Allal, L.B., Zi. Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li. J., Chim, J., et al., 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.0116 URL: http::tppp.... http://///aaaa..ooo/oo////////// wttp.//.www..aa//2....... 06161.
Li, Y. Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling. J., Gimeno, F., Lago, A.D., et al., 2022. Competition-leved code generation with alphacode. ArXiv al///bbs/2223./ preprint abs/2203.077 0.00314788444 14. URL: tetttp:://arste http://.rrrrbb///...33 07814.
Lu, S., Duan, N., Han, H., Guo, D.. Hwang. S.w., Svyatkovskiy, A., 2022. ReACC: A retrieval- augmented code completion framework, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computa- tional Linguistics, Dublin, Ireland. pp. 6227-6240. URL: http::/:///aaaaaalloosseo org/ 2022.
acl-long. 431, doi:10. (08866//1/2000 acl www..l.oooo--..eeeeeeeeee 431.
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang. D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M. GONG, M., Zhou, M.. Duan, N., Sundaresan, N.. Deng, S.K. Fu, S., LIU, S.. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation, in: Thirty-fiith Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). URL: https: //openreview wwtp:eewwweeeeefrrrueeeee www.tnenurrrr... wtw.rmmmmdd110000
Nijkamp. E., Hayashi, H., Xiong, C., Savarese, S., Zhou, Y., 2023a. Codegen2: Lessons for training Iims programming and natural languages. arXiv preprint arXiv:2305.02309 URL:
https: : //tp///rvvvvvo/////e org/ abs/a/ss///a 2005. 02309.
Nijkamp. E., Pang. B., Hayashi. H., Tu, L., Wang, H.. Zhou, Y., Savarese, S., Xiong, C., 2023b. Codegen: An open large language model for code with multi-turn program synthesis, in: Interna- tional Conference on Learning Representations, URL: https:// http::////peeeierr/eeee net/forummid= iaYcJKpY2B_.
Ouyang, L., Wu, J., Jiang. X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P.F., Leike, J., Lowe, R., 2022. Training language models to follow instructions with human feedback, in: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 17730-277744 URL: https http:///cco/ee/e/seeeee/../es.ee neurips. . cc/ wecppaper_ffee wiweffilss//eeeeeff Dalles/paper/2000 Paper- Sonferrncce.. .df.
Pei, H., Zhao, J., Lausen, L., Zha, S., Karypis, G., 2023. Better context makes better code language models: A case study on function call argument completion, Proceedings of the Thirty- Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI Press. URL: htttps://del orgooogg///o 10. 1609 aaai. v3714. 25653, doi:10. 1609 aaai. v3714. 25653.
Puri, R., Kung, D.S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., Thost, V., Buratti, L.,, Pujar, S., Ramji, S., Finkler, U., Malaika, S., Reiss, F., 2021. Codenet: A large-scale Al for code dataset for learning a diversity of coding tasks, in: Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). URL: https: httpp:/:////eeeeiieeee htt.:/oorommii--662B...o.o.
Raychev, V. Bielik, P., Vechev, M., 2016. Probabilistic model for code with decision trees, in: Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, Association for Computing Machinery, New York, NY, USA. p. 731-747. URL: atttps://dol wtt.:/i..........0.............. 11450....11111 /084/////999900 2984041, 1oi:001145/2000 2984041.
13

---

Robertson, S., Zaragoza, H., et al., 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval 3, 333-389.
Shrivastava, D., Larochelle, H., Tarlow, D., 2023. Repository-Ievel prompt generation for large language models of code, in: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, (Eds.), Proceedings of the 40th International Conference on Machine Learning, PMLR. pp. html.
31693-31715, URL: https: https://proceeddiiii
mlr
Wang. B., Komatsuzaki, A., 2021. GPT-J-6B: A Billion Parameter Autoregressive Language Model. https: /ttp::///tttttbt com
Langugg httpe//lllleeeeettaraaaeeee www.ttrrannaeeeeeee jaa.e-jaa..j.....
Wang. S., Li, Z., Qian, H., Yang. C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., Nallapati, R., Ramanathan, M.K., Roth, D., Xiang, B., 2023. ReCode: Robustness evaluation of code generation models, in: Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguisties, Toronto, Canada. pp. 13818-13844 13843. URL: httpp:://cllie hatss////aaaaahlooo/a// org/ 2023. acl acl-long. 773, doi:10. 18653vv1 20233 acl -
Wang. Y., Wang. W., Joty, S., Hoi, S.C.,, 2021. CodeT5: Identifier-aware unified pre-trained encoder- decoder models for code understanding and generation, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic. pp. 8696-8708 URL: hatas///olclaaaooggo/.. org/2021. emnlp p- main. 685, doi:10. 1ww111816131000 emnlp-mae main. 685.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A., 2020. Transformers: State- of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online. pp. 38-45. URL: https://aclanthoolggy org/2020 emmlp-demos.e doi:10 10863/////000000 emnlp-demos........--o...- demos.6 6.
Xu, F.F. Alon, U., Neubig, G., Hellendoorn, V.J., 2022. A systematic evaluation of large language models of code, in: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, Association for Computing Machinery, New York, NY, USA. p. 1-10. URI.:
https://doi. http:://ddooor//1.... 1145/3520312 353486223 doi:10.
Zhang, F., Chen, B., Zhang. Y., Liu, J., Zan, D., Mao, Y., Lou, J.G., Chen, W., 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint 1XXv:2303.125500 URL: https:://// //arxiv....vv www.ai.ow/avaaaaeeeeeeeeeoo orggabs//3//00 12570.
14

---

CROSsCODEEVAL: A Diverse and Multilingual Benchmark for Cross-File Code Completion
Appendices
A More Details of CROSSCODEEVAL Generation
A.1 Python
We presented Python example in Figure 2, Section 2.2 in the main paper. Here describe the static analysis part detail. Recall that the purpose static analysis is to detect undefined members the modified file whose imports replaced by empty classes. Hence, there's need to include project- level information at this stage, and we only run Pylint on the standalone file. We restrict the output types to be error only and ignore other types such as warnings and coding conventions. In the example in Figure 2, Section 2.2, the Pylint output for the modified file "test_utile py :0 E1101: Cl ass "CaseConverten all the errors of type E1101 and use them to identify locations of cross-file context usage.
has no camel__ to "oaaeee member (no-member)". We collect
A.2 Java
Instead of using static analysis tool, we directly leverage Java default compiler to extract the undefined member functions in the modified file, whose belonging class, originally imported from another project file, is replaced by an empty class. Figure shows the work flow the cross-file method extraction. Specifically, the calls of the cross-file file function. An illustration is provided in Figure
parse the errors with the A mark in the modified code locate
A.3 TypeScript
We go through the following steps to create examples from TypeSeript repositories.
1. Extract import statements from source files (files with "Its" or "Isxx" extension) and identify the dependencies. For example, file fileA. having the import statement import module cadule_name name from < ....feeBB.tt. depends on. .//fileB, ts. (../fileB.. ts is cross-file for fileA.ts)
2. Identify the cross-file lines by modifying the source code. followed by compiling the project using tsc."! 1I We make the following two types of modifications.
Removing the import statement and checking the compilation output message for eerror TS2304: Cannot find name 'X².".
Removing the import statement and adding a dummy class "class module_nane O and checking the compilation output message for "error TS2339: Property 'X' does not exist on type 'Y'.".
3. From the compiler error message, we use the line numbers to form references for examples.
A.4
C#
For C# repositories, the example generation procedure follows steps. In the first step, we pick source code file from a C# repository and replace the original class with dummy one. Then, we run the C# mono compiler on all C# files and gather the compilation errors. In the next step. we compare the compiler еттоr messages before and after the substitution in step-1 to locate the cross-file references based on the incremental errors, and extract the line numbers for references in CROSsCODEEVAL. examples. We repeat the process for each source code file within a repository.
15

---

Original file
Modifled file
package con. processors; inport con.utic. wonls. CaseConverter;
package con. processors: peperrt-corrtiitseeeeeeeee class CaseConverter()
Step 1: Empty class substitution
public class TextProcessor(
public class TextProcessor(
public static String ()nvertText()
public static String convertText()
String ""nnkeCasedSSrring'
String "snekeCasedSSrinng"
CassConverter c = new CaseConverter(); c.CamelTonnakeaa
CaseConverter c new CaseConverter(); CCamelTTSnaaee(a)
Step Java Compiler
cross-file falss-file context
Step Backward mapping
undefined
usage point
Figure 6: Java data creation: 1) given a original code file, we create an adapted version by replacing an imported class found in the project directory and generating dummy class accordingly. 2) Run javac on both versions. New javac error points of the modified code are actually the method calls of the original code, where the definitions in another file in the project. pair of prompt and ground truth completion is generated accordingly.
Original Code
Modified Code
package con, processocssss import con.utils.ccm Caseooieeeseeeeertteee
package pom.processors;
//inport con. won.utils..cm
class CaseConverter()
public class TextProcessorl
public class TextProcessort
public static String convertText()
String "SmakelasedStrinn";; CaseConverter
public static String convertText()
String a ""nakeCased$trinng"
CaseConverter();
CaseConverter
(aseConverter();
C. came (oSnake(a;;;
c. c.came (TSSaaaaSaa
Javac Output (Original)
Javac Output (Modified)
SoctProcessor.jvvee error: package con.utils does not exist irport con con.utils,,o wam.uutils,.esecomee CaseConverterr;
TextPracessor. java:9: cannot find symbal c. L.camelToSnake(al:
TextProcessor.jvva error: cannot find synbol CaseConverter new Casetonverter();
symbol: nethod (String) location: variable type CaseConverter
symbol: class CaseConventer
location: class TextProcessor
TextProcessor.j error: cannot find synbol Casetonverter new Casetonverter();
symbol: class CaseConverter
location: class TextProcessor
Prompt
Ground Truth
package com.processors;
inport con. uon.u. CaseCoveesse
public class TextProcessor(
public static String ()
camelToSnake
String ""nakeCasedString";
CaseConverter c new
CaseConverter());
c.
}
Figure 7: By comparing javae errors original Java code and modified code, we find the cross-file entities and create the prompt and ground truth.
B Human Annotation for Quality Control
To assess the quality and further improve CROSSCODEEVAL. conducted human annotation. We randomly sampled 100 and 50 examples from CROSSCODEEVAL Python and Java sets, respectively. Six authors annotated example each, and each example was annotated by two annotators with significant experience in the targeted language. Three questions were asked in the annotation:
Does the reference contain any name that requires looking at the associated cross-file? [Pay attention to: while answering, think if looking at the associated cross-fill helps you knowing
16

---

more info about a name mentioned in the reference.] Choose from A: Yes. / B: No, I already know everything about all the names mentioned in the reference. / C: Reference does not contain any such name.
2. Can you predict the reference given only the current-file context? [Pay attention to: while answering. think if you can guess the reference given the current-file context.] Choose from A: No, I cannot predict. / B: possibly can predict but it may not exactly match the reference. / C: can predict the reference.
3. Will you prefer to remove the example from the dataset? [Pay attention to: Is the example good enough to be included in the dataset? Use your best judgment to make decision.] Choose from A: No. Maybe. / C: Yes.
Q1
Q2
Q3
Python 98%
89%
88%
Java
100%
84%
76%
Table 5: Agreement scores for the annotation questions in Python and Java, respectively.
Q1
Q2
Q3
A
B
C
A
B
C
A
B
C
Python
99%
1%
0%
88.5%
9.5%
2%
91.5%
6%
2.5% 7%
Java
100%
0%
0%
88%
10%
2%
80%
13%
Table 6: Distribution of annotation answers to each question in Python and Java.
We calculated the agreement s2ore 12 and summarize the distribution of the annotations in Tables & 6. Overall, we see great agreement scores in most of the questions in both languages, suggesting annotators have consensus these questions. Looking at the distribution of annotation answers, we see that in almost 100% cases, the references contain names that necessities cross-file file informa- tion (QI), and only at 2% that the reference can be predicted with only current-file context (Q2). Both together suggest that CROSsCODEEVAl. serves its purpose to be dedicated cross-file code completion benchmark that accurately reflects model's capability cross-file context understanding. Besides, we see 2.5% and 7% examples that annotators think should be removed from the dataset (Q3). closer look reveals that many of such examples contain long strings in the reference, e.g.. sendCle i entSystemMessage (" ("Available scripts:") ily and also led to disagreement between annotators. We plan to improve the dataset by filtering out examples with long string in the reference in the next data revision.
which cannot be predicted eas-
C Satrieve-and-Generee Modeling Details
We present Figure illustrate more details regarding the Retrieve-and- (RG) framework.
Cross-file Context Retrieval As we have introduced in Section 3.3, the query will be constructed using the in-file context, i.e., the last 10 code lines before the cursor position, and the goal is to find the relevant cross-file context similar operation of the deposit that has been implemented in the class Bank. The code snippets in other files are chunked as code snippets with 10 lines as candidates for similarity calculation, and the most similar candidate of the in-file context turns out to be lines 5-14 in "Relevant Cross-file Context" of Figure 8. Since the in-file context is left context and the target completion is the following code, correspondingly, the retrieved context will be the following lines of the similar candidate, i.e., line 15-23 in "Relevant Cross-file Context". We can see that line 15 gives direct hint that helps with the prediction.
Cross-file Context Retrieved w/ Reference As an upper bound estimate, the query for the oracle context will be constructed by the last 10 lines of concatenating the in-file context and the target completion. Consequently, the most similar candidate is lines 8-17 in "Relevant Cross-file file Context",
12 Givvv we only have limited examples and annotations per example, and the annotation is relatively sparse. we didn't 't include other inter-coder agreement metrics like Krippendorff's Cohen's
17

---

In-file Context
from getpass import getpass
import S0L_Control__unce Funnttions as SCF
def Bankassedopiionsscccc.oooii//
while True:
print("Select option from below")
print("1. Deposit")
print("2.. Withdraw")
print("3. Details")
print("4. Show Statements")
print("S. Send Money")
print("6.. Exit")
choice (innpuutt(EEnter your choice: "))
if choice = 1:
be deposited: "))
[CURSOR_POSITION]
Target Completion
SCF. Transactiio(myydb, acc_no, amount)
Relevant Cross-file Context
# Local File: ./Progran.py
class Bank: acc_count int(0)
def (self, name, balance):
pass
def BankassedDptions(ecc_noiiii
while True:
prSeeeet option below")
print("1. Deposit")
print("2. Withdraw")
10
print("3. Show Details")
11
print("4. Exit") choice int(input("Enner your choice: "})
12
13
choice
14
amount int(input("Enttr anount to be deposited: "))
15
SCF. Transaction(mydd, acc_no, amount)
16
elif choice 2:
17
amount int(inptt("Entte anount to be withdrawn: "))
18
Bank.withdrawwmydd,, acc_no, -anount) choice 3:
20
SCF. OCF.PrintTabte(mydt "Account", acc_no)
21
elif choice 4: print("\n\nn))
23
Bank.EntryOptions((
Figure 8: An illustration demonstrating the context retrieval.
which, different from the cross-file context, will be directly used as the oracle context, since the query already contains the target completion rather than just left context. Again, it includes the direct hint, line 15, for the code completion.
D Additional Evaluation Results and Ablations
D.1 Evaluation with Additional Models
Beyond the results in Table the main paper, we further evaluate more code LMs, including more variants of CodeGen and StarcoderBase. The results are shown in Table 7,
18

---

Code Match
Model
Python
Java
TypeSeript
C#
EM
ES
EM
ES
EM
ES
EM
ES
CodeGen-350M
2.70
52.94
3.51
54.92
2.77
46.60
0.90
42.64
+ Retrieval
6,98
56,91
6.97
56.87
4.02
48.07
3.28
45.35
+ Retrieval w/ Ref.
9.64
59.31
8.04
57.75
5.21
50.25
6.50
48.58
CodeGen-2.777 2.7B
4.92
55.75
5.89
58.86
3.84
50.77
1,36
48.04
+ Retrieval
9.64
60.02
10.00
61.20
6.47
52.61
4.24
52.19
+ Retrieval w/ Ref.
13.28
62.65
11.73
62.50
7.81
54.79
7.35
54.78
CodeGen-6
5.40
55.41
5.19
58.89
3.75
49.87
1.47
45.34
+ Retrieval
10.58
60.36
8.88
61.06
6.14
51,58
4.64
47.53
+ Retrieval w/ Ref.
14.07
62.80
11.03
61.76
8.22
54.01
7.13
48.98
CodeGen- IB
6.87
57.64
7.01
60.49
4.50
52.24
1.81
45.28
+ Retrieval
12,46
62.66
11.69
62.16
7.06
54.11
5.54
48.25
+ Retrieval w/ Ref.
16.66
65.31
13.65
63.50
9.18
56.54
8.54
49.63
StarCoderBase IB
0.19
54.78
0.19
57.17
0.06
42.00
0.11
57.65
+ Retrieval
8.18
60.06
7.48
59.71
3.64
40.07
8.94
62.32
+ Retrieval w/ Ref.
12.83
63.37
10.38
62.06
6.79
42.47
15.16
65.56
StarCoderBase-B3 3B
4.77
57.54
5.66
60.26
3,55
46.74
3.62
59.27
+ Retrieval
11.74
62.99
12.39
62.33
6.73
43.89
11.99
64.65
+ Retrieval w/ Ref.
16.66
65.91
14.91
64.83
9.62
46.39
18.04
66.86
SturCoderBase-7B
6.75
59.83
8.74
62.84
5.13
49.31
4.86
59.71
+ Retrieval
13.28
64.76
15.61
65.19
8.25
45.73
14.20
65.65
+ Retrieval w/ Ref.
18.95
67.41
18.70
67.76
11.95
47.91
20.64
67.95
Identifier Match
Model
Python
Java
TypeSeript
C#
EM
F1
EM
F1
EM
F1
EM
F1
CodeGen-3500
7.92
38.51
9.40
43.36
5.07
34.44
1.98
21.07
Retrieval
13.70
44.34
13.32
46.47
7.06
37.09
5.83
26.67
Retrieval w/ Ref.
17.49
48.14
14.68
47.90
8.85
39.75
8.88
31.27
CodeGen-2.77 2.7BB
11.14
42.42
12.62
47.94
7.51
39.71
3.96
24.92
+ Retrieval
17.22
48.96
18.19
51.43
10.91
42.96
8.03
31,85
Retrieval w/ Ref.
21.54
52,40
20.06
53.19
12.96
45.58
11.43
36.50
CodeGen-6.
10.92
42.22
12.48
48.38
6.76
39.13
3.90
23.20
Retrieval
17.90
49.41
17.25
51.79
10.31
42.29
8.26
30.01
Retrieval w/ Ref.
22.40
53.14
19.45
53.12
12.87
45.72
12.10
34.20
CodeGen-16. 16.
13.28
44,98
14,40
49.97
8.76
41.95
4.13
23.49
+ Retrieval
20.53
51.95
19.50
52.97
12.13
44.65
9.33
31.08
+ Retrieval w/ Ref.
25.37
55.76
22.21
54.98
15.14
48.02
................................
12.67
35.08
StarCoderBase-
6.79
39.70
7.71
45.18
4.71
33.58
4.02
30.32
Retrieval
15.91
47.87
15.57
49.64
8.13
33.45
13.01
39.38
Retrieval w/ Ref.
21.61
52.95
19.54
52.57
11.80
36.55
18.83
44.81
SturCoderBase-B 3B
11.48
43.63
14.12
49.13
8.76
39.18
7.30
32.55
+ Retrieval
19.81
51.66
21.18
53.03
11.83
37.95
16.06
42.45
+ Retrieval w/ Ref.
25.40
55.89
24.36
56.09
15.08
41.09
21.95
47.03
StarCoderBase-75
13.92
46.65
16.88
52.18
10.04
42.15
8.71
34.04
+ Retrieval
22.29
54.
24.92
56.70
13.92
40.25
18.16
44.55
Retrieval w/ Ref.
28.26
58.14
29.64
59.99
17.79
43.40
24.04
49.12
Table 7: Evaluation of additional code LMs on CROSSCODEEVAL (cf. Table 2)
19

---

D.2 Nucleus Sampling w/ Re-ranking
Though the main results in this benchmark are reported with greedy search, we further conduct experiments to explore the effects of sampling and reranking. To this end, we apply the nucleus sampling (Holtzman et al., 2020) then ween--og-likelihood reranking (Hossain et al., 2020) during the code generation. The experiments are conducted with temperature of 0.2 for the token probability scaling. For the nucleus sampling, we set the top-p to be 0.95, and for the sean-log-likellhowee reranking, we generate top-5 prediction under the sampling setting and compute the mean log- likelihood of each generation to pick the most probable prediction.
The results are shown in Table 8. Compared to the main results (Table 2 in the main paper), we notice the results are quite comparable, and the difference is marginal. The cross-file and oracle context bring equivalent improvement to the greedy search setting. The results empirically reveal that (1) the performance difference of sampling with re-ranking and greedy decoding is marginal, and (2) the cross-file context is helpful regardless of the sampling/search algorithms.
Code Match
Model
Python
Java
TypeScript
C#
EM
ES
EM
ES
EM
ES
EM
ES
CodeGen25-7B
8.14
59.72
10.47
62.54
7.90
57.69
3.90
59.73
+ Retrieval
14.60
64.56
17.63
64.49
13.32
60.35
13.29
64.55
+ Retrieval w/ Ref.
19.51
67.67
20.48
66.92
15.79
62.88
18.21
66.46
MlarCoder-15.55
8.93
61.43
10.66
63.97
6.05
51.95
4.64
60.52
+ Retrieval
15.68
66.70
17.58
66.73
8.76
45.78
14.03
65.70
+ Retrieval w/ Ref.
21.35
69.44
20.20
68.49
11.65
47.32
19.97
68.32
Identifier Match
Model
Python
Java
TypeScript
C#
EM
F1
EM
Fl
EM
F1
EM
F1
CodeGen25-7B
14.60
46.13
17.16
52.04
12,40
47.73
7.47
34.52
+ Retrieval
22.81
53.72
25.15
55.99
18.18
51.72
17.36
43.88
+ Retrieval w/ Ref.
28.78
58.20
28.94
58.98
21.99
55.53
22.12
47.78
StarCoder-15.5B
15.91
48.08
19.07
54.08
11,50
44.13
8.31
34.58
+ Retrieval
24.62
55.75
26.46
58.46
14.78
40.59
18.33
44.55
+ Retrieval w/ Ref.
30.77
60.01
30.53
61.07
18.15
42.90
24.15
49.26
Table 8: Performance of code LMs CROSSCODEEVAL with temperature-based nucleus sampling.
(cf. Table in the main body of the paper).
D.3 Additional Results of Code Retrieval
Table 9 presents identifier match results of various retrieval methods for cross-file context.
D.4 Qualitative Analysis
To illustrate the quality of the retrieved tross-file context and in which ways they are helping to maximize the code LMs capacity, we provide two qualitative examples in Figure 9 and 10.
In Figure 9, we can see that the cross-file context provides retrieves the code snippets with a similar context to the cursor position. By capturing the repetitiveness of the repository (Le Goues et al., 2012: Barr et al., 2014), the cross-file context helps the code LMs adapt the existing, repetitive coding patterns to complete the programs. Specifically, the retrieved context also defines a function named step (), with similar goal of generating a token, and provides a direct reference for completing the API call.
20

---

Identifier Match
Model
Retriever
Python
Java
TypeSeript
C#
EM
F1
EM
F1
EM
F1
EM
Fl
CodeGen25-7B
14.26
46.02
16.60
51.43
12.46
47.75
7.69
33.81
www.
Retrieval
BM25
22.96
53.68
24.03
55.48
17.85
51.27
17.36
43.56
Retrieval w/ Ref.
BM25
28.33
57.95
27.91
57.87
21.51
55.38
21.78
47.63
Retrieval
UnixCoder
22.74
53.13
22.67
54.63
17.43
51.04
16.63
42.98
+ Retrieval w/ Ref.
UnixCoder
26.90
56.53
25.57
56.69
20.74
53.79
20.36
45.74
+ Retrieval
OpenAl Al Ada
23.53
54.17
25.01
55.67
18.06
51.67
19.30
46.08
Retrieval w/ Ref.
OpenAl A Ada
27.95
56.59
28.33
58.39
21.57
55.07
24.73
51.51
15.5
15.72
48.16
18.28
53.23
11.86
43.53
8.54
34.33
Retrieval
BM25
24.77
55.57
25.95
57.74
14.09
39.50
18.04
44.38
Retrieval w/ Ref.
BM25
30.24
59.46
29.73
60.47
17.55
42.18
24.38
49.09
Retrieval
UnixCoder
20.41
46.66
19.26
47.30
15.55
45.91
15.61
38.86
+ Retrieval w/ Ref. UnixCoder
25.25
55.42
25.90
57.88
13.32
39.13
16.23
42.36
+ Retrieval
you
OpenAl Ada
25.55
56.23
26.65
57.68
14.15
39.99
19.98
46,45
Retrieval w/ Ref.
OpenAl Ada
29.64
58.64
30.86
60.73
18.12
42.77
27.96
53.53
Table 9: Identifier Match evaluation results various methods in retrieving cross-file context (cf.
Table 4).
Different from Figure 9, Figure 10 no longer retrieves the similar usage of predicting APIs, and rather, it retrieves the implementation details of the cross-file dependencies. Concretely, the cross-file context collects the member function, store_by_texe imported by the current completing file and instantiated as self. such cross-file context, the model hallucinates a wrong function usage that is not defined in class EntitySess ionttorage.
of class EntitySeesionnStorge which Without
contrast, when the cross-file is prepended to the prompt, the model successfully predicts the ground truth, empirically revealing the limitation of existing datasets by only feeding the current-file context, which will consequently underestimate code LMs' capacity.
E Limitations
Zero-shot Evaluation Our benchmarking was done in a zero-shot fashion. We didn't perform the few-shot study the max sequence length of most benchmarked models is quite limited for prepending additional examples to the prompt. Thus, the performance will be limited as the format of cross-fil file context is never seen by the model during both training and prompting. We hope that CROssCODEEVAl encourages future research to investigate methods for efficiently retrieving and incorporating cross-file context into the model.
Cross-file Context Retrieval Quality Prepending cross-file context to the prompt has shown significant improvement to the code LM's performance. However, as we have analyzed and identified in Section 3.5, the RG retrieval framework is perfect. Due its fixed length of context window and token-based similarity calculation, RG sometimes retrieves useless information and fails to help code LMs for better generation. As for future work, we are expecting a more advanced retrieval approach to replace RG for more accurate cross-file context.
Memorization Code LMs were trained on a vast amount of unlabeled code. There is no way we could ensure that all models didn't see the evaluation data in the past. We take our best effort by excluding popular packages from annotation (see Section 2. Despite that, we suggest researchers and practitioners be cautious in interpreting the results with potential memorization mind. Future research in incorporating cross-file context may also consider deduplicating the training data with CROssCODEEvAL e.g., through methods in Lee et al. (2022).
21

---

Cross- Context
the below code fragment can be found in: examples/constraints
def _intt__ (self, prompt, can_fol :oww)
init_ii
self.context self.new wwwpneewwwetteprrneeeeeoeo
self.can_follae
can_follow
def step(self);
Generate proposed token.
token = self.sample(LL,, proooslsselt.lccllyyyoommmmmaap
In-file Context
import Ulamppl as 11p import numpy as nр
class [nfillingg(1l.MMdde)]: def
(snif_ words):
super().__nit__B
self.s = pop())
self.ctx = self.new__come (ot.e//ootttttssffsss
self.nnm retfrraeaiingg_eeementt = [self. Ulama tokenize(w) for in words]
def start(self):
self.step()
def step(self):
# Generate token =
Ge0metric((.5)) + 1 for in range(n):
self.s += self. sampseam [CURSOR_ POSITION]
w/o Cross-file Context
with Cross-file Context
cateforical(self...tx.gge. (tt.-eeetttoe_poobs(()
Transformer (wt.f.csssseffff...
Figure Qualitative Example I: the cross-file context provides a similar usage of the predicting API.
22

---

Cross-file Context
below code fragment can be found in:
doccano mint/stoorages/entty..y
entities self self.sstrrge.ger learrrggegge__stttteeeeeeeeeeeee "sssstteeeettttiss")) see return entities.get [])
storre_y_teext(eele text: str. entities: List(Entity!) None:
#
current_entitics tallssstoragggee_stttteemmmm
#
current_entitee [text] entities self. stttorraeeeeeesstteeeeeeeeeeee
#
current_entities)
In-file Context
from typing import Dict, List
from doccano_mini..toragess
import EntitysessiooSttorgge
from doccano_ninisstorggessssepp import SteperrSessionStorage
class Mannedhhtyyyccoonoooooogeeeeeeee example_path "named_enn wtt.eennttttttroonntiioonnmmmoo
def _init_ _init_iieill (self, title: str) -> None: cuper()._init_(((
self. List[str] = []
self.entiiy_reposittor Entity$essionSttrrage))
self.steeper_reoosttrr StepperSessionSttrrage))
def deffie_entitt_typesssefff
def annotate(self, examples: List(Dict)] >> [ist]):
1f len(examples) == 0:
return []
types self. define
selected_ type st. steeeeecttoo an entity type", types)
step
[step] text = [xamples[sttep ("text")
entities self. self.entiityyrepositor.. f find (texx)
entities st_nerr____ aeter_aanntttteseeeeeeteee type, text, entities, key=text) self-entityyrppsstoyy..CCoooo (CURSOR POSITION]
w/o Cross-file Context
with Cross-file Context
save(ittities)
store store_by_text tey_texxteet,, entities)
Figure 10: Qualitative Example 2: the cross-file context provides the definition of the predicting API.
23


Reviews:
[{"review_id": "pGu1Auq1Yu", "reviewer": "Reviewer_K8CN", "summary": "", "questions": "", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The strengths of this paper lie in its meticulous approach to data collection and post-processing. Preliminary experiments have validated the effectiveness of the benchmark in measuring long-distance dependencies. As the field progresses, we anticipate an increase in the number of benchmarks in this area, which will be instrumental in assessing the actual performance of code completion models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "6WOTuxeHOs", "reviewer": "Reviewer_t83P", "summary": "", "questions": "", "limitations": "The authors do not have a section discussing the limitations of their work. This is another aspect of presentation that could be improved.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper contributes a benchmark that fills an evaluation need in the code completion community. It is difficult to assess how well models perform on code completion that requires knowledge of other files within a repository. A strength of the proposed benchmark is the care taken to ensure that it does not overlap with the Stack. I also thought that the authors showed careful attention to detail in how they constructed the dataset, specifically, in ensuring that the model can't infer the call from prior context, and in the careful human annotation effort for additional verification.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "jVwl2hyINd", "reviewer": "Reviewer_C4R7", "summary": "", "questions": "", "limitations": "Limitations and potential impacts are appropriately addressed.\n", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "An important problem. We definitely need code generation benchmarks that are\nlonger than one function.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "rcsEmiqjb9", "reviewer": "Reviewer_1pD2", "summary": "", "questions": "", "limitations": "Since the majority of ground truth consists of a single statement, it only evaluates the cross file context setting. It cannot test the competence of the model to complete the given code enough. Thus they should mention about this situation and, for the accurate evaluation, the evaluator should use the existing code completion benchmark together ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Cross File Scope: CROSSCODEEVAL's strengths lie in its cross-file scope, allowing for comprehensive evaluation of code language models in Python and Java. \n\n\n2. Automated Data Generation based on Static Analysis: The methodologies automate dataset generation through static analysis, reducing manual effort and ensuring a large and representative dataset. \n\n\n3. Caution for Data Leakage: They address data leakage concerns by focusing on recently created, non-fork repositories, avoiding overlap with existing models' training data. \n\n\n4. Quality Control Process: A quality control process is implemented, verifying absence of dataset overlap and discarding false positives through static analysis. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BhCvG0QhyI", "reviewer": "Reviewer_BzTx", "summary": "", "questions": "", "limitations": "1. Limited to Line/Statement completion: One weakness of the paper is that the CROSSCODEEVAL dataset only focuses on line/statement completion. This narrow focus on completing individual lines limits the dataset's potential applications and does not capture more complex code generation tasks, such as generating entire methods or functions. Expanding the dataset to include tasks beyond line completion would make it more versatile and applicable to a wider range of code completion scenarios.\n\n2. Lack of evaluation with newer models: Another weakness is that the evaluation section does not include newer models such as OpenAI GPT-3.5. As the field of code completion evolves rapidly, it would be beneficial to include evaluations with newer models to provide a more comprehensive assessment of the state-of-the-art performance on the CROSSCODEEVAL dataset.\n\n3. Lack of execution-based metrics: The paper mentions that the evaluation metrics used are Exact Match and Edit similarity. While these metrics can provide some insights into the correctness and similarity of generated code, they do not capture the execution-based aspects of code completion. Execution-based metrics, such as test execution or runtime behavior analysis, can provide a more comprehensive evaluation of code completion models by considering the functional correctness and performance of the generated code. Incorporating execution-based metrics into the evaluation would strengthen the assessment of code completion models on the dataset.\n", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Novel dataset: The paper's introduction of the CROSSCODEEVAL dataset fills a gap in the literature by providing a benchmark that goes beyond single-file code completion tasks. By focusing on real-world software development scenarios involving multiple files and cross-file dependencies, the dataset offers a more realistic evaluation environment.\n\n2. Evaluation of existing models: The paper conducts experiments using popular code language models like CodeGen and StarCoder. By evaluating these models on the CROSSCODEEVAL dataset with different levels of cross-file context, including no context, reference-guided context, and oracle reference-guided context, the paper provides valuable insights into the performance of state-of-the-art models in handling code completion tasks that require cross-file understanding.\n\n3. Potential for evaluating retrieval strategies: CROSSCODEEVAL dataset can be used to evaluate novel retrieval strategies aimed at improving code completion. Retrieval strategies can enhance the accuracy and efficiency of code completion by leveraging relevant code snippets or documents from the repository. By considering the cross-file context and exploring retrieval-based approaches, the dataset opens up possibilities for further research and development in code completion techniques.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper first analyzes the pitfalls of previous code completion evaluation setting (single file) and proposes CROSSCODEEVAL a benchmark dataset for code completion that requires cross-file contextual understanding. This dataset uses real-world, open-source repositories in Python and Java to create examples that strictly require cross-file context to complete the code accurately. To identify code fragments that require cross-file context, the paper proposes a static-analysis-based approach. In the revised version, it covers Python, Java, typescript and C# programming languages with well curated samples to ensure an unbiased, an proper training set. The evaluation covers CodeGen, CodeGen2, SantaCoder, and StarCoder code language models in different size.  Experimental results demonstrate that CROSSCODEEVAL poses a challenge when cross-file context is not provided to the state-of-the-art code language models. Performance significantly improves when the cross-file context is included in the prompt.

All reviewers agree on the acceptance of the paper, and highlight the importance of the problem and the new research opportunities such a benchmark will open.
In addition, the revised version of the authors tackles the minor criticisms and suggestions such as recent LLM and more programming languages.
Decision: Accept (Poster)