SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Improved vs. Degraded Code Completions Table 3 presents changes in the number of correct completions (based on exact match to the references) across different prompt settings. The results suggest that all models follow a trend that the performance improves with better cross-file context (In-file  $\rightarrow$  Retrieval  $\rightarrow$  Retrieval w/ Ref.). However, the variation of correct/incorrect generation is significant; for example, when changing from Retrieval to Retrieval w/ Ref. with StarCoder in CROSSCODEEval Python, we see 327 correct generations changed to incorrect, and 468 generations changed the other way around. Upon manual inspections, we see that the retrieval of the correct cross-file context plays a huge role, as the quality of the retrieval directly correlates with whether the model is able to generate correctly. This effect is further enhanced by the fact that the retrieval happens in fixed lines of code that do not often follow code structure, making it difficult for the model to digest, especially at zero-shot settings, echoing results from Zhang et al. (2023). This highlights that the current best models are still imperfect in leveraging extensive context to make better code completion. Further, it calls for additional studies in optimizing the retrieval methods for code: we show benchmarking with various retrieval methods later in the section.
Model	Python	Java
In-file	→	Ret.	→	Ret. w/ Ref	In-file	→	Ret.	→	Ret. w/ Ref
CodeGen-350M	+72	-68 +182	+186	-166 +237	+257	+75	-71 +145	+149	-135 +158	+172
CodeGen-16.1B	+183	-162 +311	+332	-259 +371	+444	+150	-133 +233	+250	-209 +251	+292
CodeGen25-7B	+206	-172 +353	+387	-306 +430	+511	+223	-179 +317	+361	-277 +348	+432
StarCoder	+235	-192 +376	+419	-327 +468	+560	+213	-167 +328	+374	-300 +352	+426
Model	TypeScript	C#
In-file	→	Ret.	→	Ret. w/ Ref	In-file	→	Ret.	→	Ret. w/ Ref
CodeGen-350M	+93	-85 +127	+135	-128 +168	+175	+16	-16 +58	+58	-51 +108	+115
CodeGen-16.1B	+151	-140 +226	+237	-218 +289	+308	+32	-29 +95	+98	-85 +138	+151
CodeGen25-7B	+262	-226 +386	+422	-341 +434	+515	+77	-62 +215	+230	-189 +275	+316
StarCoder	+213	-198 +264	+279	-241 +338	+376	+79	-66 +227	+240	-195 +310	+355
Table 3: The numbers of correct code completions using different code generation models on the CROSSCODEEval benchmark. "In-file" refers to the prompts being constructed only with in-file context, and "Retrieval" and "Ret. w/ Ref" refer to the prompts being constructed with the retrieved contexts described in §3.3.
Scalability of Model Performance Figure 4 visualizes how the performance of CodeGen and StarCoder scales w.r.t. model sizes. We see the performance increases following the power law in all settings as expected (Kaplan et al., 2020). However, again the performance is far from perfect even with the best performing model with the best context retrieved.
Locations of Retrieved Cross-file Context To identify relevant cross-file context, we retrieve code snippets from other files of the repository. To understand which files contribute to the crossfile context, we further conduct a study on the retrieved code snippets. To analyze the code snippets retrieved for each prompt, we examine the files to determine whether they meet the following criteria: (1) they are imported by the target file, (2) they are located in the same directory as the target file, (3) they have a similar name to
the target file (with filename sharing at least one token, assuming snake-case or CamelCase style filenames), and (4) they include at least one API import within the project, similar to the target file. Our analysis shows that most of the code snippets are sourced from files that are either from the same directory with the target file (Python:  $49.0\%$ , Java:  $37.8\%$ , TypeScript:  $51.3\%$ , C#:  $51.7\%$ ), or have similar names (Python:  $33.4\%$ , Java:  $44.5\%$ , TypeScript:  $24.9\%$ , C#:  $39\%$ ). We also observed that target files and cross-files often share at least one intra-project API import statement. This result aligns with the findings of Zhang et al. (2023).
Identifier Overlap with Retrieved Cross-file Context Identifiers are a significant part of programming language constructs that cover API mentions in a source code. Therefore, we examine the distribution of the retrieved cross-file contexts for examples in CROSSCODEEval that include mentions of identifiers that are also present in the references. In Figure 5, we show the distribution and the identifier exact match performance achieved by the best performing code LM, StarCoder. In general, it is evident that an increased ratio of identifier overlap results in higher performance,
Figure 4: Performance of models in various sizes.
(a) Python
(b) Java
(c) TypeScript
(d) C#
Figure 5: Distribution of the examples according to identifier overlap between the retrieved cross-file context and the reference completion. We also show the corresponding identifier exact match scores.
Model	Retriever	Code Match
Python	Java	TypeScript	C#
EM	ES	EM	ES	EM	ES	EM	ES
CodeGen25-7B	-	7.73	59.34	10.43	62.05	7.81	57.56	4.36	58.99
+ Retrieval	BM25	14.52	64.40	16.88	64.35	12.57	60.08	13.01	63.86
+ Retrieval w/ Ref.	BM25	19.17	67.46	20.20	66.17	15.35	62.73	17.87	66.14
+ Retrieval	UniXCoder	13.73	64.21	15.61	63.67	12.10	59.82	12.39	63.82
+ Retrieval w/ Ref.	UniXCoder	18.01	66.46	18.19	65.23	14.84	61.66	16.46	65.20
+ Retrieval	OpenAI ada	14.82	65.00	17.77	64.48	12.75	60.02	14.71	65.35
+ Retrieval w/ Ref.	OpenAI ada	18.39	66.80	20.94	66.27	15.58	62.65	20.43	68.65
StarCoder-15.5B	-	8.82	61.08	9.96	63.25	6.35	51.22	4.47	59.80
+ Retrieval	BM25	15.72	66.28	17.48	66.10	8.31	44.87	13.57	65.00
+ Retrieval w/ Ref.	BM25	21.01	68.66	19.92	67.75	11.02	46.67	20.08	67.97
+ Retrieval	UniXCoder	15.87	66.07	16.83	66.09	7.87	44.67	11.93	63.90
+ Retrieval w/ Ref.	UniXCoder	19.32	68.33	19.45	67.51	10.28	46.85	16.63	66.30
+ Retrieval	OpenAI ada	16.47	66.72	17.53	65.98	8.43	45.08	15.39	66.21
+ Retrieval w/ Ref.	OpenAI ada	20.53	68.50	21.69	68.11	11.83	47.31	23.49	70.56
Table 4: Evaluation results of various sparse and neural methods in retrieving cross-file context. Identifier Match results are in Table 9.
demonstrating a positive correlation. This calls for an investigation into retrieval techniques, with a particular emphasis on key terms like identifiers for cross-file context retrieval.
CROSSCODEEval as Code Retrieval Benchmark The observations above (e.g., imperfect upper bound performance and identifier overlap) underscore the critical role of the code retrieval method. Given the strong dependency that the correct prediction requires an accurate retrieval of relevant cross-file context, we propose to use CROSSCODEEval as a code retrieval benchmark. We perform experiments with different retrievers from sparse (BM25 as we used in the rest of the experiments) to neural (UniXCoder (Guo et al., 2022) and OpenAI embedding $^{10}$ ). For UniXCoder, we use a max
10We use text-embedding-ada-002.
sequence length of 256 per 10-line chunk and for OpenAI embedding we use 8,000. We use the cosine similarity of the embedding of the prompt and the chunks to retrieve top 5 chunks.
Table 4 shows the results with these retrieval methods. On one hand, we see BM25 provides a strong baseline and, in most of the cases, can outperform UniXCoder-based retriever. On the other hand, retrieving with OpenAI's ada embedding is generally better than both BM25 and UniXCoder, especially for Java and C#. Nonetheless, the performance with the best performing retriever is still suboptimal ( $< 20$  EM in all languages), calling for future development of better code retriever.
The advent of code language models (LMs) (Feng et al., 2020; Ahmad et al., 2021; Wang et al., 2021; Guo et al., 2022) have bolstered the automation of software engineering applications. Among them, code completion has got the most attention, and as a result, generative AI powered by large language models for code (Chen et al., 2021; Xu et al., 2022; Wang and Komatsuzaki, 2021; Black et al., 2021, 2022; Nijkamp et al., 2023b; Fried et al., 2023; Li et al., 2022; CodeGeeX, 2022; Allal et al., 2023; Li et al., 2023; Nijkamp et al., 2023a) has become a reality. Benchmark datasets have been playing a pivotal role in advancing the field of generative AI for code. A large pool of recent works (Chen et al., 2021; CodeGeeX, 2022; Austin et al., 2021; Athiwaratkun et al., 2023; Cassano et al., 2023; Hendrycks et al., 2021; Raychev et al., 2016; Lu et al., 2021; Allamanis and Sutton, 2013; Puri et al., 2021; Husain et al., 2019; Clement et al., 2021; Ding et al., 2023; Wang et al., 2023; Lu et al., 2022) developed benchmarks to facilitate the evaluation of code LMs. These benchmarks typically assess code completion ability given in-file context - code prompts containing code snippets from current files (where the user is writing code). Therefore, the capability of these code LMs to generate code that requires software repository-level context has been left unexplored until recently.
A few recent works proposed repository-level code generation frameworks and benchmarks (Shrivastava et al., 2023; Ding et al., 2022; Pei et al., 2023; Zhang et al., 2023). While these works share high-level insights with CROSSCODEEval, highlighting the importance of cross-file context, their focus is mainly on proposing a new approach to incorporate such contexts, and datasets are collected to evaluate their own approaches rather than being carefully crafted as a benchmark to evaluate the code LMs in general. For example, Shrivastava et al. (2023) and Ding et al. (2022) only collect data for one single programming language, and Pei et al. (2023) narrows the completion scope to only function arguments. As a comparison, CROSSCODEEval comprehensively includes four different programming languages (Python, Java, Typescript, and C#) and targets evaluating the general code completion capacity of code LMs rather than a specific type of application. REPOEval (Zhang et al., 2023) is a concurrent work building repository-level code completion benchmark in Python, constructed from 16 GitHub repositories. These repositories are limited in a domain (mainly academia/research work), some of them overlap with popular code pre-training datasets (such as The Stack (Kocetkov et al., 2022)), and some are with non-permissive licenses. In contrast, CROSSCODEEval is derived from a diverse pool of permissively licensed GitHub repositories in 4 popular languages ( $\S 2.4$ ). Furthermore, CROSSCODEEval does not overlap with The Stack to avoid data leakage, minimizing potential memorization issues during evaluations.
We introduce CROSSCODEEval, a diverse and multilingual benchmark for cross-file code completion. CROSSCODEEval necessitates cross-file contextual understanding to complete the code accurately. We use a static-analysis-based method to identify cross-file context usages in code, and take steps to ensure the dataset is of high quality and has minimal data leakage with the pre-training dataset of popular code LMs. We experiment with popular code language models and results show that the inclusion of cross-file context significantly improves their accuracy in code completion, demonstrating that CROSSCODEEval is an effective benchmark assessing cross-file code completion capabilities. Moreover, even the top-performing model with the best retrieval method still exhibits great room for improvement, highlighting the need for further advancements in leveraging extensive context for code completion and better code retriever. In both directions, CROSSCODEEval stands as a pivotal benchmark. We envision CROSSCODEEval could fill in the gap of evaluating code completion that requires cross-file context and promote future research in all dimensions in this direction.
We would like to thank Ramana Keerthi and Akhilesh Bontala for their help in data collection, and Bryan McWhorter for various legal consultations.
Robertson, S., Zaragoza, H., et al., 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval 3, 333-389.
Shrivastava, D., Larochelle, H., Tarlow, D., 2023. Repository-level prompt generation for large language models of code, in: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (Eds.), Proceedings of the 40th International Conference on Machine Learning, PMLR. pp. 31693-31715. URL: https://proceedings.mlr.press/v202/shrivastava23a.html.
Wang, B., Komatsuzaki, A., 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., Nallapati, R., Ramanathan, M.K., Roth, D., Xiang, B., 2023. ReCode: Robustness evaluation of code generation models, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada. pp. 13818-13843. URL: https://aclanthology.org/2023.acl-long.773, doi:10.18653/v1/2023.acl-long.773.
Wang, Y., Wang, W., Joty, S., Hoi, S.C., 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic. pp. 8696-8708. URL: https://aclanthology.org/2021.emnlp-main.685, doi:10.18653/v1/2021.emnlp-main.685.
Wolf, T., Debut, L., Sanh, V., Chaumont, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A., 2020. Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online. pp. 38-45. URL: https://aclanthology.org/2020.emnlp-demos.6, doi:10.18653/v1/2020.emnlp-demos.6.
Xu, F.F., Alon, U., Neubig, G., Hellendoorn, V.J., 2022. A systematic evaluation of large language models of code, in: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, Association for Computing Machinery, New York, NY, USA. p. 1-10. URL: https://doi.org/10.1145/3520312.3534862, doi:10.1145/3520312.3534862.
Zhang, F., Chen, B., Zhang, Y., Liu, J., Zan, D., Mao, Y., Lou, J.G., Chen, W., 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570 URL: https://arxiv.org/abs/2303.12570.
A More Details of CROSSCODEVAL Generation
A.1 Python
We presented a Python example in Figure 2, Section 2.2 in the main paper. Here we describe the static analysis part in detail. Recall that the purpose of static analysis is to detect undefined members in the modified file whose imports are replaced by empty classes. Hence, there's no need to include project-level information at this stage, and we only run Pylint on the standalone file. We restrict the output types to be error only and ignore other types such as warnings and coding conventions. In the example in Figure 2, Section 2.2, the Pylint output for the modified file is "test_utils.py:10:8: E1101: Class 'CaseConverter' has no 'camel_to(snake' member (no-member)". We collect all the errors of type E1101 and use them to identify locations of cross-file context usage.
A.2 Java
Instead of using static analysis tool, we directly leverage Java default compiler to extract the undefined member functions in the modified file, whose belonging class, originally imported from another project file, is replaced by an empty class. Figure 6 shows the work flow of the cross-file method extraction. Specifically, we can parse the new errors with the  $\wedge$  mark in the modified code to locate the calls of the cross-file function. An illustration is provided in Figure 7.
A.3 TypeScript
We go through the following steps to create examples from TypeScript repositories.
1. Extract import statements from source files (files with ".ts" or ".tx" extension) and identify the dependencies. For example, a file fileA.ts having the import statement import module_name from '../fileB.ts'; depends on './fileB.ts'. (./.fileB.ts is a cross-file for fileA.ts)
2. Identify the cross-file lines by modifying the source code, followed by compiling the project using tsc.<sup>11</sup> We make the following two types of modifications.
- Removing the import statement and checking the compilation output message for "error TS2304: Cannot find name 'X'".
- Removing the import statement and adding a dummy class "class module_name {}
  and checking the compilation output message for "error TS2339: Property 'X' does not exist on type 'Y'".
3. From the compiler error message, we use the line numbers to form references for examples.
A.4 C#
For C# repositories, the example generation procedure follows two steps. In the first step, we pick a source code file from a C# repository and replace the original class with a dummy one. Then, we run the C# mono compiler on all C# files and gather the compilation errors. In the next step, we compare the compiler error messages before and after the substitution in step-1 to locate the cross-file references based on the incremental errors, and extract the line numbers for references in CROSSCODEVAL examples. We repeat the process for each source code file within a repository.
<sup>11</sup>https://www.typescriptlang.org/docs/handbook/compiler-options.html
Figure 6: Java data creation: 1) given a original code file, we create an adapted version by replacing an imported class found in the project directory and generating a dummy class accordingly. 2) Run javac on both versions. New javac error points of the modified code are actually the method calls of the original code, where the definitions are in another file in the project. A pair of prompt and ground truth completion is generated accordingly.
Figure 7: By comparing javac errors of original Java code and modified code, we find the cross-file entities and create the prompt and ground truth.
B Human Annotation for Quality Control
To assess the quality and further improve CROSSCODEEval, we conducted human annotation. We randomly sampled 100 and 50 examples from CROSSCODEEval Python and Java sets, respectively. Six authors annotated 50 example each, and each example was annotated by two annotators with significant experience in the targeted language. Three questions were asked in the annotation:
1. Does the reference contain any name that requires looking at the associated cross-file? [Pay attention to: while answering, think if looking at the associated cross-file helps you knowing
more info about a name mentioned in the reference.]
Choose from A: Yes. / B: No, I already know everything about all the names mentioned in the reference. / C: Reference does not contain any such name.
2. Can you predict the reference given only the current-file context? [Pay attention to: while answering, think if you can guess the reference given the current-file context.]
Choose from A: No, I cannot predict. / B: I possibly can predict but it may not exactly match the reference. / C: I can predict the reference.
3. Will you prefer to remove the example from the dataset? [Pay attention to: Is the example good enough to be included in the dataset? Use your best judgment to make a decision.]
Choose from A: No. / B: Maybe. / C: Yes.
Q1	Q2	Q3
Python	98%	89%	88%
Java	100%	84%	76%
Table 5: Agreement scores for the annotation questions in Python and Java, respectively.
Q1	Q2	Q3
A	B	C	A	B	C	A	B	C
Python	99%	1%	0%	88.5%	9.5%	2%	91.5%	6%	2.5%
Java	100%	0%	0%	88%	10%	2%	80%	13%	7%
Table 6: Distribution of annotation answers to each question in Python and Java.
We calculated the agreement score $^{12}$  and summarize the distribution of the annotations in Tables 5 & 6. Overall, we see great agreement scores in most of the questions in both languages, suggesting annotators have consensus in these questions. Looking at the distribution of annotation answers, we see that in almost  $100\%$  cases, the references contain names that necessities cross-file information (Q1), and only at  $2\%$  that the reference can be predicted with only current-file context (Q2). Both together suggest that CROSSCODEVAL serves its purpose to be a dedicated cross-file code completion benchmark that accurately reflects model's capability in cross-file context understanding. Besides, we see  $2.5\%$  and  $7\%$  examples that annotators think should be removed from the dataset (Q3). A closer look reveals that many of such examples contain long strings in the reference, e.g., Chat.sendClientSystemMessage("Available scripts:");, which cannot be predicted easily and also led to disagreement between annotators. We plan to improve the dataset by filtering out examples with long string in the reference in the next data revision.
C Retrieve-and-Generate Modeling Details
We present Figure 8 to illustrate more details regarding the Retrieve-and-Generate (RG) framework.
Cross-file Context Retrieval As we have introduced in Section 3.3, the query will be constructed using the in-file context, i.e., the last 10 code lines before the cursor position, and the goal is to find the relevant cross-file context - a similar operation of the deposit that has been implemented in the class Bank. The code snippets in other files are chunked as code snippets with 10 lines as candidates for similarity calculation, and the most similar candidate of the in-file context turns out to be lines 5-14 in "Relevant Cross-file Context" of Figure 8. Since the in-file context is left context and the target completion is the following code, correspondingly, the retrieved context will be the following lines of the similar candidate, i.e., line 15-23 in "Relevant Cross-file Context". We can see that line 15 gives a direct hint that helps with the prediction.
Cross-file Context Retrieved w/ Reference As an upper bound estimate, the query for the oracle context will be constructed by the last 10 lines of concatenating the in-file context and the target completion. Consequently, the most similar candidate is lines 8-17 in "Relevant Cross-file Context",
<sup>12</sup>Given we only have limited examples and annotations per example, and the annotation is relatively sparse, we didn't include other inter-coder agreement metrics like Krippendorff's  $\alpha$  or Cohen's  $\kappa$ .
Target Completion
In-file Context   
from getpass import getpass   
import SQL_Control_Functions as SCF   
def BankBasedOptions(acc_no:int): while True: print("Select an option from below") print("1. Deposit") print("2. Withdraw") print("3.Show Details") print("4.Show Statements") print("5.Send Money") print("6.Exit") choice  $=$  int(input("Enter your choice:")) if choice  $= = 1$  .. amount  $=$  int(input("Enter amount to be deposited:")) [CURSOR POSITION]
SCFTransaction(mydb, acc_no, amount)
Relevant Cross-file Context
0 # Local File: ./Program.py  
1 class Bank:  
2 acc_count = int(0)  
3 def __init__(self, name, balance):  
4 pass  
5 def BankBasedOptions(acc_no:int):  
6 while True:  
7 print("Select an option from below")  
8 print("1. Deposit")  
9 print("2. Withdraw")  
10 print("3. Show Details")  
11 print("4. Exit")  
12 choice = int(input("Enter your choice:"))  
13 if choice == 1:  
14 amount = int(input("Enter amount to be deposited:"))  
15 SCF Transaction(mydb, acc_no, amount)  
16 elif choice == 2:  
17 amount = int(input("Enter amount to be withdrawn:"))  
18 Bank.withdraw(mydb, acc_no, -amount)  
19 elif choice == 3:  
20 SCF.PrintTable(mydb, "Account", acc_no)  
21 elif choice == 4:  
22 print("\\n\n")  
23 Bank.EntryOptions()
Figure 8: An illustration demonstrating the context retrieval.
which, different from the cross-file context, will be directly used as the oracle context, since the query already contains the target completion rather than just left context. Again, it includes the direct hint, line 15, for the code completion.
D Additional Evaluation Results and Ablations
D.1 Evaluation with Additional Models