M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation

## Abstract

Abstract Repository-level code completion has drawn great attention in software engineering, and several benchmark datasets have been introduced.
However, existing repository-level code completion benchmarks usually focus on a limited
number of languages ( < < < 5), which cannot evaluate the general code intelligence abilities across different languages for existing code Large Language Models (LLMs).
Besides,
the existing benchmarks usually report overall average scores of different languages,
where the fine-grained abilities in different completion scenarios are ignored.
Therefore,
to facilitate the research of code LLMs in multilingual scenarios, we propose a massively multilingual repository-level code completion benchmark covering 18 programming languages (called M 2 rc-Eval ),
and two types of fine-grained annotations (i.e., bucket-level and semantic-level ) on different completion scenarios are provided,
where we obtain these annotations based on the parsed abstract syntax tree.
Moreover, we also curate a massively multilingual instruction corpora M 2 rc-Instruct dataset to improve the repository-level code completion abilities of existing code LLMs. Comprehensive experimental results demonstrate the effectiveness of our M 2 rc-Eval and M 2 rc-Instruct . Code and data will be released at https://github.com/M2RC-Eval-Team/M2RC-Eval .

## 1 Introduction

The emergence of Large Language Models (LLMs) specifically designed for code-related tasks has marked a significant advancement in code generation. The code LLMs (Roziere et al., ; Zheng et al., ; Guo et al., ; Hui et al., ) pre-trained on extensive datasets comprising billions of code-related tokens further revolutionize the automation of software development tasks, providing contextually relevant code suggestions and facilitating the translation from natural language to code. The generation capability of code LLMs opens up diverse applications in software development, promising to enhance productivity and streamline coding processes. As the field continues to evolve, it presents exciting opportunities for future developments and innovations in automated programming and code assistance.

The code completion task is crucial in modern software development, enhancing coding efficiency and accuracy by predicting and suggesting code segments based on context. Recent advancements in code LLMs (Bavarian et al., ) have introduced sophisticated completion techniques, such as prefix-suffix-middle (PSM) and suffix-prefix-middle (SPM) paradigms, which can complete middle code segments given the surrounding context. However, the current benchmark (Ding et al., ; Liu et al., ) mainly focuses on several programming languages.
For example,
the CrossCodeEval (Ding et al., ) includes four languages (i.e., Python, Java, TypeScript, C#).
Besides,
existing benchmarks can only provide the average score among all samples,
which can not provide a language-specific evaluation for different programming languages based on their intrinsic structure. Inspired by the multilingual in-file code generation benchmark MultiPL-E Cassano et al. () and McEval (Chai et al., ), we create a massively multilingual repository-level code completion Evaluation benchmark called M^2rc-Eval to facilitate the research of the community.

In this paper, as shown in Fig. 1,
our M^2rc-Eval includes 18 programming languages with two types of fine-grained annotations (i.e., bucket-level and semantic-level),
where each language contains 100 validation and 500 test samples,
respectively.
Specifically,
for the bucket-level annotations,
we first generate abstract syntax tree with $N$ layers using code parser (i.e., Treesitter (^1^11[https://tree-sitter.github.io/tree-sitter/](https://tree-sitter.github.io/tree-sitter/))),
and divide these $N$ into fixed $M$ buckets,
Then,
for each completion cursor position,
we annotate the corresponding bucket-level label based on
the layer to which the location belongs.
In this way,
we can obtain different code completion scenarios with different difficulties.

For the semantic-level annotations,
inspired by (Takerngsaksiri et al., ),
we first pre-define 11 major semantic labels (e.g., Program Structure, Statement) for each completion cursor position,
which aims to analyze the fine-grained performance across different code semantics.
Note that as different languages usually have specific syntax,
we carefully design the subcategories under each major semantic label for different languages.
Then,
as the code parser usually provides syntax labels (e.g., functions, variables, classes, empty lines)(^2^22Note that the syntax label provided by code parser (e.g., tree-sitter) are highly detailed.) for each completion cursor position,
we carefully define the mappings between the syntax labels to our designed semantic labels and build the semantic-level annotations for our M^2rc-Eval.
Finally,
to enhance the performance of repository-level code completion for existing code LLMs,
we also create a massively multilingual instruction corpora
M^2rc-Instruct of 18 languages.

Figure: Figure 1: Overview of our proposed M^2rc-Eval with 18 languages. Specifically, first, we provide three samples from different languages (i.e., Python, Java, TypeScript) for illustration, where the bucket label and semantic label for the corresponding cursor position are provided. Second, the code LLMs need to predict the completion results given the in-file context from the current code file and the cross file context retrieved from other code files in the current repository. Note that “$<\mathrm{INFILLING}>$” denotes that the current position will be triggered for code completion.
Refer to caption: x1.png

The contributions are summarized as follows:

- •
We propose the first massively multilingual repository-level code completion benchmark M^2rc-Eval covering 18 languages,
where two types of annotations (bucket-level and semantic-level labels) are provided based on the parsed abstract syntax tree.
- •
We introduce M^2rc-Instruct, the
massively multilingual repository-level code instruction corpora covering the multilingual code snippet from 18 languages,
which can greatly enhance the performance of repository-level code completion results.
- •
Comprehensive evaluation results and analysis demonstrate the effectiveness of our proposed M^2rc-Eval and M^2rc-Instruct.

## 2 Related Works

#### Code Large Language Models.

Code large language models (LLMs) (Chen et al., ; Zhao et al., ; Black et al., ; ; Le et al., ; Chowdhery et al., ; Nijkamp et al., ; Fried et al., ; Xu et al., ) are increasingly involved in modern programming, due to excellent capabilities of code generation (Li et al., ; Allal et al., ), code repair (Wang et al., ; ), code translation (Zheng et al., ; Li et al., ), and other coding tasks.
UniCoder (Sun et al., ) and SPT-Code (Niu et al., ) introduce the pseudo-code generation and the alignment between Abstract Syntax Tree (AST) and code.
Recent code LLMs such as Code Llama (Roziere et al., ), DeepSeek-Coder (Guo et al., ), and Qwen2.5-Coder (Hui et al., ) incorporate the fill-in-the-middle (FIM) task into their training stage for code completion.
Moreover, there is a wide variety of in-file benchmarks to evaluate different capabilities of code LLMs  (Zheng et al., ; Austin et al., ; Jain et al., ), which focus on a limited range of programming languages (e.g. Python and Java).
The recent work (Chai et al., ) extends the number of programming languages to 40 for multilingual evaluation scenarios,
which has not considered the repository-level code completion.

#### Repository-level Code Completion.

The latest repository-level code completion methods (Bairi et al., ; Phan et al., ; Liao et al., ; Shrivastava et al., ; Agrawal et al., ; Shrivastava et al., ; Pei et al., ; Zhang et al., ) are similar to RAG, aim to precisely retrieve all related code snippets across files within a repository.
Further, repository-level benchmarks are proposed to estimate the capability of code LLMs in a more realistic software engineering scenario.
But these datasets (Ding et al., ; ; Allal et al., ) are primarily concentrated on several programming languages.
Regarding difficulty categorization, most methods only consider the number of files involved in the completion content, overlooking the code’s structural and semantic context within the entire project.
Repofusion (Shrivastava et al., ) and Repocoder (Zhang et al., ) predict one line based on the prefix and suffix code, while CoderEval (Yu et al., ) measures how many third-party libraries are called.
To comprehensively evaluate the multilingual repository-based code completion of different code LLMs, we push the boundaries of programming languages into 18 languages in M^2rc-Eval with fine-grained annotations.

## 3 M 2 rc-Eval

### 3.1 Data Collection

The Overall Data Pool.
We begin by collecting The Stack v2 (Lozhkov et al., ), which consists of permissively licensed repositories from GitHub. Next, we adopt the The-stack-v2-dedup, which includes 784 million source code files spanning 619 programming languages with manual and heuristic pre-processing. Further, we keep only repositories receiving more than 5 stars and containing $[10,50]$ files. Lastly, preserving files written in 18 common languages, we have 431,353,244 files remaining, constituting the overall data pool.

Completion Cursor Position Selection.
Completion cursor position selection significantly impacts the quality of a code completion benchmark. Previous studies (Ding et al., ; Liu et al., ) randomly select a segment of consecutive characters as the completion span, which does not guarantee the integrity of identifiers and statements.
On the contrary, in M^2rc-Eval, we first parse the abstract syntax tree (AST) of each source code file, and then we randomly choose a node (e.g., the node of “Function Definition” in Fig. 2) on the AST as the completion cursor position.
After that, we obtain the corresponding code to obtain the ground-truth for the current completion cursor position.
Finally,
at inference,
the code LLMs need to predict the current code span given the in-file and cross file contexts.
Similarly,
in training,
we just use the ground-truth to supervise the tuning process of the code LLMs.

Figure: Figure 2: Illustration on generating completion cursor position and fine-grained annotations. Specifically, we first parse the source code into an abstract syntax tree (AST). Then, we choose one node as the completion cursor position and generate the bucket label based on the belonged layer number in AST, and obtain the semantic label based on the node type parsed by the Tree-sitter.
Refer to caption: x2.png

### 3.2 Quality Control

We build a suite of post-processing filters to enhance the quality of M^2rc-Instruct. We eliminate examples based on two heuristic rules: (1) The completion cursor position should be no longer than 5 lines. (2) If the completion ground truth is fewer than 20 characters, at least 20% of them should be alphabetic. To improve data independence and inference difficulty, we apply extra filters to the test cases in M^2rc-Eval. (a) Repositories in M^2rc-Eval should be absent from M^2rc-Instruct. (b) We ensure that 30% of the completion ground truth is not shorter than 2 lines. (c) The completion cursor position should not be fully white-spaced. (d) We discard test cases that could be exactly predicted by DeepSeekCoder-1.3B (Guo et al., ) without cross file contexts.

Figure: Figure 3: The average prompt length (100x tokens), completion span length (50x tokens), and cross-file dependencies (1x) in the testing set of M^2rc-Eval. We define the number of other files, which are explicitly imported and implicitly referenced by the current file, as cross-file dependencies.
Refer to caption: x3.png

**Table 1: A comparison with existing notable repository-level code completion datasets.**
| Benchmark | # Languages | Fine-grained | Training Set | # Test Repos |
| --- | --- | --- | --- | --- |
| RepoBench (Liu et al., ) | 2 | ✗ | ✓ | 1669 |
| CrossCodeEval (Ding et al., ) | 4 | ✗ | ✗ | 1002 |
| R^2C^2-Bench (Deng et al., ) | 4 | ✗ | ✓ | 1353 |
| M^2rc-Eval& M^2rc-Instruct | 18 | ✓ | ✓ | 5993 |

### 3.3 Dataset Statistics

Following the quality filters in §(3.2) from the overall data pool §(3.1).
We sample 50,000 files per language to construct our M^2rc-Instruct,
and sample 100, and 500 files per language to build the validation and test sets of our M^2rc-Eval,
respectively.
The statistics of the test set are shown in Fig. 3,
and we also provide a detailed comparison between our M^2rc-Eval with existing repository-level code completion datasets in Table 1.

### 3.4 Fine-grained Annotations

As shown in Fig. 2,
to analyze the performance in a fine-grained manner,
we further provide two types of fine-grained annotations (i.e., bucket-level and semantic-level) for each completion cursor.
Specifically,
we first generate the abstract syntax tree.
For the bucket-level annotations,
we first simply divide each tree into $M$ buckets based on the depth degree of the abstract syntax tree.
Note that we set $M$ as 10 in our M^2rc-Eval.
For example,
if the number of layers for the current abstract syntax tree is $N$,
the $i$-th layer of the tree belongs to the $\lceil\frac{i}{N/M}\rceil$ bucket.
Then,
for each completion cursor node, we annotate the bucket label based on the layer number of each node.
Similarly,
for the semantic-level annotations,
we directly annotate the semantic-level label for each completion cursor node.
Specifically,
we pre-define 11 major classes (i.e., “Program Structure”, “Declaration and Definition”, “Control Flow Structure”, “Expression”, “Data Type”, “Statement”, “Modifier and Attribute”, “Comments and Documentation”, “Preprocessing Directive”, “Identifier and Scope”, “Special Language Structure”).
Then,
as different languages have many specific designs, the subcategories under each major class are carefully annotated for different languages.
As shown in Fig. 4,
we provide the semantic-level annotations on three main-stream programming languages (Java, Go, Scala),
where the annotations on the remained 15 languages are provided in Fig. 14 of the Appendix.

Figure: (a) Java
Refer to caption: x4.png

## 4 Experiments

We perform detailed experiments on M^2rc-Eval with three popular Code LLMs (i.e., StarCoder-7B (Li et al., ), DeepSeekCoder-6.7B (Guo et al., ) and Code Llama-7B (Roziere et al., )) (See Appendix A.3 for more details).

### 4.1 Evaluation Metrics

Following (Ding et al., ),
we compare the generated code with the reference and
compute the exact match (EM) and edit similarity (ES) metrics,
which assesses the textural similarities and ignores semantic structure similarities among predictions and ground-truth.

### 4.2 Experimental Setup

Baseline.
Only the original code file, where the cursor position is located,
is provided for the code LLMs.
As no explicit inter-file context is supplied, the model must utilize its inherent knowledge-based reasoning abilities to generate appropriate code.

+ Retrieval.
In line with the approach outlined in CrossCodeEval (Ding et al., ), the retrieval process begins by examining files within the same repository. Continuous code segments of $L$ lines are extracted, where $L$ matches the length of the retrieval query and is set as 10 by default. Subsequently, these extracted candidates are prioritized based on their Jaccard similarity scores. The most relevant fragments are then appended to the beginning of the in-file context in descending order of similarity. This concatenation continues until the total length, including both the added candidates and the original in-file context, reaches the predetermined maximum token limit of 4096.

+ Retrieval & Tuning.
To further improve the performance of repository-level code completion, we fine-tune code LLMs on the M^2rc-Instruct dataset mentioned in §(3).
At inference,
we use the same inference strategy as discussed in “+ Retrieval”.

### 4.3 Main Results

**Table 2: Exact match (%) and edit similarity (%) performance on M^2rc-Eval.**
| Model | C | C# | C++ | Go | HTML | Haskell | - |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | EM | ES |  |
| Code Llama-7B | 18.6 | 47.2 | 19.6 | 52.6 | 21.8 | 51.1 | 26.0 | 53.6 | 20.6 | 40.4 | 22.6 | 48.9 | - | - |
| + Retrieval | 21.8 | 47.2 | 22.9 | 48.9 | 23.2 | 46.6 | 23.8 | 52.4 | 12.6 | 35.6 | 22.6 | 48.9 | - | - |
| + Retrieval & Tuning | 45.4 | 72.0 | 43.5 | 72.3 | 50.8 | 74.9 | 43.4 | 72.9 | 41.8 | 63.6 | 39.8 | 66.3 | - | - |
| StarCoder-7B | 20.0 | 50.4 | 20.0 | 53.3 | 22.4 | 51.8 | 25.4 | 58.2 | 17.4 | 40.7 | 25.0 | 51.1 | - | - |
| + Retrieval | 23.8 | 47.8 | 27.1 | 53.2 | 24.6 | 48.0 | 26.0 | 53.6 | 20.6 | 40.4 | 25.0 | 47.7 | - | - |
| + Retrieval & Tuning | 47.0 | 72.7 | 45.1 | 74.8 | 52.4 | 76.3 | 43.2 | 73.7 | 45.8 | 67.1 | 44.8 | 70.2 | - | - |
| DeepSeekCoder-6.7B | 22.4 | 53.7 | 21.4 | 56.2 | 23.2 | 54.2 | 29.4 | 61.4 | 17.6 | 43.4 | 25.2 | 51.3 | - | - |
| + Retrieval | 28.2 | 52.6 | 25.3 | 52.6 | 27.6 | 52.2 | 29.4 | 61.4 | 17.6 | 43.4 | 25.8 | 51.0 | - | - |
| + Retrieval & Tuning | 48.6 | 75.2 | 47.9 | 76.9 | 54.4 | 78.2 | 48.8 | 78.4 | 45.0 | 66.3 | 45.8 | 72.0 | - | - |
| Model | Java | JavaScript | Kotlin | Lua | Objective-C | PHP | - |  |  |  |  |  |  |  |
| Code Llama-7B | 23.4 | 58.5 | 17.2 | 52.0 | 23.6 | 57.0 | 20.0 | 45.7 | 17.8 | 49.5 | 19.2 | 54.9 | - | - |
| + Retrieval | 23.4 | 57.5 | 19.6 | 48.0 | 20.8 | 50.0 | 19.6 | 42.2 | 21.4 | 46.6 | 21.2 | 49.0 | - | - |
| + Retrieval & Tuning | 41.8 | 74.1 | 38.8 | 70.1 | 45.0 | 75.6 | 43.8 | 70.5 | 49.8 | 75.9 | 45.6 | 76.7 | - | - |
| StarCoder-7B | 24.0 | 59.2 | 16.6 | 52.0 | 24.4 | 59.3 | 21.4 | 48.6 | 17.6 | 49.6 | 18.6 | 54.4 | - | - |
| + Retrieval | 25.0 | 53.1 | 22.0 | 50.8 | 22.8 | 52.6 | 26.4 | 48.5 | 23.6 | 48.0 | 18.6 | 54.4 | - | - |
| + Retrieval & Tuning | 47.4 | 76.9 | 38.8 | 70.1 | 45.0 | 75.6 | 43.8 | 70.5 | 50.8 | 75.9 | 45.6 | 76.7 | - | - |
| DeepSeekCoder-6.7B | 22.2 | 61.0 | 20.4 | 56.5 | 26.0 | 61.0 | 22.0 | 48.8 | 21.0 | 55.6 | 24.2 | 58.6 | - | - |
| + Retrieval | 21.6 | 51.4 | 24.4 | 53.6 | 26.0 | 61.0 | 22.0 | 49.9 | 27.6 | 53.5 | 28.6 | 56.9 | - | - |
| + Retrieval & Tuning | 48.2 | 79.1 | 43.6 | 73.5 | 46.0 | 75.7 | 44.6 | 70.6 | 52.2 | 77.6 | 49.8 | 78.8 | - | - |
| Model | Python | R | Ruby | Rust | Scala | TypeScript | Avg. |  |  |  |  |  |  |  |
| Code Llama-7B | 24.6 | 54.2 | 15.2 | 41.2 | 17.2 | 45.8 | 26.2 | 56.0 | 22.8 | 48.5 | 23.4 | 52.3 | 19.4 | 50.3 |
| + Retrieval | 17.4 | 46.4 | 15.2 | 39.8 | 17.2 | 42.3 | 26.0 | 51.3 | 22.8 | 48.5 | 19.4 | 48.6 | 20.2 | 46.1 |
| + Retrieval & Tuning | 39.2 | 69.9 | 38.6 | 65.5 | 43.0 | 68.5 | 42.0 | 69.2 | 41.0 | 70.1 | 37.0 | 68.2 | 41.9 | 70.0 |
| StarCoder-7B | 19.4 | 52.9 | 16.4 | 43.7 | 19.4 | 47.4 | 26.2 | 56.0 | 23.6 | 53.4 | 19.8 | 53.3 | 21.0 | 52.0 |
| + Retrieval | 24.6 | 54.2 | 22.6 | 47.2 | 23.6 | 47.4 | 26.4 | 53.5 | 22.8 | 48.5 | 23.4 | 52.3 | 24.1 | 50.0 |
| + Retrieval & Tuning | 39.2 | 69.9 | 41.0 | 66.6 | 43.0 | 68.5 | 45.8 | 72.6 | 43.6 | 71.5 | 39.2 | 69.7 | 44.5 | 72.2 |
| DeepSeekCoder-6.7B | 21.8 | 55.1 | 19.4 | 48.5 | 23.6 | 52.2 | 23.8 | 54.3 | 24.6 | 56.7 | 19.4 | 55.4 | 22.6 | 54.7 |
| + Retrieval | 21.8 | 55.1 | 19.4 | 48.5 | 23.6 | 52.2 | 23.8 | 54.3 | 22.4 | 50.4 | 26.0 | 54.5 | 25.1 | 51.7 |
| + Retrieval & Tuning | 41.6 | 71.3 | 45.4 | 69.4 | 45.6 | 70.3 | 47.6 | 73.4 | 44.8 | 73.7 | 43.2 | 73.4 | 46.8 | 74.1 |

We present the results on M^2rc-Eval in Table 2.
We observe that different code LLMs have different repository-level code completion abilities for different programming languages. For instance, DeepSeekCoder-6.7B demonstrates strong completion ability for Go, while its performance is weaker with HTML, a markup language,
which demonstrates the necessity of evaluating code LLMs for multilingual capabilities.
Besides, the results indicate that cross file context is highly effective, resulting in a significant improvement compared to using only in-file context. In particular, the multilingual SFT on our created instruction corpora M^2rc-Instruct also significantly enhances performance on M^2rc-Eval. Notably, after SFT on M^2rc-Instruct, Code Llama-7B, which originally ranked lowest with in-file context, outperformed the non-finetuned StarCoder-7B, demonstrating the effectiveness of M^2rc-Instruct.

### 4.4 Analysis

Analysis on different model sizes.
In Table 4.4, we provide the performance of StarCoder with respect to different model sizes in the validation set of M^2rc-Eval. Notably, StarCoder-7B consistently outperforms StarCoder-3B under comparable conditions. However, following the application of SFT on M^2rc-Instruct, the results of StarCoder-3B exceed those of the inference-only StarCoder-7B. This finding underscores the effectiveness of our M^2rc-Instruct in augmenting the capabilities of smaller models in repository-level code completion.

**Table 3: Performance on M^2rc-Eval.**
| Model | Average |  |
| --- | --- | --- |
| EM | ES |  |
| StarCoder-3B | 14.9 | 43.5 |
| \hdashline  + Retrieval | 14.6 | 38.4 |
| \hdashline  + Retrieval & Tuning | 41.7 | 69.1 |
| StarCoder-7B | 20.6 | 49.9 |
| \hdashline  + Retrieval | 23.6 | 49.3 |
| \hdashline  + Retrieval & Tuning | 44.4 | 71.4 |

## 5 Conclusion

In this paper, we propose the first massively multilingual repository-level code completion benchmark (M^2rc-Eval) with 18 popular programming languages,
where two types of fine-grained annotations (bucket-level and semantic-level) are provided to comprehensively analyze the effectiveness of different code LLMs.
Besides,
we also curate a high-quality instruction corpus M^2rc-Instruct to enhance the performance of existing models on repository-level code completion.
Extensive experimental results and detailed discussions demonstrate the effectiveness of our proposed M^2rc-Eval and M^2rc-Instruct.
Finally, we hope M^2rc-Eval could guide the developers and researchers to understand the repository-level code completion capabilities of LLMs and facilitate the growth of code intelligence and software engineering.

## Appendix A Appendix

Figure: Figure 11: Effectiveness of different bucket levels based on StarCoder-7B for different languages.
Refer to caption: x13.png

Figure: Figure 12: Effectiveness of different bucket levels based on StarCoder-7B for different languages.
Refer to caption: x25.png

Figure: Figure 13: Effectiveness of different semantic levels based on StarCoder-7B.
Refer to caption: x31.png

Figure: (a) C
Refer to caption: x34.png

### A.1 Broader Impacts

In this paper, we propose a repository-level code completion benchmark with 18 programming languages. Therefore, we hope our work can enhance the improvements on the multilingual repository-level code completion task.

### A.2 Limitations

First,
there are several hyperparameters (e.g., training sizes, input length) to tune,
which is laborious and expensive.
Second,
the current work only focuses on the repository-level code completion task,
where other repository-level code intelligence tasks are not considered.
Third,
only textual similarity scores (EM and ES) are used and execution-based evaluation based on test cases is not applied,
which may not reflect the performance of different code LLMs well.

### A.3 Details of the Baseline Models

StarCoder (Li et al., ) is a series of generative language models (e.g., 7B, 15.5B). These decoder-only models are trained on the Stack dataset (Kocetkov et al., ) and can support 8K tokens in context.

DeepSeekCoder (Guo et al., ) is a collection of code-oriented models with capacities from 1.3B to 33B parameters. Trained on a manually curated 2-trillion-token corpus, these models leverage Fill-in-the-Middle (FIM) (Bavarian et al., ) and Rotary Position Embedding (RoPE)  (Su et al., ) techniques,
which enables efficient code generation and infilling within a 16K token window.

Code Llama (Roziere et al., ) is a family of code large language models based on Llama 2 (Touvron et al., ) with 7B, 13B, 34B, and 70B parameters.
While trained on 16K token sequences, these models can handle inputs up to 100K tokens during inference.

### A.4 Discussion on no execution-based evaluation

Current datasets for repository-level code completion evaluation, such as CrossCodeEval (Ding et al., ) and RepoBench (Liu et al., ), only assess textual similarity between predictions and ground-truth. We hypothesize this limitation stems from several challenges:
Firstly, generating comprehensive unit tests for each completion position in a repository is problematic. Single-line completions often fail to construct executable functions, and ensuring adequate test coverage is difficult.
Secondly, execution-based evaluation necessitates creating diverse environments for each repository, accommodating various software packages and hardware requirements. This process is intricate and challenging to implement.
Thirdly, existing benchmarks with unit tests typically focus on simpler scenarios, like single-file completions or function body generation. Examples include commonly used datasets such as Humaneval CodeGeeX () and MBPP Austin et al. ().
Despite these obstacles, we recognize the importance of execution-based evaluation for accurately assessing code completion effectiveness, and we will continue to investigate how to evaluate repository-level code completion well.

### A.5 More experiments

- •
We provide the analysis on the bucket levels in Fig. 11 and Fig. 12,
respectively.
- •
We analyze the effect of different semantic levels on Rust, Objective-C and Haskell in Fig. 13.
respectively.
- •
We provide the semantic-level annotations on another 15 languages in Fig. 14.
- •
We provide the visualization on the completion cases in Fig. 15 and Fig. 16.

Figure: Figure 15: Visualization on failure case for the C language.
Refer to caption: x49.png

Figure: Figure 16: Visualization on success case for the TypeScript language.
Refer to caption: x50.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: See comments above. But framed some concerns in question format below:

1. How do larger or more recent models such as StarCoder2-15B or GPT-4o perform on this benchmark?
2. Can you provide more insights from your multi-lingual transfer experiments? Specifically which languages benefit the most from Python-only fine-tuning? Can we predict such transfer ahead of time?
soundness: 4
strengths: - **Useful contributions.** The idea of having a very-large scale benchmark for repository-aware code completions across 18 programming languages is very useful. This will significantly help researchers improving Code LLMs on low resource languages such as Lua and Scala.
- **Novelty.** The fine-grained breakdown of code completion tasks is novel. However, I am not sure if this is needed or insightful. I would encourage the authors to present more analyses around this.
- **Interesting analyses.** I found the multi-lingual transfer experiment to be quite interesting and would be interested in a language-wise breakdown of this.
- **Paper exposition.** The paper is well-written and the analyses are presented in an easy-to-understand manner.
summary: The paper introduces M2rc-Eval, a new benchmarks comprising repository-level code completion tasks in many more languages than current benchmarks. Unique features of this benchmark include fine-grained annotations for different types of completion scenarios. The authors also introduce M2rc-Instruct, which a training dataset to improve performance on M2rc-Eval.
weaknesses: - **Data leakage concerns.** The presented benchmark is collected from The Stack v2 which sources code from public sources such as GitHub. As such it may have significant overlap with the training corpora of the Code LLMs already evaluated in the paper and for future LLMs. As such, this benchmark is not fool-proof from future leakage and the utility of the benchmark for future evaluations is questionable. Further, the authors have evaluated only small models, and evaluations on bigger models ≥ 15B or strong models like GPT-4o or Claude 3.5 Sonnet might reveal the effect of leakage more concretely.
- **Repository-level coding.** While the paper says that the benchmark is for repository-level code completion evaluations, the data collection described in Section 3.1 does not guarantee that the tasks in the benchmark strictly require repository-level contexts.
- **Evaluation metrics.** While EM/ES metrics are being used for evaluations and have been used in several previous works as well, I think it is time that we move on from these inaccurate metrics. We should resort to more robust metrics relying on static or execution analysis such as pass rate [1] or hallucination rate [2] for measuring correctness of code completions.
- **Nitpicks.**
    - The analyses in lines 416-428 are quite obvious, i.e., there is a negative correlation between ES and the number of tokens generated
    - Typos:
        - 047 - “can not” → cannot
        - 261 - textural → textual
        - 401 - shadow → shallow

[1] Chen, Mark, et al. "Evaluating large language models trained on code." *arXiv preprint arXiv:2107.03374* (2021).

[2] Jain, Nihal, et al. "On Mitigating Code LLM Hallucinations with API Documentation." *arXiv preprint arXiv:2407.09726* (2024).

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: In addition to the points mentioned in `Weaknesses`, I have a few questions and notes regarding the work:

1. Could the authors elaborate on how was the 5993 repos (line 204) split across eval and instruct?

2. It appears that adding retrieval seems to improve on EM but degrade on ES across models in Table 2. Why would this be expected?

3. On line 445: “StarCoder-7B base model already demonstrates strong coding proficiency, but lacks robust instruction-following capabilities” this is well expected as StarCoder-7B is a base model that has not been instruction tuned?

4. Some plots like Figure 4 look fancy but is conveying only little meaningful information in a readable way. Authors may consider using an alternative way to represent the data, e.g., a simple list or table. Similarly, Figure 5 is barely readable.
soundness: 2
strengths: - This is the first dataset that covers 18 programming languages in repo-level code completion task
- The dataset Includes both evaluation and instruction tuning datasets
- Authors have conducted various evaluation across multiple dimensions
summary: The paper introduces M$^2$RC-EVAL, a benchmark for repository-level code completion covering 18 programming languages, alongside M$^2$RC-INSTRUCT, a multilingual instruction dataset built in a similar way with M$^2$RC-EVAL. Using The Stack v2 dataset, the authors select repositories with 5+ stars containing 10-50 files, creating a dataset with 50,000 files per language for instruction tuning and 600 files per language for evaluation (w/ random 100 examples for validation and 500 for test). The benchmark introduces two types of annotations: bucket-level, an indicator of the depth of the AST node, and semantic-level, which was split in 11 major categories with language-specific subcategories. The authors benchmarked three major code LLMs (StarCoder-7B, DeepSeekCoder-6.7B, Code Llama-7B) under different settings and provide extensive analysis across multiple dimensions.
weaknesses: The most critical weakness of this paper is the use of The Stack v2 without addressing data contamination issues. Unlike previous work like Ding et al., 2023 which carefully controlled the date range to avoid overlap with model training data, this work directly uses a widely-used training corpus that contain data that was written many many years back. This fundamentally compromises the benchmark's ability to reflect real-world performance. The authors should either implement temporal splits or explicitly verify and control for overlap with model training data.

Further, the assumption that "we randomly choose a node on the AST as the completion cursor position" (Line 152) is overly restrictive and doesn't reflect real-world scenarios. Code completion can and does occur at any position in practice. A more realistic approach would be to allow arbitrary completion cursor positions while ensuring the completed code forms a valid AST node, similar to previous work cited in the paper.

In addition, the bucket-level annotation scheme appears arbitrary and lacks clear motivation. Dividing ASTs into exactly 10 buckets seems like an artificial constraint when simpler alternatives exist, such as directly using the node's layer number in the AST. It'd be great if authors could justify why this particular granularity was chosen or demonstrate its advantages over simpler approaches.

Finally, despite covering 18 programming languages, there was surprisingly very little discussion of language-specific insights. This is a great opportunity that the authors could seize to analyze how different language features, paradigms, and structures affect completion performance. Such analysis could provide valuable insights for both researchers and practitioners.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The cross-lingual transfer benchmarks may not be properly baseline. The paper uses StarCoder-1, a multi-PL pre-trained model that is continually pre-trained on Python code. This is a rather non-standard setting and would be fairer (across langs) if it were started from StarCoder-1-Base.
2. It would be interesting to explore the effect of various code retrievers instead of just Jaccard-Similarity in retrieving snippets in-context.
soundness: 3
strengths: 1. The paper enters a crowded field of repo-level code completion benchmarks but differentiates itself from the pack by the size of the data splits and the number of languages supported.
2. The data collection process is clearly detailed
summary: The paper releases a cross-file-context repo-level code completion benchmark that tests the fill-in-the-middle (FIM) completion abilities of Code LMs. GitHub repos are collected from Stack V2, and then the files' ASTs are individually processed to create varying categories of FIM holes for completion.

I believe the dataset (especially the instruct split) is a valuable OS contribution. Still, as it joins a whole host of existing benchmarks in this area, it should be fleshed out further regarding research questions.
weaknesses: 1. For a benchmark paper, it is a bit light on details of env setup, tooling and sandboxing
2. The evaluation is still based on surface form metrics like exact-match, which can be very noisy, especially for FIM settings. Scaling execution tests can be hard, but existing benchmarks like EvoCodeBench [1] and BigCodeBench [2] have done so using a combination of model generation and human validation.
3. Several very strong recent code model releases, such as Yi-Coder [3], Llama-3.1 [4], and Qwen-2.5-coder [5], are missing from the evals. This is pertinent due to their longer context support (up to 128k tokens), allowing for a much more interesting baseline than the paper currently uses (just the individual file in question). A very interesting evaluation of the paper is missing: how well the benchmark can be solved if the model just placed all the most pertinent files in context instead of just retrieving a few short snippets. Similarly, how would the various in-repo file ordering schemes previously explored by StarCoder-2 [6] and DeepSeek-Coder [7] affect this?

[1] https://arxiv.org/abs/2404.00599
[2] https://arxiv.org/abs/2406.15877
[3] https://arxiv.org/abs/2403.04652
[4] https://arxiv.org/abs/2407.21783
[5] https://arxiv.org/abs/2409.12186
[6] https://arxiv.org/abs/2402.19173
[7] https://arxiv.org/abs/2401.14196

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: Apart from the point in the 'Weaknesses' section, I have the following questions and suggestions:

**Proprietary Model Comparison**: Do the authors plan to include a comparison with advanced proprietary models, even on a smaller subset of the benchmark? This could offer insights into how fine-tuned open-source models compare against the leading commercial solutions.

**Qualitative Error Analysis (Suggestion):** It would be insightful to provide a more detailed qualitative error analysis to highlight where current code LLMs encounter the most difficulty across different languages. This could help pinpoint areas for further improvement.

**Figure Clarity (Suggestion):** While the visualizations are nicely presented, some figures (e.g., Figures 4 and 14) have very small fonts, making it difficult to read the details. Increasing the font size in these visualizations would enhance clarity and accessibility for readers.
soundness: 3
strengths: **Massive multilingual benchmark**: The paper introduces a novel multilingual benchmark that supports 18 programming languages, which is a significant expansion over existing benchmarks that only handle a limited number of languages.

**Quality control**: The dataset construction is considered thorough, with a robust collection process from github repos and clear quality control measures. 

**Clear writings**: This paper is well-organized, with clear explanations of how the benchmark and instruction dataset were constructed and evaluated. The use of tables and figures aids in understanding the experimental results.

Overall, this benchmark contributes to address the growing need for multilingual evaluation in code completion. The fine-grained annotation methodology also provides useful insights into the challenges faced by models in many programming scenarios.
summary: This paper introduces M2RC-EVAL, a new benchmark designed to assess repo-level code completion performance across 18 programming languages. The benchmark uses two types of fine-grained annotations: bucket-level, based on the depth of abstract syntax trees, and semantic-level, which focuses on code semantics.  Additionally, the paper presents a multilingual instruction corpus, M2RC-INSTRUCT, to improve code completion models.
weaknesses: **Evaluation Metrics:** While the paper mainly uses exact match and edit similarity metrics, these may not fully capture the functional correctness of the generated code. I acknowledge that implementing an execution-based evaluation across such a massive multilingual benchmark poses considerable challenges, and this is more of a broader limitation within the field rather than a direct weakness of this particular work. I do appreciate the authors’ inclusion of CodeBLEU, which offers another perspective by considering syntactic and semantic aspects of code generation. Still, it would be more insightful to explore whether there is any correlation between CodeBLEU scores and the more traditional EM and ES metrics.

Meta-review: The paper creates a repo-level code completion benchmark across 18 programming languages. The benchmark introduces two types of annotations: bucket-level, an indicator of the depth of the AST node, and semantic-level, which was split in 11 major categories with language-specific subcategories. The evaluation benchmarks three major open-weight code LLMs (StarCoder-7B, DeepSeekCoder-6.7B, Code Llama-7B) under different settings and provide extensive analysis across multiple dimensions.

While the data resource is appreciated, the main concern lies in the design of evaluation metric. The work mostly focuses surface form matching, which has shown to not have strong correlation with functional correctness in previous works. Although the authors provide results on execution-based evaluation for a small subset of the data during rebuttal, it does not cover the full evaluation set and hence cannot be considered as part of the work

Decision: Reject