M2RC-EVAL: Massively Multilingual Repository-level Code Completion Evaluation
Authors: Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ge Zhang, Yingshui Tan, Zekun Moore Wang, JinKe, Zhaoxiang Zhang, Bangyu Xiang, Guoan Zhang, Wenbo Su, Bo Zheng
Accepted at: acl25

M^2RC-EVAL: Massively Multilingual Repository-level Code Completion Evaluation

Jiaheng Liu^{1,2,*†}, Ken Deng^{2*}, Congnan Liu^{2}, Jian Yang^{2}, Shukai Liu^{2}, He Zhu^{2}, Peng Zhao^{2}, Linzheng Chai^{2}, Yanan Wu^{2}, Ke Jin^{2}, Ge Zhang^{3}, Zekun Wang^{2}, Guoan Zhang^{2}, Yingshui Tan^{2}, Bangyu Xiang^{2}, Zhaoxiang Zhang^{4}, Wenbo Su^{2}, Bo Zheng^{2}
^{1}Nanjing University, ^{2}Alibaba Group, ^{3}University of Waterloo, ^{4}CASIA
liujiaheng@nju.edu.cn

Abstract

Repository-level code completion has drawn great attention in software engineering, and several benchmarks have been introduced. However, existing repository-level code completion benchmarks usually focus on a limited number of languages (<5), which cannot evaluate the general code intelligence abilities across different languages for existing code Large Language Models (LLMs). Besides, the existing benchmarks usually report overall average scores of different languages, where the fine-grained abilities in different completion scenarios are ignored. Therefore, to facilitate the research of code LLMs in multilingual scenarios, we propose a massively multilingual repository-level code completion benchmark covering 18 programming languages (called M^2RC-EVAL), and two types of fine-grained annotations (i.e., bucket-level and semantic-level) on different completion scenarios are provided, where we obtain these annotations based on the parsed abstract syntax tree. Moreover, we also curate a massively multilingual instruction corpora M^2RC-INSTRUCT dataset to improve the repository-level code completion abilities of existing code LLMs. Comprehensive experimental results demonstrate the effectiveness of our M^2RC-EVAL and M^2RC-INSTRUCT.

1 Introduction

The emergence of Large Language Models (LLMs) has marked a significant advancement in many tasks (Liu et al., 2025; Zhang et al., 2024; Wang et al., 2024; Liu et al., 2024). The code LLMs (Roziere et al., 2023; Zheng et al., 2023; Guo et al., 2024a; Hui et al., 2024; Zhang et al., 2025; He et al., 2025; Huang et al., 2024) pre-trained on extensive datasets comprising billions of code-related tokens further revolutionize the automation of software development tasks, providing contextually relevant code suggestions and facilitating the translation from natural language to code. The generation capability of code LLMs opens up diverse applications in software development, promising to enhance productivity and streamline coding processes. As the field continues to evolve, it presents exciting opportunities for future developments and innovations in automated programming and code assistance.

The code completion task is crucial in modern software development, enhancing coding efficiency and accuracy by predicting and suggesting code segments based on context. Recent advancements in code LLMs (Bavarian et al., 2022a) have introduced sophisticated completion techniques, such as prefix-suffix-middle (PSM) and suffix-prefix-middle (SPM) paradigms, which can complete middle code segments given the surrounding context. However, the current benchmark (Ding et al., 2024; Liu et al., 2023a) mainly focuses on several programming languages. For example, the CrossCodeEval (Ding et al., 2024) includes four languages (i.e., Python, Java, TypeScript, C#). Besides, existing benchmarks can only provide the average score among all samples, which cannot provide a language-specific evaluation for different programming languages based on their intrinsic structure. Inspired by the multilingual in-file code generation benchmark MultiPL-E (Cassano et al., 2022) and McEval (Chai et al., 2024), we create a massively multilingual repository-level code completion Evaluation benchmark called M^2RC-EVAL to facilitate the research of the community.

In this paper, as shown in Fig. 1, our M^2RC-EVAL includes 18 programming languages with two types of fine-grained annotations (i.e., bucket-level and semantic-level), where each language contains 100 validation and 500 test samples, respectively. Specifically, for the bucket-level annotations, we first generate abstract syntax tree with
Figure 1: Overview of our M^2RC-EVAL with 18 languages. Specifically, first, we provide three samples from different languages (i.e., Python, Java, TypeScript) for illustration, where the bucket label and semantic label for the corresponding cursor position are provided. Second, the code LLMs need to predict the completion results given the in-file context from the current code file and the cross file context retrieved from other code files in the current repository. Note that “< INFILLING >” denotes that the current position will be triggered for code completion.

N layers using code parser (i.e., Treesitter ¹), and divide these N into fixed M buckets. Then, for each completion cursor position, we annotate the corresponding bucket-level label based on the layer to which the location belongs. In this way, we can obtain different code completion scenarios with different difficulties.

For the semantic-level annotations, inspired by (Takerngsaksiri et al., 2024), we first pre-define 11 major semantic labels (e.g., Program Structure, Statement) for each completion cursor position, which aims to analyze the fine-grained performance across different code semantics. Note that as different languages usually have specific syntax, we carefully design the subcategories under each major semantic label for different languages. Then, as the code parser usually provides syntax labels (e.g., functions, variables, classes, empty lines)² for each completion cursor position, we carefully define the mappings between the syntax labels to our designed semantic labels and build the semantic-level annotations for our M^2RC-EVAL. Finally, to enhance the performance of repository-level code completion for existing code LLMs, we also create a massively multilingual instruction corpora M^2RC-INSTRUCT of 18 languages.

The contributions are summarized as follows:

• We propose the first massively multilingual repository-level code completion benchmark M^2RC-EVAL covering 18 languages, where two types of annotations (bucket-level and semantic-level labels) are provided based on the parsed abstract syntax tree.

• We introduce M^2RC-INSTRUCT, the massively multilingual repository-level code instruction corpora covering the multilingual code snippet from 18 languages, which can greatly enhance the performance of repository-level code completion results.

• Comprehensive evaluation results and analysis demonstrate the effectiveness of our proposed M^2RC-EVAL and M^2RC-INSTRUCT.

¹https://tree-sitter.github.io/tree-sitter/
²Note that the syntax label provided by code parser (e.g., tree-sitter) are highly detailed.
2 Related Works

Code Large Language Models. Code LLMs (Chen et al., 2021; Zhao et al., 2024; Black et al., 2022; Le et al., 2022; Chowdhery et al., 2023; Nijkamp et al., 2023; Fried et al., 2023; Xu et al., 2022; Jain et al., 2024b; Zhuo et al., 2025) are increasingly involved in modern programming, due to excellent capabilities of code generation (Li et al., 2022; Allal et al., 2023), code repair (Wang et al., 2021, 2023), code translation (Li et al., 2023), and other coding tasks. Recent code LLMs such as Code Llama (Roziere et al., 2023), DeepSeek-Coder (Guo et al., 2024a), and Qwen2.5-Coder (Hui et al., 2024) incorporate the fill-in-the-middle (FIM) task into their training stage for code completion. Moreover, many in-file benchmarks are proposed to evaluate capabilities of code LLMs (Zheng et al., 2023; Austin et al., 2021; Jain et al., 2024a; Chai et al., 2024).

Repository-level Code Completion. The latest repository-level code completion methods (Bairi et al., 2023; Phan et al., 2024; Liao et al., 2023; Shrivastava et al., 2023a; Agrawal et al., 2023; Shrivastava et al., 2023b; Pei et al., 2023; Zhang et al., 2023; Yu et al., 2024; Ding et al., 2022) aim to retrieve related code snippets across files within a repository, where existing datasets mainly focus on limited programming languages. For example, RepoBench (Liu et al., 2023b) and CrossCodeEval (Ding et al., 2023) only support 2 and 4 languages, respectively. To comprehensively evaluate the multilingual repository-based code completion, we propose M^2RC-EVAL with 18 languages.

3 M^2RC-EVAL

3.1 Data Collection

The Overall Data Pool. We collect the training data from The Stack v2 (Lozhkov et al., 2024) with permissively licensed repositories from GitHub. Further, we keep only repositories receiving more than 5 stars and containing [10, 50] files. Lastly, we preserve files written in 18 common languages, and obtain 431,353,244 files.

Completion Cursor Position Selection. Completion cursor position selection significantly impacts the quality of a code completion benchmark. Previous studies (Ding et al., 2024; Liu et al., 2023a) randomly select a segment of consecutive characters as the completion span, which does not guarantee the integrity of identifiers and statements.

<table>
  <tr>
    <th>Benchmark</th>
    <th># Lang</th>
    <th>FG</th>
    <th>Training</th>
    <th># Test Repos</th>
  </tr>
  <tr>
    <td>RepoBench</td>
    <td>2</td>
    <td>x</td>
    <td>✓</td>
    <td>1669</td>
  </tr>
  <tr>
    <td>CrossCodeEval</td>
    <td>4</td>
    <td>x</td>
    <td>x</td>
    <td>1002</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>18</td>
    <td>✓</td>
    <td>✓</td>
    <td>5993</td>
  </tr>
</table>

Table 1: A comparison with existing notable other datasets. “FG” denotes “Fine-grained”.

Besides, recent works (e.g., Qwen2.5-Coder (Hui et al., 2024), aiXcoder (Jiang et al., 2024)) also claimed that developers often expect LLMs to complete the current code into a complete snippet, such as a completed code line or loop block, instead of suggesting an incomplete code snippet. Therefore, in M^2RC-EVAL, we first parse the abstract syntax tree (AST) of each source code file, and then we randomly choose a node (e.g., the node of “Function Definition” in Fig. 3) on the AST as the completion cursor position. After that, we obtain the corresponding code to obtain the ground-truth for the current completion cursor position. Finally, at inference, the code LLMs need to predict the current code span given the in-file and cross file contexts. Similarly, in training, we just use the ground-truth to tune code LLMs.

3.2 Quality Control

We build a suite of post-processing filters to enhance the quality of M^2RC-INSTRUCT. We eliminate examples based on two heuristic rules: (1) The completion cursor position should be no longer than 5 lines. (2) If the completion ground truth is fewer than 20 characters, at least 20% of them should be alphabetic. To improve data independence and inference difficulty, we apply extra filters to the test cases in M^2RC-EVAL. (a) Repositories in M^2RC-EVAL should be absent from M^2RC-INSTRUCT. (b) We ensure that 30% of the completion ground truth is not shorter than 2 lines. (c) The completion cursor position should not be fully white-spaced. (d) We discard test cases that could be exactly predicted by DeepSeekCoder-1.3B (Guo et al., 2024b) without cross file contexts.

3.3 Dataset Statistics

Following the quality filters in §(3.2) from the overall data pool §(3.1). We sample 50,000 files per language to construct our M^2RC-INSTRUCT, and sample 100, and 500 files per language to build the validation and test sets of our M^2RC-EVAL, respectively. The statistics of the test set are shown in Fig. 2, and we also provide a detailed com-
Figure 2: The average prompt length (100x tokens), completion span length (50x tokens), and cross-file dependencies (1x) in the testing set of M^2RC-EVAL. We define the number of other files, which are explicitly imported and implicitly referenced by the current file, as cross-file dependencies.

Table 2: Semantic-level annotations on different types of programming languages.

<table>
  <tr>
    <th>Major Classes</th>
    <th>Java</th>
    <th>Go</th>
    <th>Scala</th>
  </tr>
  <tr>
    <td>Program Structure</td>
    <td>"Program Entry", "Namespace", "Import/Include"</td>
    <td>"Program Entry", "Namespace", "Import/Include"</td>
    <td>"Program Entry", "Namespace", "Import/Include"</td>
  </tr>
  <tr>
    <td>Declaration and Definition</td>
    <td>"Class", "Function", "Variable"</td>
    <td>"Class", "Function", "Variable"</td>
    <td>"Class", "Function", "Variable"</td>
  </tr>
  <tr>
    <td>Control Flow Structure</td>
    <td>"Conditional", "Loop", "Jump", "Exception Handling"</td>
    <td>"Conditional", "Loop", "Jump", "Exception Handling"</td>
    <td>"Conditional", "Loop", "Jump", "Exception Handling"</td>
  </tr>
  <tr>
    <td>Expression</td>
    <td>"Arithmetic Operation", "Logical Operation", "Function Call", "Object Creation", "Type Casting", "Other", "Arithmetic Operator", "Logical Operator"</td>
    <td>"Arithmetic Operation", "Logical Operation", "Function Call", "Object Creation", "Type Casting", "Arithmetic Operator", "Logical Operator"</td>
    <td>"Arithmetic Operation", "Function Call", "Object Creation", "Type Casting", "Tuple Expression", "Logical Operator", "Special Operator"</td>
  </tr>
  <tr>
    <td>Data Type</td>
    <td>"Primitive Type", "Composite Type", "Generic", "Numeric", "String", "Boolean", "Special Value"</td>
    <td>"Primitive Type", "Composite Type", "Generic"</td>
    <td>"Primitive Type", "Composite Type", "Generic", "Numeric", "String", "Boolean", "Special Value"</td>
  </tr>
  <tr>
    <td>Statement</td>
    <td>"Expression Statement", "Compound Statement", "Other Statement"</td>
    <td>"Expression Statement", "Compound Statement"</td>
    <td>"Compound Statement"</td>
  </tr>
  <tr>
    <td>Modifier and Attribute</td>
    <td>"Access Modifiers", "Other Modifiers", "Attribute Annotation"</td>
    <td>"Access Modifiers", "Other Modifiers", "Attribute Annotation"</td>
    <td>"Access Modifiers", "Other Modifiers", "Annotation"</td>
  </tr>
  <tr>
    <td>Comments and Documentation</td>
    <td>"Single-line Comment", "Multi-line Comment"</td>
    <td>"Single-line Comment"</td>
    <td>"Single-line Comment", "Multi-line Comment"</td>
  </tr>
  <tr>
    <td>Preprocessing Directive</td>
    <td>"Conditional Compilation", "Macro Definition"</td>
    <td>"Conditional Compilation", "Macro Definition"</td>
    <td>"Macro Definition"</td>
  </tr>
  <tr>
    <td>Identifier and Scope</td>
    <td>"Identifier", "Qualified Name"</td>
    <td>"Identifier", "Qualified Name"</td>
    <td>"Identifier", "Qualified Name", "Binding", "Delimiter"</td>
  </tr>
  <tr>
    <td>Special Language Structure</td>
    <td>"Lambda Expression", "Pattern Matching", "Coroutine"</td>
    <td>"Lambda Expression", "Coroutine"</td>
    <td>"Lambda Expression", "Pattern Matching"</td>
  </tr>
</table>

parison between our M^2RC-EVAL with existing repository-level code completion datasets in Table 1. Note that the numbers of repositories for M^2RC-INSTRUCT, validation split of M^2RC-EVAL are 37439 and 1635, respectively.

3.4 Fine-grained Annotations

As shown in Fig. 3, to analyze the performance in a fine-grained manner, we further provide two types of fine-grained annotations (i.e., bucket-level and semantic-level) for each completion cursor. Specifically, we first generate the abstract syntax tree. For the bucket-level annotations, we first simply divide each tree into \( M \) buckets based on the depth degree of the abstract syntax tree. Note that we set \( M \) as 10 in our M^2RC-EVAL. For example, if the number of layers for the current abstract syntax tree is \( N \), the \( i \)-th layer of the tree belongs to the \( \lceil \frac{i}{\sqrt{M}} \rceil \) bucket. Then, for each completion cursor node, we annotate the bucket label based on the layer number of each node. Similarly, for the semantic-level annotations, we directly annotate the semantic-level label for each completion cursor node. Specifically, we pre-define 11 major classes (i.e., “Program Structure”, “Declaration and Definition”, “Control Flow Structure”, “Expression”, “Data Type”, “Statement”, “Modifier and Attribute”, “Comments and Documentation”, “Preprocessing Directive”, “Identifier and Scope”, “Special Language
Figure 3: Illustration on generating completion cursor position and fine-grained annotations. Specifically, we first parse the source code into an abstract syntax tree (AST). Then, we choose one node as the completion cursor position and generate the bucket label based on the belonged layer number in AST, and obtain the semantic label based on the node type parsed by the Tree-sitter.

Structure"). Then, as different languages have many specific designs, the subcategories under each major class are carefully annotated for different languages. In Table 2, we provide the semantic-level annotations on three main-stream programming languages (Java, Go, Scala), where the annotations on all 18 languages are provided in Fig. 10, Fig. 11 and Fig. 12 of the Appendix.

4 Experiments

4.1 Evaluation Models and Metrics
We evaluate three Code LLMs (i.e., StarCoder-7B (Li et al., 2023), DeepSeekCoder-6.7B (Guo et al., 2024b) and Code Llama-7B (Roziere et al., 2023)) (See Appendix A.2 for more details) on M^2RC-EVAL. Following (Ding et al., 2023), we compare the generated code with the reference and compute the exact match (EM) and edit similarity (ES) metrics 3, which assess the textual similarities and ignore semantic structure similarities among predictions and ground-truth.

4.2 Experimental Setup
Baseline. Only the original code file, where the cursor position is located, is provided for the code LLMs. As no explicit inter-file context is supplied, the model must utilize its inherent knowledge-based reasoning abilities to generate code.
+ Retrieval. In line with the approach outlined in CrossCodeEval (Ding et al., 2023), the retrieval process begins by examining files within the same repository. Continuous code segments of \( L \) lines are extracted, where \( L \) matches the length of the retrieval query and is set as 10 by default. Subsequently, these extracted candidates are prioritized based on their Jaccard similarity scores. The most relevant fragments are then appended to the beginning of the in-file context in descending order of similarity. This concatenation continues until the total length, including both the added candidates and the original in-file context, reaches the predetermined maximum token limit of 4096.
+ Retrieval & Tuning. To further improve the performance of repository-level code completion, we fine-tune code LLMs on M^2RC-INSTRUCT mentioned in §(3). At inference, we use the same inference strategy as discussed in “+ Retrieval”.

4.3 Main Results
We present the results on M^2RC-EVAL in Table 3. We observe that different code LLMs have different repository-level code completion abilities for different programming languages. For instance, DeepSeekCoder-6.7B demonstrates strong completion ability for Go, while its performance is weaker with HTML, a markup language, which demonstrates the necessity of evaluating code LLMs for multilingual capabilities. Besides, the results indicate that cross-file context is highly effective, resulting in a significant improvement compared to using only in-file context. In particular, the multilingual SFT on our created instruction corpora M^2RC-INSTRUCT also significantly enhances performance on M^2RC-EVAL. Notably, after SFT on M^2RC-INSTRUCT, Code Llama-7B, which originally ranked lowest with in-file context, outperformed the non-finetuned StarCoder-7B, demonstrating the effectiveness of M^2RC-INSTRUCT.

4.4 Analysis
Analysis on different model sizes. In Table 4, we provide the results on the validation set of M^2RC-EVAL. Notably, StarCoder-7B consistently outperforms StarCoder-3B under comparable conditions. However, after SFT on M^2RC-INSTRUCT, the results of StarCoder-3B exceed those of the inference-only StarCoder-7B. This finding underscores the effectiveness of our M^2RC-INSTRUCT in augmenting the capabilities of smaller models in repository-level code completion.
Analysis on different training data sizes. In Table 5, we evaluate the fine-tuned StarCoder-7B by employing varying sizes of M^2RC-INSTRUCT and report the results on the validation set of M^2RC-
Table 3: Exact match (%) and edit similarity (%) performance on M^2RC-EVAL.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">C</th>
    <th colspan="2">C#</th>
    <th colspan="2">C++</th>
    <th colspan="2">Go</th>
    <th colspan="2">HTML</th>
    <th colspan="2">Haskell</th>
    <th colspan="2">-</th>
  </tr>
  <tr>
    <th>EM</th><th>ES</th>
    <th>EM</th><th>ES</th>
    <th>EM</th><th>ES</th>
    <th>EM</th><th>ES</th>
    <th>EM</th><th>ES</th>
    <th>EM</th><th>ES</th>
    <th>EM</th><th>ES</th>
  </tr>
  <tr>
    <td>Code Llama-7B</td><td>18.6</td><td>47.2</td><td>19.6</td><td>52.6</td><td>21.8</td><td>51.1</td><td>26.0</td><td>53.6</td><td>20.6</td><td>40.4</td><td>22.6</td><td>48.9</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>21.8</td><td>47.2</td><td>22.9</td><td>48.9</td><td>23.2</td><td>46.6</td><td>23.8</td><td>52.4</td><td>12.6</td><td>35.6</td><td>22.6</td><td>48.9</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>45.4</td><td>72.0</td><td>43.5</td><td>72.3</td><td>50.8</td><td>74.9</td><td>43.4</td><td>72.9</td><td>41.8</td><td>63.6</td><td>39.8</td><td>66.3</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>StarCoder-7B</td><td>20.0</td><td>50.4</td><td>20.0</td><td>53.3</td><td>22.4</td><td>51.8</td><td>25.4</td><td>58.2</td><td>17.4</td><td>40.7</td><td>25.0</td><td>51.1</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>23.8</td><td>47.8</td><td>27.1</td><td>53.2</td><td>24.6</td><td>48.0</td><td>26.0</td><td>53.6</td><td>20.6</td><td>40.4</td><td>25.0</td><td>47.7</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>47.0</td><td>72.7</td><td>45.1</td><td>74.8</td><td>52.4</td><td>76.3</td><td>43.2</td><td>73.7</td><td>45.8</td><td>67.1</td><td>44.8</td><td>70.2</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-6.7B</td><td>22.4</td><td>53.7</td><td>21.4</td><td>56.2</td><td>23.2</td><td>54.2</td><td>29.4</td><td>61.4</td><td>17.6</td><td>43.4</td><td>25.2</td><td>51.3</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>28.2</td><td>52.6</td><td>25.3</td><td>52.6</td><td>27.6</td><td>52.2</td><td>29.4</td><td>61.4</td><td>17.6</td><td>43.4</td><td>25.8</td><td>51.0</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>48.6</td><td>75.2</td><td>47.9</td><td>76.9</td><td>54.4</td><td>78.2</td><td>48.8</td><td>78.4</td><td>45.0</td><td>66.3</td><td>45.8</td><td>72.0</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <th colspan="14">Model</th>
  </tr>
  <tr>
    <th colspan="2">Java</th>
    <th colspan="2">JavaScript</th>
    <th colspan="2">Kotlin</th>
    <th colspan="2">Lua</th>
    <th colspan="2">Objective-C</th>
    <th colspan="2">PHP</th>
    <th colspan="2">-</th>
  </tr>
  <tr>
    <td>Code Llama-7B</td><td>23.4</td><td>58.5</td><td>17.2</td><td>52.0</td><td>23.6</td><td>57.0</td><td>20.0</td><td>45.7</td><td>17.8</td><td>49.5</td><td>19.2</td><td>54.9</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>23.4</td><td>57.5</td><td>19.6</td><td>48.0</td><td>20.8</td><td>50.0</td><td>19.6</td><td>42.2</td><td>21.4</td><td>46.6</td><td>21.2</td><td>49.0</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>41.8</td><td>74.1</td><td>38.8</td><td>70.1</td><td>45.0</td><td>75.6</td><td>43.8</td><td>70.5</td><td>49.8</td><td>75.9</td><td>45.6</td><td>76.7</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>StarCoder-7B</td><td>24.0</td><td>59.2</td><td>16.6</td><td>52.0</td><td>24.4</td><td>59.3</td><td>21.4</td><td>48.6</td><td>17.6</td><td>49.6</td><td>18.6</td><td>54.4</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>25.0</td><td>53.1</td><td>22.0</td><td>50.8</td><td>22.8</td><td>52.6</td><td>26.4</td><td>48.5</td><td>23.6</td><td>48.0</td><td>18.6</td><td>54.4</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>47.4</td><td>76.9</td><td>38.8</td><td>70.1</td><td>45.0</td><td>75.6</td><td>43.8</td><td>70.5</td><td>50.8</td><td>75.9</td><td>45.6</td><td>76.7</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-6.7B</td><td>22.2</td><td>61.0</td><td>20.4</td><td>56.5</td><td>26.0</td><td>61.0</td><td>22.0</td><td>48.8</td><td>21.0</td><td>55.6</td><td>24.2</td><td>58.6</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>21.6</td><td>51.4</td><td>24.4</td><td>53.6</td><td>26.0</td><td>61.0</td><td>22.0</td><td>49.9</td><td>27.6</td><td>53.5</td><td>28.6</td><td>56.9</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>48.2</td><td>79.1</td><td>43.6</td><td>73.5</td><td>46.0</td><td>75.7</td><td>44.6</td><td>70.6</td><td>52.2</td><td>77.6</td><td>49.8</td><td>78.8</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <th colspan="14">Model</th>
  </tr>
  <tr>
    <th colspan="2">Python</th>
    <th colspan="2">R</th>
    <th colspan="2">Ruby</th>
    <th colspan="2">Rust</th>
    <th colspan="2">Scala</th>
    <th colspan="2">TypeScript</th>
    <th colspan="2">Avg.</th>
  </tr>
  <tr>
    <td>Code Llama-7B</td><td>24.6</td><td>54.2</td><td>15.2</td><td>41.2</td><td>17.2</td><td>45.8</td><td>26.2</td><td>56.0</td><td>22.8</td><td>48.5</td><td>23.4</td><td>52.3</td><td>19.4</td><td>50.3</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>17.4</td><td>46.4</td><td>15.2</td><td>39.8</td><td>17.2</td><td>42.3</td><td>26.0</td><td>51.3</td><td>22.8</td><td>48.5</td><td>19.4</td><td>48.6</td><td>20.2</td><td>46.1</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>39.2</td><td>69.9</td><td>38.6</td><td>65.5</td><td>43.0</td><td>68.5</td><td>42.0</td><td>69.2</td><td>41.0</td><td>70.1</td><td>37.0</td><td>68.2</td><td>41.9</td><td>70.0</td>
  </tr>
  <tr>
    <td>StarCoder-7B</td><td>19.4</td><td>52.9</td><td>16.4</td><td>43.7</td><td>19.4</td><td>47.4</td><td>26.2</td><td>56.0</td><td>23.6</td><td>53.4</td><td>19.8</td><td>53.3</td><td>21.0</td><td>52.0</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>24.6</td><td>54.2</td><td>22.6</td><td>47.2</td><td>23.6</td><td>47.4</td><td>26.4</td><td>53.5</td><td>22.8</td><td>48.5</td><td>23.4</td><td>52.3</td><td>24.1</td><td>50.0</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>39.2</td><td>69.9</td><td>41.0</td><td>66.6</td><td>43.0</td><td>68.5</td><td>45.8</td><td>72.6</td><td>43.6</td><td>71.5</td><td>39.2</td><td>69.7</td><td>44.5</td><td>72.2</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-6.7B</td><td>21.8</td><td>55.1</td><td>19.4</td><td>48.5</td><td>23.6</td><td>52.2</td><td>23.8</td><td>54.3</td><td>24.6</td><td>56.7</td><td>19.4</td><td>55.4</td><td>22.6</td><td>54.7</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>21.8</td><td>55.1</td><td>19.4</td><td>48.5</td><td>23.6</td><td>52.2</td><td>23.8</td><td>54.3</td><td>22.4</td><td>50.4</td><td>26.0</td><td>54.5</td><td>25.1</td><td>51.7</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>41.6</td><td>71.3</td><td>45.4</td><td>69.4</td><td>45.6</td><td>70.3</td><td>47.6</td><td>73.4</td><td>44.8</td><td>73.7</td><td>43.2</td><td>73.4</td><td>46.8</td><td>74.1</td>
  </tr>
</table>

Table 4: Performance on M^2RC-EVAL.

<table>
  <tr>
    <th>Model</th>
    <th>EM</th>
    <th>ES</th>
  </tr>
  <tr>
    <td>StarCoder-3B</td><td>14.9</td><td>43.5</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>14.6</td><td>38.4</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>41.7</td><td>69.1</td>
  </tr>
  <tr>
    <td>StarCoder-7B</td><td>20.6</td><td>49.9</td>
  </tr>
  <tr>
    <td>+ Retrieval</td><td>23.6</td><td>49.3</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td><td>44.4</td><td>71.4</td>
  </tr>
</table>

Table 5: Performance under different training data sizes.

<table>
  <tr>
    <th>Data Size (Per lang.)</th>
    <th>100</th>
    <th>1k</th>
    <th>5k</th>
    <th>10k</th>
    <th>50k</th>
  </tr>
  <tr>
    <td>EM (Avg.)</td>
    <td>23.4</td>
    <td>35.7</td>
    <td>40.5</td>
    <td>42.4</td>
    <td>44.4</td>
  </tr>
  <tr>
    <td>ES (Avg.)</td>
    <td>49.1</td>
    <td>62.9</td>
    <td>68.2</td>
    <td>69.4</td>
    <td>71.4</td>
  </tr>
</table>

EVAL. Our observations indicate that increasing the dataset from 0.1k to 50k samples per language yields improved results. This suggests that more training data can help boost the model’s performance. Therefore, we select 50k samples per language as the default training set size.

Analysis on the granularity of different bucket levels. As mentioned in §(3.4), we categorize M^2RC-EVAL into ten bucket levels based on the positions of the code requiring completion within the abstract syntax tree. As shown in Fig. 4, we presents the performance of StarCoder-7B on the test set of M^2RC-EVAL across these different bucket levels, and we observe that as the bucket level decreases, the performance of StarCoder-7B correspondingly declines, which means that the
Table 6: CodeBLEU results on ten representative programming languages.

<table>
  <tr>
    <th>Model</th>
    <th>C</th>
    <th>C#</th>
    <th>C++</th>
    <th>Go</th>
    <th>Java</th>
    <th>JavaScript</th>
    <th>PHP</th>
    <th>Python</th>
    <th>Ruby</th>
    <th>Rust</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>StarCoder-7B</td>
    <td>48.3</td>
    <td>48.9</td>
    <td>50.4</td>
    <td>51.5</td>
    <td>50.6</td>
    <td>46.4</td>
    <td>48.2</td>
    <td>46.4</td>
    <td>46.1</td>
    <td>50.4</td>
    <td>48.7</td>
  </tr>
  <tr>
    <td>+ Retrieval</td>
    <td>50.1</td>
    <td>52.3</td>
    <td>51.1</td>
    <td>52.5</td>
    <td>51.4</td>
    <td>49.3</td>
    <td>52.2</td>
    <td>49.3</td>
    <td>49.1</td>
    <td>51.4</td>
    <td>50.9</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td>
    <td>56.0</td>
    <td>57.4</td>
    <td>57.6</td>
    <td>57.0</td>
    <td>57.6</td>
    <td>54.8</td>
    <td>57.8</td>
    <td>52.0</td>
    <td>52.9</td>
    <td>55.5</td>
    <td><b>55.9</b></td>
  </tr>
</table>

![Line graph showing performance of different bucket levels](page_186_370_482_246.png)

Figure 4: Effectiveness of different bucket levels.

Table 7: Performance on M^2RC-EVAL.

<table>
  <tr>
    <th>Model</th>
    <th>Syntax Accuracy</th>
  </tr>
  <tr>
    <td>StarCoder-7B</td>
    <td>82.8</td>
  </tr>
  <tr>
    <td>+ Retrieval</td>
    <td>83.9</td>
  </tr>
  <tr>
    <td>+ Retrieval & Tuning</td>
    <td>96.9</td>
  </tr>
</table>

code completion on the shallow layer is usually more challenging than on the deep layer. For more experimental data on single-language completion performance and its relation to bucket levels, please refer to Fig.7 and Fig.8 in the Appendix. These findings suggest that the code LLMs encounter challenges when addressing shallow nodes within the syntax tree during the code completion process.

Analysis on the granularity of different semantic levels. Similarly, in §(3.4), we also categorize the nodes within the abstract syntax tree into eleven primary semantic levels based on their semantic characteristics, and we provide the performance of StarCoder-7B for these various semantic levels across multilingual languages on the test set of the M^2RC-EVAL, as illustrated in Fig. 5. Notably, we observe significant performance disparities across different semantic levels. Specifically, StarCoder-7B shows superior performance on “Identifier and Scope”, while it exhibits lower efficacy on “Special Language Structure”. This suggests that current code LLMs are proficient at completing tasks related to variable definitions and references, yet their capacity to handle characteristics of different languages requires further enhancement. For single-language completion performance across various node types, please refer to Fig. 9 in the Appendix.

Analysis on completion on different lines. As shown in Fig.6, StarCoder-7B can effectively complete tasks involving a small number of lines. However, as the number of lines to be completed increases, the scores of the generated code gradually decline. This indicates that completing multi-line code remains a challenge for code LLMs.

Analysis on various input lengths. In Fig. 6, we report the results produced by StarCoder-7B (“Retrieval & Tuning”) on our M^2RC-EVAL when the input lengths of range in {512, 1024, 2048, 4096} tokens. In Fig. 6, we observe that a scaling law exists, where better performance is achieved when the input length is larger. Thus, we set the default input length as 4096 tokens.

Analysis on CodeBLEU metric. In Table 3, we mainly report the EM and ES metrics based on the textual similarity, which neglects important syntactic and semantic features of codes and underestimates different outputs with the same semantic logic. Thus, the CodeBLEU (Ren et al., 2020)⁴ is proposed, which considers information from not only the shallow match, but also the syntactic match and the semantic match. In Table 6, we report the results of 10 popular programming languages using the test split of M^2RC-EVAL based on the StarCoder-7B model and observe that we can still achieve better performance by fine-tuning on our constructed M^2RC-INSTRUCT, which further demonstrates the effectiveness of our M^2RC-INSTRUCT on repository-level code completion.

Analysis on syntax accuracy. In Table 7, for syntax static analysis, to further verify the syntax correctness of the predicted code snippets, we use the code static checking tools (Tree-Sitter) for all predicted code snippets of M^2RC-EVAL. Specifically, we parse the code snippet into the abstract syntax tree and filter out the code snippet, where the parsed nodes in the code snippet have parsing errors. For

⁴We test the CodeBLEU metric based on https://github.com/k4black/codebleu.
Figure 5: Effectiveness of different semantic levels based on StarCoder-7B.

Table 8: Results of \( \mathcal{M}^2\mathrm{RC}\)-Eval based on paradigm types.

<table>
  <tr>
    <th>Paradigm Types</th>
    <th>StarCoder</th>
    <th>+ Retrieval</th>
    <th>+ Retrieval & Tuning</th>
  </tr>
  <tr>
    <td>Procedural</td>
    <td>19.2</td>
    <td>23.7</td>
    <td>47.6</td>
  </tr>
  <tr>
    <td>Object Oriented</td>
    <td>21.3</td>
    <td>24.2</td>
    <td>47.4</td>
  </tr>
  <tr>
    <td>Multiple Paradigms</td>
    <td>21.1</td>
    <td>24.3</td>
    <td>43.4</td>
  </tr>
  <tr>
    <td>Functional</td>
    <td>25.1</td>
    <td>24.8</td>
    <td>44.6</td>
  </tr>
  <tr>
    <td>Markup Language</td>
    <td>17.2</td>
    <td>21.3</td>
    <td>46.7</td>
  </tr>
</table>

Table 9: Results of \( \mathcal{M}^2\mathrm{RC}\)-Eval based on application scenarios.

<table>
  <tr>
    <th>Application Scenarios</th>
    <th>StarCoder</th>
    <th>+ Retrieval</th>
    <th>+ Retrieval & Tuning</th>
  </tr>
  <tr>
    <td>Mobile</td>
    <td>21.2</td>
    <td>22.9</td>
    <td>48.0</td>
  </tr>
  <tr>
    <td>Cross Platform</td>
    <td>24.2</td>
    <td>25.3</td>
    <td>48.1</td>
  </tr>
  <tr>
    <td>Desktop Application</td>
    <td>18.7</td>
    <td>26.1</td>
    <td>45.3</td>
  </tr>
  <tr>
    <td>Web Frontend</td>
    <td>17.9</td>
    <td>22.2</td>
    <td>41.5</td>
  </tr>
  <tr>
    <td>Web Backend</td>
    <td>22.8</td>
    <td>24.8</td>
    <td>44.6</td>
  </tr>
  <tr>
    <td>Scientific Computing</td>
    <td>18.2</td>
    <td>23.5</td>
    <td>40.3</td>
  </tr>
  <tr>
    <td>System & Software</td>
    <td>21.3</td>
    <td>24.0</td>
    <td>50.1</td>
  </tr>
  <tr>
    <td>Education & Research</td>
    <td>25.1</td>
    <td>24.8</td>
    <td>44.6</td>
  </tr>
  <tr>
    <td>Automation Scripts</td>
    <td>21.7</td>
    <td>26.3</td>
    <td>43.7</td>
  </tr>
</table>

Figure 6: Left figure: Results of different lines. Right figure: Results of various input lengths.

execution analysis, generating unit test cases and providing execution sandboxes for repository-level code completion are very challenging. In Table 7, we observe that tuning leads to better performance in both syntax and execution analysis. Notably, we observe that syntax accuracy improves a lot after tuning, which means that code LLMs can easily learn the basic syntax rules for programming languages.

Analysis on language-specific insights. We have classified 18 programming languages in \( \mathcal{M}^2\mathrm{RC}\)-Eval into 5 programming paradigms and 9 application scenarios as shown in Table 11 and Table 12: Based on the above programming classification structure, we also report the EM results based on StarCoder-7B as shown in Table 8 and Table 9, and have the following observations: (1). For different programming paradigms, we observe that the markup language paradigm has the lowest performance, and the functional paradigm has the best performance. Besides, after tuning, the performance of the markup language paradigm improves greatly. We assume that the syntax rules for markup language are easy, and these code LLMs can quickly obtain these rules after tuning. (2). For different application scenarios, the performance varies significantly. Specifically, the Web Frontend and Scientific Computing have relatively low performance, which needs to be improved for existing code LLMs. (3). We observe these LLMs share some common strengths and weaknesses and we
will continue to investigate more language-specific insights to better improve the code completion abilities of existing code LLMs.

5 Conclusion

In this paper, we propose the first massively multilingual repository-level code completion benchmark (M^2RC-EVAL) with 18 popular programming languages, where two types of fine-grained annotations (bucket-level and semantic-level) are provided to comprehensively analyze the effectiveness of different code LLMs. Besides, we also curate a high-quality instruction corpus M^2RC-INSTRUCT to enhance the performance of existing models on repository-level code completion. Finally, we hope M^2RC-EVAL could facilitate the growth of code intelligence and software engineering.

6 Limitations

First, there are several hyperparameters (e.g., training sizes, input length) to tune, which is laborious and expensive. Second, the current work only focuses on the repository-level code completion task, where other repository-level code intelligence tasks are not considered. Third, while M^2RC-EVAL aims to simulate realistic debugging scenarios, the tasks and data may not fully capture the complexity and diversity of real-world software development. We will continue include challenging and complex real-world repos into the benchmark.

7 Acknowledgement

This work was supported by the Jiangsu Science and Technology Major Project (BG2024031) and Nanjing University AI & AI for Science Funding (2024300540).

References

Lakshya A Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K Lahiri, and Sriram K Rajamani. 2023. Guiding language models of code with global context using monitors.

Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. Santacoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B Ashok, Shashank Shet, et al. 2023. Codeplan: Repository-level coding using llms and planning. arXiv preprint arXiv:2309.12499.

Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022a. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255.

Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022b. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255.

Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pielet, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95–136, virtual+Dublin. Association for Computational Linguistics.

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2022. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227.

Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. 2024. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. 24(240):1–113.

Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. 2024. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36.
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Han-tian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2022. Cocomic: Code completion by jointly modeling in-file and cross-file context. arXiv preprint arXiv:2212.10007.

Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024a. Deepseek-coder: When the large language model meets programming—the rise of code intelligence. arXiv preprint arXiv:2401.14196.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024b. Deepseek-coder: When the large language model meets programming—the rise of code intelligence. arXiv preprint arXiv:2401.14196.

Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, and Bo Zheng. 2025. Can large language models detect errors in long chain-of-thought reasoning? Preprint, arXiv:2502.19361.

Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. 2024. Opencoder: The open cookbook for top-tier code large language models.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tiantyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024a. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.

Nihal Jain, Robert Kwiatkowski, Baishakhi Ray, Murali Krishna Ramanathan, and Varun Kumar. 2024b. On mitigating code llm hallucinations with api documentation. ArXiv, abs/2407.09726.

Siyuan Jiang, Jia Li, He Zong, Huanyu Liu, Hao Zhu, Shukai Hu, Erliu Li, Jiazheng Ding, Yu Han, Wei Ning, Gen Wang, Yihong Dong, Kechi Zhang, and Ge Li. 2024. aixocoder-7b: A lightweight and effective large language model for code completion.

Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. 2022. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533.

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. ArXiv, abs/2207.01780.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.

Yuji Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. ArXiv preprint, abs/2203.07814.

Dianshu Liao, Shidong Pan, Qing Huang, Xiaoxue Ren, Zhencheng Xing, Huan Jin, and Qinying Li. 2023. Context-aware code generation framework for code repositories: Local, global, and third-party library awareness.

Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Jie Liu, Ge Zhang, Yanan Wu, Congnan Liu, et al. 2024. Ddk: Distilling domain knowledge for efficient large language models. Advances in Neural Information Processing Systems, 37:98297–98319.

Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. 2025. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407.

Tianyang Liu, Canwen Xu, and Julian McAuley. 2023a. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091.

Tianyang Liu, Canwen Xu, and Julian J. McAuley. 2023b. Repobench: Benchmarking repository-level code auto-completion systems. abs/2306.03091.

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtiar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representations.

Hengzhi Pei, Jinman Zhao, Leonard Lausen, Sheng Zha, and George Karypis. 2023. Better context makes better code language models: A case study on function call argument completion. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 23/IAAI 23/EAII’23. AAAI Press.

Huy Nhat Phan, Hoang N. Phan, Tien N. Nguyen, and Nghĩa D. Q. Bui. 2024. Repohyper: Search-expand-refine on semantic graphs for repository-level code completion.

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code.

Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, and Torsten Scholak. 2023a. Repofusion: Training code models to understand your repository. arXiv preprint arXiv:2306.10998.

Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023b. Repository-level prompt generation for large language models of code. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 31693–31715. PMLR.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. 568:127063.

Wannita Takerngsaksiri, Chakkrit Tantithamthavorn, and Yuan-Fang Li. 2024. Syntax-aware on-the-fly code completion. Inf. Softw. Technol., 165(C).

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models.

Pei Wang, Yanan Wu, Noah Wang, Jiaheng Liu, Xiaoshuai Song, Z.Y. Peng, Ken Deng, Chenchen Zhang, JiakaiWang, Junran Peng, Ge Zhang, Hangyu Guo, Zhaoxiang Zhang, Wenbo Su, and Bo Zheng. 2024. Mtu-bench: A multi-granularity tool-use benchmark for large language models.

Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation.

Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696–8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Frank F Xu, Uri Alon, Graham Neubig, and Vincent Joshua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1–10.

Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1–12.

Alexander Zhang, Marcus Dong, Jiaheng Liu, Wei Zhang, Yejie Wang, Jian Yang, Ge Zhang, Tianyu Liu, Zhongyuan Peng, Yingshui Tan, Yuanxing Zhang, Zhexu Wang, Weixun Wang, Yancheng He, Ken Deng, Wangchunshu Zhou, Wenhao Huang, and Zhaoxiang Zhang. 2025. Codecriticbench: A holistic code critique benchmark for large language models. Preprint, arXiv:2502.16614.

Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570.

Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Ying-hao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhui Chen. 2024. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv: 2405.19327.

CodeGemma Team Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A. Choquette-Choo, Jingyue Shen, Joe Kelley, Kshi tij Bansal, Luke Vilnis, Mateo Wirth, Paul Michel, Peter Choy, Pratik Joshi, Ravin Kumar, Sar-mad Hashmi, Shubham Agrawal, Zhitao Gong, Jane
Fine, Tris Brian Warkentin, Ale Jakse Hartman, Bin Ni, Kathy Korevec, Kelly Schaefer, and Scott Huffman. 2024. Codegemma: Open code models based on gemma. ArXiv, abs/2406.11409.

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. Codegex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568, abs/2303.17568.

Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kad-dour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. 2025. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations.
A Appendix

A.1 Broader Impacts & Potential Risks

In this paper, we propose a repository-level code completion benchmark with 18 programming languages. Therefore, we hope our work can enhance the improvements on the multilingual repository-level code completion task. For potential risks, we have not seen any risks in our M^2RC-EVAL.

A.2 Details of the Baseline Models

StarCoder (Li et al., 2023) is a series of generative language models (e.g., 7B, 15.5B). These decoder-only models are trained on the Stack dataset (Kocetkov et al., 2022) and can support 8K tokens in context.
DeepSeekCoder (Guo et al., 2024b) is a collection of code-oriented models with capacities from 1.3B to 33B parameters. Trained on a manually curated 2-trillion-token corpus, these models leverage Fill-in-the-Middle (FIM) (Bavarian et al., 2022b) and Rotary Position Embedding (RoPE) (Su et al., 2024) techniques, which enables efficient code generation and infilling within a 16K token window.
Code Llama (Roziere et al., 2023) is a family of code large language models based on Llama 2 (Touvron et al., 2023) with 7B, 13B, 34B, and 70B parameters. While trained on 16K token sequences, these models can handle inputs up to 100K tokens during inference.
Note that we just use the base model versions of these three models.

A.3 Analysis on the Quality Control

In §(3.2), we discard test samples that could be exactly predicted by DeepSeekCoder-1.3B without cross-file contexts. Meanwhile, to discuss more clearly, we also use the DeepSeekCoder-6.7B, StarCoder-7B, and DeepSeekCoder-33B to analyze the ratios of evaluation cases with or without using repository-level contexts. Specifically, we prompt DeepSeekCoder-6.7B, StarCoder-7B, and DeepSeekCoder-33B using the in-file contexts of each sample and obtain three predictions. If one prediction is exactly matched ground-truth, this sample is considered to be predicted without requiring repository-level contexts. Finally, we observe that 71% samples cannot be well predicted only using in-file contexts, which indicates that it is necessary to use the cross-file contexts to achieve better performance in our M^2RC-EVAL..

A.4 Analysis on Inference Cases in Different Languages

We manually inspect the behavior of StarCoder-7B (Li et al., 2023) on completion cases in different programming languages. As shown in Fig. 13, the model successively predicts the attribute position.y, which is an easy pattern that could be inferred from position.x in the prefix. Besides, the (x, y) pattern that occurs multiple times in the cross-file context. On the contrary, the model seems to struggle with complex expressions and statements. In Fig. 14, the model should complete the function with a combined condition and return statement. However, the retriever could not provide useful references and the model only predicts half of the condition correctly. Fig. 15 illustrates a Python script to execute memory calculations. Although some calculations appear in the cross-file context, there are no precisely matched calculation procedures. The recurrent conditions in the ground truth require calculations on the data shape, but the model clumsily guesses the data shape. Moreover, we observe that the model prediction is usually affected by frequent identifiers in the retrieved contents. In Fig. 16, the model repeats the gc.Client and results in a hallucination for object PullRequests, where the ground truth is gc. Similarly, in Fig. 17, the model blindly catches the “err” with error level. Yet the correct log level is “warn”, which could be judged from c.on('error', console.warn) in the Cross file Context 1. Further, in Fig. 18, the ground truth and the model prediction differ by only two characters “()” from a textual perspective, but the ground truth passes the method reference while the model prediction passes the method return.

A.5 Analysis on the Number of Bucket Granularity

In Table 10, we have analyzed the minimum, maximum, and average depths of these 18 programming languages, and observed that depths of AST for different languages are different greatly, where the minimum depth is from 1 to 7, and the maximum depth is from 23 to 51. Therefore, we cannot use
the node’s layer number in the AST as the fine-grained annotation well. Besides, we observe that the average depth of different languages is from 9.7 to 20.2, and the overall average depth is 14.6. In our implementation, to explain clearly, we just chose 10 as the bucket number. Note that we will provide the tree depth number for each completion sample, and users can select the corresponding bucket number freely.

Table 10: Depth statistics of different languages.

<table>
  <tr>
    <th>Language</th>
    <th>Minimum</th>
    <th>Maximum</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Objective-C</td>
    <td>3</td>
    <td>32</td>
    <td>11.3</td>
  </tr>
  <tr>
    <td>C++</td>
    <td>3</td>
    <td>32</td>
    <td>11.3</td>
  </tr>
  <tr>
    <td>C</td>
    <td>4</td>
    <td>41</td>
    <td>13.8</td>
  </tr>
  <tr>
    <td>Haskell</td>
    <td>1</td>
    <td>28</td>
    <td>10.9</td>
  </tr>
  <tr>
    <td>Ruby</td>
    <td>2</td>
    <td>45</td>
    <td>13.7</td>
  </tr>
  <tr>
    <td>Kotlin</td>
    <td>1</td>
    <td>34</td>
    <td>14.9</td>
  </tr>
  <tr>
    <td>Rust</td>
    <td>5</td>
    <td>51</td>
    <td>20.2</td>
  </tr>
  <tr>
    <td>C#</td>
    <td>3</td>
    <td>41</td>
    <td>16.4</td>
  </tr>
  <tr>
    <td>PHP</td>
    <td>7</td>
    <td>51</td>
    <td>17.7</td>
  </tr>
  <tr>
    <td>Go</td>
    <td>1</td>
    <td>40</td>
    <td>13.6</td>
  </tr>
  <tr>
    <td>Java</td>
    <td>1</td>
    <td>46</td>
    <td>14.3</td>
  </tr>
  <tr>
    <td>HTML</td>
    <td>2</td>
    <td>46</td>
    <td>14.0</td>
  </tr>
  <tr>
    <td>R</td>
    <td>1</td>
    <td>23</td>
    <td>9.7</td>
  </tr>
  <tr>
    <td>JavaScript</td>
    <td>1</td>
    <td>31</td>
    <td>13.4</td>
  </tr>
  <tr>
    <td>TypeScript</td>
    <td>3</td>
    <td>51</td>
    <td>18.3</td>
  </tr>
  <tr>
    <td>Scala</td>
    <td>3</td>
    <td>51</td>
    <td>17.5</td>
  </tr>
  <tr>
    <td>Lua</td>
    <td>2</td>
    <td>51</td>
    <td>16.1</td>
  </tr>
  <tr>
    <td>Python</td>
    <td>1</td>
    <td>38</td>
    <td>15.7</td>
  </tr>
</table>

Table 11: Classification of M^2RC-EVAL based on paradigm types.

<table>
  <tr>
    <th>Paradigm Types</th>
    <th>Languages</th>
  </tr>
  <tr>
    <td>Procedural<br>Object Oriented<br>Multiple Paradigms<br>Functional<br>Markup Language</td>
    <td>C<br>C#, Java, Kotlin, Objective-C<br>C++, Go, JavaScript, Lua, PHP, Python, R, Ruby, Rust, Scala, TypeScript<br>Haskell<br>HTML</td>
  </tr>
</table>

Table 12: Classification of M^2RC-EVAL based on application scenarios.

<table>
  <tr>
    <th>Application Scenarios</th>
    <th>Languages</th>
  </tr>
  <tr>
    <td>Mobile</td>
    <td>Kotlin, Objective-C</td>
  </tr>
  <tr>
    <td>Cross Platform</td>
    <td>Java</td>
  </tr>
  <tr>
    <td>Desktop Application</td>
    <td>C#</td>
  </tr>
  <tr>
    <td>Web Frontend</td>
    <td>JavaScript, TypeScript, HTML</td>
  </tr>
  <tr>
    <td>Web Backend</td>
    <td>Go, PHP, Ruby, Rust, Scala</td>
  </tr>
  <tr>
    <td>Scientific Computing</td>
    <td>Python, R</td>
  </tr>
  <tr>
    <td>System & Software</td>
    <td>C, C++</td>
  </tr>
  <tr>
    <td>Education & Research</td>
    <td>Haskell</td>
  </tr>
  <tr>
    <td>Automation Scripts</td>
    <td>Lua</td>
  </tr>
</table>
A.6  More Experiments
• We provide the analysis on the bucket levels in Fig. 7 and Fig. 8, respectively.

• We analyze the effect of different semantic levels on Rust, Objective-C, and Haskell in Fig. 9. respectively.

• We provide the semantic-level annotations on 18 languages in Fig. 10, Fig. 11 and Fig. 12.

• We provide the results of problems from different difficulty levels in Fig. 19, where we define completion on 1 line, completion on 2-3 lines and completion on 4-5 lines as easy, middle, and hard settings, respectively.
Figure 7: Effectiveness of different bucket levels based on StarCoder-7B for different languages.
Figure 8: Effectiveness of different bucket levels based on StarCoder-7B for different languages.
Figure 9: Effectiveness of different semantic levels based on StarCoder-7B.
Figure 10: Semantic-level annotations on different types of programming languages. "none" is used if this language does not have corresponding subcategories.
Figure 11: Semantic-level annotations on different types of programming languages. "none" is used if this language does not have corresponding subcategories.
Figure 12: Semantic-level annotations on different types of programming languages. “none” is used if this language does not have corresponding subcategories.
Figure 13: Visualization on success case for TypeScript. (Semantic label: Modifier and Attribute)

Figure 14: Visualization on failure case for the C language. (Semantic label: Statement)
Figure 15: Visualization on failure case for Python. (Semantic label: Expression)

Figure 16: Visualization on failure case for Go. (Semantic label: Expression)
Figure 17: Visualization on failure case for Javascript. (Semantic label: Statement)

Figure 18: Visualization on failure case for Scala. (Semantic label: Expression)

Figure 19: Performance on M^2RC-EVAL for problems of different difficulty levels.

Reviewer paper summaries: This paper introduces a repository-level evaluation benchmark dataset called M2RC-EVAL, which aims to evaluate the code completion capabilities of large code language models (including general large language models) in multiple programming languages. The paper describes in detail the dataset construction process including data collection, quality control, and fine-grained annotation. The paper also proposes an instruction fine-tuning dataset called M2RC-INSTRUCT to improve the model's code completion capabilities by fine-tuning this dataset.

This paper introduces M2RC-EVAL, a comprehensive benchmark for evaluating repository-level code completion across 18 programming languages, addressing the limitations of existing benchmarks that focus on few languages and offer only coarse-grained evaluations. To enable fine-grained assessment, the authors provide two types of annotations, including bucket-level (based on AST depth) and semantic-level (based on code functionality), for each completion instance. Alongside the benchmark, they curate M2RC-INSTRUCT, a multilingual instruction dataset designed to improve LLM performance via fine-tuning. The authors evaluate several code LLMs, including Code Llama, StarCoder, and DeepSeekCoder, under different settings (in-file only, retrieval-augmented, and retrieval plus fine-tuning), and show that both cross-file context retrieval and instruction tuning substantially boost completion performance. They further analyze performance by language, semantic category, AST depth, model size, and training data size, offering detailed insights into multilingual code understanding.

This paper introduces M²RC-EVAL, a new benchmark for evaluating repository-level code completion capabilities across 18 programming languages. M²RC-EVAL includes fine-grained annotations at both bucket-level (based on abstract syntax tree layers) and semantic-level (based on code functionality). Additionally, the paper develops M²RC-INSTRUCT, a multilingual instruction dataset to improve the performance of LLMs on repository-level code completion tasks. Through experimentation with different code LLMs (StarCoder-7B, DeepSeekCoder-6.7B, and Code Llama-7B), they demonstrate the effectiveness of both their benchmark and instruction corpus.
Summary of strengths: - This paper is a benchmark covering 18 programming languages, which is a great improvement over previous work[1], and also shows the author's analysis of the performance differences of the model in multiple languages.
- The process of constructing the dataset is very clear. The author also consider the problem of data leakage and conduct corresponding verification experiments.
- The paper is very well written, clearly structured, and comes with a rich discussion and appendices.

[1] CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion

- Good writing and easy to follow.
- Introduces M2RC-EVAL, the first repository-level code completion benchmark covering 18 programming languages
- Presents rigorous experiments across multiple code LLMs, yielding valuable insights into the behavior and limitations of multilingual code models

1. The proposed benchmark (multilingual + 18 languages) significantly advances the field beyond current benchmarks (typically cover only 2-4 programming languages). In addition, the bucket-level and semantic-level annotations allow for detailed model performance analysis across different code completion contexts. 

2. This work also provides an instruction dataset. This is a valuable resource for improving model performance on multilingual code completion tasks.

3. The paper describes quality control steps to ensure benchmark examples are representative of real-world scenarios.

4. Well-written and easy to follow.
Summary of weaknesses: - In the Table 2, the performance of the +Retrieval version of the model is degraded compared to the original model, such as the performance of Code Llama-7B on Python. Can you explain why this happens?
- The semantic-level labels for 18 programming languages in the paper seem to be carefully designed. Can you provide more analysis and reasons for this?

- For the left graph of the Figure 5, the performance of the model gradually decreases as the number of lines to be completed increases, which is very natural. But I noticed that when the number of lines is 5, the performance of the model in terms of ES is actually improved compared to the number of lines 4. I wonder if this is a special case only for the StarCoder-7B?

- While the benchmark is multilingual and fine-grained, the core task, repository-level code completion, has already been explored in prior benchmarks
- The paper relies on several heuristic decisions (e.g., AST bucketization into 10 levels, semantic label mapping across languages, filtering criteria for completion spans) that are not extensively justified or ablated.
- Despite positioning M2RC-EVAL as a repository-level benchmark, it focuses solely on the code completion task.

1. The retrieval step used for cross-file context (Jaccard similarity) is relatively simple. Can the author provide more explanations to illustrate the comprehensiveness or effectiveness of this method? Or more sophisticated retrieval methods might be explored?

2. The benchmark is comprehensive. However, there are too many results just in the table.  A more comprehensive qualitative analysis or insight would help researchers better understand this benchmark and challenge in this domain. 

3. Would like to see more discussion about how language-specific features affect model performance.
Comments, suggestions and typos: - About the Table 3: Place the model name (including +Retrieval and Retrieval & Tuning versions) horizontally and the 18 programming languages vertically. I think that this will make it easier to observe the differences between models in different programming languages.

- Consider clarifying the rationale behind the selection of the 18 programming languages
- Consider extending M2RC-EVAL beyond just code completion to include other repository-level tasks such as code summarization, function naming, bug fixing, or cross-file code navigation
- It would be helpful to include more illustrative examples in the main text (not just figures) showing how different semantic labels affect model performance

N/A
Overall assessment scores: 4

2.5

3.5
Metareview score: 3.5
Metareview: This paper proposes M2RC-EVAL, a new benchmark dedicated to massively multilingual repository-level code completion across 18 programming languages. It also introduces M2RC-INSTRUCT, a multilingual instruction dataset aimed at improving the performance of large language models on repository-level completion tasks through additional fine-tuning. The benchmark’s fine-grained annotations, which include “bucket-level” categories derived from AST depth and “semantic-level” categories reflecting functional code aspects, allow a more nuanced evaluation than prior work. Experiments with various code LLMs (e.g., StarCoder, DeepSeekCoder, Code Llama) show that both the benchmark and instruction corpus provide informative diagnostics and measurable gains, especially when cross-file context retrieval is employed.

Reviewers appreciate the substantial increase in language coverage—18 languages compared to the more common 2–4 in existing benchmarks—and appreciate that the bucket-level and semantic-level labels help uncover subtle performance differences. The authors also offer thorough appendices detailing heuristics, case studies, and ablation analyses, demonstrating the real-world relevance of repository-level completion. They also highlight four main concerns: (1) Clarifying and comparing alternative retrieval strategies; (2) Providing more rationale for heuristic choices (e.g., AST bucket sizes, line constraints); (3) Offering stronger language-specific examples; and (4) Considering broader repository-level tasks (e.g., bug fixing, code summarization). 

In their rebuttal, the authors address these concerns by providing additional experiments, describing the costs and benefits of advanced retrievers, and detailing future plans for more repository-level tasks. The reviewers agree that these clarifications addressed most concerns. Overall, the paper tackles an important and timely challenge: evaluating and improving code LLMs in a multilingual, repository-scale setting.  I am therefore recommending an overall assessment of "3.5 = Borderline Conference".
Summary of reasons to publish: - Covering 18 programming languages is a notable leap forward compared to existing code completion benchmarks, which typically include only a few languages.

- The bucket-level (AST depth) and semantic-level (functional labels) annotations illuminate nuanced performance differentials that coarse-grained metrics might miss.

-  Introducing M2RC-INSTRUCT and analyzing retrieval-augmented approaches demonstrate real-world relevance for code LLM tasks in large, evolving repositories.