M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation

Authors: Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ge Zhang, Noah Wang, JinKe, Bangyu Xiang, Guoan Zhang, Wenbo Su, Bo Zheng
Primary area: datasets and benchmarks
Keywords: Large Language Models, Code Completion, Benchmark

Abstract:
Repository-level code completion has drawn great attention in software engineering, and several benchmark datasets have been introduced. However, existing repository-level code completion benchmarks usually focus on a limited number of languages (<5), which cannot evaluate the general code intelligence abilities across different languages for existing code Large Language Models (LLMs). Besides, the existing benchmarks usually report overall average scores of different languages, where the fine-grained abilities in different completion scenarios are ignored. Therefore, to facilitate the research of code LLMs in multilingual scenarios, we propose a massively multilingual repository-level code completion benchmark covering 18 programming languages (called M2RC-EVAL), and two types of fine-grained annotations (i.e., bucket-level and semantic-level) on different completion scenarios are provided, where we obtain these annotations based on the parsed abstract syntax tree. Moreover, we also curate a massively multilingual instruction corpora M2RC- INSTRUCT dataset to improve the repository-level code completion abilities of existing code LLMs. Comprehensive experimental results demonstrate the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.

Decision: reject
Reviewer scores: 5, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1730079488265

thread_type:
official_review

<thread_object>
creator: reviewer_5s7r

<content>
summary:
This paper introduces M2RC-EVAL, a new benchmark designed to assess repo-level code completion performance across 18 programming languages. The benchmark uses two types of fine-grained annotations: bucket-level, based on the depth of abstract syntax trees, and semantic-level, which focuses on code semantics.  Additionally, the paper presents a multilingual instruction corpus, M2RC-INSTRUCT, to improve code completion models.

strengths:
**Massive multilingual benchmark**: The paper introduces a novel multilingual benchmark that supports 18 programming languages, which is a significant expansion over existing benchmarks that only handle a limited number of languages.

**Quality control**: The dataset construction is considered thorough, with a robust collection process from github repos and clear quality control measures. 

**Clear writings**: This paper is well-organized, with clear explanations of how the benchmark and instruction dataset were constructed and evaluated. The use of tables and figures aids in understanding the experimental results.

Overall, this benchmark contributes to address the growing need for multilingual evaluation in code completion. The fine-grained annotation methodology also provides useful insights into the challenges faced by models in many programming scenarios.

weaknesses:
**Evaluation Metrics:** While the paper mainly uses exact match and edit similarity metrics, these may not fully capture the functional correctness of the generated code. I acknowledge that implementing an execution-based evaluation across such a massive multilingual benchmark poses considerable challenges, and this is more of a broader limitation within the field rather than a direct weakness of this particular work. I do appreciate the authors’ inclusion of CodeBLEU, which offers another perspective by considering syntactic and semantic aspects of code generation. Still, it would be more insightful to explore whether there is any correlation between CodeBLEU scores and the more traditional EM and ES metrics.

questions:
Apart from the point in the 'Weaknesses' section, I have the following questions and suggestions:

**Proprietary Model Comparison**: Do the authors plan to include a comparison with advanced proprietary models, even on a smaller subset of the benchmark? This could offer insights into how fine-tuned open-source models compare against the leading commercial solutions.

**Qualitative Error Analysis (Suggestion):** It would be insightful to provide a more detailed qualitative error analysis to highlight where current code LLMs encounter the most difficulty across different languages. This could help pinpoint areas for further improvement.

**Figure Clarity (Suggestion):** While the visualizations are nicely presented, some figures (e.g., Figures 4 and 14) have very small fonts, making it difficult to read the details. Increasing the font size in these visualizations would enhance clarity and accessibility for readers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666199432

thread_type:
official_review

<thread_object>
creator: reviewer_pyqh

<content>
summary:
The paper releases a cross-file-context repo-level code completion benchmark that tests the fill-in-the-middle (FIM) completion abilities of Code LMs. GitHub repos are collected from Stack V2, and then the files' ASTs are individually processed to create varying categories of FIM holes for completion.

I believe the dataset (especially the instruct split) is a valuable OS contribution. Still, as it joins a whole host of existing benchmarks in this area, it should be fleshed out further regarding research questions.

strengths:
1. The paper enters a crowded field of repo-level code completion benchmarks but differentiates itself from the pack by the size of the data splits and the number of languages supported.
2. The data collection process is clearly detailed

weaknesses:
1. For a benchmark paper, it is a bit light on details of env setup, tooling and sandboxing
2. The evaluation is still based on surface form metrics like exact-match, which can be very noisy, especially for FIM settings. Scaling execution tests can be hard, but existing benchmarks like EvoCodeBench [1] and BigCodeBench [2] have done so using a combination of model generation and human validation.
3. Several very strong recent code model releases, such as Yi-Coder [3], Llama-3.1 [4], and Qwen-2.5-coder [5], are missing from the evals. This is pertinent due to their longer context support (up to 128k tokens), allowing for a much more interesting baseline than the paper currently uses (just the individual file in question). A very interesting evaluation of the paper is missing: how well the benchmark can be solved if the model just placed all the most pertinent files in context instead of just retrieving a few short snippets. Similarly, how would the various in-repo file ordering schemes previously explored by StarCoder-2 [6] and DeepSeek-Coder [7] affect this?

[1] https://arxiv.org/abs/2404.00599
[2] https://arxiv.org/abs/2406.15877
[3] https://arxiv.org/abs/2403.04652
[4] https://arxiv.org/abs/2407.21783
[5] https://arxiv.org/abs/2409.12186
[6] https://arxiv.org/abs/2402.19173
[7] https://arxiv.org/abs/2401.14196

questions:
1. The cross-lingual transfer benchmarks may not be properly baseline. The paper uses StarCoder-1, a multi-PL pre-trained model that is continually pre-trained on Python code. This is a rather non-standard setting and would be fairer (across langs) if it were started from StarCoder-1-Base.
2. It would be interesting to explore the effect of various code retrievers instead of just Jaccard-Similarity in retrieving snippets in-context.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690061255

thread_type:
official_review

<thread_object>
creator: reviewer_ts5v

<content>
summary:
The paper introduces M$^2$RC-EVAL, a benchmark for repository-level code completion covering 18 programming languages, alongside M$^2$RC-INSTRUCT, a multilingual instruction dataset built in a similar way with M$^2$RC-EVAL. Using The Stack v2 dataset, the authors select repositories with 5+ stars containing 10-50 files, creating a dataset with 50,000 files per language for instruction tuning and 600 files per language for evaluation (w/ random 100 examples for validation and 500 for test). The benchmark introduces two types of annotations: bucket-level, an indicator of the depth of the AST node, and semantic-level, which was split in 11 major categories with language-specific subcategories. The authors benchmarked three major code LLMs (StarCoder-7B, DeepSeekCoder-6.7B, Code Llama-7B) under different settings and provide extensive analysis across multiple dimensions.

strengths:
- This is the first dataset that covers 18 programming languages in repo-level code completion task
- The dataset Includes both evaluation and instruction tuning datasets
- Authors have conducted various evaluation across multiple dimensions

weaknesses:
The most critical weakness of this paper is the use of The Stack v2 without addressing data contamination issues. Unlike previous work like Ding et al., 2023 which carefully controlled the date range to avoid overlap with model training data, this work directly uses a widely-used training corpus that contain data that was written many many years back. This fundamentally compromises the benchmark's ability to reflect real-world performance. The authors should either implement temporal splits or explicitly verify and control for overlap with model training data.

Further, the assumption that "we randomly choose a node on the AST as the completion cursor position" (Line 152) is overly restrictive and doesn't reflect real-world scenarios. Code completion can and does occur at any position in practice. A more realistic approach would be to allow arbitrary completion cursor positions while ensuring the completed code forms a valid AST node, similar to previous work cited in the paper.

In addition, the bucket-level annotation scheme appears arbitrary and lacks clear motivation. Dividing ASTs into exactly 10 buckets seems like an artificial constraint when simpler alternatives exist, such as directly using the node's layer number in the AST. It'd be great if authors could justify why this particular granularity was chosen or demonstrate its advantages over simpler approaches.

Finally, despite covering 18 programming languages, there was surprisingly very little discussion of language-specific insights. This is a great opportunity that the authors could seize to analyze how different language features, paradigms, and structures affect completion performance. Such analysis could provide valuable insights for both researchers and practitioners.

questions:
In addition to the points mentioned in `Weaknesses`, I have a few questions and notes regarding the work:

1. Could the authors elaborate on how was the 5993 repos (line 204) split across eval and instruct?

2. It appears that adding retrieval seems to improve on EM but degrade on ES across models in Table 2. Why would this be expected?

3. On line 445: “StarCoder-7B base model already demonstrates strong coding proficiency, but lacks robust instruction-following capabilities” this is well expected as StarCoder-7B is a base model that has not been instruction tuned?

4. Some plots like Figure 4 look fancy but is conveying only little meaningful information in a readable way. Authors may consider using an alternative way to represent the data, e.g., a simple list or table. Similarly, Figure 5 is barely readable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730734780171

thread_type:
official_review

<thread_object>
creator: reviewer_uhra

<content>
summary:
The paper introduces M2rc-Eval, a new benchmarks comprising repository-level code completion tasks in many more languages than current benchmarks. Unique features of this benchmark include fine-grained annotations for different types of completion scenarios. The authors also introduce M2rc-Instruct, which a training dataset to improve performance on M2rc-Eval.

strengths:
- **Useful contributions.** The idea of having a very-large scale benchmark for repository-aware code completions across 18 programming languages is very useful. This will significantly help researchers improving Code LLMs on low resource languages such as Lua and Scala.
- **Novelty.** The fine-grained breakdown of code completion tasks is novel. However, I am not sure if this is needed or insightful. I would encourage the authors to present more analyses around this.
- **Interesting analyses.** I found the multi-lingual transfer experiment to be quite interesting and would be interested in a language-wise breakdown of this.
- **Paper exposition.** The paper is well-written and the analyses are presented in an easy-to-understand manner.

weaknesses:
- **Data leakage concerns.** The presented benchmark is collected from The Stack v2 which sources code from public sources such as GitHub. As such it may have significant overlap with the training corpora of the Code LLMs already evaluated in the paper and for future LLMs. As such, this benchmark is not fool-proof from future leakage and the utility of the benchmark for future evaluations is questionable. Further, the authors have evaluated only small models, and evaluations on bigger models ≥ 15B or strong models like GPT-4o or Claude 3.5 Sonnet might reveal the effect of leakage more concretely.
- **Repository-level coding.** While the paper says that the benchmark is for repository-level code completion evaluations, the data collection described in Section 3.1 does not guarantee that the tasks in the benchmark strictly require repository-level contexts.
- **Evaluation metrics.** While EM/ES metrics are being used for evaluations and have been used in several previous works as well, I think it is time that we move on from these inaccurate metrics. We should resort to more robust metrics relying on static or execution analysis such as pass rate [1] or hallucination rate [2] for measuring correctness of code completions.
- **Nitpicks.**
    - The analyses in lines 416-428 are quite obvious, i.e., there is a negative correlation between ES and the number of tokens generated
    - Typos:
        - 047 - “can not” → cannot
        - 261 - textural → textual
        - 401 - shadow → shallow

[1] Chen, Mark, et al. "Evaluating large language models trained on code." *arXiv preprint arXiv:2107.03374* (2021).

[2] Jain, Nihal, et al. "On Mitigating Code LLM Hallucinations with API Documentation." *arXiv preprint arXiv:2407.09726* (2024).

questions:
See comments above. But framed some concerns in question format below:

1. How do larger or more recent models such as StarCoder2-15B or GPT-4o perform on this benchmark?
2. Can you provide more insights from your multi-lingual transfer experiments? Specifically which languages benefit the most from Python-only fine-tuning? Can we predict such transfer ahead of time?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732412537235

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
# General Response
#### **G.Q1: Data leakage concerns and evaluation on more strong models.** 

**A1**: Following LiveCodeBench and EvoCodeBench, after submission, we also build a dynamically updating M2rc-Eval dataset, where the M2rc-Eval-2403 and M2rc-Eval-2406 are produced.

Specifically, we collect repositories from 2024.03.01-2024.05.31 and then build the **M2rc-Eval-2403** split based on the data collection process of Sections 3.1 and 3.2. Similarly, we build the **M2rc-Eval-2406** using repositories from 2024.06.01-2024.08.30. In the following table, we report the results (EM/ES) of different splits for different LLMs (Code Llama-7B, StarCoder-7B, DeepSeekCoder-6.7B, GPT-4o, LLama3.1, Qwen2.5, Claude 3.5 and DeepSeek-V2.5). Note that as some models (i.e., general LLMs) do not support the FIM pattern, we directly use prompt engineering strategy to obtain the repository-level code completion results, where the prompt template is provided in our new version.

We have the following observations. (1). When introducing cross-file context using retrieval, better performance results are usually obtained for both code LLMs and General LLMs. (2). For existing LLMs, the performance on different testing splits (M2rc-Eval, M2rc-Eval-2403, M2rc-Eval-2406) are relatively stable, which means that data leakage or contamination concern does not have a significant impact on evaluating the code completion abilities in M2rc-Eval. Besides, for many knowledge-based benchmarks (e.g., MMLU, SimpleQA),  this knowledge information widely exists in web and book corpus, which have been trained in existing LLMs. However, these benchmarks are still effective tools for evaluating the knowledge coverage degree in existing LLMs. (3) Meanwhile, although our M2rc-Eval has been trained in several LLMs, we still find existing LLMs cannot achieve competitive performance results, and our M2rc-Eval can still be used as an effective benchmark to evaluate the code completion abilities of existing LLMs. (4) These powerful API LLMs or open-source LLMs (e.g., LLama3.1-70B, Qwen2.5-72B) have strong code generation abilities in many benchmarks (e.g., HumanEval, MBPP, LiveCodeBench), but the repository-level code completion results are still limited when compared to these code-specific LLMs. We assume that these code LLMs usually introduce an FIM loss objective in training, which is the same as the testing scenes and greatly improves the repository-level code completion.

| **Model**          | **M2rc-Eval** | **M2rc-Eval-2403** | **M2rc-Eval-2406** |
| ------------------ | ------------- | ------------------ | ------------------ |
| **Code LLMs**      |               |                    |                    |
| Code Llama-7B      | 19.4/50.3     | 19.1/52.9          | 21.5/52.7          |
|  + Retrieval          | 20.2/46.1     | 23.1/50.8          | 25.0/51.5          |
| StarCoder-7B       | 21.0/52.0     | 20.4/53.1          | 20.1/51.6          |
|  + Retrieval          | 24.1/50.0     | 26.0/54.9          | 28.6/55.9          |
| StarCoder2-15B       | 23.2/53.6     | 22.4/52.7          | 22.9/52.7          |
|  + Retrieval          | 25.3/54.2     | 26.2/55.1          | 25.9/54.8          |
| DeepSeekCoder-6.7B | 22.6/54.7     | 20.4/51.9          | 23.0/55.6          |
|  + Retrieval          | 25.1/51.7     | 24.0/52.7          | 30.3/56.4          |
| DeepSeekCoder-33B  | 26.8/51.6     | 24.0/43.7          | 23.9/49.7          |
| + Retrieval          | 27.3/52.9     | 27.1/51.8          | 27.5/49.8          |
| Qwen2.5-Coder-7B         | 18.8/46.5     | 20.5/49.7          | 21.0/48.1          |
| + Retrieval          | 27.2/52.2     | 31.0/57.2          | 32.4/56.7          |
| Qwen2.5-Coder-32B        | 34.7/65.7     | 35.0/66.2          | 37.3/67.6          |
| + Retrieval          | 41.7/68.0     | 43.9/69.5          | 45.9/71.2          |
| **General LLMs**   |               |                    |                    |
| LLama3.1-70B       | 6.4/31.9      | 5.0/31.5           | 5.4/31.3           |
| + Retrieval          | 6.8/33.0      | 6.1/33.3           | 6.1/32.8           |
| Qwen2.5-72B        | 6.7/39.1      | 11.6/49.6          | 10.2/45.3          |
| + Retrieval          | 12.2/44.8     | 12.4/51.1          | 13.4/50.9          |
| GPT-4o             | 12.2/45.5     | 11.5/54.0          | 11.1/47.2          |
| + Retrieval          | 17.8/56.7     | 15.0/57.4          | 17.3/54.0          |
| Claude 3.5 Sonnet  | 22.4/55.3     | 23.2/63.8          | 23.1/59.5          |
| + Retrieval          | 29.9/62.8     | 28.4/65.9          | 30.5/67.1          |
| DeepSeekV2.5       | 16.1/50.5     | 23.9/61.0          | 25.2/56.9          |
| + Retrieval          | 27.2/60.6     | 28.3/64.1          | 26.0/61.1          |

We have added this discussion in our new version.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732412663285

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
# General Response



#### **G.Q2: More evaluation metrics.**

**A2**: Thanks for your suggestions. We have additionally provided the syntax static analysis and execution analysis.

For syntax static analysis, following Qwen2.5-Coder, to further verify the syntax correctness of the predicted code snippets, we use the code static checking tools (Tree-Sitter) for all predicted code snippets of test split of M2rc-Eval. Specifically, we parse the code snippet into the abstract syntax tree and filter out the code snippet, where the parsed nodes in the code snippet have parsing errors. 

The results for syntax static analysis are as follows, and we observe that syntax accuracy improves a lot after tuning. Notably, the syntax accuracy is close to 100% after tuning, which means that existing code LLMs can easily learn the basic syntax rules for existing programming languages.

| Metrics                       | Code Llama-7B (+ Retrieval) | Code Llama-7B( + Retrieval & Tuning) | StarCoder-7B(+ Retrieval) | StarCoder-7B( + Retrieval & Tuning) | DeepSeekCoder-6.7B(+ Retrieval) | DeepSeekCoder-6.7B( + Retrieval & Tuning) |
| ----------------------------- | --------------------------- | ------------------------------------ | ------------------------- | ----------------------------------- | ------------------------------- | ----------------------------------------- |
| Syntax static analysis (Acc.) | 71.3                        | 96.1                                 | 83.9                      | 96.9                                | 80.4                            | 97.2                                      |



For execution analysis, as discussed in Appendix A.4, generating unit test cases and providing execution sandboxes for repository-level code completion are very challenging. In this rebuttal phase, we follow RepoCoder to provide the execution test samples in 17 languages. Note that HTML language cannot be executed, and we do not provide execution test samples for HTML. Specifically, as running tests can be time-consuming and computationally expensive, we first randomly select a separate set of smaller-scale repositories that are easy to deploy. Besides, as collecting unit tests is challenging, we directly utilize the unit tests available in these repositories and annotate corresponding functions covered by these unit tests. Finally, we utilize unit tests present in the repository to evaluate the functional correctness of the completed function body, where we report the Pass@1(Pass rate is 1 if the code passes all the corresponding test cases, and 0 otherwise). Note that the number of samples for execution analysis is 815.  The average execution accuracy results in 17 languages are as follows, and we observe that retrieval and tuning on our M2rc-Instruct both lead to better performance.

| Model                | Avg Acc. |
| -------------------- | ---- |
| StarCoder-7B         | 29.9 |
| + Retrieval          | 32.3 |
| + Retrieval & Tuning | 56.6 |
| DeepSeekCoder-6.7B   | 33.7 |
| + Retrieval          | 39.1 |
| + Retrieval & Tuning | 60.5 |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732412831703

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
**Prompt template for G.Q1.**

The prompt template for **G.Q1** for evaluation on some general LLMs, which do not support the FIM pattern.


```
## First, input a segment of <LANG> code that needs completion. Please help complete the code at the corresponding position.

## The format of the input code is as follows:
<fim_start> prefix <fim_hole> suffix <fim_end> 

Explanation:
1. <fim_start>, <fim_hole>, and <fim_end> are special characters.
2. <fim_hole> is the position that needs completion.
3. The prefix after <fim_start> represents the context before the content that needs completion.
4. The suffix after <fim_hole> represents the context following the content that needs completion.

## The output format is as follows:
    1. Only the code completion result for the position <fim_hole> is needed. 
    2. Do not use markdown format.
    3. Do not include the surrounding context.
    4. Do not provide any explanation or description.

## The content of the input code is as follows:
<CODE>
</content>
</thread_object>

</thread>
<thread>
creation date:
1732502305776

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summarization on the Responses

comment:
Thanks for handling/reviewing our submitted manuscript: "M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation". We would like to thank the reviewers for their insightful and constructive comments and suggestions. By addressing each of the issues raised by the reviewers, we believe that the quality and clarity of our M2rc-Eval can be improved a lot. The major responses are summarized as follows:

(1). We have carefully discussed the data leakage concerns by introducing two newly updated test sets based on our data collection process (See  Reviewer uHra.Q1, Reviewer TS5v.Q1).

(2). We have additionally reported the results of recent code-specific LLMs and the powerful general LLMs (See Reviewer uHra.Q6, Reviewer PyQh.Q3, Reviewer 5s7R.Q3).

(3). We have provided more evaluation results based on syntax static analysis and execution analysis (See Reviewer uHra.Q3, Reviewer PyQh.Q2, Reviewer 5s7R.Q1).

(4). We have provided more clarification and discussion on data collection process and multi-lingual transfer (See Reviewer uHra.Q2&Q4&Q7).

(5). We have discussed more details on another data construction strategy, justification on the bucket granularity, language-specific insights, and degradation for several LLMs (See Reviewer TS5v.Q2&Q3&Q4&Q6).

(6). We have discussed more details on evaluation setup, results based on long-context LLMs, and the effect of different retrievers (See Reviewer PyQh.Q1&Q4&Q6).

(7). We have discussed the correlation between CodeBLEU and EM/ES, and qualitative error analysis (See Reviewer 5s7R.Q2&Q4).

(8). We have significantly improved the writing quality based on comments of typos and figure presentation (See Reviewer uHra.Q5, Reviewer TS5v.Q5&Q7&Q8, Reviewer PyQh.Q5, Reviewer 5s7R.Q5).

Again, we would like to sincerely thank you very much for these constructive comments and evaluation for our manuscript.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732502419434

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to feedback on the Responses.

comment:
Dear Reviewers:

Hello! We have updated the responses to your constructive and insightful comments, and we would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address them. We appreciate your time and consideration!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732976879086

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Hello! Since the discussion period is short, we kindly ask you to review our responses and reevaluate our work given our clarifications.  If any concerns remain, we would be grateful for further clarification and are happy to continue the discussion in this rebuttal process.

We appreciate your valuable time and consideration!

Thanks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper creates a repo-level code completion benchmark across 18 programming languages. The benchmark introduces two types of annotations: bucket-level, an indicator of the depth of the AST node, and semantic-level, which was split in 11 major categories with language-specific subcategories. The evaluation benchmarks three major open-weight code LLMs (StarCoder-7B, DeepSeekCoder-6.7B, Code Llama-7B) under different settings and provide extensive analysis across multiple dimensions.

While the data resource is appreciated, the main concern lies in the design of evaluation metric. The work mostly focuses surface form matching, which has shown to not have strong correlation with functional correctness in previous works. Although the authors provide results on execution-based evaluation for a small subset of the data during rebuttal, it does not cover the full evaluation set and hence cannot be considered as part of the work