Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a way to make an existing dataset (MBPP), an execution-based code completion benchmark, multilingual: while the original MBPP only contained python code, the new dataset (MBXP) contains a total of 10 programming languages. Problems in function completion datasets consists of *prompt*, *test statement*, and *canonical solution*. The proposed dataset creation is based on the insight that, for an execution-based benchmark, only translation of the prompt and test statement are necessary.

The paper further evaluates multiple monolingual and multilingual models on MBXP. This evaluation results in multiple interesting findings, for instance, models have a non-zero performance on languages they have not been trained on, what the authors attribute to code in multiple languages appearing in the same files. The authors further explore few-shot approaches and translation. Multilingual models mostly outperform monolingual ones.
strength_and_weaknesses: Strengths:
- The paper is very clear and contains a coherent set of experiments.
- The paper presents a new dataset, which will be useful for future work.

Weaknesses:
- The proposed dataset creation and the experimental setup aren't very creative -- but the results are interesting, so this isn't a big weakness.
clarity,_quality,_novelty_and_reproducibility: The work is very clear and rigorous. It's also original, even though it might not be very creative.
summary_of_the_review: As the paper provides interesting insights and contributes a useful dataset, I believe it should be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes a new framework to convert prompt and test cases, for code generation models, to a new target programming language. The framework is used to generate an execution-based code completion benchmark for more than 10 programming languages.
This enables an easier evaluation of a "multilingual" model versus the standard monolingual model.

This work shows that multilingual models tend to be better than the monolingual model and show that code generation models have zero-shot capabilities.
  
strength_and_weaknesses: Strengths:
- the paper is very well written and illustrated. For someone like me who is not an expert of this area, I find this paper relatively easy to follow.
- the framework proposed is well-motivated and very simple
- this work can easily be applied to extend many benchmarks to many languages.


Weaknesses:
- The appendix is huge. It contain tons of information that would be more useful in the main paper. The limits for instance should definitely be included before the conclusion.
- most of  the details needed for reproducibility and fully understand the experiments are in the appendix. I couldn't check it thouroughly.

clarity,_quality,_novelty_and_reproducibility: The method is clear and the paper well-written.

It looks novel (although I might have missed previous work since I'm not an expert of this area).

Based on the main paper, I don't think this is easily reproducible (of course I didn't read the 77 pages to check it...).
summary_of_the_review: The paper is very interesting and the method will be useful. My only concern is that the size of the PDF file submitted is huge and that I couldn't check all of it thouroughly. Some sections of the appendix must be move to the main paper before acceptance.
I would definitely recommend the authors to submit the entire paper to a journal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new benchmark MBXP for 10+ programming languages to facilitate the evaluation of execution-based code completion task. By the design of the conversion framework to translate prompts and test cases from the MBPP dataset, MBXP can evaluate code generation models in a multi-lingual manner. Furthermore, by the code generation model and some defined strategies, they can also provide the canonical samples in different programming languages and these data can also used for different code-related tasks such as source code summarization, code translation etc.
strength_and_weaknesses: Strength:

- The provided benchmark is valuable for AI4Code community. Currently, MBPP is a dataset designed for python, which is limited for evaluation, This paper proposes a new benchmark based on MBPP to enrich the evaluated programming languages in Python, Java, Go, etc. (13 in total) and this dataset is very valuable for the community.

- The extensive evaluation on this benchmark is very appreciated. Besides the code completion task including  multi-lingual version and  mono-lingual version, this paper also evaluates on zero-shot translation, prompt robustness, code insertion and the summarization task to show the effectiveness of the constructed benchmark.

- There are also some findings by the extensive evaluation and these findings can provide the readers with some insights about these large-scale pre-trained models.

Although I appreciated the authors for the extensive evaluation, I also find some problems and hope to fix these issues in the next version.

- The synthetic canonical solutions are generated by the models, so I do not know how diversity they have compared with the real semantic-equivalent data?

- In the abstract section, the authors mention that they will release part of the benchmark, but the main contribution for this paper is this benchmark, why not release the entire benchmark for the following-up researchers? 

- I suggest authors provide more details about the quality check via reviewers part. How many reviewers do you have? How to evaluate the converted version? How to ensure the converted problem is correct? How long do you take to evaluate these converted problems?

- In Figure 3, I cannot know which line belongs to which model.  It should add some clarifications.


clarity,_quality,_novelty_and_reproducibility: - Clarity The paper is easy to follow
- Novelty: The technical novelty is limited, however this paper conducts extensive experiments to provide some empirical findings. Furthermore, the constructed benchmark is valuable.
- Quality: The experiments are extensive and some findings are interesting.
- Reproducibility: The experiments should be reproducible if all data and models are released.


summary_of_the_review: This paper propose a framework to construct a new benchmark across different programming languages for different code-related tasks. Based on this bechmark, an entensive experiment is conducted. Although the technique novelty is limited, I appreciate the efforts in this paper for some interesting findings.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a framework for deriving multi-lingual data from an execution-based dataset (e.g., MBPP, HumanEval, MathQA), which entails applying rule-based transformations based on static analysis to the prompt and test cases. The key idea is that the canonical solution does not need to be translated since evaluation is done by running the test cases. Through this, they translate MBPP into 10+ languages. They train large decoder-only monolingual and multi-lingual models, of various sizes, using three languages, and they evaluate them in out-of-domain and in-domain settings. They highlight the benefits of the large multi-lingual models especially in the out-of-domain settings. They also consider few-shot learning, which results in fewer errors, and is most advantageous in out-of-domain evaluation. They further consider other evaluation tasks like zero-shot translation, code insertion, and summarization. Additionally, their framework also allows deriving synthetic canonical solutions for tasks which require it.

Contributions:
- A framework for acquiring multi-lingual data from a monolingual dataset, without directly requiring translation of code solutions.
- A large multilingual dataset which can be used for studying a variety of code-related end tasks, spanning multiple languages.
- Empirical results and observations, spanning many languages, model sizes, settings, and tasks.

strength_and_weaknesses: Strengths:
- Deriving multilingual code data without having to translate code solutions (especially considering how error-prone code translation models are) is a neat idea and useful.
- Since it is challenging to get parallel data to study tasks like code translation, this framework which can give synthetic canonical solutions could be particularly useful for studying that tasks.
- There are extensive empirical results, including supplementary analyses (e.g., best combos of source/target pairs for zero-shot translation) and even minor observations (e.g., sampling efficiency, validation losses) that can be beneficial for guiding future work.

Weaknesses:
- One concern I have is regarding whether potential biases/assumptions in the original MBPP dataset can transfer over to the MBXP datasets. These could be possible ambiguities (as the authors have mentioned  in Section 2.1) but also more fundamental biases/assumptions, such as language-specific ones. Could the formulation of the prompt and the types of tests that are needed drastically change based on the specific language, depending on for example, whether it is a language that requires explicit memory management?
- While the synthetic canonical solutions may be useful for training during different tasks like translation, I find that using them for test to be slightly problematic because these are model-generated, and possible errors/biases in this output could propagate, and we don’t want to be evaluating using such references. However, these execution-based sets like MBPP are quite small, so it is not clear whether they offer enough training data to have a meaningful impact.
- It is not clear whether training the monolingual and multilingual models in Section 3.1 was necessary. As described in Appendix M, CodeGen entails monolingual and multilingual models trained on a subset of the languages studied in this work, including Python Java and Javascript, which were used in this work. Based on the results shown in  Appendix M, it seems that many of those models are better performing than the ones trained in this work. Most (if not all) of the analyses could have been done with these.


clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is well-written and easy to follow. One minor point is that there is some ambiguity in which model is being shown in Figure 3.

Novelty:
The idea to effectively “translate” the prompt and test cases, which may have simpler structures, is novel and interesting. It can also have broader applications for different natural languages, when extrinsic-evaluation is used. There is empirical novelty in terms of a new dataset that spans multiple programming languages, as well as with a lot of new empirical results for many different programming languages (e.g., in-domain vs out-of-domain training, monolingual vs multi-lingual training, model sizes, source/target pairs for translation). Additionally, the technical novelty seems to be somewhat limited, as the rule-based static analysis techniques used to build the framework seem to be derived from existing work/tools. 

Quality:
The experiments are well-designed and findings are well-grounded. 

Reproducibility:
If the datasets are publicly released (including the pretraining ones), the experiments should be reproducible. There is a detailed Appendix with training details provided in Appendix N.



summary_of_the_review: Overall, the intuition and motivation behind this work are neat. The paper presents interesting empirical results and observations related to multilingual aspects of code generation, especially considering much of the focus in recent years has been on only a few languages like Python and Java. They also present a dataset spanning multiple languages, which could potentially be useful for the broader research community, although I have some concerns about whether their framework which effectively “translates” prompts and test cases in existing datasets might be propagating biases. Additionally, the technical novelty behind the framework seems to be somewhat limited. Nonetheless, I would say that the empirical novelty is still relatively significant.

[After Author Response]
I have read the response, and the authors have addressed most of my concerns. I have increased my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work presents a new execution-based benchmark for code across multiple programming languages. Code generation benchmark datasets consists of a prompt, a test statement, and a canonical solution. The key idea is that the canonical solution does not need to be translated per language since evaluation can be done by running the test cases. Finally, the authors show how this benchmark performs in a series of tasks and LLMs.

The presented benchmark is novel and needed in the community and its clear that the authors have put a lot work into it. The conclusions drawn on the performance of LLMs on this benchmark are also useful. One weakness noted by the reviewers is that it is unclear what kind of biases and limitations the original (and translated benchmark) introduce to the measurement of a model's ability to generate code. For example, does performance in these datasets correlate with performance in code completion in library code?
justification_for_why_not_higher_score: The paper itself is not exceptionally novel or present a significant advance in machine learning to justify an oral presentation.
justification_for_why_not_lower_score: Accepting the paper as a poster would be okay, but would reduce its visibility. Since the community seems to be interested in code generation, having good benchmarks widely known seems useful.
Decision: Accept: notable-top-25%