Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a benchmark to measure the quality of the systems in the growing domain of code intelligence that includes code search, code completion, and code-to-code translations among others. The authors presented three Transformer-based systems (with BERT-style encoder, GPT-style decoder, and encoder-decoder) to set up baselines. Some of the tasks are novel or improve over the original creation process. 

=========EDIT=========\
After the discussion phase, the rating was increased from the initial 4 to 7.\
======================
strengths: 1. It is a relevant and up-date challenge for language models. 
2. The choice of baselines is satisfying.
3. The experimental section is well designed and strikes the right balance between providing as much information as needed, and being easy to digest.
4. The website and hosting platform are carefully designed. 
5. I like the specific information about the training and inference costs in the Appendix.
6. The novel datasets are described and released publicly.
weaknesses: ## Major
### 1. The motivation behind the choice of tasks.
The most important missing puzzle is the lack of proper motivation and desiderata that would support the choice of tasks. Measuring the quality of systems aimed to deliver code intelligence demands a proper statement of the problem, and approaching the benchmark in a more top-down manner, where the tasks are chosen on purpose to serve a specific need that is explicitly stated and supported. I would expect a discussion on what steps the developer needs to take in order to come up with the working code, and how the specific tasks are helping him in this regard.  It would be perfectly OK if the authors state that the code-writing process consists of X steps, and here we only took some of them, and here is why. Unfortunately, the paper approaches the benchmarking from the bottom, by selecting some code-related tasks (based on the unspecific terms such as 'popularity', 'reputation', or 'widespread adoption') and then clustering them into similar buckets, based on their features, but leaving the reviewer with the feeling that the scientific community may confuse itself in the longterm when choosing this suite as the main target for future development. I would like to avoid inconsistencies between these chosen tasks and the downstream applications. The paper is not convincing in this regard, and I am not going to recommend it to the community in its current version, but I believe the future is bright for this benchmark.\
Questions: Could you provide some rationale behind the choice of the tasks, and compare the chosen tasks with the ones that were not included?  How are these tasks relevant to the process of software development?
I am willing to substantially increase the score if this aspect will be improved.

### 2. Cloze test tasks
It is unclear how cloze test tasks (when the model has to choose from the limited vocabulary or 2 tokens), would solve real-life problems. These tasks seem somewhat artificial and not grounded in the experience of a programmer.
Are there specific situations, in which the developer is supposed to choose from the limited vocabulary, or is it rather a toy task? What is it aimed to measure? At the moment it demands adding the extra classification heads to the Encoder models, but it seems it may be easily simplified and generalized to the 'next token prediction' task. Is there any argument against that?

### 3. Human baselines are missing
In order to measure progress in the development of code-helping language models, the paper should discuss how well humans perform on these tasks.  Is it possible to perform human evaluation and take into account the experience of a software developer? I believe knowing where we stand with relation to the human baseline should not only be provided, but also be the main criterion when choosing tasks.

## Minor
### 4. Confidence Interval are not provided
As it is stated in the reproducibility checklist, the models were run 2-3 times. The paper would greatly benefit from this information.

### 5. The novelty of some datasets is questionable
The caption in Table 1. states that the 'Code completion' task contains a novel dataset, however, it is not clear what is novelty there, as, in Section 2.4 (Code completion), it is written that the previously developed datasets are used. From the provided description it is hard to understand where the novelty is. Does it lie in the reformulation of the original dataset, or in the performed processing steps that slightly differ from the originally devised?
correctness: The technical side of the evaluation and experimental section is correct. Some issues with the dataset's novelty were raised in 'Weaknesses'
clarity: The writing is very good.\
Only minor flaws or typos can be spotted:\
1. line #49 'test test sets' -> 'test sets'\
2. In Table 1: Two tasks lack numbers in the size column, and it should be explained how to understand it.\
3. #118 - dot on the next line
4. #153  - 'AdvTest ' - It is unclear what is the full name. Is it Adversarial, Advanced, or something else?
5. Related Works - I think it is perfectly OK to drop the first paragraph as the work does not benefit from it.
relation to prior work: Yes, the work is novel in its narrow field, and a wider comparison to other benchmarks is also provided.
additional feedback: ## Questions of minor importance
### 1. Programming languages chosen
What was the driven force behind the choice of programming languages for tasks? Is it data availability or is there a different rationale? How does it stand with the most popular used languages at the moment?

### 2. Aggregating the results and future expansion
It is not described how the results are aggregated in the 'Overall' leaderboard. My concern is that the future extension of the benchmark will change the balance between the tasks and groups of tasks(e.g., code-code). How do you plan to expand the benchmark? Is it going to be backward-compatible? 

### 3. Tutorials on how to run the baselines
As the code is provided(which is great!) it may be very helpful to deliver a simple readme on how to reproduce results from the paper and perform inference with the already hosted models.
review_title: Important topic, good execution, but the paper is missing the bigger picture, and how the tasks combine together to solve a real-world problem.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a comprehensive benchmark dataset for program understanding/generation, consisting of 10 tasks across 14 datasets, where 5 datasets are newly introduced. The collection of these tasks cover code-code, text-code, code-text and text-text settings, providing a complete evaluation platform.
strengths: Program understanding and generation is an important and interesting topic that has attracted considerable amount of research efforts recently. The biggest contribution of this paper is to put together a large collection of program-related tasks (several of these tasks are introduced by the authors to fill in the missing parts) that cover (almost) all aspects of this area. I believe this benchmark dataset has the potential to become a standard benchmark in further research and will facilitate the development of program understanding and generation research.

weaknesses: My only major complaint is that there is no single baseline model that can be applied to all tasks, or a joint model trained on all datasets to provide an overall performance (the model can very simple and naïve). The value of a benchmark dataset consisting of many tasks might be greatly compromised if the tasks are only processed separately.
correctness: The design and implementation of the benchmark is appropriate. An overall metric for evaluating a system on all subtasks simultaneously is not discussed.
clarity: The paper is well-written and easy to follow.
relation to prior work: Most of the related datasets are covered. If space permits, the authors may also include more existing successful models for the tasks in the benchmark.
additional feedback: Please see the weaknesses section.
review_title: A comprehensive benchmark dataset for program understanding/generation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a benchmark dataset focused on source code processing (understanding and generation) tasks. The dataset is rather comprehensive, including code-to-code, code-to-text, text-to-code and text-to-text tasks. It includes 8 preexisting datasets that were already used to evaluate some of the tasks, together with 6 newly-introduced datasets. The paper describes each of the tasks and datasets, and also provides strong baseline models based on pretrained language models and the corresponding results.
strengths: This is a relevant contribution that will likely have a strong impact and be used by the community. Benchmark datasets are a useful asset to evaluate new general-purpose systems and models, and while areas like natural language or image processing have their own dataset suites, this was not true (to my knowledge) in the realm of source code.

The presented dataset suite covers a wide range of tasks (with the intention of extending it in the future), and also competitive baselines.

The dataset is publicly accessible under a license that permits computational use.
weaknesses: The description of some of the datasets could be clearer. As the description paper for the benchmark dataset, this will be probably a go-to resource and first point of contact for researchers that want to work with these benchmarks. As such, I think it should be written (as far as possible) in such a way that the description of each dataset can be understood without being familiar with the original papers or even tasks, which I think is not true in some cases (concrete examples and suggestions are provided in the "Clarity" section below).

Some items in the checklist are in [TODO] status.
correctness: The dataset design and experimental design in the paper look sound. I just have a few minor questions about the choice of some accuracy metrics (in the "Clarity" suggestions below).
clarity: The paper is generally well-written, but some things should be clarified more, as outlined in sections above.

In particular:

In Table 1, rather than marking the tasks that have new datasets in the task column, it would be clearer to mark the datasets directly in the dataset column. With the current version of the table, it is not clear exactly which datasets are new, one has to go to the text.

In the description of BigCloneBench, please clarify the distinction between "code fragments", "clone pairs" (true and false) and "examples". The dataset (after filtering) is claimed to have ~9K fragments but millions of clone pairs and examples, so I suppose a fragment is a larger piece of code and one fragment can contain N clone pairs (which are the examples) but this is not obvious from the description. A benchmark dataset description paper like this should not assume that the reader knows the dataset (I understand that these details will be described in the original dataset papers, but this paper will be the go-to point for researchers that want to evaluate against the benchmark so it should aim to be reasonably self-contained).

It's also worth explaining why F1 is picked as the metric for that task (which I understand is a binary classification task) while accuracy is picked for e.g. defect detection - I suppose is due to class balance/imbalance?

In the description of CodeRepair, it would be helpful to mention what the scope of the intended repairs is - i.e., whether repairs affect single tokens or lines or they could be wider refactorings.

For the documentation translation task, please specify if the sentence pairs in the dataset have been translated by humans or they have some degree of machine translation.
relation to prior work: The discussion of related work is good, but it would make sense to mention SuperGLUE (https://arxiv.org/abs/1905.00537) as another relevant benchmark dataset in the field of natural language processing.
additional feedback: I confirm that I have read the author rebuttal. It has reaffirmed my positive recommendation for the paper.

The checklist seems to have a contradiction, it says that no license is specified because the code is proprietary, but the github repository does specify a license for the code.

Two typos:

In line 118, the full stop is separated from the previous line.

In line 157, missing space before opening parenthesis.
review_title: A useful benchmark dataset for code

Meta-review: The paper proposes a comprehensive benchmark for program understanding and generation consisting of 10 tasks across 14 datasets. All the reviewers agree that this dataset has potentially be used as a standard benchmark to measure progress in this domain. Therefore, I recommend acceptance of this paper.
Decision: Accept