Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Reasons for score:
A benchmark for evaluating source code ML models will help to accelerate the progress in the right direction. However, the analysis to support the dataset and the proposed tasks as a benchmark does not address some critical concerns (please see weakness section below). 

Summary:
The paper presents a dataset of source code that allows experimenting with different representations and proposes five tasks to evaluate local and global reasoning capabilities of a source code machine learning model. 

Strength:
1. GLUECode provides a labeled dataset with different representations by compiling ~5300 Java projects extracted from Github. This could be useful for future research in ML modeling for source code.
2. Evaluation results of five baseline models on five proposed benchmark tasks demonstrate varying performances over different tasks. 

Weakness
1. Although overall construction of the dataset could be useful for the community, sufficient evidence is not provided to establish the utility of the dataset compared to other existing datasets. An arbitrary minimum number of 50 files in a project is selected as a filtering method without presenting any supporting analysis. “NullToken” task is presented as the task that benefits most from global reasoning, which is the primary contribution of this paper. However, the dataset size for NullToken task is small, which significantly reduces the usefulness of the task in evaluating the models.
2. The evaluation results of the baseline models are not well-explained. According to Table-2 the “Completion” task requires increased non-structural and global reasoning compared to the “Naming” task. However, all the baseline models are showing poor performance in the “Naming” task compared to the “Completion” task. 
3. The tasks that require global reasoning are mostly generation tasks in the benchmark. Therefore, the evaluation metrics could be less representative of global-reasoning performance of classification models.

Questions to author:
Please address and clarify the cons above.

review_title: GLUECode: A Benchmark for Source Code Machine Learning Models

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary ###

The paper presents GlueCode, a new benchmark suite for evaluating source code learning models. The suite includes 5 tasks, two of which are classification tasks and three are sequence generation tasks.

### Strengths ###

* A standard benchmark for evaluating source code models would be a blessing. 

* The selected tasks are interesting and compelling. Particularly interesting are the tasks that require fine-grained reasoning about the control and data flow of programs. The balance between classification and generation tasks is also solid. Other design choices like focusing on the scope of a single method also seem well justified considering the common practice in this area. 

### Weaknesses ###

* It is hard (impossible) to evaluate the contribution of this paper without looking at the actual code and data. The devil is in the details. On the face of it, the suggested benchmark suite seems reasonable. 

* The only contribution of this paper is the benchmark suite. There is no additional novelty. This is not really a weakness, just a comment. I think that we should accept benchmark papers that help move the research area forward.


### Comments ###

* The operator prediction task seems “too easy” when only a single operator is masked. It is worth considering variations of this task when masking multiple operators. 

* For the code completion task, it should be clear whether comments are part of the permitted/desired prediction context. In recent work, we are seeing increasing importance of natural language hints, and an explicit decision is required about this in the benchmark suite. 

### Questions for Authors ###

* Can you please make the code and data available? All the described tasks make sense, the choice of baselines looks good. 

### Minor questions and comments ###

* "Across all representations, source code entities (methods and classes) are identified via a Universally Unique Identifier (UUID), and can be linked together. ?? provides details and examples."

### A general comment about benchmarking papers ###

As a benchmark suite, this seems like a good step in the right direction, and I am happy to increase my score based on that. However, my current score is calibrated to take novelty into account when comparing to other papers. 

review_title: useful suite 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The objective of this paper is to present a benchmark of code understanding tasks in the spirit of GLUE benchmarks in NLP. Towards this, it designs 5 Java language tasks: NPath complexity, operator prediction, method naming, completion of method calls, and null dereference prediction. An evaluation on some common neural architectures is performed.

The first weakness of the paper is that the benchmark tasks do not fulfil the stated objective of the paper. The main argument of the paper is that many approaches in the literature focus on local (intra-procedural) prediction tasks and use either sequence representations or structured representations (e.g., ASTs, control and data flow graphs). This paper seeks to present a benchmark of tasks which requires going beyond this, by requiring global (inter-procedural) analyses and structured representations. This is a good objective and the community would certainly benefit from such a benchmark. However, among the proposed tasks, except for the null deference analysis, none of the tasks particularly require global reasoning. The NPath complexity, operator prediction, method naming and code completion (whose special case focussing on method calls in considered in this paper) are local in scope and have been solved in the literature as such.

For the null dereference prediction task, the paper uses a static analysis tool, Infer, to obtain labels. Infer is stated to return an execution path exhibiting null pointer deference. However, the paper does not give the exact number or percentage of examples from the dataset in which the paths do span multiple methods. Among all the tasks and data points, only these can be said to be truly requring global reasoning; but these details are missing and compared to the entire benchmark, this represents a small fraction. The paper conjectures that method naming and method call completion can benefit from global reasoning, but offers no evidence to that effect. I also have some concerns about the call-graph precision and ambiguity in null dereference task; these are listed in the detailed comments below.

This leads to the second weakness of the paper: the baselines. First, the baseline methods consider only sequence based models even though the paper explicitly wants to promote structured representations. They are also not tuned enough. Second, the paper does not use any global reasoning in the baselines. The paper would be more convincing if it were to show that such a global model outperforms local models. This would help to concretely claim that at least some of the benchmark tasks require global reasoning, including method naming and method call completion as conjectured by the authors. I also have other concerns above the experiments, which I list below.

Now, some detailed comments:
* The abstract says that "However, these models are commonly designed to perform well on a single task, failing to capture code’s multifaceted nature." I don't agree that just because a paper targets a single task, it fails to capture the multi-faceted nature of code. There are ample examples in the literature which take many views (e.g., ASTs, control flow, data flow, etc.) into account while solving a particular task.
* I like that the benchmarks come with pre-processed inputs in different formats. However, are the call-graphs over-approximate or under-approximate? Which call-graph construction algorithm is used? Does it construct context-sensitive call-graphs? It is important to spell out these details since different call-graph construction algorithms offer different precisions and these would impact the precision of the models build using those representations.
* There is an unresolved ref in Sec 2.2.
* The code completion task is restricted to method calls. Does this include predicting method names or their parameters also?
* The labels of the null dereference prediction task are tokens from the vocabulary (a classification problem). Such a token may occur in multiple places in the method. As stated in the paper, Infer provides the actual dereference token susceptible to null dereference. So this task should use a pointer that localizes the bug to the specific token, along the lines of "Neural Program Repair by Jointly Learning to Localize and Repair" (ICLR'19).
* The baselines are not tuned enough. There is no hyper-parameter search. The datasets vary in sizes and characteristics and would benefit from appropriate hyper-parameters.
* The paper uses a closed vocabulary of 10K. It should report on the prevelance of OOV tokens in inputs and output labels.
* The seq2seq baseline could benefit by an attention layer.
* There is no description of the task-specific layers in the Transformer baseline. The results for the completion task are not made available for review.
* The relative performance of the baselines on the NullToken task is surprisingly. The authors should explain this.
* I did not understand the argument against comparison with previous work in Sec 4.1.
* It seems that code deplication between training and test sets is not entirely ruled out. This should be fixed.
review_title: Good objective but weak tasks and baselines

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents GLUECode, a benchmark for evaluating machine learning models of source code. GLUECode considers both global and local contexts of source code, and aims to help researchers experiment with multiple source code representations and evaluate their models. The authors also presented results of several baselines on the benchmark. 

Machine learning for source code has attracted a lot of interests in recent years. It is good to see a benchmark consists of 5000+ projects, which could help advance this area of research. The authors also performed some GLUECode tasks and presented results for several baselines, which show that there is ample room for progress on GLUECode. Overall, the paper is well written.

Concerns: 

The proposed work considers both global and local contexts of code (the benchmark’s name is Global and Local Understanding Evaluation of Code). Section 2.1 also dedicates to this. However, it is not clear what global context is considered and how it is incorporated by the benchmark. In a ML for SE work, researchers may use various global contexts such as UML diagrams, library/API dependency, inter-procedural data/control flow, commit data, etc. It is not clear how these global context information can be satisfied by the benchmark. 

The authors can also describe more about the unique advantages of using the proposed benchmark. Currently, they are already many public datasets released by various papers in this field (thanks to the open science policy). Also, it is easy for researchers to download a large amount of source code from open source websites (such as Github) themselves. They can also process the source code using existing static analysis tools to obtain the data they need and share the data. 

Currently, GLUECode only provides a few types of source code representations. In recent years, researchers have proposed many different ways of representing source code tokens and ASTs. As an example, the following works use different AST-based source code representations (and it is not clear if the benchmark could provide necessary information to support these representations):
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. Improving automatic source code summarization via deep reinforcement learning. In ASE, pages 397–407. ACM, 2018.

J. Zhang, et al., A Novel Neural Source Code Representation based on Abstract Syntax Tree, In Proc. the 41th International Conference on Software Engineering (ICSE 2019), Montreal, Canada, 2019.

The data quality should be discussed in detail, as low quality data will bias the analysis results. This is particularly important for a public benchmark. For example, if the benchmark contains a lot of duplicated code, the follow-up analysis will be misleading. Furthermore, software evolves. Very soon, new versions/commits will emerge. It is not clear if the evolution will degrade the data quality and the validity of the benchmark.  

The proposed benchmark data and code are not available for replication purpose.

In Table 2, the baseline result for Transformer-based method completion is missing. 

The paper is generally well-written. There are a few typos. For example:

In page 3, ”?? provides details and examples...”

review_title: GLUECode: A Benchmark for Source Code Machine Learning Models

Meta-review: This paper proposes a new source code modeling benchmark, with the unique twist being that we not only have code source text, but we also have build information, which allows extracting richer information to construct labels from. This enables, for example, a null pointer prediction task with labels coming from an inter-procedural static analysis tool. AC and reviewers agree that this is a valuable framing for a benchmark suite. Unfortunately, it’s not clear that the benchmark in its current form delivers on the promise of the framing. Much of the interest and novelty is limited to just the one NullToken task, and reviewers raise a number of concerns including dataset size and whether the task truly measures the inter-procedural reasoning that it sets out to measure. AnonReviewer2 raised some good questions here that the authors promised to address in a forthcoming comment, but that didn’t come before the discussion deadline. I’d encourage the authors to use the reviewer suggestions to more strongly establish that these tasks measure what they set out to measure, and also to consider adding other tasks that measure whether our ML models are capable of deeper / longer-range reasoning. In total, there is a lot of potential here, but the work needs another iteration before it’s ready for publication.
Decision: Reject