Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: # Summary

This paper presents a BERT-inspired pretraining/finetuning setup for source code tasks. It collects a corpus of
unlabeled Python files for BERT pretraining, designs or adopts 5 tasks on established smaller-scale Python corpora, and
adjusts the BERT model to tokenize and encode source code snippets appropriately.

# Strengths

* The idea of applying the pretraining/finetuning paradigm to program analysis tasks makes sense, and has been
  informally attempted by multiple groups in the community in 2019. This is the first high-quality submission to a
  top-tier ML conference I've seen on the subject, though.
* The authors exercised commendable care and diligence in preparing the training data, adopting BERT to source code
  inputs, and ensuring correctness of the experimental setup. I appreciated all the provided details on tokenization
  (Section 3.3), deduplication (Sections 3.1-3.2), and task setup (Section 3.5). This should become a technical standard
  in the community.
* The paper is written clearly and concisely, and is generally a pleasure to read.

# Weaknesses

I have a gripe with the authors' choice to ignore program structure (e.g. abstract syntax trees) or features (e.g.
types) in their program representation. Without this extra information (easily available from a compiler/interpreter
API) the pipeline is not substantially different from the original NLP pipeline of BERT et al. The main program-related
representation insight comes in tokenization (Section 3.3) and the definition of "sentences". To repeat, I appreciate
the effort the authors put in making tokenization appropriate for BERT processing of source code, but this is a drop in
the bucket compared to the all the other program-related features the work is leaving off the table. Programs are not
natural language.
The argument that source code analysis would "pass on the burden ... to downstream tasks" (Page 3) is odd. First, most
downstream tasks of interest occur in the settings where this analysis is already available: IDEs, code review
assistants, linters, etc. Second, one often needs program analysis to even define downstream tasks in the first place --
for example, determining whether function arguments are swapped required detecting a function call and boundaries of its
arguments, thus parsing the program!

This work obtains (and nicely analyzes) impressive results obtained by applying CuBERT. However, it does not put the
results in context with prior work based on structured program representations. Without this, it is difficult to say
whether the improvement comes from pretraining or from the language model simply learning a better "parsed"
representation of an input program from all the unlabeled corpus. If it's the latter, one might argue that supplying the
model with structured program features explicitly might eliminate much of the need for the unlabeled corpus.
I personally think that there will still be a gap between pretraining and finetuning even with structured program
features simply due to the sheer volume of available data, which, as the authors showed, is crucial for good
generalization of Transformer. However, this still needs to be shown empirically.

The "Function-Docstring Mismatch" task, as presented, seems too easy. If the distractors (negative examples) are truly
chosen at random, most of them are going to use obviously different vocabulary from the original function signature (as
Figure 4 demonstrates). A well designed task would somehow bias the sampling toward subtle distractors such as `get` vs.
`set` docstrings, but this seems challenging.
This also explains why the task is not influenced as much by reduction of training data (Table 3).

The Next Sentence Prediction pretraining task, as adapted for CuBERT, seems too difficult, in contrast. If the paired
sentences (i.e. code lines) are chosen at random, the model would lack most of the context required to make a decision
about the logical relationship between them, such as which variables are defined and available in context, which
functionality is being implemented, etc. I wonder, can the authors experiment with pretraining CuBERT only with the
Masked Language Model task? Will it worsen the results substantially or at all?

# Questions

Section 3.2: "similar files according to the same similarity metric..."
What are these metrics?

What is the fraction of positive/negative examples in the constructed finetuning datasets?

What is the motivation for making Variable Misuse and Wrong Operator/Operand into a simple classification tasks instead
of the original (more useful) correction task?

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes a transformer based approach to address a number of problems for machine learning from code. Then, the paper adds BERT-based pretraining to significantly improve on the results. The paper is nicely written and easy to follow, but with relatively thin contributions besides the large amount of experiments.

Initially, I was quite positive on the entire paper, but as I read it in more details, I got less convinced that this is something I want to see at ICLR. First, there are no good baselines. BiLSTM gets quite high accuracy on most of the tasks, which is unexpected because most prior works show that the tasks benefit from some semantic understanding of code. I cannot relate any of the numbers with a previous work. Right now, I even have reduced confidence that the authors do not have bugs in the tasks or the reported numbers. Then, for the Operator and Operand tasks, BiLSTM is also doing impressively well (these tasks were done differently in prior works). Interestingly, things get reversed on the last two tasks. Given that most of the experiments are not the same as in any previous paper, I would strongly appreciate if much more details are given in the appendix. In fact, the appendix right now does not have much useful information besides praising how good job the attention is doing. What would be needed is information on how many samples were generated, how large were they, was any data thrown out? Table 1 is a good start, but it actually raises questions. You split the Python dataset into separate functions like Vasic et al and the number of functions 2x higher (counting the artificial samples, I guess), did you put a limit on how large functions you consider? 250 tokens was the limit of Vasic et al. To which of the tasks in the Vasic et al paper can I relate? Is the BiLSTM on the level of that work or it is substantially worse or better? Also, are the results coming from the paper SOTA or uncomparable to other works?

Five tasks are evaluated, which is impressive. This is one reason I want to see the results published. The problem is also quite important. The experiments that show the effect on reduced labelled data are quite important and interesting - in fact, for many tasks, we can start curating datasets and having model working on small data will be crucial. However, I think the paper needs more work before it is something to present, cite or build upon.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper describes a BERT-based pre-training for source code related task. By pre-training on BERT-like models on source code and finetuning on a set of 5 tasks, the authors show good performance improvements over non-pretrained models. The authors make a series of ablation studies showing that pre-training is indeed useful.

Overall, I find this work relevant and interesting, albeit somewhat unsurprising. Nevertheless, I see no reason to reject this paper. To make my "weak accept" to a "strong accept" I would like to see experiments on more tasks, preferably more complex tasks. For example, such tasks could include (a) variable naming (b) method naming (c) docstring prediction/summarization (d) language modeling/autocompletion. I believe it's unclear to the reader if pre-training is also helpful for any of those tasks too and adding such experiments would significantly strengthen the paper.

Some clarifications comments/questions to the authors:

* I would insist that the authors rename the "Variable Misuse" task to "Variable Misuse Localization". To my understanding the current model points to the misused variable (if any), but does not attempt to suggest a fix. This tackles only a part of the task discussed in Vasic et al. (2019), Allamanis et al (2018) and this might confuse readers who want to compare with those works.

* For the Function-Docstring Mismatch task (Section 3.4):
    * It's unclear to me which dataset is used. Is it the Py150 dataset or the Barone & Sennrich (2017)?
    * I believe that the citations [a], [b] would be appropriate here.

* Overall, for the all the tasks except from "Exception Type", there is a replicability issue: Since the authors manually mutate the code (e.g. introduce a variable misuse, swap an operand), for anyone to compare directly, they would need access to the mutated samples. I would strongly encourage the authors to provide more details on how they create mutated samples and (eventually) the source code that achieves that.

* For the Variable Misuse, Wrong Binary Operator, Swapped Operand tasks. There are a few things that need to be clarified:
   * How long is each code snippet? One would expect that the longer the code snippet the harder the task. Do the authors pass a whole function?
   * What is the proportion of positive/negative examples in each task?



[a] Cambronero, Jose, et al. "When Deep Learning Met Code Search." arXiv preprint arXiv:1905.03813 (2019).
[b] Louis, Annie, et al. "Deep learning to detect redundant method comments." arXiv preprint arXiv:1806.04616 (2018).
review_title: Official Blind Review #2

Meta-review: The paper presents  CuBERT (Code Understanding BERT), which is BERT-inspired pretraining/finetuning setup, for source code contextual embedding. The embedding results are tested on classification tasks to demonstrate the effectiveness of CuBERT. 

This is an interesting application paper that extends existing models to source code analysis. The authors did a good job at motivating the applications, describing the proposed models and discussing the experiments. The authors also agree to share all the datasets and source code so that the experiment results can be replicated and compared with by other researchers. 

One major concern is the lack of strong baselines. All reviewers are concerned about this issue. The paper could lead to a good publication in the future if the issues can be addressed. 
Decision: Reject