Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This work address the pretraining over code and text. It proposes to leverage data flow as additional inputs, and add two structure aware pre-training tasks besides the masked token prediction task. The pretrained model is evaluated on four different tasks and outperforms the CodeBERT baselines as well as other pretrained models. Further analysis confirmed the benefits from the additional tasks and data flow input. 

Strength:

1. The use of data flow as additional input and the proposal of structure aware tasks are well motivated adaptations of pre-training to code. 

2. The extensive experiments and comparisons supported the claim and additional ablation studies and analysis confirmed the benefits from the additional structure-aware tasks and the data flow representation. 

3. The comparison with syntactic structures (AST) is interesting and demonstrated the higher potential of semantic structures like data flow. This could be beneficial to other code related tasks. 

4. The paper is well written and easy to follow. 

Weakness:

1. Since the proposed approach is adding extra inputs, and given the O(n^2) complexity of Transformer, it would help to add some information regarding the increase in compute cost, for example, additional flops or latency due to additional data flow inputs. 

Related to the point above, although linearizing the data flow into sequence seems the simplest approach, it might be computationally inefficient given that the data flow graph is probably sparse, thus large number of attention position needs to masked. But maybe the data flow input is much smaller than the code input so that it is not adding too much overhead.  

It would also help to add more statistics about the inputs, for example, the average length of the code, the length of the linearized data flow, the length of the linearized AST, etc. 

2. Not so much a weakness, but it would help to show some examples, in appendix if the space is limited, where GraphCodeBERT improves over CodeBERT or other models without semantic structures, to give some more intuition. 
review_title: Solid work on introducing structure aware tasks and data flow representation to pre-training on code

final_score: rating: 7: Good paper, accept
review_content: 
Overview:
The authors present Graph Code BERT, the first language model that leverages data flow to learn code representation. They use three objective functions: Masked Language Modeling, Edge Prediction, and Node Alignment. They claim their structure-aware pre-training can help improving performance on code-related downstream tasks, including code search, clone detection, code translation, and code refinement.


Reasons to accept:
* The paper is well-written and easy-to-follow.

* It is the first pre-trained model that leverages the data flow structure of code to learn code representation.

* They propose two pre-training tasks for learning representation from source code and data flow. Edge Prediction is more interesting in my opinion.

* They show improvement on four downstream tasks, even though some of them are marginal in my opinion compared to CodeBERT.

Reasons to reject:
* This work is not the first work to conduct language model pretraining for code understanding applications. I will expect more comprehensive comparisons with related work. For example, Svyatkovskiy et al. (2020) propose GPT-C, which is actually a transformer architecture that can also use for both generation and classification tasks. Missing C-BERT (Buratti et al.) as well.

* The 2.4M functions used for "pre-training" is not very persuasive. I think it is more reasonable to crawl existing code repositories for Pre-training. I do not see a particular reason because pre-training without natural language descriptions still makes sense.

Questions & Suggestions:
* Do you find natural language descriptions useful in your experiments? Why if we discard it from pre-training?

* What is the ratio of edge connection? 10%? 20%? The reason I am asking this is that when you do the randomly sampling 20% of nodes Vs in the data flow, how many node pairs you used to calculate the loss? Is the positive-negative ratio balanced? Please give more details about the training objectives.

* Please give more prediction examples or qualitative analysis for each downstream tasks in the Appendix.
review_title: Review of Graph Code BERT

final_score: rating: 7: Good paper, accept
review_content: This paper extends CodeBERT to include elements of dataflow in
addition to the comments and sequence of code tokens. Also, the
pretraining includes two tasks that are structure-aware: predicting
dataflow edges and aligning dataflow nodes to code. The paper is well
written, well motivated and the empirical evaluation is quite
thorough. The performance compared to CodeBERT is around 1-2%
additional accuracy for a series of diverse tasks.

Detailed comments:
- In general the paper is well written. There were a few parts that
  required further clarification for me. For example, it is not clear
  how the variable names are represented textually (e.g. x1 - is it
  really x1 or a different way of expressing this in the sequence of
  tokens that is sent to the model). The architecture seems to me to
  be pretty identical to BERT with the exception of the attention
  matrix for the dataflow portion. Is this matrix kept constant
  throughout training? Not clear to me if the dataflow graph is a
  traditional dataflow (as in the one used in compiler analysis). It
  appears to me that it's a slightly simplified version based on AST
  and some idea of data flow. I would contrast and compare with
  "traditional" dataflow graphs (that use SSA - single static
  assignment form - for example). Not exactly clear what you mean by
  "neat". I would find a different word. Neat sounds like "pretty" in
  this context; I don't find it scientific. Do you mean less complex?
  The Noe alignment task was not clear to me at all. Perhaps an
  example would be helpful showing the actual task and what is
  provided as input (I'm guessing the same thing as usual) and how the
  prediction looks like.

- I'm not sure there is a discussion on the experimental methodology
  and hyperparameter tuning. Are the results presented some average
  over several runs? If they are single runs, the difference in
  accuracy for most task is within noise, I would think. Also, I
  particularly find the third decimal a bit much for "stretching"
  result improvements. I think the idea is simple, it gives some
  limited performance improvement over CodeBERT but the model
  architecture is overall similar so it is probably worth it
review_title: Simple extension over CodeBERT with small performance improvement for a series of tasks

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes GraphCodeBERT as a Transformer-based pretrained model for programming language that incorporates data flow information in the graph representation of variables in the code. The data flow graph encodes the structure of variables based on “where-the-value-comes-from” from the AST parse. The pretrained model is jointly trained on the code, the natural language comment of the code, and the data flow graph of the code. In addition to the Masked Language Modeling objective, two new pretraining objectives are proposed including predicting the edge of the data flow graph and predicting the alignment of variables between data flow graph and code. The graph-guided masked attention is used such that the attention can only occur if two variables have an edge in the data flow graph or there is an alignment between data flow graph and code. The experiments show that GraphCodeBERT can deliver improvements on Natural Language Code Search, Code Clone Detection, Code Translation, and Code Refinement.

Strengths

- The approach is well motivated, and I totally agree that we should consider the code structure for a pretrained code language model.

- There are consistent, though small, improvements on all four tasks.

- The case study and ablation study are very helpful to understand why GraphCodeSearch is better than other models with better representation of code semantics on the Code Search task.

Weakness

- This paper uses the data flow graph and claims this is better than AST because AST has an "unnecessarily deep hierarchy". However, as the data flow graph is extracted from AST, data flow contains only partial information of AST, especially the information of “where-the-value-comes-from”, and throws out other useful information such as function name, condition control, operator. It is not clear to me if this design choice is a good one. 

- The experiments should have a comprehensive comparison using AST and the data flow graph to answer the questions such as: What is the performance of using AST instead of the data flow graph? What information in AST is useful for which tasks? In addition to “where-the-value-comes-from” information, what other information shall we incorporate into a pretrained language model for code? Right now, there is only analysis on the code search task and there is not enough discussion about AST approaches.

- The improvement of GraphCodeBERT over previous SOTA, especially CodeBERT, is marginal. There is no significance test in the results. On Code Clone Detection, the improvement is 0.6% F1. On code translation, the improvement is less than 1% for accuracy. The most improvement is on 5.1 Natural Language Code Search, but the data flow graph and the two newly introduced objectives are not relevant to natural language comment.

- I don't think BLEU is a meaningful metric for code translation and code refinement, and it can be misleading. For example, in Table 4, BLEU is 90+ while the accuracy is less than 10%. The author can consider not reporting BLEU.

Questions
-  Can you please give more details about how to use AST Preorder Traversal and AST Subtree Masking for code search? I am wondering if this is a good usage of AST on this task. Thanks!

Minor
- Notation W is used to denote both parameters in self-attention (Eq (1)) and the code comment. Consider using two different notations.

review_title: AnonReviewer4

Meta-review: This paper proposes a simple extension to BERT-like pre-training for source code models, which allows incorporation of data flow information. This is a new way of incorporating code structural information into models, and it appears practical and effective. Reviewers are all in favor of accepting the paper.
Decision: Accept (Poster)