GraphCodeBERT: Pre-training Code Representations with Data Flow
Authors: Daya Guo
Shuo Ren
Shuai Lu
Zhangyin Feng
Duyu Tang
Shujie LIU
Long Zhou
Nan Duan
Alexey Svyatkovskiy
Shengyu Fu
Michele Tufano
Shao Kun Deng
Colin Clement
Dawn Drain
Neel Sundaresan
Jian Yin
Daxin Jiang
Ming Zhou
Venue: ICLR 2021 Poster (2021)
Keywords: Pre-training
BERT
Code Representations
Code Structure
Data Flow

Abstract: Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc. However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process. We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables. Such a semantic-level structure is neat and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efficient. We develop GraphCodeBERT based on Transformer. In addition to using the task of masked language modeling, we introduce two structure-aware pre-training tasks. One is to predict code structure edges, and the other is to align representations between source code and code structure. We implement the model in an efficient way with a graph-guided masked attention function to incorporate the code structure. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refinement. Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks. We further show that the model prefers structure-level attentions over token-level attentions in the task of code search.

Published as a conference paper at ICLR 2021
GRAPHCODEBERT: PRE-TRAINING CODE REPRESEN- TATIONS WITH DATA FLOW
Daya Guo Shuo Ren Shuai Lu Zhangyin Feng**, Duyu Tang", Shujie Liu Long Zhou", Nan Duan', Alexey Svyatkovskiy*, Shennny Fu", Michele Tufano", Shao Kun Deng", Colin Clement", Dawn Drain Neel Sundaresan Jaan Yin', Daxin Jiang', and Ming Zhou Scchool of Computer Science and Engineering. Sun Yat-sen University. "Beihang University, PPeking University, "Harbin Institute of Technology. 6 Microsoft Research Asia, Microsoft Devdiv, Microsoft STCA
ABSTRACT
Pre-trained models for programming language have achieved dramatic empirical improvements variety of code-related tasks such as code search, code comple- tion, code summarization, etc. However, existing pre-trained models regard code snippet as sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semanties and would enhance the code understanding process. We present GraphCodeBERT, pre-trained model for programming language that considers the inherent structure of code. Instead taking syntactic-level structure of code like abstract syntax tree (AST), use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where- wwwa-valllee--rroooooe
between variables. Such a semantic-level structure is less complex and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efficient. We develop GraphCodeBERT based on Transformer. In addition to using the task of masked language modeling, we introduce two structure-ane aware pre-training tasks. One to predict code structure edges, and the other is to align representations between source code and code structure. We implement the model an efficient way with a graph-guided masked attention function to incorporate the code structure. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refine- ment. Results show that code structure and newly introduced pre-training tasks improve GraphCodeBERT and achieves state-of-the-ae performance on the four downstream tasks. We further show that the model prefers structure-level attentions over token-level attentions in the task of code search []]
INTRODUCTION
Pre-trained models such as ELMo (Peters et al. 2018), GPT (Radford et al. 2018) and BERT (Devlin et al. 2018) have led to strong improvement numerous natural language processing (NLP) tasks. These pre-trained models are first pre-trained on a large unsupervised text corpus, and then fine-tuned on downstream tasks. The success of pre-trained models in NLP also promotes the development of
pre-trained models for programming language. Existing works (Kanade et al. 2019 Karampatsis & Sutton, 2020; Feng et al., (2020; Svyatkovskiy et al., 2020 Buratti et al.) (020) regard source code as a sequence of tokens and pre-train models on source code to support code-related tasks such as code search, code completion, code summarization, etc. However, previous works only utilize source code for pre- pre-training, while ignoring the inherent structure of code. Such code structure provides useful semantic information of code, which would benefit the code understanding process. Taking the expression v maz value min value as an example, u is computed from maz value and min_valuue. Programmers do not always follow the naming conventions so that it's hard to understand the semantic of the variable only from its name. The semantic structure of code provides way to understand the semantic of the variable o by leveraging dependency relation between variables.
'Wor done while this author was an intern at Microsoft Research Asia. Contact: Daya Guo
(guody5 @ mail22syssu.dd..cm
'll the codes and data are available /tttppp:/ggttiit.cccc///ooooooop

---

Published as a conference paper at ICLR 2021
In this work, we present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. Instead of taking syntactic-level structure of code like abstract syntax tree (AST), leverage semantic-level information of code, i.e. data flow, for pre- training. Data flow a graph, in which nodes represent variables and edges represent the relation of tathere-the-vlle-ccome frresss--'mm between variables. Compared with AST, data flow is less complex and does not bring an unnecessarily deep hierarchy, the property of which makes the model more efficient. In order learn code representation from source code and code structure, we introduce two new structure-awe awarr pre- pre-training tasks. One is data flow edges prediction for learning representation from code structure, and the other is criiable-alggmment across source code and data flow for aligning representation between source code and code structure. GraphCodeBERTT is based on Transformer neural architecture (Vaswani et al. 2017) and we extend it by introducing graph-guided masked attention function to incorporate the code structure.
We pre-train GraphCodeBERT on the CodeSearchNet dataset (Husain et al. 2019), which includes 2.3M functions of six programming languages paired with natural language documents. We evaluate the model on four downstream tasks: natural language code search, clone detection, code translation, and code refinement. Experiments show that model achieves state-of-the-an performance on the four tasks. Further analysis shows that code structure and newly introduced pre- pre-training tasks can improve GraphCodeBERRT and the model has consistent preference for attending data flow.
In summary, the contributions of this paper are: (1) GraphCodeBBRRT is the first pre-trained model that leverages semantic structure of code learn code representation. (2) We introduce
new structure-awe pre-training tasks for learning representation from source code and data flow, (3) GraphCodeBERY provides significant improvement on four downstream tasks, i.e. code search, clone detection, code translation, and code refinement.
2 RELATED WORKS
Pre-Trained Models for Programming Languages Inspired by the big success of pre-training in NLP (Devlin et al. 2018 Yang et al.) 22019: Liu et al. 2019 Raffel et al. 2019). pre-trained models
for programming languages also promotes the development of code intelligence (Knade et al. 2019
Feng et al., 2020; Karampatsis & Sutton, 2020)) Svyatkovskiy et al.) 2020) Buratti et al.) 2020).
Kanade et al. (2019) pre-train BERT model on a massive corpus of Python source codes by masked language modeling and next sentence prediction objectives. Feng et al. (2200) propose CodeBERT,
bimodal pre-tra pre-trained model for programming and natural languages by masked language modeling and replaced token detection to support text-code tasks such as code search. |arampatsis & Sutton (2020)
pre-train contextual embeddings on a JavaScript corpus using the ELMo framework for program repair task. (Svyatkovskiy et al. (2020) propose GPT-C, which is variant of the GPT-2 trained from scratch on source code data to support generative tasks like code completion. |Buratti et al. (2020) present C-BERT, transformer-basee language model pre-trained on collection of repositories written in C language, and achieve high accuracy in the abstract syntax tree (AST) tagging task.
Different with previous works, GraphCodeBERT is the first pre-trained model that leverages code structure to learn code representation to improve code understanding. We further introduce a graph- guided masked attention function incorporate the code structure into Transformer and two new structure-aware pre-training tasks to learn representation from source code and code structure.
Neural Networks with Code Structure In recent years, some neural networks leveraging code structure such as AST have been proposed and achieved strong performance in code-related tasks like code completion (Li et al. Aloo,Alon Alon et al., 22099110. Kim et al., 2020), code generation (Rabinovich al. 2017 Yin & Neubig 2017 Brockschmidt et al.) 2018), code clone detection (Wei & 2017] Zhang et al. 2019 Wang et al. (2020), code summarization (Alon et al.) 2018 Hu et 20.18)018) and so on (Nguyen & Nguyen 2015 Allamanis et al. 2018 8 Hellendoorn et al. 2019). 9.. Nguyen & Nguyen (2015) propose an AST-based language model to support the detection and suggestion of syntactic template at the current editing location. Allamanis et al. (2018) use graphs to represent programs and graph neural network reason over program structures. Hellennoorm et al. (2019) propose two different architectures using a gated graph neural network and Transformers for combining local and global information to leverage richly structured representations of source code. However, these
2

---

Published as a conference paper at ICLR 2021
works leverage code structure to learn models specific tasks from scratch without using pre- pre-trained models. In this work, we study how to leverage code structure for pre- pre-training code representation.
DATA FLOW
In this section, we describe the basic concept and extraction data flow. In next section, we will describe how use data flow for pre-traine pre-training.
Data flow a graph that represents dependency relation between variables, in which nodes represent variables and edges represent where the value of cach variable comes from. Unlike AST, data flow same under different abstract grammars for the same source code. Such code structure provides crucial code semantic information for code understanding. Taking an example, programmers do not always follow the naming conventions so that it is hard to understand the semantic of the variable. Data flow provides way understand the semantic of the variable to some extent, i.e. the value of comes from mor_value and min_calue in data flow. Besides, data flow supports the model consider long-range dependencies induced by using the same variable or function in distant locations. Taking Figure I an example, there are four variables with same name (i.e. x²3 x²,, _ and 11 but with different semantic. The graph in the figure shows dependency relation between these variables and supports x to pay more attention to x and x instead of x Next, we describe how to extract data flow from a source code.
valuue min_vallnaaaoooao valuee as
Source code
Parse into AST
Identify variable sequence
Variable relation
def maxia, b): x=0 if b>a:: x=b else:
def max(al Inj: x=0 a x=b' else: x-a
x=a
return
return x"
commes from
Complier Tool
Idbeniily voriable sequer ice in AST
Extrett vor ablle relation from AST
The procedure of extracting data flow given source code. The graph in the rightmost is data flow that represents the relation of wwwheer-iie-valle-cone------e between variables.
Figure 1:
Figure I shows the extraction of data flow through a source code. Given source code C J. we first parse the code into abstract syntax tree (AST) by a standard compiler tool] The AST includes syntax information of the code and terminals (leaves) are used to identify node of the graph and an direct edge E = (V,, Uz) from v; to By refers that the value of j-th variable comes from i-th expr
{ci .2, en
the variable sequence, denoted V {U1, U2, Jg We take each variable
variable. Taking
example, edges from all variables in expr to are added into the graph. We denote the set of directed edges as E = {& E2, and the graph G(C") = (V, E) is data flow used to represent dependency relation between variables of the source code C.
4 GRAPHCODEBERT
In this section, describe GraphCodeBERI,, a graph-based pre-trained model based Transformer for programming language. We introduce model architecture, graph-guided masked attention and pre-training tasks including standard masked language model and newly introduced ones. More details about model pre-training setting are provided in the Appendix A.

---

Published as a conference paper at ICLR 2021
Source code
data flow edge prediction among variables
def maxantts if ii-a" DF else:
voriable diignment code and
moximum
Moxied tonguage Modeing
returni
Comment
GraphCodeBERT
Return maximum volue
Data Flow
[CLS]
Value connes from
Text
Variable Sequence
An illustration about GraphCodeBERT pre-training. The model takes source code paired with comment and the corresponding data flow as the input, and is pre-trained using standard masked language modeling (Devlin et al. 2018) and two structure- aware tasks. One structure-awar task is to predict where a variable is identified from (marked with orange lines) and the other is data flow edges prediction between variables (marked with blue lines).
Figure 2:
4.1 MODEL ARCHITECTURE
Figure Shours shows the model architecture of GraphCodeBBRRT. We follow BERT (Devlin et al. 2018) and use the multi-layer bidirectional Transformer (Vaswani al. 2017) as the model backbone. Instead of only using source code, we also utilize paired comments to pre-train the model to support more code-related tasks involving natural language such as natural language code search (Feng et al.) 2020). We further take data flow, which is a graph, as part of the input to the model.
Given a source code C' {or, C2. ..., Cn } with its comment W {wi, W2, Wm. J. we can obtain the corresponding data flow (CC) (V, F) as discussed in the Section 3 where V= {01, U2,-.., 0% a set of variables and = {E1, E2; EI.,Er} a set of direct edges that represent where the value of each variable comes from. We concatenate the comment, source code and the set of variables as the sequence input X = [CLLS W, [SEP], C, [SEP], VJ. where [CLS]] a special token front of three segments and [SEP] is special symbol to split two kinds of data types.
GraphCode takes the sequence X as the input and then converts sequence into input vectors H For each token, its input vector is constructed by summing the corresponding token and position embeddings. We use a special position embedding for all variables to indicate that they are nodes of data flow. The model applies N transformer layers over the input vectors produce contextual representations H = trans fonnsform former, (H*-l) )),n E [1, N). Each transformer layer contains an architecturally identical transformer that applies a multi-headed self-attention operation (Vaswani et al. 2017) followed by feed forward layer over the input H=-1 in the nith layer.
G H (G"FFN(G") GT))
LN (M ulti Att((HH + H=-1)
(2)
where AultiAttn a multi-headed self-attention mechanism, FFN is a two layers feed forward network, and LN represents layer normalization operation. For the n-th transformer layer, the output 6"h of multi-headed self-attention is computed via:
Qi H= K; H Hn-lW V, H
(3)
QiK
head, softmax Vdx
MMV;
(4)
G [headi;... =
head,,ww
(5)
where the previous layer's output H=-1 1 E [X|xx is linearly projected to a triplet of queries, keys respectively. u the number of heads, dy the dimension of a head, and W E Rdd xd is the model parameters. M E X|||||X| is mask matrix, where Mij is 0 if i-th token is allowed to attend j-th token otherwise -00.
and values using model parameters W.º.W.t ..V
4

---

Published as a conference paper at ICLR 2021
4.2 GRAPH-GUIDED MASKED ATTENTION
To incorporate the graph structure into Transformer, we define a graph-guided masked attention function to filter out irrelevant signals. The attention masking function could avoid the key k; attended by the query Qj by adding the attention score a k, an infinitely negative value so that the attention weight becomes zero after using softmax function. To represent dependency relation between variables, node-query qu, is allowed to attend to a node-key ku, if there is direct edge from the node ; to the node V/ (i.e. (uy, u/) E E)) or they are the same node (i.e. = j). Otherwise, the attention is masked by adding an infinitely negative value into the attention score. To represent the relation between source code tokens and nodes of the data flow, we first define set E', where (ue, c,)/(c,, u.) E E' if the variable 2y is identified from the source code token cj. We then allow the node Qu, and code ke attend each other if and only if (vs, cj) (i), vi) E E', More formally, we use the following graph- huph-ggided masked attention matrix as the mask matrix M in the equation
if E ([CT.S), [SEP]} 4s ki E W U or kg) E EUUE otherwise
Myj
(6)
4.3 PRE-TNAINING TASKS
We describe three pre-training tasks used for pre-training GraphCodeBERT in this section. The first task is masked language modeling (Devlin et al. 2018) for learning representation from the source code. The second task is data flow edge prediction for learning representation from data flow, where we first mask some variables' data flow edges and then let GraphCodeBERT predict those edges. The last task is variable-alignment across source code and data flow for aligning representation between source code and data flow, which predicts where a variable is identified from.
Masked Language Modeling We follow Devlin et al. 20188) to apply masked language modeling (MLM) pre- training task. Specially, we sample randomly 15% of the tokens from the source code and paired comment. We replace them with a [MASK] token 80% of the time, with a random token 10% of the time, and leave them unchanged 10% of the time. The MLM objective is to predict original tokens of these sampled tokens, which has proven effective in previous works (Devlin et al., 2018 Liu et al. 2019; Feng et al.] 2020). In particular, the model can leverage the comment context if the source code context is not sufficient to infer the masked code token, encouraging the model to align the natural language and programming language representations.
Edge Prediction To learn representation from data flow, we introduce a pre-training task of data flow edges prediction. The motivation is to encourage the model to learn structure- aware repre- sentation that encodes the relation of "wwere-the- theee---alle--ome-frrom for better code understanding. Specially, we randomly sample 20% of nodes V, in data flow, mask direct edges connecting these sampled nodes by add an infinitely negative value in the mask matrix, and then predict these masked edges Emask. Taking the variable = in Figure 2 for an example, we first mask edges (x",, ,x")) and (x.", ,) in the graph and then let the model to predict these edges. Formally, the pre-training objective of the task is calculated as Equation / where E. V, V U V, is a set of candidates for edge prediction, Ö(eij E E) is if (us, E E otherwise 0, and the probability Pri, of existing an edge from i-th to j-th node is calculated by dot product following a sigmoid function using representations of two nodes from GraphCodeBERT: To balance positive-negative ratio of examples, we sample negative and positive samples with the same number for E..
EdgePed =
[ō(e.z E Moggst + (1 - 5(esy E Eloos))log(11 - Pe.g)) ) E.yEE.
Σ
Node Alignment To align representation between source code and data flow, we introduce a pre- training task of node alignment across source code and data flow, which is similar to data flow edge prediction. Instead of predicting edges between nodes, we predict edges between code tokens and nodes. The motivation is to encourage the model to align variables and source code according to data flow. Taking Figure 3 for an example, we first mask edges between the variable x!' in data flow and code tokens, and then predict which code token the variable x!1 in data flow is identified from. As we can see, the model could predict that the variable x1l is identified form the variable x in the expression "return x" according data flow information (i.e. the value of Ill comes from x² or x²).
5

---

Published as a conference paper at ICLR 2021
Source code
mextii 'ol oghag FRSSWw
Predict which code token the vaniotle 2 flow identtfeed from
(CLS) (MASK] volue (SSP) def max (o, bJ:x-0 ebeex-o (539)
BraphCodeBERT
Comment
Aeuurs mastmumn HouR
Data Flow
Mask edges between the variable x data flow and code tokenns
Text
Code
Variable Sequence
Figure 3: An example of the Node Alignment task.
Specially, we randomly sample 20% nodes V in the graph, mask edges between code tokens and sampled nodes, and then predict masked edges Emaek. The pre-training objective of this task similar to Equation 7 where F. = V." x C is a set of candidates for node alignment. Similarly, we also sample negative and positive samples with the same number for E..
loss Align
Σ eiEEE
1ô(egs E Emmas (1 Ö(eij )og() )ogg(1 Pe..))
5
EXPERIMENTS
We evaluate our model on four downstream tasks, including code search, clone detection, code translation and code refinement. Detailed experimental settings can be found in the Appendix.
5.1 NATURAL LANGUAGE CODE SEARCH
Given natural language as the input, the task aims to find the most semantically related code from collection of candidate codes. We conduct experiments on the CodeScarchNet code corpus (Husain al. 2019), which includes six programming languages. Different from the dataset and the setting
used in the Husain et al.) (2019), we filter low-quality queries by handcrafted rules and expand 1000 candidates the whole code corpus, which closer the real-life scenario. We use Mean Reciprocal
Rank (MRR) as our evaluation metric and report results existing methods the Table T We provide more details about the filtered dataset and also give results using the same setting of Husain
et al. (019)) in the Appendix B.
model
Ruby
Javascript
Go
Python
Java
Php
Overall
NBow
0.162
0.157
0.330
0.161
0.171
0.152
0.189
CNN
0.276
0.224
0.680
0.242
0.263
0.260
0.324
BiRNN
0.213
0.193
0.688
0.290
0.304
0.338
0.338
selfAtt
0.275
0.287
0.723
0.398
0.404
0.426
0.419
RoBERTa
0.587
0.517
0.850
0.587
0.599
0.560
0.617
RoBERTa (code)
0.628
0.562
0.859
0.610
0.620
0.579
0.643
CodeBERT
0.679
0.620
0.882
0.672
0.676
0.628
0.693
GraphCodeBERT
0.703
0.644
0.897
0.692
0.691
0.649
0.713
Table 1: Results on code search. GraphCodeBERR outperforms other models significantly (p < 0.01).
All models calculate inner product of code and query encodings as relevance scores to rank candidate codes. We follow Husain et al. (011)) to implement four methods baselines in the first group to obtain the encodings, including bag-of-worde convolutional neural network, bidirectional recurrent neural network, and multi-head attention. The second group is the results of pre-trained models. Roberta (Liu et al. 2019) pre-trained model on text corpus with MLM learning objective, while RoBERTa (code) pre-trained only code. CodeBERT (Feng et al. 2020) is pre-trained

---

Published as a conference paper at ICLR 2021
code-text pairs with MLM and replaced token detection learning objectives. As see, GraphCodeBERT that leverages code structure for pre- pre-training brings a 2% gain of MRR, achieving the state-of-art performance. We also conducted t-test between GraphCodeBERT and other baselines, and the results show the improvements are significant with p < 0.01.
5.2 CODE CLONE DETECTION
Code clones multiple code fragments that output similar results when given the same input. The task aims to measure the similarity between two code fragments, which can help reduce the cost of software maintenance and prevent bugs. We conduct experiments on the BigCloneBench dataset (vajjenko et al. 2014) and report results in the Table 2
Deckard (Jian al. 2007) is compute vectors for structural information within ASTs and then a Locality Sensitive Hashing (LSH) (Datar et al. (004) is used to cluster similar vectors for detection. RtyNN (White al. 2016) trains recursive autoencoder to Iearn representations for AST. CDLH (Wei & Li, (017) learn representations of code fragments via AST-baaed LSTM and hamming distance is used to optimize the distance between the vector representation of AST pairs.
ASTNN Zhang et al. (2019) uses RNNs to encode AST subtrees for statements, then feed the encodings of all statement trees into an RNN to learn representation for a pro- gram. FA-AST-GMN (Wang al.] 2020) uses GNNs over a flow-augmented AST to leverages explicit control and data flow in- formation for code clone detection. Results show that our GraphCodeBERT that lever- ages code structure information significantly outperforms other methods with p < 0.01, which demonstrates the effectiveness of our pre-trained model for the task of code clone detection.
Model
Precision
Recall
F1
Deckard RtvNN
0.93
0.02
0.03
0.95
0.01
0.01
CDLH
0.92
0.74
0.82
ASTNN
0.92
0.94
0.93
FA-AST-GMN
0.96
0,94
RoBERTa (code)
0,960
0,955
0.957
CodeBERT
0.964
0.966
0.965
GraphCodeBERT 0.973
0.968
0.971
Table 2: Results on code clone detection. Graph- CodeBERT outperforms other pre-trained methods significantly < 0.01).
5.3
CODE TRANSLATION
Code translation aims to migrate legacy software from one programming language a platform to another. Following Nguyen et al. 20155) and Chen et al. (2018), we conduct experiments on a dataset crawled from the same several open-source projects as them and report results in the Table 3
The Naive method is directly copying the source code the translation result. PBSMT is short for phrase-base based statistical machine translation (Koehn et al., (003). and has been exploited in previous works (Nguyen et al. 2013) Karaivanov et al. 2014). As for the Transformer,
the
same number of layers and hidden size as pre-trained models. To leverage the pre- trained models for translation, we initialize the encoder with pre-trained models and ran- domly initialize parameters of the decoder and the source-to- target attention. Results show that the models initialized with pre- trained models (i.e the second group) signifi- cantly outperform PBSMT and Transformer models. Among them, GraphCodeBERT achieves state-of-art performance, which demonstrates the effectiveness our model for code translation.
Java +C#
C# #a-+Java
Method
BLEU
Acc
BLEU
Acc
Naive
18.54
0.0
18.69
0.0
PBSMT
43.53
12.5
40.06
16.1
Transformer
55.84
33.0
50.47
37.9
RoBERTa (code)
77.46
56.1
71.99
57.9
CodeBERT
79.92
59.0
72.14
58.0
GraphCodeBERT
80.58
59.4
72.64
58.8
Table Results on code translation. GraphCodeBERT outperforms other models significantly (p < 0.05).
5.4 CODE REFNNEMENT
Code refinement aims to automatically fix bugs in the code, which can contribute to reducing the cost of bug-fixes. We the dataset released by Tufano al. (2019) and report results in the Table

---

Published as a conference paper at ICLR 2021
The Naive method directly copies the buggy code the refinement result. For the Transformer. use the same number of layers and hidden size as the pre-trained models. Same as the Section [.3
we initialize the encoder with pre-trained models and randomly initialize parameters of the decoder and the source-to-target attention. Then use the training data fine-tune the whole model. In the table, we see that the Transformer significantly out- performs LSTM. Results in the second group shows that pre-trained models out- perform Transformer models further, and GraphCodeBERT achieves better per- formance than other pre-trained models on both datasets, which shows leveraging code structure information are helpful to the task of code refinement.
small
medium
Method
BLEU
Acc
BLEU
Acc
Naive
78.06
0.0
90,91
0.0
LSTM
76.76
10.0
72.08
2.5
Transformer
77.21
14.7
89.25
3.7
RoBERTa (code)
77.30
15.9
90.07
4.1
CodeBERT
77.42
16.4
91.07
5.2
GraphCodeBERT
80.02
17.3
91.31
9.1
Table 4: Results on code refinement.
5.5 MODEL ANALYSIS
Ablation Study We conduct ablation study on the task of natural language code search under- stand various components in our approach impact overall performance. We remove two pre-training tasks and data flow, respectively, to analyze their contribution. Table [ables]shhoss shows that the overall perfor- mance drops from 71.3% to 70.3%~~70.79 when removing Node ATignment and Edge Prediction pre-training tasks, respectively, which reveals the importance of two structure-aware pre-training tasks, After ablating the data flow totally, we can see that the performance drops from 71.3% to 69.3%, which means leveraging data flow to learn code representation could improve GraphCodeBERT.
Methods
Ruby
Javascript
Go
Python
Java
Php
Overall
GraphCodeBBRT
0.703
0.644
0.897
0.692
0.691
0.649
0.713
-woo EdgePred
0.701
0,632
0.894
0,687
0.688
0.640
0.707
wwo NodeAlign
0.685
0.635
0.887
0,682
0.690
0.640
0,703
-w/o Data Flow
0.679
0.620
0.882
0.672
0.676
0.628
0.693
Table 5: Ablation study on natural language code search
Node-vs. Token-level Attention Table shows how frequently special token [C'LS] that is used to calculate probability of correct candidate attends to code tokens (Codes) and variables (Nodes). We see that although the number of nodes account for 5%.200, attentions over nodes overwhelm node/code ratio (around 10% to 32%) across all programming languages. The results indicate that data flow plays an important role in code understanding process and the model pays more attention to nodes in data flow than code tokens.
Ruby
Javascript
Go
Python
Java
Php
Codes/Nodes
90. 1/9.9
94.6/5.4
95.01 95.0/5.03
80.6/19.4
93.2/6.8 87.8/12.2
87.5/12.5 79.4/20.6
[CLS]] -Codes/Nodes
82.3/17.7
89.7/10.3
91.0/9.0
67.7/32.3
Table Attention distribution (%) between code tokens (codes) and variables (nodes) across different programming language on natural language code search test sets. The first row is the ratio of the number of code tokens to nodes, and the second row is attention distribution of (CLSS) token.
Comparison between AST and Data Flow Figure 4 shows MRR score with respect to input sequence length the validation dataset of Ruby programming language for the task of code search. AST Pre-order Traversal regards AST as a sequence by linearizing all AST nodes us- ing pre-order traversal algorithm. AST Subtree Masking regards AST as tree and introduce subtree masking (Nguyen et al. 2019) for self-attention of the Transformer. In subtree masking, each node-query in AST attends only to its own subtree descendants, and each leaf-query only attends to leaves of AST. Transformer has self-attention component with (n²) time and memory complexity where n is the input sequence length, and thus is not efficient to scale to long inputs.
8

---

Published as a conference paper at ICLR 2021
We observe that injecting AST even hurts the performance when the sequence length is short (e.g. shorter than 128), while Graph- CodeBERT consistently brings performance boost on varying se- quence length and obtains better MRR score than AST-based meth- ods. The main reason that data flow is less complex and the num- ber of nodes account for 5% 20% (see 6), which does not bring an unnecessarily deep hierar- chy of AST and makes the model more accurate and efficient.
0.74
0.72
0.50
128
255
Sequence Length
Figure 4: MRR score on the validation dataset of Ruby for code search with varying length of input sequence.
Case Study We also give understanding process. Given a source code and a comment, we use GraphCodeBBRRT with and without data flow to predict whether the comment correctly describes the source code. Results are given in Figure We can see that both models make correct prediction in the original example, where the threshold 0.5 (left panel). To study the code understanding ability of models, we change the source code (center panel) and the comment (right panel), respectively. Although we make a small change on the source code (refurn a + return b) and the comment (sum value - mean value), completely different and corresponding gold
study demonstrate that data flow would enhance the code
the semantic of the source code and the comment labels change from to As we can see in the figure, GraphCodeBERT without using data flow fails these tests and still outputs high probability for negative examples. After leveraging data flow, GraphCodeBERT better understands the semantic of source code and makes correct predictions on all tests, which demonstrates that data flow could improve the code understanding ability of the model.
Unchanged
Code: return return b
NL: sum value mean value
Return sum value of an array
Return sum value an array
Return mean value an array
Code:
Code: import numpy
Code:
import numpy as nр def f(array):
import numpy as nр def f(array):
Input
def f(array):
aFnp. -=mp..um(arrra) banp Demp.mmen(arrray return
@=mp.sum(array)
g=mp.sum(array)
s=mp.mean(array)
b=np. (aroma)) return
return b
Label
0
GraphCodeBERT: 0.6563 (1)
GraphCodeBERT: 0.4815 (0)
GraphCodeBERT: 0.2884 (0)
Prediction
GraphCodeBERT: 0.8728 (1) (Wlo Data Flow)
GraphCodeBERT: 0.8808 (1) (wlo Data Flow)
GraphCodeBERT: 0.9048 (1)
(wlo Data Flow)
Figure 5: We take comment and source code the input (first row), and use GraphCodeBERT with and without data flow to predict the probability of the source code matching the comment (third row). The label is 1 if the comment correctly describes the source code otherwise O (second row).
CONCLUSION
In this paper, we present GraphCodeBERT that leverages data flow learn code representation. To the best of our knowledge, this is the first pre-trained model that considers code structure for pre-training code representations. that GERRR achieves state-of-the-art performance on four code-related downstream tasks, including code search, clone detection, code translation and code refinement. Further analysis shows that code structure and newly introduced pre-trat pre-training tasks boost the performance. Additionally, study in the task of code search shows that applying data flow in the pre- pre-trained trainedie model improves code understanding.
introduce two structure-aware pre-training tasks and show

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
Daya Guo and Jian Yin are supported by the Research Foundation of Science and Technology Plan Project in Guangdong Province (2017B03030800).
REFERENCES
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations, 2018.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. arXiv preprint 1808.0110 2018.
Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. arXiv, pp. arXiv-1910, 2019.
Mare Brockschmidt, Miltiadis Allamanis, Alexander L Gaunt, and Oleksandr Polozov. Generative code modeling with graphs. arXiv preprint (rxxiv:8005.04900 2018.
Luca Buratti, Saurabh Pujar, Mihacla Bornaa, Scott McCarley, Yunhui Zheng, Gaetano Rossiello, Alessandro Morari, Jim Laredo, Veronika Thost, Yufan Zhuang, et al. Exploring software natural- ness throughneural language models. arXiv preprint arXiv:2006. /2641, 2020.
Xinyun Chen, Chang I Liu, and Dawn Song. Tree-to-tree neural networks for program translation. In Advances in neural information processing systems, pp. 2547-2557, 2018
Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry, pp. 253 262. 2004.
Jacob Devlin, Ming-Wor Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.0811 08155, 2020.
Daya Guo, Duyu Tang, Nan Duan, M. Zhou, and Jian Yin. Pallo-to-acction: Conversational question answering over large-scale scale knowledge base. In NeurlPS, 2018.
Daya Guo, Duyu Tang, Nan Duan, M. Zhou, and Jian Yin. Coupling retrieval and meta- leta-learning for context-dependent semantic parsing. ArXiv, abs/1906.00 108, 2019.
Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In International Conference on Learning Representations, 2019.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC), pp. 200-20010. IEEE, 2018.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt, Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint 2019.
I Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. Deckard: Scalable and accurate tree-based detection of code clones. In 29th International Conference on Software Engineering (ICSE'07), pp. 96-105. IEEE, 2007.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Pre-trained contextual embedding of source code. arXiv preprint arXiv:2001.00059 2019,
Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections Programming & Software, pp. 173-184, 2014.
10

---

Published as a conference paper at ICLR 2021
Rafael-Michael Karampatsis and Charles Sutton. Scelmo: Source code embeddings from language models. arXiv preprint arXiv:2004.132 2020.
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. arXiv preprint vrXiv:2003.13844 2020.
Philipp Koehn, Franz: J Och, and Daniel Marcu. Statistical phrase-based translation. Technical report, UNIVERSITY OF SOUTHERN CALIFORNIA MARINA DEL REY INFORMATION SCIENCES INST, 2003.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining, arXiv preprint arXiv:1901.07291, 2019.
Jian Li, Yue Wang, Michael R Lyu, and Irwin King. Code completion with neural attention and pointer networks. arXiv preprint arXiv: 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1td.:1997.11660 2019.
Anh Tuan Nguyen and Tien N Nguyen. Graph-based based statistical language model for code. In 2015 IEEF/ACM 37th IEEE International Conference on Software Engineering, volume pp. 858-868. IEEE, 2015.
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 651 1654,, 2013.
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Divide-and-conquer approach for multi-phase statistical migration for source code (t). In 2015 30th JEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 585-596. IEEE, 2015.
Xuan-Phi Nguyen, Shafiq Joty, Steven Hoi, and Richard Socher. Tree-structured attention with hierarchical accumulation. In International Conference Learning Representations, 2019.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint 1rXiv:1802.0555 05365, 2018.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation and semantic parsing. arXiv preprint XXXvv/7704005333 2017.
Alec Radford, Karthik Narasimhan. Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre- pre-training. URL https://s3-us-wett amazonaws. com/openai- understanding paper. pdf. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text 10683, 2019.
transformer. arXiv preprint
Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, pp. 476-480. IEEE, 2014.
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: Code generation using transformer. arXiv preprint arXiv:2005.0888 080225, 2020.
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. An empirical study learning bug-fixim patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1-29, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
11

---

Published as a conference paper at ICLR 2021
Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Detecting code clones with graph neural networkand flow-augmented abstract syntax tree. arXiv preprint arXiv:2002.08653, 2020.
Huihui Wei and Ming Li. Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code. In IJCAI, pp. 3034 3044-0040, 2017.
Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. Deep learning code fragments for code clone detection. In 2016 3 IEEE/ACM International Conference Automated Software Engineering (ASE), pp. 87-98. IEEE, 2016.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V
Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019.
Pengcheng Yin and Graham Neubig. A syntactic neural model for gemeral-purposs code generation. In The 55th Annual Meeting of the Association for Computational Linguisties (ACL), Vancouver, Canada, July 2017. Utthttps: /ttp///aavvv////a//ai//eeeeaaaaa org/ abs 01 01696
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. A novel neural source code representation based on abstract syntax tree. In 20/9 IEEE/ACM 4 International Conference on Software Engineering (ICSE), pp. 783-794. IEEE, 2019.
A PRE-TRAINING DETAILS
GrapRCodeBERT includes 12 layers Transformer with 768 dimensional hidden states and 12 attention heads. For fair comparison, we use the same dataset as CodeBERT (Feng et al.) 2020) to pretrain our model. The dataset is the CodeSearchNet datase (usain al. 2019). which includes 2.3M functions with document pairs for six programming languages. We train the model on two DGX-2 machines, each having 16 NVIDIA Tesla V100 with 32GB memory. We set the max length of sequences and nodes as 512 and 128, respectively. We use the Adam optimizer to update model parameters with 1.024 batch size and 2e-4 learning rate. To accelerate the training process, we adopt the parameters of CodeBERT released by Feng al. (2020) initialize the model. The model is trained with 200K batches and costs about 83 hours.
At each iteration, we alternate EdgePred and NodeAlign objectives in combination with MLM to pre-train the model. And we follow Lample & Conneau (2019) to sample each batch from the same programming language according
multinomial distribution with probabilities {q: J= i.....N: N: where n, is number of examples for i-th programming language and a=0.7. Sampling with this distribution could alleviates the bias towards high-resource languages.
pf MN
n2
with P.
(9)
N
P
ΣN 71
B NATURAL LANGUAGE CODE SEARCH
Given a natural language as the input, code search aims to find the most semantically related code from a collection of candidate codes. We conduct experiments on the CodeSearchNet code corpus (Husain et al. 2019) and follow Husain et al. (2019) to take the first paragraph of the documentation as the query for the corresponding function. However, we observe that some queries contain content unrelated to the code, such as a link "http://.."" that refers to external resources. Therefore, we filter following examples to improve the quality of the dataset.
(I) Examples whose code could not be parsed into abstract syntax tree.
(2) Examples whose query tokens number is shorter than 3 or larger than 256.
(3) Examples whose query contains special tokens such as "http://".
(4) Examples whose query is empty not written in English.
12

---

Published as a conference paper at ICLR 2021
Different from the setting of Husain al. (2019), the answer of each query is retrieved from the whole development and testing code corpus instead of 1,000 candidate codes. We list data statistics about the filtered dataset in Table
Code Search
Training examples
Dev queries
Testing queries
Candidate codes
167,288
7,325
8, 8,122
28,120 120
Java
164,923
5,183
10,955
40,347
JavaScript
58,025
3,885
3,291
13,981
PHP
241,241
12,982
14,014
52,660
Python
251,820
13,914
14,918
43,827
Ruby
24,927
1,400
1,261
4,360
Table 7: Data statisties about the filtered dataset. For each query in the development and testing sets, the answer is retrieved from the whole candidate codes (i.e. the last row).
We use GraphCodeBERT separately encode query and source code with data flow, and calculate inner product of their representations of the special token [C'LS] as relevance scores to rank candidate codes. In the fine-turning step, we set the learning rate as 2e-5, the batch size as 32, the max sequence length of queries and codes as 128 and 256, and the max number of nodes as 64. We use the Adam optimizer to update model parameters and perform early stopping on the development set.
We also report the results using the same setting of Husain al. 2019) in Table 8 In this setting, models are required retrieve an answer for query from 1000 candidates. The results show that GraphCodeBERT also achieves the state-of-the-arr performance.
model
Ruby
Javascript
Go
Python
Java
Php
Overall
NBow
0.429
0.461
0.641
0.581
0.514
0.484
0.518
CNN
0.245
0.352
0.627
0.571
0.527
0.529
0.475
BiRNN
0.084
0.153
0.452
0.321
0.287
0,251
0.258
selfAtt
0,365
0.451
0.681
0.692
0,587
0.601
0.563
RoBERTa
0.625
0.606
0.820
0.809
0.666
0.658
0.697
RoBERTa (code)
0.661
0.640
0.819
0.844
0.721
0.671
0.726
CodeBERT
0.693
0.706
0.840
0.869
0.748
0.706
0.760
GraphCodeBERT
0.732
0.711
0.841
0.879
0.757
0.725
0.774
Table 8: Results natural language code search using the setting of Husain al. (2019)
C CODE CLONE DETECTION
Code clone detection aims measure the similarity between two code fragments. We Big- CloneBench dataset (Svajlenko et al.) 2014). which contains over 6,000,000 true clone pairs and 260,000 false clone pairs from 10 different functionalities. We follow the settings in Wei & Li (2017). discarding code fragments without any tagged true and false clone pairs and using 9.134 remaining code fragments. Finally, the dataset provided by Wang et al. (020) includes 901,724/416, 6.2441 16,328 examples for vaaiiing/alldatiin/neesss
We treat the task as binary classification to fine-tur tueetune Graph- CodeBERT, where we use source code and data flow the input. The probability of true clone is calculated by dot product from the representation of [CT.S). In the fine-turning step, we set the learning rate as 2e-5, the batch size as 16, the max sequence length as 512 the max number of nodes as 128. We use the Adam optimizer to update model parameters and tune ppper-parametess and perform early stopping on the development set.
We give case of the GraphCodeBERT output for this task in Figure 6 In this example, two Java source codes both download content from a given URL and convert the type of the content into string type. Therefore, two codes semantically similar since they output similar results when given the same input. As we can see, our model gives high score for this case and the pair classified as true clone pair.
13

---

Published as a conference paper at ICLR 2021
Input: Two source codes
Output: Semantically similar (score: 0.983)
profected String downooeiURLoossiinggur unij throws IOException
public static String Tetchlnn(String urittring)
BufferedReader in new BufferedFRederr1ew
try BufferedReader reader BufferndRRoodetteew
ImpusSiremmReadeturr.o SrpprdrrrrrtopenSSrmmfj.i
URL uri new URL(urSString):
SurngBuffer sb StingSSuffer1000 1024);
String str;
while ((str readLine)))) null)
String lnne null;
appenndsstr):
Stringuulder builder new StrinBuiddrr);:
while (fine rew.dderedddiiieieiee null) builder appenddline):
in.close():
return sb toString]:
buikder tnString('
catch (MalforredURRLEe RRLee e)
catch (IOExeeption
retum
Figure 6: A case of GraphCodeBERT output for the code clone detection task.
D CODE TRANSLATION
Code translation aims migrate legacy software from one programming language platform another. We conduct experiments on a dataset crawled from the same several open- projects as Nguyen et (2015) and Chen al. (2018), i.e. Lucene PO JGi and Antle"] We do not use Itex and ITS as they do because of the license problem. Those projects have both Java and C# implementation. We pair the methods the two languages based on their file names and method names. After removing duplication and methods with null function body, the total number of method pairs 11,800, and we split 500 pairs from them as the development set and another 1,000 pairs for test. To demonstrate the effectiveness of GraphCodeBERI on the task of code translation, various pre-trained models as encoders and stay hyperparameters consistent. We set the learning rate le-4, the batch size 32, the max sequence length as 256 and the max number of nodes 64. We use the Adam optimizer to update model parameters and tune hyper-parameteit and perform early stopping on the development
adopt
We give case of the GraphCodeBERT output for this task in Figure In this example, the model successfully translates piece of Java code into its C# version. The differences include the type name (from "boolean" to "bool") and the usage of getting a string value of a bool variable (from "String. valueOf((" "b.ToString(")..
Input: A Java method
Output: Its C# version
public void print(boolean b)
public void print(bool b)
{
print(tting.valueef(b)})
print(b.TooString)))
}
Figure A of GraphCodeBERT output for the code translation task.
http://luceee.apacchh.ooo
"http://poi.ppache.ory
"https:/ggitubb.cm
'https//giihhub.oom/aa
'http://sourceforge..m htwp://reeeeeeeeeeeeppccsseeeede
14

---

Published as a conference paper at ICLR 2021
E CODE REFINEMENT
Code refinement aims to automatically fix bugs in the code. We use the dataset released by Tufano al. 22019). The source buggy Java functions while the target the according fixed ones. Almost all the names of variables and custom methods are normalized. The dataset contains two subsets based on the code length. For the small dataset, the numbers training, development and test samples are 46,680, 5,835 and 5,835. For the medium dataset, the numbers are 52,364, 6,545 and Transformer model to conduct the experiments. In the
6,545. We also use the peuuueccett-sequeeee fine-tuning step, adopt various pre-trained models encoders. We set the learning rate as 1e-4, the batch size as 32, the max sequence length as 256 and the max number of nodes as 64. We use the Adam optimizer update model parameters and perform early stopping on the development set.
We give two cases of the GraphCodeBERT output for this task in Figure 8 In the first example, the model successfully fixes the operation bug (from "*" to "+") to match the function name "add". In the second case, the source function and type names are normalized. The return type of this function is "void" but the buggy code gives return value. Our model successfully removes the "return" word so that the return type of the function matches its declaration.
Input: buggy Java method
Output: The fixed one
public int add int a, in:
public int add int a, b)
return
return a b;;
public void METHOD_1 (TYPE_1 c)
public void METHOD_1 (TYPE_1 C.)
return VAR_1 remove (c);
VAR_1 remove (c);
Figure 8: Two cases of GraphCodeBERT output for the code refinement task.
F CASE STUDY
F.1 NATURAL LANGUAGE CODE SEARCH
We give a case study to illustrate retrieved results by GraphCodeBERT on the natural language code search task, with comparison to CodeBERT and RoBERTa (code) models. Two examples are given in Figure 9 and we can see that GraphCodeBERT successfully retrieves correct source codes for given queries both examples. As Graph-CodeBERT better understand the complicated expression "[(k, v) for k, v self.teem() if not self.EMPTY]" by leveraging dependency relation among variables in data flow graph. In the second case, the terminology "% Y-% %dd in Python program language is a format of date time. BERRT and Code both successfully search the correct function. Compared with RoBERTa (code), the second case shows that utilizing natural language descriptions for pre-training helps models do better semantic matching between source codes and queries on the code search task.
see in the first case, incorporating data flow will help
F.2 CODE CLONE DETECTION
We give case study compare GraphCodeBERT with CodeBERT and RoBERTa (code) models on code clone detection task. An example is shown in Figure 10 The first source code is to return the HTML content from a given URL, while the second source code is return the last www.ttgggoogllcode.coom
line from fixed URL similar due to their different outputs. Data flow could help GraphCodeBERT better understand that the return value "pageHTML" in first source code comes from "pagel[TML..ppenn((nne) and the
Their semantics are not
pageHTTML.appen((()rrir """""" instead of "buffered www.eeepaggeHHMM..oS...n.... return value "version" in the second source code comes from "version inputLine" "version null;". Although two source codes are highly overlapped (marked in yellow), GraphCodeBERT successfully predict the gold label compared with other models without data flow.
15

---

Published as a conference paper at ICLR 2021
Case
Case
Query: Retum copy instance, cnniting cutries EMPTY
Query: parsing
Souree Cade:
Gold Source Code: del parve daiass)
defined, ltemssselff:
return self clase
ty:
[ik, for K selEltems)) Lt'v is nos seCEMPTY), eape
return datetime (n))))) intef557J) intge 10j9) exeept ValucFrooe: datetime strptiess. dan))
Scarch Results (Top.)
Search Results (Topl)
GraphCudeBEKT defined, Herrsaeiffy return self, class ( [0k, for k, selfittems)) if'v not seEEEMPTY] [V], is empoy-Talse
GraphCodeeMERT def _dauuss:
try: retura datetime (n)) imtsf5/]], intoe::
except ValueEnooe
retura datetime datetieettteteiiieeeee suptimess, "Mdd NB "S""),
CodeBERT:
copy(sal):
CodeBEETT:
contrat Cl.Comestt)
daturss:
for Nem in dir(dd)):
try:
itmm00 1- and irem not ('opyy. 'wiite, heatens')
retura datetime dater inns(:4), imtsf5/]], imfe(:
sctattfcoeteet, tien geuttifeelf, Heema)
except ValueEnno:
retura datetime datetine supiimesss....ss "Aid NB daaned)))))
(code]:
copy(self)
lolf dusi)
parseiself, cancnialiizrraiael
s.popseeff pleyy)
return self.request "panse" jssn-["JoblCCc "'JobIICL"" hel, "Canoricalizee: conotiicalaae,, metiod- "post", allow redirects- www.:/ieett--teeejjeeeee.. Trackjsout)
seturn self class (*))
Figure 9: Two examples code search task and retrieved results from different models.
Sourre
Source
private Sirin gStrriii pegeURL, String encoding, String dirPutt) drross 10Uxception
peivase state String getVersion))
debug print ""
StingBuilder PageHTML SeingBulldrr);
String veniua sull;
Hipt RI Cunnertinn cornection rull;
String vemion,
'Iapp:/kmiig gouglssnl combonbbanbleennnn
new URL
connection (HiIUUL Comecciaa) open Connection
connection conectlonseRRqqeeee wew.seeshhopprtty...... hoppeyyyuueeeeee User Agent" "MSIE 7.0% comaaction comsec)).
URL Connection coo openop conectionn:
BullsveRRenaeer in new BultnnlBBenkannn
Warfensditedden be new indfrrediiendettee
ImponSimemReedeeeoo getlnputStrrmm()) String inpill inci
InpueSn commmeeaeeeeeeeeeiiiii iepuannneaee' mendingl)
String
(imputtinee (n.reaci.met)) mulll verson inputLine:
(fline inntl mill'
pogeETML. apppet..pppltttee Back
basch EEsception exll
|igHTMM apper""l-spppd"" appenk Yew"):
version mall;
cands (Eseppion
retum version,
finally:
commetiio.dissocmssttte
Gold Label: No senantically sinilbe
(ddiPat sull)
Fil: fic FilddirPahh)
Predintion: CrapkcodcBERTE No serianticall sinika
Wilee WeieerelWWeerr new BullesaWWitteeoee
File FieWiter(Filjjs
CodeBERT semoniically smilar
buffenddwrmer closel:
PigeHTMI ssStiin(:
RatIEITa (endajs
amooiiallly emilaar
Figure 10: An examples code clone detection task and model prediction from different models. Overlapped code snippets between two source codes are marked yellow.
F.3 CODE TRANSLATION AND CODE REFINEMENT
We give a case study to compare GraphCodeBERT with Transformer without using data flow on code generation tasks, including code translation and code refinement. We list three cases in Table 9 and Table 10 respectively. [src] represents the source input, [ref] represents the reference, [sys] represents Transformer without data flow and [ours] represents GraphCodeBERT. We can see that the Transformer ([sys]) baseline makes several mistakes, including repeating tokens, logic errors and syntax errors, while GraphCodeBERT ([ours]) encoder could improve the generation.
16

---

Published as a conference paper at ICLR 2021
Cascl: Transformer outputs repcating tokens
[src]
public static final WeightedTerm[] getTerms(Query ( ge(Termss(uerry.aaase))
rrtt]
public static Weighted Get'Terms( Qunnry query))return Gerrerms(query. false);
|sys]
public static WeightedTTer)) GetTerms((uuen query)) Gefserms(false, new static
static static static static static WeightTerms):
[ours]
public static WeightedTermi] GetTerms (eurrs query) (return GetTerms(query. false);)
Case2: Transformer oulputs codes with severe logic and syntax errors
[src]
public long skip(Iong ni/int s (int) Math min(aviilabll()) Math.max() n)):ptr += suetum s:}
[ref]]
public override long Skip(long n) [ini s = (ini)Math. Min(Availablee), Math. Max(O,, n));
ptr += streturn s:}
[sys]
public override long Skip(long n) [int = Math.Min(n) = 0 Mith.Min(00.0::
Math.Min(n, s.Leengh):retuum s:}
[ours]
public override long Skip(long (int s (int)Math (ivAAvallaabl.. Math. Max(O, n));
ptr += s,return s:}
Case3: Transformer uses the wrong variable as a parameter.
[src]
public UnbufferedCChaaSteaammm bufferSize) 0,data new int int[bu]ferSize]:)
[rf]
public TinbuficeedCharStreemn bufferSize) n 0,data new ini bi
[sys]
bufferSize) data new int[data]:] public UnbufferedCCharStreamll bufferSize) O,data new int int(bufferSizel:)
public
[ours]
Table 9: Three examples that translate from Java to C# programming language on code translation
task. [src] represents the source input, [ref] represents the reference, [sys] represents Transformer without data flow and [ours] represents GraphCodeBERT.
Casel: Transformer adds redundant parameters (android view. View view)
[src]
public void METHOD_ 1 O android. Intent VAR_1 = new android.content.. content.
(VAR_2) ; METHOD_2 2 VAR_1. .0) weatroid.coneent.Ineeeeee
new android.comtent. Intent
this, , VAR class) ); METHOD (i):) }
[rcf]
public void METHOD_ O { android..coter content..nnentt VAR_I = new wwwdroid.contett.mmgg Intent
(VAR_2) ;; METHOD_2 ( VAR_1. }
[sys]
public void METHOD.1 ( android. view. View view { wndroid.content.o Intent VAR.I
new android.content. content.Intea (AR.2) ) METHOD.2 2 (VAR_.. .
[ours]
public void METHON 1 {){ cow.oit...otttttttteee.. VAR_ = new www.rridd.coneet..ttt.....
(VAR.2) ): METHOD _2 ( \VAR1. :)
Casc2: Transformer outputs codes with severe logic irrelevant codes
[src] [ref]
javaicaava.uti java.util. wtt..uuii..aare Date METHOD _1 O { return VAR_1 METHOD. 1 O. METHOD 2):
public Date METHOD_I O ( i ( [VAR_ METHOD. I () != null)
{ return VAR_1 . METHOD_I O. METHOD 2 () else { return null }
[sys]
public Java.util.Date METHOD_ O ( == null)
{ return new java.util. ():) return VAR_I. METHOD_ O (O:
[ours]
public java. Date METHOD O { ( (VA.I. METHOD ()) 1= null
{ return VAR_1. METHOD_I O. METHOD 2 ()) } else { return null:
Case3: Transformer makes no change
[src]
public Dava.lang.Strime METHOD 1 (TYPE VAR 1) { (if( VAR null) return null; ; return VAR METHOD O.. getText )
[ref]
public java.lang String METHOD_ (TYPE_] VAR_1) { return VAR_1 METHOD_2 ().
getText ():}. }
[sys]
public java.lang. String METHOD (TYPE VAR.I) { ( VAR_1 null) ) return null:
return VAR_1. METHOD_2 O. get'ext 0 ();)
[ours]
public wwvallang.Strnng METHOD_I (TYPE_1 VAR.I) return VAR_I. METHOD.2(
getText )
Table 10: Three examples code refinement task. [src] represents the source input, [ref] represents the reference, [sys] represents Transformer without data flow and [ours] represents GraPHCodeBERTT
17

---

Published as a conference paper at ICLR 2021
G ERROR ANALYSIS
We also conduct error analysis and summary two main classes of errors for both code understanding and generation tasks.
Figure [1 gives three error cases of GraphCodeBERT on the natural language code search task. We observe that GRRpRCodeBERR mainly fails to retrieve those source code that involves functions of the library like "tf" (Tensorflow) in the first case and GcoggeCloudStorageeoooo case. It's difficult for GraphCodeBERR to understand meanings of APIs like "tf.io.read_fib" and "tf. meded decode image" without relevant information. A potential direction to mitigate the problem to incorporate definitions of the library. The other major problem that there are some termi- nologies like "unistr" in the query (corresponding to "decode(" (orrdde(utf-8)))) ytf-8")" in Python code) in third case. Incorporating more text-code pairs for pre-training might alleviate this problem.
in the second
As for the code generation task, Table 11 shows two cases of GraphCodeBERT on the code translation task. We find that the major problems include semantic errors like identifiers from nowhere in the first case and syntax errors like missing "]" symbol before 'return n" in the second case. This problem might be mitigated by incorporating a dedicated decoder that takes into account grammar of programming languages and different generation paradigm like generating sequence of production
rules (Yin & Neubig, 2017 7
Guo et al. 2018 2019) in a context-free grammar manner.
Cave
Case Pytnasi. Czeveri
Query Retun image loeae.
Querr: koads Gold Searce Codi: orf Jinn Feeelay
food Souree Cade:
def mastillquahi
flacttayeeh
haattr fara, kaoodd's
In- triug deodl_ jnageinn dovedeCCHANNEL))
Elsna
dopeliz, d.Ooal))
doss dro decadeearter
remm
exege LeioodeDevoiiinee...
ast inngeted, floe, key dandon
'laperme vald UTFS for ISUN dam, gat for (5dda)))
teciltruul interre lonne" adtvelee, fox key cher)
soum rupllaidal lesa: hanont, slanyfl, tenees)
GraphCCadetEMi:
del pareo unateet
Case
ne joee -jor (oafitersine))
Oeery: Upleads the Google aboud strgge
data mry jact
Gold Suurce
beok GoogleTee Sioeppplll
if'aa dud isce dea shtakaa))
seegleslead aeorag odlignnkesholbbttngg..e
delouaic jo zdfakeppa. ppokl
der
becket, sitersofiitt.
inse
ttted schera - it. acherna uneree evert: acherso retan [ffhol Jnee__kan]]
CodeBEIE
peroppeetoonee local yahr
skeuge TORAGESS-F5)
und SaCss dergee stange) cettapkartienl par,, sereae part)
Figure 11: Error cases of GraphCodeBERT on the natural language code search.
Casel: semantic error - identifiers from nowhere.
[src]
public String toString( (return getKey()+ + ": + gatuel } public override string ToStringO(retur GetKey() + " GetValueO:)
[ref]
[ours]
public override string ToString( returm Name + +" GerValueO3)
Case2: syntax errors missing "Y" before "return n"))
[src]
public static int numNonnul[Ohjeet]] data) (int n 0;if (data null return for (Object o: data) {if o!=:uul) ) n++:; n+++) m:}
[ref]
data) n= Ozif (data == returr n:} foreach (object o in (if (o t= null)(n++:)
public static int NumNonnull (ull))n)++))rruumm |retum n;)
[ours]
public static int NumNonNNulloojeec
data) (int Ofif (data (retumm n:}
foreach (object o in data) (if (o != null) (al+)) n:}
Table 11: Error cases of GraPhCodeBERT on the code translation task. [src] represents the source input, [ref] represents the reference and [ours] represents GRRRhCodeBERT.
18


Reviews:
[{"review_id": "bj8YqWK2Wf2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nOverview:\nThe authors present Graph Code BERT, the first language model that leverages data flow to learn code representation. They use three objective functions: Masked Language Modeling, Edge Prediction, and Node Alignment. They claim their structure-aware pre-training can help improving performance on code-related downstream tasks, including code search, clone detection, code translation, and code refinement.\n\n\nReasons to accept:\n* The paper is well-written and easy-to-follow.\n\n* It is the first pre-trained model that leverages the data flow structure of code to learn code representation.\n\n* They propose two pre-training tasks for learning representation from source code and data flow. Edge Prediction is more interesting in my opinion.\n\n* They show improvement on four downstream tasks, even though some of them are marginal in my opinion compared to CodeBERT.\n\nReasons to reject:\n* This work is not the first work to conduct language model pretraining for code understanding applications. I will expect more comprehensive comparisons with related work. For example, Svyatkovskiy et al. (2020) propose GPT-C, which is actually a transformer architecture that can also use for both generation and classification tasks. Missing C-BERT (Buratti et al.) as well.\n\n* The 2.4M functions used for \"pre-training\" is not very persuasive. I think it is more reasonable to crawl existing code repositories for Pre-training. I do not see a particular reason because pre-training without natural language descriptions still makes sense.\n\nQuestions & Suggestions:\n* Do you find natural language descriptions useful in your experiments? Why if we discard it from pre-training?\n\n* What is the ratio of edge connection? 10%? 20%? The reason I am asking this is that when you do the randomly sampling 20% of nodes Vs in the data flow, how many node pairs you used to calculate the loss? Is the positive-negative ratio balanced? Please give more details about the training objectives.\n\n* Please give more prediction examples or qualitative analysis for each downstream tasks in the Appendix.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HQq2rKwSkFs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work address the pretraining over code and text. It proposes to leverage data flow as additional inputs, and add two structure aware pre-training tasks besides the masked token prediction task. The pretrained model is evaluated on four different tasks and outperforms the CodeBERT baselines as well as other pretrained models. Further analysis confirmed the benefits from the additional tasks and data flow input. \n\nStrength:\n\n1. The use of data flow as additional input and the proposal of structure aware tasks are well motivated adaptations of pre-training to code. \n\n2. The extensive experiments and comparisons supported the claim and additional ablation studies and analysis confirmed the benefits from the additional structure-aware tasks and the data flow representation. \n\n3. The comparison with syntactic structures (AST) is interesting and demonstrated the higher potential of semantic structures like data flow. This could be beneficial to other code related tasks. \n\n4. The paper is well written and easy to follow. \n\nWeakness:\n\n1. Since the proposed approach is adding extra inputs, and given the O(n^2) complexity of Transformer, it would help to add some information regarding the increase in compute cost, for example, additional flops or latency due to additional data flow inputs. \n\nRelated to the point above, although linearizing the data flow into sequence seems the simplest approach, it might be computationally inefficient given that the data flow graph is probably sparse, thus large number of attention position needs to masked. But maybe the data flow input is much smaller than the code input so that it is not adding too much overhead.  \n\nIt would also help to add more statistics about the inputs, for example, the average length of the code, the length of the linearized data flow, the length of the linearized AST, etc. \n\n2. Not so much a weakness, but it would help to show some examples, in appendix if the space is limited, where GraphCodeBERT improves over CodeBERT or other models without semantic structures, to give some more intuition. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EtePPHYyFEI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends CodeBERT to include elements of dataflow in\naddition to the comments and sequence of code tokens. Also, the\npretraining includes two tasks that are structure-aware: predicting\ndataflow edges and aligning dataflow nodes to code. The paper is well\nwritten, well motivated and the empirical evaluation is quite\nthorough. The performance compared to CodeBERT is around 1-2%\nadditional accuracy for a series of diverse tasks.\n\nDetailed comments:\n- In general the paper is well written. There were a few parts that\n  required further clarification for me. For example, it is not clear\n  how the variable names are represented textually (e.g. x1 - is it\n  really x1 or a different way of expressing this in the sequence of\n  tokens that is sent to the model). The architecture seems to me to\n  be pretty identical to BERT with the exception of the attention\n  matrix for the dataflow portion. Is this matrix kept constant\n  throughout training? Not clear to me if the dataflow graph is a\n  traditional dataflow (as in the one used in compiler analysis). It\n  appears to me that it's a slightly simplified version based on AST\n  and some idea of data flow. I would contrast and compare with\n  \"traditional\" dataflow graphs (that use SSA - single static\n  assignment form - for example). Not exactly clear what you mean by\n  \"neat\". I would find a different word. Neat sounds like \"pretty\" in\n  this context; I don't find it scientific. Do you mean less complex?\n  The Noe alignment task was not clear to me at all. Perhaps an\n  example would be helpful showing the actual task and what is\n  provided as input (I'm guessing the same thing as usual) and how the\n  prediction looks like.\n\n- I'm not sure there is a discussion on the experimental methodology\n  and hyperparameter tuning. Are the results presented some average\n  over several runs? If they are single runs, the difference in\n  accuracy for most task is within noise, I would think. Also, I\n  particularly find the third decimal a bit much for \"stretching\"\n  result improvements. I think the idea is simple, it gives some\n  limited performance improvement over CodeBERT but the model\n  architecture is overall similar so it is probably worth it", "extra_scores": {}, "extra_text": {}}, {"review_id": "2KLdo-CCc1p", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes GraphCodeBERT as a Transformer-based pretrained model for programming language that incorporates data flow information in the graph representation of variables in the code. The data flow graph encodes the structure of variables based on “where-the-value-comes-from” from the AST parse. The pretrained model is jointly trained on the code, the natural language comment of the code, and the data flow graph of the code. In addition to the Masked Language Modeling objective, two new pretraining objectives are proposed including predicting the edge of the data flow graph and predicting the alignment of variables between data flow graph and code. The graph-guided masked attention is used such that the attention can only occur if two variables have an edge in the data flow graph or there is an alignment between data flow graph and code. The experiments show that GraphCodeBERT can deliver improvements on Natural Language Code Search, Code Clone Detection, Code Translation, and Code Refinement.\n\nStrengths\n\n- The approach is well motivated, and I totally agree that we should consider the code structure for a pretrained code language model.\n\n- There are consistent, though small, improvements on all four tasks.\n\n- The case study and ablation study are very helpful to understand why GraphCodeSearch is better than other models with better representation of code semantics on the Code Search task.\n\nWeakness\n\n- This paper uses the data flow graph and claims this is better than AST because AST has an \"unnecessarily deep hierarchy\". However, as the data flow graph is extracted from AST, data flow contains only partial information of AST, especially the information of “where-the-value-comes-from”, and throws out other useful information such as function name, condition control, operator. It is not clear to me if this design choice is a good one. \n\n- The experiments should have a comprehensive comparison using AST and the data flow graph to answer the questions such as: What is the performance of using AST instead of the data flow graph? What information in AST is useful for which tasks? In addition to “where-the-value-comes-from” information, what other information shall we incorporate into a pretrained language model for code? Right now, there is only analysis on the code search task and there is not enough discussion about AST approaches.\n\n- The improvement of GraphCodeBERT over previous SOTA, especially CodeBERT, is marginal. There is no significance test in the results. On Code Clone Detection, the improvement is 0.6% F1. On code translation, the improvement is less than 1% for accuracy. The most improvement is on 5.1 Natural Language Code Search, but the data flow graph and the two newly introduced objectives are not relevant to natural language comment.\n\n- I don't think BLEU is a meaningful metric for code translation and code refinement, and it can be misleading. For example, in Table 4, BLEU is 90+ while the accuracy is less than 10%. The author can consider not reporting BLEU.\n\nQuestions\n-  Can you please give more details about how to use AST Preorder Traversal and AST Subtree Masking for code search? I am wondering if this is a good usage of AST on this task. Thanks!\n\nMinor\n- Notation W is used to denote both parameters in self-attention (Eq (1)) and the code comment. Consider using two different notations.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple extension to BERT-like pre-training for source code models, which allows incorporation of data flow information. This is a new way of incorporating code structural information into models, and it appears practical and effective. Reviewers are all in favor of accepting the paper.
Decision: Accept (Poster)