Language-Agnostic Representation Learning of Source Code from Structure and Context
Authors: Daniel Zügner
Tobias Kirschstein
Michele Catasta
Jure Leskovec
Stephan Günnemann
Venue: ICLR 2021 Poster (2021)
Keywords: machine learning for code
code summarization

Abstract: Source code (Context) and its parsed abstract syntax tree (AST; Structure) are two complementary representations of the same computer program. Traditionally, designers of machine learning models have relied predominantly either on Structure or Context. We propose a new model, which jointly learns on Context and Structure of source code. In contrast to previous approaches, our model uses only language-agnostic features, i.e., source code and features that can be computed directly from the AST. Besides obtaining state-of-the-art on monolingual code summarization on all five programming languages considered in this work, we propose the first multilingual code summarization model. We show that jointly training on non-parallel data from multiple programming languages improves results on all individual languages, where the strongest gains are on low-resource languages. Remarkably, multilingual training only from Context does not lead to the same improvements, highlighting the benefits of combining Structure and Context for representation learning on code.

Published as a conference paper at ICLR 2021
LANGUAGE-AGNOSTEE REPRESENTATION LEARNING OF SOURCE CODE FROM STRUCTURE AND CONTEXT
Daniel Zügner, Tobias Kirschstein
Michele Catasta
Technical University of Munich (zuegnerd, ki rrschto @ t.m.de
Stanford University
pirrohêcs sta.osrsstttooodddoooooo
Jure Leskovec
Stephan Günnemann
Stanford University
Technical University of Munich guennemann @in tum. de
jureêcs. stanford.
ABSTRACT
Source code (Context) and its parsed abstract syntax tree (AST; Structure) are two complementary representations of the same computer program. Tradition- ally, designers of machine learning models have relied predominantly either on Structure or Context. We propose a new model, which jointly learns on Context and Structure of source code. In contrast to previous approaches, model uses only language-agnostic features, i.e., source code and features that can be com- puted directly from the AST. Besides obtaining code summarization on all five programming languages considered this work, we propose the first multilingual code summarization model. We show that jointly training on non-parte non-parallel data from multiple programming languages improves re- sults all individual languages, where the strongest gains are on low-resource languages. Remarkably, multilingual training only from Context does not lead the same improvements, highlighting the benefits of combining Structure and Context for representation learning on code.
monolingual
INTRODUCTION
Machine learning for code an active and growing area of research which aims at building models that can learn semantically meaningful representations of programs. These embeddings can be used on downstream tasks, such as code generation, bug detection, or code summarization. We focus work on two complementary data representations of programs: the source code (referred to as Context in this work), and the abstract syntax tree (AST; referred to as Structure). Traditionally, researchers and practitioners have decided to predominantly leverage either Structure Context in their machine learning models. In this work, we show that jointly learning on Context and Structure improves representation learning on source code (see Fig.
The source code representation naturally lends itself to models from natural language process- ing (NLP), e.g., long short-term memory networks (Hochreiter Schmidhuber] 1997) (LSTM) or Transformers (Vaswani et al. 2017: RRadfodd et al.) 2019) et al. [2019; Yang et al. 2019 Shaw
al. 2018). On the other hand, models leveraging the structure representations are typically based
on graph neural networks (GNNs) (Kipf & Welling 2017 Xu et al. 2019, Veličković et al. 2018
You al. 2019 Hamilton al. 2017 et al. 2015 Kliepera et al. 2020 While the AST repre-
sentation makes the highly structured nature of source code explicit to the models, since most GNNs use the message-passing framework, their learned representations are inherently local and struggle to leverage long-range interactions.
Recently, Hellendoorn et al. (2020) have explored models that can leverage several representations, including both Structure and Context. Their Graph Relational Embedding Attention Transformer (GREAT) extends Shaw et al. (008)). which biases the self-attention computation in a localized way given the underlying graph. The language-speciff representations used by GREAT include combination of the data flow graph, control flow graph, syntactic edges (inspired by Allamanis et al. (2018)) etc. which require specialized pipelines and static analysis tools to be obtained.

---

Published as a conference paper at ICLR 2021
Source Code as Sequence of Tokens
Source Code as Abstract Syntax Tree
def get model condition:
hop
2 hope
train () [20 takens]
elae:
return model
hops
Context
Structure
Figure 1: Context and Structure both encapsulate valuable information about source code. In this realistic example, token and 4 are distant in the sequence of tokens (Contex(), but only 5 hops away when traversing the Abstract Syntax Tree (Structure). As such, method that relies only on the sequence of tokens could neglect the relationship between method name and its return variable. Conversely, token 1 and 2 showcase the opposite setting. Hence, unifying Structure and Context leads to a more powerful representation of source code.
We propose the CODE TRANSFORMER + which combines distances computed on Structure and Context in the self-attention operation. Tn contrast to the localized treatment via edges described above, we make the full Structure accessible to the model at each layer by computing pairwise distances on the AST, such as shortest path lengths. To this end, we draw inspiration from the XLNet architecture YYng et al,, 2019), which uses relative distances instead of absolute positions in the attention computation. Importantly, all our features are language-agnoste computed for any programming language based on the source code and AST.
i.e., can easily be
We use two datasets comprising 5 different programming languages in total, and evaluate the rep- resentations learned by our model on the task of code summarization, where the model predicts a method's name based on its body. Besides setting the state-of-the on all five languages for single- language training, we also train the first multilingual model for code summarization. This enabled by the fact that our model uses only language-agnoste features that casily be obtained for any programming language. Remarkably, training our model on multiple programming languages sub- stantially improves the performance Context does not lead to the same improvements, highlighting the benefits of combining Structure and Context for representation learning on code.
all languages. Moreover, multilingual training only from
RELATED WORK
Machine Learning for Code. Early research learned language models on raw text data, e.g., (Wang et al., [016)] Raychev et al.] 2014] Dam et al.] (2016), providing evidence for the naturalness assump- tion (Hindle et al. 2012). For example, Allamanis et al. (2015) learned distributed representations of variables and methods, finding that they were indeed able to encode common semantic properties from the regularities present in source code. Alon et al. (2019b) also found evidence of semantic arithmetic in their embedding space, dubbed code2vec. These representations- and their variants like (Mou et al. 2016)- can then used to predict sequences identifier sub-tokens (Allamanis al. (0155) or API calls (Acharya al. 2007 Nguyen et al.) 2017). They can be used as advanced auto-completion tools (Hindle et al.] 2012; Bhoopchand et al. 2016).. including for user-provided
tokens like Variable Names (Raychev al. 2014]] Allamanis et al. (0014). These useful for deobfuscating Android applications (Bichsel et al. 2016) for example.
Several works leverage structured graphical models for probabilistic models of source code, usually through parse trees (Maddison & Tarlow| [2044] Bielik et al. (0166). Unlike previous works where hand-crafted features were used as node features (Raychev et al. (014)) or explicit semantic edges (Allamanis et al. 2018), our work does not augment the existing syntactic relationships between the
I !Code www. dam1 in. tum wwwmm//ee//eeeeeeeeeeeee www.ddd-aaaaaormeeee..oomm demo at code-ttaansoormeerrr .org We use the term anguage-agnostte to highlight that our model does not rely on language-spest specific features (e.g., program analysis edges), thus facilitating multi-language training, as it is possible to generate unified AST representations for different programming languages.
2

---

Published as a conference paper at ICLR 2021
different elements enhance the predictive capabilities of the model. Other approaches (Alon al.
2018 Li et al. 2017) also leverage the AST structure, but linearize the graph by first traversing it.
Learning representations of structured languages. While models of language have dramatically improved in their ability to learn structure (syntax) and semantics from scratch, can be argued that directly providing the model with the underlying structure of the language
help with generaliza-
tion (Battagla et al., (018). managing long-ranging dependencies (Tai et al. 2015), or representing the compositional aspect of natural language (Socher et al. 2013). Notably, tree structures have shown promising results and inspired new architectures (Shen et al. 2019). including in the domain of source code (Fernandes et al. 2019), where the underlying syntax is directly available. Our work pursues this line of research, showing the benefits of explicitly integrating structural information as an inductive bias. [Siv & Quirk (2019) propose positional encodings for nodes on trees; however, their approach assumes regular trees, which is an unrealistic assumption when working with Ab- stract Syntax Trees, as an AST node can have arbitrarily many children, e.g., the arguments of function.
Graph Neural Networks. GNNs provide a powerful tool for machine learning on graphs, thanks to their ability to recursively incorporate information from neighboring nodes in the network (Battaglia
al. 2018). naturally capturing the graph structure simultaneously with the nodes' features. (Gori
et al. 2005 (Scrreeli et al.) (2008) are able to learn vector representations of nodes and graphs in
an end-to-end fashion, encoding structural and feature information in the embedding space. Under this model, GNNs have achieved sate-of-the-all performance across variety of tasks, such as node
classification (Kipf & Welling 2017 7 Hamilton ton et al., 2017] Kliepera et al. 2019a). link prediction
(Zhang & Chen 2018 Schlichtkrull al.) 2018), ) graph clustering (Defferrard al. 2016 Ying
ct al. 2018) or graph classification (Ying et al. 2018 Dai et al. 2016 Duvenaud et al.
2015)
INTEGRATING STRUCTURE AND CONTEXT IN THE CODE TRANSFORMER
self-attention is the core operation powering the Transformer. enables the model selectively focus on relevant parts of the input. The matrix form equation for attention with a single head is
QKQ
Attention (Q, K, V) = softmax
V.
dd
where Q, E N and V E Nxd. N is the number of input tokens, de the key dimension, and de the value dimension (typically we have de = de). The attention score of query Q; and key K, before softmax is
= E. W,EEE.
where E., E, E Bd are the d-dimensional embeddings of tokens and j. and Wg, W. E Rdexd are the query and key projection matrices, respectively.
Observe that Eq. 2 contains no assumption about potential structure in the input domain: in the attention operation we compute all dot products of query and key vectors equally, effectivvly viewing them as unordered sets of vectors. This means, however, that the model is oblivious to structured inputs (such text graphs) and therefore is unable distinguish, for example, variable name occurring as an argument and in the return statement of a method.
In NLP, it is common to bias Transformers towards sequential inputs by adding positional encodings to the token embeddings. These positional encodings are obtained by applying an encoding function IR + Rd to each token's position Pi. These positional encodings make the information about the sequence of tokens available to the model. Eq. (2) becomes:
Aij (E. + ((p))) W Wx(EE p(p))):
(3)
which factorizes into
Aij E W W.E.
E WT Wko(p)) + ((p)) W.T.EE + d(p)) T W NyTTaoli Wx)))))
(4)
(2) 4
(b) A
(c) A
(8) A
We can interpret the terms (d)-(d) as follows. (a) A is the contribution from the 'match' between the content embeddings of tokens and (b) A steers the attention towards certain positions based
3

---

Published as a conference paper at ICLR 2021
on the content of token i; (c) A. biases towards content embeddings based the position of token (d) A controls which positions should attend which other positions.
In our model, we adopt the formulation of Dai et al. (2019;; Yang et al. (2019). They modify Eq.
(4)
by replacing the absolute position encodings ((p)) with relative position encodings (((s-))):
W W.E, Ej E W W, ((r-))))))))))))))))))))))())( u && + D'W,o(rrs))
(5)
where ri-aj is the relative distance from token token in the sequence, u, v E Rd are learnable bias vectors, and W, is a key projection matrix for the relative distances. Besides fixing issues with absolute position encodings such ambiguity when processing enables native application of the powerful self-attention operation on domains such as graphs, where absolute coordinates are not available. We adopt the (non-trainable) sinusoidal encoding function proposed by (Vaswani et al., 2017) for all relations; see Appendix A.1 for details on the distance encoding function.
sentences
time, Eq. (
3.1 INTEGRATING SOURCE CODE AND AST REPRESENTATIONS OF PROGRAMS.
To enable the model to integrate information both the Context and Structure of programs, modify Eq. (5) to be able to incorporate multiple different relations. To this end, we use one key projection matrix W per relation s, and sum their contributions in the raw attention score. This enables CODE TRANSFORMER to combine information from multiple relations between tokens in the attention computation. Besides the token distance in the Context, include pairwise relations based on the AST as described in the following. See Fig. for a visualization of the Structure distances use.
Shortest path length. We include the number of hops required to reach node j starting from node i and vice versa.
shortest path length
Here, treat the AST an undirected graph, since otherwise most distances would be undefined: e.g., all other nodes in the AST would be unreachable from the leaves.
hops
22'
ancestor distance
PPR score node 5
Similar to the distance of two tokens on the source code sequence, the shortest- path length is a global distance. This makes the whole graph structure acces- sible to the model at each layer. In contrast, Hellendoorn et al. (2020) add bias terms to the attention computation only for edges (i.e. shortest-path distance of 1), which local operation that only exchanges information between immediate neighbors (similar to message passing in GNNs). The cquivalent localized operation on the source code sequence would be to treat the sequence as a chain graph and only compute attention terms for neighboring tokens, which in turn highlights the benefit of non-local attention operations.
sibling distance
Figure 2: Structure distances used by our model.
Ancestor distance. Since we treat the ASTs undirected for the computation of the shortest-pan -athh length, we lose the direction information of the edges. To avoid this, we also include the distance on the ordered set of ancestors and descendants of node in the AST (red arrow in Fig. 2). Again, we include number (vertical) hops to avoid locality in the attention computation. For example, node r = 2 for ggrand-childred
of and rj 2 in the other direction.
Sibling distance. The neighbor sets in graphs are typically considered to be unordered, but in an AST, the order of children encodes their order of occurrence in the source code. To avoid information loss when encoding the AST, we further include the distance on the ordered set of siblings {u,) of node, where again avoid locality by encoding the number of hops, i.e. Tei Tig -01 2.
and
Personalized PageRank (Page et al.) 1999) (PPR). PPR is well-studied proximity measure which has been shown to be very effective in learning with graphs (Klicpera et al.] 2019a Bojchevski al. 2020 PPR captures the local graph structure around a pair of nodes (1,7). B.g. if has
4

---

Published as a conference paper at ICLR 2021
Source Code as Sequence of Tokens
Code Transformer
Code Summarization Task
def get model ()
55 PPR(1.4) 0.27
if condition.
(11144
def [MASK]
train ()
(20 takens]
hops
else:
return model
Token distance encodings
AST distance encodings
Inpul encodings
Source Code as Abstract Syntax Tree
Attention Layer
DECODER
Attention
values
(0.9) train model
(0.5) fåt
ENCODER
valid
hops
encodings
Oulpul
(0.1) infer
(0.2) zezo zezo_grad grad
Figure 3: Left: Sequence (Context) and AST (Structure) representation of an input code snippet. Center: The CODE TRANSFORMER jointly leverages the sequence of tokens and the Abstract Syn- tax Tree to learn expressive representations of source code. In addition to the input token and node embeddings the model uses different distances between the tokens, e.g.. shortest paths on the AST or personalized PageRank, to reason about their relative positions. The output embeddings can be used for downstream tasks such as code summarization (right).
many neighbors, its PPR score for will be low when they are only few hops apart, which complements the purely hop-based distances described above.
Input embeddings to the model. To combine the Context and Structure information, assign each token in the sequence to an AST node by selecting the AST node whose range in the source containing the token. We concatenate the (sub-) token embeddings with
code is the shortest the embedding of the token's assigned AST node type as well as the token type returned by the tokenizer. That is, among all the internal nodes, in the sequence; however, the remaining internal nodes can used by the model since their presence affects the distances between the remaining AST nodes. See Appendices and A.4 for details.
as input only those corresponding to token
3.2 EFFICIENT RELATIVE ATTENTION COMPUTATION.
Naïvely, we need to compute and materialize a tensor dimension N N d to hold all pairwise relative position encodings ø (ri=+)) in Eq. (5) where N is the input length. This is prohibitive for fast GPU training. While for discrete distance values (e.g., sequence distance or shortest-path length on a graph) we only need to compute unique distance values occurring in the input, this does not gen- eralize to continuous distances such as PPR. Therefore, we propose a constant-time approximation of the relational attention computation by grouping the values into i < N bins. Since closer sam- ples are typically more relevant for a query sample, we increase the bin widths exponentially with experiments have found the CODE TRANSFORMER
growing distance values. Throughout be relatively insensitive to the number of bins; we thus set k = 32 in our experiments.
4 EXPERIMENTAL SETUP
Code summarization is one of the most popular tasks in machine learning for code. Given the body of a function, the task is to predict the function's name. As observed by Alln et al. (2019b) and Allamanis et al. (2016), this is a useful benchmark as method names in open-source projects tend to be precise and descriptive, and functions typically form complete logical units. See Fig. 3 (right) for a visual overview the task. We use two complementary representations of programs: the source code as sequence of tokens (Context) and the AST (Structure). As shown in Fig. 3 (left), tokens that are far away on the sequence may be very close on the AST and vice versa. In this task we make of the CODE TRANSFORMER 's ability jointly leverage both Structure and Context and show that
5

---

Published as a conference paper at ICLR 2021
it improves learning. Further, we show the benefit of using only language-agnoste features in our model by training the first multilingual model for code summarization.
Datasets. To highlight the benefit of only relying language-agnostic representations such as source code and abstract syntax trees, we evaluate on challenging datasets in four programming lan- guages introduced in the CodeSearchNet (CSN) Challenge (Husain et al. 2019): Python, Javascript, Go, and Ruby. Similar to Java-small, the datasets from CodeSearchNet have been carefully dedu- plicated by the creators avoid data leakage from the training set, e.g., via copy-and- paste code.
We further evaluate on Java-small (Allamanis et al. 2016), popular and challenging code summarization dataset. It contains 11 open-source Java projects. We use the split as in Alon al. (2019a), where of these projects are used for training, one for validation, and one for test. The dataset contains roughly 700K sam- ples (function definitions). Moreover, we also experi- ment with pre-training our model on Java-medium and Java-large (Alon et al., (2019)) before fine-tuning on Java-small, making sure to avoid leakage by removing the test and validation projects of Java-small from the pre- pre-training dataset. See Table for a summary of the datasets we use in this work.
Samples per partition Val.
Dataset
Train
Test
CSN- CSN-Python
412,178
23,107
22,176
CSN -av-ssrser
123,889
8,253
6,483
ISN-Ruby CSN-Go
48,791
2,209
2,279
317,832
14.242
14,291
Java-small
691,974
23,844
57,088
Table 1: Dataset statistics.
Preprocessing. Each token of the source code is split into subtokens respective to code naming conventions, i.e., Get.Trainiiinne aiggataa is converted to [get, training, data]. Following Alon et al. (2019a) we use at most six subtokens for the method names, truncating longer function names if necessary. In addition to the tokenized source code we produce an AST for each method using the open-source AST parser Semantic] We limit the vocabulary to subtokens with at least 100 occurrences in the training set, and only consider snippets with 512 or fewer tokens (after removing punctuation). We refer the reader the appendix for further details the data preprocessing.
Pointer network. We add a pointer network (Vinyals et al. 2015) (as described in Fernandes et al. ) to the decoders of all Transformer-
models. This enables them enhance their predic- tions by pointing at positions in the input sequence. For instance, when predicting the method name get_url, the model results for less frequent tokens, and even enables the model to predict tokens which are not in the vocabulary by pointing at their positions in the input.
point directly to occurrences of the variable url. This often improves
We compare with code2seq (Alon et al. 2019a), the Graph Relational Embedding At- tention Transformer (GREAT) (Hellendoorn al. 2020), and the BILLSMM-GNN-LLSM-PPOP model presented in Fernandes et al. (2019). Code2seq is a non-Trans -on-Transformer model and state of the art for code summarization using only AST information. GREAT using the framework presented in (Shaw et al. 2018) to bias the attention via edges. In the origi- nal formulation, GREAT additionally uses hand craffer language-specific edges such as dataflow, 'computed from', or 'next lexical use' edges, which require specialized preprocessing and static analysis tools. While this approach of leveraging language-specific features can certainly improve results specific tasks and programming languages, our goal is to have a flexible model that can be used on any programming language. Since the specialized preprocessing used by GREAT is pro- prietary and not public, we produce the results for GREAT using edges from the AST instead, i.e. it has access to the same information as our proposed model. Note that the preprocessing of Fernandes et al. (209)) is language specific, which why only compare with their results Java-small.
Baselines.
recent Transformer model
RESULTS
5.1
MONOLINGUAL CODE SUMMARIZATION
CSN dataset. First, we study the performance (measured by Fl score) of our model and the base- lines on the traditional setting, where training and evaluation are performed on a single programming language. The results are shown in the upper part of Table 2 The CODE TRANSFORMER (without
https://github..

---

Published as a conference paper at ICLR 2021
Python Rec.
Javascript Rec.
Ruby Rec.
Model
Prec.
F1
Prec.
F1
Prec.
F1
Prec.
Rec.
F1
code2seq
35.79
24.85
29.34
30.18
19.88
23.97
23.23
10.31
14.28
52.30
43.43
47.45
GREAT
35.07
31.59
33.24
31.20
26.84
28.86
24.64
22.23
23.38
50.01
46.51
48.20
Ours w/o structure
37.38
31.98
34.47
33.17
26.70
29.59
29.85
25.87
27.72
51.78
47.57
49.59
Ours pointer
37.74
31.85
28.70
30.75
23.32
25.21
24.23
54.31
50.12
52.13
Ours
36.40
33,66
34.97
35.06
29.61
32.11
31.42
24,46
27.50
55.10
48.05
51.34
code2seq (Multilanguuge)
34.49
25.49
29.32
31.62
22. 16
26.06
23.97
17.06
19.93
52.70
44,36
48.17
GREAT (Mulhilanguage)
36.75
31.54
33,94
27.7%
30.05
24.33
Ours w/o structure (Mult.)
38.48
30.14
33.80
35.38
27.41
30.89
32.61
26.76
29,40
55.03
47.34
50.90
Ours w/o pointer (Mult.)
38.91
33.12
35.78
37.21
29.75
33.07
34.52
27.31
30.50
56.07
50.76
53.28
Ours (Multilanguugee)
38.89
33.82
36.18
36.95
29.98
33.10
33.93
28.94
31.24
56.00
50.44
53.07
Ours (Mult. + Finetune)
39.85
32.79
35.98
37.00
29.79
33.00
35.85
27.75
31 228
55.63
51.12
53.28
Ours (Mult. + LM Pretrain)
39.67
35.29
37.35
37.06
31.94
34.31
35.19
29.36
32.01
57.73
51.89
54.68
Table 2: Code summarization results on the CSN dataset (micro F1).
multi-language training) substantially outperforms all other models on all but one language, high- lighting the effectiveness of jointly learning from Structure and Context. The only exception is Ruby, where it performs on par with its Context-only variant. We attribute this to the fact that there are relatively few samples in the Ruby dataset, and that Ruby is an dynamically typed language, which could make the Structure less powerful for learning. Interestingly, the Context-only CODE TRANS- FORMER outperforms GREAT on all languages. We attribute this to the fact that GREAT uses the Structure of the programs only in localized way (see Sec. 3.I. Another noteworthy finding is that code2seq performs comparably to the Transformer-basee baselines on Go. We hypothesize that ASTs more informative on Go since it is a compiled and strongly typed language.
Java-small results. In Table 3 we present code summarization results on the Java-small dataset. Among all models equipped with a pointer network, the CODE TRANSFORMER (without pre- training) obtains state-of-the-an on code summarization, outperforming all baselines, including the previous state-of-the-an on Java-small proposed by |ernandes et al. (2019). Further, pre-training on mewwaammemmmmmmm and Java-large the permutation language modeling objective (Yang et al. 2019) substantially improves precision, recall, and Fl score after fine-tumin fine-tuning on Java-small. To avoid leak- age, exclude the projects used in the validation and test splits of Java-small from pre-training.
Ablation study. We further perform ablations where we remove our model's 's access to the Context or Structure, also presented in Table
With pointer network. We find that both ab- lations lead to a substantial drop in perfor- mance, highlighting the benefit of learning jointly from Structure and Context. Interest- ingly, the model without access to the Struc- ture performs slightly better than the variant without Context. Note that model with- out Structure related to the XLNet (Yang) et al. 2019) model, where add a pointer net- work to the decoder and concatenate the token types their respective input tokens (see Ap- pendix A.4). Without pointer network. We re- peat the ablation on the variants without pointer network. Here, the variant without Context per- forms better than the variant without Structure, indicating that the pointer network helps to compensate for the lack of access to Structure. The Structure-only variant (w/o pointer net) of our model even outperforms the full variant this sce- mario. Inspection of the results revealed that the Structure-only variant has better performance longer method names, which have an outsize influence on the micro-Fl score used in this work.
Model
Prec.
Rec.
F1
Without pointer net code2seq
51.23
37.31
43.18
Ours w/o structure
50.70
45.49
47.96
Ours w/o context Ours
51.81
46.04
48.75
50.33
46.80
48.50
With pointer net Fernandes al. 2019
$1.4
GREAT
53.60
46.41
49.75
Ours w/o structure
55.48
46.07
50.34
Ours w/o context Ours
54.45
45.29
49.45
54.85
49.84
52.22
Ours + Pretrain
57.02
50.87
53.77
Table 3: Results on Java-small and ablation study.
Ablation of the AST-based distances. In Table we compare the performance of our model when trained with each of the four different AST distances (sibling shortest paths, ancestor shortest paths, shortest paths, personalized PageRank; in the Structure-oni only setting and without pointer network. For reference, we also show the results of training our model using all four AST distance functions (c.f. Table 3). We find that, while the
Section 3.1). Here, the model is trained on Java-small

---

Published as a conference paper at ICLR 2021
personalized PageRank distance performs best on its own, each of the individual distances on their own performs substantially worse than their combination, highlighting the usefulness of combining the distances in our model well as their complementary nature.
5.2 MULTILINGUAL CODE SUMMARIZATION
Setup. A key contribution of our proposed architecture is that it only uses language-agnostic features, i.e. the source code and features that can be directly computed from the AST. We use this fact to study the first multi- language code summarization model. We train our model jointly on Python, Javascript, Ruby, and Go. The shared sub-token vocabulary is the union of the individual vocab- ularies, enabling us to evaluate the multi-language model on the individual languages and compare with the
AST distance
Fl score
Sibling shortest paths Ancestor shortest paths
46. 17
47.89
Shortest paths Personalized PageRank All the above (c.f. Table 3
47.76
48.47
48.75
Table 4: AST distance ablation study.
models. As proposed by Conneau & Lample 20199) we add a learned language embedding to each input embedding.
Results. In the lower part of Table 2 we can see the results of training our CODE TRANSFORMER
jointly on all four programming languages. Our multi-lingual variants substantially outperform the mono-lingual models on all languages. The strongest improvement is on Ruby, which is also the programming language with the smallest number of samples in the dataset. Fine-tuning on the individual languages after joint training on code summarization only has marginal effect on performance, indicating that the multilingual objective is well-aligned with the individual languages. In the last row, we have a variant of our model where we pre-train on the multi multi-lingual masked language modeling task, followed by finetuning on code summarization on the individual languages.
Further, we observe that similar to the results on Java-small, removing the pointer network generally leads to weaker performance. One notable exception is Go, where the variant without the pointer network performs better in terms Fl score. Our investigation revealed that there seems to be some violation of the i.i.d. assumption in the split provided by the creators of the dataset. In Figure 7 we show that in the test partition of the Go dataset, the share of tokens from the labels also occurring in the methods' bodies exactly the scenario where the pointer network can improve predictions is substantially lower compared to the vrain/valiaation partitions.
Remarkably, the multi-languagee Context-only variant (i.e. without access to the Structure) performs substantially worse than the full multi-language variant. This highlights that Structure is crucial to exploit the commonalities of different programming languages. Also notably, the GREAT baseline's results also improve substantially when trained in the multi-language setting, though it is still outper- formed by our model. However, our results indicate that any representation learning model for code can benefit from multi-language training, especially when evaluating on low-resource languages.
we present results using the sample-F1 score. At the time of submission. our mono- lingual model on Python outperforms the state of the art on the ogbg- sgbg-cooe22 2 (Hu et al. 2020 leaderboard by 12%, and our multilanguage variant with LM pretraining outperforms by 122%.
In Table 16
Qualitative analysis of multilingual representations. Learning the CODE TRANSFORMER on multiple programming languages jointly provides us with embeddings a shared representation space. In Fig. 4 we show a t-SNE (Maaten & Hinton (0088) visualization of the ca. 40,000 snippets from the validation sets of four programming languages from the CSN dataset. For the embedding of a snippet, we use the representation of the method name in the final layer of the encoder. Note that the true method names are masked, i.e., inaccessible to the model. Further, note that in contrast to the monolingual embeddings learned by Kanade et al. (2020), the embeddings we evaluate are learned on the task of code summarization (though a similar study could be performed by using our model that was trained the traditional language modeling pretraining task on multiple languages).
While snippets from the same language tend to be grouped together, there are interesting intersec- tions of the different programming languages. For example, we highlight all methods whose names start with the subtoken parse or main. We see that snippets starting with parse are predom- inantly in an intersection region of Python and Javascript. From these snippets, we display the
8

---

Published as a conference paper at ICLR 2021
python
javascript
go
ruby
Name starts with 'parse'
Name starts with 'main'
Figure 4: t-SNE visualization of the CODE TRANSFORMER learned multilingual representations.
def parsaBool (G):
function jacoverage Sawprerrrg_ggettoooe Root leanVa
1 B.lower 11 if 1 in (*Lzue", "L", "17):
5. s.zoLowe:tase !I
return True ("Calse", "E", "0"):
15
'noo'
--- 'n"
"OEE" 1 8 101) {
return Falze
return false;
zaise Exception(
"Unable le to convert string 185""
return true;
"Lo a boolean value" &
Figure 5: Example snippet starting with parse (left) and its best embedding match from other languages (right). Both methods parse an imput string convert it into a boolean value. Note that even though they are semantically very similar, their method names are not: nonetheless, their representations in the CODE TRANSFORMER encoder reflect their semantic similarity.
cross- loos-language pair with smallest Euclidean embedding distance in Fig. 5 Remarkably, both snip- pets are effectively the same method in Javascript and Python - it is worth reminding that the model has never seen any parallel data during training. On the other hand, snippets starting with ma in tend to lie at an intersectional region of Python, Javascript, and Go. In Table the appendix we show additional cross-lingual pairs with similar embeddings, including failure case of a ma in function, where embedding distance is not representative of semantic similarity. We attribute this to the fact that we used the encoder output embedding of the masked method name the representation used by the decoder to predict the method name as snippet's representation. Thus, snippets with com- pletely different semanties (as is to expected for very generie method names starting with main) have similar representations because they are predictive of the method name.
As another qualitative insight into the representations learned by the CODE TRANSFORMER we have found that the language embeddings of languages with similar roots in language design are close; see Table in the appendix for the pairwise similarity matrix of the learned language embeddings.
CONCLUSION
We present the CODE TRANSFORMER, which learns jointly from Structure and Context of programs while only relying on language-agnostic features. Our model obtains state-of-the-art performance on code summarization on five different programming languages. Besides these results for train- ing on individual languages, the language-agnosti nature of our model allows us train it jointly on multiple programming languages. The resulting multilingual model substantially outperforms its mono-lingual variant on all programming languages, setting the state of the art on each lan- guage. We observe the largest improvement from multilingual training on the language with fewest resources, indicating that multilingual training can improve learning for less widely used program- ming languages. Remarkably, multilingual training only from Context does not lead to the same improvements, highlighting the benefits of combining Structure and Context.

---

Published as a conference paper at ICLR 2021
REFERENCES
Mithun Acharya, Tao Xie, Jian Pei, and Jun Xu. Mining api patterns as partial orders from source code: From usage scenarios to specifications. In Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Founda- tions of Software Engineering, ESEC-FSE '07, pp. 25 34, New York, NY, USA, 2007. Associ- ation for Computing Machinery. ISBN 9781 9781599338111 14. doi: 10.1 10.1145/1287624 1287663.. URL 022244.2273333
https:://di.org//10 1145/
Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. Learning natural coding conventions. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Founda- tions of Software Engineering, FSE 2014, pp. 281-293, New York, NY, USA, 2014. Associa- tion for Computing Machinery. ISBN 9781450330565.
doi: 04114472263886.266355555 URL
/tttps://doi.org/ 111///...1111225888888 Ttt.3355868666668868 2635883
Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 38-49 28-49, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 4503366588 doi: 10.1145/278680022786444 1tt.//0.111//278888
URL https://dol.
2786849
Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for extreme summarization of source code. In International conference on machine learning, pp. 2091-2100, 2016.
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning represent programs with graphs. In International Conference on Learning Representations, 2018. URL https: //openzeview. wttp::/ttwooommmmcommooo...
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A general path-based representa- tion for predicting program properties. In Proceedings of the 39th ACM SIGPLAN Confer- ence on Programming Language Design and Implementation, PLDI 2018, pp. 404 404-419, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450356985. 10. 10.1145/3192366.19999 1192222 URL
doi:
1145/11111111 01733366.19220 3192412
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from struc- tured representations of code. In International Conference on Learning Representations, 2019a. URL httpthts:t//peeeeeeeees ittp://rwweeeeeeettooreeeeeieeee iirrummidddlllgxx000t00
net/ forum?
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed repre- sentations of code. Proceedings of the ACM on Programming Languages, 3(POPL): 2019b.
Peter W Battaglia, Jessica B Hamrick. Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arxXiv/806.01/
2018.
Avishkar Bhoopchand, Tim Rocktäschel, Earl Barr, and Sebastian Riedel. Learning python code suggestion with sparse pointer network. arXiv preprint arXiv:1611.08307. 2016.
Benjamin Bichsel, Veselin Raychev, Petar Tsankov, and Martin Vechev. Statistical deobfuscation of android applications. In Proceedingy of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS 116, pp. 343-355, New York, NY. USA, 2016. Association for Computing Machinery. ISBN 9781450341394, doi: (0.111452976749.29788222 URL https:
1145/2976740 2978422
Pavol Bielik, Veselin Raychev, and Martin Vechev. Phog: probabilistic model for code. In Interna- tional Conference on Machine Learning, pp. 2933-2942, 2016.
Aleksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Günnemann. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery amp; Data Mining, KDD '20, pp. 2464-2473, New York, NY, USA, 2020.
10

---

Published as a conference paper at ICLR 2021
Association for Computing Machinery, ISBN 9781450379984, doi: 10. 10.1145/339448663033396
URL U///ttpps///od..ccr/0011111////3 .3429 3403296
Dylan Bourgeois. Learning representations of source code from structure and context. 2019. URL
Alexis Conneau and Guillaume Lample.
Cross-lingual language model pretrain- ing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and
R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 7059- Curran Associates, Inc., 2019. URL httpp:/ppapers. wwwpeerss..is..ommm cctpserc/ 8928 caa--rosss--1
7069.
Hanjun Dai, Bo Dai, and Le Song. Discriminative embeddings of latent variable models for struc- tured data. In International conference on machine learning, pp. 2702-2711, 2016.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 2019.
Hoa Khanh Dam, Truyen Tran, and Trang Pham. A deep language model for software code. In FSE 2016: Proceedings of the Foundations Software Engineering International Symposium [TTee Conference), 2016.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtering. Advances in neural information processing systems, pp. 3844-3852, 2016.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks graphs for learning molecular fingerprints. In Advances in neural information processing systems, pp. 2224-2232, 2015.
Patrick Fernandes, Miltiadis Allamanis, and Mare Brockschmidt. Structured neural summa-
rization. In International Conference Learning Representations, 2019. URL https:
//openreview. http://ww.....tte.foooooooooo-ee
Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 729-734. IEEE, 2005.
William I. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning large graphs. In NIPS, 2017.
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. International Conference on Learning Representations, 2020.
URL http:::::///peeeviee...ee net forum? Botoruumid-BBlllnaane
Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. On the natural- ness of software. In Proceedings of the 34th International Conference on Software Engineering, ICSE 12. pp. 837-847. IEEE Press, 2012. ISBN 978 9781467310673.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()):
1735-1780, 1997.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 22118-22133, Curran As- sociates, Inc., 2020. URL Fb60d41 1 5a5c5b 7 2tt.555222222772522cc2f2oo3222
neurips. wctppapee/222//111D.. pddf
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv: /90.9099990936 2019.
11

---

Published as a conference paper at ICLR 2021
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning. 2020.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Johannes Kliepera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR), 2019a.
Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learn- ing. In Conference on Neural Information Processing Systems (NeurIPS), 2019b.
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molec- ular graphs. In International Conference Learning Representations (ICLR), 2020.
Jian Li, Yue Wang, Michael R Lyu, and Irwin King. Code completion with neural attention and pointer networks. arXiv preprint arXiv: 2017.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural 2015.
networks. arXiv preprint
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov):2559-22655 2008.
Chris Maddison and Daniel Tarlow. Structured generative models of natural source code. In Inter- national Conference on Machine Learning, pp. 649-657, 2014.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree struc- tures for programming language processing. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI' 16. pp. 1287-1293. AAAI Press, 2016
Trong Due Nguyen, Anh Tuan Nguyen, Hung Dang Phan, and Tien N. Nguyen. Exploring api embedding for api usages and applications. In Proceedings of the 39th International Conference on Software Engineering, ICSE 17. pp. 438 449. IEEE Press, 2017. ISBN 9781538638622. doi: / doi org /tw.g//..... 1109/ ICSE. 2017.201..4
10.1109//CSE.2011..77
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation. PLDI 14, pp. 419-428, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 4503277888
doi: 10.1 10.1145/2594291 2594321. URL https:://oo..
org/10.1100 105e/0001445/2942911. 25943211
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20 (0(1))1--0,, 2008
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. Aldo Gangemi, Roberto Navigli, Maria- Essther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and Mehwish Alam (eds.), The Semantic Web, pp. 593-607, Cham, 2018. Springer International Pub- lishing. ISBN 978-3-319-93417-44
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp- 464-468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: N0.18653/v1/N 1-7444.74 URL htttst:::: wwwww acl web. wwwp./e....///tttoooogggo///o///
12

---

Published as a conference paper at ICLR 2021
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. International Conference on Learning Repre- sentations, 2019. URL/ttts:///pp/ee///// wwt.:/ewwwwwwweeeeeeeeeeeeeeeeee net/ htt.://oorrm..ddl1ll1113g3333
Vighnesh Shiv and Chris Quirk. Novel positional encodings enable tree-based transformers. In Advances in Neural Information Processing Systems, pp. 12081-12091, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cexsing, pp. 1631-1642, 2013.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXivv1503.00077 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-599999 6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in neural information processing systems, pp. 2692-2700, 2015. 2015.
Song Wang, Devin Chollak, Dana Movshovitz-Atiias and Lin Tan. Bugram:
Bug detection with n-gram language models. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, pp. 708-719, New York, NY, USA, 2016. Associ- ation for Computing Machinery. ISBN 9781450338455. 1145 21224///227766... 2970341
doi: 10.1145/2970226.290033 URL.
https://doi. /tg//...oggg..../// 1tt.://....11/997710
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learming Representations, 2019. URL https:
//openreviee ew. net./forum? d= A5Km
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, pp. 5754-5764, 2019.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hi- erarchical graph representation learning with differentiable pooling. In Advances in neural infor- mation processing systems, pp. 4800 2018.
Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7134- URL http: http:::/pppeeeiinn........eeeee.
7143. PMLR, 09-15 Jun 2019, html
yb.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems, pp. 5165-5175, 2018.
13

---

Published as a conference paper at ICLR 2021
Python
Javascript
Go
Ruby
Python
1.00
0.43
0.42
0.79
Javascript
0..43
1.00
0.84
0.39
Go
0.43
0.84
1.00
0.38
Ruby 0.79
0.39
0.38
1.00
Table 5: Pairwise cosine similarities of the learned language embeddings of the CODE TRANS- FORMER.
ACKNOWLEDGEMENTS
We are grateful to Dylan Bourgeois for having paved the way to this research contribution with his thesis work (Bourgeois 2019). further thank Simon Geisler for his helpful suggestions and proofreading the paper, as well the anonymous reviewers for their constructive feedback and fruitful discussions.
This research was supported by the TUM International Graduate School of Science and Engineering (IGSSE). Stanford University is supported by DARPA under Nos. N66001 1924033 (MCS); ARO under Nos. W911NF-16-1- 0342 (MURI), 1911NF-16-1-011 (DURIP); NSF under Nos. OAC- 1835598 (CINES), OAC-1934578 (HDR), CCF-191 19940 (Expeditions), IIS TIS-2030477 (RAPID); Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Chan Zuckerberg Biohub, Ama- zon, JPMorgan Chase, Docomo, Hitachi, JD.com, KDDI, NVIDIA, Dell, Toshiba, Intel, and Unit- edHealth Group. Jure Leskovec is Chan Zuckerberg Biohub investigator.
A APPENDIX
A.1 DISTANCE ENCODING FUNCTION
For encoding scalar relation values via vectors employ encoding functions o: Rd where d the model's embedding dimension. We choose the popular sinusoidal encoding function presented in Vaswani al. (2017):
6(r=4))24 sin
Ti-t M2k/d
6r(--j22441
Ti-s M2k/a
where < < d/2 the position the encoding vector and M is some constant; we adopt M 10. 000 chosen by /Vaswani et al. 2017). Note that the distance encoding functions have no trainable parameters.
A.2 MULTILINGUAL REPRESENTATION ANALYSIS
In Table 5 we show the pairwise cosine similarities of the learned language embeddings of the CODE TRANSFORMER. We see that the pairs Python-Rube and Javascript- Go have similar language embeddings. This aligns well with roots of language design and common use cases of the languages.
Moreover, in Table 6 we show selected snippets starting with is, ma main, or load (left) and their best embedding matches from other languages (right).
A.3 DATA PREPROCESSING
A.3.1 TEXTUAL CODE SNIPPET PREPROCESSING
1. Tokenize code snippets with Pygments sanguage-speciffe tokenizer.
2. Remove comments (both multi-line, single-line and doc comments). pygments. token. Comment and pygments. tat.toknn Literal. String. Doc that are generated by Pygments are used to identify comments.
The comment token types.
14

---

Published as a conference paper at ICLR 2021
function JeBquelAAreyy (dy b)
if (a men
return true;
[10 undofinedi II
aceSameflcat if len 1- len false
(] {
(b mas undefinedi]
false;
length;
for (e) 1+4
if [1 b. Length)
if a[i] 1- b[i] return false
return false;
while [1--)
[a]]] b[i])
return false;
Function
var rawbata $ "e"_HeadeeTextinn (data (data-1og loginn
user-smail]
TasksayRello +tasking.T)
if tsawDatal [
username := t.Flags. Sttring ("rame"""mme""" ("name")
as
name: rawData, 1og inae,,
moil: M.wData..oginnseee.....
usec, 1- user. Cucrent ()
17
getCartid (functioon (cart) {
me cart cart;;
if L.3lags. Dool (eerbose") {
injectMenu (1:
t.Togf ("Hel("Hel10) $8, the time now tennn, username, Sima.Now (3)
refreshlsers (actions waw.:lons.udatteeeeeeee.... .upattelserat
Listenon:Caager tonChangel
Listenon:Orerronn www.oooooeeeoeeeneiniiiioa (onCunEimm)
else t.LogE ("ilello tsta", username]
)17
else
callback (Nno (
fune (a:kee(E::eeaam string) error !
info, eer :- os.Stat (filename)
if err !- nil
os.IaNctExist (erF) return nil
return err
if info. Size() return nil
def tload_rcle_ttiejseell :ilename):
if not lcs. sya-stderr.urritar writtt sya.st
files, ezr filepath. filename L_AAACUU_SUEFPR
"rtlint: la: No such file or 'directoryla' & filenane
if arr !- nil
return
try:
basename = cs.psth barenare (filename) (nate, ext] os ps.p.th. splitext (basensme) imp. Looa__suuree Iname, elnam)
nunRackup :- byta(1)
except Exception
(en(riles) !-
krrtte sts.rt writei
lastFile :iles|len(tties)) num3sckup Lestriiee[Len(Lsstiieee- if aumbackup > 1g' numSackup 111
'rélint: 9s: exception while
"oadiigg tain" (filename, scr(e))
else num3ackup '1'
return
Copy (oi
Sprints ("Gs+ke" Eilename, string (nunBackupt
Table 6: Selected snippets starting with iss, ma main, or load (left) and their best embedding matches from other languages (right).
15

---

Published as a conference paper at ICLR 2021
Example mippet
1: def test() -> NoReturn
Statements
Has children with overlapping ranges
Annotation
range: 1:1 2:9
range:
1:15 1:23
Function
Identifier
NoRetuum
Identifier
Statements
NoOp
Empty
Figure 6: Example snippet and its corresponding AST obtained from GitHub Semantic.
3. Empty lines are removed.
4. Hard coded strings and numbers are replaced with a special TEASK_STRING and token.
5. Indentation style of the code snippet is detected and whitespace characters at the beginning line are replaced with single (INDENT] or [DEDENT] token when indentation changes.
6. Tokens are further split into sub tokens, c.g., sciBottomHeight + ['set', 'bottom', 'height'J. Throughout our experiments, we use 5 input sub tokens. If a token consists of less than 5 sub tokens, the remaining spaces are filled with special [PAD] token.
7. Any remaining tokens that only consist of white spaces are removed. The only white space characters that kept line breaks "\n".
8. Any code snippets where the Pygments tokenizer cannot parse token discarded.
A.3.2 STAGE PREPROCESSING (GENERATION OF ASTS)
1. Stripped code snippets are used generate language-specific ASTs. For Java, we use the AST parser from the java-parser project. The ASTs contain node types and source ranges. For Python, JavaScript, Ruby and Go, we use semantic.
2. Snippets that lead to AST parse error are discarded.
3. We caleulate mapping between tokens and nodes in the AST. Every token is assigned to the node in the AST with shortest source range that still encompasses the source range of the token.
To find such node, we originally intended to make use of the assumption that source ranges of child nodes do not overlap. Then, one could easily find the node with smallest encompassing source range by greedily selecting at every layer in the AST the child that encompasses the token's source range (there can only at most one child that fulfills this). However, this assumption does not hold for all ASTs (see Figure 6 for an example). As heuristic, we greedily select child node with the shorter source range case there were multiple child nodes with encompassing source ranges. This approximation seems to be sufficient in our case, and limits runtime as we do not have to consider multiple paths in the AST. It is also sufficient to stop when child node encompasses the source range of the token, as ASTs the source ranges of child nodes are always contained in the source ranges of their parent.
16

---

Published as a conference paper at ICLR 2021
A.3.3 STAGE 2 PREPROCESSING (CALCULATION OF DISTANCE MATRICES)
1. Tokens are vocabularized. Any token occurring less than 100 times in the training set is replaced by <unk> token.
2. We calculate multiple pair-wise relations between nodes in the AST:
Personalized Page Rank (PPR)
We interpret the negative logarithm of PPR as a distance. We use a teleport probability of a 0.15 and a threshold of i... anything with - log PPR > 5 considered unreachable
Shortest path length between two nodes
Ancestor shortest paths (bidirectional). That is, the parent has an ancestor shortest path distance of 1 to all its children and the child has a distance of to its parents. We consider nodes that are not ancestors or descendants of a node (i.e. not reachable by following only parent or only child relations) as not connected in the ancestor shortest
paths relation. We encode this with a very large value in their distance; we have found value of I, 000 to work well in practice.
Next sibling shortest paths (bidirectional, analogous the ancestor shortest paths)
Note that the ancestor shortest paths and next sibling shortest paths are required because treating the AST as normal graph leads to ambiguity. In graph, the neighbors of node have no ordering; however in the AST, the order of the children of a node reflects their order in the code. Therefore, we explicitly include the next sibling shortest paths. The ancestor shortest paths would not be required if we treated the AST as a directed graph: in this case, however, leaf node could not reach any other node in the AST, and therefore both PPR and shortest path length are not useful this case. Therefore, we model the AST undirected and inject the ancestor / child edges to avoid ambiguity.
3. Distance values are binned into 32 bins using area-based exponential binning with growth factor of 1.3, i.e., the area of a bin's rectangle (r: bin range, y: number of val- ues bin) will be approximately 1.3 times bigger for the next bin (going away from the bin that contains the zero value). Additionally, for discrete distance measures (such as se- quence distance or shortest path length), we hard-code values around have their own bins. For instance, on the sequence distance the values -4, -4....... ....... 4 have their individual bins, and around those values we employ the exponential binning.
4. Punctuation tokens (such as points or brackets) are removed from the input sequence, as experiments showed that their presence does not improve performance but slows down training due to bigger input sizes.
5. Snippets that are longer than NUX_NUMMTOOENN after punctuation tokens are removed are discarded from the training set. Throughout experiments, use MAX MAA_NUMM_
= 512. During evaluation on the test set, we use MAX WWW.NMM..OOKEEE = 1000.
A.4 INPUT EMBEDDINGS TO THE MODEL
Besides its five subtokens (e.g.. O'ge'' 'data', [[PAD] '[PAD]' [[PAD] }), each input token has token type (coming from the Pygments tokenizer) and an AST node type. The AST node type is the type of the node assigned to each respective token, as described in Section A.3.2 We concatenate the embeddings of the five subtokens, the token type, and the AST node type. Then, we apply linear layer (without activation function) to project down to the model's embedding dimension.
A.5 INPUT TO THE GREAT BASELINE
As mentioned in the main text, we also compare with GREAT Hellendoorn et al. (2020). Since their preprocessing pipeline is proprietary and could not be shared with even after contacting the authors, we provide to GREAT the same AST distances as our model. Since GREAT uses edges in- stead of distances to encode relations in the Structure, we essentially threshold the ancestor, sibling, and soottsst-pathss distances and provide the edges where the distances are equal to I (including their edge types) to the model.
17

---

Published as a conference paper at ICLR 2021
(a) CODE TRANSFORMER
(b) GREAT
(Hellendoorn
al.
2020
Hyperparameter
Value
Hyperparameter
Value
Activation
GELU
Input Nonlinearity
tanh
Activation
GELU
Num. layers
Num. layers
d dFF
1024
1024
2048
dFF
2048
Pdropout
0.2
Paropout
0.2
Num. heads
Num. heads
Table 7: Code Summarization hyperparameters
A.6 EXPERIMENTAL. SETUP
Table shows hyperparameters of our models for code summarization. For all our experiments, we Transformer Decoder with layer and teacher forcing to generate 6 output sub tokens. We also employ label smoothing of 0.1. As optimizer, we use Adam with a learning rate of 8e-6 and weight decay of 3e-5, Batch size during training is with simulated batch size of 128 achieved by gradient accumulation.
Apart from comparing the CODE TRANSFORMER to baselines, we performed the following hyper- parameter comparisons and ablation studies:
CODE TRANSFORMER (structure-only) Using only AST information as input, i.e., masking all tokens that do not correspond to a leaf of the AST, and removing the token distance relation to be used by the model. Further, token types are not fed into the model.
CODE TRANSFORMER (context-only)
Here, we do not include any information on the AST (i.e. node types and distances on the AST). This is effectively the XLNet backbone plus encoding of the token type returned by the tokenizer.
CODE TRANSFORMER (Max-Dist.)
Applying a Max Distance Mask of 5 to the shortest paths distance (i.e., model cannot see node that is more than 5 hops away matter how small the other distances are). Early results showed that, as expected, results deteriorate substantially when limiting our model's receptive field. Hence, do not include these results in this work.
Using 16 and 64 bins instead of 32 bins. This had no noticeable effect performance.
A.7 CODE SUMMARIZATION EXAMPLES
we present example functions from the Java-small dataset along with the different models predictions for the function name.
In the Tables
TO
2
13
and 15
18

---

Published as a conference paper at ICLR 2021
public Summation ion next () return parts]itt];
{
Model
Prediction
GREAT
get x map get parts get get next
code2seq
Ours w/o structure
CODE TRANSFORMER
Ground Truth
next
Table The CODE TRANSFORMER the only model correctly identify the notion of getting the next entry.
private Path FindCacheFile (Path[) cacheFiles, String fileName) 1 if (cacheFiles !- null && cacheF les. lenghh 0)
for (Path file cacheFiles) { if return file;
() .equals (fi 1eName)
return null;
Model
Prediction
GREAT
get path
code2seq
find file
Ours w/o structure
get file
CODE TRANSFORMER
find cache
Ground Truth
find cache file
Table The CODE MRARSFFRMER is the only model to both recognize that the task is to find a file well the fact that it is about the cache. However, it did not correctly predict the file part of the method name.
public int compare (Pair<ioggedJob,, JobTraccReader> p1,
PaircloggedJJb, JobTraceReader>> p2) {
LoggedJeb j1 pl.first ()
LoggedJcb j2 f2.first ()
return (j1. get SubmitTime () ge..getSumiitiim ()) 2 -1
(i1, Suomitiii () == 12, get Submitttieeme ()) 0
Model
Prediction
GREAT
run get submit time compare
code2seg
Ours w/o
CODE TRANSFORMER
compare
Ground Truth
compare
Table 10: The CODE TRANSFORMER and the its context-only variant are the only models correctly recognizing the 'compare' template in the method body.
19

---

Published as a conference paper at ICLR 2021
public static MNTPROC fromValue (int value) if (value value >= values () length)) {
return null;
return values () [value];
Model
Prediction
GREAT
get value get value
code2seq
Ours w/o structure
CODE TRANSFORMER
from value
Ground Truth
from value
Table 11: The CODE TRANSFORMER is the only model to recognize that the snippet is similar to a static factory method which is often preceded with from.
private IterabllediistRloblee listRootBloo obs (String aPrefix,
boolean useP1atHloble obl.istoooliiiiiiii
ingteesctttlllsssssnneeetta 1s> 1istingDetails,
BlosRequestOptions options, OperaticnContext opContext)
throws StorageException, URSSyntaxException
cloudillbbirretttryyyaaaaa directory return directory, listBlobs (null, useflattlobbissinng listingpetails, options, opContext);
(aPrefix),
Model
Prediction
GREAT
list blobs
code2seq
list blobs
w/o structure
list blobs
CODE TRANSFORMER
list blobs by prefix
Ground Truth
list root blobs
Table 12: All models could correctly identify the listBlobs() call in the return statement. However, the CODE TRANSFORMER additionally comprehended that the specified prefix is quite important.
private static void dumpOpCounts MaaunMap<FFSEEE tSEditoogpppCdees Holder<Integer>>> opCounts) { StringBuilder sb new StringBuilder ;
sb.append ("Summary of operations loaded from edit log:An "); Joiner, ("An ") www.hnkeeyaalessppaatttto ("=") appenddo (sb, opCounts);
FSImage. LOG, debug (ab toString ());
Model
Prediction
GREAT code2seg
append
add log
Ours w/o structure
CODE TRANSFORMER
log op counts
Ground Truth
dump op counts
Table 13: Only the CODE TRANSFORMER could correctly identify that it is the op counts that should be logged.
20

---

Published as a conference paper at ICLR 2021
static String execCommand
String... emd) throws ICException
Stringl args new String cmd. length 11; bystem.arraycee (amd, 0, args, 0, cmd. length);
args [cmd. length] - f L.getCanenicalPathh. string output Shell. chell.eecccommandkaarrg (args); return output;
Model
Prediction
GREAT
get canonical path
code2seq
Ours w/o structure
get output
CODE TRANSFORMER
cxce command
Ground Truth
exec command
Table 14: Only the CODE TRANSFORMER and code2seq could identify that the relevant part of the method is concerned with executing command instead of returning something.
protected void subView (Class<? extends SubView> cls)
indent (of (ENDTAG));
sb #b.setLength(0)>
out.print (sb append (" appeend (ciis (cls, getName ()) append ("J " .toString () out.printin();
Model
Prediction
GREAT
print print
code2seq
Ours w/o structure
print
CODE TRANSFORMER
print sub view
Ground Truth
sub view
Table 15: Only the CODE TRANSFORMER was able link the print functionality to the object that should be printed, which can only be inferred from the object's 's class in the method parameters.
21

---

Published as a conference paper at ICLR 2021
Tainn
Amhoutt of tabe) takend 3ppcarra bodyy
60%
50%
40%
30%
20%
10%
0%
javesmall
pythoh
raby
Figure 7: Share of tokens labels also occurring in the bodies methods.
Python Rec.
Javaseript Rec.
Ruby Rec.
Model
Prec.
F1
Prec.
F1 Prec.
F!
Prec.
Rec.
F1
code2seq
GREAT
34,93
31.12
31661
29.69
25.55
19.69
21.49
22.18
48.38
Ours w/o structure
36.87
32,17
32.97
31.30
25.03
26.64
31.43
25.34
26.63
49.78
46.73
46.69
Ours pointer
38.77
31.72
33.27
32.70
25.50
27.33
32.12
30.17
29.36
$3.09
49.26
Ours
36.68
33,86
33.84
33.36
27.55
29.02
31.53
24.72
26.43
52.00
47.35
47.93
code2seq (Multilanguage)
GREAT (Multilanguuge)
Ours w/o structure (Mult.)
36.78
29.92
31.58
32.60
26.02
27.74
31.71
26.07
27.24
51.91
47.58
48.15
Ours w/o pointer (Mult.)
37.18
30.52
32.04
33.95
25.92
28.11
32.76
25.04
27.01
53.50
48.54
49.35
Ours (Multilanguage)
38.10
33.32
34.18
34.29
28.69
30.08
33.30
28.33
29.29
53.86
50,46
50.61
Ours (Mult. Finetune)
38.29
32.41
33.65
34.43
28.28
29.91
32.89
27.15
28.49
53.85
50.85
50.81
Ours (Mult. + LM Pretrain)
38.97
34.77
35.34
35.23
30.26
31.38
33.73
29.15
29.94
55.31
52.03
52.13
Table 16: Code summarization results on the CSN dataset (sample-F1).
A.8 ESTIMATION OF POINTER NETWORK POTENTIAL
In Table we observe that the pointer network improves the score for all languages except Go, where counterintuitively it leads to reduced performance measured by Fl score the test set (while it improves by about 3 points on validation). To investigate this, in Figure 7 we plot the share of tokens in the labels also occurring in the bodies of methods in the different languages. Intuitively, this gives an indication on how much gain we can expect from using pointer network. If the share pointer network cannot improve the prediction by pointing at the input. We see that for Go, there is strong mismatch between the test partition and the train/validation partitions, which much fewer tokens from the labels occurring in the bodies of methods on test compared to train/validation. Thus, we attribute the drop in performance observed by adding a pointer network on Go this apparent violation of the i.i.d. assumption.
zero, then no token in the labels
occur the bodies of the methods, so the
A.9
CODE SUMMARIZATION RESULTS ON THE CSN DATASET (SAMPLE-FL)
In Table 16 we present our results on the CSN dataset as measured by the sample-F1 score.
22


Reviews:
[{"review_id": "zeg1z985Jj9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper wants to combine sequence (called Context) and AST (called Structure) representations of source code in a Transformer encoder model. For this, it makes clever use of relative position embedding of Transformer-XL. Different pairwise relations based on ASTs and sequence ordering are computed and each of them is encoded as a separate distance matrix, with its own learnable projection. This model is evaluated on the task of code summarization and compared against code2seq and GREAT models, and against different configurations of the proposed model, called Code Transformer. The results show that Code Transformer achieves results better than these models. This evaluation is conducted on five languages. A separate evaluation with the model trained on all the languages together is also performed. This multi-lingual model outperforms the mono-lingual models.\n\n**Design** \nThe paper considers four types of paths in the ASTs and quantifies the path lengths and directions. The ancestor and sibling distances are also defined over internal (non-terminal) nodes of the AST. But the input to the model is a sequence of sub-tokens (grouped by tokens). As discussed in appendix A.2 and A.3, the leaf nodes are mapped to some internal nodes whose types are concatenated to the token representations. Thus, the internal nodes themselves are not part of the input. From section 3.1, it appears as if all the internal nodes will be somehow encoded. This does not seem to be the case and the non-leaf-to-leaf paths are perhaps projected only one certain internal nodes. Please clarify this.\n\nA separate projection matrix $W_r^{(s)}$ is used for each relation r. Is the positional encoding $\\phi$ shared between them?\n\nThe paper calls the proposed model as Code Transformer. Technically, it is only the encoder. It would be good to make this clear.\n\nIn the appendix, the paper states that a token span may have overlap among its ancestors. A concrete example would help.\n\n**Language-agnostic representation**\nI find the use of the term \"language-agnostic representation\" confusing. The proposed method uses features computed from ASTs which are language dependent and even dependent on the parser. If a different parser for the same language is used, the features would change (because even if the grammars might be equivalent they may not be identical). I think the authors want to emphasize that they don't use program analysis information such as control and data flow. It would be more accurate to state that as such.\n\n**Learned representations**\nThe paper presents nice visualizations, and examples of methods and their embeddings. However, it is important to highlight in the paper that these representations are learned in the context of a specific supervised task, unlike unsupervised representations, eg, in \"Learning and evaluating contextual embedding of source code\" (ICML'20). Note that those can also be used for code summarization with a decoder similar to this paper or by using other decoding methods such as \"Blank language models\" (https://arxiv.org/abs/2002.03079). \n\nThe paper talks about pre-training on Java-large and Java-medium datasets and fine-tuning on Java-small. What is the pre-training objective? If masking (similar to BERT) is involved, then how are the distance matrices computed and masked?\n\n**Baselines**\nThe GREAT model uses semantic and syntactic relations between tokens (called leaves-only graph). However, this paper uses the ASTs, which is different from the leaves-only representation. First, how exactly the ASTs are used with GREAT is not explained. Second, the results do not imply that using Structure, as defined in this paper, can outperform edge-level representation and relational attention for the edge types considered in the GREAT paper.\n\nAnother baseline considered is Fernandes et al. (2019) on the Java-small dataset. The performance improvement over this baseline is marginal (F1 of 51.4 vs 51.83 with pre-training).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "qIsImVMxo3B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors develop a Transformer model for language-agnostic code summarization. The Transformer is provided both sequential tokenized code and a parsed AST as inputs, and tasked with generating the name of the corresponding code function as output. The authors show that the inclusion of this added structural AST information improves performance on the task, and also improves the cross-language transfer learning abilities of the model. They demonstrate state-of-the-art performance on this task when testing against comparable architectures and datasets. This reviewer believes that the paper is deserving of acceptance to ICLR 2021.\n\nReasons to Accept:\n- The authors take pains to fairly compare their model to prior works, including testing on two separate datasets, and ablating their Pointer Network decoder.\n- The authors achieve state-of-the-art results for the code-summarization task on both the CodeSearchNet dataset (Python, Javascript, Ruby, Go) and the Java-small dataset.\n- The authors demonstrate the surprising extent to which structural information can help with cross-language generalization, particularly for undersampled languages.\n- The authors visualize the embeddings learned by the Transformer, and compare them across languages.\n\nGeneral Issues to resolve:\n- Given the centrality of AST code representations to this paper, there should be some figure showing a sample code snippet alongside its AST in full. This can go in the appendix if need be.\n- Clarify how Ancestor (and Sibling) distance are defined when one node is not a direct ancestor (or sibling) of another node.\n- Add a more detailed explanation to Figure 3, particularly regarding the \"Attention Transformer\" subfigure.\n\nGrammar/Syntax Issues:\n- Bottom of Page 2: \"learn structure (syntax) and *semantics* from scratch\"\n- Bottom of Page 4: $r_{v_1 \\to v_3} = -2$  should be  $r_{v_3 \\to v_1} = -2$\n- Bottom of Page 4: \"PPR is a well-studied proximity measure *which* has\"\n- Middle of Page 13: ['set', 'bottom', 'height'] should be rendered using backticks\n- Middle of Page 13: '\\n' should be rendered using backticks\n- Middle of Page 15: \"the child *has* a distance\"\n- Numbering of Tables goes from 6 to 8, skipping 7\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TGqn-7QjgEX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nThis paper presents a representation of source code based on the AST. By adding positional relational information, such as shortest path length, and ancestor distance the transformers learn to better represent code, without language specific features. On the code summarization task, the model improves on baselines, while training among many languages further improves results. The representations learned seem to share semantic similarities among languages.\n\n#### Overview\n(+) State-of-the-art results for single/multiple language method naming task.\n\n(-) Limited novelty on the machine learning modelling (paper adopts a pre-existing formulation of relational transformers from Dai _et al._ (2019) and Yang _et al._ (2019)).\n\n(-) Only one task evaluated. Does this method offer a language-agnostic representation that generalizes to other tasks on source code?\n\n~~(-) Some additional evaluation would be useful to discern how the proposed model and others compare.~~ [Addressed at response]\n\n#### Comments\n* It is not clear to which extent the representation used here is language agnostic vs. the model is able to learn language-agnostic features: GitHub Semantic (the package used to extract the ASTs in this work) goes into a lot of effort to convert code into a language-agnostic AST format (for its downstream analyses). In that sense, the input AST is mostly language-agnostic, which may let any neural model to be language-agnostic. \nThis suggests that other models that could accept Semantic's ASTs (e.g. [a], [Alon 2019a], [Fernandes 2019]) could also act as language-agnostic models, however, none of these are evaluated in the multilingual setting.\n* An ablation to study the effects of the four different relations considered (Section 3.1) would be useful to understand how much they improve the model (shortest path length, vs. ancestor distance vs. PPR vs. some combinations)\n\n\n#### Clarification Questions\nFinally, I have a few clarification questions (which I think need to also be clarified in the paper):\n* Are the relational encodings an input on all transformer layers or just the first one (like the \"standard\" BERT positional encoding)?\n* The \"context\" has the \"token distance\" is this the absolute (BERT-like) or the relative (like Dai _et al._ 2019, Yang _et al._ 2019)\n* In 5.2 it's unclear what is the representation of each function. Is it the representation of the masked method name token? (the \"encoder output of the method name\" is a bit vague, but suggests that)\n* Are all AST nodes input elements to the \"Structure+Context\" model?\n* How does GREAT differ from the \"context\" model? Is it just the relational transformer (scalar bias) formulation, the 1-step AST relations and the absolute positional encodings?\n\n#### Additional References\n[a] Kim, Seohyun, et al. \"Code Prediction by Feeding Trees to Transformers.\" arXiv preprint arXiv:2003.13848 (2020). \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7u0_66yUWG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There is not much not to like about this paper as it has a simple idea to extend the transformer model. The paper proposed to not only take positional information of each token, but to add additional structural information about distances of tokens in the abstract syntax tree. This positional information gives an edge of this model on several code summarization datasets. The simplicity of the proposed model (assuming it is released by the authors) puts the work in the state-of-the-art category in machine learning for code.\n\nThe paper is also easy to follow and the contribution, while small, is clear and well explained. The related work is also thoroughly covered. While it looks standard, the idea seems to deliver well in the evaluation results. This is also the first work that shows cross-language improvements for programming language models.\n\nIn terms of writing, my main complaint is that the paper can better relate to existing works. For example, “ours without structure” seems to be equivalent to a “plain” transformer with a pointer network in the decoders and using relative distances. It would also help if Table 2 includes a row without the pointer network called a transformer model.\n\nIn the light of this, it also looks like the comparison with the GREAT model is not quite fair. One possibility to fix this is to encode some program analysis edges at least for one language. The program analysis edges are in fact not difficult to build, especially if the authors want to claim they did this comparison correctly.\n\nAs another possible improvement, I find the terms “structure” and “context” confusing and I would suggest the authors use something that directly refers to trees and sequences.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper gives an extension of the transformer model that is suited to computing representations of source code. The main difference from transformers is that the model takes in a program's abstract syntax tree (AST) in addition to its sequence representation, and utilizes several pairwise distance measures between AST nodes in the self-attention operation. The model is evaluated on the task of code summarization for 5 different languages and shown to beat two state-of-the-art models. One interesting observation is that a model trained on data from all languages outperforms the monolingual version of the model.

The reviewers generally liked the paper. The technical idea is simple, but the evaluation is substantial and makes a convincing case about setting a new state of the art. The observation about multilingual models is also interesting. While there were a few concerns, many of these were addressed in the authors' responses, and the ones that remain seem minor. Given this, I am recommending acceptance as a poster. Please incorporate the reviewers' comments in the final version. 
Decision: Accept (Poster)