GN-Transformer: Fusing AST and Source Code information in Graph Networks
Authors: Junyan Cheng
Iordanis Fostiropoulos
Barry Boehm
Venue:  (2021)

Abstract: As opposed to natural languages, source code understanding is influenced by grammar relations between tokens regardless of their identifier name. Considering graph representation of source code such as Abstract Syntax Tree (AST) and Control Flow Graph (CFG), can capture a token’s grammatical relationships that are not obvious from the source code. Most existing methods are late fusion and underperform when supplementing the source code text with a graph representation. We propose a novel method called GN-Transformer to fuse representations learned from graph and text modalities under the Graph Networks (GN) framework with attention mechanism. Our method learns the embedding on a constructed graph called Syntax-Code Graph (SCG). We perform experiments on the structure of SCG, an ablation study on the model design and the hyper-paramaters to conclude that the performance advantage is from the fusion method and not the specific details of the model. The proposed method achieved state of the art performance in two code summarization datasets and across three metrics.

Under review as a conference paper at ICLR 2021
- :RANSSSSFFRM:R: FUSING AST AND SOURCE CODE INFORMATION IN GRAPH NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
As opposed to natural languages, source code understanding is influenced by grammar relations between tokens regardless of their identifier name. Consid- ering graph representation of source code such as Abstract Syntax Tree (AST) and Control Flow Graph (CFG), can capture token's 's grammatical relationships that are not obvious from the source code. Most existing methods are late fu- sion and underperform when supplementing the source code text with a graph representation. We propose a novel method called GN-Transformer to fuse rep- resentations learned from graph and text modalities under the Graph Networks (GN) framework with attention mechanism. Our method learns the embedding on a constructed graph called Syntax-Code Graph (SCG). We perform experiments on the structure of SCG, an ablation study on the model design and the hyper- paramaters to conclude that the performance advantage is from the fusion method and not the specific details of the model. The proposed method achieved state of the art performance in two code summarization datasets and across three metrics.
INTRODUCTION
Code summarization is the task of generating readable summary that describes the functionality of a snippet. Such task requires a high-level comprehension of source code snippet thus it is an effective task to evaluate whether Deep Learning Model is able to capture complex relations and structures inside code. Programming languages are context-free formal language, an unambigu- ous representation, Abstract Syntax Tree (AST), could be derived from a source code snippet. A parse tree based representation of code is precise and without noise. An AST accurately describes the structure of a snippet and relationships between tokens which provides valuable supplementary information for code understanding.
Using graph representations of source code has been the focus of multiple methods that perform code summarization. For example, Alon et al. (2019a) encoded AST paths between tokens and ag- gregated them by an attention mechanism. Huo et al, (2020) used CNN and DeepWalk on Control Flow Graph (CFG) to learn statements representations. LeClair al. (2020) applied Graph Convo- lutional Networks (GCNs) to learn AST representation. These methods proposed different ways to
private cameethuiatttiinggg valus, Wilddat..BVaaae
def hww.tjdirrrrcoom rarnal:
String bretegi bree
"igott part iodiydie, rammet:
Refer
Taee Ge
formatiike nonntt)
the fllee in directery tree.
avit)
for gust, drnu frest walkuur e narms, Beptosere trust I' sear 2W reatt:
generate the hash
refar trigger.annennttiiin
directory.
Tranaformer (Armed
Raderence: senvert atring to an treating emety atr leges defuul value, Gi www-rraaaoomrr (Dursj: converts
irject Fatartow Eh and just search) fill
2000): return hash
returning the provided
The, name paisspeesoooct
dictionary
defuuttva vadua iFttls natl.
wra roformmee
Truraformer jótamud uil 20201: convert sit comvert Int. Tranaformer (tub) jAhrmad 2010) converta string to int. this will be paraad only the string it mell.
addSuppaaehhaarile wotfeoooo) Rowelgest0
opentfiee come, reart)
2010) computer
neturn mab headigea)))
hesh hew hosh by rounsiod.
Figure Examples of generated summarizations on Java (red blocks) and Python (blue blocks) test set. Transformer is a vanilla Transformer, Transformer (full) implements relative positional encoding and copy mechanism from Ahmad al. (022))

---

Under review as a conference paper at ICLR 2021
extract AST features, however the cross modal interaction Veličkovic 2019) is very limited since the AST and code features are independently extracted by separate models then simply concatenated summed.
In this paper we propose a novel architecture GN-Transformer shown in Figure to fuse Graph information with equivalent sequence representation. In summary:
We extend Graph Networks (GN) (Battaglia al. 2018 novel GN- Transffooree archi- tecture that is sequence of GN encoder blocks followed by vanilla Transformer decoder.
We propose a novel method for early fusion of the AST representation and that of a code snippet sequence called Syntax-Covd Graph (SCG)
We evaluate our approach on the task of code summarization and outperform the previous state of the art in two datasets and across three metrics.
Initialize Node Features
GN Transformer Encoder
Fetch token embeddinge
Output
Graph Batch
Embeddinng
Mock
Transformer Decoder
GN- SNN-Trnnffrmma black
108
AssignEapr
Era
Nochex
Edgaas
Embedding
50002: Num
tz
(LUL
50000. ItSnn
1:1ce
NameExpr
Edge block
Node block
Target
50103: =PAD>
Mane
Node embaddings of batch
weecsspprreeegee butch!
Figure 2: Overall structure of our model. Encoder consists of multiple GN-Transformer blocks. We denote '+' as a residual connection followed by normalization layer. In 'Node embeddings of graph batch', each black bar represents the nodes embedding of a graph in the input batch. Blue dots represent token nodes, grey dots denote padding. Nodes embedding in the grey box are fetched as input to the decoder and AST nodes embedding (red dots) are discarded.
We evaluated our model on Java and Python datasets used by Ahmad al. (020). We compared our results to those of Ahmad et al. (2020). Two qualitative results are presented in Figure We make available code, trained models and pre-processed datasets in supplementary package, and we will open-source it after the review process concludes.
FUSING GRAPH AND SEQUENCE INFORMATION
modalities that are processed indepen- dently. For sequence, recurrent architectures such as RNNs, LSTMs, GRUs are commonly used. CNNs have also been applied on sliding windows of sequences (Kim 2014). Transformers (Vaswani
Previous methods consider sequences and graphs
et al. 2017) became a popular choice for sequences in recent years. For graph data, spectrum preserum-based methods like GCNs (Bruna al. 2014) capture graph structure through a spectrum. non-sppetrum methods like GraphSAGE (Hamilton et al. 2017) aggregates information from neighboring nodes using different aggregators, GAT (Veličković et al.) 2018) introduced attention mechanism to ag- gregate neighboring information. Early fusion of multiple modalities is challenging task. As result, late fusion methods are used when considering multi-modal information in code summariza- tion tasks. The cross-modal interactions are less efficient in late fusion as compared to early fusion Veličković (2019). In Section we discuss early fusion approaches of code sequence with an AST. In Section2 2.2 we discuss representing sequence in graph.
2.1 EARLY FUSION OF SEQUENCE AND GRAPH
For early fusion of sequence and a graph, it is common to represent them under a single uni- fied representation and input deep learning model. Random walks (Perozzi et al. 2014) and
2

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
Figure 3: (a) Simplest fully-connected graph representation sequence. Each node corresponds to token. Self- are omitted. (b) Deep blue nodes are AST for the statement 'a[i]=4+2', shallow blue nodes shows the correspondence between AST and source code. (c) Standard SCG structure. It preserves the AST structure with additional edges between AST nodes and tokens.
structure-based traversal (SBT) (Hu et al. 2018a) demonstrated the advantages of flattening graph structure into sequence. SBT converts AST into sequence that be used by seq2seq chitectures. However, structural information contained in the AST is lost when flattened into an unstructured sequence.
The main motivation of previous methods to flatten a graph structure into a sequence instead of the opposite is due the power of sequence models when compared to their graph counter-parts. Recent advances in general frameworks such as Graph Networks (Battaglia et al. 2018), Message- Passing Neural Networks (MPNN) (Gilmer et 2017) proposed unified graphical representation of data. BBtttagia et al. 2018) proposed an extension and generalization of previous approaches that can learn a graphical representation of data under configurable computation unit, GN block. We discuss our GN block based encoder in Section
There are several benefits in graph representation. Firstly can contain different information sources with arbitrary relational inductive bias (Battaglia et al. 2018). Secondly, graph representation can have explicit relational inductive biases among graph entities. Thirdly, flexible augmentation of graph structure of input through expert knowledge. Additionally, better combinatorial generaliza- tion (Battaglia et al. 2018) due to the reuse of multiple information sources simultaneously unified representation. Finally, measures for analyzing performance through graph structures could be naturally incorporated, such as average path length L and clustering coefficient C.. We discuss in better detail in Section 3.2
2.2 GRAPHI REPRESENTATIONS FOR SEQUENCE
In Graph Nerworks, defining the graph structure of input data the main way to introduce relational inductive biases (Battaglia et al. 2018). Relational inductive biases impose constraints the lationship among entities. In Graph Networks, the assumption that two entities have a relation or interaction is expressed by an edge between the corresponding entity nodes. The absence an edge expresses the assumption of isolation which means no relation or direct influence of two nodes.
Sequences are unstructured data in which relationships represented with each token as graph node that is fully connected with all other nodes. Such representation allows each node interact with every other node with no explicit assumptions on isolation. Thus it allows the model to infer the relationships between them implicitly. Transformers could be regarded as inferring on such a fully connected graph, see Figure 3(a). Each token in an input sequence corresponds to node. The relationship between tokens is thus represented by attention weights, high attention values correspond to strong interaction while low attention means isolation.
implicit. Implicit relationships can be
less efficient for a model to learn to infer relationships without any explicit assumptions of interactions and isolation. AST provides precise information about interaction and isolation among tokens in source code since it's a representation without noise on how the tokens interact during the execution of code snippet. Thus natural way of utilizing information from AST is fusing
3

---

apart from Under review as a conference paper at ICLR 2021
the graph structure with input sequence. We find explicit mapping between tokens in a sequence and nodes in the AST through the scope information provided by parser for a given programming language. We are then able to find relations thus build edges between sequence and a graph. We'll further discuss the graph structure of input data and how we fuse AST with it in Section [] The idea of fusing sequence and graph can be extended to broader
AST and code snippet when the mapping is not explicit. There are techniques that are applied in knowledge graphs to find a mapping between entities in knowledge graphs and words in sequence (Peters et al. 2019). For our problem statement, we make the reasonable assumption that explicit mapping between the sequence and graph can be provided.
3 GRAPH STRUCTURE OF AST AND CODE
First, we propose simple joint graph representation of source code and AST called standard Syntax- Code Graph (SCG) Section 3.1 In Section 3.2 we discuss the influence of the structure of SCG and the existence of theoretically optimal graph structure.
170
10
160
aucrogg Poth Longth/
Avg. tedei
250
2.0
130
40
60
100
0.0
0.2
64
0.8
10
BLEU ccone threchold
Clustering dlency (C)
(a)
(b)
Figure 4: (a) Average code length of subsets of samples under different sentence BLEU score thresholds on the test set of Java dataset. (b) The average path length and clustering coefficient of our SCG structures are calculated on the test set of Java dataset and Transformer.
3.1 SYNTAX-CODE GRAPII
Standard SCG consists of AST nodes directly from AST and token nodes created for each code snippet token or simply referred to the remainder of the paper token node. The attribute of an AST node is the type on AST such "NameFxpr"". The attribute of token node is the identifier name such as "a", "int", "+". Standard SCG preserves the AST structure and introduces additional edges that connect token nodes with their direct parent node in AST. The direct parent node for token node found through the scope information for AST nodes. Taking the statement 'a[i]=4+2' as example, the AST is shown in Figure 3(b). The scope of an AST node is determined by the positional mark given by a compiler. The positional mark the AST node 'AdditiveExpr' line I, col 6 ~ line I, col 8 which corresponds to '4+2', the scope covers token nodes '4', '+', and '2'. However, a token may be covered by the scopes of multiple AST nodes, we only connect the token with its direct parent AST. The direct parent of token is the deepest AST node among all AST nodes which scope covered the token. Figure 3(c) shows standard SCG.
Standard SCG directly builds on top of the AST structure without introducing any additional rela- tional assumptions. Thus it objectively reflects the program structure depicted by a compiler.
3.2 OPTIMAL SYNTAX-CODE GRAPH STRUCTURE
The graph structure of SCG plays critical role in our method. You al. (2020) represented neural network structure as relational graphs. Message passing in a relational graph is analogous to infor- mation propagation in Graph Networks. SCG can be directly defined the relational graph of our model. Node features could defined as node embedding in SCG. Likewise, the message function node-wise FEN and the aggregation function is MHA. As result, the graph structure of SCG
4

---

Under review as a conference paper at ICLR 2021
will influence the performance of the model. The standard SCG structure discussed in Section 3.1 is efficient but may not be optimal.
Qualitatively, observed two problems of standard SCG, The first problem is the existence of long range dependencies. It is hard to pass information between nodes separated by long paths in SCG. In Graph Networks, each node is executing an information propagation and aggregation simultaneously with neighboring nodes. For example, One graph neural networks (GNN) layer could be regarded as executing one turn of information propagation and aggregation. In Figure 3(c), it would require three GNN layers for information to propagate from leaf node 'a" to root node 'AssignExpr'. Thus it is difficult to propagate information between nodes a large AST. The second problem is the isolation among token nodes. Source code token nodes can only indirectly interact with each other through AST nodes. Taking '4+2' in Figure 3(c) as an example, there no direct edge between token nodes '4' and '2', they can only indirectly interact through the AST node 'AdditiveExpr'. However, the information that will be passed from 'AdditiveExpr" to '4' and '2' is identical. Therefore, makes it difficult for the token nodes to learn more complex relationships with other token nodes when they are under the same expression. We analyzed the test set of the Java dataset and present the results in Figure 4(a). Longer code usually contains more long range dependencies and complex expressions that correspond to the above two problems.
Hand enginnerinn graph structure with expert knowledge may alleviate these problems thus improve performance. However, introducing redundant edges could be lead to performance degradation. We perform two experiments on opposite sides of the spectrum in our graph structure examine the problems above. Variant 1 added shortcut edges between a code token and AST nodes with the scope covered by that token thus the distance between all AST nodes token nodes within its scope was shortened to 1. Variant 2 makes token nodes fully connected, thus there is no isolation at all. We reference those variants as Variant 1 and Variant 2 in the remainder of this text. More details on the two variants are described in Appendix D Both variants failed to improve performance due to loss of structural information and redundant connections respectively, leading us to conclude that a trade-off of both is required. We discuss it in further detail at Section 5.2
The optimal graph structure was also analyzed through quantitative experiments. You et al. (2020) proposed to measure the effectiveness of the message passing in relational graph by the Average path length (L) and the clustering coefficient (C). The authors claim that the optimal structure is a balance between the value of C' and L. Figure 4(b) shows L and C' for variants, a Standard AST, and that of a culle-connected relational graph such as in a Transformer which could be formulated into relational graph by the same way of ours with different relational graph structure. A vanilla Transformer encoder has a maximum C' of and a minimum T. of 1. Our standard SCG, at the other extreme, has minimum C' of 0 and relatively high I. due its tree structure. The average L of standard SCG in Java and Python dataset is 6.28 and 6.35 respectively. Figure 4(b) shows that we only explored small area of the entire design space of the possible graph structure for various L and C. You et al. (2020) proposed that the optimal structure is usually located around a "sweet spot" in design space between extreme cases of the tree and fully-connected structures. Model performance will improve nearing this "sweet spot". Thus we conclude that there is large potential of improving model performance by improving the input structure either through hand-engineering or additional rules tailored for the specific problem domain.
4 MODEL ARCHITECTURE
Our model follows the generic Transformer encoder-decoder architecture. The encoder is ex- tended architecture of Graph Networks with multiple GN blocks. The decoder is a vanilla Trans- former decoder. The overview of our architecture is presented in Figure 2. We'll introduce our overall structure in Section 4.1 Then propose our Graph Networks based encoder in Section 4.2)
4.1 ENCODER-DECODER ARCHITECTURE
The encoder consists of a stack of N Transfooffrerr blocks which are derived from GN blocks, the main computation unit in Graph Networks, it takes a graph as input and returns graph as output (Battaglia et al. 2018). Each block in our model implements a multi-hea head attention (MHA) sublayer and a feed-forward network (FFN) sublayer equivalent to Transformer encoder layer.
5

---

Under review as a conference paper at ICLR 2021
Encoder accepts graph 9 with the node features h as input. {V, E} where V is node set and E is the edge set. h E pdlixdmoded are the node features in G. where dmodel is the imput and output dimension of the encoder. The node features in the imput graph are initialized through an embedding layer. AST and token nodes fetch embedding vector according to their type and identifier names respectively. For our implementation, we handle token code identifiers and AST node type separately when performing an embedding lookup, as avoid naming conflicts between the two. Encoder outputs the graph with the updated node features h'. Feature vectors of only token nodes are fetched the decoder input. Feature vectors of AST nodes discarded and the token embeddings are padded for batching (see Figure 2). Residual connections and layer normalization are used in Transformers and are also employed within each block in our model. Moreover, due to equivalence in network architectures, parameters number of our model is the same as the vanilla Transformer with the same configurations.
4.2 GN-TRANSFORMER BLOCKS
We extend GN blocks proposed by BBattaliia al. (2018) and define MHA and FFN in this context and we call it GN-Transformer block. One GN- EN-Transformer block will execute one round of information propagation and aggregation with the neighboring nodes that updates the node and edge attributes on a graph. This is done by two sub-blocks an edge block that updates edge attributes, then node block that updates node attributes. Notice that for our problem the initial edge attribute matrix of a SCG is the presence of connection between two nodes.
Edge block E is updated by an edge update function (h,,, hy) = h. (h, where h; and hy
Va.
are attributes of node i and j respectively and EX( is the updated attribute of edge from node to through attention head y. Each EN-Transformer block has H attention heads. W E I dmoded xdy and W IR IRdmmdet are parameter matrices for attention head %
And an edge aggregate function pE Concat(headd" : ..., head head!")) w with head EjEN, h, a which aggregates updates of edges with a (EE(T))) )j- the attention from
node to the set of neighboring nodes No. a-)) is softmax function, wy E Rmoddee xdv Wo
R Hdd, xdmedel are parameter matrices.
A Node block V updated by a node update function d" d' "0) = FFN(h)) + h4 where h h4 pf- and FFN(2) (x) maz rW, + b2) b2) b2-
6f together with pf > define the MHA. o" implemented a nose-wise FFN with residual connections similar to wosnt-wise FFN in a Transformer. For our experiments, also use dropout and layer normalization the same as Vaswani et al. (007)).
5 EXPERIMENT
We evaluated our model in two code summarization datasets. Additionally, we perform experiments on the hyperparameters, model structure, and variants of the graph structure. Our experiment settings are presented in Section 5.1 Results are presented and analyzed in Section 5.2
5.1 SETTINGS
The experiments are conducted on a Java dataset (Hu et al., 2018b) and Python dataset (Barone & Sennrich, 2017). We used JavaParser to extract the AST and javalang for parsing Java source code, python ast to parse and get the AST for Python.
We chose Transformer as our main comparison baseline which achieved state of the art in the two datasets by Ahmad et al. (2020). Ahmad et al.] (2020) proposed a base model which is a vanilla Transformer and a full model that implements relative position embedding and copy mechanism.
https::/havaarrer..orr

---

Under review as a conference paper at ICLR 2021
Table 1: Overall results on Java and Python datasets.
Java
Python
Models
BLEU
METEOR
ROUGE-L
BLEU
METEOR
ROUGE-L
DeepCom
39.75
23.06
52.67
20.78
9.98
37.35
Dual Model
42.39
25.77
53.61
21.80
11.14
39.45
Transformer
43.99
26.40
53.30
31.22
18.56
44.22
Transformer (full) 6
44.84
26.89
54.80
32.79
19.63
46.51
GN- ANnTrrasformer (Ours)
45.48
26.91
55.29
33.31
19.66
46.56
We reproduced their results on our preprocessed datasets. Our preprocessed datasets are composed of the source code, corresponding AST, and text summary. We also compared our method with the results of other baselines reported in Ahmad et al. (2020). Additional details about the datasets and preprocessing are in Appendix B We used the same metries that are reported by all baselines to evaluate our model which are BLUE, ROUGE-L, and METEOR. We applied the same hyper- parameters as Ahmad et al. 20200, they are listed Appendix A.
5.2 RESULTS AND ANALYSIS
Results on Java and Python datasets. Table shows comparisons of our model GN- Transfforrrr with previous works code summarization. Our method outperformed all previous works in all metrics. The most suitable comparison of our approach from all previous works is that of a vanilla Transformer rather than Transformer (full). The only difference between our work and vanilla Trans- former by Ahmad al. (2020) is the GN-rransformer Encoder block and the SCG structure, which the scope of this paper. In contrast Transformer (full) makes use of positional embedding and copy mechanism introduced by [See et al.) (2017); [Shaw al. (2018). Such improvements apply to all sequence models irrespective of the domain but they do not directly apply to a graph struc- ture. Analogous techniques introduced Graph Networks should correspond to improvements in our model. However, we consider this to be outside the scope of this paper. Our experiments on positioning encoding and copy mechanism are presented in Appendix
We applied the same hyperparameters and network configurations with the vanilla Transformer as proposed in Ahhmad al. (2020). The result shows an improvement of 1.49, 2.09 BLEU, 0.51, 1.10 METEOR and 1.99, 2.34 ROUGE-L in Java and Python dataset respectively. Comparisons with more baselines are shown Table in Appendix
Additional experiments on Java dataset We conducted additional experiments in the following three directions on the Java dataset. Results are shown Table Appendix D provided more details about additional experiment settings.
(A) Hyper parameters. Experiments on different combinations of hyper parameters including the number of layers, embedding size, width of FFN, and configurations of attention heads.
(B) Model structure. We firstly tested a 2-hop GN-Transformer block by using two MHA sublayers in each block thus aggregating two hops of information. Result shows that it harms performance. Collecting two hops of information without a FFN in between is less expressive. We then did an ab- lation study to show the advantage of the GN-aransformer blocks. We replaced GN-Transformer block with Graph Attention Networks (GAT) layer with the same hyperparameters as that of a base model. GAT did not perform optimally in our problem when modeled text sequence Result shows that the GN-Transformer block largely outperformed GAT even when we compared configurations with similar parameter numbers as (A). Then we added FFN to GAT layers for an ablation study on MHA. This was to see the impact of MHA. Results show that MHA brings a sig-
graph.
Hu et al. 2018a
Wei ct all
2019a
Ahmad
2020
7

---

Under review as a conference paper at ICLR 2021
nificant improvement. We thus conclude that both the FFN and MHA used by our -N-Transformer blocks are necessary and greatly improve performance.
(C) Variants of graph structures. We tested two variants of graph structures discussed in Section 3.2 Both variants underperform with Variant I being much closer to a SCG when compared to Variant 2. Variant underperforms because it introduces redundant edges and leads redundant interac- tions between the AST and tokens. It degrades performance little since the structural information and isolation among tokens introduced by the AST were preserved. For Variant 2, the AST code sequence tokens are fully connected. Isolation among tokens is completely lost which leads to the loss of structural information.
The above experiments explain why Ahmad et al. (2020) did not improve performance when using SBT (Hu et al., 2018a). The AST nodes are fully connected. The structural information from the AST graph is not preserved, similar to Variant 2. Moreover, SBT introduced redundant interactions between AST nodes and token nodes similar Variant However, Variant 2 still achieved 0.43 BLEU, 0.47 METEOR, 0.78 ROUGE-L improvements compared to a vanilla Transformer which shows the usefulness of AST. From our ablation study on the graph structure variants, we conclude that the cause of degradation is the redundant edges and the loss of structural information.
RELATED WORKS
Integrating valuable information from graph representations like AST. CFG, PDG has long been the focus of deep learning in code understanding. Alon et al. (2019b) retrieves AST embeddings with random walks then concatenates them with token embeddings and finally aggregates them by an attention mechanism as a context vector. Hu et al. (2018a) proposed Structure- Based Traversal (SBT) that flattens the AST into sequence. |Huo et al. (2020) applied DeepWalk with CNN and LSTM to learn a CFG representation which then concatenated with the source code representation. LeClair et al. (2020) used GCN to learn AST embeddings then concatenated them with source code embeddings. Wang et al. 2020) augmented AST features by adding edge type information on AST that representing the control and data flow. then applied gated GNN to learn this augmented-AST embedding. Most existing methods fuse AST and code information by late fusion (Baltrušaitis et al. 2019) that concatenates embeddings of the different modalities with few interactions of cross- modal information. Veličkovid (2019) investigated the possibility of early fusion. They proposed cross-connections between models to enable the sharing of cross-modal information. TrecLSTM Tai et al. 2015) fused tree information into sequence by a tree-structured LSTM. Another category of methods modeled information sources as heterogeneous graphs that are then processed by GNNs. Yao et al. (2019) constructed heterogeneous graphs of documents and text. Ren & Zhang (2020) modeled text as a heterogeneous graph of topics and entities.
Research on general deep learning frameworks proposed a method to train on uniform represen- tation and model framework (Battaglia et al., 2018 Gilmer et al. 2017] Wang et al. 20188). Graph Networks unified deep learning models on powerful graph representation which could represent ar- bitrary relational inductive biases between entities (Battaglia et al. 2018). On the other hand, You et al. (2020) reveals the relation between the graph structure and neural network structure which provided a more comprehensive view on the impact of graph structure in deep learning models.
7 CONCLUSION
In this paper, we analyzed the fusion of a sequence and graph from a novel perspective of graph networks. We proposed our GN-Transformer with Syntax-Code Graph. Our method achieved state of the art in two code summarization datasets. We also did experiments on hyperparameters, model structure, and graph structures. Future works include finding the optimal structure of SCG and fus- ing supplementary information like CFG. Due to the similarity with Transformer, ideas like masked pretraining, positional encoding, copy mechanisms for Transformers also worth interpreting and implementing in the context of Graph Networks. A decoder designed for graph representation may also improve performance,
method discarded AST node embedding while they naturally con- tained additional structural information which should be useful to the decoder. The simplicity of our method could expand in other domains that there is duality in sequence and graph representations.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. A transformer-basee approach for souree code summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from struc- tured representations of code. In International Conference on Learning Representations, 2019a. URL httphttts////ppeeeeeeee net. forue//orumma 1treee//oooummmi0Hogoo000033
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. Code2vee: Learning distributed repre- sentations of code. Proc. ACM Program. I Lang., 3(POPL), January 2019b. doi: 10.1145/3290353.
URL httpp::///eet //ts::://doo.ee http://..........0......... 2t10///....1//39035355555
T. Baltrušaitis, C. Ahuja, and L.. Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423 443--44331111 2019.
Antonio Valerio Miceli Barone and Rico Sennrich. A parallel corpus of python functions and doc- umentation strings for automated code documentation and code generation. In The 8th Interna- tional Joint Conference on Natural Language Processing (IJCNLP 2017), volume 2, pp. 314-319, December 2017.
Peter Battaglia, Jessica Blake Chandler Hamrick, Victor Bapst, Alvaro Sanchez, Vinicius Zam- baldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song. Andy Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Jayne Langston, Chris Dyer, Nicolas Heess, Daan Wier- stra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Re- lational inductive biases, deep learning, and graph networks. arXiv, 2018. URL https:
//arxiv. wtggpd// 01.61.
Joan Bruna, W. Zaremba, Arthur Szlam, and Y. LeCun. Spectral networks and locally connected networks on graphs. CoRR, abs/1312.6203, 2014.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 1263-1272. JMLR.org, 2017.
William L. Hamilton, Zhitao Ying, and J. Leskovec. Inductive representation learning on large graphs. NIPS, 2017.
X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin. Deep code comment generation. In 2018 IEEE/ACM 26th International Conference Program Comprehension (ICPC), pp. 200-20010, 2018a
Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. Summarizing source code with transferred api knowledge. In Proceedings of the Twenty-Seventh International Joint Confer- ence on Artificial Intelligence, IJCAI-18, pp. 2269-2275. International Joint Conferences on Artificial Intelligence Organization, 7 2018b. doi: 10.24963/jjca..011 20ticaii..1//1144 8/3144
URL https:
7/doi. org 10. 24922429433333 http:///11aaa1111/111.1 2018/314
Xuan Huo, M. Li, and Zhi-Hua Zhou. Control flow graph embedding based on multi-instance decomposition for bug localization. In AAAI, 2020.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Summarizing source code using neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2073-2083, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10.18655. 1/11116/////111119/99 URL https:
7/ //ww.. ac www.wwww.bbb.. o ant http://w.w...r.oooooooooo///////
Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP. 2014.
Alex LeClair, Sakib Haque, Lingfei Wu, and Collin McMillan. Improved code summarization via graph neural network. In 2020 JEEE/ACM International Conference on Program Comprehension, Oct. 2020.

---

Under review as a conference paper at ICLR 2021
Bryan Perozzi, Rami Al-Rfou, and S. Skiena. Deepwalk: online learning of social representations. In KDD "14 2014.
Matthew E. Peters, Mark Neumann, IV RogertLLogan, Roy Schwartz, V. Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In EMNLP//JCNLP, 2019.
Yuxiang Ren and J. Zhang. Hgat: Hierarchical graph attention network for fake news detection. ArXiv, abs/2002.04399 003977, 2020.
Abigail See, Peter Liu, and Christopher Manning. Get to the point: Summarization with pointer- generator networks. In Association for Computational Linguistics, 2017. URL https:
04368
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464 468, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 1..18633/1/N118-277 URL https: www wwe... org/ httpaaoooooo//11-22000
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representa- tions from tree-structured long short-term memory networks. In Proceedings of the 53rd An- mual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 1556-1566 Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/115-1155 11500 URL https: / www ac web. org/ antho waatthooogy/Ps11.. ogtpp15-1155
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, un- definedukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017, Curran Associates Inc. ISBN 97815 108666966
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
accepted as poster.
Petar Veličković. The resurgence of structure in deep neural networks. PhD thesis, 01 2019.
Y. Wan, 7. Zhao, M. Yang, G. Xu, H. Ying, J. Wu, and P. S. Yu. Improving automatic source code summarization via deep reinforcement learning. In 2018 33rd IEEF/ACM International Conference on Automated Software Engineering (ASE), pp. 397-407, 2018.
W. Wang, G. Li, B. Ma, X. Xia, and Z. Jin. Detecting code clones with graph neural network and flow-augmented abstract syntax tree. In 2020 IEEF 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 261-271, 2020.
X. Wang, R. Girshick, Gupta, and K. He. Non-local neural networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7794-7803, 2018.
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. Code generation dual task of code summa- rization. arXiv preprint arXiv: 1910.0090. 2019a.
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. Code generation as dual task of code summariza- tion. NeurlPS, pp. 6559, 2019b.
Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7370-7377, 2019.
Jiaxuan You, Rex Ying, and J. Leskovec. Position-aware graph neural networks. In ICMI., 2019.
Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks. In
37th International Conference on Machine Learning (ICML), 2020.
10

---

Under review as a conference paper at ICLR 2021
A SUMMARY OF HYPER-PARAMETERS
Hyper-parameters
Value
Num of layers
6
Attention heads
8
64
d,
64
dmoode
512
Hidden units in FFN
2048
Dropout rate Optimizer
0.2
Adam
Initial learning rate
0.0001
Decay rate
0.99
Max epoch num
200
Early stop epochs
20
Training Batch
30
Testing Beam size
4
Max src. vocab size
50000
Max tgt. vocab size
30000
Max Java training code len.
150
Max Python training code len.
400
Max Java training summary len.
50
Max Python training summary len.
50
We used the same apper-parameters as (Ahmad et al. 2020). For the data preprocessing, the code and summary data will be truncated if they exceed the maximum length. We will first truncate the tokens that exceed the max sequence length. For all AST nodes, if the scope contains a truncated token, this AST node will be truncated. We removed all nodes that have scope which contains truncated tokens. We set a vocabulary size limit, we store only the highest frequent words. Words that are in the vocabulary will be recogmized Unknown_word Our methodology con- sistent with Ahmad et al. (2020) with the exception of truncating AST since they do not use that information.
B DETAILS OF DATASET AND PREPROCESSING
Dataset
Java
Python
Examples Train 69593
64939
Examples Validation
8694
21605
Examples Test
8689
21670
Unique Function Tokens
66569
104839
Unique Summary Tokens
46859
64898
Avg. Function Length
120.29
132.64
Avg. Summary Length
17.73
9,56
Avg. AST Nodes
50.30
70.10
Avg. AST Edges
45.89
68.16
The statistics of our preprocessed datasets are shown above. Despite the difference in implementa- tion, we kept our methodology consistent with Ahmad et al. (2020). We used the same CamelCase and snake_case tokenizer from Ahmad et al. (2020) to preprocess source code data in both Java and Python datasets. SCG for subtokens is a little bit different, see Figure 6. We also replaced the strings and numbers in source code by *(ST R) and "(UM)) For the code summary, raw corpus for the Java dataset, and used the same method with Wan et al.) (2018) to process code summaries in the Python dataset. We used the train/valid/test split from the original corpus for the
used

---

Under review as a conference paper at ICLR 2021
Java dataset and split the Python dataset by 6:2:2, All of the above consistent with Ahmad et al. (2020).
There two differences between preprocessed dataset Ahmaa et al. (2020):
1. Data cleaning: We discarded the samples that cannot be parsed by the compiler. There are 160 out of 187136 136 and 4894 out of 13108 samples in Java and Python dataset that discarded respectively. In the Java dataset, there are no samples removed by Ahmad et al. (2020). For the Python dataset, Ahmad et al. (2020) follow the same cleaning process by Wei et al. (2019b). They removed the samples that exceed length threshold. They remove 20563 out of 1 113108 samples.
2. Python Dataset: We used the method for Python
did for the Java dataset. Ahmad al. (2020) deleted special characters the Python source codes while we preserved them. The result is that the average code length of the dataset preprocessed by them is 47.98 while ours 132.64.
Apart from the above two differences, the preprocessing methodology is consistent with Ahmad al. 2020) as well as other baselines reported by them.
C ADDITIONAL EXPERIMENT RESULTS
Table 2: Other baselines on Java and Python datasets.
Java
Python
Models
BLEU
METEOR
ROUGE-L
BLEU
METEOR
ROUGE-L
CODE-NN
27.60
12.61
41.10
17.36
9.29
37.81
Tree2Seq
37.88
22.55
51.50
20.07
8.96
35.64
RI.+Hybrid2eqq
38.22
22.75
51.91
19.28
9.75
39.34
API+CODE
10
41.31
23.73
52.25
15.36
8.57
33.65
GN- ON-TTrnsformer (Ours)
45,48
26.91
55.29
33.31
19.66
46.56
The comparison between method and additional baselines shown in Table 2. Table 3 showed the results of additional experiments. Table 4 showed experimental results on positional encoding and copy mechanism.
For experiments on positional encoding. we applied a learnable absolute positional embedding (APE) layer, which is often used Transformers. We then the summation of APE and node embeddings fetched from the input embedding layer as the input to the encoder. We tested APE only on token nodes and used padding AST nodes.
For the experiments on relative positional encoding (RPE) on our model, we had to adapt the original definition for sequence models that of graph representation. When RPE applied to sequences requires the sequence nodes to be fully connected, which is not the case for SCG. Instead, we applied two-layered PGNN You al. 2019 that learns relative positional information on graph to Iearn a RPE for each node. We used an APE as the input to PGNN. We set an embedding size of 512. We applied the fixed number of anchor sets with copies each instead of an adjusted anchor set number in You et al. (2019). We concatenate RPE and node embeddings fetched from the input embedding layer. We then reduce its dimension dmoodet by a linear layer and provide that as input to the encoder. Next, we tested applying RPE only token nodes by replacing the RPE for AST nodes by padding. For the copy mechanism, we tested the same copy mechanism as Ahmad et al. 2020). All hypeparrameters are the same and listed in Appendix A
Iyer et al.
2016
Tai et al.
2015)
Wan ct al
2018
Hu al.
20185
12

---

Under review as a conference paper at ICLR 2021
Table 3: Additional experimental results. Unlisted values identical the base model. Parameter number not including the embedding layer.
params
N
dmodel
dff
H
dg
dy
BLEU
METEOR
ROUGE-L
(x 10%)
base
512
2048
8
64
64
45.48
26.91
55.29
44.1
2
37.37
20.66
49.20
14.7
43.41
24.86
53.62
29.4
45.76
27.18
55.59
58.9
512
512
43.13
24.77
53,24
4
128
128
44.79
26.20
54.59
(A)
16
32
32
45.52
27.00
55.39
32
16
16
45.50
27.11
55.49
256
40.01
22.35
51.16
22.1
1024
46.19
28.02
55.86
88.2
1024
43.99
25.40
54.34
31.5
4096
45.82
27.63
55.71
69.3
2-hop GN- Transoosfrrrrr block
43.90
25.24
53.49
50.4
(B)
GAT
38.51
21.14
48.39
26.8
GAT with FFN
40.60
22.71
50.05
39.4
Variant 1
45.25
26.56
55.11
(C)
Variant 2
44.42
26.87
54.08
Table 4: Experimental results positional encoding and copy mechanism. APE/RPE (token) means only applied APE/RPE on token nodes. Parameter number not including the embedding layers.
Java
Python
params (x10')
BLEU
METEOR
ROUGE-L
BLEU
METEOR
ROUGE-L
base
45.48
26.91
55.29
33.33
19.67
46.57
44.1
APE
44.18
25.68
54.41
29.40
17.08
43.17
44.1
APE (token)
44.42
25.81
54.59
31.40
18.37
45.10
44.1
RPE
45.32
27.08
55.33
32.49
19.62
46.30
46.2
RPE (token)
45.07
26.58
55.12
31.55
19.09
45.70
46.2
Copy mech.
45.03
26.50
54.91
33.04
19.64
46.59
44.1
The results show that APE will harm the performance of both Java and Python dataset. When applying APE only in token nodes, the degradation is minor. We hypothesize that this could be because APE not useful for AST nodes, APE in token nodes represent their absolute positions in the input sequence. On the contrary, the positions for AST nodes in the input sequence should be the scope instead of single absolute position. The absolute positional information is not useful in standard SCG for the AST nodes.
The results for RPE are also not promising in the interpretation we made for graph representations. We chose fixed number of anchor sets for all graphs. While in You et al. (2019), they dynamically choosing anchor sets number for different sizes of graphs. Our anchor sets number may be inade- quate capture accurate relative positional information in large graphs and it's revealed by the fact that the average node number of the Python dataset is 70.10 compared to 50.30 in the Java dataset. This reflects on the results since we obtain improvement for some metries in the Java dataset while performance degradation for Python.
Both positional encoding and copy mechanism did not improve the performance across all met- rics but were close to the base model. We hypothesize that both mechanisms have shortcomings when applied for graphs and SCG. Our results show we can not apply tricks that are designed for Transformer in model without further consideration of the problem domain and graph modality. The ideas found in sequence models can still be designed to fit the graph representation but require additional investigation.
13

---

Under review as a conference paper at ICLR 2021
D DETAILS OF ADDITIONAL EXPERIMENTS
Here we showed more details about our additional experiments discussed in Section 5.2
For additional experiments (B), We used the GAT layer implementation from DGI [T The difference between MHA GAT and GN- Transfforrrrr that GN-Transformer used the multiplication of features of two nodes in edge to calculate edge feature while GAT used summation in DGL's implementation. Secondly, Transformer applied four linear layers on key, query, value, and output of MHA respectively while GAT only used one input features.
(a)
(b)
Figure 5: (a) Introducing shortcut edges for AST node 'AdditiveExpr' Variant 1, the three token nodes '4', '+', '2' will connect with it. We marked token nodes by shallow blue, AST nodes by deep blue, edges from token nodes to AST nodes by red, orange for the converse direction. (b) In Variant 2, all token nodes are fully connected with each other.
For additional experiments (C). In Variant I, we solve the long range dependence problem discussed in Section 3.2 by introducing shortcut edges between each AST node and the token nodes that are related to this AST node. The distance between AST nodes and all nodes within its scope are shortened to 1. It doesn't break any isolation but introduced additional direct interactions. We define the related token nodes of an AST node as all token nodes within scope of the AST node. Figure 5(a) shows how shortcut edges are added for the 'AdditiveExpr" node, add bi-directional shortcut edges between the AST node 'AdditiveEppr' and token nodes '4', '+', '2' which are within the scope of this AST node. The edges allow for the AST node propagate information through the node and edge update rules. This can ameliorate the long range dependence problem in the graph. In the second variant, make the token nodes fully connected the Transformer. The nodes not isolated from each other anymore. This causes all nodes to interact with each other. See Figure 5(b).
E EXAMPLES GENERATED SUMMARIZATION
Here presented ten examples of the summarization generated by model and the baselines on the test set. The first five ones are Java examples, the remaining five are Python examples.
public boolean selselected (ItemSelectionChoice {-choice) return sel array [p choice. ordinal 0
Reference: looks, if the input item type selected.
:N-TTranffrmer: whether a choice is selected, or not
Transformer: get if the selection should be selection
httpps:/ htt.///wwwwee www.dgll.i.
14

---

Under review as a conference paper at ICLR 2021
ST
pr
Camenine
(a)
(b)
Figure 6: Example of building connections for subtokens. Edges between other tokens and AST are omitted. The figure also shows how shortcut edges are connected subtoken nodes. (a) Connection for original token node. (b) Connection for subtoken nodes. All subtoken nodes copied the same edges from the original token node.
Transformer (full): is selection find possible
private static hoolean aqualsHandeesNllls (Object a. Object b){ return = b (a t= null && a.equals(b));
Reference: returns true if a and b are equal or are both null.
G--Transformer: returns true if two possibly null objects equal.
Transformer: tell whether two objects are equal.
Transformer (full): checks equality of two objects. returns true if one possibly null.
public static String formalLis(ToString ( (List String list {
String result=
(String s list ) result 4=s + "(A)""
r trim
return result toUpperCase();
Reference: formats the list objects from the pokemodel into formatted strings that are easily read- able.
GN-Transformer: converts a list of strings to upper case strings.
Transformer: returns comma separated list of strings.
Transformer (full): encloses a list of strings from standard separator
public final boolean isExceptinnHandeeeEqqqilllln (BasicBlock other {
if (exceptionHandlers != other. exceptionHandllers)
Enumeration <asicBlook>>> Enumerratoon-BBsiclllccc>>>>>>>> el=geeExceppionHHandless
Enumeration Baaierrttioon/BiiiBlllooee. e2=other Heaooeerege..cceeppimannnnnngs
while (el. hasMorrellementt())
15

---

Under review as a conference paper at ICLR 2021
if (le2. ....aMoMoelEleeetts))) return false:
(el nextElement() != c2. nexeelleemm))) return false;
if (c2. ho..hMoMollllmnntt)) return false ;
true;
Reference: compare the in scope exception handlers of two blocks.
GN-Transler :N-Trrastormer: return true if the block is equivalent to the exception handlers, false otherwise.
Transformer: determines whether this exception has an exception?
Transformer (full): returns true if this instruction is exception.
protected static void quickSort Instances insts int indices int attidx int int right { if (left < right) {
int middle= partition insts indices, attidx, left right) ); quickSort ( insts indices, attidx left middle); quickSort insts indices, attidx middle right);
sorts the instances according to the given attribute/dimmnsionn. the sorting is done on the master index array and not on the actual instances object.
Reference:
GN-Transtormer: sorts the specified range of the array using the specified items
Transformer: src the ordinal field array into ascending order
Transformer (full): sorts the specified range of the array using the given workspace array
def is ..uoer2(nuum):
return isinstance (num, numbers. Integral) (num > 0) and (not (num (num 1)))
Reference: test num is positive integer power 2.
GN-Transformert return true if the power of
Transformer: returns true if and number a user-defined power.
Transformer (full): return whether or not the argument is a power.
delete (queue, items. backend= sqlite '): junuc_funes = salt loader . queues( _opts...
cmd= = (0). delete
(bankenb))))))
if (emd not in (neue_funcs):
raise SaltInvocationEErro ('Function "(0)" is available (omd))(cmm)) ret qqueueefuncssccmdfiteees-ii return ret
queue=queue)
Reference: delete an item or items from a queue cli example:
GN-Transformer: delete one or more or all items from queue cli example:
Transformer: delete an item from a queue cli example:
Transformer (full): delete message(s) from queue cli example:
16

---

Under review as a conference paper at ICLR 2021
def issues closed since ppriod=timmddeta (days=365), project jpplect=ippyhoonippyhhnn pulls =False): which ( pulls pulls else "issues")
if isinstance (period. timedelta): since found_hourte datetime utcnow() period))
else:
since period
=(' httpp api github. comdrepos//////sssttttteeee http://swss.ttttooooooooooeeeenn pageo=/
% (project, which, since. strftime (ISO8601), PER PAGE))
allclosed paged (url, headers=make auth headde(e)))))
filtered [i for in allclosed if (-parse.datetime
datetime (if' closed
since)]
if pulls:
filtered [i for i in filtered if ( (-parse parse.datetima (i [e'mmrred_aa at') }) > since)]
filtered = [i for i in filtered if (il [[["base] ][ ref"] == "master"))
clse:
filtered for i in filtered if (not reqqeet (i))
return
filtered
Reference: get all issues closed since particular point in time.
GN-Transtormer: return a list of closed issues
Transformer: find/download for constructed tol by several /viewfinder convention.
Transformer (full): return list of closed since promotions time.
def pporgeOptionssinpuutttttioi overrideOptions) if inputOptions. pickledOptions:
try:
inputOptions base64unpickle (inpuuuui
except Exception as ex:
errMsg = ("provided invalid value "%s" for option -- pitionn %
inputOptions, inpurtOptiooss.iiklllltpttpttt pickledOptions)
errMsg += (("("ss")" "" ex) ex. message else ??)
raise SyllnnSSytaxxxxcciiinnnnnggg
if inputOptions. configFile
configFilcParser (inpurOptions. configFile
if hasattr (inputOptions. 'items")
inputOptionsItems inputOptions. items ()
else:
inputOptionsstemss inputOptions. dict .items ()
for (key, value) in inputOptionsitems:
((key not conf) or (value not (None, False)) or overrideOptions)> ):
conf[key] = value
for (key, value) in conf. conf.itemsi ():
if (value is not None):
kb. explicitSettings .add(key)
for (key, value) defaults iitems
hasattr (conf, key) and (conf[key] is None)):
conf[key] value
=()
for (key, value) in os. environ. items (): key. startswith (SQLMAP ENVIRONMENT PREFIX):
[kee|Iee ENVIRONMENT PREFIX):).upper() = value
types. = ()
for group :ey:::
uppes..udatte. o
for key in conf:
if ((key upper() in and (key in types.)):
value = [_kkeyupperr)
if (types [key] OPTION TYPE:BOOLEAN:
17

---

Under review as a conference paper at ICLR 2021
uy:
value bool
except ValucError:
value False
elif (types_ [key] = OPTONNYVPE.NTEGR)
tr:
value int (value)
except ValueError:
value =0
elif (types OPTION.TYPE.FLOTT.
try:
value float (value)
except ValucError:
value 0.0
conf[key] value
mergeel@Opiions.uddaaleccocmmm
Reference: merge command line options with configuration file and default options.
GN-Transformer: merges options from a config file.
Transformer: merges all of the data used into option.
Transformer (full): loads configuration attributes and add attributes..
def LoadFromSStrigg(yaml product-yaml.key required.clienttvalles. optional,produuct.vlluss ):
if (_PY_ VERSION MAJOR 2);
(.PY_VERSION MINOR = 7) and WWW_VVERIONN.IE < 9));
Jogger, warning( (EERREATEEDDDDDD BEEECATE.VEEEIIOO VERSION, TEMPLATE, _______VEEEEEE CERRSION-MAIR. .PY PY_VERSION _IIOR, PY_VERSION (MRSION-MICRO) elif (.PY_VERSION MINOR 7):
.ogger warning( DEPRECATED. VERSION TEMPLATE, P...P.VVVE... VERSION MAJOR, PY VERSION MINOR, PY V.VEIIION MICRO)
data = (yaml. safe load (yaml doc) or {})
try: :
product.data dadalppoouut.yymll.ky
except KeyError:
raise googleads. errors GoogleAAsVVllueFrrooo "%s" configuration is missing' %
(product.yyn) yaml k))
if (not isinstance (roduuct data dict )):
raise googleads. errors VoogeeeAdVVlluEEooo(( The "%s" configuration empty or invalid & (product.yamll_ey_.)
ValueError(("
UebluduilliiessmUssrAA... (data, (JUTILITY L REGISTER, YAML KEY, Truc)) original.keys list product-data () client_kwarg { try:
for key in sequired_client_vall
client kwargs [key] product data [key] del product data [key]
except KeyError:
raise googleads. emors ValueEroo Some of the required values are missing.
Required values are: 56s, actual values are %s' % required__lientt.all values
original.keys )))
proxy_config. data data, aet[_POXX CONFIG KEY, !))
proxy_come config EExtrractrroyyoonnggg (product yaml_key, proxy_coo mroxy_confi.ddate
client kwargs [' proxy proxx_config' ]I proxy .onfig
client kwargs [' oliett = ExtractOAuth2CCient (product-yaml key, product.date
proxy proxy_config)
kwargs [ENABLE COMPRESSION KEY] = data.get(ENABLE COMPRESSION KEY, False) for value in if (value in product_data)) ):
client_kwargs value] product_data [value]
del product data [value]
18

---

Under review as a conference paper at ICLR 2021
if product.data:
warnings. warm(( Could recognize the following keys: %s. They were ignored. (product.data ) stacklevel =3)
return kwargs
Reference: loads the data necessary for instantiating a client from file storage.
GN- :NNTTrasformer: loads the header data necessary for instantiating.
Transformer: loads data necessary for credit types
Transformer (full): loads key: data from loader_context.
F ATTENTION VISUALIZATION FOR ENTAX-CODE GRAPH
The attention visualization for this short program is shown here, we could get more insights about how each nodes paid attention its neighboring nodes:
public StartListener (Object resouree) resource rresuree=;
The first figure attention visualization for standard SCG. We marked the edges from token nodes to AST nodes by red, edges from AST nodes to token nodes by black, edges within AST by orange and red. The remaining figures are attention visualization based on Variant Each figure shows the attention visualization of the nodes in each layer of AST. The edges from token nodes to AST nodes are marked by red, edges from AST nodes to token nodes by black, edges within AST nodes by orange.
Constructon
Block
Exp es crntt
Fam otter
Adil Ad pr
enterfichyyp
19

---

Under review as a conference paper at ICLR 2021
Constructor Derlattion
DocckSunnt
Express
Param
Aetignnan
N
NameExte
Clasoo
coti
Variab
Dediaratard
N
Hameak eee
qublicc
Object
rssource
resource
BlockStnt
Expr eesson
Parambter
AdtionEatt
NamwEr
attri
20

---

Under review as a conference paper at ICLR 2021
Constructor Dellatiion
DocckSuntt
Expressios
Param eter
Assignaan
NamoExtr
Clasoo
coppe
Variab dpard
Oijjett
resource
resource
Bookktent
Exp soresston mmt
PA Paramote o
AdiionEaco or
NamwEr
aruri
21

---

Under review as a conference paper at ICLR 2021
Constru leration
Actrointaao
NamoExte
DeRso
CoTyy
Veytab Bnyciarotard
N
Start
Object
rssource
rssource
22


Reviews:
[{"review_id": "sPFHDu4AusC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n \nThe paper provides a interesting direction in representing source code. In particular, it proposes a method called GN-Transformer to fuse representations learned from graph and text modalities under the Graph Networks.\nThe proposed method achieved state of the art performance in two code summarization datasets.\n\n##########################################################################\n\nReasons for score: \n\n \nOverall, I vote for rejection. I like the idea of injecting code structure in the code representation learning process. My major concern is about the novelty of the paper. Even though the figure looks fancy, the methodology itself seems only standard mask attention. A more recent work has explored ways towards the integration of tree into Transformer, which should be considered. https://arxiv.org/pdf/2002.08046.pdf. Moreover, the claim of \"Most existing methods are late fusion and underperform when supplementing the source code text with a graph representation.\" needs evidence to support. The claim of the title is about a generic method for code representation learning, however, the experiments are only conducted on code summarization task, which is not convincing enough to support such claim. Last, paper writing and organization needs to be improved. Many important details like number of a part of baselines are given in appendix. Appendix should not be used in this way. \n\n  \n##########################################################################\n\nConcerns: \n\n \n1. My major concern is about the novelty of the paper. Even though the figure looks fancy, the methodology itself seems only standard mask attention. A more recent work has explored ways towards the integration of tree into Transformer, which should be considered. https://arxiv.org/pdf/2002.08046.pdf. \n\n2. The claim of \"Most existing methods are late fusion and underperform when supplementing the source code text with a graph representation.\" needs evidence to support. \n\n3. The claim of the title is about a generic method for code representation learning, however, the experiments are only conducted on code summarization task, which is not convincing enough to support such claim. \n\n4. Paper writing and organization needs to be improved. Many important details like number of a part of baselines are given in appendix. Appendix should not be used in this way. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kJOtRRZgZTL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  \n\n In this work, authors propose a new direction in summarizing code snippets by combining their AST and lexical code features in the form of a graph called SCG (which is shown not to be optimal). Their model, GN-Transformer, further extracts information from SCG to summarize the code snippet. This model is a combination of transformers and graph network and is benchmarked on two datasets of Java and Python source code. The results show that the model marginally outperforms the selected benchmarks. \n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%  \n\nStrengths of the paper:  \n\nThe idea of combining syntactic features of the code (AST) with its lexical feature into a single graph structure is interesting. \n\nThe related work section is strong and captures the majority of recent work on the topic. \n\nThe paper is well-written and easy to follow. It is well-motivated and unfolds the intuition behind the arguments clearly. \n\n  \n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \n\nWeakness of the paper: \n\nSee detailed review below. \n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%  \n\n  \n\nDetailed Review:  \n\n  \n\nMajor concerns:  \n\nThe results of the experiments are rather underwhelming and show marginal improvement compared to the selected benchmarks, specifically the one from Ahmad et al. As such, it is not evident how practical the proposed method is in real-world scenarios. It also does not support the claim by the authors that considering SCG design adds more value compared to capturing AST and sequence information separately. \n\nI find the choice of baselines rather limited. The paper has a rich related work study. However, the focus of the paper is put on the comparison with the work of Ahmad et al.. Many similar research mentioned by authors are not included as baselines. For example, it is far more interesting to see how the SCG-based summarization can compete with CFG-based model of Huo et al., than seeing the comparison with baselines in Appendix C. This can help with proving the value of the model under practical setting in which variety of options are available and not only those based on AST and sequence modeling. \n\nThe design of SCG (which is not optimal as authors mentioned as well) does not seem to help much with improving the results. Authors discuss possible ways to improve SCG (e.g. hand-engineering or the use of additional rules), but do not implement them in their experiments. Why is that? Their argument about finding the “sweet spot” between C and L does not go into details of how it can be done and where that “spot” lies in. \n\nIn section 3.2, it is mentioned the average path length L in Python and Java corpus is above 6. However, we see L for test data is around 4. I think it is more realistic to test on scripts with L smaller and larger than the average L in the training data to see how it affects the results. \n\nIn section 3.2 where the two variants are introduces to alleviate the problems of standard SCG, I was wondering why a combination of the two is not tested as well. Why authors did not test that? \n\n \n\n %%%%%%%%%%%%%%%%%%%%%%%%%%%%% \n\n \n\nMinor Concerns:  \n\nIn figure 3c, as far as I understood, the black edges belong to original AST and the red ones are added. However, there are some AST edges that are red in 3c. Is that a mistake or it is intended? If the latter, why? \n\nThe term “code features” is broad. A code snippet has lexical, layout, syntactic, and dynamic features. I believe this study uses “code features” to refer to lexical features only, so it is best to specify that. \n\nThe paper has to be self-sufficient, so in section 5.1, it is best to have a brief summary of the selected baselines and their shortcomings or at least how they differ from the propose model in the paper. \n\nThe fonts in figure 3 are not readable. \n\nSome abbreviations are used before they are defined. For example, FFA and MHA in page 4, and “SBT” in section 5.2. \n\n Some typos and grammatical errors. Examples: \n\nFigure 1, in the caption: “blocks” -> block \n\nFigure 4b, x-axis: “clustering coefficiency”  \n\nPage 5: “could be lead to...” \n\n \n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%  \n\n  \n\nQuestions:  \n\n  \n\nPlease answer the questions raised in “detailed review” above.  \n\n  \n\n  \n\n \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "hk_i7RzMqx_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper focuses on the problem of training a neural model to understand source code. The authors argue that both graph information (such as the parsed abstract syntax tree) and sequence information (such as the raw program tokens) are useful for understanding code, and describe a particular method of adding raw program tokens to a graph called SCS. They also describe a modification of a transformer (called a GN-Transformer) that uses this SCS graph representation, and present results on code summarization tasks.\n\nThe proposed encoding does not seem novel, and the authors appear to be unaware of significant prior work in this space. In particular, the SCS graph representation seems like a simpler version of the representation described by Allemanis et al. (2018) [1]. The GN-Transformer model also seems to have only minor differences from previously-used architectures for code understanding. Additionally, the paper has multiple issues with clarity and style that make it hard to understand, and also has some claims that seem unsupported by the evidence.\n\nThe experimental results are somewhat interesting, and show that this kind of method can achieve good results on source code summarization tasks (whereas most of the closely-related prior work that I am aware of has instead focused on automatic fixing of bugs). Overall, however, given the lack of novelty of the proposed method and the issues with clarity, I recommend rejection for this paper.\n\n\n### Detailed comments\n\nIn \"Learning to Represent Programs with Graphs\" [1] (see section 4 heading \"Program Graphs\"), Allamanis et al. describe a graph representation very similar to the SCS representation described here, which includes both raw tokens and AST nodes, linking the AST nodes to the raw tokens they contain, and connecting each token to the previous and following tokens. The graph representation in [1] also includes a large variety of additional edges with explicit edge labels, based on static analyses of program behaviors. The SCS representation seems to be simpler than this, only including AST and token edges, and not including next/previous relationships for tokens (although the authors do describe a variant where each token is connected to all other tokens) or having any notion of different edge types or directed edges.\n\nOn page 3, the authors claim that \"structural information contained in the AST is lost\" when represented with a structure-based traversal. However, Hu et al. state that the SBT traversal is lossless and that the AST can be recovered exactly from it, so this claim seems to be incorrect.\n\nI don't understand the discussion of \"optimal graph structure\" in section 3.2. The authors cite work by You et al. (2020), which describes a way to analyze the structure of the connectivity of arbitrary neural networks (including feedforward nets, convolutional nets, etc); they then analyze various properties of these graphs to draw conclusions about the corresponding networks, and find that certain connectivity patterns work better than others. I don't see how this argument translates to graph neural networks, for which the graph is the input to the model. It's not obvious that connectivity of network layers and connectivity of the input graph are the same kind of thing, and also just because connectivity correlates with accuracy in some sense on one dataset that doesn't mean that those connectivity patterns are *necessary* to achieve good accuracy. Furthermore figure 4b in this work suggests that the SCG encoding has a clustering coefficient of exactly 0, which is strange and does not seem to be the same kind of connectivity pattern analyzed by You et al.\n\nThe description of the GN-Transformer model is cluttered and hard to follow, but from my understanding it is only a minor variation of existing models. In particular, it appears to be the same as a vanilla transformer, but with masking applied so that each graph node can only attend to neighboring nodes. (Or, equivalently, it is a standard multi-head GAT block except that dot-product attention is used instead of leaky ReLU attention, and a dense feedforward layer is added after the attention mechanism.) There are also similarities to the GREAT model described in \"Global relational models of source code\" [2], which is also a transformer-based model applied to nodes of an \"early-fusion\" graph (containing source tokens and AST nodes), but the GREAT model seems more powerful because it is allowed to attend to nodes that are not neighbors (avoiding the long-range dependency problem that the authors of this paper mention in section 3.2).\n\nAlso, if my understanding of the GN-Transformer model is correct, I don't see why it is necessary to invoke the formalism of GN blocks (proposed by Battaglia et al). The \"edge update function\" seems to discard all previous information about each edge, and is just a more complicated way of writing the dot-product attention computation. While it may be true that GN-Transformer fits into the GN block framework, I think using the terminology of GN blocks obscures what the proposed model is doing.\n\nThe experimental results seem somewhat promising, and suggest that using graph structure and source tokens together when doing code summarization works better than just operating on sequences or doing a late-fusion approach. However, it seems to me that you could likely obtain even better performance by using a more powerful graph representation and model such as the GREAT model in [2]; it would be important to compare against that approach to determine what, if anything, the SCS representation and GT-Transformer models add.\n\nThe claim that the experiments with graph variants \"explain\" the results of Ahmad et al. (2020) seems too strong. These experiments are conducted with an entirely different architecture and input structure. Maybe these experiments suggest a possible explanation, but there could be many other alternate explanations.\n\nThe paper contains some grammatical and notational errors; please carefully reread for these. I have listed some of the things I noticed below, although there may be others.\n\n### Minor errors\n\nIntroduction: The sentence \"Programming languages are context-free formal language, an unambiguous representation, Abstract Syntax Tree (AST), could be derived from a source code snippet\" seems grammatically incorrect.\n\nFigure 2 caption: Some quotation marks face the wrong direction.\n\nLast paragraph of 2.1: some of these sentences are sentence fragments.\n\nThird paragraph of 2.2 \"it’s a representation without noise on how the tokens interact\": What do you mean by noise here?\n\nSecond paragraph of 3.2 \"For example, One graph neural networks (GNN) layer\": typo in capitalization\n\nFirst paragraph of 4.2 \"This is done by two sub-blocks an edge block\": should there be punctuation between \"sub-blocks\" and \"an\"?\n\nSecond paragraph of 4.2: The notation used here seems inconsistent; $\\phi^e$ is defined but never use it, then ${E'_{ij}}^{(\\gamma)}$ is described and used but never defined. Are those the same?\n\n\n### References\n\n[1] Allamanis, Miltiadis, Marc Brockschmidt, and Mahmoud Khademi. \"Learning to represent programs with graphs.\" International Conference on Learning Representations. 2018.\n\n[2]: Hellendoorn, Vincent J., et al. \"Global relational models of source code.\" International Conference on Learning Representations. 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3cOpxk6i2OJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the analysis for AST (program code) by graph network-based transformer. This paper is well-motivated but the organization and presentation remain to be improved. Regarding the methodology, it is too simple but still makes sense.\n\nMajor Concerns:\n1. The presentation should be improved. For example, the motivations are revealed to the reader in Sec. 3.2. I suggest the authors re-organize and rephrase this paper for better presentation. To be honest, this paper is hard to follow, though the idea is good.\n2. The methodology though is effective but too simple. It seems this paper only makes an incremental contribution to combine GN and Transformer. I suggest the authors focus on the tasks, more specifically. Because the experimental results only serve as marginal improvements relative to the Transformer, I think I confirmed the oversimplified property of this methodology. \n\nMinor Concerns:\n1. Fig.2: Very confusing figure, because you shall mention 50000 in the caption and specify the process in the caption also. Because of weak description, This paper is hard to read.\n2. Fig.3: What is the meaning of showing (a)? The entire Fig.3 seems redundant to me.\n3. Did you perform the significance test for the experimental results? Because your improvements are marginal. \n\nDiscussion:\n\nThis paper really proposed an interesting idea. But for me, it seems like an end-to-end framework to transform tree-based to sequential texts. I have a question: Will the description accurate if we don't see the whole picture of the codes? I just want to provide a new perspective for you to look into.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there are some potentially interesting aspects to this work, it doesn’t acknowledge a significant amount of relevant literature, and there are some unsupported claims. All reviewers believe the paper is not ready for acceptance. Reviewers provided some good thorough reviews and suggestions, but the authors did not choose to respond or engage in discussions to improve the paper. 
Decision: Reject