Retrieval-Augmented Generation for Code Summarization via Hybrid GNN
Authors: Shangqing Liu
Yu Chen
Xiaofei Xie
Jing Kai Siow
Yang Liu
Venue: ICLR 2021 Spotlight (2021)
Keywords: Code Summarization
Graph Neural Network
Retrieval
Generation

Abstract: Source code summarization aims to generate natural language summaries from structured code snippets for better understanding code functionalities. However, automatic code summarization is challenging due to the complexity of the source code and the language gap between the source code and natural language summaries. Most previous approaches either rely on retrieval-based (which can take advantage of similar examples seen from the retrieval database, but have low generalization performance) or generation-based methods (which have better generalization performance, but cannot take advantage of similar examples).
This paper proposes a novel retrieval-augmented mechanism to combine the benefits of both worlds.
Furthermore, to mitigate the limitation of Graph Neural Networks (GNNs) on capturing global graph structure information of source code, we propose a novel attention-based dynamic graph to complement the static graph representation of the source code, and design a hybrid message passing GNN for capturing both the local and global structural information. To evaluate the proposed approach, we release a new challenging benchmark, crawled from diversified large-scale open-source C projects (total 95k+ unique functions in the dataset). Our method achieves the state-of-the-art performance, improving existing methods by 1.42, 2.44 and 1.29 in terms of BLEU-4, ROUGE-L and METEOR.

Published as a conference paper at ICLR 2021
RETRIEVAL-AUGMENTED GENERATION FOR CODE SUMMARIZATION VIA HYBRID GNN
Shangqing Liu'; Yu Chen Xiaofei Jingkai Siow!, Yang Liu'
Naayynn Technology University
2 Rensselaer Polytechnic Institute
ABSTRACT
Source code summarization aims to generate natural language summaries from structured code snippets for better understanding code functionalities. However, au- tomatic code summarization is challenging due to the complexity of the source code and the language gap between the source code and natural language summaries. Most previous approaches either rely on retrieval-based (which can take advantage of similar examples seen from the retrieval database, but have low generalization performance) or generation- baasd methods (which have better generalization per- formance, but cannot take advantage of similar examples). This paper proposes a novel -augmeeal- mechanism to combine the benefits of both worlds. Furthermore, to mitigate the limitation of Graph Neural Networks (GNNs) on capturing global graph structure information of source code, we propose a novel ditention-based dynamic graph to complement the static graph representation of the source code, and design a hybrid message passing GNN for capturing both the local and global structural information. To evaluate the proposed approach, we release a new challenging benchmark, crawled from diversified large-scale open-source C projects (total 95k+ unique functions the dataset). Our method achieves the state-of- state-of-the-ar performance, improving existing methods by 1.42, 2.44 and 1.29 in terms of BLEU-4, ROUGE-L and METEOR.
INTRODUCTION
With software growing in size and complexity, developers tend to spend nearly 90% (Wan et al., 2018) effort on software maintenance (e.g., version iteration and bug fix) in the completed life cycle of software development. Source code summary, in the form of natural language, plays a critical role in the comprehension and maintenance process and greatly reduces the effort of reading and comprehending programs. However, manually writing code summaries tedious and time- consuming, and with the acceleration of software iteration, it has become heavy burden for software developers. Hence, source code summarization which automates concise descriptions of programs is meaningful.
Automatic source code summarization is crucial yet far from the settled problem. The key challenges include: 1) the source code and the natural language summary are heterogeneous, which means they may not share common lexical tokens, synonyms, or language structures and 2) the source code complex with complicated logic and variable grammatical structure, making hard to learn the semantics. Conventionally, information retrieval (IR) techniques have been widely used in code summarization (Eddy et al., 2013; Haidue al., 2010; Wong al., 2015; 2013). Since code duplication (Kamiya et al., 2002; Li et al., 2006) is common in "big code" (Allamanis et al., 2018), carly works summarize the new programs by retrieving the similar code snippet in the existing code database and use its summary directly. Essentially, the retrieval-based approaches transform the code summarization to the code similarity calculation task, which may achieve promising performance on similar programs, but are limited in generalization, i.e. they have poorer performance programs that are very different from the code database.
"Contactshannginn00 @e. @e.ntu.edu.sg 'Coresponding authors

---

Published as a conference paper at ICLR 2021
To improve the generalization performance, recent works focus generation- based approaches. Some works explore Seq2Seq architectures (Bahdanau et al., 2014; Luong et al., 2015) to generate summaries from the given source code. The Seq2Seq-basee approaches (lyer et al., 2016; Hu et al., 2018;; Alon et al., 2018) usually treat the source code or abstract syntax tree parsed from the source code as sequence and follow paradigm of encoder-decoder with the attention mechanism for generating a summary. However, these works only rely on sequential models, which are struggling to capture the rich semantics of source code e.g., control dependencies and data dependencies. In addition, inperaatio-basel approaches typically cannot take advantage of similar examples from the retrieval database, as retrieval- based approaches do.
To better learn the semantics of the source code, Allamanis et al. (Allamanis et al.. 2017) lighted up this field by representing programs as graphs. Some follow-up works (Fernandes et al., 2018) attempted to encode more code structures (e.g., control flow, program dependencies) into code graphs with graph neural networks (GNNs), and achieved the promising performance than the sequence- based approaches. Existing works (Allamanis al., 2017; Fernandes et al., 2018) usually convert code into graph-structured input during preprocessing, and directly consume it via modern neural networks (e.g., GNNs) for computing node and graph embeddings. However, most GNNNbased encoders only allow message passing among nodes within a k-hopp neighborhood (where & is usually small number such as 4) to avoid over-smoothing (Zhao & Akoglu, 2019; Chen et al., 2020a), thus capture only local neighborhood information and ignore global interactions among nodes. Even there are some works (Li et al., 2019) that try address this challenging with deep GCNs (i.c.. 56 layers) (Kipf& & Welling, 2016) by the residual connection (He et al.. 2016), however, the computation cost cannot endure in the program especially for a large and complex program.
To address these challenges, we propose a framework for automatic code summarization, namely Hybrid GNN (HGNN). Specifically, from the source code, we first construct a code property graph (CPG) based on the abstract syntax tree (AST) with different types of edges (i.e., Flow To, Reach). In order to combine the benefits of both retrieval-bar based and generation- based methods, we propose a retrieval-based based augmentation mechanism to retrieve the source code that is most similar to the current program from the retrieval database (excluding the current program itself), and add the retrieved code as well as the corresponding summary as auxiliary information for training the model. In order to go beyond local graph neighborhood information, and capture global interactions in the program, further propose an attention- -aaedd dynamic graph by learning global attention scores (i.e., edge weights) in the augmented static CPG. Then, a hybrid message passing (HMP) is performed on both static and dynamic graphs. We also release new code summarization benchmark by crawling data from popular and diversified projects containing 95k+ functions in C' programming language and make it public !. We highlight our main contributions follows:
We propose general- framework for automatic code summarization, which combines the benefits of both retrieval-based and generation-based methods via retrieval-based augmentation mechanism.
We innovate a Hybrid GNN by fusing the static graph (based on code property graph) and dynamic graph (via structure-aware global attention mechanism) to mitigate the limitation of the GNN on capturing global graph information.
We release a new challenging C benchmark for the task of source code summarization.
We conduct an extensive experiment to evaluate our framework. The proposed approach achieves the
performance and improves existing approaches by 1.42, 2.44 and 1.29 in terms of BLEU-4, ROUGE-L and METEOR metrics.
2 HYBRID GNN FRAMEWORK
In this section, we introduce the proposed framework Hybrid GNN (HGGN), as shown in Figure which mainly includes four components: 1) Retrieval-augmmenter Static Graph Construction (c.f. Section 2.2), which incorporates retrieved code-summary pairs augment the original code for learning. 2) Attention-based Dynamic Graph Construction (c.f. Section 2.3), which allows message passing among any pair of nodes via a structure-aware global attention mechanism. 3) HGNN, (c.f.
2

---

Published as a conference paper at ICLR 2021
Retrieval-uggmented Graph
Hybrid
Code
Static Message
Passing
Aug
Node Embeddings
Retrievol Database
Hybrid
CPC
Dynamic Message
GRU Updates
Passing
Attention- -aased Graph
Retrieved
Summary Encoding
Aggregation
Graph Embedding
Decoder
Generated
Summary
Figure 1: The overall architecture of the proposed HGNN framework.
Section 2.4), which incorporates information from both static graphs and dynamic graphs with Hybrid Message Passing. 4) Decoder (c,f. Section 2.5), which utilizes an attention- based LSTM (Hochreiter Schmidhuber, 1997) model to generate a summary.
2.1 PROBLEM FORMULATION
In this work, we focus on generating natural language summaries for the given functions (Wan et al., 2018; Zhang al., 2020). A simple example is illustrated in Listing which is crawled from Linux Kernel. Our goal is to generate the best summary "set the time of day clock" based on the given source code. Formally, we define dataset as D {(c, s) E C, S). where is the source code of function in the function set C' and represents its targeted summary in the summary set S. The task of code summarization is, given a source code c, to generate the best summary consisting of sequence of tokens å = (f1, tt.,,, (r) that maximizes the conditional likelihood é argmax, P(s|c).
Source int pdc_tod_set (unsigned long sec, unsigned long usec)
int retval; unsigned long flags;
spin_lock_ing irquava (4pde_lock flags);
zetval men pdo call (POC_ 20C_rco_ KRITE, sec, usec);
upin_unlock. unlock. irqreatore (spde__occk Elage); return retval;
Ground -rounddTruuh:: set
day clock
Listing I: An example our dataset crawled from Linux Kernel.
2.2
RETRIEVAL-AUGMENTER STATIC GRAPH
2.2.1 GRAPH INITIALIZATION
The source code of funetion can be represented as Code Property Graph (CPG) (Yamaguchi et al., 2014), which is built on the abstract syntax tree (AST) with different type of edges (i.e., Flow To, Control, Define/Use, Reach). Formally, one raw function e could be represented by a multi-edged graph (V,,)), where V is the set of AST nodes, (v,u) E & denotes the edge between the node and the node u. A node e consists two parts: the node sequence and the node type. An illustrative example is shown in Figure 2. For example. in the red node, a%2 == 0 is the node sequence and Condition is the node type. An edge (v, u) has a type, named edge type, e.g., AST type and Flow To type. For more details about the CPG, please refer to Appendix A.
Initialization Representation. Given CPG, we utilize a BiLSTM encode its nodes. We represent each token of the node sequence and each edge type using the learned embedding matrix Esettoken and Bodgetyre respectively. Then nodes and edges of the CPG can be encoded as:
hy BiLSTM Beegokenn ...., E encode (odeee)_odew)
Bangtoken
[ encode_edgefe u) = pedgetupe if (v, u) E else 0
(I)
3

---

Published as a conference paper at ICLR 2021
Inter)and]
exampee 1} int rand)] Hl % we 0) int cullit
rand
2
simmll
Flo
Control
Define/U
Reach
Figure 2: An example of Code Property Graph (CPG).
where / is the number of tokens in the node sequence of u. For the sake of simplicity, in the following section, we use h, and uu to represent the embedding of the node v and the edge (v, u), respectively. i.e., encode no)e(v) and seccod_edgee(( u). Given the source code e of function as well as the CPG g(V, E). Ho E IRd denotes the initial node matrix of the CPG, where m is the total number of nodes in the CPG and d is the dimension of the node embedding.
2.2.2 RETRIEVAL-BASED AUGMENTATION
While retrieval- basse methods can perform reasonably well examples that are similar to those examples from a retrieval database, they typically have low generalization performance and might perform poorly dissimilar examples. On the contrary, generation- -baddd methods usually have better generalization performance, but cannot take advantage of similar examples from the retrieval database. In this work, we propose to combine the benefits of the two worlds, and design serieval-augmennee generation framework for the task of code summarization.
In principle, the goal of code summarization is learn mapping from source code c the natural language summary 8 =(c)). In other words, for any source code c', a code summarization system can produce its summary s' f(c'). Inspired by this observation, conceptually, we can derive the following formulation semantic difference between and c', and further obtain the desired summary s for e by considering both the above semantic difference and s' which is the summary for c'. Mathmatically, our goal becomes to learn function which takes as input (c, ,'), and outputs the summary s for c, that is, s g(c, s'). This motivates to design our Retrieval-based Augmentation mechanism, as detailed below.
f(c) f(c') + s'. This tells us that we can actually compute the
Step 1: Retrieving. For each sample (c, 8) E D, we retrieve the most similar sample: (c,8') argmax s'' sim c'), where c + d, D' is a given retrieval database and sim(c, c') is the text utilize Lucene for retrieval and calculate the similarity score z between the source code e and the retrieved code c' via dynamic programming (Bellman, dis(c.e') where dis(c, c') is the text edit distance. 1966), namely, 2 = mext]cl..eT]>>
similarity. Following Zhang et al. (2020),
Step 2: Retrieved Code-based Augmentation. Given the retrieved source code c' for the current sample we adopt fusion strategy to inject retrieved semantics into the current sample. The fusion strategy based on their initial graph representations (He and H.)) with an attention mechanism:
To capture the relevance between and c', we design an attention function, which computes the attention score matrix A aug based on the embeddings of each pair of nodes in CPGs of c and d':
A aug x (
(2)
where WC wº E Raxd is the weight matrix with d-dim embedding size and ReLU is the rectified linear unit.
We then multiply the attention matrix Aaug with the retrieved representation He to inject the retrieved features into Ho:
=== aug H.
(3)
where E [0, 1] is the similarity score and computed from Step 1, which is introduced to weaken the negative impact of d' on the original training data c, i.e., when the similarity of e and d is low. Finally, we merge H', and the original H. to get the final representation of c.
comp
H. H'
(4)
4

---

Published as a conference paper at ICLR 2021
where comp is the augmented node representation additionally encoding the retrieved semantics.
Step 3: Retrieved Summary-based Augmentation. We further encode the retrieved summary 8 with another BiLSTM model. We represent each token t of s' using the learned embedding matrix Esegtoken Then s' can be encoded as:
(5) where he; is the hidden state of the BiLSTM model for the token tj in s' and T is the length of s'. We multiply he;; he2 with the similarity score 2, computed from Step 1, and concatenate it with the graph encoding results (i.e., the GNN encoder outputs) obtain the input, namely, [GNNoutputi sh.. zhee J, to the decoder.
p Myg BiLSTM ( Esegtokenn Bi
Esegtoken
2.3 IYTENTION-BASED DYNAMIC GRAPH
Due that GNN-based encoders usually consider the k-hop neighborhood, the global relation among nodes in the static graph (see Section 2.2.1) may be ignored. In order to better capture the global semanties of source code, based on the static graph, we propose to dynamically construct a graph via structure-aware global attention mechanism, which allows message passing among any pair of nodes. The attention-based dynamic graph can better capture the global dependency among nodes, and thus supplement the static graph.
Structure-aware Global Attention. The construction of the dynamic graph is motivated by the structure-awart self-attention mechanism proposed in Zhu et al. (2019). Given the static graph, compute a corresponding dense adjacency matrix dgm based on a structure-aware global attention mechanism, and obtain the constructed graph, namely, attention-based dynamic graph.
ReLU(h? W (ReLU W ReLU W
dyn U,
(6)
Vd
where h. h. comp are the augmented node embedding for any node pair (v, u) in the CPG. Note that the global attention considers each pair of nodes of the CPG, regardless whether there an edge between them. Gv.u E Rd. is the edge embedding and WQ. W idddd W E Rd, xd are parameter matrices, de and are the dimensions edge embedding and node embedding. respectively.
The adjacency matrix A dgn will be further row normalized to obtain À dyn which will be used to compute dynamic message passing (see Section 2.4).
Á dyn softmax (A dgn)
2.4 HYBRID GNN
To better incorporate the information of the static graph and the dynamic graph, we propose the Hybrid Message Passing (HMP), which are performed both eetrieval-augmenne static graph and attention- based dynamic graph.
Static Message Passing, For every node v each computation hop k in the static graph, we apply an aggregation function to calculate the aggregated vector h by considering a set of neighboring node embeddings computed from the previous hop.
h = SUM( - E N(
where N(v) is set of the neighboring nodes which are directly connected with v. For each node v, hy is the initial augmented node embedding u, ie.,, h, E comp.
Dynamic Message Passing. The node information and edge information are propagated on the attention-based dynamic graph with the adjacency matrices Adyn defined as
h
dan (WV h 1k-1, + WF e
where e and are any pair of nodes, WV E Rdxd, WF Rddx are learned matrices, and 6 the embedding of the edge connecting v and u. Similarly, h.o is the initial augmented node embedding of in comp.
5

---

Published as a conference paper at ICLR 2021
Hybrid Message Passing. Given the static/dynamic aggregated vectors h for static and dynamic graphs, we fuse both vectors and feed the resulting vector to a Gated Recurrent Unit (GRU) to update node representations.
GRU(F Fusee h2)))
(10)
where f is the augmented node initialization in comp. The fusion function Fuse is designed as a gated sum of two inputs.
Fuse b) = = C a (1 2) @ = (Wala; o b - bj bz)
(11)
where W. and bg are learnable weight matrix and vector, is the component multiplication, is a sigmoid function and z is gating vector. After n hops of GNN computation, we obtain the final
node representation f and then apply max-pooling over all nodes {ʃ (Vu E V} to get the graph representation.
2.5 DECODER
The decoder is similar with other state-of-the-art Seq2seq models (Bahdanau et al., 2014; Luong et al., 2015) where attention-based LSTM decoder is used. The decoder takes the input of the concatenation of the node representation and the representation of the retrieved summary s', namely, where is the number of nodes in the input CPG graph. The initial hidden state of the decoder is the fusion (Eq. 11) of the graph representation and the weighted (i.c., multiply similarity score 2) final state of the retrieved summary BiLSTM encoder.
if...... f ;
We train the model with the cross-entropy loss, defined as L = E. Plo s²x),, where st is the word at the 1-th position of the ground-truth output and is the source code of the function. To alleviate the exposure bias, we utilize schedule teacher forcing (Bengio et al., 2015). During the inference, we use beam search to generate final results.
EXPERIMENTS
3.1
SETUP
We evaluate our proposed framework against a number of state-of-the-art methods. Specifically, we classify the selected baseline methods into three groups: 1) Retrieval-basee approaches: TF- approaches: CODE- NN (lyer et al., 2016; Barone & Sennrich, 2017). Transformer (Ahmad et al., 2020), Hybrid- DRL (Wan et al., 2018), Rencos (Zhang et al., 2020) and Dual model (Wei et al., 2019), 3) Graph- based approaches: SeqGNN (Fernandes et al., 2018). In addition, we implemented two another graph-based baselines: GCN2Seq and GAT2Seq, which respectively adopt the Graph Convolution (Kipf & Welling, 2016) and Graph Attention (Velickovic et al., 2018) as the encoder and a LSTM as the decoder for generating summaries. Note that Rencos (Zhang et al., 2020) combines the retrieval information into Seq2Seq model, we classify into Sequence- based approaches. More detailed description about baselines and the configuration of HGNN can be found in the Appendix B and C.
IDF (Haiduc et al., 2010) and NNGen (Liu et al., 2018), 2) Sequence-
Existing benchmarks (Barone & Sennrich, 2017; Hu et al., 2018b) are all based on high-level programming language i.e., Java, Python. Furthermore, they have been confirmed have extensive duplication, making the model overfit to the training data that overlapped with the testset (Fernandes al., 2018; Allamanis, 2019). We are the first explore neural summarization C programming language, and make our C Code Summarization Dataset (CCSD) public to benefit academia and pairs
industry. We crawled from popular repositories on GitHub and extracted function-summen based on the documents of functions. After a deduplication process, we kept 95k+ unique function- summary pairs. To further test the model generalization ability, we construct in-domain functions and out-of-domain functions by dividing the projects into two sets, denoted as a and b. For each project in a, we randomly select some of the functions in this project as the training data and the unselected functions are the in-domain validation/test data. All functions in projects b are regarded as test data. Finally, we obtain 84,316 training functions, 4,432 in in-domain validation test functions. For the retrieval
functions, 4,203 in-domain test functions and 2,330 augmentation, we use the training set as the retrieval database, i.e., D' = D (see Step in Section 2.2.2). For more details about data processing, please refer to Appendix D)

---

Published as a conference paper at ICLR 2021
Table 1: Automatic evaluation results (in %) on the CCSD test set.
In-dooain ROUGET
Out-of-domnun
Overall
Methods
BLEU4
METEOR
ROUGET
METEOR
BLEU 4 ROUGEL METEOR
TF-ID
15.20
27.98
13.74
5.50
15.37
6.34
12.19
23,49
11.43
NNGien
15.97
28.14
13.82
5.74
16.33
7.18
12.76
23,93
11,58
CODE-NN
10.08
26.17
11.35
3.86
15.25
6.19
8.24
22.28
9.61
Hybrid-DRL
9,29
30.00
12.47
6.30
24.19
10.30
8.42
28.64
11.73
Transformer
12.91
28.04
13.83
5.75
18.62
9.89
10.69
24.65
12.02
Dual Model
11.49
29.20
13.24
5.25
21.31
9.14
9.61
26.40
11.87
Remens
14,80
31.41
14.64
7.54
23.12
10.35
12.59
28.45
13.21
GUN2Seq
9.79
26.59
11.65
4.96
18.96
7.76
7.91
25.67
10.23
GAT2Seq
26.17
11.88
3.80
16.94
SegGNNN
10.51
29.84
13.14
4.94
20.80
9.50
8.87
26.34
11,93
HGNN w/o dugmennt & stotte
11.75
29.59
13.86
5.37
22.14
9.41
9.98
26.94
12.05
HGNN augment & dynamic
5.45
HGNN vlo augment
12.33
29.99
13.78
5.45
22.07
9.46
10.26
27.17
12.32
HGNN mttic
33.67
15.67
7.72
24.69
10.63
30.47
HGNN who dynamic
15.77
33.84
15.67
7.64
24.72
10.73
13.31
30.59
14.01
HGNN
16.72
34.29
16.25
7.85
24.74
11.05
14.01
30.89
14.50
Similar to previous works (Zhang et al., 2020; Wan et al., 2018; Fernandes et al., 2018; Iyer et al., 2016), BLEU (Papineni et al., 2002), METEOR (Banerjee Lavie, 2005) and ROUGE-L (Lin, 2004) are used as our automatic evaluation metrics. These metrics are popular for evaluating machine translation and text summarization tasks. Except for these automatic metrics, we also conduct a human evaluation study. We invite 5 PhD students and 10 master students volunteers, who have rich C programming experiences. The volunteers are asked to rank summaries generated from the anonymized approaches from 1 to 5 (iee., 1: Poor, 2: Marginal, 3: Acceptable, 4: Good, 5: Excellent) based on the relevance of the generated summary to the source code and the degree of similarity between the generated summary and the actual summary. Specifically, we randomly choose 50 functions for each model with the corresponding generated summaries and ground-truths. We calculate the average score and the higher the score, the better the quality.
3.2 COMPARISON WITH THE BASELINES
Table shows the evaluation results including two parts: the comparison with baselines and the ablation study. Consider the comparison with state-of oh-the-rrt baselines, in general, we find that proposed model outperforms existing methods by significant margin on both in-domain and out-of-domain datasets, and shows good generalization performance. Compared with others, on in-domain dataset, the retrieval- -based approaches could achieve competitive performance BLEU-4, however ROUGE-L and METEOR are fare less than ours. Moreover, they do not perform well on the out-of-domain dataset. Compared with the graph-based approaches (i.e., GCN2Seq, GAT2Seq and SeqGNN), even without augmentation (HGNN w/o augment), our approach still outperforms them, which further demonstrates the effectiveness Hybrid GNN for additionally capturing global graph information. Compared with Rencos that also considers the retrieved information in the Seq2Seq model, its performance still lower than HGNN. On the overall dataset including both of in-domain and out-of-domain data, our model achieves 14.01, 30.89 and 14.50, outperforming current state-of of-the-art method Rencos by 1.42, 2.44 and 1.29 in terms of BLEU-4, ROUGE-L and METEOR metrics.
3.3
ABLATION STUDY
We also conduct an ablation study to evaluate the impact of different components of our framework, e.g., retrieval-basee augmentation, static graph and dynamic graph the last row of Table Overall, we found that 1) retrieval-aute retrieval-augmentee augmented mechanism could contribute to the overall model performance (HGNN vs. HGNN w/o augment). Compared with HGNN, we see that the performance HGNN w/o static and HGNN w/o dynamic decreases, which demonstrates the effectiveness of the Hybrid GNN and 2) the performance without static graph is worse than the performance without dynamic graph in ROUGE-L and METEOR, however, BLEU-4 is higher than the performance without dynamic graph. To further understand the impact of the static graph and dynamic graph, we evaluate the performance without augmentation and static graph/dynamic graph (see HGNN w/o augment& static and HGNN w/o augment& dynamic). Compared with HGNN w/o augment, the results further confirm the effectiveness of the Hybrid GNN (Lee,, static graph and dynamic graph).
7

---

Published as a conference paper at ICLR 2021
Table 2: Human evaluation results the CCSD test
Metrics
NNGco
Transtonmeer
Rencos
SoIGNN
HGNN
Relevance
1223
3.17
3.48
Similarity
3
3.02
3.32
3.06
3,51
Table Examples of generated summaries on the CCSD test set.
Example
Example
Example 2
void ReleaseCedar (CEDAR .c)
static void strimit (Str +p)
if (c NULL)
g->2 p->nAlloc 0:
Source Code
(Release (c-bzeE)
p-sntsed 0;
CleanupCadar
Ground Truth
initialize a sir object
release relerence of the cedar
NNGcn
the string
release the virtual
Transformer
reset string
release of the cancel object
Remcos
uppend a raw string to the json string
releuse of the cuncel object
SegGNN
initialize string
release cedarr communication
HGNN
initialize string oltject
release reference of cedar
We also conduct experiments to investigate the impact of code-based augmentation and summary- based augmentation. Overall, found that the summary-based augmentation could contribute more than the code-basee augmentation. For example, after adding the bode-based augmentation. the performance can be 10.22, 27.54 and 12.49 in terms of BLUE-4, ROUGE-L and METEOR on the overall dataset. With the summary-based augmentation, the results can reach to 13.76, 30.59 and 14.11. Compared with the results without augmentation (i.e., 10.26. 27.17, 12.32 with HGNN w/o augment), we can see that code-based augmentation could have some improvement, but the effect is not significant compared with summary-basee augmentation. We conjecture that, due to that the code and summary are heterogeneous, the summary-basee augmentation has a more direct impact on the code summarization task. When combining both code based augmentation and summary-base based augmentation, we can achieve the best results (i.e., 14.01, 30.89, 14.50). We plan to explore more code-based augmentation (e.g., semantic-equivalent code transformation) in our future work.
3.4 HUMAN EVALUATION
As shown in Table 2, we perform human evaluation on the overall dataset to assess the quality of the generated summaries by our approach, NNGen, Transformer, Rencos and SeqGNN in terms of relevance and similarity. As depicted in Table 1, NNGen, Rencos and SeqGNN are the best retrieval-basse sequence-based, and graph-based approaches. respectively. We also compare with Transformer as it has been widely used in natural language processing. The results show that method can generate better summaries which are more relevant with the source code and more similar with the ground-truth summaries.
3.5
CASE STUDY
To perform qualitative analysis, we present two examples with generated summaries by different methods from the overall data set, shown in Table 3. We can that, in the first example, our approach can learn more code semantics, i.e., p is a self-defined struct variable. Thus, we could generate token object for the variable p. However, other models can only produce string. Example a more difficult function with the functionality to "release reference of cedar", as compared to other baselines, our approach effectively captures the functionality and generates more precise summary.
3.6 EXTENSION ON THE PYTHON DATASET
We conducted additional experiments on a public dataset, i.e., the Python Code Summarization Dataset (PCSD), which was also used in Rencos (the most competitive baseline in our paper). We follow the setting of Rencos and split PCSD into the training set, validation set and testing set with fractions of 60%, 20% and 20%. We construct the static graph based on AST. The decoding step set to 50, followed by Rencos, and the other settings are the same with CCSD. We compare our methods PCSD against various competitive baselines, i.e., NNGen, CO Rencos and
8

---

Published as a conference paper at ICLR 2021
Table 4: Automatic evaluation results (in %)
the PCSD test set.
Methods
BLEU-4
ROUGE-L
METEOR
NNGcn
21.60
31.61
15,96
CODE-NN
16.39
28.99
13.68
Transformer
17.06
31.16
14.37
Renens
24.02
36.21
18.07
HGNN w/o statte
24.06
38.28
HGNN w/o dynamic
24.13
38.64
18.93
HGNN
24.42
39.91
19.48
Transformer, which are either retrieval-basee generation- baaeed or hybrid methods, The results are shown in Table 4. The results indicate that, compared with the best results from NNGen, CODE-NN, Rencos and Transformer, our method can improve the performance by 0.40, 3.70 and 1.41 in terms of BLEU-4, ROUGE-L and METEOR. We also conduct the ablation study PCSD to demonstrate the usefulness of the static graph (i.e., HGNN w/o dynamic) and dynamic graph (i.e., HGNN w/o static). The results also demonstrate that both static graph and dynamic graph can contribute to framework. In summary, the results on both our released benchmark (CCSD) and existing benchmark (PCSD) demonstrate the effectiveness and the scalability of our method.
RELATED WORK
Source Code Summarization Early works (Eddy et al., 2013; Haidue al., 2010; Wong et al., 2015; 2013) for code summarization focused on using information retrieval to retrieve summaries. Later works attempted employ attentional Seq2Seq model the source code (Iyer et al., 2016; Siow et al., 2020) or some variants, i.e., AST (Hu et al., 201 8a; Alon et al., 2018; Liu et al., 2020) for generation. However, these works based sequential models, ignoring rich code semantics. Some latest attempts (LeClair et al., 2020: Fernandes et al., 2018) embedded program semantics into GNNs. but they mainly rely on simple representations, which are limited learn full semantics.
Graph Neural Networks Over the past few years, GNNs (Li et al., 2015; Hamilton et al., 2017; Kipf Welling, 2016; Chen et al., 2020b) have attracted increasing attention with many successful applications in computer vision (Norcliffe-Broww 2018a; Chen et al., 2020d;c;e). Because by design GNNs can model graph-structured data, recently, some works have extended the widely used Seq2Seq architectures to Graph2Seq architectures for various tasks including machine translation (Beck et al., 2018), and graph (c.g., AMR, SQL)-to-text generation (Zhu et al., 2019; Xu et al., 2018b). Some works have also attempted encode programs with graphs for diverse tasks e.g., VARNAMING/VARMISUU
et al., 2018), natural language processing (Xu et al.,
(Allamanis et al., 2017), Source Code Vulnerability Detection (Zhou al., 2019). As compared to these works, innovate hybrid message passing GNN performed on both static graph and dynamic graph for message fusion.
CONCLUSION AND FUTURE WORK
In this paper, we proposed general-purpose framework for automatic code summarization. novel serieval-augmenttd mechanism is proposed for combining the benefits of both retrieval-based and generation- based approaches. Moreover, to capture global semantics among nodes, we develop a hybrid message passing GNN based on both static and dynamic graphs. The evaluation shows that our approach improves state-of- of-the-arttttttt techniques substantially. Our future work includes: 1) we plan to introduce more information such API knowledge to learn the better semantics of programs, 2) we explore more baded augmentation techniques to improve the performance and 3) we plan to adopt the existing techniques such (Du al., 2019; Xie al., 2019;;b; Ma et al., 2018) to evaluate the robustness of the trained model.
ACKNOWLEDGMENTT
This research is partially supported by the National Research Foundation, Singapore under its the AI Singapore Programme Office, Singapore under its National Cybersecurity R&D Program (Award No. NRF2018NCE NCR005-0001, NRF Investigatorship SRRF-NNFI0--2020-00 through its National Satellite of Excellence in Trustworthy Software Systems (NSOE- -SSOE-TSS) project under the National Cybersecurity R&D (NCR) Grant award no. NRF201 122018NCR-NN EXCR-NSOFO00S0000
the National Research Foundation, Prime Ministers
the National Research Foundation

---

Published as a conference paper at ICLR 2021
REFERENCES
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. A transformer-based approach for source code summarization. arXiv preprint arXiv:2005. 00653, 2020.
Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections Programming and Software, pp. 143 143-153, 2019.
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. arXiv preprint arXiv: 2017.
Miltiadis Allamanis, Earl Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4):1-37, 2018.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq; Generating sequences from structured representations of code. arXiv preprint 1808: 00000 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint weXiv:1409.047.1 2014.
Satanjeev Banerjee and Alon Lavie. Meteor:
An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Antonio Valerio Miceli Barone and Rico Sennrich. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv: 2017.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-sequencc learning using gated graph neural networks. arXiv preprint 1rXivv1806.000 2018.
Richard Bellman. Dynamic programming. Science, 153(3731):34-37, 1966.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in Neural Information Processing Systems, pp. 1171-1179 2015.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topological view. In AAAI, pp. 3438-3445, 2020a.
Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances in Neural Information Processing Systems, 33, 2020b.
Yu Chen, Lingfei Wu, and Mohammed J. Zaki. Graphflow:
Exploiting conversation flow with graph neural networks for conversational machine comprehension. In Proceedings of the Twenty- Ninth International Joint Conference on Artificial Intelligence, pp. 1230-1236. International Joint Conferences on Artificial Intelligence Organization, 2020c.
Yu Chen, Lingfei Wu, and Mohammed J. Zaki. Reinforcement learning based graph-to Sroph-to-sequee sequence model for natural question generation. In Proceedings of the 8th International Conference on Learning Representationss Apr. 26-30, 2020d.
Yu Chen, Lingfei Wu, and Mohammed Zaki. Toward subgraph guided knowledge graph question generation with graph neural networks. arXiv preprint arXiv:2004. 06015, 2020e.
Xiaoning Du, Xiaofei Xie, Yi Li, Lei Ma, Yang Liu, and Jianjun Zhao. Deepstellar: Model-based quantitative analysis of stateful deep learning systems. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 477 477-487, 2019.
10

---

Published as a conference paper at ICLR 2021
Brian P Eddy, Jeffrey A Robinson, Nicholas A Kraft, and Jeffrey C Carver. Evaluating source code summarization techniques: Replication and expansion. 2013 21st International Conference on Program Comprehension (ICPC), pp. 13-22. IEEE, 2013.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. Structured neural summarization. arXiv preprint arXiv: 1.01824, 2018.
Sonia Haiduc, Jairo Aponte, Laura Moreno, and Andrian Marcus. On the use of automated text summarization techniques for summarizing source code. In 2010 17th Working Conference on Reverse Engineering, pp. 44. IEEE, 2010.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, pp. 1024-1034, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735 1780, 1997.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, pp. 200-210, 2018a.
Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. Summarizing source code with transferred api knowledge. 2018b.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Summarizing source code using neural attention model. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 2073-2083, 2016.
Siyuan Jiang, Ameer Armaly, and Collin McMillan. Automatically generating commit messages from diffs using neural machine translation. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 135-146. IEEE, 2017.
Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. Cefinder: multilinguistic loken-based code clone detection system for large scale source code. IEEE Transactions on Software Engineering, 28(7):654 670, 2002.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609 12090077 2016.
Alexander LeClair, Sakib Haque, Linfgei Wu, and Collin McMillan. Improved code summarization via a graph neural network. arXiv preprint arXiv:2004.0268 2020.
Guohao Li, Matthias Müller, Ali K. Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns? In 2019 IEFF/CVE International Conference on Computer Vision ICCV 2019, Seoul, Korea (South), October 27 November 2019, pp. 9276-99775.. IEEE, 2019. doi: 10.1 10.1109//CCV. 2019,00936. URL https: : /oii org 11 11 ICCV 201..0999...09.. 00936.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:1571.05493, 2015.
Zhenmin Li, Shan Lu. Suvda Myagmar, and Yuanyuan Zhou. Cp-miner: Finding copy copy-paste and related bugs in large-scale software code. IEFE Transactions on software Engineering, 32(3): 176-192, 2006.
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Association for Computational Linguistics, July 2004.
Shangqing Liu, Cuiyun Gao, Sen Chen, Nie Lun Yiu, and Yang Liu. Atom: Commit message generation based on abstract syntax tree and hybrid ranking. IEEE Transactions on Software Fingineering, 2020.
11

---

Published as a conference paper at ICLR 2021
Zhongxin Liu, Xin Xia, Ahmed E Hassan, David Lo, Zhenchang Xing, and Xinyu Wang. Neural- machine- machine-translationn twwastationn-aasettteeea commit message generation: how far are we? In Proceedings of the 33rd ACM/IEEE International Conference Automated Software Engineering, pp- 373-384, 2018.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprini arXiv:1508 04025, 2015.
Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, et al. Deepgauge: Multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. pp. 120- 101, 2018.
Will Norcliffe-Broww Stathis Vafeias, and Sarah Parisot. Learning conditioned graph structures for interpretable visual question answering. In Advances Neural Information Processing Systems, pp. $334-8353, 2018.
Kishore Papineni, Salim Roukos, Todd Ward. and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
Jing Kai Siow Cuiyun Gao, Lingling Fan, Sen Chen, and Yang Liu. Core: Automating review recommendation for code changes. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 284 295. IEEE, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Petar Velickovic, Guillem Cucurull, A. Casanova, A. Romero, P. Liò, and Yoshua Bengio. Graph attention networks. ArXiv,
2018.
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S Yu. Improving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 397-407, 2018.
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. Code generation dual task of code summariza- tion. In Advances in Neural Information Processing Systems, pp. 6563-6573, 2019.
Edmund Wong, Jinqiu Yang, and Lin Tan, Autocomment: Mining question and answer sites for automatic comment generation. In 2013 28th IEEE/ACM International Conference Automated Software Engineering (ASE), pp. 562-567. IEEE, 2013.
Edmund Wong, Taiyue Liu, and Lin Tan. Clocom: Mining existing source code for automatic comment generation. In 2015 IEEF 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 380-389. IEEE, 2015.
Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Minhui Xue, Hongxu Chen, Yang Liu, Jianjun Zhao, Bo Li, Jianxiong Yin, and Simon See. Deephunter: a coverage-guidee guided fuzz testing framework for deep neural networks. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 146-157, 2019a.
Xiaofei Xie, Lei Ma, Haijun Wang, Yuekang Li, Yang Liu, and Xiaohong Li. Diffchaser: Detecting disagreements for deep neural networks. In IJCAI, pp. 5772-5778, 2019b.
Kun Xu, Lingfei Wu, Zhiguo Wang, and Vadim Sheinin. Graph2seq: Graph to sequence learning with attention neural networks. arXiv preprint 1rXiv:1804.0000 1804.00823, 2018a.
Kun Xu, Lingfei Wu, Zhiguo Wang, Mo Yu, Liwei Chen, and Vadim Sheinin. Sql-to-text generation with graph-to- seequee model. arXiv preprint wwXivv1809.05255 2018b.
Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. Modeling and discovering vulnera- bilities with code property graphs. In 2014 IEEE Symposium on Security and Privacy, pp. 590-604. IEFE, 2014.
12

---

Published as a conference paper at ICLR 2021
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. Retrieval-base based neural source code summarization. In Proceedings of the 42nd International Conference on Software Engineering. IEEE, 2020.
Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. arXiv preprint arXiv: 12223, 2019.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnera- bility identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems, pp. 10197-10207, 2019.
Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. Modeling graph structure in transformer for better amr-to-text generation. arXiv preprint arXiv: 2019.
13

---

Published as a conference paper at ICLR 2021
Appendices
A
DETAILS ON CODE PROPERTY GRAPH
Code Property Graph (CPG) (Yamaguchi et al., 2014), which constructed on abstract syntax tree (AST), combines different edges (i.c.. Flow to, Control) to represent the semantics of the program. We describe each representation combining with Figure follows:
Abstract Syntax Tree (AST). AST contains syntactic information for program and omits irrelevant details that have no effect on the semantics. Figure shows the completed AST nodes the left simple program and each node has a code sequence in the first line and type attribute in the second line. The black arrows represent the child-parent relations among ASTs.
Control Flow Graph (CFG). Compared with AST highlighting the syntactic structure, CFG displays statement execution order, i.e., the possible order in which statements may be executed and the conditions that must be met for this happen. Each statement in the program treated as an independent node as well as a designated entry and exit node. Based on the keywords if. for, goto, break and continue, control flow graphs can be easily built and "Flow to" with green dashed arrows in Figure represents this flow order.
Program Dependency Graph (PDG). PDG includes data dependencies and control dependen- cies: 1) data dependencies are described as the definition of variable in a statement reaches the usage the same variable at another statement. In Figure 2, the variable "b" defined in the statement "int b a++" and used in "call (b)". Hence, there is "Reach" edge with blue arrows point from "int = ++++*+++ to "call (b)". Furthermore, Define/Use edge with orange double arrows denotes the definition and usage of the variable. 2) different from CFG displaying the execution process of the complete program, control dependencies define the execution of a statement may be dependent the value of predicate, which more focus the statement itself. For instance, the statements "int = a++* and "call(b)"' are only performed "if a is even". Therefore, red double arrow "Control" points from "if (a % 2) == O' to "int a++" and "call(b)".
B DETAILS ON BASELINE METHODS
We compare our approach with existing baselines. They can be divided into three groups: Retrieval- based approaches, Sequence-basee based approaches and Graph- based approaches. For papers that provide the source code, we directly reproduce their methods on CCSD dataset. Otherwise, we reimplement their approaches with reference to the papers.
B. RETRIEVAL- BASEDD APPROACHES
TF-IDF (Haiduc al., 2010) is the abbreviation of Term -nequency-Inverrse Document Frequency, which is adopted in the early code summarization (Haidue et al., 2010). It transforms programs into weight vectors by calculating term frequency and inverse document frequency. We retrieve the summary of the most similar programs by calculating the cosine similarity on the weighted vectors.
NNGen (Liu et al., 2018) is retrieved-baateee approach produce commit messages for code changes. We reproduce such an algorithm on code summarization. Specifically, we retrieve the most similar top-k code snippets bag-of-worts model and prioritizes the summary in terms of BL.U-4 scores in top-k code snippets.
B.2 SEQUENCE- BASED APPROACHES
CODE-NN (lyer et al., 2016; Barone & Sennrich, 2017) adopts an sreention-based Seq2Seq model to generate summaries the source code.
Transformer (Ahmad et al., 2020) adopts the transformer architecture (Vaswani et al., 2017) with self-attention capture long dependency in the code for source code summrization.
14

---

Published as a conference paper at ICLR 2021
Hybrid-DRL (Wan al., 2018) is reinforcement learning-bas basedd approach, which incorporates AST and sequential code snippets into a deep reinforcement learning framework and employ evaluation metrics e.g., BLEU the reward.
Dual Model (Wei et al., 2019) propose a dual training framework by training code summarization and code generation tasks simultaneously to boost each task performance.
Rencos (Zhang et al., 2020) is the retrieval- based Seq2Seq model for code summarization, it utilized pretrained Seq2Seq model during the testing phase by computing a joint probability conditioned on both the original source code and retrieved the most similar source code for the summary generation. Compared with Rencos, propose a novel aarreetal-uggmented mechanism for the similar source code and use it at the model training phase.
B.3 GRAPH-BASED APPROACHES
We also compared with some latest GNN-based works, employing graph neural network for source code summarization.
GCN2Seq, GAT2Seq modify Graph Convolution Network (Kipf & Welling, 2016) and Graph Attention Network (Velickovic al., 2018) perform convolution operation and attention operation on the code property graph for learning and followed by a LSTM to generate summaries. We implement the related code from scratch.
SegGNN (Fernandes et al., 2018) combines GGNNs and standard sequence encoders for summa- rization. They take the code and relationships between elements of the code as input. Specially, BiLSTM is employed on the code sequence to learn representations and each source code token is modelled as a node in the graph, and employed GGNN for graph-level learning. Since our node sequences are sub- uub-sequence of source code rather than individual token, adjust to slice the output of BiLSTM and sum each token representation in node sequences as node initial representation for summarization. Furthermore, we implement the related code from scratch.
C MODEL SETTINGS
We embed the most frequent 40,000 words in the training set with 5 112-dims and set the hidden size of BiLSTM to 256 and the concatenated state size for both directions is 512. The dropout is set to 0.3 after the word embedding layer and BiLSTM. We set GNN hops to for the best performance. The optimizer is selected with Adam with initial learning rate of 0.001. The batch size is set to 64 and early stop for 10. The beam search width is set to 5 as usual. All experiments are conducted on the DGX server with four Nvidia Graphics Tesla V100 and each epoch takes 6 minutes averagely. All hyperparameters are tuned with grid search on the validation set.
D DETAILS ON DATA PREPARATION
It is non-trivial to obtain high-quality datasets for code summarization. We noticed that despite some previous works (Barone & Sennrich, 2017; 7; Hu et al., 201 8b) released their datasets, however, they all based high-level programming languages i.e. Java, Python. We are the first explore summarization on C programming language. Specifically, we crawled from popular C repositories (e.g., Linux and QEMU) GitHub, and then extracted separate function-summan pairs from these projects. Specifically, we extracted functions and associated comments marked by special characters "/**" and "*'" over the function declaration. These comments can be considered as explanations of the functions. We filtered out functions with line exceeding 1000 and any other comments inside the function, and the first sentence was selected as the summary. A similar practice can be found in (Jiang et al., 2017). Totally, we collected 500k+ raw fanction-summann functions with token size greater than 150 were removed for computational efficiency and there were 130k+ functions left. Since duplication is very common in existing datasets (Fernandes et al., 2018), followed by Allamanis (2019), we performed de-duplication process and removed functions with similarity over 80%. Specifically, functions into vectors with sklearn. Finally, we kept 95k+ unique functions. We name this dataset Code Summarization Dataset (CCSD). To testify model generalization ability, randomly selected
pairs. Furthermore,
calculated the cosine similarity by encoding the raw
15

---

Published as a conference paper at ICLR 2021
Table 5: More Examples of generated summaries the CCSD test set.
Example
Example
Example
static vold counterbitexFFoe
(sglite3_muter xp)
g. Nee.eeFreeeeeeeeeeeee (p->pReal);
ig.inii)));
static
subays exit (void)
Source Code
if( porpetpe==oLTTELLOOTT MOTEK__AF
p-selypess
winax id table r))ease(); genl unregiater family (6wimax fanilyy
QLITE
free
Ground-Trun
free countable mutex
shutdown the wimax stack
NNGen
enter countable mutex
unregisters pmnrrid event family retur value none
Transformer
mulex
de initialize wimax
Rencos
enter mulex
unregister the wimax device subsystem
SeqGNN
free a mutex allocated by sqlite3 mutex release a mutex
this function is called when the driver is not held free the wimax stack
HGNN
static int init. wimax subays init (void)
int reault; a_ NOLL, (e");) d_parse_ LEVEL, SIaE, wimax dabug params, "winax debug"); result - genl renl_rreiatte_aanii
stabic int counterkutaxfty (aglited xp]
assert g.isInit pr>etypa>=d
(6winax gnl family);
[unlikely (cesult 0))
assert p->etype NOTSXES ):
prr ('cannot segister genaric netlink family: adin", result); goto regiates family;) d (4,
Retrieved_code
paCountrrpp-eefype if dimableizy
SQLIVE_
O%a");
return g.m. (p
return 0;
ezzor register. family:
fnend NULL, "() vdin", result); result;
Retrieved summary Example
tyy enter Example 3
shutdown Example 4
void ReleaseSockEvent (SOCK_ revent) (event HULL)
static void uda dd free! struct 1pc32xx uda aude, struct 1pc32xx__sbe _sbd_dd__aa +dd)
Sourze Code
(Ralease (2vant)))))) 0)
ama_pool__fee free uud->ddd_achhee dd, dd-sthis_dda)
CleanupfockEvent (evant);
Ground Truth
a dana descriptor
socket
NNGie
allocatc dma descriptor
Clean of the socket event
Transformer
free the ush device
set the event
Rencus
allocate descriptor
set sock event
SeqGNN
free dma buffers
telease of the socket
HGNN
free dma descriptor
release the sock event
static struct 1pc32xx usbd dd gad 1pc32xx__de audc) f
L2TP_ SERVER +12tp, BOCK, EVENT +0) if (12tp = NULL)
dna addr dns;
struct 1pc32xx uabd dd gad vdd; dna_pool_
return:]
Retrieved_code
GFP ATOMIC GFP DMA, édma); if (dd)
(udo->dd eache,
if I= NULL)
if (12tp=>SochBvent 1- NULL)
(e-sze))))
dd-sthis_dee dma dna;
ReleaseBockBvent (12tp==Bock8veat))
12tp=> NOLL;
12tp-> );
Retneved summary
allocate descriptor
SCt the 12tp server
some projects as the out-of-domain test set with 2,330 examples and the remaining were randomly split into train/valiattionneest with 14.316/4,43224,4 examples. The open-source code analysis platform Joern (Yamaguchi et al., 2014) was applied to construct the code property graph.
E
MORE EXAMPLES
We show more examples along with the retrieved code and summary by dynamic programming in Table and we can find that I/GNN can generate more high-quality summries based approach.
16


Reviews:
[{"review_id": "_5DmQwJWuo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nOverview:\n\nThe authors tackle the code summarization problem. They retrieve the most similar code and use it as additional input features. They also using GNN to get features from static and dynamic graphs. They evaluate their results on their collected C projects (C-Code-Summarization Benchmark) with 1-2% improvement on automatic evaluation.\n\n\nReasons to accept:\n* They collect and release a new challenging C benchmark for code summarization.\n* They propose a hybrid-GNN solution to capture global graph information.\n\nReasons to reject:\n* No evaluation on any publicly available datasets. Even though there are some issues about duplication, I will still expect to see such a comparison with other baselines.\n\n* Attention-based dynamic message passing of graph model is not proposed in this paper, and I don't think it is necessary to have this \"hybrid\" design, maybe only dynamic one is enough. As shown in Table 1, the dynamic is more important than static (Although we do see an overall performance, except the out-of-domain meteor score, using both are still better, but it is marginal).\n\n\nQuestions & Suggestions:\n* Can you provide more dataset information? For example, the average lines of the code, the average length of the natural language summarization. It seems to me from the examples that each example only has few lines of code and the summarization is very short, it is more like a topic modeling task. \n\n* To my understanding, this work is not the first work combining retrieval solution with generation model for code summarization (e.g., Retrieval-based Neural Source Code Summarization) so please modify some of the claims in the paper.\n\n* Can you provide more details about the human evaluation? What is the agreement value? \n\n* \"z is the similarity score, which is introduced to weaken the negative impact of c′ on the original training data c\", how do you find the best c?\n\n* When you run baselines on your dataset, did you do a hyper-parameter search or just use their default setting (especially the Rencos model)?\n\n* There are some retrieved-augment language models in the NLP field that the authors may want to take a look and compare with, for example, \"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks\".\n\n* Did you run any baselines that are based on pre-trained language model, such as BERT, BART, T5, or even more code related like CodeBERT: A Pre-Trained Model for Programming and Natural Languages?", "extra_scores": {}, "extra_text": {}}, {"review_id": "US1E9bhU1QJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper leverages similar code-summary pairs from existing data to assist code summary generation. The model first retrieves a similar code snippet from the existing database. Then, the author applied GNN over the code property graphs (CPGs).  A challenge is that CPGs are typically deep therefore it is difficult to capture long dependencies. The author proposed an attention mechanism to capture global information between nodes, and then a hybrid GNN layer encodes the retrieve-augmented graph.  Finally, a generator takes both GNN's output and the retrieved text summary and predict outputs. Experimental results over a new C code indicates that the proposed method outperforms both IR and neural generation methods.\n\n########################################\n\nReason for score: \n\nOverall, I vote for accepting. Both the idea of leveraging existing code and also the adaptive layer to capture long dependencies are interesting and the experiments look solid. Although I would still like to see the results from previous existing datasets.\n\n########################################\nSome comments about the experiments:\n\na. As an application study, it is still necessary to compare the model over previous benchmarks, even though there are some issues with those datasets. \n\nb. A pair of missing ablation studies are: a generator still takes the text summary of retrieved code, but not use the augmented graph; and vice versa, the generator only takes the graph information but not the retrieved text summary. This can further indicate which part of the retrieved information is more useful. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "3wrO0jIvyCU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a retrieval-augmented method for generating code summarization. The model encodes the input code based on its graph structure (Code Property Graph) with a hybrid GNN architecture. The model augments the initial graph representation of the input code based on the representation of the top-1 retrieval result. It also augments the final graph encoding with the BiLSTM encoding of the retrieved summary.\n\nThe proposed model is evaluated on a newly curated C code summarization data set and shows state of the art performance compared against previous systems.\n\nStrengths\n- Releases a new C code summarization data, which will be beneficial for the community.\n- This paper reports human evaluation results.\n- Ablation study shows the retrieval augmentation and the new hybrid GNN architecture is helpful.\n\nWeaknesses\n- The model is not evaluated on any existing code summarization benchmarks. Showing that the proposed architecture is generally applicable by getting good results on more benchmarks will make the story a lot more convincing.\n- Could include more analysis (see below for details)\n\nOther questions/comments\n- Would be nice to provide the mathematical formulation of “Step 3: Retrieved Summary-based Augmentation”\n- Would it be possible to ablate code-based summarization vs. summary-based augmentation separately? It would be interesting to see their relative impact.\n- Suggestion: it would be helpful to provide an input-output example earlier in the paper.\n- Suggestion: would be nice to include real examples of retrieval results in the analysis section.\n- Have you considered using top-k retrieval results instead of top-1?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an interesting method for combining retrieval-based models and graph neural networks for source code summarization. Finding new ways of bringing in additional context for graph-based models is an important research direction in this space, and the paper presents a novel and effective approach. The initial submission was missing experiments on existing benchmarks, but new experiments presented in the discussion phase are enough to resolve that concern. Reviewers are unanimously in support of acceptance. 
Decision: Accept (Spotlight)