Pyramid Attention For Source Code Summarization
Authors: Lei Chai
Ming Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning
source code summarization
software
code understanding

Abstract: This paper presents a multi-granularity method for source code summarization, which generates a concise functional description for the given code snippet. We notice that skilled programmers write and read source codes hierarchically and pay close attention to conceptual entities like statements, tokens, sub-tokens, and the mapping relations between them. The entities have specific emphasis according to their granularities, e.g., statements in coarse-granularity reveal the global logical semantics of code, and the sub-tokens in fine-granularity are more related to the textual semantics. Driven by this observation, we demonstrate that a multi-granularity formulation incorporating these conceptual entities benefit the code summarization task. Concretely, the source code is transformed into a pyramidal representation, and then a pyramid attention mechanism is applied for efficient feature aggregation among different hierarchies in it. We instantiate our multi-granularity method using the proposed pyramid attention and name it PA-former (Pyramid Attention transformer). We evaluated it on two source code summarization benchmarks where it surpasses the prior works and achieves new state-of-the-art results. Our code and data are available at https://github.com/leichainju/pa-former.

Pyramid Attention For Source Code Summarization
Lei Chai and Ming Li" National Key Laboratory for Novel Software Technology. Nanjing University, Nanjing 210023, China {chail, lima .nju. edu.cn
Abstract
This paper presents multi-granularity method for source code summarization, which generates concise functional description for the given code snippet. We notice that skilled programmers write and read source codes hierarchically and pay close attention to conceptual entities like statements, tokens, sub-tokens, and the mapping relations between them. The entities have specific empha- sis according to their granularities, e.g., statements in coasse-ranullarity reveal the global logical semantics code, and the sub-tokens in fine-gramularig are more related to the textual semantics. Driven by this observation, demon- strate that a multi-granularity formulation incorporating these conceptual enti- ties benefit the code summarization task. Concretely, the source code is trans- formed into a pyramidal representation, and then a pyramid attention mecha- nism is applied for efficient feature aggregation among different hierarchies in it. We instantiate our multi-granularity method using the proposed pyramid at- tention and name it PA-former (Pyramid Attention transformer). We evaluated it on two source code summarization benchmarks where it surpasses the prior works and achieves new state-of-the-art results. Our code and data available at https: / ////gttii github...... wwtptt....meeeciiiiii..eeea com//eechainjjupa---aa--ae former.
Introduction
Automatic source code summarization is drawing increasing attention to its promising application prospects in software development and maintenance. This task is challenging due to the complex syntax structure in programs and the arbitrariness of variable naming. Summarization methods need to take advantage of textual and grammatical information in programs to learn a meaningful representation for concise description generation. Following the early stage works [12, 30] that treated the code as a sequence text and directly adopted the well-developed seq2seq [28] models for this task, recent leading methods [27, 25] further exploit the explicit structure (i.e., abstract syntax tree or pair-wise relations extracted from it) in programs to boost the model performance.
Current structure-basee based code summarization methods are either in hybrid way [27, 1, 13, 3] or in structure-guided way [25, 35, 9, 19]. Works hybrid fashion encode the AST-based structure using structure-awar models like GGNN [16] or Tre [29], and then combine the structural repre- sentation with the textual representation learned by sequential models for the decoding process. The structure-guides approaches use the pair-wise relations between tokens (or sub-tokens) extracted from code structure as inductive biases to guide the learning process. While these methods achieve perspective, which can
excellent results, they only model the source code from be further improved in a multi- multt-granullartyy manner.
To understand given code snippet, skilled programmer hierarchically breaks it down into fine- grained entities (i.e., sub-tokens) and builds up conceptual semantics gradually from fine-grained
Ming Li is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

public parseStringksinn(Sernngg in> tryl
return (1))
course course-grained
catch Kunberfoorattxccption ex Uhrow IndexParsef wtw.e/eeeeeeeeeeeeeeetotoooooooe e]i
semi-grained
label the spon with stalement type
fine-grained
declaration
paraneters
try
return stnt
catch
throw_staf
(d) the multi muutti-raanlaritt pyramid structure of source cade.
(a) code span segmentation in sddement-leve granularity.
(ars:Int(in);;
label the spon with token-wise grammar type
Reference
purses string
retuen
class_ nome
mathed, none
argunent
PA former
parses string integer if the parsed inseger:
(ours)
(b) intra statement span segmeniation in loken-leve level granularity.
TPTraans
parses string value from string representatinn
lobel the span with the subtoken text
parseInt
CAST
parse string as number in the pursed declaration.
NeuralCS
parses string as unid.
int
(c) intra-token token span segmentation subtoken- granularity.
(e) summaries predicts by various
Figure 1: Example of dlmann-lik code comprchension. Skilled programmers hierarchically break the source code down to fine-grained entities (i.e., sub-tokens) and gradually build up conceptual semantics understanding for the code from fine-grained entities to coarse-grained ones.
entities to coarse-graines grassed ones. We now use the example shown in Figure illustrate the multi- granularity process for code understanding. Concretely, we first segment the whole code into several coarse-graine searse-grained spans according to punctuations and grammar, where span corresponds ment conceptually (Figure ((a)). Given the coarse-grained sequence, we can roughly summarize the code as return something if something is done. After that, we dive into each statement and do intra-statement segmentation for it (Figure 1(b)). Now we can detail the first something as return something produced by method invocation. To further clarify which method invoked and what is produced, we go deep into each token and further segment the token into sub-tokens. Here, we can figure out that an int parsing method is invoked and an int is produced (Figure (c))). We fi- nally roll back the details from fine-gra fine-grained entities to coarse-grained ones and obtain the overall understanding of the given code.
state-
Driven by the above analysis, we propose our marization by mimicking human behavior. We first construct pyramidal input, which is divided into tme-grained layer, semi-graines layer and coarse soarse-raiind layer from bottom to top (Figure 1(d)). And novel pyramid attention mechanism proposed for efficient feature aggregation among these layers and produces a pyramidal representation with rich semantics. Because both -earse-grained logical patterns and fine-grained textual features are considered simultaneously, our model can pre- dict more comprehensive summary. Taking the code illustrated in Figure 1(e) as example, compared with current leading methods, our model notices the try ... catch ... structure in this code snippet and uses the additional if the string can be parsed handling pattern.
formulation for source code sum-
int. to tell the captured exception
Without bells and whistles, vanilla Transformer [31] equipped with the proposed Pyramid Attention (PA-former) achieves about 5% performance improvement (evaluated by BLEU [24]) on a Java dataset. PA-former is evaluated on two source code summarization benchmarks where it surpasses the prior works and reaches new state-of-the the---- results. And ablation studies are conducted to show the efficiency of the proposed method.
2 Related Work
Source code representation. Learning-based source code comprehension drawing increasing attention as its promising application prospects in the field of software development and software maintenance. A meaningful representation can be used for various downstream tasks, such as code clone detection [34, 33], code classification [23, 37, 3], code summarization [10, 30, 3], code search

---

[7], etc. Early learning-basse works treated source code sequence of text ignoring the structure features. Mainstream base models like CNN [2]. Attention-based LSTM [12] and Transformer [30] were all used directly to capture the textual patterns in programs.
Many follow-ups leveraged both textual and structural information to learn more meaningful code representation. Tree-based models like TBCNN [23], Code-GRU [17], ASTNN [37] were proposed to represent the source code by considering the tree structure. Some works [1] extended ASTs into graphs by introducing control flows and data flows, then a GGNN was applied to represent the code graphs. The set of paths [3, 4] extracted from AST was also used as a representation of the corresponding code. Some works [11. 8, 10] flatten the ASTs into sequence, and then the off-the-shellf sequential models are applied to learn the textual and syntax features. Currently, most attention is focused on guiding the attention computation in Transformer by using the structural bias in programs [25, 9, 38).
Source code summarization. To generate a functional description for the given code snippet, code summarization models should capture both textual and structural information in programs. Some early works [12, 30, 32] just treated the source code as sequence or flatten the AST using a structure-based traversal (SBT) method [10] and then performed a conventional seq2seq learning. The hybrid works represented both textual and structural features separately, then hybrid mecha- nism is used to combine the learned representations for the summary generation. HDeepcom [11] applied the SBT method for structural representation and CAST [27] learning the structure via hi- erarchical splitting and reconstruction of abstract syntax trees. Many Transformer-base methods [27, 1, 13, 3] used structure information to guide the attention computation. TPTrans [25] encoded the pairwise path between code tokens and incorporated the path embeddings like relative position encoding. Unlike the above-mentioned methods which view the source code as single-granularity data, we propose a multi-granularrty source code summarization method.
Multi-scale feature learning. The multi-scale feature processing is widely used in the computer vision community. FPN [21] leverages the pyramid-shaped features of ConvNet and creates a feature pyramid that has rich semantics at all scales. Recently, the multi-scale feature hierarchies were also introduced into Vision Transformer (ViT) by a variety of works like MViT [6]. Swin [22] and HRFormer [36], which show strong evidence that Transformer could work well with the multi-scale features.
Approach
As shown in Figure 2, given source code, we first construct sequences with three granularities and form them into a pyramid-shaped structure based on the mapping constraints between different granularities. After that, the proposed pyramid attention mechanism (in Figure 3) is applied layer by layer in the encoding stage to produce a feature pyramid which is used by a vanilla Transformer decoder to generate the summary. The details of the proposed pyramidal input constructor and pyramid attention mechanism are introduced in this section.
3.1 Pyramidal Input Constructor
The pyramidal input is constructed following the human code comprehension process described in Section Entities in different granularities have their own emphases, telling specific attributes of source code from different viewpoints. For instance, the yoarse-graine statement sequence reveals the global logical semanties of code, the semi-grained token sequence shows the code grammar details, while the fine-graine, sub-token split from each token is attached with textual information. We introduce the details of the proposed pyramid input constructor the following.
Fine-grained sequence. Each leaf node in the AST corresponds to a token (like boolean, value) in the source code. Since method names and variable names in programs are usually composed of multiple natural language words with CamelCase or snakee case, we can further split these combined tokens into sub-tokens, which reveal the textual information of source code. To construct the fine- grained layer (i.e., sub-token sequence), we first obtain the leaf nodes sequence of AST by depth-first first pre-order traversal and then split each token into sub-tokens. Such a sub-token technique is widely used by source code summarization methods [30, 25, 27] and proved to be effective. Formally, let
3

---

Feature Pyramid
Output
Probabilities
Copy
Add Norm
Feed
Linear &Softmax
Nx
Forward
Pyramid Input Constructor
Pyramid
Vanilla Decoder
Attention
XN
Position
Source Code
Pyramid Input
Outputs
The proposed pyramidal input with three granularities, and then novel pyramid attention based encoder is applied to produce feature pyramid which is used for decoding.
Figure 2:
source code summarization pipeline. We first construct a
T T= 42, 11.,,,,..,44 [N] (T for text) denote the ordered sub-token sequence, that is, the fine-grained input representation situated in the bottom of the pyramidal input. For the example in Figure 1, T is [public, static, int, parse, String, As, Int, ....
Semingrained sequence. The attribute of the parent node of each token (i.e. leaf) in parsed AST represents its grammatical property. We construct the semi-grained sequence by replac- ing the element ordered token sequence with its grammatical property directly. Formally, let g= [91, 92, ... 99M] (g for grammar) denote the ordered token grammatical sequence, that is, the semi-grained input representation situated in the middle of the pyramidal input. For the example in Figure 1, G is [modifier, modifier, type, identifier, (, ... In addition, one-to- many mapping relationship Mc between token and sub-token is introduced. Mc (j) = if the sub-token is split from 9i. which will be used to guide the information aggregation process from fin-nne hierarchy to semi -rainer hierarchy.
Coarse-grained sequence. We use the nodes corresponding to statements to form our coarse- grained sequence, which reveals the logical information of the source code. As the statement type limited, we directly identify the AST node as a statement-level node if the tag belongs to the pre-defined statement type set (see Appendix C for more details). Formally, let L (2.,2...., [x] (C for logic) denote the sequence of the ordered statements, that is, the coarse- grained input representation situated in the top of the pyramidal input. For the code in Figure 1, Ls [declaration, parameters, try, return_ ceccuunnstttemeeee stent ...]. Similar to the semi-grained sequence, we also construct one-to-many mapping function Mc-. between statement and token, which bridges the coarse- saase-grained hierarchy with the semi-grained hierarchy. Mg-a0(1 = the statement I; is the most recent ancestor of token 9j among L in AST.
In summary, we construct sequence set (T, 9, L} in three granularities which helps the model to capture patterns from different scales and extract mapping relationships MM---.,M->>> between adjacent granularities which introduce inductive bias to aid the model to learn accurate semantics. Thus, the sequences and the mapping relationships all together form the pyramidal input representa- tion of the given code.
3.2 Pyramid Attention
The pyramidal layers (T,9,4) complement each other following the mapping relationships MyrTh. We propose novel pyramid attention mechanism for efficient feature ag-

---

Multi-Head Attention
Cross- cross-granularity interaction
C
Concat
C
E
Linear & Norm
Bottom-to-top aggregation
Aggregate & Add
T
Figure 3: Pyramid attention mechanism. The pyramid attention takes a pyramid-shaped data de- scribed in Section 3.1 as input, and outputs multi-scale representation without changing the orig- inal pyramid shape. 'he whole process consists of bottom-to-top aggregation pathway and a cross-granularity information interaction process.
gregation among these layers. As illustrated in Figure 3, the pyramid attention module includes a bottom-to-top aggregation pathway and granularalrrrt information interaction process. We describe the details in the following.
Bottom-to-top aggregation. The bottom-to-top pathway does a fine-to-coarse information ag- gregation, which ensures to enrich the higher-level representation using the lower-level semantics. Given the pyramid input data: {T.9, C} and the mapping function [MC-g.MM-rT]) we aggre- gate the representation of T into w.r.t MG_T. The formulation of the aggregation process is given as:
9: LayerNorm (
9, + W ΣJ=1 Mo-r Σ5 (g
Bagg
(1)
Where W agg E DDD bagg E DRD are learnable parameters, 9ú, la E R.D are the i-th representation vectors in and T respectively, 1) represents the dimension the representation, and 9; is the aggregated representation of 9,- The aggregation process from G into L w.r.t. Me- is the same as above. further conduct aggregation directly from into w.r.t. MC-7, which obtained by:
Mc-T(1, V.E(I....m)) (Mc-g 9w) Mg (gx. t2))
(2)
In summary, we update [}.L} into (G. using T and (T.9} respectively. The mapping- based aggregation reasonable and necessary as the semantics of the element high levels come from the combination of its descendants' semantics. The semi-grained sequence and maare--gainned sequence are initialized as the grammatical type of the tokens and statements. Figuratively speaking, they are like torsos with empty skeletons, and we need to fill them with textual information to make them complete. There an ablation experiment demonstrating its effectiveness in Section 4.3.
Cross-granularity interaction. The aggregated semantics of the elements in semi/coarse-grained sequences tend be local. To alleviate this dilemma, further make the elements specific granularity to query all elements at three granularities, which guarantees that the model can capture both grained and coarse grainedd features.
Formally, we now have (T, G. З after aggregation. Taking sequence ğ MXD of length M in certain granularity as an example, G will query all of (T,ğ, 7, L) in the cross-granularity interaction process. We first concatenate the pyramid representation
C = Concat [T, G, 2) E TR (K+KM+N)
here N, M. K are the sequence lengths of T. T. å respectively. relative position encoded multi- head attention [26] is applied here and we use the relative position map R² Mx make the
5

---

model granular-award The multi-head cross-attentin (MHCA) process is calculated as:
= MMutiHead((,.C) W. E RXXD
(4)
MultiHead = Concat (onaat(Head,
(,)))(( E RAXD
(5)
Head, (8, = [21, 22,
xM] E RM*#
(6)
[cl
2, exp Kij
C, W R.g W retw) E I
1=1 j'=1 expoxi Oiij'
w
T
0gj = VD/H C + Raj W relk)
E
(8)
R Embedding σ (Rē) E M
99)
where W
q: W k: w E RR E [1...., H] are projection matrices of Query. Key and Value for /-th head. W. E RDDD; is projection matrix for attention output. H represents the number of heads and X represents the output of multi-head self-attention process. w w IR relkk* rel_k
projection matrices of Key and Value of relative position embeddings, the G means that they specific to G.
T
C
5
5
5
5
5
5
5
0
2
3
4
4
5
5
5
5
5
5
5
1
1
2
4
4
g
5
5
5
5
5
5
5
2
0
4
4
5
5
5
5
5
5
5
5
5
3
1
0
4
4
Figure 4: toy example of granular-aware relative position map for G cross-granularity interaction.
For the construction of relative position map R. we calculate the clipped relative distance with max distance as 32 for the pair within granularity and assign the relative distance default value (e.g. max distance + and max distance + 2) for all the pair across the specific granularity. Figure shows toy example of granular-aware relative position map with max distance as 3 for G cross- granularity interaction. The cross- granulartyy interaction for T and L is similar to G. We share attention parameters (i.e. Wh. W/. W and W. across three granularities, but assign specific rel- ative path parameters for each granularity. In summary, in this cross grosularanulrrtty interaction process, the semantics of the feature pyramid are updated as:
Σ.g,ī= MMCAA C),
(10)
3.3 Pyramid Attention Transformer
As illustrated in Figure 2, the overall model is similar the vanilla Transformer. The self-attention module in the encoder is replaced with the proposed pyramid attention module for an efficient pro- cess of the multi-scale input. Notably, we only use the fine-grained sequence in the pyramidal representation for decoding, which already contains enough semantics for summary generation and reduces the complexity of the model design. Following prior works [30], we also apply copy mechanism in our model which reduces the risk caused by vutcof-vocabulryy and speeds up training convergence.
Although there is overlap between the vocabularies of different granularities, the meanings of these common words are quite different when differentiated at different granularities. For instance, else may be just a natural word in the textual sequence, but a keyword for control in a grammatical sequence. Thus, we assign individual embedding layers for each granularity avoid such confusion.
For efficient parameter usage, a parameter sharing strategy also applied the cross-granularity interaction process as stated in Section 3.2, We find the self-attention mechanism is strong enough

---

to handle this multi-role mission. Thus, our model doesn't t add too many parameters compared to Transformer. What's more, such parameter-sharing strategy also brings efficient model implemen- tation. We only need to concatenate these three sequences and the relative position maps separately, then directly adopt a relative position encoded multi-head self-attention layer to implement the cross- granularity interaction module.
4 Experiments
4.1 Experiments Setup
Dataset and Metries. To demonstrate the effectiveness of the proposed method, we conduct ex- periments on two widely-used and well-developed java datasets: EMSB-DeepComm [11] which is collected from GitHubs Java repositories and FunCom² [14] which has ~2 million java method- comment pairs. We filter out the examples that cannot be parsed properly or are of too large size for both datasets. And deduplication is done on EMSE-DeepCom avoid data leakage. Table I shows the statistics of the datasets.
Table 1:
Dataset statistics. #train and #test mean the number of examples for training and testing, #sub-token means the max sequence length when the code is tokenized as sub-token, and the rest are similar to this.
Name EMSE-Deepcom FunCom 1,017.964
#train
#test
#sub-token
#token
#statement
#summary
295,967
12,226
196
160
16
24
53,936
256
196
32
32
We use 6 metrics to evaluate the model performance on the datasets. BLEU [24]. Rouge-L [20] and Meteor [5] are widely used to evaluate the quality of text similarity in the domain of natural language generation. Another 3 metrics are used to qualify the token-wise prediction performance. We use beam search with a beam size of 3 for all models in the inference phase.
Data preprocessing. All data preprocessing of code is based the parsed syntax tree. We obtain the AST using the open-sourced tool Tree-sitter'. Please refer to 3.1 for details of the proposed pyra- mid data construction. We re-implement the data preprocessing pipeline for all compared baselines based on the AST parsed by Tree-sitter, please refer to Appendix C for more details.
Baselines. Early works viewed source code as a sequence of text and used the sequential model like LSTM or Transformer, we mainly compare with CODE-NN [12] and NeuralCodeSum [30] for this type of methods. A few works only learned the structure features for this generation task, compare with TreeLSTM [29] here. Most works exploited the textual and structural information for this task. One group of works learned represent these two types of information separately, then hybrid mechanism is used to combine them in the decoding phase. We do comparisons with HDeepCom [11], ASTAttGRRU [13] and CAST [27] for the hybrid fashion. Another line of methods used structure relations as an inductive bias to guide the information interactions in the model. Here, we compare with SiT [35], GREAT [9] and TPTrans [25]. For elarity, the backbone and taxonomy of each method are listed in Table 2.
And copy mechanism is added for all transformer-based models. We directly adopt the model implementation script if the code is publicly available, and re-implement the method according to the description in the corresponding paper the code is not provided or not available. What's 's more, all baselines are integrated under unified PyTorch-based code summarization framework developed by ourselves for fair comparisons. We provide implement details for baselines in Appendix A.3.
Hyperparameters and training setup. For fair comparisons, all the Transformer-base models use the default Transformer configurations with embedding dimension as 512, feedforward dimen- sion as 2048, head number as 8, and layer number for encoder/decoder as 6 and all RNN-based
http://ecccair...cccaaaaaaa//a/a 'tttps:/ttreessitter
7

---

Table 2: Comparisons with other code summarization methods on middle-sial dataset and large-size
dataset.
Methods
Backbone
Hybrid
Structure
Epochs
BLEU
Rouge-L
Meteor
Precision
Recall
Fi
DeeSp-Deepoom (middle-size)
CODE-NN [12]
LSTM
24
28. 448
43.506
17.886
47.266
45.165
44.774
TreLLTTM [29]
LSTM
43.985
18.179
48.486
45.115
45.287
IDDeepCom [1]]
GRU
24
32.179
49.029
21.528
54,273
50.584
50.754
ASTAnnGRU [13]
22.205
54.519
51.754
51.467
SiT [35]
Transformer
30
35.689
53.750
24.196
60.784
54.243
55.717
GREAT
Transformer
36.382
53.606
24.181
60.0088
$4.230
55.460
NeuralCodeSumm [30]
Translormer
30
37.133
54,800
25.051
61.356
55.413
56.682
CAST [27]
TPTrans [25]
Transformer
30
37.248
54.996
25.022
62.022
55.354
56.884
PA-former (ours)
Transformer
38.848
56,095
25.895
62.498
56.642
57.903
Funcom (large-siza)
CODE-NN [12]
LSTM
24
31. 168662
48.8% 48.896
19,108
54.018
49.241
49.924
TreLLTMM [29]
LSTM
18.869
53.237
HDeepCom [1]]
GRU
24
35,063
53.350
22.645
59,460
53,693
54.808
ASTAmaGRU [13]
GRU
37.001
55,034
23.753
61.244
55.315
56.523
SiT [35]
Transformer
36
42,121
59.330
26.819
65,562
59 39.427
60.839
GREAT
Transformer
60.364
27.439
66.186
60.580
61.831
NeuralCadSSum [30]
Transformer
36
43,355
60.405
27.540
66,488
60:459
61.860
TPTras [25)
Transformer
43,450
60.566
27.607
66.765
60.568
62.030
CAST 127)
Transtormer
36
43,580
60.524
27.665
66.458
60.657
61.976
PA-former (ours)
Transfoemer
44.649
61.450
28.274
67.210
61.593
62.863
models use the hidden dimension with 512. For embeddings, we use the embedding dimension with 512 for all approaches. The size of vocabularies limited by 50000 over code sequences and 30000 over summary sequences in EMSE-DeepCom, and the corresponding configurations are 35000 and 30000 in FunCom. All models are trained using NVIDIA Tesla A100 GPUs with a batch size of 64. We train all baselines including our models using Adam W optimizer with a multi mullii_seep learning rate scheduler, and set the initial learning rate to 0.0002 and 0.003 for Transformer based and RNN- based models, respectively. And warmup strategy is used for stable training in the Transformer- based model. The training epochs for each method are listed in Table
4.2
Main Results
Comparisons with baselines. The comparisons with baselines are listed in Table 2. Without bells and whistles, we find that our proposed PA-former substantially surpasses all baselines and achieves both middle-size and large-size datasets. Compared with NeuralCodeSun
new state-of-the-an which is a relative positional encoded Transformer, our method has a 5% improvement in BLEU on the middle-size dataset. For the current leading methods CAST and TPTrans, which both take all the nodes in ASTs and the sub-token sequence as input, our model only takes a subset of AST nodes and tub-tokens surpass both of them over 1.0+ BLEU scores implementation and parameter sharing strategy, our model requires far less training time to converge, while TPTrans needs very long time converge the large size pair-wise paths and the sparse scatter algorithm. The leading results demonstrate that our multi-scale formulation for the task of source code summarization is reasonable. To further evaluate the effectiveness of our method, provide a case study in Appendix D. Moreover, we provide the results on RMSE-Deepcom dataset with error bars to show that the improvement achieved by our method is statistically significant in Table 11 at Appendix B.
datasets. And thanks to the efficient
Human evaluation. As a complement, human evaluation is conducted over the vransformer-based methods following the previous works [12, 27, 15]. We invite 5 volunteers with over 5 years of programming experience and ask them to score (0- 10, 10 is the best) the generated summaries over 100 examples sampled randomly from the testing sets (20 from RMSE-Deepcom and 80 from Fun- com). Here we focus on two key points: naturalness (grammaticality and fluency of the generated summary) and usefulness (what extent the generated summary is useful to understand the code). We divide the 100 examples into 5 groups (each one has 20 examples) and assign each group into

---

Table 3: Human evaluation results over 100 examples sampled randomly from the testing sets (20 from RMSE- Deepcom and 80 from Funcom).
Methods
Naturalness
Usefulness
NeuralCodeSum
5.12
4.98
TPTrans
5.51
5.23
CAST
5.55
5.19
PA-former (ours)
6.09
5.86
different volunteers, and the final score of each example is the average of its 3 "review" scores, As shown in Table 3, our PA-former outperforms others in both naturalness and usefulness.
4.3 Ablation Study
In this part, analyze the importance of designs in proposed PA-former using series of ablation studies on EMSE-DeepCom dataset. Unless otherwise specified, the training setup is the same above.
Multi-scale v.s. single-scale. To demonstrate the effectiveness of multi-granularity formulation, degenerate the pyramid input into single large sequence by concatenating the three granularity sequences and directly feed the same sequence into a relative positional encoded Transformer with copy mechanism. As shown Tabel 4, more data in the input sequence obtains performance im- provement compared to NeuralCodeSum which only takes a sub-token sequence as input. However, processing the same data using our proposed multi-scale way achieves higher performance.
Table 4: Evaluation results of multi--cale single-scale.
Methods
BLEU
Rouge-L
Meteor
Precision
Recall
F1
multi-scale
38.848
56.095
25.895
62.498
56.642
57.903
single-scale 37.329
55.032
25.114
61.586
55.750
56.946
Effectiveness of pyramid attention. The core components of pyramid attention are the bottom- to-top aggregation and cross-granularity interaction. We show the necessity for both of these two components here. To demonstrate effectiveness of bottom-to-top aggregation, we evaluate vari- ant of PA-former whose bottom-to-top pathway is disabled. As shown in Table 5, the model perfor- mance drops by about 1.1 BLEU scores without such a component. demonstrate the effectiveness of cross- gross-granularity interaction, we evaluate variant of PA-former whose cross-granularig inter- action is disabled which directly degenerates into NeuralCodeSum as we only use the fine- gre-grained encoder output for decoding. As shown in Table 5. the model performance drops by about 1.7 BLEU scores without such component. Such results reveal the importance of cross- goass-rranllriiics interac- tion. We further disable the granular-aware relative embedding to investigate its importance, and observe a big drop in performance shown in Table 5, which shows its effectiveness.
Table 5: Evaluation results of pyramid attention ablation.
Methods
BLEU
Rouge-L L
Meteor
Precision
Recall
PA-former
38,848
56.095
25.895
62,498
56.642
57.903
w/o to-
37.532
55.153
25.131
61.815
55.590
56.977
w/o cross-gre moss-granularity
37.133
54.800
25.051
61.356
55,413
56.682
w/o granular-awer -anare rel.
36.905
54.690
24.896
61.443
55.224
56.560
Decoding strategy. The output our PA-former encoder pyramid representation, thus there are various ways for decoding. Our PA-former directly uses the fine-grained sequence just for sim- plicity. Here, we study other implementations that utilize all three granularity of representation. The most intuitive method is concating the three granularity representations, and another way is using

---

them in serial strategy [18]. As Table 6 shows, using the fine-graim representation is good enough in this task benefit from the design of information interaction across granularities. The results bit counterintuitive, we think a reasonable explanation is that the fine-grained representation al- ready contains enough information benefited from the proposed pyramid attention. When three granularities are used, there is severe information redundancy between different granularities, which complicates the task.
Table 6: Evaluation results of decoding strategy.
Decoding strategy
BLEU
Rouge-L Meteor
Precision
Recall
F1
fine-grained serial
38.848
56.095
25.895
62.498
56.642
57.903
38.729
55.894
25,803
62.223
56.425
57.669
concat
38.556
55.819
25.797
62.174
56.456
57.638
Grammar input. When constructing the pyramid input, intentionally enforce the bias between different granularities by assigning different types of content. For example, we bias the semi-graine sequence into grammar features by assigning token grammar types to it. To evaluate the rationality of assigning different types of content, we compare the proposed pyramid input without grammar information, which the initial content of semi-grained and yoarse-graings sequences to empty. The results in Table 7 prove the rationality of pyramid input design.
Table Evaluation results grammar input.
pyramid imput content
BLEU
Rouge-L
Meteor
Precision
Recall
F1
PA-former
38.848
56.095
25.895
62.498 62.198
56.642
57.903
w/o grammar
38.394
55.744
25.706
56.458
57.637
Conclusion
In this paper, present multi-scale formulation for source code summarization. Given code snippet, we construct pyramidal input with three granularities. And a novel pyramid attention mechanism is proposed for efficient information interaction across granularities. Without bells and whistles, the proposed method obtains 5% improvement over the strong baseline and achieves new state-of-the-ar results. Additional ablation studies show the effectiveness of the proposed modules. Furthermore, our multi-scale formulation for the source code can easily extended to other code- understanding tasks. In future work, would try scale our model up and pre-tret pre-train it larger source code corpora to further improve its capability and generalizability on source code compre- hension tasks.
6 Acknowledgement
This research was supported by NSFC (62076121, 61921006). The authors would like to thank Ying Li, Hao- Yuoo He, Ren-Biao Liu, Yu-Xi Sun and Zhi- Yu Shen for helpful discussions. Finally, We are grateful to all reviewers for their insightful comments.
10

---

References
[1] M. Allamanis, M. Brockschmidt, and M. Khademi. Learning represent programs with graphs. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
[2] Allamanis, H. Peng, and Sutton. A convolutional attention network for extreme summarization of source code. In International Conference on Machine Learning (ICML), 2016.
[3] U. Alon, S. Brody, O. Levy, and E. Yahav. code2seq: Generating sequences from structured represen- tations of code. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[4] U. Alon, M. Zilberstein, O. Levy, and E. Yahav. code2vee:
Learning distributed representations of code. In Proceedings of the ACM Programming Languages (POPL., 2019.
[5] S. Banerjee and A. Lavie. Meteor: An automatic metrie for evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Trunslation and/or Summurization, 2005.
[6] H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer. Multiscale vision trans- formers. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2021.
[7] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou. Codebert: pre-trained model for programming and natural languages. In International Conference on Learning Representations (ICLR), 2020.
[8] D. Guo, Lu, N. Duan, Wang. Zhou, and J. Yin. Unixcoder: Unified cross- modsl pre-training for code representation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
[9] V. J. Hellendoom, P. Maniatis, R. Singh, C. Sutton, and D. Bieber. Global relational models of source code. In Proceedings of the 8th International Conference Learning Representations (ICLR), 2020.
[10] X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, Deep code comment generation. In Proceedings of IEEF/ACM International Conference on Program Comprehension (ICPC), 2018.
[11] X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin. Deep code comment generation with hybrid lexical and syntactical information. Empirical Software Engineering, 2019.
[12] S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer. Summarizing source code using neural attention model. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (ACL), 2016.
[13] A. LeClair. S. Haque, L. Wu, and C. McMillan. Improved code summarization via graph neural network. In Proceedings of International Conference Program Comprehension (ICPC), 2020.
[14] A. LeClair and C. McMillan. Recommendations for datasets for source code summarization. In Proceed- ings of NAACL-HLT, 2019.
[15] J. Li, Y. Li, G. Li, X. Hu, X. Xia, and Z. Jin. Editsum: retrieve-and-cdit framework for source code summarization. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021.
[16] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated graph sequence neural networks. In Proceedings of the International Conference Learning Representations (ICLR), 2016.
[17] Y. Liang and K. Q. Zhu. Automatic generation of text descriptive comments for code blocks. In Proceed- ings of the 32nd AAAI Conference Artificial Intelligence (AAAI), 2018.
[18] J. Libovický, J. Helcl, and D. Marecek. Input combination strategies for multi-source transformer decoder. In Proceedings of the Third Conference on Machine Translation (WMT), 2018
[19] C. Lin, Z. Ouyang. J. Zhuang, Chen, H. Li, and Wu. Improving code summarization with block wise abstract syntax tree splitting. In Proceedings of 29th IEEE/ACM International Conference on Progrum Comprehension (ICPC), 2021.
[20] C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Proceedings of the 42th Annual Meeting of the Association for Computational Linguistics (ACL), 2004.
11

---

[21] T.-Y. Lin, P. Dollt'arl, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[22] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang. S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
[23] L. Mou, G. Li, L. Zhang, Wang, and Z. Jin. Convolutional neural networks over tree structures for programming language processing, In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2016.
[24] K. Papineni, S. Roukos. T. Ward, and W.-J Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics (ACL), 2002.
[25] H. Peng. G. Li, W. Wang. Y. Zhao, and Z. Jin. Integrating tree path in transformer for code representation. In Advances of 35th Conference on Neural Information Processing Systems (NeurIPS), 2021.
[26] P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations, In In Proceed- ings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018
[27] E. Shi, Y. Wang, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun. Cast: Enhancing code summariza- tion with hicrarchical splitting and reconstruction of abstract syntax trees. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
[28] Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. Advances of the Conference on Neural Information Processing Systems (NeurIPS), 2014.
[29] K. Tai, Socher, and D. Manning. Improved semantic representations from tree-structured long Short-temn memory networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2015.
[30] A. W. Uddin, C. Saikat, R. Baishakhi, and C. Kai-Wei, A transformer-baset approach for source code summarization. In Proceedings the 58th Annual Meeting of the Association for Computational Linguis- tics (ACL), 2020.
[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, and ukasz. Kaiser, Attention is all you need. In Advances of the 31st Conference on Neural Information Processing Systems (NeurlPS), 2017.
[32] B. Wei, G. Li, X. Xia. Z. Fu., and Z. Jin. Code generation dual task of code summarization. In Advances of 33th Conference on Neural Information Processing Systems (NeurlPS), 2019.
[33] H.-H. Wei and M. Li. Supervised dcep features for software functional clone detection by exploiting lexi- cal and syntactical information in source code. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (ICCAI), 2017.
[34] M. White, M. Tufano, C. Vendome, and D. Poshyvanyk. Deep learning code fragments for code clone detection, Proceedings of the 31st IEFE/ACM International Conference on Automated Software Engi- neering (ASE), 2016.
[35] H. Wu, H. Zhao, and M. Zhang. Code summarization with structure-induced transformer. In Findings of the Association for Computational Linguistics: ACL-IJCNLP, 2021.
[36] Y. Yuan, R. Fu, L. Huang, W. Lin, C. Zhang, X. Chen, and J. Wang. Hrformer: High-resolution trans- former for dense prediction. In Advances of 35th Conference on Neural Information Processing Systems (NeurlPS), 2021.
[37] J. Zhang, X. Wang, H. Zhang, H. Sun, K. Wang, and X. Liu. A novel neural source code representation based on abstract syntax tree. In Proceedings of the IEEF/ACM International Conference on Software Engineering (ICSE), 2019.
[38] D. Zügner. T. Kirschstein, M. Catasta, J. Leskovec, and S. Günnemann. Language-agnostic representation learning of source code from structure and context. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (cither the supplemental material or as a URL)? [Yes] See direc- tory named pa_former and we also provide training logs (rmse_ setrsed_deeomm.me .log and funcom. log) for our results
the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] In Table 11 at Appendix B
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifi- able information or offensive content? [No]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "uWGQ7nA49A0", "reviewer": "Reviewer_26RW", "summary": "This paper proposes a multi-granularity method for the code summarization task. Specially, one code snippet is first hierarchically broken into three granularities: statements, tokens, and sub-tokens levels. Then a new mechanism named pyramid attention mechanism is proposed for the aggregation of multi-granularity features. The authors also conducted extensive experiments to demonstrate the performance of their model and the effectiveness of the model components.\n", "questions": "1. Are the improvements achieved by the proposed method over existing work significant?\n\n2. How did you calculate the BLEU scores of your approach and the baselines in Tables 2-6? \n", "limitations": "No, the authors did not adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper targets automatic generation of natural language descriptions for source code, which could be useful in a variety of source code related tasks.  The paper illustrates and explains the idea well. The paper is generally well organized and clearly describes the architecture of the model. The proposed multi-granularity method for code representation is novel to me.\n\nThe improvement of the proposed method over related work is rather small. Although the paper says that “the proposed methods obtain 5% improvement over the strong NeuralCodeSum on a middle-size dataset and achieve new state-of-the-art results”, actually the results achieved by the proposed method are very similar to the results of CAST and TPTrans, as shown in Table 2. The differences are around or below 1.0 in almost all metrics. It is not clear if those improvements are indeed significant.\n\nAlso, there is lack of human evaluation to show the effectiveness of the proposed method in practice. Automatic evaluation (using BLEU metric, etc) may not be sufficient. Some papers (e.g. [a] and [b] below) on code summarization show that automatic metrics mainly calculate the textual similarity between the reference and the generated summaries, rather than the semantic similarity. Thus, human evaluation is needed to confirm the effectiveness of a code summarization method in practice.\n\n[a] Li, J., Li, Y., Li, G., Hu, X., Xia, X., & Jin, Z., EDITSUM : A Retrieve-and-Edit Framework for Source Code Summarization.  Proc. ASE2021.\n\n[b]J. Zhang, X. Wang, H. Zhang, H. Sun, and X. Liu, “Retrieval-based Neural Source Code Summarization.” Proc. ICSE2020.\n\nFurthermore, the below reference [c] shows that the BLEU metric, which is widely used in existing code summarization work, actually has many variants. Ignoring the differences among these variants could greatly affect the validity of the claimed results. Please check the BLEU scores in Tables 2-6 and make sure that they are calculated in the same way.\n\n[c] Shi, E., et al., On the Evaluation of Neural Code Summarization, in Proc.  ICSE2022.\n\nMinor:\n\n. The reference [26] is wrong. CAST is published at EMNLP2021 instead of ICPC2021.\n\n. Line 259. Bleu -> BLEU. Please keep the term consistent in the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tsVBH_DYdHe", "reviewer": "Reviewer_Q2Zz", "summary": "The proposed idea presented in this work is based on the observation that skilled programmers write and read source codes in a hierarchical way and pay close attention to the conceptual entities like statements, tokens, sub-tokens, and the mapping relations between them. Driven by this observation, the authors argue that a multi-granularities formulation incorporating the entities in different granularities may benefit the code summarization task. To this end, the paper proposes constructing a pyramid-shaped input representation based on a pyramid attention mechanism for efficient feature aggregation and distribution across different hierarchies. The proposed framework is evaluated on two source code summarization benchmarks where it surpasses the prior works and reaches new state-of-the-art results.", "questions": "- Are the experiments done with multiple seeds? Are the improvements achieved by PA-former is significant? I just want to make sure that the improvements are not due to some randomness in the experiments.\n\n- Can we apply the proposed framework for other languages, e.g., python, go, PHP? Performing experiments on the CodeXGlue summarization benchmark may add value to the work.\n\n- Is the proposed idea scalable? Can we apply the idea to fine-tune a large language model?\n\n- Figure 1 shows that PA-former generates a better summary by producing additional useful information. But a comparison with ground truth, especially based on BLEU score-like metric, should not be able to demonstrate the efficacy of the proposed approach. What do the authors think about it?\n\n- The paper shows improvements on the target task. But does the paper show whether the proposed model truly learns a pyramid-shaped input representation? Is there any way the authors can validate the claim?\n\n", "limitations": "My concern is around the authors' observation that skilled programmers write and read source codes in a hierarchical way and pay close attention to the conceptual entities like statements, tokens, sub-tokens, and the mapping relations between them. I am skeptical about accepting this argument. As Figure 1 shows, the PA-former generates a summary with more information, but the reference (human-written summary) does not have the extra information \"if the string can be parsed as an integer\". Therefore, I felt the observation of the authors is not well supported. Performing a human study to demonstrate that the proposed model-generated summaries are of higher quality would be a plus for this work.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\nGenerally, the paper did a good job of presenting the motivation, describing the proposed method, and demonstrating the effectiveness of the proposed approach based on experiment and evaluation.\n\n**Weaknesses**\n\n- The paper argues the existing structure-aware code summarization methods are inconsistent with the multi-grained nature of source code. Then the paper linked human behavior to the multi-granularity formulation of source code which I am skeptical to accept. Consequently, it is not clear why the multi-granularity nature of source code would encourage programmers to write summaries accordingly. The paper should verify the main claim of the work.\n- Evaluation is only in one language.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gbHanZ0AOSn", "reviewer": "Reviewer_Hb7s", "summary": "The paper introduces pyramid attention for source code summarization and\nintegrates it into a transformer architecture (they call the resulting\ntransformer PA-former). The pyramid attention combines three levels of looking\nat the source code (statement, token, and subtoken level). The levels are\ncombined bottom up from subtoken to statement level, and also with a\ncross-granularity interaction across levels where each level can query every\nother level (including its own).\n", "questions": "Table 5 shows that using just the fine-grained representation for decoding\nslightly outperforms using all three representations. I find this\ncounterintuitive. Could you explain this finding?\n\nIn Table 6, the difference between PA-former and w/o grammar is very low. Could\nit mean that the input design (e.g., the token grammar types) is less important,\nand the good results are obtained because of the pyramid attention and the\nstructure the input provides for it ($M_{G \\rightarrow T}$ and $M_{L \\rightarrow\nG}$?\n", "limitations": "I cannot think of any limitations which are not addressed.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nI think that the presented method is elegant, logical, and not overcomplicated.\nIt also improves state-of-the-art results.\n\n## Weaknesses\n\nI think the writing could be improved. Overall the paper is written well and is\neasy to follow, but there are numerous typos and errors. I found Section 3.1 a\nlittle bit hard to understand at first reading, especially the \"Semi-grained\nsequence\" part, the writing could be much improved there. Correcting the typos\nand the plurals would help, and maybe a figure about the AST like Figure 5 in\nthe Appendix and concrete examples if there is space for that.\n\nThere is also a typo in Eq. (2), the disjunction should go over k instead of j.\n\nSome examples of typos and errors:\n- line 19: as => because of\n- line 124 and 125:  order=> ordered, sequenc => sequence, denotes => denote\n- lines 131 and 132: corresponds => corresponding, reveals logic side of program => reveals the logic side of the program.A\n- line 172: lenght => length\n- line 290: We => we\n- line 304: The => the\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ct-rTZ6Suw4", "reviewer": "Reviewer_Xtyo", "summary": "The paper presents the method to tackle the source code summarization task by leveraging multi-scale features that appear in it. Such a human-like multi-scale comprehension is modeled relatively easily using the Transformer architecture enhanced with the information at each granularity level and cross-granularity interactions.", "questions": "1. How much longer are inputs being passed to the multi-head cross-attention?  \n1a. How much did the computational cost increase due to this? \n2. I wish the paper provided the results with error bars. Is it possible to add this? \n3. How was the segmentation on each level performed? Were there any additional tools used for such a segmentation? Is the description of \"Data preprocessing\" section enough?", "limitations": "Authors do not provide limitation section nor analyze the results qualitatively. I believe one important factor that would be fair to disclose is the computational expense of the model, relative to the Vanilla Transformer baseline.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Pros*:\n1. I like how the paper is structured and how illustrations benefit the narrative. A reader gets a nice overview before digging into details (Section 3.1 provides details on something that was previously elegantly explained on a high level, making it easy to process)\n2. Design choices are motivated well (e.g., use of the vocabulary, graining of the sequence, cross-granular interactions). Important ablations are performed; in general, paper naturally answers most questions that a reader wants to ask.\n3. The proposed method is elegant, does not overcomplicate the process, and seems easily reproducible. Moreover, the results suggest it provides stable gains on both used datasets.\n\n*Cons*:\n1. Since page 6, the quality of grammar and writing radically decreases, suggesting it needs proofreading, as, e.g., \"All model is trained using...\" is one among other trivial mistakes.\n2. A bit of obscured equations (4)-(9) suddenly makes processing this part much harder than the rest of the paper. Interestingly, this gracious note made me understand the steps easily: \"we just need concat the three sequences and the relative position maps, then directly adopt a relative position encoded multi-head self-attention layer to implement the cross-granularity interaction module.\" I recommend reworking the \"Cross-granularity interaction\" section a bit and, again, allowing someone to proofread that.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept