Integrating Tree Path in Transformer for Code Representation
Authors: Han Peng
Ge Li
Wenhan Wang
Yunfei Zhao
Zhi Jin
Venue: NeurIPS 2021 Poster (2021)
Keywords: machine learning for code
code summarization

Abstract: Learning distributed representation of source code requires modelling its syntax and semantics. Recent state-of-the-art models leverage highly structured source code representations, such as the syntax trees and paths therein. In this paper, we investigate two representative path encoding methods shown in previous research work and integrate them into the attention module of Transformer. We draw inspiration from the ideas of positional encoding and modify them to incorporate these path encoding. Specifically, we encode both the pairwise path between tokens of source code and the path from the leaf node to the tree root for each token in the syntax tree. We explore the interaction between these two kinds of paths by integrating them into the unified Transformer framework. The detailed empirical study for path encoding methods also leads to our novel state-of-the-art representation model TPTrans, which finally outperforms strong baselines. Extensive experiments and ablation studies on code summarization across four different languages demonstrate the effectiveness of our approaches. We release our code at \url{https://github.com/AwdHanPeng/TPTrans}.

Integrating Tree Path in Transformer for Code Representation
Han Peng', Ge Li Wenhan Wang', Yunfei Zhao', Zhi Jin
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education; Institute of Software, EECS, Peking University, Beijing, China (phan, lige, wwhjacob, zhaoyunfei, zhijin) @pku.edu.cn
Abstract
Learning distributed representation of source code requires modelling its syntax and semantics. Recent state-of-the- models leverage highly structured source code representations, such as the syntax trees and paths therein. In this paper, we investigate two representative path encoding methods shown in previous research work and integrate them into the attention module of Transformer. We draw inspiration from the ideas of positional encoding and modify them to incorporate these path encoding. Specifically, we encode both the pairwise path between tokens of source code and the path from the leaf node the tree root for each token in the syntax tree. We explore the interaction between these two kinds of paths by integrating them into the unified Transformer framework. The detailed empirical study for path encoding methods also leads our novel state-of ofiiee-f-the- art representation model TPTrans, which finally outperforms strong baselines. Extensive experiments and ablation studies code summarization across four different languages demonstrate the effectiveness of our approaches. We release code at ptttps://githune wwwam//vvHaaannn///rraaaa TPTraas
Introduction
Machine learning for source code aims at building models that learn semantic embedding of programs. The initial representation of source code relied on sequential models adopted from natural language processing, such as n- n-gram language model [Hindle et al., 2016 Hellendoorn and Devanbu, 2017). Recurrent Neural Networks (RNNs) (Wei et al. 2019) and on. However, source code more logical than natural languages, rich structured information such as Abstract Syntax Tree (AST). Therefore, these previous works struggle to capture the structural complexity of source code.
Some works leverage the program AST model source code structure and linearize the graph by traversing it. Code2sey [Alon et al.) 2018) represents source code as a set of pairwise paths over AST where each path is compressed to a vector using LSTMs [Hochreiter and Schmidhuber 1997). Code2seq obtains state-of-the-an for code representation using only AST information, demonstrating the effectiveness of path encoding. [Kim al.) 2020) leveraged another kind of path from the leaf node to the root of AST by traversing up its ancestors and then coupled the representation of paths with the source code.
On the other hand, several works also leverage structured graph neural networks for modelling source code directly. [Allaaanis al. 2017 used Gated Graph Neural Network (GGNN) embed the semantically meaningful relationships in the code. [Zhou et al., 2019) proposed a novel graph neural network with composite programming representation for vulnerability identification. However, GNNs
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Funcilef
def Fantoria)(nnm)): res <1
Parameters
Block
fictoral
Eepritmt
for in range(1, num 1):
result result * return result
Anlgament
for
Call
Block
relurn
reault
range
Argeliit
Exprštaıt
BinaryOp
Assigmment
Relative
Path:
Relative
Path:
mum
BinaryOp
Absolute
Path:
Absolute
Path:
result
(a)
(b)
Figure 1: Example (a) python code snippet and (b) the syntax tree of it. The relative path across the syntax tree between tokens for and result represents the relationship between them, showing the pattern of for loop in the code snippet. Mcanwhile, the absolute path for token result starts at the leaf node and ends at the tree root, which reveals the behaviour of assignment for this token.
typically rely on synchronous message passing, requiring message passing iterations to aggregate information Alllamani et al. 2017 Fernandes et al. 2018). Besides, primarily for computational reasons, GNNs for programs usually compute few pesssage-passing iterations. For these reasons, the representation of GNNs tends to be local and struggle to leverage long-range interactions.
On the contrary, Transformer [Vaswani et al. 2017] allows global information flow at each step but lacks the ability to model the structural complexity of source code. Recently, some works have explored introducing structural inductive bias Transformer access the global structural representation of source codes. [Hellendoorn et al., 2019) proposed Graph Relation Embedding Attention Transformer (GREAT), which extends (Shaw et al. 2018) and biases Transformer with relational information from graph edge types. After that. [Zügner et al. [0221] proposed Code Transformer based on XLNet (Yang et al. [219], which computes pairwise distances on AST and integrate multiple relations into the attention module.
In this paper, we pursue the research line of combining Transformer with additional structure information of source code. Our first starting point is the struetural model of Code2seq [Alon et al.) 2018), which obtains state-of-the-an in AST. However, Code2seq lacks the modelling of context, which leads us explore combining path representation with source code context. The other starting point is Code Transformer [uügner] al. 2021]. which counts node distances cross AST to capture source code structure. However, the different nodes combinations of paths contain plenty of structure information, which is overlooked by only encoding distances.
code summarization using only pairwise path information
To overcome the drawback shown in previous works, we adopt the ideas of encoding path to represent source code and integrate them to bias the attention module of Transformer. Specifically, we encode the path between source code tokens across AST and the path from leaf node to tree root for each token. Then we draw inspiration from both relative and absolute positional encoding methods in NLP field and modify them integrate path encodings into Transformer. These paths introduce inductive bias into the attention module of Transformer, powering it to know the structure of source codes. For clarity, we name the first path as the relative path and the last one as the absolute path (see I.. since much similarities between path and position relative or absolute encoding exist.
Intuitively, the relative path represents the structural relationship between tokens and shows specific patterns of code block such as For loop, etc. Meanwhile, the absolute path complements the structural information for each token and reveals the program behaviour on it, such as À inggent and Call, etc. Both two kinds of paths contain plenty of structural information of code syntax. Actually, lots of works have introduced these paths into many novel models [Kim et al. 2020 Alon al. 2018 2020). However, to our best knowledge, the feature relationship between these two paths for code representation learning still not completely studied. In this paper, we integrate these paths into the unified Transformer framework to analyze each effectiveness and their relationship. The detailed

---

empirical study also leads to our novel code representation model TPTrans, which means encoding the Tree Path into the Transformer. We show the effectiveness of our approaches on the code summarization task across four different languages datasets, in which the model predicts a method's name based on body. Our model finally outperforms solid baselines and obtains state-of-the-art on most datasets. The contributions of this paper are summarized as follows:
We propose TPTRANS which integrates path encoding in Transformer, powers it to know the structure of source code, and our model significantly outperforms existing basclines.
2.
We present the empirical study to investigate the relationship between two kinds of path encoding methods proposed in previous works, shedding light on this line of future work.
2 Related work
Representation learning for source code The growing availability of open-source code creates opportunities for machine learning from big code. Early research works learned language model on raw text data for code [Dam al. 2016 Wang al. 20166] Allamanis et al. 2016 Iyer et al. 2016). providing evidence for the naturalness assumption [Hindle et al. 2016]. These models treat codes as sequences rather than trees and struggle to model the structural complexity of source code. After
that, some works leveraged structural information to model source code. [Mou et al. 2016) proposed tree-based convolutional neural networks represent source code. [Alon al. 2018 2019 treated source code as weighted combination of pairwise paths in abstract syntax trees. Several works also use structured graphical models to represent structural aspects of source code directly. For example, [Allamanis et al. 2017] proposed GGNN to represent program graphs consisting of AST together with control flow and data-flow edges. After that, (Hellendoorn et al., 2019 Zügner et al.) 202T) proposed to bias the self-attention computation of Transformer given the underlying graph structure. In this paper, we explore integrating path representation of syntax tree into Transformer, powering to capture the structure of code snippets.
Structural representation of languages has been proved that modelling the language structure helps to represent the compositional aspects of natural language (Socher et al.) 2011] and improves the generalization of models [Chen et al., 2016). Several works [Shen et al.) 2018 Shiv and Quirk] 2019] proposed new architectures for modeling tree structures, including in the domain of natural languages processing [Wang et al., 2019] and code representation learning [Hellendoorn et al. 20199] Zügnnrr et al. [202T] We pursue this research line and show the benefits of integrating tree structure as inductive bias.
Positional encoding for Transformer The absolute positional encoding used in Transformer has been proved inefficient to model the order information of sequence. (Shaw et al. 2018 firstly
proposed leverage relative positional encoding Transformer. After that, Transformer-Xi [Dai
et al. 2019] re-parameterized the self attention of Transformer to integrate relative positional encoding
and T5 Raffel al., 2019] simplified the vector representation of relative positions in (Shaw et al., 2018) to scalars. Recently, several works also tried to enhance positional encodings in Transformer to integrate structural inductive bias. For example, (Shiv and Quirk 2019) extended the sequential positional encoding to tree-based positional encoding in Transformer, and [Zügner et al. 2021 modified the relative positional encoding of Transformer-Xl. to incorporate multiple relations in tree structural. In this paper, we adopt the idea to modify positional encodings of Transformer and introduce structural bias of path information in the syntax tree for self-attention computing.
Approaches
Self-attention is one of the key components of Transformer. The attention module be formulated as querying a dictionary with key-value pairs. The matrix form equation for attention with a single head is
QKT
Attention (Q, K, V) = softmax
J Vd
(1)
where the dimension of the hidden representations, and Q(Query), K(Key), V(Value) are specified as the hidden representations of the previous layer. For simplicity, we omit the index of
3

---

layer, denote z = (#1, 22. ... zn) and 2 = (21, 22 ... (n)) as the input and output of self attentonn module in the same layer respectively, where n is the sequence length. Then Eq can be rewritten as
exp (ay) 2 exp (asjr)
(I, W. where oij Va (1, (r;
(2)
where W Rdxd, W Rdxdd WV E Rdxdx and we set de d, d. Eq 2 is oblivious to structured input since it views imput as unordered vector sets. In the NLP field, several works (Shaw al. 2018 Ke al. 2020 Raffel al. 2019, Dai al. 2019 propose different ways to bias Transformer towards sequential inputs by incorporating positional information into the self-attention module. Generally, there are two categories of positional encoding methods for self-attention: absolute positional encoding and relative positional encoding.
The original Transformer [Vaswani et al. 2017] absolute positional encoding to represent positions. Given a sentence, a real-valued vector E R will be added to the word embedding wi at position i, and the sum vector will be used the input to Transformer. (Ke et al. 2020 further proposed TUPE to untie the correlations between absolute positions and words and directly model the relationship between a pair of words or positions using different projection matrices, that
03
1
(x, (I, + V2d (p,
(3)
/2d
where U2, U.K Bddd are projection matrices for positional embedding and scaling term v2d is used retain the magnitude of azy.
Using different helps Transformer distinguish the words at different positions i. However, the absolute positional encoding is not effective in capturing the relative word orders. Therefore, (Shaw et al. 2018 proposed the relative positional encoding, that is
exp (atss) Ef== =1 exp (asp) (V + aay), where ouj = 1 (r, + tz vd
4
(4)
where b a Rd are learnable parameters and can be viewed the relative position embedding between position and j. Eggeextend self-attention to consider the pairwise positional relationship and models it as a directed edge between head and tail word entities [Shaw al. 2018), which means the tail entity should be close to the embedding of the head entity plus vector depending on the relationship [Bordes al. [013].
We draw inspiration from the ideas of encoding absolute and relative positional information and modify 3nd to incorporate path information in the syntax tree of source code.
3.1 Integrating path encodings in attention module
syntax tree uniquely represents a source code snippet in the given language and grammar. In a syntax tree, the leaves are called terminals, and the non-leaf nodes are called nonterminals. [Alon] jet al. 2018 2019] used AST to represent a source code snippet. They considered pairwise paths between terminals, representing them sequences of the terminal and nonterminal nodes in AST. [Kim et al. 2020) used another kind of path from the terminal to AST root by traversing up its ancestors, and then coupled paths representation with tokens and fed the sum representation into Transformer.
We adopt the above methods of leveraging paths and integrate them into the self-attention module of the unified Transformer framework. We name the path between terminals as the relative path and the path from terminal root as the absolute path. Technically, we modify the positional encoding methods in the NLP field to integrate path encoding into Transformer, which shows natural analogies from the positional to path encoding ways as follows.
All kinds of paths are composed of nonterminals from a limited vocabulary of syntax grammar. We use learned embedding matrix to convert nonterminals to vectors first.

---

Relative path encoding The embed vectors of a relative path are first fed into sequence encoder. We use a bi-directional GRU [Cho al. 2014) to encode the path and use the final state represent it, that is
Ti GRU,
(5)
where Pathg,-2, is the nonterminal vector sequence of the relative path between tokens 24 and zj. and vig E Rd the final vector representation of this path. Specifically, Pat is a vector list [no. ni...... .. where m the path length and each the embedding vector of each nonterminal node which is looked up from the embedding matrix of node type.
The pairwise path across the syntax tree reveals the structural relationship between two code tokens. Similarly, [og44models the positional relationship between two words, so it is natural to modify the Eq to integrate path encoding rij into self-attention module. We firstly integrate Ftj into the Query-Key product, that is
1 0i Val (1, W + +ryg )..
(6)
and then we integrate Ftj into the weighted sum of Value, that is
exp (aig) exp (arrr) (R,WY + tr
24
where W W.Y E Rdxd are projection matrices of Key and Value for riy. respectively. For clarity, we label the modifications for Key and Value as and 2, respectively.
Absolute path encoding The embed vectors of an absolute path are fed into another bi-directional GRU Cho et al. 2014], that is
a₂ = (Ptthe
(8)
where Pathz is the nonterminals vectors sequence of the absolute path from tokens z; to the syntax tree root, and a; Rdd is the final vector representation of this path.
The absolute path for each source code token can be viewed as a directed edge connecting the terminal with the tree root, which means the absolute path can calibrate the coordinates for each terminal in the special structural space that assigns the tree root as the origin. So is natural to modify the absolute positional encoding in Eq 3 to integrate path vectors into the self-attention module, that is
Oii (a, (x, + (a,
where Wº,, WK E &dd are projection matrices of Query and Key for absolute path representation, respectively
We think it is meaningful to investigate the feature interaction between absolute and relative paths and focus on both of them instead of overlooking either. So we further combine the relative and absolute path encoding shown in previous paragraphs and merge Eq Eg as
1 [x, W +ry + (W,
04j =
(10)
In the paragraphs above, we propose several modifications to integrate paths into self-attention. To sum up. we combine them and propose two versions of models. We firstly name the method of
only relative path encoding as TPTRANS (Fq Eq 7). Then, to analyze the relationship between the relative and absolute path encoding further, we name the method of combining both of them as TPTRANS-O [0] [7
We tried the method adding path representation with token directly like Kim et al. 2020 but improve- ment, which we think is similar to the conclusion untying correlations in TUPE, Besides, we tried to keep the scaling term / 2d in Eq but improvement.
5

---

Experiment setup
Code summarization is one of the most popular tasks in code representation learning. Given a function body, this task is to predict the function name. As observed by [Allamanis et al. 2016 Alon et al. 2019], code summarization is a good benchmark for code representation learning, as the method body typically forms complete logical units and the method name tends to be descriptive and precise. In this task, our approaches jointly learn code contexts and path representation in AST and predict the target method name sequence of sub-tokens. We measure precision, recall, and F1 score over target sequences in the case insensitive like [Alon et al. 2018 Zügner et al. [022]]
Datasets To show the effectiveness of our approaches across different source code languages. we experiment on four datasets introduced in the CodeSearchNet (CSN) Challenge (Husain et al. 2019): Python, Javascript, Go, and Ruby. The datasets from CodeSearchNet have been carefully deduplicated to avoid data leakage from training sets. See Table
I for a summary of the datasets.
Table I:
Dataset statistics
Samples per partition
Dataset
Train
Valid
Test
CSN-Python 412,178
23.107
22,176
CSN-Javascript
123,889
8,253
6,483
CSN-Ruby
48,791
2,209
2,279
CSN-Go 317,832
14,242
14,291
Preprocessing We produce the AST for each method using the open-source parser Tree-Sitter in which all the code tokens are natively mapped as terminals (see [. We adopt the way to split sub-tokens following [Alon et al., 2018, Züggner et al. 202T]. in which each code token is split into sub-tokens respective to code naming conventions, c.g., setComeettionsPerSerr split into [set, connections, per, server]. We limit the vocabulary of sub-tokens with at least 100 occurrences training sets, remove all punctuations and restrict the max code length 512. We remove all anonymous Javascript functions following (Zügner et al. 2021]. We assign different ids for nonterminals and do not split the literal of them. We set the max path length 32 covering almost all paths and make padding for short paths. For paths longer than 32, we sample nodes with equal intervals to maintain max length instead of truncating the last nodes since assume the nodes at both left and right ends of the path are equally important.
is
Parameter sharing In our implementation, the embedding matrix for nonterminals shared for both absolute and relative paths since both kinds of paths are made up of nonterminals within the same syntax tree. We do not share the path encoder GRU for two different kinds of paths. Since the number of nonterminal types is far less than the size of source code vocabulary, we set different embedding matrices for nonterminals and source tokens, respectively. For efficiency, we share in different layers and heads for the projection matrices W.K, W.Y in relative path encoding and Wº W in absolute path encoding. We also share the path representation across different layers and heads, which means pre-encode paths only once by GRU before feeding features into Transformer.
Complexity optimization The naive way for relative path encoding needs to encode all the pairwise paths for each code snippet and compose the relative path representation matrix R E Raxxxd where n is code length and d is the dimension of hidden representations. This approach costs (n²)) complexity since the number of pairwise paths encoded is n(n-1) To optimize it, we notice that much repetition exists in all relative paths of sample That means we only need encode all the unique pairwise paths for each sample. We record the mapping between token pairs and unique paths as matrix M e gnx=, After that, we compute the Query- product between tokens and unique paths, and gather the product scores using matrix M. After calculating attention distribution, we
"natps//gituu..com//ee
For example, if we parse a function that has three formal arguments to syntax tree, we could see that the pairwise relative path between any two arguments will be the same.

---

scatter attention probability to each unique path and weighted sum using matrix M again. This new approach avoids encoding On²)) paths and reduces into (1) where I is the number of unique paths and is far less than n². It also reduces matrix R from ((n"d) to (ld). In our experiment, we set the max 5122 for the relative path, which covers most of the samples. Besides, the complexity of
absolute path encoding is original (n), but we also reduce it in the same way and set the max 256 for it. We refer readers to the appendix for further details.
Normalization We integrate vector sequence of path nodes as the inductive bias for the attention module of Transformer, which is not just trivial technically. Firstly, directly integrating standard RNN into the deep backbone of Transformer most likely results in gradient exploding or vanishing. Secondly, the attention procedure of Transformer is similarity calculation between vectors. The Q. K, and V of the attention module are normalized while standard RNN does not limit vector norm, leading to size instability for dot-product attention between word and path representation. To overcome these issues, we first replace the standard GRU as a aayer-normalized GRU [Ba et al. (2066] getting more stable parameters updates. After getting the final state of GRU, we feed it into another normalization layer to keep it the same vector norm with Q, K, and V. The normalization is much efficient shown in the following ablation result.
Hyperparameters We denote the number of layers for encoder and decoder as Lg and LD. the hidden size as D, the feed-forward dimension as DFF: F. and the number of heads as H. We primarily report results on two model size for both TPTrans and TPTrans-a: base (Ln=1, D 20044) and large (L==3, Dr 40966). For base and large settings, we all set Lg=3, D=1024, and H=8. We add a pointer network [Vinyals et al. 2015) to the decoder as same as our baselines. The base model setting keeps the same size as other baselines for fair comparisons. The embedding dimension of the word and path node are 512 and 64, and we apply linear layer to project word embedding to the hidden size of Transformer. We use one layer Bi-GRU, set the hidden size to 64, and concat the final states of forward and backward as output with the size of the single head dimension. We set the batch size and dropout rate to 128 and 0.2 and employ label smoothing of 0.1. All models and baselines are trained from random initial parameters. As the optimizer, we use Adam Kingma and Ba 2014] with a learning rate and weight decay of le In our experiment, we use Tesla V100 GPUs for training.
Baselines Combining two views of context and structure for source code has been widely discussed in Code Intelligence. The context mainly refers to representing code as a sequence of text, and the structure refers to extracting knowledge from code syntax. Here we mainly compare with Code2seq [Alon et al., 2018), Graph Relational Embedding Attention Transformer (GREAT) [Hellendoorn et al. 2019]. XLNet (Yang et al., 2019) and Code Transformer (Zügner et al. 2021]_ Unlike the Transformer- based baselines, Code2seq uses only pairwise path information in AST to represent code snippets, which means it mainly focuses on code structure within the syntax tree and lacks context modelling. On the contrary, our proposed models and other baselines are designed based on Transformer naturally focused on context. Specifically, GREAT uses the relative positional encoding present in [Shaw et al., 2018) to bias the attention via manually designed edges, and Code Transformer a recent Transformer model based XLNet, incorporating multiple relations to learning both structure and context jointly. XLNet is designed initially for huge-corpus natural language pretraining, but its novel two-stream self-attention also shows powerful performance in code representation learning.
Table 2: Model comparison
Model
Context
Structure
Code2Seq
No
Relative Path Manually Designed Edges
Great
Yes
XLNet
Yes
No
Codefransformer
Yes
Multiply Structural Distances
TPTrans
Yes
Relative Path
TPTrans-a
Yes
Relative and Absolute Path
Essentially, the difference between our model and baselines GREAT and Code Transformer is encoding path vs encoding edges in GREAT and distances in Code Transformer. The different node combinations of paths naturally contain plenty of structure information, which overlooked by encoding distances. See Table 2 for clear comparisons between our models and baselines.
7

---

5 Results
5.1
Overall comparison
Table 3: Code summarization result on the CSN dataset
Python Rec.
Ruby Rec.
Javascript Rec.
Go
Model
Prec.
F1
Prec.
Prec.
Prec.
Rec.
F1
Code2seq
35.79
24.85
29.34
23.23
10.31
14.28
30.18
19.88
23.97
52.30
43.43
47,45
Great
35.09
31.62
33.26
24,66
22.25
23.39
31.25
26.87
28.89
50.02
46.52
48.21
XLNet
37.39
32.01
34.49
29. 88
25.89
27.74
33.33
26.86
29.75
$1.79
47.58
49.60
CodTTransformer
36.41
33.68
34,99
31.46
24.50
27.55
35.07
29.65
32.13
55.09
48.05
51.33
TPTiass
38.45
33,63
35.88
32.70
27.75
30,02
33.47
28.27
30.65
56,19
51.14
53.54
[Base]
#0.17
10.06
+0.05
#1.32
+0.38
1:0.72
1:0.62
#0.52
20.01
+0.21
+:0 .56
=0.22
TPTrans-ux
38.48
33,99
36.09
32.54
26.77
29.38
34,06
28.42
30,99
56.00
50,97
53.37
[Base]
+0.07
+-0.99
+-0.18
+-0.15
+0.50
+-0.35
+-0.21
++0.12
+-0.44
+-0.33
+-0.28
-0.22
TPTrans
38.76
34,66
36.59
32.40
27.63
29.82
33.83
28.37
30.86
$5.79
51.05
53.30
[Large]
+0.49
10.35
10.02
10.84
10.78
1:0.79
10.55
20.54
10.53
+1.30
+1.30
=0.83
TPTruns-o
38.39
34.70
36.45
33,07
28.34
30.52
33.68
28.95
31.14
55.67
51.31
53,39
[Large]
#0.18
+0.26
+0.22
#0.86
=0.61
1:053
10.31
=0.54
20.46
+0.92
40.76
=0.13
The overall comparison results are shown in Table 3 We show the performance of two different settings for both TPTrans and TPTrans-ox. The base setting keeps the same Transformer backbone baselines, and the large setting additionally widens the feea-forward
other Transformer-basee dimension and increases the number of decoder layers. We find that our proposed models TPTrans and TPTrans-a substantially outperform all baselines on all but one language, highlighting the effectiveness of learning path information. The only exception is Javascript, while our models still outperform most of the baselines this dataset. Besides, we find that the large setting further improves the performance on most datasets.
It is worth mentioning that Code Transformer is designed based on XLNet, which natively equips the relative positional encoding of context and is effective for modelling long sequences. this work, we mainly focus on studying the effectiveness of encoding path, so we use the original Transformer as our backbone. Even so, we still obtain advanced results on almost all datasets. We also find that compared to the improvement of Code Transformer over XLNet, models get more remarkable rise over Code Transformer.
To investigate the exception of Javascript dataset, we focus on TPTrans and count the mean and variance of all relative paths length across four datasets. We also count the total numbers of unique relative paths in training datasets and present them in Table 4 We firstly find that the path length of Javascript is not only longer but also has a significantly larger variance than other datasets. We then find that the unique paths num of Javascript is much large and does not match its dataset size well, which likely indicates much noise exists in the training dataset. So we assume that due to such property of Javascript, it is harder to extract useful structural information from path than others. Accordingly, our models achieve more significant performance on Ruby dataset, and the Ruby dataset has smaller mean length and variance than others and also keeps an ideal num of unique paths. As for the outlier of Javascript, one possible reasoning line is the inherent property of Javascript language results in complicated syntax. We also suspect that it is due to the internal design of language parsers, and perhaps more suitable parser might alleviate this problem.
Table 4: Path statistics
Train
Valid
Test
Train
Dataset
Mean
Var.
Mean
Var.
Mean
Var.
Path nums
Data size
Ruby
9.29
17.62
9.76
18.84
9.60
18.82
10,97M
48,791
Go
11.18
20.16
11.56
22.40
10.68
18.83
45.79M
317,832
Python
11.96
24.61
11.81
23.64
12.05
25.36
81.42M
412,178
Javascript
14.59
41.46
14.40
40.02
14.57
40.59
52.60M
81,487
Lastly, we find that the overall performance of TPTrans and TPTrans-a does not differ so much, although the latter seems to obtain additional information about the absolute path. We investigate the effectiveness of both paths and analyze the relationship between them further in the following.
For Javascript, we present the sample num of oon-anonymuus functions.
8

---

5.2
Empirical study
In this work, we propose two versions of models, namely TPTaans and TPTrans-o. The first one encodes the pairwise relative paths in the syntax tree, while the last one additionally encodes the absolute path. Here we explore the feature interaction of relative and absolute path encodings and mainly experiment on the large model setting.
Table 5: Empirical study for path relationship
Python Rec.
Ruby Rec.
Javascript Rec.
Go
Model
Prec.
Prec.
Prec.
Rec.
IPTrans
38.76
34.66
36.59
32.40
27.63
29.82
33.83
28.37
30.86
55.79
51.05
53.30
10.49
1.0.35
100.02
10.84
-0.78
1:0.79
10.55
1.0.54
10053
1:1.30
1:130
100.3
V
37.00
25.87
27.76
31,93
27.31
+0.38
+0.77
+0.46
+0.64
=0.90
+0.76
+0071
+0.77
+0.66
+1.12
+0.68
+0.34
w/o K, V
34,30
28.99
31.42
24.78
20.32
22.33
27.94
23.08
25.28
51.47
47.71
49.52
+0.35
+0.44
+20.1
+0051
+0.48
+0037
++0.4
+0038
+:0.44
+:0.22
+0.10
TPTrans-o
38.39
34,70
36.45
33.077
28.34
30.52
33.68
28.95
31.14
55.67
51.31
53.39
+0.18
+0.26
+0.22
+0.86
=0.61
+:053
=0034
+00.54
=0.46
+:0.92
+0.76
=0.13
win Rel V
37.28
33.63
35.36
30.44
25.27
27.58
32.93
28.29
30.43
54.89
51.35
53.06
+0.35
+0.44
2:0.44
+1.47
=1.11
+0.16
+0.33
+0.54
-0.43
+:0.39
+0.17
+00.27
w/o V
36.35
31,2
33.70
30.05
22.68
25.83
31.17
26.
28.41
50.58
52.27
10.34
1.062
20.31
10.84
1.04
10.52
20.85
10.06
0.558
1:0.99
10.72
20.61
Relative path encoding For TPTrans 66 [. We first remove the relative path encoding in
the weighted sum of Value (2 in 7 and then further remove the relative path in Query-Key product in 66) showing in the top half of Table 5 We find that with the elimination of relative path encoding gradually, the performance of TPTrans decreases obviously. After removing all submodules of relative path encoding in 6 and in Eq 77). TPTrans totally degenerates to vanilla Transformer and performs the worst. These results highlight the usefulness of relative path encoding, both in Query-Key product (6 and the weighted sum Value (Eq
Absolute path encoding TPTrans-a extra adds absolute path encoding on Query-Key product of attention score (Eq[TO) compared TPTrans. We remove all the relative path encoding of TPTrans-a and compare it to vanilla Transformer shown in the last rows of both top and bottom halves of Table 5 respectively. We find that absolute path encoding significantly improves vanilla Transformer, highlighting its effectiveness.
Relationship between them We analyze the interaction between these two kinds of paths further. In the bottom half of Table 3 we firstly find that with the elimination of two submodules of relative path encoding gradually (1 [q[ and 2 in (). the performance of TPTrans-a decreases at the same time. This phenomenon Is consistent with the ablation study for relative path encoding shown in the previous paragraph. We then compare the top and bottom halves of Table row by row from down to up and find that with the introduction of relative path gradually, the performance gap between TPTrans and TPTrans-a correspondingly decreases, which means the improvement brought by absolute path becomes smaller at the same time. After introducing full of the relative path encoding, the gain from absolute path encoding is almost weak. These results show that these two kinds of paths are not orthogonal and much feature overlap indeed exists between them. Actually, if two sub-tokens are too far away from each other in the syntax tree, the relative path between them then spans the root of AST, in which the relative and absolute path encodings play the same role for modelling structure. The experimental results also show that is preferable to encode the relative path than the absolute path.
5.3 Ablation study
We explore the roles of each part of our approaches. We choose TPTrans with the base setting and mainly experiment on Python and Ruby datasets.
We also tried another method adding absolute path representation to word embedding directly in Kim et al. 2020 as mentioned before, but the overall conclusion remains all the same.

---

Path vs. distance To verify the benefit of learning from paths than distances presented in [Zügner al. 202T]. we try convert all nodes paths the same one before feeding paths into TPTras and present in Table After such special preprocessing. what TPTaas learns completely degenerates into hops across the syntax tree between two terminals. We find that the performance of our proposed model shows noticeable declines after information ablation. It indicates TPTaas indeed learns from rich node combinations within paths, and it is better to model path than distance.
Table 6: Ablation study on path information
Python Rec.
Ruby Rec.
Model
Prec.
Fl
Prec.
F1
TPTrans
33.63-0.06 38.455:0.17
35.88
32.70=1.32
27.75
30.02:10.72
degenerate to hops
37.31
32.76=:0224
34.89:0.09
29.68-0.00
24.85
27.05:+0.10
Normalization
mentioned before, embedding vector sequences into the attention module of Transformer is not just trivial, so we carefully design the normalization method and present the ablation results in Table[7 These results prove that the normalization method is crucial for embedding vector sequence into attention, which is also helpful to guide the design of neural networks.
Table 7: Ablation study on normalization
Python Rec.
Ruby Rec.
Model
Prec.
F1
Prec.
F1
TPTrans
38.45:10.17
33.63:10.06
35.88:10.05
32.70=1.33
27.75
30.02:10.7 72
w/o normalization
37.15+0.44
30.33+0..3
34.74+0.22
26.81 - 0055
21.81 +0.11+
24.05+0.19
Deeper model architecture We re-design a deeper model architecture for TPTrans and vanilla Transformer and set Lg=Ld==,, D=512, D and H=8. For TPTrans, the input and output sizes of GRU are both 64. We present the results in Table 8 and conclude that our approaches of integrating path encodings still work for the deep model setting.
Table 8: Ablation study on deeper model
Python Rec.
Ruby Rec.
Model
Prec.
Fl
Prec.
Fl
TPTrans
37.27--0.36
33.28-L0.23 27.63++0.10
35.22+10.28
31.10-0.411 24.26++0.33
26.87+00.7
28.82-+0.30
Transformer
32.71++0.27
29.96++0.16
19.66+0.115 21.71++0.25
Conclusion
In this paper, we investigate the interaction between the absolute and relative path encodings by integrating them into the unified Transformer framework, and confirm that feature overlap exists between these two kinds of paths. The detailed empirical study for path encodings also leads to our novel code representation model TPTrans. Extensive experiments and ablation study demonstrate the effectiveness of our approaches on code summarization across four different programming languages.
7
Acknowledgements
We thank all reviewers for their constructive comments. This research is supported by the National Key R&D Program of China under Grant No. 2020AAA0109400,, and the National Natural Science Foundation of China under Grant Nos. 62072007, 61 61832009, 61620106007.
References
Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. On the naturalness of software. Communications of the ACM, 122-113111331 2016.
10

---

Vincent J Hellendoorn and Premkumar Devanbu. Are deep neural networks the best choice for modeling source code? Proceedings of the 2017 1 11th Joint Meeting on Foundations of Software Engineering, pages 763-773, 2017.
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. Code generation dual task of code summariza- tion. arXiv preprint arXiv:1910.0500 2019.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv: 01400, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees transformers. arXiv preprint arXiv:2003. /3844, 2020.
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. arXiv preprint arXiv:1711.00744 2017.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerabil- ity identification by learning comprehensive program semantics via graph neural networks. arXiv preprint arXiv: 0009,,3996 2019.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. Structured neural summarization. arXiv preprini DeIXiv:/811.001 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint
2017.
Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In International conference on learning representations, 2019. 2019
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.0211 2018
Daniel Zilgner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. Language-agnostee representation learning of source code from structure and context. arXiv preprint arXiv:2103.11318, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.000 09037 2019.
Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In International Conference on Machine Learning, pages 245-256. PMLR, 2020.
Hoa Khanh Dam, Truyen Tran, and Trang Pham. A deep language model for software code. arXiv preprint 1001: 2016.
Song Wang, Devin Chollak, Dana Movshovitz-Atiia, and Lin Tan. Bugram: bug detection with n-gram language models. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, pages 708-719, 2016.
Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for extreme summarization of source code. In International conference on machine learning, pages 2091-2100. PMLR, 2016.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2073-2083, 2016.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for programming language processing. Thirtieth AAAI Conference on Artificial Intelligence, 2016.
11

---

Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed rep- resentations of code. Proceedings of the ACM on Programming Languages, 3(POPL): 2019.
Richard Socher, Cliff Chiung-Wi Yu Lin, Andrew Y Ng. and Christopher D Manning. Parsing natural scenes and natural language with recursive neural networks. In ICML, 2011.
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced Istm for natural language inference. arXiv preprint arXiv: /w0.:vv:9..0000000 06038, 2016.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating 2018.
tree structures into recurrent neural networks. arXiv preprint
Vighnesh Shiv and Chris Quirk. Novel positional encodings to enable tree- based transformers. Advances in Neural Information Processing Systems, 32: 2081-12001, 2019.
Yau-Shian Wang, Hung-Yi Yi Lee, and Yun-Nung Chen. Tree transformer: Integrating tree structures into self-attention. arXiv preprint
2019.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02880 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Guolin Ke, Di He, and Tie- Yan Liu. Rethinking the positional encoding in language pre-training. arXiv preprint arXiv:2006. / 55595, 2020.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoderr approaches. arXiv preprint arXiv: /409. /259, 2014 4.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint 1rX1v:190..09436.
2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06-55 2016.
arXiv preprint
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks.
arXiv preprint
150.0:/506.03334, 2015.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv: 1422,660, 2014.
arXiv preprint
12


Reviews:
[{"review_id": "j5M8N0V5KoA", "reviewer": "Reviewer_86qv", "summary": "The paper proposes an Transformer-based architecture for embedding source code snippets that integrates encoding of\nabsolute and relative AST paths for all code tokens into self-attention. Following up on prior work in relative position\nencoding, they map AST paths into the position encoding framework, modify the self-attention computation accordingly,\nand evaluate on method summarization. The approach is compared to recent Transformer-based baselines on the standard\nmethod summarization benchmark, as well as to code2seq, which pioneered path-only source code encoding. The results\nimprove performance significantly on almost all benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Discussion update\n\nThe authors have conducted numerous experiments to clarify my (and other reviewers') concerns, answer our questions, and compile a clearer picture of the paper's findings. As of now, it presents a key simple idea – an AST-dependent way of incorporating learned relative positions into code Transformers – and evaluates it on multiple domains and dimensions against SOTA single-model baselines. The paper has extensive empirical backing even for a simple idea. In fact, the idea's simplicity works in its favor here.\n\nMany of our questions and new experiments would be appreciated in the new version of the paper, including:\n- \"Deep and narrow\" Transformer baseline\n- Discussion of model sizes\n- Discussion of the distribution of unique paths (and JavaScript outliers)\n- Code classification task\n- More appropriate discussion/speculation on the relationship between absolute and relative positions' performance\n- Various perspectives on prior work discussed during the rebuttal phase, which make the paper clearer.\n\nAfter the discussion, I would love to see this work in NeurIPS, and increasing my score accordingly.\n\n---\n\nTPTrans extends the ideas of code2seq and CodeTransformer to encode concrete paths over the AST as a substitute for\n\"positions\" in the Relational Transformer framework. It is a neat and simple idea, made possible by the unification of\nmulti-lingual parsing by `tree-sitter`, and experiments show that it improves code representations further. The\nintegration of prior ideas into this framework is relatively straightforward but I believe it's a reasonable and useful\ncontribution to the field.\n\nThe authors experiment with both \"relative-like\" and \"absolute-like\" AST-derived positions, and find that absolute\npositions don't add any value over the relative ones, and in fact hurt performance. I appreciate the inclusion of the\nnegative result, it is useful and informative in context. Interestingly, this contrasts with natural language, where\nstate-of-the-art language models still use absolute position embeddings, likely due to computational challenges involved\nwith storing $n \\times n$ embeddings for position pairs. For code, the number of practical different paths is lower,\nwhich allows the sparse optimization the authors introduce in Section 4. The implementation of this sparse relative path\nencoding is another nice contribution of this paper.\n\n**Q1:** Can you plot the actual number of different paths for different programming languages and their distribution\n(perhaps faceted by program size)? A canonical vocabulary of \"most useful code positions\" would help follow-up work.\n\n**Q2:** Intuitively, mere information redundancy (Table 3) would not _lower_ performance, especially given that\nTPTrans-$\\alpha$ has two more parameter matrices. An alternative explanation might be that additional parameters are\nspent _only_ on absolute position projection, which might over-emphasize importance of absolute positions and overfit\nthe learned code representation to them.\n\nThe evaluation follows the same standard as prior work on the same topic, with proper ablation experiments and\nstate-of-the-art baselines. Qualitative evaluation is a bit lacking. Please include an example that demonstrates the\nvalue of joint absolute/relative encoding -- e.g., something not handled by either code2seq or CodeTransformer. Some\nother qualitative evaluation of representations (e.g., clustering) would also help the reader.\n\n\n## Evaluation questions\n\n**Q3:** What is the standard error in Table 2? The prior work does not include them either, thus a significant\nimprovement on this dataset is hard to estimate. That said, the delta improvement of TPTrans seems similar to such of\nstate-of-the-art works to their predecessors.\n\n**Q4:** Addition of domain-specific inductive bias (like source code paths) usually compensates for model capacity. What\nare the sizes of all evaluated models in Table 2? Furthermore, does TPTrans show an even more noticeable improvement\nupon e.g. CodeTransformer when evaluated on equal _small_ model configurations?\n\n**Q5:** I could not understand L266-268. Does it describe TPTrans (as evidently), or does it state that your\nre-implementation of CodeTransformer is different from Zügner et al.? If it's the latter, the comparison in Table 2 is\nnot strictly equivalent.\n\n\n## Minor\n\nProofreading:\n- L269-271: \"realize\" and \"plus\" seem out of place / ungrammatical.\n- Please sort the references.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nSome limitations are present in the paper as negative results or performance discussions. No other dedicated sections\ndiscuss societal impact. However, it does not significantly differ in potential impact from prior similar work.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No issues or concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "WtwdbyrFx6x", "reviewer": "Reviewer_5kuu", "summary": "The paper proposes encoding the relative positions of the source code tokens in the Concrete Syntax Tree (CST) and their absolute positions into the self-attention of a transformer and show the value of learned representations in the downstream task of code summarization on four different languages and compare with related baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think that the idea of using relative path encoding in the transformer self-attention proposed in the paper is very similar to the one in Zugner et. al 2021, except that instead of AST they use CST. The incorporation of absolute path encoding ( TPTRANS-$\\alpha$ model) may be a bit different but the authors show in experiments that the absolute path encodings don't make much difference and in the end, they use the relative path encoding model only.  Though I agree that this difference may result in differences in performance (as shown in results), I personally think that just this change in itself does not suffice to contribute to enough novelty of the work that can lead to acceptance at a conference like NeurIPS. On the contrary, I feel that the approach used in Zugner et. al 2021 that uses both structure and context is more meaningful.  The source code context treated as a text sequence in addition to the structure provided by the tree representations can be quite helpful especially for code segments containing comments. Besides this main point, I have the following comments:\n\n(i) The contributions stated are overly broad and grossly misleading. The first contribution *We show the feasibility of embedding sequential information as an inductive bias for the attention module of Transformers and are the first ones to do so to our best knowledge.* (Line 77-78) is not at all a contribution. As stated by the authors themselves there are several previous works like GREAT, Zugner et. al 2021, etc. that have already done this. \n\n(ii) What exactly are the x's and z's in the context of the CST? \n\n(iii) Please include error bars for performances.\n\n(iv) I think it should be W^Q in Eqn (10) instead of W^(V)?\n\n(v) I didn't exactly understand the reasoning for parameter sharing as stated in Lines 228-1229.\n\n(v) Why does the incorporation of absolute path encoding doesn't help? It seems like the authors spend a lot of time justifying their choice and in the end decided by using only the relative path encoding model. \n\n(vi) The statement made in lines 54-55 is very vague. Please expand on it for better motivation.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I didn't see any limitations or potential negative impact discussed in the paper. Some suggestions are:\n\n(i) Mention the computational complexity of the model in terms of getting the CST and carrying the whole inference process, especially for large code fragments.\n\n(ii) The approach is language-dependent as the parser that creates the CST has to be dependent on the programming language being considered.\n\n(iii) The splitting of tokens into subtokens relies on using variable naming conventions that may not always be followed by developers making the splitting difficult.\n\n(iv) Blind trust in the code summarization systems based on this approach can be dangerous and hence should be exercised with caution along with a human developer.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "FItNYB4zqi8", "reviewer": "Reviewer_y3EX", "summary": "Integrating the rich structural information available in source code is a key research topic. This work proposes to embed paths in syntax trees into attention-based models, demonstrating that this excells over both simpler path-based models and various Transformers for source code.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Update, following discussion\nThe authors provided thorough responses and relevant new results during the response phase. Therefore, I am updating my assessment to a (slightly) more positive rating. Please incorporate the many helpful comments you received. In particular, the authors are strongly encouraged to include a further analysis of \"deep and narrow\" models (ideally, also larger ones), since, to the best of my knowledge, the \"squat and wide\" (3 layer, 1,024 setup) was only used by Zügner et al. and may not be representative of commonly used Transformer architectures. Enriching the table in the below response with the other models and datasets would be a good start, as would including a bigger model, such as the 10 layer variant used by Hellendoorn et al.\n\n### Original Review\nThe main contribution of this work is to merge the intuition from Alon et al.'s AST-paths and work on biasing Transformer attention towards code's structural features. This is a worthwhile contribution, and results suggest that it translates into performance improvements.\n\nThe primary concern is a fairly straightforward flaw in the comparison with other models: TPTrans adds a non-trivial number of parameters through its various RNNs, but apparently does not account for this discrepancy in model capacity in its comparisons. The results may well hold even after compensating for this imbalance, e.g. by employing more layers or larger hidden dimensions in the baseline models, but until this comparison is run it is hard to interpret any performance improvements.\n\nThis aside, I articulate a number of smaller concerns below. Primarily, the writing is quite poor and occasionally glosses over details that seem important. Please significantly improve the writing of the paper throughout.\n\nL203: code summarization by specifically generating a function's name has been considered a dubious task for some time now, primarily due to both high ambiguity and repetitiveness in function names in practice. Consequently, he paper would be strengthened by including other tasks, such as defect detection.\n\nP6: the \"Efficiency\" section lacks a lot of details, some of which seem rather important (e.g. L242), but are relegated to the appendix. I would encourage trying to include more details directly here (see note under presentation on freeing up space), ideally at least Figure 2 or something equivalent. In the absence of such room, and really regardless, please ensure that the parts explained here can stand alone; especially the phrase \"Finally, we reduce the costs to be linear.\" needs to be expanded.\n\nThe explanation of the JavaScript results relative to other languages feels a bit handwavy. This is not a major concern, but illustrating this concretely, for instance with examples of outliers or representative samples, would help.\n\nI did not find any record of the number of layers used for the models. Please add this to p7.\n\nThe explanation of the \"absolute\" path performance, being worse than the relative one only, feels off-base. If the information were redundant, we should not expect a drop in performance by including it. L319 further mentions a potential \"conflict\" between these sources of information, which, as far as I can tell, is not further substantiated. This all deserves further exploration.\n\nTable 5 seems to be missing the ablation for other models. Or did those never use the sentinel pointer setup?\n\n\n### Presentation\nThe text on the first four pages or so is a bit redundant, repeatedly discussing the related literature and the relation of TPTrans across the introduction (rendering it over 2 pages long), section 2, and section 3.1. Please concentrate this and remove duplicate explanations, which will also free up space to discuss some of the important details that are currently relegated to the appendix.\n\nThroughout, avoid using references as nouns. E.g., instead of \"[Alon et al., 2018] proposed\", use \"Alon et al. [2018] proposed\". In Latex, use the \\citet command instead of \\citep for these cases.\n\nThe names \"relative\" and \"absolute\" are odd names for the these two types of paths. Consider revising, to something more direct, like \"leaf-to-leaf\" and  \"leaf-to-root\".\n\n### Typos:\nThe below is only a sample, please make a thorough pass to improve writing quality.\n- L9: \"attentions\" -> \"attention\"\n- L17: \"semantically embedding\" -> \"semantic embeddings\"\n- L26: \"over AST\" -> \"over an AST\"\n- L27: \"The code2seq\" -> remove \"the\"\n- L28: \"Besides\" -> here and throughout, prefer using another word (or simply drop this) when the purpose is to describe a series of options.\n- L53: \"counts the distances\" -> unclear\n- L73: \"on most of all\" -> \"on most\"\n- L111: \"works\" -> not the plural of \"work\"\n- L120: use tall parentheses for the RHS. In Latex, use \\left( and \\right) around the faction.\n- L126: $d_k$ is never used, presumably it was supposed to be for $W^K$? Given the next clause, that all d's are equal, this distinction seems unnecessary. Also, remove \"The\" before \"Eq.2\"\n- L269: \"the realize\" -> unclear (same L270)\n- L270: \"which plus\" -> unclear\n- L278-9: several typos, revise. \"much weak\", \"the encoding path\", \"count distance\" (unclear)\n- L315: \"full of the\" -> \"the full\" ?\n- L331: \"is degenerate\" -> \"degenerates\"\n- L336: \"these results also prove workable\" -> unclear\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not address either aspect of this work. The former is especially important.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "3N0YVhxpM_5", "reviewer": "Reviewer_ZRpb", "summary": "The authors present a technique for modeling source code. They point out that modeling code is different from language because of the tight binding between ASTs, data, and control flow. Also they mention some weakness of GNNs. To compete with GNNs, they introduce a new method which is a sequence-based transformer that incorporates graph structure through the attention weights. The model is shown to be effective for code summarization.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- The proposed technique is simple and effective.\n- The proposed technique outperforms other models in all categories besides Javascript.\n\nWeaknesses\n\n- It is very difficult to gauge which modeling decisions are introduced in this paper, and which are borrowed from other work. There is clearly a lot of overlap with Code Transforme, such as the use of the pointer network.\n- There are results in the Code Transformer paper that outperform results shown here. I realize that this is a very recent related work, and that it is perhaps with different training settings, but Code Transformer was mentioned many times and it seems important to address.\n- It is not especially clear why performance on Javascript is not as strong. I would recommend the authors double check this setting and see if there is some issue? It is not such a major issue if the model does not outperform in all settings, but the explanation that Javascript is somehow less structure dependent would likely need more justification.\n- The presentation of the analysis could be improved. Although the subsections and tables clearly indicate what was ablated, it is not clear to me what is learned from these experiments.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This was a controversial paper. On the one hand, the paper's central idea is straightforward and also closely related to multiple recent efforts. On the other hand, a simple idea can be a valuable contribution if established to be effective. While the original version of the paper left some doubts about the method's effectiveness,  the authors have engaged with the reviewers thoroughly during the discussion period and provided compelling additional results. The majority of the reviewers appreciated this, and based on their advice, I am recommending that the paper be accepted as a poster. Please make sure to address the suggestions made in the discussion thread.
Decision: Accept (Poster)