Integrating Tree Path in Transformer for Code Representation
Authors: Han Peng
Ge Li
Wenhan Wang
Yunfei Zhao
Zhi Jin
Venue: NeurIPS 2021 Poster (2021)
Keywords: machine learning for code
code summarization

Abstract: Learning distributed representation of source code requires modelling its syntax and semantics. Recent state-of-the-art models leverage highly structured source code representations, such as the syntax trees and paths therein. In this paper, we investigate two representative path encoding methods shown in previous research work and integrate them into the attention module of Transformer. We draw inspiration from the ideas of positional encoding and modify them to incorporate these path encoding. Specifically, we encode both the pairwise path between tokens of source code and the path from the leaf node to the tree root for each token in the syntax tree. We explore the interaction between these two kinds of paths by integrating them into the unified Transformer framework. The detailed empirical study for path encoding methods also leads to our novel state-of-the-art representation model TPTrans, which finally outperforms strong baselines. Extensive experiments and ablation studies on code summarization across four different languages demonstrate the effectiveness of our approaches. We release our code at \url{https://github.com/AwdHanPeng/TPTrans}.

Integrating Tree Path in Transformer for Code Representation
Han Peng', Ge Li Wenhan Wang', Yunfei Zhao', Zhi Jin
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education; Institute of Software, EECS, Peking University, Beijing, China (phan, lige, wwhjacob, zhaoyunfei, zhijin) @ @pku.edu.cn
Abstract
Learning distributed representation of source code requires modelling its syntax and semantics. Recent state-of-the- models leverage highly structured source code representations, such as the syntax trees and paths therein. In this paper, we investigate two representative path encoding methods shown in previous research work and integrate them into the attention module of Transformer. We draw inspiration from the ideas of positional encoding and modify them to incorporate these path encoding. Specifically, we encode both the pairwise path between tokens of source code and the path from the leaf node the tree root for each token in the syntax tree. We explore the interaction between these two kinds of paths by integrating them into the unified Transformer framework. The detailed empirical study for path encoding methods also leads our novel state-of ofiiee-f-thee art representation model TPTrans, which finally outperforms strong baselines. Extensive experiments and ablation studies code summarization across four different languages demonstrate the effectiveness of our approaches. We release code at ptttps://githune wwwam//viHaaannn///reaaaa TPTrans
Introduction
Machine learning for source code aims at building models that learn semantic embedding of programs. The initial representation of source code relied on sequential models adopted from natural language processing, such as n- n-gram language model [Hindle et al., 2016 Hellendoorn and Devanbu, 2017). Recurrent Neural Networks (RNNs) (Wei et al. 2019 and on. However, source code more logical than natural languages, rich structured information such as Abstract Syntax Tree (AST). Therefore, these previous works struggle to capture the structural complexity of source code.
Some works leverage the program AST model source code structure and linearize the graph by traversing it. Code2sey [Alon et al.) 2018) represents source code as a set of pairwise paths over AST where each path is compressed to a vector using LSTMs [Hochreiter and Schmidhuber 1997). Code2seq obtains state-of-the-an for code representation using only AST information, demonstrating the effectiveness of path encoding. [Kim al.) 2020) leveraged another kind of path from the leaf node to the root of AST by traversing up its ancestors and then coupled the representation of paths with the source code.
On the other hand, several works also leverage structured graph neural networks for modelling source code directly. [Allaaanis al. 2017 used Gated Graph Neural Network (GGNN) embed the semantically meaningful relationships in the code. [Zhou et al., 2019) proposed a novel graph neural network with composite programming representation for vulnerability identification. However, GNNs
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Funcilef
def Fantoria)(nnm)): res <1
Parameters
Block
fictoral
Eepritmt
for in range(1, num 1):
result result * return result
Anlgament
for
Call
Block
relurn
reault
range
Argelist
Exprštaıt
BinaryOp
Assigmment
Relative
Path:
Relative
Path:
mum
BinaryOp
Absolute
Path:
Absolute
Path:
result
(a)
(b)
Figure 1: Example (a) python code snippet and (b) the syntax tree of it. The relative path across the syntax tree between tokens for and result represents the relationship between them, showing the pattern of for loop in the code snippet. Mcanwhile, the absolute path for token result starts at the leaf node and ends at the tree root, which reveals the behaviour of assignment for this token.
typically rely on synchronous message passing, requiring message passing iterations to aggregate information Alllamani et al. 2017 Fernandes et al. 2018). Besides, primarily for computational reasons, GNNs for programs usually compute few pesssage-passing iterations. For these reasons, the representation of GNNs tends to be local and struggle to leverage long-range interactions.
On the contrary, Transformer [Vaswani et al. 2017] allows global information flow at each step but lacks the ability to model the structural complexity of source code. Recently, some works have explored introducing structural inductive bias Transformer access the global structural representation of source codes. [Hellendoorn et al., 2019) proposed Graph Relation Embedding Attention Transformer (GREAT), which extends (Shaw et al. 2018) and biases Transformer with relational information from graph edge types. After that. [Zügner et al. [0221] proposed Code Transformer based on XLNet (Yang et al. [219], which computes pairwise distances on AST and integrate multiple relations into the attention module.
In this paper, we pursue the research line of combining Transformer with additional structure information of source code. Our first starting point is the struetural model of Code2seq [Alon et al.) 2018), which obtains state-of-the-an in AST. However, Code2seq lacks the modelling of context, which leads us explore combining path representation with source code context. The other starting point is Code Transformer [uügner] al. 2021]. which counts node distances cross AST to capture source code structure. However, the different nodes combinations of paths contain plenty of structure information, which is overlooked by only encoding distances.
code summarization using only pairwise path information
To overcome the drawback shown in previous works, we adopt the ideas of encoding path to represent source code and integrate them to bias the attention module of Transformer. Specifically, we encode the path between source code tokens across AST and the path from leaf node to tree root for each token. Then we draw inspiration from both relative and absolute positional encoding methods in NLP field and modify them integrate path encodings into Transformer. These paths introduce inductive bias into the attention module of Transformer, powering it to know the structure of source codes. For clarity, we name the first path as the relative path and the last one as the absolute path (see I.. since much similarities between path and position relative or absolute encoding exist.
Intuitively, the relative path represents the structural relationship between tokens and shows specific patterns of code block such as For loop, etc. Meanwhile, the absolute path complements the structural information for each token and reveals the program behaviour on it, such as A inggent and Call, etc. Both two kinds of paths contain plenty of structural information of code syntax. Actually, lots of works have introduced these paths into many novel models [Kim et al. 2020 Alon al. 2018 2020). However, to our best knowledge, the feature relationship between these two paths for code representation learning still not completely studied. In this paper, we integrate these paths into the unified Transformer framework to analyze each effectiveness and their relationship. The detailed

---

empirical study also leads to our novel code representation model TPTrans, which means encoding the Tree Path into the Transformer. We show the effectiveness of our approaches on the code summarization task across four different languages datasets, in which the model predicts a method's name based on body. Our model finally outperforms solid baselines and obtains state-of-the-art on most datasets. The contributions of this paper are summarized as follows:
We propose TPTRANS which integrates path encoding in Transformer, powers it to know the structure of source code, and our model significantly outperforms existing basclines.
2.
We present the empirical study to investigate the relationship between two kinds of path encoding methods proposed in previous works, shedding light on this line of future work.
2 Related work
Representation learning for source code The growing availability of open-source code creates opportunities for machine learning from big code. Early research works learned language model on raw text data for code [Dam al. 2016 Wang al. 20166] Allamanis et al. 2016 Iyer et al. 2016). providing evidence for the naturalness assumption [Hindle et al. 2016]. These models treat codes as sequences rather than trees and struggle to model the structural complexity of source code. After
that, some works leveraged structural information to model source code. [Mou et al. 2016) proposed tree-based convolutional neural networks represent source code. [Alon al. 2018 2019 treated source code as weighted combination of pairwise paths in abstract syntax trees. Several works also use structured graphical models to represent structural aspects of source code directly. For example, [Allamanis et al. 2017] proposed GGNN to represent program graphs consisting of AST together with control flow and data-flow edges. After that, (Hellendoorn et al., 2019 Zügner et al.) 202T) proposed to bias the self-attention computation of Transformer given the underlying graph structure. In this paper, we explore integrating path representation of syntax tree into Transformer, powering to capture the structure of code snippets.
Structural representation of languages has been proved that modelling the language structure helps to represent the compositional aspects of natural language (Socher et al.) 2011] and improves the generalization of models [Chen et al., 2016). Several works [Shen et al.) 2018 Shiv and Quirk] 2019] proposed new architectures for modeling tree structures, including in the domain of natural languages processing [Wang et al., 2019] and code representation learning [Hellendoorn et al. 20199] Zügnnrr et al. [202T] We pursue this research line and show the benefits of integrating tree structure as inductive bias.
Positional encoding for Transformer The absolute positional encoding used in Transformer has been proved inefficient to model the order information of sequence. (Shaw et al. 2018 firstly proposed leverage relative positional encoding Transformer. After that, Transformer-Xi [Dai
et al. 2019] re-parameterized the self attention of Transformer to integrate relative positional encoding
and T5 Raffel al., 2019] simplified the vector representation of relative positions in (Shaw et al., 2018) to scalars. Recently, several works also tried to enhance positional encodings in Transformer to integrate structural inductive bias. For example, (Shiv and Quirk 2019) extended the sequential positional encoding to tree-based positional encoding in Transformer, and [Zügner et al. 2021 modified the relative positional encoding of Transformer-Xl. to incorporate multiple relations in tree structural. In this paper, we adopt the idea to modify positional encodings of Transformer and introduce structural bias of path information in the syntax tree for self-attention computing.
Approaches
Self-attention is one of the key components of Transformer. The attention module be formulated as querying a dictionary with key-value pairs. The matrix form equation for attention with a single head is
QKT
Attention (Q, K, V) = softmax
J Vd
(1)
where the dimension of the hidden representations, and Q(Query), K(Key), V(Value) are specified as the hidden representations of the previous layer. For simplicity, we omit the index of
3

---

layer, denote z = (#1, 22... ................................ zn) and 2 = (21, 22 ... (n)) as the input and output of self attentonn module in the same layer respectively, where n is the sequence length. Then Eq can be rewritten as
exp (ay) 2 exp (aajr)
(I, W. where oij Va (1, (r;
(2)
where W Rdxd, W Rdxdd WV E Rdxdx and we set de d, d. Eq 2 is oblivious to structured input since it views imput as unordered vector sets. In the NLP field, several works (Shaw al. 2018 Ke al. 2020 Raffel al. 2019, Dai al. 2019 propose different ways to bias Transformer towards sequential inputs by incorporating positional information into the self-attention module. Generally, there are two categories of positional encoding methods for self-attention: absolute positional encoding and relative positional encoding.
The original Transformer [Vaswani et al. 2017] absolute positional encoding to represent positions. Given a sentence, a real-valued vector E R will be added to the word embedding wi at position i, and the sum vector will be used the input to Transformer. (Ke et al. 2020 further proposed TUPE to untie the correlations between absolute positions and words and directly model the relationship between a pair of words or positions using different projection matrices, that
03
1
(x, (I, + V2d (p,
(3)
/2d
where U2, U.K Bddd are projection matrices for positional embedding and scaling term v2d is used retain the magnitude of azy.
Using different helps Transformer distinguish the words at different positions i. However, the absolute positional encoding is not effective in capturing the relative word orders. Therefore, (Shaw et al. 2018 proposed the relative positional encoding, that is
exp (atss) Ef== =1 exp (asp) (V + aay), where ouj = 1 (r, + tz vd
4
(4)
where b a Rd are learnable parameters and can be viewed the relative position embedding between position and j. Eggeextend self-attention to consider the pairwise positional relationship and models it as a directed edge between head and tail word entities [Shaw al. 2018), which means the tail entity should be close to the embedding of the head entity plus vector depending on the relationship [Bordes al. [013].
We draw inspiration from the ideas of encoding absolute and relative positional information and modify 3nd to incorporate path information in the syntax tree of source code.
3.1 Integrating path encodings in attention module
syntax tree uniquely represents a source code snippet in the given language and grammar. In a syntax tree, the leaves are called terminals, and the non-leaf nodes are called nonterminals. [Alon] jet al. 2018 2019] used AST to represent a source code snippet. They considered pairwise paths between terminals, representing them sequences of the terminal and nonterminal nodes in AST. [Kim et al. 2020) used another kind of path from the terminal to AST root by traversing up its ancestors, and then coupled paths representation with tokens and fed the sum representation into Transformer.
We adopt the above methods of leveraging paths and integrate them into the self-attention module of the unified Transformer framework. We name the path between terminals as the relative path and the path from terminal root as the absolute path. Technically, we modify the positional encoding methods in the NLP field to integrate path encoding into Transformer, which shows natural analogies from the positional to path encoding ways as follows.
All kinds of paths are composed of nonterminals from a limited vocabulary of syntax grammar. We use learned embedding matrix to convert nonterminals to vectors first.

---

Relative path encoding The embed vectors of a relative path are first fed into sequence encoder. We use a bi-directional GRU [Cho al. 2014) to encode the path and use the final state represent it, that is
Ti GRU,
(5)
where Pathg,-2, is the nonterminal vector sequence of the relative path between tokens 24 and zj. and Vig E Rd the final vector representation of this path. Specifically, Pat is a vector list [no. ni...... ..m] where m the path length and each the embedding vector of each nonterminal node which is looked up from the embedding matrix of node type.
The pairwise path across the syntax tree reveals the structural relationship between two code tokens. Similarly, [og44models the positional relationship between two words, so it is natural to modify the Eq to integrate path encoding rij into self-attention module. We firstly integrate Ftj into the Query-Key product, that is
1 0i Val (1, W + +ryg )..
(6)
and then we integrate Ftj into the weighted sum of Value, that is
exp (aig) exp (arrr) (R,WY + tr
24
where W W.Y E Rdxd are projection matrices of Key and Value for riy. respectively. For clarity, we label the modifications for Key and Value as and 2, respectively.
Absolute path encoding The embed vectors of an absolute path are fed into another bi-directional GRU Cho et al. 2014], that is
a₂ = (Ptthe
(8)
where Pathz is the nonterminals vectors sequence of the absolute path from tokens z; to the syntax tree root, and a; Rd is the final vector representation of this path.
The absolute path for each source code token can be viewed as a directed edge connecting the terminal with the tree root, which means the absolute path can calibrate the coordinates for each terminal in the special structural space that assigns the tree root as the origin. So is natural to modify the absolute positional encoding in Eq 3 to integrate path vectors into the self-attention module, that is
Oii (a, (x, + (a,
where Wº,, WK E &dd are projection matrices of Query and Key for absolute path representation, respectively
We think it is meaningful to investigate the feature interaction between absolute and relative paths and focus on both of them instead of overlooking either. So we further combine the relative and absolute path encoding shown in previous paragraphs and merge Eq Eq as
1 (x, W +ry + (W,
04j =
(10)
In the paragraphs above, we propose several modifications to integrate paths into self-attention. To sum up. we combine them and propose two versions of models. We firstly name the method of
only relative path encoding as TPTRANS (Fq Eq 7). Then, to analyze the relationship between the relative and absolute path encoding further, we name the method of combining both of them as TPTRANS-O [0] [7
We tried the method adding path representation with token directly like Kim et al. 2020 but improve- ment, which we think is similar to the conclusion untying correlations in TUPE, Besides, we tried to keep the scaling term / 2d in Eq but improvement.
5

---

Experiment setup
Code summarization is one of the most popular tasks in code representation learning. Given a function body, this task is to predict the function name. As observed by [Allamanis et al. 2016 Alon et al. 2019], code summarization is a good benchmark for code representation learning, as the method body typically forms complete logical units and the method name tends to be descriptive and precise. In this task, our approaches jointly learn code contexts and path representation in AST and predict the target method name sequence of sub-tokens. We measure precision, recall, and F1 score over target sequences in the case insensitive like [Alon et al. 2018 Zügner et al. [022]]
Datasets To show the effectiveness of our approaches across different source code languages. we experiment on four datasets introduced in the CodeSearchNet (CSN) Challenge (Husain et al. 2019): Python, Javascript, Go, and Ruby. The datasets from CodeSearchNet have been carefully deduplicated to avoid data leakage from training sets. See Table
I for a summary of the datasets.
Table I:
Dataset statistics
Samples per partition
Dataset
Train
Valid
Test
CSN-Python 412,178
23.107
22,176
CSN-Javascript
123,889
8,253
6,483
CSN-Ruby
48,791
2,209
2,279
CSN-Go 317,832
14,242
14,291
Preprocessing We produce the AST for each method using the open-source parser Tree-Sitter in which all the code tokens are natively mapped as terminals (see [. We adopt the way to split sub-tokens following [Alon et al., 2018, Züggner et al. 202T]. in which each code token is split into sub-tokens respective to code naming conventions, c.g., setComeettionsPerSerr split into [set, connections, per, server]. We limit the vocabulary of sub-tokens with at least 100 occurrences training sets, remove all punctuations and restrict the max code length 512. We remove all anonymous Javascript functions following (Zügner et al. 2021]. We assign different ids for nonterminals and do not split the literal of them. We set the max path length 32 covering almost all paths and make padding for short paths. For paths longer than 32, we sample nodes with equal intervals to maintain max length instead of truncating the last nodes since assume the nodes at both left and right ends of the path are equally important.
is
Parameter sharing In our implementation, the embedding matrix for nonterminals shared for both absolute and relative paths since both kinds of paths are made up of nonterminals within the same syntax tree. We do not share the path encoder GRU for two different kinds of paths. Since the number of nonterminal types is far less than the size of source code vocabulary, we set different embedding matrices for nonterminals and source tokens, respectively. For efficiency, we share in different layers and heads for the projection matrices W.K, W.Y in relative path encoding and Wº W in absolute path encoding. We also share the path representation across different layers and heads, which means pre-encode paths only once by GRU before feeding features into Transformer.
Complexity optimization The naive way for relative path encoding needs to encode all the pairwise paths for each code snippet and compose the relative path representation matrix R E Raxxxd where n is code length and d is the dimension of hidden representations. This approach costs (n²)) complexity since the number of pairwise paths encoded is n(n-1) To optimize it, we notice that much repetition exists in all relative paths of sample That means we only need encode all the unique pairwise paths for each sample. We record the mapping between token pairs and unique paths as matrix M e gnx=, After that, we compute the Query- product between tokens and unique paths, and gather the product scores using matrix M. After calculating attention distribution, we
"natps//gituu..com//ee
For example, if we parse a function that has three formal arguments to syntax tree, we could see that the pairwise relative path between any two arguments will be the same.

---

scatter attention probability to each unique path and weighted sum using matrix M again. This new approach avoids encoding On²)) paths and reduces into (1) where I is the number of unique paths and is far less than n². It also reduces matrix R from ((n"d) to (ld). In our experiment, we set the max 5122 for the relative path, which covers most of the samples. Besides, the complexity of
absolute path encoding is original (n), but we also reduce it in the same way and set the max 256 for it. We refer readers to the appendix for further details.
Normalization We integrate vector sequence of path nodes as the inductive bias for the attention module of Transformer, which is not just trivial technically. Firstly, directly integrating standard RNN into the deep backbone of Transformer most likely results in gradient exploding or vanishing. Secondly, the attention procedure of Transformer is similarity calculation between vectors. The Q. K, and V of the attention module are normalized while standard RNN does not limit vector norm, leading to size instability for dot-product attention between word and path representation. To overcome these issues, we first replace the standard GRU as a aayer-normalized GRU [Ba et al. (2066] getting more stable parameters updates. After getting the final state of GRU, we feed it into another normalization layer to keep it the same vector norm with Q, K, and V. The normalization is much efficient shown in the following ablation result.
Hyperparameters We denote the number of layers for encoder and decoder as Lg and LD. the hidden size as D, the feed-forward dimension as DFF: F. and the number of heads as H. We primarily report results on two model size for both TPTrans and TPTrans-a: base (Ln=1, D 20044) and large (L==3, Dr 40966). For base and large settings, we all set Lg=3, D=1024, and H=8. We add a pointer network [Vinyals et al. 2015) to the decoder as same as our baselines. The base model setting keeps the same size as other baselines for fair comparisons. The embedding dimension of the word and path node are 512 and 64, and we apply linear layer to project word embedding to the hidden size of Transformer. We use one layer Bi-GRU, set the hidden size to 64, and concat the final states of forward and backward as output with the size of the single head dimension. We set the batch size and dropout rate to 128 and 0.2 and employ label smoothing of 0.1. All models and baselines are trained from random initial parameters. As the optimizer, we use Adam Kingma and Ba 2014] with a learning rate and weight decay of le In our experiment, we use Tesla V100 GPUs for training.
Baselines Combining two views of context and structure for source code has been widely discussed in Code Intelligence. The context mainly refers to representing code as a sequence of text, and the structure refers to extracting knowledge from code syntax. Here we mainly compare with Code2seq [Alon et al., 2018), Graph Relational Embedding Attention Transformer (GREAT) [Hellendoorn et al. 2019]. XLNet (Yang et al., 2019) and Code Transformer (Zügner et al. 2021]_ Unlike the Transformer- based baselines, Code2seq uses only pairwise path information in AST to represent code snippets, which means it mainly focuses on code structure within the syntax tree and lacks context modelling. On the contrary, our proposed models and other baselines are designed based on Transformer naturally focused on context. Specifically, GREAT uses the relative positional encoding present in [Shaw et al., 2018) to bias the attention via manually designed edges, and Code Transformer a recent Transformer model based XLNet, incorporating multiple relations to learning both structure and context jointly. XLNet is designed initially for huge-corpus natural language pretraining, but its novel two-stream self-attention also shows powerful performance in code representation learning.
Table 2: Model comparison
Model
Context
Structure
Code2Seq
No
Relative Path Manually Designed Edges
Great
Yes
XLNet
Yes
No
Codefransformer
Yes
Multiply Structural Distances
TPTrans
Yes
Relative Path
TPTrans-a
Yes
Relative and Absolute Path
Essentially, the difference between our model and baselines GREAT and Code Transformer is encoding path vs encoding edges in GREAT and distances in Code Transformer. The different node combinations of paths naturally contain plenty of structure information, which overlooked by encoding distances. See Table 2 for clear comparisons between our models and baselines.
7

---

5 Results
5.1
Overall comparison
Table 3: Code summarization result on the CSN dataset
Python Rec.
Ruby Rec.
Javascript Rec.
Go
Model
Prec.
F1
Prec.
Prec.
Prec.
Rec.
F1
Code2seq
35.79
24.85
29.34
23.23
10.31
14.28
30.18
19.88
23.97
52.30
43.43
47,45
Great
35.09
31.62
33.26
24,66
22.25
23.39
31.25
26.87
28.89
50.02
46.52
48.21
XLNet
37.39
32.01
34.49
29. 88
25.89
27.74
33.33
26.86
29.75
$1.79
47.58
49.60
CodTTransformer
36.41
33.68
34,99
31.46
24.50
27.55
35.07
29.65
32.13
55.09
48.05
51.33
TPTiass
38.45
33,63
35.88
32.70
27.75
30,02
33.47
28.27
30.65
56,19
51.14
53.54
[Base]
#0.17
10.06
+0.05
#1.32
+0.38
1:0.72
1:0.62
#0.52
20.01
+0.21
+:0 .56
=0.22
TPTrans-ux
38.48
33,99
36.09
32.54
26.77
29.38
34,06
28.42
30,99
56.00
50,97
53.37
[Base]
+0.07
+-0.99
+-0.18
+-0.15
+0.55
+-0.35
+-0.21
++0.12
+-0.44
+-0.33
+-0.28
-0.22
TPTrans
38.76
34,66
36.59
32.40
27.63
29.82
33.83
28.37
30.86
$5.79
51.05
53.30
[Large]
+0.49
10.35
10.02
10.84
10.78
1:0.79
10.55
20.54
10.53
+1.30
+1.30
=0.83
TPTruns-o
38.39
34.70
36.45
33,07
28.34
30.52
33.68
28.95
31.14
55.67
51.31
53,39
[Large]
#0.18
+0.26
+0.22
#0.86
=0.61
1:053
10.31
=0.54
20.46
+0.92
40.76
=0.13
The overall comparison results are shown in Table 3 We show the performance of two different settings for both TPTrans and TPTrans-ox. The base setting keeps the same Transformer backbone baselines, and the large setting additionally widens the feea-forward
other Transformer-basee dimension and increases the number of decoder layers. We find that our proposed models TPTrans and TPTrans-a substantially outperform all baselines on all but one language, highlighting the effectiveness of learning path information. The only exception is Javascript, while our models still outperform most of the baselines this dataset. Besides, we find that the large setting further improves the performance on most datasets.
It is worth mentioning that Code Transformer is designed based on XLNet, which natively equips the relative positional encoding of context and is effective for modelling long sequences. this work, we mainly focus on studying the effectiveness of encoding path, so we use the original Transformer as our backbone. Even so, we still obtain advanced results on almost all datasets. We also find that compared to the improvement of Code Transformer over XLNet, models get more remarkable rise over Code Transformer.
To investigate the exception of Javascript dataset, we focus on TPTrans and count the mean and variance of all relative paths length across four datasets. We also count the total numbers of unique relative paths in training datasets and present them in Table 4 We firstly find that the path length of Javascript is not only longer but also has a significantly larger variance than other datasets. We then find that the unique paths num of Javascript is much large and does not match its dataset size well, which likely indicates much noise exists in the training dataset. So we assume that due to such property of Javascript, it is harder to extract useful structural information from path than others. Accordingly, our models achieve more significant performance on Ruby dataset, and the Ruby dataset has smaller mean length and variance than others and also keeps an ideal num of unique paths. As for the outlier of Javascript, one possible reasoning line is the inherent property of Javascript language results in complicated syntax. We also suspect that it is due to the internal design of language parsers, and perhaps more suitable parser might alleviate this problem.
Table 4: Path statistics
Train
Valid
Test
Train
Dataset
Mean
Var.
Mean
Var.
Mean
Var.
Path nums
Data size
Ruby
9.29
17.62
9.76
18.84
9.60
18.82
10,97M
48,791
Go
11.18
20.16
11.56
22.40
10.68
18.83
45.79M
317,832
Python
11.96
24.61
11.81
23.64
12.05
25.36
81.42M
412,178
Javascript
14.59
41.46
14.40
40.02
14.57
40.59
52.60M
81,487
Lastly, we find that the overall performance of TPTrans and TPTrans-a does not differ so much, although the latter seems to obtain additional information about the absolute path. We investigate the effectiveness of both paths and analyze the relationship between them further in the following.
For Javascript, we present the sample num of ooo-anonymuus functions.
8

---

5.2
Empirical study
In this work, we propose two versions of models, namely TPTaans and TPTrans-o. The first one encodes the pairwise relative paths in the syntax tree, while the last one additionally encodes the absolute path. Here we explore the feature interaction of relative and absolute path encodings and mainly experiment on the large model setting.
Table 5: Empirical study for path relationship
Python Rec.
Ruby Rec.
Javascript Rec.
Go
Model
Prec.
Prec.
Prec.
Rec.
IPTrans
38.76
34.66
36.59
32.40
27.63
29.82
33.83
28.37
30.86
55.79
51.05
53.30
10.49
1.0.35
100.02
10.84
-0.78
1:0.79
10.55
1.0.54
1.053
1:1.30
1:130
100.3
V
37.00
25.87
27.76
31,93
27.31
+0.38
+0.77
+0.46
+0.64
=0.90
+0.76
+0071
+0.77
+0.66
+1.12
+0.68
+0.34
w/o K, V
34,30
28.99
31.42
24.78
20.32
22.33
27.94
23.08
25.28
51.47
47.71
49.52
+0.35
+0.44
+20.1
+0051
+0.48
+0037
++0.4
+0038
+:0.44
+:0.22
+0.10
TPTrans-o
38.39
34,70
36.45
33.077
28.34
30.52
33.68
28.95
31.14
55.67
51.31
53.39
+0.18
+0.26
+0.22
+0.86
=0.61
+:053
=0034
+00.54
=0.46
+:0.92
+0.76
=0.13
win Rel V
37.28
33.63
35.36
30.44
25.27
27.58
32.93
28.29
30.43
54.89
51.35
53.06
+0.35
+0.44
2:0.44
+1.47
=1.11
+0.16
+0.33
+0.54
-0.43
+:0.39
+0.17
+00.27
w/o V
36.35
31,2
33.70
30.05
22.68
25.83
31.17
26.11
28.41
50.58
52.27
10.34
1.062
20.31
10.84
1.04
10.52
20.85
10.06
0.558
1:0.99
10.72
20.61
Relative path encoding For TPTrans 66 [. We first remove the relative path encoding in
the weighted sum of Value (2 in 7 and then further remove the relative path in Query-Key product in 66 showing in the top half of Table 5 We find that with the elimination of relative path encoding gradually, the performance of TPTrans decreases obviously. After removing all submodules of relative path encoding in 6 and in Eq 77). TPTrans totally degenerates to vanilla Transformer and performs the worst. These results highlight the usefulness of relative path encoding, both in Query-Key product (6 and the weighted sum Value (Eq
Absolute path encoding TPTrans-a extra adds absolute path encoding on Query-Key product of attention score (Eq[TO) compared TPTrans. We remove all the relative path encoding of TPTrans-a and compare it to vanilla Transformer shown in the last rows of both top and bottom halves of Table 5 respectively. We find that absolute path encoding significantly improves vanilla Transformer, highlighting its effectiveness.
Relationship between them We analyze the interaction between these two kinds of paths further. In the bottom half of Table 3 we firstly find that with the elimination of two submodules of relative path encoding gradually (1 [q[ and 2 in (). the performance of TPTrans-a decreases at the same time. This phenomenon Is consistent with the ablation study for relative path encoding shown in the previous paragraph. We then compare the top and bottom halves of Table row by row from down to up and find that with the introduction of relative path gradually, the performance gap between TPTrans and TPTrans-a correspondingly decreases, which means the improvement brought by absolute path becomes smaller at the same time. After introducing full of the relative path encoding, the gain from absolute path encoding is almost weak. These results show that these two kinds of paths are not orthogonal and much feature overlap indeed exists between them. Actually, if two sub-tokens are too far away from each other in the syntax tree, the relative path between them then spans the root of AST, in which the relative and absolute path encodings play the same role for modelling structure. The experimental results also show that is preferable to encode the relative path than the absolute path.
5.3 Ablation study
We explore the roles of each part of our approaches. We choose TPTrans with the base setting and mainly experiment on Python and Ruby datasets.
We also tried another method adding absolute path representation to word embedding directly in Kim et al. 2020 as mentioned before, but the overall conclusion remains all the same.

---

Path vs. distance To verify the benefit of learning from paths than distances presented in [Zügner al. 202T]. we try convert all nodes paths the same one before feeding paths into TPTras and present in Table After such special preprocessing. what TPTaas learns completely degenerates into hops across the syntax tree between two terminals. We find that the performance of our proposed model shows noticeable declines after information ablation. It indicates TPTaas indeed learns from rich node combinations within paths, and it is better to model path than distance.
Table 6: Ablation study on path information
Python Rec.
Ruby Rec.
Model
Prec.
Fl
Prec.
F1
TPTrans
33.63-0.06 38.455:0.17
35.88
32.70=1.32
27.75
30.02:10.72
degenerate to hops
37.31
32.76=:0224
34.89:0.09
29.68-0.00
24.85
27.05:+0.10
Normalization
mentioned before, embedding vector sequences into the attention module of Transformer is not just trivial, so we carefully design the normalization method and present the ablation results in Table[7 These results prove that the normalization method is crucial for embedding vector sequence into attention, which is also helpful to guide the design of neural networks.
Table 7: Ablation study on normalization
Python Rec.
Ruby Rec.
Model
Prec.
F1
Prec.
F1
TPTrans
38.45:10.17
33.63:10.06
35.88:10.05
32.70=1.33
27.75
30.02:10.7 72
w/o normalization
37.15+0.44
30.33+0..3
34.74+0.22
26.81 - 0055
21.81 +0.11+0.31
24.05+0.19
Deeper model architecture We re-design a deeper model architecture for TPTrans and vanilla Transformer and set Lg=Ld==,, D=512, D and H=8. For TPTrans, the input and output sizes of GRU are both 64. We present the results in Table 8 and conclude that our approaches of integrating path encodings still work for the deep model setting.
Table 8: Ablation study on deeper model
Python Rec.
Ruby Rec.
Model
Prec.
Fl
Prec.
Fl
TPTrans
37.27--0.36
33.28-L0.23 27.63++0.10
35.22+10.28
31.10-0.411 24.26++0.33
26.87+00.7
28.82-+0.30
Transformer
32.71++0.27
29.96++0.16
19.66+0.115 21.71++0.25
Conclusion
In this paper, we investigate the interaction between the absolute and relative path encodings by integrating them into the unified Transformer framework, and confirm that feature overlap exists between these two kinds of paths. The detailed empirical study for path encodings also leads to our novel code representation model TPTrans. Extensive experiments and ablation study demonstrate the effectiveness of our approaches on code summarization across four different programming languages.
7
Acknowledgements
We thank all reviewers for their constructive comments. This research is supported by the National Key R&D Program of China under Grant No. 2020AAA0109400,, and the National Natural Science Foundation of China under Grant Nos. 62072007, 61 61832009, 61620106007.
References
Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. On the naturalness of software. Communications of the ACM, 122-113111331 2016.
10

---

Vincent J Hellendoorn and Premkumar Devanbu. Are deep neural networks the best choice for modeling source code? Proceedings of the 2017 1 11th Joint Meeting on Foundations of Software Engineering, pages 763-773, 2017.
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. Code generation dual task of code summariza- tion. arXiv preprint arXiv:1910.0500 2019.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv: 01400, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees transformers. arXiv preprint arXiv:2003. /3844, 2020.
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. arXiv preprint arXiv:1711.00744 2017.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerabil- ity identification by learning comprehensive program semantics via graph neural networks. arXiv preprint arXiv: 0009,,3996 2019.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. Structured neural summarization. arXiv preprini PeIXiv:/811.001 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint
2017.
Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In International conference on learning representations, 2019. 2019
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.0211 2018
Daniel Zilgner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. Language-agnostee representation learning of source code from structure and context. arXiv preprint arXiv:2103.11318, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.000 19037 2019.
Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In International Conference on Machine Learning, pages 245-256. PMLR, 2020.
Hoa Khanh Dam, Truyen Tran, and Trang Pham. A deep language model for software code. arXiv preprint 1001: 2016.
Song Wang, Devin Chollak, Dana Movshovitz-Atiia, and Lin Tan. Bugram: bug detection with n-gram language models. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, pages 708-719, 2016.
Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for extreme summarization of source code. In International conference on machine learning, pages 2091-2100. PMLR, 2016.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2073-2083, 2016.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for programming language processing. Thirtieth AAAI Conference on Artificial Intelligence, 2016.
11

---

Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed rep- resentations of code. Proceedings of the ACM on Programming Languages, 3(POPL): 2019.
Richard Socher, Cliff Chiung-Wi Yu Lin, Andrew Y Ng. and Christopher D Manning. Parsing natural scenes and natural language with recursive neural networks. In ICML, 2011.
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced Istm for natural language inference. arXiv preprint arXiv: ww0.:vv:9..0000000 06038, 2016.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating 2018.
tree structures into recurrent neural networks. arXiv preprint
Vighnesh Shiv and Chris Quirk. Novel positional encodings to enable tree- based transformers. Advances in Neural Information Processing Systems, 32: 2081-12001, 2019.
Yau-Shian Wang, Hung-Yi Yi Lee, and Yun-Nung Chen. Tree transformer: Integrating tree structures into self-attention. arXiv preprint
2019.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02880 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Guolin Ke, Di He, and Tie- Yan Liu. Rethinking the positional encoding in language pre-training. arXiv preprint arXiv:2006. / 55595, 2020.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoderr approaches. arXiv preprint arXiv: /409. /259, 2014 4.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint 1rX1v:190..09436.
2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06455 2016.
arXiv preprint
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks.
arXiv preprint
150.0:/506.03334, 2015.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv: 1422,660, 2014.
arXiv preprint
12


Reviews:
[{"review_id": "bCnGbSQBiVs", "reviewer": "Reviewer_43xz", "summary": "The paper presents a way to incorporate the tree-structure information into a transformer architecture to better learn representation for code. The tree structures are first dispatched as leaf-to-leaf \"relative paths\" and leaf-to-root \"absolute paths\", and then embedded and modulated as additional learnable parts of the query, key, and value components within the transformer attention. The authors evaluate their proposal on the task of extreme code summarization -- predicting function names. Results show improvements over 4 previous non-transformer and transformer models on 3 out of 4 programming languages tested. The authors also carry out ablation studies to understand the effect of absolute paths and pointer networks but many important questions remain unanswered. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The writing is clear and easy to follow. The paper follows the current trend of using transformers to learn code representation and incorporating richer structures into the originally sequence-based transformer model.\n\nThe proposed way for adding relative paths and absolute paths into the transformer attention seems natural but the exact places and terms added are somewhat arbitrary. For example, why the relative-path terms are added at the K part (Eq. 6) and the V part (Eq. 7) but not the Q part? For absolute paths, why the responding terms are only by themselves (Eq. 9) but not with any of the Q, K, V parts?\n\nThe paper also needs a clearer description of experimental settings, especially training settings (see Questions below), to fully make sense of the results.\n\nThe main results (Table 1) show a majority but not definitive gain. Even for where it gains, the gains are modest. \n\nAdditionally, why JS is such an outlier? Besides the under-supported claim that \"JS's structure is weak\", are there deeper reasons related to the dataset and the proposed model? It would be great if the authors could shed more light on the case.\n\nI appreciate the effort the authors put into the ablation study. However, it also raises questions. Why is adding absolute paths -- more information for the model to rely on, degrades the metrics? Shall we try to add more relative-path terms since the trend in Table 3 suggests more is better? Rather than conclude that absolute paths do not really help, shall we try other ways of adding the terms and/or adding the terms to different places? Table 4 and 5 generally shows the proposal still works well in combination with sequence bias and pointer network, which is good.\n\nIn general, I see a solid effort and a sensible way to incorporating tree structures into transformers for learning code representation. However, the exact ways of incorporating paths lack justification, and the improvements in numbers are limited and the unclarity of the setting further decreases the credibility. The significance of the paper could be improved by giving more supported insight into how the tree structure, in the way it is used, is helping or not helping with learning code representation under different scenarios. \n\n### Pros\n- The writing is clear and easy to follow.\n- Many potential audiences in learning code representations using structures.\n- Natural way of adding tree structures as paths into transformer attention.\n- Experiments show the proposal generally improves metrics.\n\n### Cons\n- The exact way of modifying attention seems arbitrary and lacks justification from either intuition or ablation study.\n- Experimental results show a majority but not definitive gain, and the gains are modest. The no-gain case (JavaScript) is not well-explained.\n- The efficacy of the proposed architecture is only evaluated over one training dataset and one task.\n- Not too much insight was generated about how and why it works or not for future research.\n- The title says \"for Code Representation\" but the authors only experimented on code summarization. \n\n### Questions\n- Justifications for the seemingly arbitrary places to add relative- and absolute-path terms in Eq. 5-10.\n\n- Main results in Table 2. What are all the data used to train each model? For transformer-based baselines, did the authors train them from scratch, or used a pre-trained model and then fine-tuned over the CSN datasets? This question is related to fair comparison as well as if the models with a large number of parameters will become under-trained or overfitting to a relatively small number of training instances.\n\n- Table 2, Table 5. Why is the trend so different for JavaScript? Besides the under-supported claim that \"JS's structure is weak\", could it be because the dataset has biases or unusual properties, or the proposed model itself has some \"flaws\"?\n\n- L320 \"If two sub-tokens are too far away ... the relative and absolute path encoding play the same role ...\" Can we find more evidence to justify or falsify this? Local pairs can still benefit from absolute paths. Additionally, if there is any trimming of the paths (for example, due to input-length constraints), then absolute paths can keep information that would be otherwise discarded from (the longer) relative paths.\n\n### Other points\n- L278 \"... since the structure of Javascript is much weak, ...\" \n- L339 \"... the structure of Javascript is weaker than other languages, ...\" I don't find references or supporting evidence for this claim. What makes the authors conclude that JS is less structured?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper could benefit from a dedicated discussion about limitations. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "TXcVtCsdfgE", "reviewer": "Reviewer_FGbv", "summary": "This work continues on earlier advances in how positional and relational encoding is used by the self attention primitive of Transformers, in particular in the context of source-code understanding. Prior work has biased self attention to encode relative relational information between inputs (e.g., edges or distance on some graph), and to separate the self-attention term due to position from that due to input.\n\nFor the former, TPTrans runs the path between any two tokens (specifically, the path in the code Parse Tree between two leaves -- the authors call this the \"relative path\") through an RNN, and uses the resulting embedding to bias the self attention between those tokens.\n\nFor the latter, TPTrans-$\\alpha$ runs the path from the parse-tree root to a token through an RNN (the \"absolute path\"), and uses that to construct a separate positional self-attention term.\n\nThe results show that both TPTrans and TPTrans-$\\alpha$ outperform prior work (most notably, Code Transformer), but TPTrans seems to dominate TPTrans-$\\alpha$ with a smaller set of trainable parameters.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overall\n\n* Originality: This is an interesting twist to prior advances (GREAT and Code Transformer, and perhaps TUPE). It's a natural twist, but a non-trivial one.\n\n* Quality: The quality of the evaluation was mostly good, with small issues (e.g., the lack of confidence intervals). I particularly appreciated the authors showing TPTrans-$\\alpha$, even though it was largely a negative result. \n\n* Clarity: The presentation was good, especially Section 3. It gave a very precise and contextualized description of what's new and what isn't, which is greatly appreciated. The experimental section was harder to understand, and I don't have a good understanding of what precise baselines were run, and whether the comparisons are fair.\n\n* Significance: This is still a new area, and the proposal here is likely to influence future Transformer-like ML architectures for code  or other structured inputs. As such, it's a non-trivial new contribution.\n\n\n# Other Comments\n\nI really appreciated the detailed contrast between what had been done before (Section 3.0) and what TPTrans is introducing (Section 3.1). This helps distill the differences and novel contributions, especially contextualizing them to provide motivation. In this effort, it would have been even more helpful to the reader to use the same notation where it wouldn't affect semantics. For examples, there's no reason why $a_i$ and $a_j$ in Equations 8 and 9 can't be denoted with $p_i$ and $p_j$ from Equation 3. Simialrly for $W_a^{Q/K}$ and $U^{Q/K}$.\n\nIt would be helpful to provide confidence intervals for your results. It's not clear if your improvements are statistically significant otherwise.\n\nI found some parallels in absolute/relative paths to AnyCodeGen's evolution of code2seq (Alon, Uri, Roy Sadaka, Omer Levy, and Eran Yahav. \"Structural language models of code.\" In International Conference on Machine Learning, pp. 245-256. PMLR, 2020.) I found it rather odd that it was not cited and mentioned.\n\nAlthough removing the layer index from equations (e.g., 6, 7, ...) is helpful, I found it problematic for Equation 5. Please add a precise definition of $\\mathit{Path}_{x_i\\rightarrow x_j}$. Does this change for every layer, or is it dependent solely on the initial embeddings of node types?\n\n# Smaller Issues\n\n* Your Hellendoorn 2019 paper was at ICLR2020, not ICLR2019.\n\n* Line 216: Table 4 should be Table 1.\n\n* Line 221: Please explain equal interval sampling. How do you choose the interval? Is it fixed or chosen to provide a sample of length 32?\n\n* Lines 245--249: Pointer decoders have been used in the relevant literature for different purposes. Copying (which is what you do here) or to identify positions for an edit (e.g., in HOPPITY and Hellendoorn et al 2020). Please clarify that you're using pointers for copying, not as part of the task definition. Also, please explain how you use the pointer network exactly. The information you provide does not allow the reader to reproduce your architecture.\n\n* Lines 260--271: The text here is very hard to parse and to understand how exactly you have constructed your baselines.\n\n* References: I found the order of citations very distracting. It was very hard to locate a citation, given that the references are provided in the order of appearance rather than alphabetical order.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "JhIrSsoeVDT", "reviewer": "Reviewer_SWC5", "summary": "This paper introduces a new neural network architecture for code, TPTrans (Tree Path Transformer), by modifying the attention calculation in the Transformer architecture to use information about paths in the syntax trees of programs. The modification works by computing path embeddings between pairs of leaf nodes (\"relative path embeddings\") and between leaf nodes and the root (forming nodes' \"absolute path embeddings\"), and incorporating these into the Transformer's attention module. The paper evaluates TPTrans, including baselines and variants, on code summarization -- the task of predicting a function's name given its body -- for four different programming languages. The main experiments reveal the value of including embeddings of sequences (the paths) in the attention calculation over baselines, and subsequent experimentation compares the impact of relative and absolute paths, and the effect of pointer network outputs on the model.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper's core contribution is the introduction of TPTrans, a method of incorporating paths from a syntax tree into the attention module of the Transformer architecture. By evaluating this approach with two kinds of paths on code summarization, and by finding it outperforms baseline methods, the paper indeed shows the feasibility of incorporating sequential information into transformer attention as an inductive bias, as claimed.\n\nThis is an important area of study, as machine learning for source code is an ever increasingly important domain, yet techniques from natural language processing such as the transformer architecture do not take advantage of properties of source code. Identifying models that learn better representations of source code and fully take advantage of the information and structure present in source code, including its semantics, is a critical challenge. This paper is a step toward this, identifying a method of incorporating program structure for better learned representations of code.\n\nThe method proposed for incorporating relative and absolute path encodings into Transformer attention is conceptually simple, and is explained clearly in Section 3.1. The conceptual simplicity of the proposal is a strength of the paper, as is the clarity of the description in this section. The main result, showing on most tasks that the relative path variant TPTrans outperform all baselines including the absolute path variant TPTrans-$\\alpha$, is another strength.\n\nThe specific choices of absolute path for each node and relative path for each node-pair makes the method specific to code. These paths are embedded using a GRU over the sequence representation of the path. Beyond these choices, the approach is more general, applicable for incorporating any pairwise and element-wise node information about a sequence\n\nConsidered individually these components are not novel. GRUs are a standard way of encoding sequential information, the transformer is a standard sequence-processing architecture, the use of absolute position encodings is present in the original transformer and relative position encodings are introduced in [1]. The paper's strength is in it's combination of these components, demonstrating that sequence information is suitable for both relative and absolute position encodings, and that in the context of source code choosing these position encodings improves performance on an important benchmark task. That said, since TPTrans is primarily a novel combination of existing architecture components and an extension of the transformer attention module similar to ways it has already been extended, the size and significance of the contribution is limited.\n\nEvaluations are done on the task of code summarization, which is a sensible task for evaluating performance on representation learning of source code. Four programming languages are considered, Python, Javascript, Go, and Ruby, and TPTrans outperforms the baselines considered on all languages but Javascript. One area for improvement in the experimental setup is clarifying the axes along which the models are comparable. Do they have comparable parameter counts? If the increase in performance were only due to an increase in parameter count, which could also be achieved by scaling up existing models, that would decrease the significance of the result.\n\nA weakness of the experiments is they lack error bars or any measure of confidence or variation. Independent of this, the experimental evidence is also somewhat thin, as a single task is used to demonstrate improvements on the quite broad goal of representation learning.\n\nOne potential way to improve the paper would be to demonstrate why the path embeddings are useful. For example, the model admits inspecting which path embeddings are attended to. Do these exhibit any patterns or have any particular semantic meaning?\n\nAs an aside, the paper unfortunately contains many grammatical and typographic errors. Usually these do not impede clarity too much, and in my review score I do not count them against the paper. Some of them do make the paper more difficult to read, however (e.g. lines 269-270), but in all cases I think I was able to determine the intent of the sentence.\n\n[1] Self-attention with relative position representations.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A technical limitation of the work is: The method relies on having a parse for the program, which limits the approach to well formed code and languages supported by tree-sitter. The baseline Transformer does not have this limitation. The paper does not address potential societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "Parts of the introduction to Section 3 are lifted from Section 3 of [Language-Agnostic Representation Learning of Source Code from Structure and Context](https://arxiv.org/abs/2103.11318).", "needs_ethics_review": "Yes", "ethics_review_area": ["Research Integrity Issues (e.g., plagiarism)"], "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that even though the general aspects of using recurrent units for embedding paths, and using relative and absolute paths for embedding source code are not completely novel, the paper presents a nice novel combinations of these ideas. There were many clarity issues in the writing, but the response helped clarify many of them. One of the major concerns that came up was that evaluation did not clearly show that the proposed architecture was consistently better, and the evaluation was only performed on a single task of code summarization. The new results for code classification task are interesting, but it is not clear if TPTrans performs better than stronger baselines. Finally, there were also some concerns regarding better understanding of the benefits and limitations of the approach, and how incorporating paths might improve (or worsen) the performance of the model.
Decision: Accept (Poster)