Integrating Tree Path in Transformer for Code Representation

Abstract: Learning distributed representation of source code requires modelling its syntax and semantics. Recent state-of-the-art models leverage highly structured source code representations, such as the syntax trees and paths therein. In this paper, we investigate two representative path encoding methods shown in previous research work and integrate them into the attention module of Transformer. We draw inspiration from the ideas of positional encoding and modify them to incorporate these path encoding. Specifically, we encode both the pairwise path between tokens of source code and the path from the leaf node to the tree root for each token in the syntax tree. We explore the interaction between these two kinds of paths by integrating them into the unified Transformer framework. The detailed empirical study for path encoding methods also leads to our novel state-of-the-art representation model TPTrans, which finally outperforms strong baselines. Extensive experiments and ablation studies on code summarization across four different languages demonstrate the effectiveness of our approaches. We release our code at \url{https://github.com/AwdHanPeng/TPTrans}.


Reviews and author responses:

 Thanks for your reading. 

As you point out, incorporating syntax _structure_ of code snippet is much essential for code representation learning, which is worthy of in-depth research. Much appreciate your valuable suggestions and positive comments again.

Besides, we have added many experiments and post new public comments about these critical issues. Please refer to them and point out any other possible confusing statement further, and we will try our best to eliminate it. 

Thanks so much again.

The paper proposes an Transformer-based architecture for embedding source code snippets that integrates encoding of
absolute and relative AST paths for all code tokens into self-attention. Following up on prior work in relative position
encoding, they map AST paths into the position encoding framework, modify the self-attention computation accordingly,
and evaluate on method summarization. The approach is compared to recent Transformer-based baselines on the standard
method summarization benchmark, as well as to code2seq, which pioneered path-only source code encoding. The results
improve performance significantly on almost all benchmarks.  ## Discussion update

The authors have conducted numerous experiments to clarify my (and other reviewers') concerns, answer our questions, and compile a clearer picture of the paper's findings. As of now, it presents a key simple idea – an AST-dependent way of incorporating learned relative positions into code Transformers – and evaluates it on multiple domains and dimensions against SOTA single-model baselines. The paper has extensive empirical backing even for a simple idea. In fact, the idea's simplicity works in its favor here.

Many of our questions and new experiments would be appreciated in the new version of the paper, including:
- "Deep and narrow" Transformer baseline
- Discussion of model sizes
- Discussion of the distribution of unique paths (and JavaScript outliers)
- Code classification task
- More appropriate discussion/speculation on the relationship between absolute and relative positions' performance
- Various perspectives on prior work discussed during the rebuttal phase, which make the paper clearer.

After the discussion, I would love to see this work in NeurIPS, and increasing my score accordingly.

---

TPTrans extends the ideas of code2seq and CodeTransformer to encode concrete paths over the AST as a substitute for
"positions" in the Relational Transformer framework. It is a neat and simple idea, made possible by the unification of
multi-lingual parsing by `tree-sitter`, and experiments show that it improves code representations further. The
integration of prior ideas into this framework is relatively straightforward but I believe it's a reasonable and useful
contribution to the field.

The authors experiment with both "relative-like" and "absolute-like" AST-derived positions, and find that absolute
positions don't add any value over the relative ones, and in fact hurt performance. I appreciate the inclusion of the
negative result, it is useful and informative in context. Interestingly, this contrasts with natural language, where
state-of-the-art language models still use absolute position embeddings, likely due to computational challenges involved
with storing $n \times n$ embeddings for position pairs. For code, the number of practical different paths is lower,
which allows the sparse optimization the authors introduce in Section 4. The implementation of this sparse relative path
encoding is another nice contribution of this paper.

**Q1:** Can you plot the actual number of different paths for different programming languages and their distribution
(perhaps faceted by program size)? A canonical vocabulary of "most useful code positions" would help follow-up work.

**Q2:** Intuitively, mere information redundancy (Table 3) would not _lower_ performance, especially given that
TPTrans-$\alpha$ has two more parameter matrices. An alternative explanation might be that additional parameters are
spent _only_ on absolute position projection, which might over-emphasize importance of absolute positions and overfit
the learned code representation to them.

The evaluation follows the same standard as prior work on the same topic, with proper ablation experiments and
state-of-the-art baselines. Qualitative evaluation is a bit lacking. Please include an example that demonstrates the
value of joint absolute/relative encoding -- e.g., something not handled by either code2seq or CodeTransformer. Some
other qualitative evaluation of representations (e.g., clustering) would also help the reader.


## Evaluation questions

**Q3:** What is the standard error in Table 2? The prior work does not include them either, thus a significant
improvement on this dataset is hard to estimate. That said, the delta improvement of TPTrans seems similar to such of
state-of-the-art works to their predecessors.

**Q4:** Addition of domain-specific inductive bias (like source code paths) usually compensates for model capacity. What
are the sizes of all evaluated models in Table 2? Furthermore, does TPTrans show an even more noticeable improvement
upon e.g. CodeTransformer when evaluated on equal _small_ model configurations?

**Q5:** I could not understand L266-268. Does it describe TPTrans (as evidently), or does it state that your
re-implementation of CodeTransformer is different from Zügner et al.? If it's the latter, the comparison in Table 2 is
not strictly equivalent.


## Minor

Proofreading:
- L269-271: "realize" and "plus" seem out of place / ungrammatical.
- Please sort the references. 
Some limitations are present in the paper as negative results or performance discussions. No other dedicated sections
discuss societal impact. However, it does not significantly differ in potential impact from prior similar work.


 Some Reviewer shows concerns about the first contribution we present, which says _We show the feasibility of embedding sequential information as an inductive bias for the
attention module of Transformers and are the first ones to do so to our best knowledge_ at L76. Here we explain further about this.

Above all, the _sequential_ information we say here means encoding a vector sequence, specifically referring to the path's node sequence in our paper. We firmly believe that integrating sequential information into Transformer as an inductive bias is not just trivial. 

1)Firstly, directly integrating standard RNN into the Transformer's deep backbone most likely results in exploding or vanishing gradient.

2)Secondly, as we all know, the attention procedure of the Transformer can be seen as the similarity calculation for vectors. The _K_, _Q_ and _V_ of the Transformer's attention module all be normalized while the RNN's recursive process does not limit it, which may lead to the calculation instability for attention scores. 

To overcome these issues, we carefully design the normalization method. Firstly, we replace the standard GRU cell as a layer-normalized GRU, leading to more stable parameters updates. Secondly, after getting the final state of a sequence, we feed it into a LayerNorm module to keep the same vector norm with _K_, _Q_ and _V_.


The result is:

|Model|Ruby.P|Ruby.R|Ruby.F1|JS.P|JS.R|JS.F1|
|----|----|----|----|----|----|----|
|TPTrans|32.40 $\pm$ 0.84|27.63 $\pm$ 0.78|29.82 $\pm$ 0.79|33.83 $\pm$ 0.55|28.37 $\pm$ 0.54|30.86 $\pm$ 0.53|
|  w/o Norm|26.91 $\pm$ 0.42|23.99 $\pm$ 0.31|25.37 $\pm$ 0.36|31.42 $\pm$ 0.16|26.34 $\pm$ 0.18|28.66 $\pm$ 0.10|


From this table, we conclude the normalization method is very effective. These results show that it is harder for the composite model to fully extract _sequential_ information without norm control. 

So we say that we prove the feasibility of embedding sequential information as an inductive bias for the attention module of Transformers. And we think that our approach also contributes to the design of _neural network architectures_, not only to the community of Code Intelligence. 

We have mentioned this design in L254, although many details still have not been presented in our paper. But we are also glad to add the details in the later version if reviewers suggest it.

 For your concerns about the first contribution of embedding _sequential information_, we explain further in the public comment as 'The Further explanation about sequential information integrating'. Please refer to it for more details. 

Thanks for your suggestions again.

 Since some reviewers concern that our proposed model is only evaluated on the code summarization task, we further experiment on the code classification task using the POJ-104 dataset[1], which contains 52000 C programs of 104 classes. We remove the Transformer decoder of our model and equip an attention-based weighted sum module to merge encoder features and then feed it into an MLP classifier. 
Please note that:

1) most baselines of POJ classification use PycParser to parse code into AST while we still pick Tree-Sitter parser here;

2) unlike the standard experimental setup, many baselines omit each AST node's _Token_ field and only consider the _Type_ field to show the exclusive _structure_ learning power. For _approximate_ comparison, we mask all the variable names of CST, which means only  _Structure_ information is considered for our model. After that, only language Keywords are shown in the code sequence, and then the total vocab size reduces to 38. These results are present as POJ-Structure setting. 
 
The baselines' accuracy:

|Setting|BiLSTM|Code2Vec|TBCNN|ASTNN|Treecaps|
|----|----|----|----|----|----|
|POJ|83.51|86.21|95.21|98.2|98.32|

|Setting|TBCNN|Treecaps|
|----|----|----|
|POJ-Structure|92.64|95.88|

And the performance of our models and ablation study:

|Model|POJ|POJ-Structure|
|----|----|----|
|TPTrans|94.94|93.03|
|w/o relative path in V|93.87|88.01|
|   w/o relative path in K,V|89.34|77.89|

First of all, we conclude that our model gets remarkable performance although not receives state of the art. The ablation study then shows that the path integrating method is still powerful on the classification task and improves much compared with the Vanilla Transformer (presented as 'w/o relative path in K,V').

Comparing the performance of our model with the baselines, we guess that the parser we use may not be the best choice for the code classification task. The authors of [2] also prove that different parsers can indeed result in much difference. So we think that the further research for different parsers would perhaps be valuable for the whole research community of code representation learning. 

[1]Convolutional neural networks over tree structures for programming language processing

[2]TreeCaps: Tree-Based Capsule Networks for Source Code Processing


 Thanks for your valuable suggestions. After careful consideration, we confirm that your suggestions are reasonable.  As you point out, our statement seems to be overly broad, which results in the unintentional exaggeration of our contribution. We will carefully fix it in the later version, including adding more limiting conditions and perhaps reorganizing the presentation order of each contribution. Thanks again for your advice, which makes our paper more accurate. 

Please point out any other confusing statement in this paper, and we will try our best to eliminate it.

The authors present a technique for modeling source code. They point out that modeling code is different from language because of the tight binding between ASTs, data, and control flow. Also they mention some weakness of GNNs. To compete with GNNs, they introduce a new method which is a sequence-based transformer that incorporates graph structure through the attention weights. The model is shown to be effective for code summarization.  Strengths

- The proposed technique is simple and effective.
- The proposed technique outperforms other models in all categories besides Javascript.

Weaknesses

- It is very difficult to gauge which modeling decisions are introduced in this paper, and which are borrowed from other work. There is clearly a lot of overlap with Code Transforme, such as the use of the pointer network.
- There are results in the Code Transformer paper that outperform results shown here. I realize that this is a very recent related work, and that it is perhaps with different training settings, but Code Transformer was mentioned many times and it seems important to address.
- It is not especially clear why performance on Javascript is not as strong. I would recommend the authors double check this setting and see if there is some issue? It is not such a major issue if the model does not outperform in all settings, but the explanation that Javascript is somehow less structure dependent would likely need more justification.
- The presentation of the analysis could be improved. Although the subsections and tables clearly indicate what was ablated, it is not clear to me what is learned from these experiments.
 n/a

 The finding wrt. to javascript is oddly encouraging and the results seem more reasonable given this new analysis. In general, I think this paper would benefit greatly from continued refinement of presenting the methods, and would benefit even more from in-depth analysis of the structure being incorporated. Nonetheless, with the response thus far I am happy to improve my score to weak accept.

 *We show the feasibility of embedding sequential information as an inductive bias for the attention module of Transformers and are the first ones to do so to our best knowledge* - this is a stand-alone statement and stated as your first contribution. Please provide additional details of what sequential information means here before claiming it as a contribution especially when you are saying something like you are the first ones to do so. Sequential information can be anything. It just means information present in a sequence.

 1)Motivation:

we think it is meaningful to make an empirical study for the relationship between these two kinds of paths(called _relative_ path and _absolute_ path in our paper). Above all, lots of works have introduced either kind of path into many novel models[1-4]. However, to our best knowledge, none of them has assembled both paths simultaneously, and the relationship between these two kinds of paths is still unknown. On the other hand, as mentioned in our paper(L66-L70), both paths intuitively contain plenty of structural information of code syntax. For this reason, we think it is reasonable to focus on both of them instead of overlook either.

To further explain our empirical study's motivation, we also want to make an analogy from the research community of NLP. Technically, the ways we integrate path encodings into Transformer are mostly derived from the NLP field's _positional_ encoding methods. That is: 

Relative path encoding vs. Relative positional encoding

Absolute path encoding vs. Absolute positional encoding

In the NLP community, the discussion about positional encoding has been going on for a long time, and tons of encoding method has been presented[5-7]. In contrast, as we mentioned above, the research of different paths is just emerging. From this point of view, we think our research is necessary and valuable for the Code Intelligence community.

2)Experiment Result and Conclusion

From Table3 presented in Section 5.2(Ablation Study), we conclude that after introducing full of the relative path encoding, the improvement from the absolute path encoding almost diminishes. It shows that these two kinds of paths are not orthogonal, which is still intuitive.

We feel much sorry for that, because of lacking error bars for Table3, readers maybe deduce a misguided conclusion that a concrete conflict definitely exists between two kinds of paths. Our somewhat misleading explanation at L319 further deepens it. On the contrary, actually, the only thing we firmly conclude here is that only feature overlap certainly exists rather than conflict. 

For further explanation, we show the performance of TPTrans-alpha equipped with a pointer network (we repeat the experiment several times for confidence). Unlike TPTrans(Eq.6+Eq.7), TPTrans-alpha fully equips both the relative and absolute path encodings (Eq.7+Eq.10). The result is:

|Model|Ruby.P|Ruby.R|Ruby.F1|JS.P|JS.R|JS.F1|
|----|----|----|----|----|----|----|
|TPTrans|32.40 $\pm$ 0.84|27.63 $\pm$ 0.78|29.82 $\pm$ 0.79|33.83 $\pm$ 0.55|28.37 $\pm$ 0.54|30.86 $\pm$ 0.53|
|TPTrans-alpha|33.07 $\pm$ 0.86|28.34 $\pm$ 0.61|30.52 $\pm$ 0.53|33.68 $\pm$ 0.34|28.95 $\pm$ 0.54|31.14 $\pm$ 0.46|

|Model|Python.P|Python.R|Python.F1|Go.P|Go.R|Go.F1|
|----|----|----|----|----|----|----|
|TPTrans|38.76 $\pm$ 0.49|34.66 $\pm$ 0.35|36.59 $\pm$ 0.02|55.79 $\pm$ 1.3|51.05 $\pm$ 1.3|53.30 $\pm$ 0.83|
|TPTrans-alpha|38.39 $\pm$ 0.18|34.70 $\pm$ 0.26|36.45 $\pm$ 0.22|55.67 $\pm$ 0.92|51.31 $\pm$ 0.76|53.39 $\pm$ 0.13|

From this table, we conclude that the improvement of TPTrans-alpha over TPTrans is not consistently significant, and the so-called _conflict_ is also not a certainty. Actually, these results show that the feature overlap indeed exists. For these reasons, we finally select the TPTrans as our main proposed model, which is also consistent with the Principle of Occam's Razor. 

We feel much sorry for any confusion. We will carefully fix them in the later version.



[1]Code2seq: Generating Sequences from Structured Representations of Code

[2]Code prediction by feeding trees to transformers

[3]Structural language models of code

[4]Code2vec: Learning Distributed Representations of Code

[5]Self-Attention with Relative Position Representations 

[6]Rethinking the positional encoding in language pre-training 

[7]Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

 Combining the two views about _Context_ and _Structure_ for source code has always been widely discussed in the research field of Code Intelligence. The _Context_ mainly refers to representing code as a sequence of text, and the _Structure_ refers to extracting knowledge from code snippets' syntax. 

Unlike the Transformer-based baselines we picked, the Code2seq uses only pairwise path information in AST to represent code snippets. In other words, code2seq mainly focus on the code _Structure_ within the syntax tree and, as a result, lacks the modelling of _Context_. On the contrary, our proposed model and other Transformer-based baselines are designed based on Transformer, which means the _Context_ is naturally focused.

Essentially, the difference between our model and our baselines(Great, CodeTransformer) is encoding path vs encoding manually designed edges (Great) or distances/hops(CodeTransformer). The path's different node combinations contain plenty of structure information, which is overlooked by encoding distances. To verify the benefit of learning from paths than distance, we conduct ablation studies at L326-L340.

The XLNet is not designed for learning source code representation, and it is designed initially for huge-corpus natural language pretraining. The reason we list XLNet for comparison is that the Code Transformer is designed based on XLNet. Code Transformer additionally incorporates multiply structural relations within code syntax trees over XLNet. Although the XLnet is not specially designed for source code, its novel architecture of two-stream self-attention still shows powerful performance in code representation learning.

Please refer to the table below for clear comparisons.

|Model|Context|Structure|
|----|----|----|
|Code2Seq|No|Pairwise Path|
|Great|Yes|Manually Designed Edges|
|XLNet|Yes|No|
|CodeTransformer|Yes|Multiply Structural Distances|
|TPTrans|Yes|Pairwise Path|

Integrating the rich structural information available in source code is a key research topic. This work proposes to embed paths in syntax trees into attention-based models, demonstrating that this excells over both simpler path-based models and various Transformers for source code.  ### Update, following discussion
The authors provided thorough responses and relevant new results during the response phase. Therefore, I am updating my assessment to a (slightly) more positive rating. Please incorporate the many helpful comments you received. In particular, the authors are strongly encouraged to include a further analysis of "deep and narrow" models (ideally, also larger ones), since, to the best of my knowledge, the "squat and wide" (3 layer, 1,024 setup) was only used by Zügner et al. and may not be representative of commonly used Transformer architectures. Enriching the table in the below response with the other models and datasets would be a good start, as would including a bigger model, such as the 10 layer variant used by Hellendoorn et al.

### Original Review
The main contribution of this work is to merge the intuition from Alon et al.'s AST-paths and work on biasing Transformer attention towards code's structural features. This is a worthwhile contribution, and results suggest that it translates into performance improvements.

The primary concern is a fairly straightforward flaw in the comparison with other models: TPTrans adds a non-trivial number of parameters through its various RNNs, but apparently does not account for this discrepancy in model capacity in its comparisons. The results may well hold even after compensating for this imbalance, e.g. by employing more layers or larger hidden dimensions in the baseline models, but until this comparison is run it is hard to interpret any performance improvements.

This aside, I articulate a number of smaller concerns below. Primarily, the writing is quite poor and occasionally glosses over details that seem important. Please significantly improve the writing of the paper throughout.

L203: code summarization by specifically generating a function's name has been considered a dubious task for some time now, primarily due to both high ambiguity and repetitiveness in function names in practice. Consequently, he paper would be strengthened by including other tasks, such as defect detection.

P6: the "Efficiency" section lacks a lot of details, some of which seem rather important (e.g. L242), but are relegated to the appendix. I would encourage trying to include more details directly here (see note under presentation on freeing up space), ideally at least Figure 2 or something equivalent. In the absence of such room, and really regardless, please ensure that the parts explained here can stand alone; especially the phrase "Finally, we reduce the costs to be linear." needs to be expanded.

The explanation of the JavaScript results relative to other languages feels a bit handwavy. This is not a major concern, but illustrating this concretely, for instance with examples of outliers or representative samples, would help.

I did not find any record of the number of layers used for the models. Please add this to p7.

The explanation of the "absolute" path performance, being worse than the relative one only, feels off-base. If the information were redundant, we should not expect a drop in performance by including it. L319 further mentions a potential "conflict" between these sources of information, which, as far as I can tell, is not further substantiated. This all deserves further exploration.

Table 5 seems to be missing the ablation for other models. Or did those never use the sentinel pointer setup?


### Presentation
The text on the first four pages or so is a bit redundant, repeatedly discussing the related literature and the relation of TPTrans across the introduction (rendering it over 2 pages long), section 2, and section 3.1. Please concentrate this and remove duplicate explanations, which will also free up space to discuss some of the important details that are currently relegated to the appendix.

Throughout, avoid using references as nouns. E.g., instead of "[Alon et al., 2018] proposed", use "Alon et al. [2018] proposed". In Latex, use the \citet command instead of \citep for these cases.

The names "relative" and "absolute" are odd names for the these two types of paths. Consider revising, to something more direct, like "leaf-to-leaf" and  "leaf-to-root".

### Typos:
The below is only a sample, please make a thorough pass to improve writing quality.
- L9: "attentions" -> "attention"
- L17: "semantically embedding" -> "semantic embeddings"
- L26: "over AST" -> "over an AST"
- L27: "The code2seq" -> remove "the"
- L28: "Besides" -> here and throughout, prefer using another word (or simply drop this) when the purpose is to describe a series of options.
- L53: "counts the distances" -> unclear
- L73: "on most of all" -> "on most"
- L111: "works" -> not the plural of "work"
- L120: use tall parentheses for the RHS. In Latex, use \left( and \right) around the faction.
- L126: $d_k$ is never used, presumably it was supposed to be for $W^K$? Given the next clause, that all d's are equal, this distinction seems unnecessary. Also, remove "The" before "Eq.2"
- L269: "the realize" -> unclear (same L270)
- L270: "which plus" -> unclear
- L278-9: several typos, revise. "much weak", "the encoding path", "count distance" (unclear)
- L315: "full of the" -> "the full" ?
- L331: "is degenerate" -> "degenerates"
- L336: "these results also prove workable" -> unclear
 The paper does not address either aspect of this work. The former is especially important.

 Q2: It is very difficult to gauge which modeling decisions are introduced in this paper, and which are borrowed from other work. There is clearly a lot of overlap with Code Transformer, such as the use of the pointer network.

A2: The whole story of our paper derives from two parts: 

1) Our first starting point is to encode the path of AST for code representation. This idea is first presented at Code2Vec and Code2Seq. The Code2seq obtains state-of-the-art of code summarization using only pairwise path information in AST. However, code2seq lacks the modelling of context, which leads us to explore combining the path representation with the source code context. Since the Transformer can model context well, we naturally consider integrating tree path encoding into Transformer.

2) The other point comes from our baselines(Great and CodeTransformer). These models design based on the Transformer, and both introduce the structural inductive bias in Transformer. Technically, they modify the positional encoding of the Transformer and use either manually designed structural edges or distance across AST to represent structural information. In this paper, we pursue the research line of combining the Transformer-based model with additional
structure information of source codes. 

Essentially, the difference between our model and our baselines is encoding path vs manually designed edges or distances. The path's different node combinations contain plenty of structure information, which is overlooked by encoding distances. To verify the benefit of learning from paths, we conduct ablation studies: before feeding the path into Transformer, we first convert all nodes in the path to the same one. In such a case, what the model learns is degenerate completely into hops across the syntax tree between two terminals.
For more details about ablation studies, please refer to L326-L340. 

Please refer to the table below for clear comparisons.

|Model|Context|Structure|
|----|----|----|
|Code2Seq|No|Pairwise Path|
|Great|Yes|Manually Designed Edges|
|XLNet|Yes|No|
|CodeTransformer|Yes|Multiply Structural Distances|
|TPTrans|Yes|Pairwise Path|

The pointer network enhances predictions by pointing at positions, and it has already been widely adopted in lots of seq2seq models (in both NLP and Code Intelligence). For more details about the pointer network, please refer to the A3 for y3EX.



Q3: There are results in the Code Transformer paper that outperform results shown here. I realize that this is a very recent related work, and that it is perhaps with different training settings, but Code Transformer was mentioned many times and it seems important to address.

A3: Very keen observation. One of the contributions of CodeTransformer is showing the benefits of training multilingual models across different languages. The code parser Semantic facilitates their multi-language setting by providing _unified_ ASTs for different languages. So as you see, after training across different languages, the performance of CodeTransformer outperforms the monolingual one. 

In this work, we mainly focus on studying the effectiveness of integrating different kinds of path encoding methods and the relationship between them, instead of multilingual training. So we mainly compare the result of monolingual CodeTransformer with our model. But it does not mean that our models cannot benefit from the multilingual training. 

For training across different languages, a _unified_ multilingual parser is the key. But unfortunately, the parser we used Tree-Sitter can not provide _unified_ syntax trees for different languages yet. As a result, the node type vocab for different languages is also different (for the Go language, the nodes vocab size is 94, and 105 for Ruby, Js and Python). For example, the node type 'statement_block' of JS is not present in Python language. So for the multilingual training, one possible solution is to modify the TreeSitter's parse syntax to provide _unified_ syntax trees for different languages. Another possible solution is to adopt a large node vocab, which is the union of different languages' node vocab. After such a process, we believe that our model can also benefit from multilingual training. Training multilingual models is indeed an interesting topic, and perhaps we will investigate it further in future.

 Q2: I think that the idea of using relative path encoding in the transformer self-attention proposed in the paper is very similar to the one in Zugner et. al 2021, except that instead of AST they use CST. On the contrary, I feel that the approach used in Zugner et. al 2021 that uses both structure and context is more meaningful. The source code context treated as a text sequence in addition to the structure provided by the tree representations can be quite helpful especially for code segments containing comments.

A2: As for the difference between Great, CodeTransformer and our proposed model, please refer to the A2 for ZRpb.

Combining the two views about _Context_ and _Structure_ for source code has always been widely discussed in the research field of Code Intelligence. The _Context_ mainly refers to representing code as a sequence of text, and the _Structure_ refers to extracting knowledge from code snippets' syntax. 

Unlike the Transformer-based baselines we picked (Great and CodeTransformer), the Code2seq uses only pairwise path information in AST to represent code snippets. In other words, code2seq mainly focus on the code _Structure_ within the syntax tree and, as a result, lacks the modelling of _Context_. On the contrary, our proposed model and two baselines are designed based on The Transformer, which means the _Context_ is naturally focused. Please refer to the table below for clear comparisons.

|Model|Context|Structure|
|----|----|----|
|Code2Seq|No|Pairwise Path|
|Great|Yes|Manually Designed Edges|
|XLNet|Yes|No|
|CodeTransformer|Yes|Multiply Structural Distances|
|TPTrans|Yes|Pairwise Path|

PS: If you are confused about XLNet and the relationship between it and CodeTransformer, please refer to the A1 for 86qv for more details.


Q3: The statement made in lines 54-55 is very vague. Please expand on it for better motivation.

A3: The L54-L55 is 'However, the different nodes combinations in the path contain plenty of structure information, which is overlooked by encoding distances'. 

For example, in Figure 1(b), please focus on those two tokens: the 'num' in 'def factorial(num):' and the 'result' in 'result=1'. The relative path between these two tokens is ['num', _Paramerters_, _FuncDef_, _Block_, _ExprStmT_, _Assignment_, 'result']. The distance(hops) between these two tokens obviously is 5. Unlike our proposed model, the CodeTransformer counts the distance between two tokens across the syntax tree to represent the _Structure_ information. Obviously, only having equal length does not mean that two paths are the same. The pairwise path represents the plenty structural relationship between two tokens, which is overlooked by only counting distance.


Q4: Why does the incorporation of absolute path encoding doesn't help? It seems like the authors spend a lot of time justifying their choice and in the end decided by using only the relative path encoding model.

A4: For the relationship between these two kind of paths, please refer to the A2 for y3EX. 


Q5: I think it should be W^Q in Eqn (10) instead of W^(V)?

A5: You're right. Thanks for your pointing.


Q6: What exactly are the x's and z's in the context of the CST?

A6: Pardon me?

Q7: Please include error bars for performances.

A7: Please refer to the A1 for 86qv.

Q8: I didn't exactly understand the reasoning for parameter sharing as stated in Lines 228-229.

A8: L228-L229 says, 'In our implementation, the path embedding matrix for nonterminals is shared for both absolute and relative paths while the path encoders are not'. Both absolute and relative paths are made up of non-terminals nodes within the syntax tree. So it is natural to share the lookup table embedding matrix of nodes for absolute and relative paths.

 A8: One salient point emerged from re-reading L250 and your discussion of (hyper-)parameters: the choice of a Transformer with just three layers (but a very wide hidden dimension) is very odd, and potentially undercuts a fair comparison. Deeper models are naturally able to infer more complex connections. Robbing the baselines of this potential would naturally augment a model with a heavy dependence on a sequential component like the one proposed here. If at all possible, I urge the authors to add results for a more conventional architecture.

Q8: Very keen observation. We design a shallow and wide model because all of our Transformer-based baselines set this configuration. So we also design like others for fairness. As for why all baselines make this decision, we speculate that a wider model means that the dimensions of structured embedding vectors(edge embedding for Great, distance embedding for CodeTransformer) will also be wider, ensuring the representation capability of structured embedding vector. But such speculations still do not mean that structural encoding cannot work on deeper models.

We re-design deeper TPTrans and vanilla Transformer and set 6 layers, 512 dims and 8 heads. For TPTrans, the input and output dim of GRUs are 64 and 32*2 ( *2 for bi-direction). The total parameters of the vanilla Transformer are about 49M (100M for the previous wider model). The total parameters of GRUs are 18816, and the proportion of the full model is 0.0387% (0.046% for the previous wider one). We repeat the experiment several times at Ruby dataset and get that:


|Model|Ruby.P|Ruby.R|Ruby.F1|
|----|----|----|----|
|TPTrans|25.42 $\pm$ 0.42|24.53 $\pm$ 0.68|24.97 $\pm$ 0.51|
|Vanilla Transformer|21.06 $\pm$ 0.72|19.47 $\pm$ 0.32|20.23 $\pm$ 0.50|

From this table, we conclude that our way of integrating path encodings still works for the deep and narrow model.  

 Thank you for your thorough reply and clarifications. The hyper-parameter trade-off, in terms of added parameters vs. the corresponding improvements, indeed feels more convincing now. Definitely do update the text ca. L315-319, as you mention.

One salient point emerged from re-reading L250 and your discussion of (hyper-)parameters: the choice of a Transformer with just three layers (but a very wide hidden dimension) is _very_ odd, and potentially undercuts a fair comparison. Deeper models are naturally able to infer more complex connections. Robbing the baselines of this potential would naturally augment a model with a heavy dependence on a sequential component like the one proposed here.

If at all possible, I urge the authors to add results for a more conventional architecture, e.g. 8 layers x d=512 (& dff=2048) dimensions. Without those results, it is hard to guess whether the proposed addition will still yield improvements on "normal" Transformers. Alternatively, please carefully defend why this method _requires_ a wide and short Transformer to work, and add this to the limitations of the approach.

 Q3: Can you plot the actual number of different paths for different programming languages and their distribution (perhaps faceted by program size)? A canonical vocabulary of "most useful code positions" would help follow-up work.

A3: We appreciate your concern about the distribution of different paths. For the Python language, the vocab size for nodes is 105 (105 for Ruby and Js, 94 for Go). The max number of all possible permutations of nodes is approximately equal to $m^n$, where m is the vocab size, and n is the path length. This number is huge, but the number of all the real paths are much smaller than it. So it makes sense to learn a distribution of real paths in a data-driven manner. 

Since the total number of paths is so big, calculating the distributions of different paths may not be appropriate. So we first try to count all unique relative paths presented in training datasets and get that:

|Dataset|Paths num|Dataset size|
|----|----|----|
|Ruby|10971671|48791|
|Javascript|52600545|81487|
|Python|81419629|412178|
|Go|45794921|317832|

The larger the datasets, the more unique paths should appear. But a wired phenomenon arises that the all paths num of Javascript is much big, which does not match its dataset size well. This phenomenon may reflect why our model's performance on JS is not as good as expected. To investigate this phenomenon further, we try to count the mean and variance of the length. Please refer to A1 for ZRpb for more details.



Q4:  Intuitively, mere information redundancy (Table 3) would not lower performance, especially given that TPTrans- has two more parameter matrices. An alternative explanation might be that additional parameters are spent only on absolute position projection, which might over-emphasize importance of absolute positions and overfit the learned code representation to them.

A4: Please refer to the A2 for y3EX for more details.

Q5: I could not understand L266-268. Does it describe TPTrans (as evidently), or does it state that your re-implementation of CodeTransformer is different from Zügner et al.? If it's the latter, the comparison in Table 2 is not strictly equivalent.

A5: Much sorry for any confusion. L266-268 mainly describe the different design between our model and Code Transformer. The implementation of CodeTransformer keeps the same to Zügner et al. For more details of parameter setting, please refer to the A1 for y3EX.

 Thanks for your careful and valuable comments.

Q1: What is the standard error in Table 2? The prior work does not include them either, thus a significant improvement on this dataset is hard to estimate. That said, the delta improvement of TPTrans seems similar to such of state-of-the-art works to their predecessors.

A1: For the concern about the main result in Table2, we repeat the experiment several times for confidence measure, and get that:

|Model|Python.P|Python.R|Python.F1|Ruby.P|Ruby.R|Ruby.F1|
|----|----|----|----|----|----|----|
|TPTrans|38.76 $\pm$ 0.49|34.66 $\pm$ 0.35|36.59 $\pm$ 0.02|32.40 $\pm$ 0.84|27.63 $\pm$ 0.78|29.82 $\pm$ 0.79|

|Model|JS.P|JS.R|JS.F1|Go.P|Go.R|Go.F1|
|----|----|----|----|----|----|----|
|TPTrans|33.83 $\pm$ 0.55|28.37 $\pm$ 0.54|30.86 $\pm$ 0.53|55.79 $\pm$ 1.3|51.05 $\pm$ 1.3|53.30 $\pm$ 0.83|

We further compare the improvement of the SOTA(Code Transformer) against the previous model(Here we pick up XLNet for comparison), 
and the gain of our proposed model over Code Transformer.

The gain of Code Transformer over XLnet:

|Model|Python.F1|Ruby.F1|JS.F1|Go.F1|
|----|----|----|----|----|
|CT minus XLNet|0.5|-0.19|2.38|1.73|

And the gain of our model over Code Transformer:

|Model|Python.F1|Ruby.F1|JS.F1|Go.F1|
|----|----|----|----|----|
|TPTrans minus CT|1.6|2.27|-1.27|1.97|

As mentioned in A2, the code transformer is designed based XLNet and enhanced with multiply structural relations of code AST.
We can find that in most datasets, compared the improvement of code transformer over XLNet, our model obtains more remarkable improvement over code transformer, although Code Transformer is well designed for code and XLNet is not.

Q2: Addition of domain-specific inductive bias (like source code paths) usually compensates for model capacity. What are the sizes of all evaluated models in Table 2? 

A2: Please refer to the A1 for Reviewer y3EX 

 Q4: L203: code summarization by specifically generating a function's name has been considered a dubious task for some time now, primarily due to both high ambiguity and repetitiveness in function names in practice. Consequently, he paper would be strengthened by including other tasks, such as defect detection.

A4: Appreciate your criticism. As we all know, the method naming task is a popular and useful benchmark for code representation learning. Many works pick this task to show the ability of code representation learning[1][4]. In our paper, we experiment on four datasets across different languages to show our approaches' effectiveness. We're indeed happy to explore other tasks such as classification and retrieval, and we also have confidence in our proposed model.


Q5: The explanation of the JavaScript results relative to other languages feels a bit handwavy. This is not a major concern, but illustrating this concretely, for instance with examples of outliers or representative samples, would help.

A5: Please refer to the A1 for Reviewer ZRpb.

Q6: I did not find any record of the number of layers used for the models. Please add this to p7.

A6: Please refer to L250-L259(Hyperparameters). And also please refer to the A1 for more details.

Q7: Please significantly improve the writing of the paper throughout. The "Efficiency" section lacks a lot of details, some of which seem rather important (e.g. L242), but are relegated to the appendix. And other smaller issues and Typos.

A7: We appreciate your kind reminder. After overall consideration, we will carefully try to fix them in the later version.

[4]Code2vec: Learning Distributed Representations of Code


 Thanks for your careful and valuable comments.

Q1: The contributions stated are overly broad and grossly misleading. The first contribution We show the feasibility of embedding sequential information as an inductive bias for the attention module of Transformers and are the first ones to do so to our best knowledge. (Line 77-78) is not at all a contribution. As stated by the authors themselves there are several previous works like GREAT, Zugner et. al 2021, etc. that have already done this.

A1: The sequential information here means the encoding for a vector sequence. For example, The $Path(x_i\rightarrow x_j)$ in Eq.5 is a vector list consisted with [$n_0$,$n_1$,$n_2$,...,$n_m$], where $m$ is the path length and each $n$ is the embedding vector for each node which is looked up from the embedding matrix table of node type. We use a bi-directional GRU to encode the input sequence and get the final state of the path. After that, we integrate this sequential information into Transformer as an inductive bias. 

Unlike our model, the Great biases the attention vias manually designed structural edges and Code Transformer counts multiple distances to leverage code structure. The baselines convert one edge or distance into a embedding vector as a bias for the attention module. However, the path's different node combinations contain plenty of structure information, which is overlooked by encoding distances. To verify the benefit of learning from paths than distance, we also conduct ablation studies presented at L326-L340. Please refer to it for more details.

 Thanks for your careful and valuable comments.

Q1: It is not especially clear why performance on Javascript is not as strong. I would recommend the authors double check this setting and see if there is some issue? It is not such a major issue if the model does not outperform in all settings, but the explanation that Javascript is somehow less structure dependent would likely need more justification.

A1: We appreciate your concern about the exception of the Javascript dataset.
From Table2, we can find that the benefit of encoding relative path for Javascript dataset is not as powerful as the performances in other datasets (while the improvement over vanilla Transformer is still significant, see Table3).
To investigate this phenomenon, we try to count the mean and variance of the length of all relative paths across four different datasets and get that:

| Python | Mean | Var |  Ruby | Mean | Var |JS | Mean | Var| Go | Mean | Var|
| ----- | ----- | ----- | ----- | ----- | -----|----- | ----- | -----| ----- | ----- | ----- |
| Train |11.96 | 24.61| Train |9.29 | 17.62|Train |14.59 |41.46| Train |11.18 | 20.16|
| Valid |11.81 |23.64| Valid |9.76 |18.84 | Valid |14.40| 40.02| Valid |11.56 | 22.40 |
| Test |12.05| 25.36| Test |9.60 |18.82| Test |14.57| 40.59| Test |10.68| 18.83|

In this table, we can find that the path length of Javascript is not only longer but also has a significantly bigger variance than other datasets. So we assume that due to such property of the JS dataset, it is harder to extract useful structural information from the path than other datasets. In addition, we are also surprised to find that the length and variance of paths of the Ruby dataset are smaller than other datasets. Meanwhile, the gain of TPTrans over baselines on Ruby datasets are more significant than others(please refer to A3 for details). 

But unfortunately, as to the outlier of path length and variance of JS languages, we have not yet reached an accurate conclusion. 
One possible line of reasoning is that the inherent property of Js language results in a much complicated syntax tree. 
We also suspect that this is due to the internal design of the language parser, and perhaps a better parser might alleviate this problem.

 Thanks for your careful and valuable comments.

Q1:The primary concern is a fairly straightforward flaw in the comparison with other models: TPTrans adds a non-trivial number of parameters through its various RNNs, but apparently does not account for this discrepancy in model capacity in its comparisons.

A1: Thanks for your concern. For fair comparisons, we set 1024 hidden dims and 3 layers for Transformer, which keeps a similar setting with the backbone of Code Transformer present in [1]. The additional parameters we introduce for the encoding path come from two parts: the lookup table embedding matrix for nodes and two 1-layer bidirectional GRUs(one for relative path and another for absolute path).

1) The dimension of the embedding matrix of nodes is 64, while for the Python language, the vocab size for nodes is 105 (105 for Ruby and Js, 94 for Go). So the num of parameters is 6720 for Python (6720 for Ruby and JS, 6016 for Go). 

2) The input and output dim of GRUS are 64 and 64*2( *2 for bi-direction), respectively, so the total parameters is 49920.

The total parameters of the vanilla Transformer are about 100M (including the parameters of embedding matrix for word's lookup table). So we can find that the proportion of additional parameters for integrating tree path is only about 0.057%. This shows that the improvement of our methods mainly comes from good design instead of parameters' increase.

Q2: The explanation of the "absolute" path performance, being worse than the relative one only, feels off-base. If the information were redundant, we should not expect a drop in performance by including it. L319 further mentions a potential "conflict" between these sources of information, which, as far as I can tell, is not further substantiated. This all deserves further exploration.

A2: From Table3 presented in Section 5.2(Ablation Study), we conclude that after introducing full of the relative path encoding, the improvement from the absolute path encoding almost diminishes. We feel much sorry for that, because of lacking error bars for Table3, readers maybe deduce a misguided conclusion that a concrete conflict definitely exists between two kinds of paths. On the contrary, actually, the only thing we firmly conclude here is that these two kinds of path encoding methods are not orthogonal, which means overlap exists rather than conflict. 

For further explanation, we show the performance of TPTrans-alpha equipped with a pointer network (we repeat the experiment several times for confidence). Compared to TPTrans, TPTrans-alpha fully equips both the relative and absolute path encodings (Eq.7+Eq.10) (Ps: We are so sorry for only showing the results of Ruby and Js datasets due to tight time. And we will try our best to complete the results of the other two datasets getting enough time.), that is:

|Model|Ruby.P|Ruby.R|Ruby.F1|JS.P|JS.R|JS.F1|
|----|----|----|----|----|----|----|
|TPTrans|32.40+0.84|27.63+0.78|29.82+0.79|33.83+0.55|28.37+0.54|30.86+0.53|
|TPTrans-alpha|33.07+0.86|28.34+0.61|30.52+0.53|33.68+0.34|28.95+0.54|31.14+0.46|


From this table, we conclude that although the improvement of TPTrans-alpha over TPTrans is not consistently significant, the so-called 'conflict is also not a certainty. 

We feel much sorry for any confusion. We will fix the misleading and even somewhat exaggerated explanation at L315-L319 and update the experiment results with error bars for Table3 in the later version.


Q3: Table 5 seems to be missing the ablation for other models. Or did those never use the sentinel pointer setup?

A3: Much sorry for the missing details for the pointer network. Overall, we refer to the source code of Code Transformer released in [1] and add the same pointer network on our model for fair comparisons. The pointer network enhances predictions by pointing at positions and is often presented in many seq2seq models. Concretely, both Code Transformer and our model use the pointer network equipped with the sentinel[2]. At each decoding step, the decoder additionally calculates a dot-product between the target sequence features and the sentinel (presented as learnable vector parameters) and takes the product as the gated probability of whether to copy, which is slightly different from the pointer network present in [3]. We will release our source code for result reproduction and more details.

[1]]Language-Agnostic Representation Learning of Source Code from Structure and Context

[2]Pointer sentinel mixture models

[3]Pointer networks


The paper proposes encoding the relative positions of the source code tokens in the Concrete Syntax Tree (CST) and their absolute positions into the self-attention of a transformer and show the value of learned representations in the downstream task of code summarization on four different languages and compare with related baselines.  I think that the idea of using relative path encoding in the transformer self-attention proposed in the paper is very similar to the one in Zugner et. al 2021, except that instead of AST they use CST. The incorporation of absolute path encoding ( TPTRANS-$\alpha$ model) may be a bit different but the authors show in experiments that the absolute path encodings don't make much difference and in the end, they use the relative path encoding model only.  Though I agree that this difference may result in differences in performance (as shown in results), I personally think that just this change in itself does not suffice to contribute to enough novelty of the work that can lead to acceptance at a conference like NeurIPS. On the contrary, I feel that the approach used in Zugner et. al 2021 that uses both structure and context is more meaningful.  The source code context treated as a text sequence in addition to the structure provided by the tree representations can be quite helpful especially for code segments containing comments. Besides this main point, I have the following comments:

(i) The contributions stated are overly broad and grossly misleading. The first contribution *We show the feasibility of embedding sequential information as an inductive bias for the attention module of Transformers and are the first ones to do so to our best knowledge.* (Line 77-78) is not at all a contribution. As stated by the authors themselves there are several previous works like GREAT, Zugner et. al 2021, etc. that have already done this. 

(ii) What exactly are the x's and z's in the context of the CST? 

(iii) Please include error bars for performances.

(iv) I think it should be W^Q in Eqn (10) instead of W^(V)?

(v) I didn't exactly understand the reasoning for parameter sharing as stated in Lines 228-1229.

(v) Why does the incorporation of absolute path encoding doesn't help? It seems like the authors spend a lot of time justifying their choice and in the end decided by using only the relative path encoding model. 

(vi) The statement made in lines 54-55 is very vague. Please expand on it for better motivation.

 I didn't see any limitations or potential negative impact discussed in the paper. Some suggestions are:

(i) Mention the computational complexity of the model in terms of getting the CST and carrying the whole inference process, especially for large code fragments.

(ii) The approach is language-dependent as the parser that creates the CST has to be dependent on the programming language being considered.

(iii) The splitting of tokens into subtokens relies on using variable naming conventions that may not always be followed by developers making the splitting difficult.

(iv) Blind trust in the code summarization systems based on this approach can be dangerous and hence should be exercised with caution along with a human developer.

Meta-review: This was a controversial paper. On the one hand, the paper's central idea is straightforward and also closely related to multiple recent efforts. On the other hand, a simple idea can be a valuable contribution if established to be effective. While the original version of the paper left some doubts about the method's effectiveness,  the authors have engaged with the reviewers thoroughly during the discussion period and provided compelling additional results. The majority of the reviewers appreciated this, and based on their advice, I am recommending that the paper be accepted as a poster. Please make sure to address the suggestions made in the discussion thread.

Decision: accept