Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents a way to incorporate the tree-structure information into a transformer architecture to better learn representation for code. The tree structures are first dispatched as leaf-to-leaf "relative paths" and leaf-to-root "absolute paths", and then embedded and modulated as additional learnable parts of the query, key, and value components within the transformer attention. The authors evaluate their proposal on the task of extreme code summarization -- predicting function names. Results show improvements over 4 previous non-transformer and transformer models on 3 out of 4 programming languages tested. The authors also carry out ablation studies to understand the effect of absolute paths and pointer networks but many important questions remain unanswered. 
main_review: The writing is clear and easy to follow. The paper follows the current trend of using transformers to learn code representation and incorporating richer structures into the originally sequence-based transformer model.

The proposed way for adding relative paths and absolute paths into the transformer attention seems natural but the exact places and terms added are somewhat arbitrary. For example, why the relative-path terms are added at the K part (Eq. 6) and the V part (Eq. 7) but not the Q part? For absolute paths, why the responding terms are only by themselves (Eq. 9) but not with any of the Q, K, V parts?

The paper also needs a clearer description of experimental settings, especially training settings (see Questions below), to fully make sense of the results.

The main results (Table 1) show a majority but not definitive gain. Even for where it gains, the gains are modest. 

Additionally, why JS is such an outlier? Besides the under-supported claim that "JS's structure is weak", are there deeper reasons related to the dataset and the proposed model? It would be great if the authors could shed more light on the case.

I appreciate the effort the authors put into the ablation study. However, it also raises questions. Why is adding absolute paths -- more information for the model to rely on, degrades the metrics? Shall we try to add more relative-path terms since the trend in Table 3 suggests more is better? Rather than conclude that absolute paths do not really help, shall we try other ways of adding the terms and/or adding the terms to different places? Table 4 and 5 generally shows the proposal still works well in combination with sequence bias and pointer network, which is good.

In general, I see a solid effort and a sensible way to incorporating tree structures into transformers for learning code representation. However, the exact ways of incorporating paths lack justification, and the improvements in numbers are limited and the unclarity of the setting further decreases the credibility. The significance of the paper could be improved by giving more supported insight into how the tree structure, in the way it is used, is helping or not helping with learning code representation under different scenarios. 

### Pros
- The writing is clear and easy to follow.
- Many potential audiences in learning code representations using structures.
- Natural way of adding tree structures as paths into transformer attention.
- Experiments show the proposal generally improves metrics.

### Cons
- The exact way of modifying attention seems arbitrary and lacks justification from either intuition or ablation study.
- Experimental results show a majority but not definitive gain, and the gains are modest. The no-gain case (JavaScript) is not well-explained.
- The efficacy of the proposed architecture is only evaluated over one training dataset and one task.
- Not too much insight was generated about how and why it works or not for future research.
- The title says "for Code Representation" but the authors only experimented on code summarization. 

### Questions
- Justifications for the seemingly arbitrary places to add relative- and absolute-path terms in Eq. 5-10.

- Main results in Table 2. What are all the data used to train each model? For transformer-based baselines, did the authors train them from scratch, or used a pre-trained model and then fine-tuned over the CSN datasets? This question is related to fair comparison as well as if the models with a large number of parameters will become under-trained or overfitting to a relatively small number of training instances.

- Table 2, Table 5. Why is the trend so different for JavaScript? Besides the under-supported claim that "JS's structure is weak", could it be because the dataset has biases or unusual properties, or the proposed model itself has some "flaws"?

- L320 "If two sub-tokens are too far away ... the relative and absolute path encoding play the same role ..." Can we find more evidence to justify or falsify this? Local pairs can still benefit from absolute paths. Additionally, if there is any trimming of the paths (for example, due to input-length constraints), then absolute paths can keep information that would be otherwise discarded from (the longer) relative paths.

### Other points
- L278 "... since the structure of Javascript is much weak, ..." 
- L339 "... the structure of Javascript is weaker than other languages, ..." I don't find references or supporting evidence for this claim. What makes the authors conclude that JS is less structured?
limitations_and_societal_impact: The paper could benefit from a dedicated discussion about limitations.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper introduces a new neural network architecture for code, TPTrans (Tree Path Transformer), by modifying the attention calculation in the Transformer architecture to use information about paths in the syntax trees of programs. The modification works by computing path embeddings between pairs of leaf nodes ("relative path embeddings") and between leaf nodes and the root (forming nodes' "absolute path embeddings"), and incorporating these into the Transformer's attention module. The paper evaluates TPTrans, including baselines and variants, on code summarization -- the task of predicting a function's name given its body -- for four different programming languages. The main experiments reveal the value of including embeddings of sequences (the paths) in the attention calculation over baselines, and subsequent experimentation compares the impact of relative and absolute paths, and the effect of pointer network outputs on the model.
main_review: The paper's core contribution is the introduction of TPTrans, a method of incorporating paths from a syntax tree into the attention module of the Transformer architecture. By evaluating this approach with two kinds of paths on code summarization, and by finding it outperforms baseline methods, the paper indeed shows the feasibility of incorporating sequential information into transformer attention as an inductive bias, as claimed.

This is an important area of study, as machine learning for source code is an ever increasingly important domain, yet techniques from natural language processing such as the transformer architecture do not take advantage of properties of source code. Identifying models that learn better representations of source code and fully take advantage of the information and structure present in source code, including its semantics, is a critical challenge. This paper is a step toward this, identifying a method of incorporating program structure for better learned representations of code.

The method proposed for incorporating relative and absolute path encodings into Transformer attention is conceptually simple, and is explained clearly in Section 3.1. The conceptual simplicity of the proposal is a strength of the paper, as is the clarity of the description in this section. The main result, showing on most tasks that the relative path variant TPTrans outperform all baselines including the absolute path variant TPTrans-$\alpha$, is another strength.

The specific choices of absolute path for each node and relative path for each node-pair makes the method specific to code. These paths are embedded using a GRU over the sequence representation of the path. Beyond these choices, the approach is more general, applicable for incorporating any pairwise and element-wise node information about a sequence

Considered individually these components are not novel. GRUs are a standard way of encoding sequential information, the transformer is a standard sequence-processing architecture, the use of absolute position encodings is present in the original transformer and relative position encodings are introduced in [1]. The paper's strength is in it's combination of these components, demonstrating that sequence information is suitable for both relative and absolute position encodings, and that in the context of source code choosing these position encodings improves performance on an important benchmark task. That said, since TPTrans is primarily a novel combination of existing architecture components and an extension of the transformer attention module similar to ways it has already been extended, the size and significance of the contribution is limited.

Evaluations are done on the task of code summarization, which is a sensible task for evaluating performance on representation learning of source code. Four programming languages are considered, Python, Javascript, Go, and Ruby, and TPTrans outperforms the baselines considered on all languages but Javascript. One area for improvement in the experimental setup is clarifying the axes along which the models are comparable. Do they have comparable parameter counts? If the increase in performance were only due to an increase in parameter count, which could also be achieved by scaling up existing models, that would decrease the significance of the result.

A weakness of the experiments is they lack error bars or any measure of confidence or variation. Independent of this, the experimental evidence is also somewhat thin, as a single task is used to demonstrate improvements on the quite broad goal of representation learning.

One potential way to improve the paper would be to demonstrate why the path embeddings are useful. For example, the model admits inspecting which path embeddings are attended to. Do these exhibit any patterns or have any particular semantic meaning?

As an aside, the paper unfortunately contains many grammatical and typographic errors. Usually these do not impede clarity too much, and in my review score I do not count them against the paper. Some of them do make the paper more difficult to read, however (e.g. lines 269-270), but in all cases I think I was able to determine the intent of the sentence.

[1] Self-attention with relative position representations.
limitations_and_societal_impact: A technical limitation of the work is: The method relies on having a parse for the program, which limits the approach to well formed code and languages supported by tree-sitter. The baseline Transformer does not have this limitation. The paper does not address potential societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work continues on earlier advances in how positional and relational encoding is used by the self attention primitive of Transformers, in particular in the context of source-code understanding. Prior work has biased self attention to encode relative relational information between inputs (e.g., edges or distance on some graph), and to separate the self-attention term due to position from that due to input.

For the former, TPTrans runs the path between any two tokens (specifically, the path in the code Parse Tree between two leaves -- the authors call this the "relative path") through an RNN, and uses the resulting embedding to bias the self attention between those tokens.

For the latter, TPTrans-$\alpha$ runs the path from the parse-tree root to a token through an RNN (the "absolute path"), and uses that to construct a separate positional self-attention term.

The results show that both TPTrans and TPTrans-$\alpha$ outperform prior work (most notably, Code Transformer), but TPTrans seems to dominate TPTrans-$\alpha$ with a smaller set of trainable parameters.
main_review: # Overall

* Originality: This is an interesting twist to prior advances (GREAT and Code Transformer, and perhaps TUPE). It's a natural twist, but a non-trivial one.

* Quality: The quality of the evaluation was mostly good, with small issues (e.g., the lack of confidence intervals). I particularly appreciated the authors showing TPTrans-$\alpha$, even though it was largely a negative result. 

* Clarity: The presentation was good, especially Section 3. It gave a very precise and contextualized description of what's new and what isn't, which is greatly appreciated. The experimental section was harder to understand, and I don't have a good understanding of what precise baselines were run, and whether the comparisons are fair.

* Significance: This is still a new area, and the proposal here is likely to influence future Transformer-like ML architectures for code  or other structured inputs. As such, it's a non-trivial new contribution.


# Other Comments

I really appreciated the detailed contrast between what had been done before (Section 3.0) and what TPTrans is introducing (Section 3.1). This helps distill the differences and novel contributions, especially contextualizing them to provide motivation. In this effort, it would have been even more helpful to the reader to use the same notation where it wouldn't affect semantics. For examples, there's no reason why $a_i$ and $a_j$ in Equations 8 and 9 can't be denoted with $p_i$ and $p_j$ from Equation 3. Simialrly for $W_a^{Q/K}$ and $U^{Q/K}$.

It would be helpful to provide confidence intervals for your results. It's not clear if your improvements are statistically significant otherwise.

I found some parallels in absolute/relative paths to AnyCodeGen's evolution of code2seq (Alon, Uri, Roy Sadaka, Omer Levy, and Eran Yahav. "Structural language models of code." In International Conference on Machine Learning, pp. 245-256. PMLR, 2020.) I found it rather odd that it was not cited and mentioned.

Although removing the layer index from equations (e.g., 6, 7, ...) is helpful, I found it problematic for Equation 5. Please add a precise definition of $\mathit{Path}_{x_i\rightarrow x_j}$. Does this change for every layer, or is it dependent solely on the initial embeddings of node types?

# Smaller Issues

* Your Hellendoorn 2019 paper was at ICLR2020, not ICLR2019.

* Line 216: Table 4 should be Table 1.

* Line 221: Please explain equal interval sampling. How do you choose the interval? Is it fixed or chosen to provide a sample of length 32?

* Lines 245--249: Pointer decoders have been used in the relevant literature for different purposes. Copying (which is what you do here) or to identify positions for an edit (e.g., in HOPPITY and Hellendoorn et al 2020). Please clarify that you're using pointers for copying, not as part of the task definition. Also, please explain how you use the pointer network exactly. The information you provide does not allow the reader to reproduce your architecture.

* Lines 260--271: The text here is very hard to parse and to understand how exactly you have constructed your baselines.

* References: I found the order of citations very distracting. It was very hard to locate a citation, given that the references are provided in the order of appearance rather than alphabetical order.

limitations_and_societal_impact: No concerns.
review_title: null

Meta-review: The reviewers agreed that even though the general aspects of using recurrent units for embedding paths, and using relative and absolute paths for embedding source code are not completely novel, the paper presents a nice novel combinations of these ideas. There were many clarity issues in the writing, but the response helped clarify many of them. One of the major concerns that came up was that evaluation did not clearly show that the proposed architecture was consistently better, and the evaluation was only performed on a single task of code summarization. The new results for code classification task are interesting, but it is not clear if TPTrans performs better than stronger baselines. Finally, there were also some concerns regarding better understanding of the benefits and limitations of the approach, and how incorporating paths might improve (or worsen) the performance of the model.
Decision: Accept (Poster)