Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

The authors develop a Transformer model for language-agnostic code summarization. The Transformer is provided both sequential tokenized code and a parsed AST as inputs, and tasked with generating the name of the corresponding code function as output. The authors show that the inclusion of this added structural AST information improves performance on the task, and also improves the cross-language transfer learning abilities of the model. They demonstrate state-of-the-art performance on this task when testing against comparable architectures and datasets. This reviewer believes that the paper is deserving of acceptance to ICLR 2021.

Reasons to Accept:
- The authors take pains to fairly compare their model to prior works, including testing on two separate datasets, and ablating their Pointer Network decoder.
- The authors achieve state-of-the-art results for the code-summarization task on both the CodeSearchNet dataset (Python, Javascript, Ruby, Go) and the Java-small dataset.
- The authors demonstrate the surprising extent to which structural information can help with cross-language generalization, particularly for undersampled languages.
- The authors visualize the embeddings learned by the Transformer, and compare them across languages.

General Issues to resolve:
- Given the centrality of AST code representations to this paper, there should be some figure showing a sample code snippet alongside its AST in full. This can go in the appendix if need be.
- Clarify how Ancestor (and Sibling) distance are defined when one node is not a direct ancestor (or sibling) of another node.
- Add a more detailed explanation to Figure 3, particularly regarding the "Attention Transformer" subfigure.

Grammar/Syntax Issues:
- Bottom of Page 2: "learn structure (syntax) and *semantics* from scratch"
- Bottom of Page 4: $r_{v_1 \to v_3} = -2$  should be  $r_{v_3 \to v_1} = -2$
- Bottom of Page 4: "PPR is a well-studied proximity measure *which* has"
- Middle of Page 13: ['set', 'bottom', 'height'] should be rendered using backticks
- Middle of Page 13: '\n' should be rendered using backticks
- Middle of Page 15: "the child *has* a distance"
- Numbering of Tables goes from 6 to 8, skipping 7

review_title: Review of "Language-Agnostic Representation Learning of Source Code from Structure and Context"

final_score: rating: 7: Good paper, accept
review_content: There is not much not to like about this paper as it has a simple idea to extend the transformer model. The paper proposed to not only take positional information of each token, but to add additional structural information about distances of tokens in the abstract syntax tree. This positional information gives an edge of this model on several code summarization datasets. The simplicity of the proposed model (assuming it is released by the authors) puts the work in the state-of-the-art category in machine learning for code.

The paper is also easy to follow and the contribution, while small, is clear and well explained. The related work is also thoroughly covered. While it looks standard, the idea seems to deliver well in the evaluation results. This is also the first work that shows cross-language improvements for programming language models.

In terms of writing, my main complaint is that the paper can better relate to existing works. For example, “ours without structure” seems to be equivalent to a “plain” transformer with a pointer network in the decoders and using relative distances. It would also help if Table 2 includes a row without the pointer network called a transformer model.

In the light of this, it also looks like the comparison with the GREAT model is not quite fair. One possibility to fix this is to encode some program analysis edges at least for one language. The program analysis edges are in fact not difficult to build, especially if the authors want to claim they did this comparison correctly.

As another possible improvement, I find the terms “structure” and “context” confusing and I would suggest the authors use something that directly refers to trees and sequences.

review_title: Simple extension of transformers for code that works well.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper wants to combine sequence (called Context) and AST (called Structure) representations of source code in a Transformer encoder model. For this, it makes clever use of relative position embedding of Transformer-XL. Different pairwise relations based on ASTs and sequence ordering are computed and each of them is encoded as a separate distance matrix, with its own learnable projection. This model is evaluated on the task of code summarization and compared against code2seq and GREAT models, and against different configurations of the proposed model, called Code Transformer. The results show that Code Transformer achieves results better than these models. This evaluation is conducted on five languages. A separate evaluation with the model trained on all the languages together is also performed. This multi-lingual model outperforms the mono-lingual models.

**Design** 
The paper considers four types of paths in the ASTs and quantifies the path lengths and directions. The ancestor and sibling distances are also defined over internal (non-terminal) nodes of the AST. But the input to the model is a sequence of sub-tokens (grouped by tokens). As discussed in appendix A.2 and A.3, the leaf nodes are mapped to some internal nodes whose types are concatenated to the token representations. Thus, the internal nodes themselves are not part of the input. From section 3.1, it appears as if all the internal nodes will be somehow encoded. This does not seem to be the case and the non-leaf-to-leaf paths are perhaps projected only one certain internal nodes. Please clarify this.

A separate projection matrix $W_r^{(s)}$ is used for each relation r. Is the positional encoding $\phi$ shared between them?

The paper calls the proposed model as Code Transformer. Technically, it is only the encoder. It would be good to make this clear.

In the appendix, the paper states that a token span may have overlap among its ancestors. A concrete example would help.

**Language-agnostic representation**
I find the use of the term "language-agnostic representation" confusing. The proposed method uses features computed from ASTs which are language dependent and even dependent on the parser. If a different parser for the same language is used, the features would change (because even if the grammars might be equivalent they may not be identical). I think the authors want to emphasize that they don't use program analysis information such as control and data flow. It would be more accurate to state that as such.

**Learned representations**
The paper presents nice visualizations, and examples of methods and their embeddings. However, it is important to highlight in the paper that these representations are learned in the context of a specific supervised task, unlike unsupervised representations, eg, in "Learning and evaluating contextual embedding of source code" (ICML'20). Note that those can also be used for code summarization with a decoder similar to this paper or by using other decoding methods such as "Blank language models" (https://arxiv.org/abs/2002.03079). 

The paper talks about pre-training on Java-large and Java-medium datasets and fine-tuning on Java-small. What is the pre-training objective? If masking (similar to BERT) is involved, then how are the distance matrices computed and masked?

**Baselines**
The GREAT model uses semantic and syntactic relations between tokens (called leaves-only graph). However, this paper uses the ASTs, which is different from the leaves-only representation. First, how exactly the ASTs are used with GREAT is not explained. Second, the results do not imply that using Structure, as defined in this paper, can outperform edge-level representation and relational attention for the edge types considered in the GREAT paper.

Another baseline considered is Fernandes et al. (2019) on the Java-small dataset. The performance improvement over this baseline is marginal (F1 of 51.4 vs 51.83 with pre-training).


review_title: Interesting design, need better experimental comparison

final_score: rating: 6: Marginally above acceptance threshold
review_content: #### Summary
This paper presents a representation of source code based on the AST. By adding positional relational information, such as shortest path length, and ancestor distance the transformers learn to better represent code, without language specific features. On the code summarization task, the model improves on baselines, while training among many languages further improves results. The representations learned seem to share semantic similarities among languages.

#### Overview
(+) State-of-the-art results for single/multiple language method naming task.

(-) Limited novelty on the machine learning modelling (paper adopts a pre-existing formulation of relational transformers from Dai _et al._ (2019) and Yang _et al._ (2019)).

(-) Only one task evaluated. Does this method offer a language-agnostic representation that generalizes to other tasks on source code?

~~(-) Some additional evaluation would be useful to discern how the proposed model and others compare.~~ [Addressed at response]

#### Comments
* It is not clear to which extent the representation used here is language agnostic vs. the model is able to learn language-agnostic features: GitHub Semantic (the package used to extract the ASTs in this work) goes into a lot of effort to convert code into a language-agnostic AST format (for its downstream analyses). In that sense, the input AST is mostly language-agnostic, which may let any neural model to be language-agnostic. 
This suggests that other models that could accept Semantic's ASTs (e.g. [a], [Alon 2019a], [Fernandes 2019]) could also act as language-agnostic models, however, none of these are evaluated in the multilingual setting.
* An ablation to study the effects of the four different relations considered (Section 3.1) would be useful to understand how much they improve the model (shortest path length, vs. ancestor distance vs. PPR vs. some combinations)


#### Clarification Questions
Finally, I have a few clarification questions (which I think need to also be clarified in the paper):
* Are the relational encodings an input on all transformer layers or just the first one (like the "standard" BERT positional encoding)?
* The "context" has the "token distance" is this the absolute (BERT-like) or the relative (like Dai _et al._ 2019, Yang _et al._ 2019)
* In 5.2 it's unclear what is the representation of each function. Is it the representation of the masked method name token? (the "encoder output of the method name" is a bit vague, but suggests that)
* Are all AST nodes input elements to the "Structure+Context" model?
* How does GREAT differ from the "context" model? Is it just the relational transformer (scalar bias) formulation, the 1-step AST relations and the absolute positional encodings?

#### Additional References
[a] Kim, Seohyun, et al. "Code Prediction by Feeding Trees to Transformers." arXiv preprint arXiv:2003.13848 (2020). 



review_title: Review

Meta-review: The paper gives an extension of the transformer model that is suited to computing representations of source code. The main difference from transformers is that the model takes in a program's abstract syntax tree (AST) in addition to its sequence representation, and utilizes several pairwise distance measures between AST nodes in the self-attention operation. The model is evaluated on the task of code summarization for 5 different languages and shown to beat two state-of-the-art models. One interesting observation is that a model trained on data from all languages outperforms the monolingual version of the model.

The reviewers generally liked the paper. The technical idea is simple, but the evaluation is substantial and makes a convincing case about setting a new state of the art. The observation about multilingual models is also interesting. While there were a few concerns, many of these were addressed in the authors' responses, and the ones that remain seem minor. Given this, I am recommending acceptance as a poster. Please incorporate the reviewers' comments in the final version. 
Decision: Accept (Poster)