Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes improvements on existing probabilistic models for code that predicts and repairs variable misuses. This is a variant of the task, proposed by Vasic et al. The task takes a dataset of python functions, introduces errors in these functions and makes a classifier that would identify what errors were introduced and effectively reconstruct the original code.

The paper claims to improve state-of-the-art results published by Vasic et al for this task, however the RNN model by Vasic et al was known to be far from optimal when the work was published. Furthermore, that task was evaluated on artificially introduced changes (the original code could contain an error), but it is not clear that the improvements would have any practical effect. In fact, I conjecture that the bug-detector is in fact worse, because the entire dataset is not sufficiently large for millions of parameters and it is not clear that bugs that were originally the dataset were not learned by the better model, making it worse at spotting them. Given the relatively thinner contribution on the rest of the paper, I think this would be a valid question to be addressed to show the effectiveness of the model beyond accuracy on the artificial task.

The paper does a number of contributions to the neural architecture. The most important change precision-wise is to use transformer model instead of RNN (the model used by Vasic et al).  This change is also what makes the work perform as well or better than GGNN-based approaches. The paper then proposes to improve the transformer model by modifying the attention where there are edges. The rest of the contributions seem to be addressing the problem of aster convergence speed.  The other contribution of the paper is by selecting which edges to include and it is also shown to improve convergence speed.

Given that most of the work talks about performance, it would also help if the authors clarify what kind of hardware was used and which optimizer.

Q: Why a larger transformer model was not evaluated?

More minor issues:
“We conjecture that the Transformer learns to infer many of the same connections”. There is no confirmation for this besides similar accuracy, but if this is the case, why would I change the architecture and not just try with initializing the vectors to values corresponding to this knowledge and get faster convergence?
page 3, “where q and k correspond to the query and key vectors as described above,”. It seems it is q_i and k_j?

Update after the rebuttal:

 - I thank the authors for running additional experiments on a short notice. I have some reservations about their correctness though (we still do not know if there were bugs fixed in these commits, their number is very low and the authors seem to have cherrypicked the numbers to show - their first updated revision had recall at 20%, then decided to show it at 10%, the RNN baseline is actually having the highest recall of all although it is not highlighted). I am not sure that data cleaning of this small evaluation sample will not show a different picture.

 - I actually increase my score a bit (I was torn in the beginning), because this is one of the first papers to run transformer model on code. I still think the actual contributions of the paper are minor.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Strength:
-- Interesting problem
--The paper is well written and easy to follow
-- The proposed approach seems very effective

Weakness:
-- the novelty of the proposed is marginal
-- Some of the claims are not right in the paper

This paper studied learning the representations of source codes by combining sequential-based approaches (RNN, Transformers) and graph neural network to model both the local and global dependency between the tokens. Experimental results on both synthetic and real-world datasets prove the effectiveness of the proposed approach.

Overall, the paper is well written and easy to follow. However, the novelty of the proposed technique seems to be marginal to me. Some of the claims in the paper are not right. In the abstract, the authors said the graph neural network is more local-based while the transformer is more global-based. The essential difference between the two approaches lie in the way of constructing the graphs since transformer used the fully-connected graph (more local dependency) while graph neural networks usually capture the long-range dependency. 

And there are actually some existing work that have already explored this idea in the context of natural language understanding, e.g.,
Contextualized Non-local Neural Networks for Sequence Learning. https://arxiv.org/abs/1811.08600
The authors should clarify the difference between these approaches.


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors proposed a new method to model the source code for the bug repairing task. Traditional methods use either a global sequence based model or a local graph based model. The authors proposed a new sandwich model like [RNN GNN RNN]. The experiments show that such simple combination of models significantly improve the localization and repair accuracy. 

The idea is simple, so the technical contribution is a bit low. But the message is clear. The sandwich model can benefit from GNN model that can achieve a higher accuracy at the beginning of training where transformer did a poor job. At the end of training, the sandwich model outperforms both kinds of models.

Here are some detailed comments:
1. It would be interesting to have the complete result from Transformer in Figure 1 and Figure 2 which is missing.
2. The results from GGNN (smaller model) in Figure 1 and Figure 2 seems to be not the same.
3. One major benefit of GNN is its efficient local computation. Some industrial applications have also used GNN for recommendation that can be trained very fast. Why GGNN is so slow in this paper? Is this because of implementation? 

review_title: Official Blind Review #3

Meta-review: The paper investigates hybrid NN architectures to represent programs, involving both local (RNN, Transformer) and global (Gated Graph NN) structures, with the goal of exploiting the program structure while permitting the fast flow of information through the whole program.

The proof of concept for the quality of the representation is the performance on the VarMisuse task (identifying where a variable was replaced by another one, and which variable was the correct one). Other criteria regard the computational cost of training and number of parameters.

Varied architectures, involving fast and local transmission with and without attention mechanisms, are investigated, comparing full graphs and compressed (leaves-only) graphs. The lessons learned concern the trade-off between the architecture of the model, the computational time and the learning curve. It is suggested that the Transformer learns from scratch to connect the tokens as appropriate; and that interleaving RNN and GNN allows for more effective processing, with less message passes and less parameters with improved accuracy.

A first issue raised by the reviewers concerns the computational time (ca 100 hours on P100 GPUs); the authors focus on the performance gain w.r.t. GGNN in terms of computational time (significant) and in terms of epochs. Another concern raised by the reviewers is the moderate originality of the proposed architecture. I strongly recommend that the authors make their architecture public; this is imo the best way to evidence the originality of the proposed solution. 

The authors did a good job in answering the other concerns, in particular concerning the computational time and the choice of the samples. I thus recommend acceptance. 
Decision: Accept (Poster)