Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed a general method called PLUR for solving ML4code problems. PLUR is based on mapping a graph to a sequence of tokens and pointers. The author showed that 15 different ML4Code tasks can be casted in the form of PLUR. Experiments show that PLUR achieves near or above state-of-the-art results for nearly all tasks. In addition, the unified framework enables the author to study a few important research questions, including 1) performance of different encoders (GREAT, Transformer, GGNN), 2) importance of the pointer mechanism, 3) effect of different graph representations, 4) effect of multi-task learning. To be more specific, the author showed that 1) GREAT and Transformer encoders outperform GGNN, 2) multi-task learning with PLUR improves the performance in most datasets, 3) copy mechanism and pointer network improves the performance.

main_review: 
- Originality

Although the PLUR framework is based on the Graph2ToCoPo formulation, it is original because it is the first unified framework for solving multiple types of ML4Code tasks. In addition, the author answers several important research questions and demonstrated that PLUR can be used for multi-task learning, which is also novel.

- Technical Quality

The paper is technically strong. The author evaluated PLUR on 15 different datasets and conducted ablation study to answer the research questions. In addition, the multi-tasking results proved the advantage of the PLUR formulation.


- Clarity

The paper is well-written.

- Significance

The study conducted in the paper is significant and will boost the community's understanding of how to build a deep learning model for code data.


limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a unified encoder-decoder framework, PLUR, that can cover 15 recently published source code tasks (e.g. code classification, code repair, function name prediction). The encoder generalizes sequence-based modeling (e.g. Transformer) and relation/graph-aware modeling. The decoder includes copying and pointer mechanisms. Their unified model, PLUR, outperforms task-specific models in previous literature, and also enables multi-task learning. The authors also study important research questions such as Transformer vs GNN. I think this paper deserves acceptance to NeurIPS.
main_review: Reasons to Accept:
- Various tasks and models have been proposed in the field of ML4Code, but the format is not standardized and hence it has been difficult to make systematic comparison of models (e.g. what model choices really matter in code understanding?). This work provides a nice step to unify source code tasks and modeling framework. Under this modeling framework, one can study research questions in a more controlled and reproducible way
- The unified model also facilitates multi-task learning for source code, opening up a new research avenue.
- The author also shows that their general model can outperform existing models designed for specific tasks, which is a notable insight for the ML4Code field.
- The paper is clear and well-written. The experiments are extensive.

Weakness/questions:
- PLUR covers diverse tasks, but there are some common tasks not included (e.g. code completion, C code repair like DeepFix). Is it because they do not fit the PLUG framework? It'd be great to discuss a bit more 1) what kinds of tasks are not included or do not fit in the current framework, and 2) if there is a prospect to cover more tasks by extending the PLUG framework.
- Were there any technical challenges that needed to be addressed when developing the unified framework? If there were, it might be worthwhile to mention these insights in the paper, as they may help future work to build on the proposed framework.

Typos/grammar:
- L373: "one risk is that"
limitations_and_societal_impact: The authors discuss limitations and broader impact of their work in L373-379.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper presents a general architecture which unifies fifteen previously
separate ML4Code tasks and achieves state-of-the-art or near state-of-the-art
results on them.

The inputs of the architecture are graphs where the nodes have labels and
positions, and both the nodes and the edges have types. The outputs follow the
ToCoPo output formulation where each output element can be a token, a copy, or a
pointer. This makes the architecture flexible enough to accomodate most ML4Code
tasks.

main_review: I believe that the paper is an excellent and important contribution to ML4Code
as it unifies previously separate tasks utilizing different algorithms into a
single consistent state-of-the-art framework.

The framework will be open-sourced and can be the base of future research: new
models can be tested and ablated across the diverse 15 tasks, and new tasks can
be incorporated into the framework.

This is demonstrated in the paper with various ablation studies which give
interesting, novel insights. The authors compare different encoders across all
the tasks, the usefulness of pointers and copies across two tasks, multi-task
learning, and different graph representations for code2seq.

Overall I really like the paper, and only have a few concerns/questions:
1. The reported values for grid search in Appendix C
   suggest |learning_rate|\*|dropout_rate|\*|batch_size_per_device| = 3\*2\*2 = 12
   hyperparameter combinations for Table 4, and 2*2 = 4 combinations for Table
   5, yet the paper reports 24 combinations.
2. The hyperparameters were set separately for each task with grid search. The
   values tried by grid search are reported in Appendix C, but the concrete
   hyperparameters are not.
   - It would be good to see some summary about how different the
     hyperparameters are across the tasks. For example, it would be instructive
     to know whether dropout is useful for all the tasks, or there are
     exceptions.
   - Are there hyperparameters for which the architecture performs well across
     all tasks?
3. I think that the cells should be bolded differently in Table 2 (d). The
   purpose of that table is to compare between using pointers and copy and not
   using them, so the best result in each column should be bolded in my opinion.
   For example, the bolding of the cell CuBERT-VMR/GREAT/No pointer is
   particularly misleading, as that is the only bolded cell in the column and it
   suggests to me that No pointer is better than Full at a first glance.
4. Maybe the Discussion would be better as the Conclusion?
5. There is a small typo in line 102: "See and details of the software
   architecture"

# Update

I would like to thank the authors for their insightful and transparent answers
and their corrections to the paper. I'm increasing my score to a clear accept.

limitations_and_societal_impact: The authors address the limitations and the societal impact appropriately .
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a framework, called PLUR to unify a number of code-related tasks into a general form. Specifically, it pairs a graph encoder with a Transformer-style decoder augmented with pointers and a copy mechanism as a unified form and uses GREAT as the graph encoder.  On 15 different tasks e.g., function name prediction, program repair, variable misuse, PLUR has achieved at or above state-of-the-art results. Furthermore, PLUR also investigates multi-task learning by the proposed framework and yields promising results.
main_review: This paper attempts to unify many ML4Code tasks into a general formulation, which is interesting and meaningful. However, it is also very complicated since different code task has its characteristic and needs customized network architecture to achieve promising performance. This paper utilizes the transformer as the base modelling architecture combing with GREAT for learning and it has performed a large number of experiments over 15 tasks to illustrate the effectiveness of the proposed framework compared with the original paper. The improvements on some tasks e.g., Hoppity, CuBERT-VMR are significant.


limitations_and_societal_impact: - The architecture of the network is not novel, GREAT and TOCOPO have already proved the effectiveness in the related papers.

- Missing the important baselines CodeBERT and GraphCodeBERT for comparison since CodeBERT has already achieved state-of-the-art performance on many ML4Code tasks. In the chosen tasks, PLUR has already compared with CuBERT, to make the approach solid, it is better to compare with the latest techniques i.e., CodeBERT and GraphCodeBERT.

- The selected tasks can roughly be categorized into four groups i.e.,  function classification (ManySStuBs4J, CuBERT), function name prediction (code2seq, ogb-code, convattn), code summary (funcom) and bug repair (VarMisuseH, Hoppity),  whether the selected tasks are diverse and fair enough to cover all tasks? 

- It seems pointer and copy mechanisms just severs for VarMisuse task,  is it necessary to equip in the decoder?
review_title: null

Meta-review: This paper presents a unifying, graph-based ML architecture for understanding and editing source code. The method, called PLUR, takes the general approach of mapping a graph to a sequence of tokens and pointers, a form into which many ML4Code tasks can be cast. The authors show that PLUR admits 15 recently published, diverse tasks, enabling a single model architecture to achieve near or above state-of-the-art results on them. PLUR is extensible to new problems beyond these initial 15. Significantly, its unifying framework enables approaches like multi-task learning and experiments on the importance of different modelling choices for ML4Code. These experiments, several of which are conducted in the present work, should guide the community toward robust improvements in ML4Code model development.

Reviewers are in consensus that the paper is above the NeurIPS acceptance threshold, with one championing it as a top-50% submission. They agree that it is technically strong, clear, and significant, as do I. The variety of tasks covered by the experiments is persuasive. Two reviewers mentioned that the framework does not exactly innovate on the modelling side, since it is based on the existing Graph2ToCoPo formulation; however, its unification of previously disconnected ML4Code approaches simplifies this important problem space in an insightful, clarifying manner that will almost certainly impact the field going forward. I recommend the paper for acceptance.
Decision: Accept (Spotlight)