Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a model over sequential structural tree edits, used for program repair based on ASTs. The model itself uses a graph encode and decoder to predict the sequential tree edits. The authors also introduce a method for pretraining the model on existing (non program repair) code data: they delete subtrees of arbitrary size from the code, and predict their reconstruction. The resulting model performs comparably to several other state of the art code repair models on the Patches in the Wild Java repair dataset, but with fewer parameters than several of the best-performing pre-trained models (CodeBERT, CodeT5).

main_review: This is an ambitious paper that tries to pack a lot in: between the main paper and the appendix, the authors introduce a (non-trivial, graph-aware) model; a pre-training regime for the code repair problem; quantitative evaluations against many benchmark models; and qualitative examples of some code repair predictions. I really do admire and appreciate the effort that went into this work, which seems significant!

However, the net effect is of many, many parts to this paper without a convincing demonstration of model novelty or of significant improvement to the SOTA on the provided baseline. I provide more details below: 

Strengths:
1. Important and interesting application. Program repair is clearly useful, and the structured task also bears close relation to other tree and graph structured tasks (knowledge graph and structured schema editing; protein editing; language and grammar correction) that are clearly of interest to many. 
2. Model is structure aware and predicts graph edits - an interesting choice with downstream possibilities. The authors present comparisons both to SOTA models that are not generally graph-aware (code pre-trained versions of BERT, Transformers, and T5) and at least two explicitly graph/AST-based models (GraphCodeBERT , HOPPITY). Unlike all the models evaluated except HOPPITY, the authors model is both structure-aware and predicts explicit edits to graphs, which feels like an interesting feature that may allow it to integrate more easily with other, non-neural systems (like a linter). It also outperforms the HOPPITY baseline by a large margin.
3. Extensive evaluation efforts against many comparable models. The authors compare their model against many SOTA models on two versions of the same dataset; and provide additional ablations on several parts of the model (eg. number of graph encoder heads) in the appendix.

Weaknesses and/or concerns:
1. Model novelty. The model architecture seems very closely based on the HOPPITY baseline: the primary change seems to be to the multi-head graph encoder (which itself uses an existing GIN network for each head). While it’s exciting that this addition improves over the original, the resulting model therefore doesn’t seem to introduce a fundamentally new idea: it also encodes graphs, and predicts edits to them. As a presentation nit, the bulk of the model architecture itself (beyond the ‘multi-head’ change) is described in the appendix.
2. Quantitative evaluations show marginal improvement at best in comparison to the large pre-trained models. I’d further break down some comments on the qualitative results section:
* The authors heavily emphasize that the ‘smaller number of parameters’ of their model in comparison to the best-performing SOTA. This feels marginally interesting -- and the ‘CodeT5-small’ model seems to perform very similarly to the model in this paper, with about the same order of magnitude of parameters. In fact, all of the best performing pre-trained models seem to perform about the same.
* The evaluation omits a comparison to the closely related Yao 2021 model that is cited throughout the paper.
* The pre-training regime, which is presented as a contribution of this paper, does not seem to produce markedly stronger results. Further, for a fair comparison, it might seem valid to apply the same pre-training regime to some of the other models (like HOPPITY; or the Yao 2021 model if that was added.)
* Presentationally, a good deal of space is taken up by discussing a quirk of the particular dataset evaluation chosen (the abstract vs. concrete versions of the dataset.) As it turns out, the performance of all of the better-performing pre-trained models seems to be about the same between these. This is useful knowledge for future users of this dataset, but not particularly enlightening for this paper. It seems like this space might have been better used with evaluation on another code repair dataset or domain. 

3. Qualitative evaluation is intriguing but doesn’t yield clear conclusions. In particular, I’d be looking for evidence that ‘GRAPHIX demonstrates strong inductive biases in learning complex bug-fix patterns’ - or otherwise, some conclusion on the kinds of bugs that GRAPHIX tends to fix (largely syntactic or catchable by a linter? Requiring multiple edits in graph-space? Concentrated on specific kinds of operators?). One possibility is hand-coding the analyzed edits to categorize them and summarizing the results: without that, these examples are interesting but inconclusive.

4. Pre-training regime is not particularly novel. The authors suggest deleting elements of subtrees from existing code and reconstructing them. This is perhaps given more space in the paper than necessary, and offered as a contribution. The significance of this is a matter of reviewer opinion; I did not personally find it sufficiently novel.

summary_of_the_review: In short, while I appreciate the significant effort that went into this paper, I found it comprised of many parts that fell just short of convincing execution. The model architecture presents only minor changes over an existing one; the quantitative evaluations show only minor improvements over the SOTA (and the authors rely heavily on the qualified claim that they are the best performing ‘mid sized model’ to distinguish themselves, a claim only somewhat supported, given the t5-small model); and present inconclusive and hard to interpret qualitative results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new approach to abstract syntax tree-based automatic program repair. The novel technique called deleted-subtree reconstruction is based on dropping parts of the syntax tree and training the model to grow them back. The method is evaluated against edit-based and sequence-based approaches, where it outperforms only the edit-based ones.
main_review: ## Comments
* It is unclear to me what it means that the state of the art was not beaten. I am missing a discussion on why the presented work is still relevant despite that.
* The anecdotal examples are not compared to the kind of results other approaches give, so it is unclear what to make of them. 
* Repeatedly, the claim is made that the model is more than 10x smaller than “current large scale sequence models” but out of the evaluated baselines only “BART” is that much larger. The presented approach is even outperformed by CodeT5-small, which is less than 2x as large.
* I would have liked a short discussion on how accurate you think it is to detect bug fixes by filtering for “fix”, “bug”, “error” and ”exception” (page 6).
* “To build the final dataset, the authors extracted about 10M GitHub commits whose messages textually match one of these (“fix”, “bug”, “error”, and “exception”) patterns.” -- Remove “,” before the “and” (page 6)
* What is the reasoning behind having exactly 600 nodes (page 7)?
* Table 1 might benefit from a visual separation between edit based and sequence based approaches, maybe using color?
* “Additionally, we notice that while the pre-training provides an additional 10% relative gain on the small subsets, GRAPHIX-P does not achieve similar gains on the medium set, which is inherently longer and harder.” Seems a bit euphemistic as the results got slightly worse.

summary_of_the_review: ## Strengths:
* very well written and clearly structured text
* good motivating example
* interesting approach in scope of the conference
* gives a concise intro to previous work and ties back throughout the paper
* achieves respectable performance at comparable small model size

## Weaknesses:
* underperforms in terms of the state of the art
* anecdotal examples offer very limited insights in current form
* approach mostly orchestrates existing architectural concepts with little actual novelty
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents GRAPHIX, a graph edit model for program repair. The work is directly related to Hoppity (Dinella et. al. 2020) which proposed using a sequence of graph edit for program repair. GRAPHIX employs multi-head graph encoder which improves upon Hoppity in terms of accuracy and complexity. Notably GRAPHIX is able to learn longer edit sequence and thus work on more program repair samples. The work has also proposed a pre-training task to improve model performance. Empirically the authors evaluated GRAPHIX on the *Patches in the Wild* Java bug-fix benchmark. It outperforms various baselines without pre-training. With pre-training, GRAPHIX-P stays roughly on par despite having much smaller model.
main_review: ### Pros

- The paper tackles the task of program repair, which is relatively new and can have practical impact. Although solving a very hard problem, the work is able to outperform various baselines and have on-par performance with very large models.
- The proposed architecture is able to generate longer graph edits which can inherently fix more programs.
- Empirical study covers many recent works and is fairly comprehensive.
- The anecdotal examples as well as the examples provided in the Appendix are indeed interesting.

### Cons

Major

A. Although claimed to be a novel pre-training task, it is not THAT novel. Moreover, although "masking tokens" is natural in pre-training sequence encoders, "masking sub-trees" on AST is essentially just asking the model to do "missing code prediction", encompassing only a small subset of general program repair samples. The pre-training task would be far more "novel" if more thoughts has been put into generating the pre-training dataset. For example, how to generate data for pre-training other edits such as copy, remove, and update.

B. Empirically, pre-training is costly (according to the Appendix) but not giving a lot of performance gain (<2%). The problem is much worse on medium sized data. Here are two questions:
	 1. Why is pre-training only done on "sub-trees between 2 and 6 descendants"? How about increasing the size in order to cover longer edit sequences during pre-training? Is it possible to add such an experiment?
	 2. (Related to A) Is it because the pre-training task only asks the model to predict a sequence of addition operations? Can we add more diverse tasks to let the model to predict a variety of edits?

C. Multi-head graph encoder does not sound too novel to me.


Other Issues

. Miss related works: 

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

Learning semantic program embeddings with graph interval neural network

. The last paragraph in Section 4.1 shows that there is a single-head "base" model, which is not mentioned in any experiment in the main paper. I found the "base" in Appendix but it would be better if you move this description into Appendix as it is not that relevant. Alternatively you can also put the "base"  model and its performance inside Table 1/2.

. Listing 2 Top: the code is not valid if the highlighted green part is removed. What is actually the before and after of that code snippet?

. The paper claims "these observations suggest that the code abstraction may not be necessary". Despite finding that the argument is pretty weak, I also don't think "code abstraction is necessary" in the first place. For example, with abstraction one would not find the fix for Listing 3 (bottom example).

### Nitpicks

- Highlight colors (green and red) used in code snippets are too intense. Can be toned down for ease of read
- Listing 2 Bottom: trailing whitespace is also included in the highlight.
- Table 1 and 2 are hard to decode. Maybe try separating them into multiple tables (with pre-train and without pre-train).
summary_of_the_review: Overall, I think the paper has merits. Program repair in the wild using end-to-end neural network is inherently a difficult problem. The proposed architecture shows a good performance while using a much smaller model, and can spawn further discussion of algorithms using graph edits for program repair. The paper has also shown a pretty comprehensive evaluation. However, the paper will be in a much better shape if the pre-training task is designed more throughly. We would also hope there is a better performance gain from incorporating pre-training. I lean towards accepting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Automated program repair benefits from knowledge of its many properties, which includes its inherent (parse-)tree structure and graphical properties such as data-flow. This work proposes a graph-based encoder coupled with a tree-edit decoder, and optionally pretrained on a tree-based objective comparable to masked language modeling. The resulting model efficiently leverages relatively few parameters to achieve near-SOTA performance on a benchmark compiled from real-world bug fixes.
main_review: Strengths:
+ The problem formulation of mapping a graph to a series of AST edits lends itself naturally to modeling source code, and repairs in particular.
+ Incorporating a large set of baselines and variations helps contextualize Graphix' performance.

Weaknesses:
- Contributions over prior work are relatively slim and hard to assess because the work lacks a comprehensive comparison.
- The efficacy of the proposed multihead GNN scheme is not adequately ablated.
- Some concerns about the scalability given the need to encode the entire tree at each intermediate step.

This work explores a promising direction, injecting richer inductive bias into neural models of source code and its edits. The scope of its technical contributions is rather narrow and although the paper includes a fair set of comparisons and ablations, their practical usefulness is difficult to assess because several key comparisons are missing. Please focus on these in the response/future revisions.

The first main contribution over Hoppity is the use of a multi-head scheme in the GNN encoder. While the name might suggest otherwise, this is an ensemble of 8 independently trained GNNs whose final states are concatenated. This naturally increases the parameter cost of the GNN by a factor 8, which requires ablations with models of the same size. The paper only offers a few such comparisons, all of which compensate by equipping the non-multi-head model with a larger hidden dimension, which is somewhat naturally prone to overfitting. This really calls for another variant that instead increases the layer count; that number is set to a surprisingly low number (4) in this work, and it seems entirely plausible that using an ensemble would compensate for this depth rather than width. This hypothesis needs to be tested. Also consider combinations of increasing both depth and width.

The second contribution is the ASDL-guided decoder. This aspect is only ablated through comparison with a small instantiation of Hoppity in Table 5 (E.2., where it should be made more clear that GRAPHIX-B refers to a single-headed model). This points to a larger problem: Hoppity is never analyzed with more than 1M parameters, despite Graphix using 32M by default. Comparisons with Graphix-B (also using 1M parameters) need not translate to the higher parameter domain. This is especially relevant since the multi-head idea discussed earlier does not seem quite as impactful, perhaps especially when properly ablated, nor does the pretraining (as evidenced later), as the ASDL contribution here. A similar problem is present in E.3. where the per-layer and last-layer performance is virtually identical in the middle two rows, so why was per-layer not evaluated at 32M parameters? (And why only using averaging?). On a smaller note, the runtime cost of operating Graphix vs. Hoppity should be reported and considered as well.

Overall, the paper uses a natural approach to this task, but much of it is similar to prior work, and the new components feel inadequately analyzed. Please provide at least equal comparisons of Hoppity and one or more deeper non-multihead Graphix models in the 32M parameter domain. Besides these salient issues, several more problems, some minor and others moderately significant, are identified below.

Detailed Comments:
- The first contribution on page 2 seems to claim a very specific "first". Was the goal to claim it as the "first medium-scale graph edit model" and list the rest of the sentences as its properties?
- The motivational example is rather generic and does not really articulate why Graphix should work better than other edit-based methods. That edits (and graphs) are a better formalism has been shown quite extensively in prior work (e.g., Ding et al., ASE'20, or many of the cited works).
- The idea of using multiple heads in graph encoders is naturally reminiscent of Transformers, and more specifically of relationally biased attention such as proposed in RAT (Wang et al., ACL'20) or GREAT (Hellendoorn et al., ICLR'20). This probably deserves a comparison.
- Reinstantiating and encoding the program graph on every edit would seem to present major scalability problems, especially on longer chains of edits. Please analyze and discuss.
- It is not quite clear whether the ensemble-of-graphs model is meant to be a contribution in general or just for source code; the former would naturally require more analysis on different datasets, but the latter probably calls for a discussion of why we expect this to just/primarily be relevant for code.
- The contribution of pretraining seems quite slim; performance only increases a little on the Small dataset and actually decreases on the Medium data. The latter probably deserves more analysis if this is to be considered a part of the contribution of this work: did you attempt to pretrain with removing larger subtrees?
- The introduction and conclusion decidedly overpromise on performance, claiming that Graphix performs well and is 10x smaller than recent models -- those recent models also significantly outperform it in many settings, and models like PL-BART are just 4 times larger than Graphix (much less than the 32x difference of Graphix with Hoppity).
- The examples in section 6 suffer from the same problem as the motivation: they do not shed any light on why the specific additions of Graphix over comparable models are helpful.
- Why are most baselines in Table 1 not evaluated on the concrete setting?
summary_of_the_review: The work proposes two modeling contributions over prior work, neither of which is ablated quite carefully enough, primarily in terms of comparing models of more reasonable size. The eventual results are not particularly strong compared to many other baselines, which, while equipped with more parameters, also use models that scale better with larger parameter budgets.
review_title: null

Meta-review: This paper presents an approach for machine learning to fix programming errors via edits to abstract syntax trees. The main contributions are a pretraining scheme based on masking out subtrees and some minor architectural modifications compared to previous work. Reviewers found the paper to contain a significant amount of work, but there are some questions about significance relative to previous work that framed the problem similarly, and about experimental methodology. Authors did a great deal of work in the rebuttal to address many of the experimental methodology questions, but this also introduced substantial unreviewed changes to the model, the pretraining approach, and the experiments. In total, the remaining concerns about significance and the substantial changes lead us to recommend that this paper be revised and resubmitted to the next conference.
Decision: Reject