Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a self-supervised approach that trains bug detectors by co-training a bug selector that learns to create bugs. A high-level framework is formulated, and then a python implementation using graph neural networks and relational transformers is provided. Experiments on a manually curated bug dataset demonstrate its effectiveness.
main_review: The paper is well written, and the formula-based explanation of the method is clear.  It is an interesting idea of incorporating a bug selector with bug detectors, which resembles GAN though the overall framework is non-differentiable. But the overall training procedure seems very complicated. As mentioned by the authors, it costs more than one week to train the models. 

The main problem of this paper is in the evaluation, making the experimental result somewhat unconvincing. Some important baselines (e.g., HOPPITY[1],  deepdebug[2]) are missing, making it difficult to verify the superiority of BUGLAB over existing techniques. Simply comparing with CuBert on only bug prediction task is somewhat weak. Or the authors can instantiate the BUGLAB framework based on other program-repair datasets used by previous works to verify the its generalization ability. I believe there can be a setting where BUGLAB can be compared with previous representative works, though it may involve tedious implementation work. Given current experimental results, it is not ready to be published at a high-profile conference.

-----After seeing the authors' response-----

In my view, though the method is claimed to be "self-supervised," the main contribution is more about a refinement strategy on auto-generated training data. Based on some heuristic rules, the bug selector will actually generate and select buggy code (of specific types ) that serves as training data for the bug detector. Some other works (e.g., DeepBugs) will also automatically generate training data (though not in a learning way), targeting similar restricted bug types (e.g., wrong operation and argument swapping) and based on similar rewrite rules. More comparison and analysis are necessary.  The data refinement idea does make sense, though its technical design is not novel to me.

[1] Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. Hoppity: Learning graph transformations to detect and fix bugs in programs. In International Conference on Learning Representations (ICLR), 2020.

[2] Dawn Drain, Chen Wu, Alexey Svyatkovskiy, and Neel Sundaresan. Generating bug-fixes using pretrained transformers. arXiv preprint arXiv:2104.07896, 2021. 4
limitations_and_societal_impact: This paper has no potential negative societal impact.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper describes an adversarial approach towards detecting and repairing local bugs in human-written software. The approach involves training in tandem two neural networks, the detector, which discovers and repairs bugs in a given piece of code, and the selector, which produces bugs that the detector is unable to detect. At test time, only the detector is used.

The architecture used for each of the networks is similar: first the program is represented by a graph with labeled edges, then each node is embedded using a language-like model (to take into account semantic information in variable names), then embeds the graph using either a GREAT relational transformer or a GNN. The code rewriting model then predicts a distribution over locations to rewrite using a pointer network, and a distribution of rewrites at that given location using a softmax over potential rewrites.

The types of bugs considered are local and shallow bugs such as variable misnamings or use of the wrong operator or literal in an expression. The authors create a corpus of code by crawling pypi and create a corpus of known shallow bugs by looking for version updates that include edits that fit their definition of a shallow bug. They additionally include a corpus of random generated bugs. In general, their approach works better on the random bug corpus than on the real bug corpus, and also works much better at repairing bugs given localization than it does at localization. They are able to achieve better performance than the special purpose CuBERT model using a F1 metric. Additionally, they were able to detect 19 bugs in open source projects, but with a TPR of 0.19% this is probably not yet a productionizable tool in this context.
main_review: Originality: the central topic of the paper, the adversarial approach, is novel in this domain as far as I can tell. The neural architectures are not original, but they are not the main focus of the paper and are adequately cited.
 
Quality: The paper makes claims that are clearly supported by the evidence provided in the paper, and does not shy away from presenting limitations. I especially appreciated the real world experimentation as well as  the clean structure of the training process and the similar neural architectures.
 
I have a few minor nits that are listed below:
 
121: I think mentioning earlier (perhaps towards the beginning of the introduction) that the bug set you are targeting is simple bugs might be helpful, as it contextualizes the local techniques being used. I would also avoid characterizing the bugs as “stupid simple bugs” and instead characterize them as local, as many bugs that are local are not simple to detect and many bugs that are simple to detect are not local.
 
296: If possible, a comparison to a CuBERT model that has been specifically conditioned to have the same precision as the PyBugLab model or vice versa would be slightly more convincing (just change one of the detection thresholds so that the precisions match). Not a huge deal if this is too difficult to do since there’s such a big gap in the recall.
 
Clarity: The submission is clearly written and conveys all the necessary information. In general it is easy to read and the key results being highlighted with the bug icon and underline makes it easier to quickly find them.
 
Significance: The results seem pretty significant. While I don’t know how much this specific task actually needs to be solved (as the authors themselves acknowledge), it certainly is a real world task. More importantly, the broader approach here is not at all specific to the kinds of bugs this paper tries to address. I can see this approach applied to bug repair of human programs and even potentially to domains such as program synthesis.

limitations_and_societal_impact: The paper very clearly addresses its limitations. I do not think the approach provided here has a substantial potential negative social impact (software tooling probably will not lead to job displacement or surveillance), so it’s probably fine that it is not addressed in detail.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work tackles programming bug detection and repair, introducing BugLab, which uses two models trained in an adversarial fashion together. One model is a bug detector that learns to identify and repair bugs, while the other is a bug selector that introduces buggy mutations in code that are difficult for the detector to identify. When trained together, the detector model is able to see more training data for bugs that it is worse at detecting, leading to better performance after sufficient training in this fashion. BugLab also uses semantics-preserving code modifications as a form of data augmentation. The experiments show that the BugLab training procedure leads to improved performance in bug localization and repair when applied to prior models (GNN and GREAT), compared to baselines of supervised learning and selecting buggy mutations at random instead of sampling from the trained selector model.
main_review: ## Positives:
* Intuitive idea of training two models together such that they help each other train better, which is easily understood because it is similar to other works like GANs
* The main idea is presented clearly and well-justified
* Detailed analysis of experiments

## Negatives:
* The experiments have some minor gaps, although I don’t expect these gaps to affect the conclusions drawn by the paper
* The text could use some editing for clarity and typos

---

**Originality: 3/5** The idea of using two models trained together is not new (nor did the authors claim that this is new), but this is the first work I’m aware of that applies this technique to code repair. I think the technique is particularly applicable here specifically due to the lack of large datasets of real bugs, and random and synthetically generated bugs likely leads to a distribution quite different from real bugs. By using a bug selector to choose bugs that the detector struggles with, the detector is trained on increasingly difficult bugs and has more of its bases covered.

**Quality: 4/5** The paper is generally well written and the experiments and analysis are generally good too. However there are some gaps (errors in experiment setup or obvious questions left unanswered) in the experiments that prevent them from feeling complete, although I suspect the main conclusions of the paper to remain the same regardless. See questions below.

**Clarity: 3/5** The paper communicates its main ideas clearly, but some details could use clarification or better wording (see below). The paper also introduces a lot of notation without many concrete examples to ground the notation with ideas in the reader’s mind, and I am wondering if all of the notation is actually necessary, or if some parts of the presentation could be simplified and made more intuitive.

**Significance: 4/5** Code repair is a significant research topic with obvious practical impact. Although BugLab is not yet good enough for use in real coding environments due to its impractical false positive rate, it makes good progress toward a significant goal.


## Questions and comments
The most important items, in my opinion, are marked with **(!)**.
* Line 181: do you consider swapping `a == b` with `b == a`? What about swapping operands of commutative operations like + or * (on numbers, not on sequences)?
* **(!)** Line 196-205: What do the “pools” of training data actually contain? What specifically are the “new elements for the data pools”? Are there separate data pools for training D versus S?  I read this paragraph a couple times but didn’t quite understand the details, so clarification here would be helpful.
* **(!!)** 294: CuBERT considers more “operators” in its WrongOp task than just l11 and l20. In particular, `in`, `not in`, `is`, `is not`, `and`, `or`, etc. are all “operators” considered by CuBERT, and thus the bug locations could be l2, l22, l14, l16, and l20 -- see Table 7 of the CuBERT paper for a more complete list. The corresponding experiment should be updated. _I view this as high priority to fix, since otherwise the paper would be making factually incorrect claims about CuBERT._ Adequately addressing this issue will improve my opinion of the paper -- I think this is a good paper overall but, at the moment, I can’t suggest acceptance due to the factually incorrect claims.
* **(!)** Why no NoBug category for PyPIBugs? Considering that PyPIBugs consists of code that was updated with a rewrite captured by Sec. 4.1, and we’re assuming that the “before” version of the code is buggy, I think it’s similarly safe to assume that the “after” version of the code is bug-free.
* Section 4.1: Similar to Argument Swapping, why not consider Operand Swapping? I.e., instead of changing the order of arguments to a function call, consider swapping the operands on the two sides of a binary operator. (This is the Swapped Operand task in CuBERT.)
* Checklist 4(b): Did you check that all of the individual open-source licenses are compatible with how you’re using+releasing the code? For example, viral licenses may require that your dataset/code also be released under the same viral license, which may cause conflicts with requirements imposed by other licenses.

## Minor Suggestions
* Line 93: The text says relations in E are mapped to “elements of V_tok” (i.e., individual tokens) but the elaboration seems to imply that each relation is an edge between 2 elements of V_tok
* Line 116: When looking for “three concrete rule score functions in Sec. 4”, I wanted to find mention of $w_p$ since this is the notation for rule score functions (line 113). Although I believe I know what the 3 rule score functions are in Sec. 4, this $w_p$ notation is not used.
* Line 209: On a first read, I thought, “Why are some packages excluded from training?” The question is answered on line 221 (the remaining packages form the test set), but I think clarification could be added to line 209
* 259: double quotes going the wrong way
* 270: “results the” -> “results of the”
* 274: “seems to is” -> “seems to be”
* 306: the truncation is in line 4
* 378: “offer” -> “offers”, since “analysis” is singular
* 383: “relative” -> “relatively”

---
Update after author response: changed score from 5 to 7
limitations_and_societal_impact: The authors did a good job here.
review_title: null

Meta-review: Using modern (deep unsupervised) methods to support bug detection in programs is an important direction. The reviewers generally liked the approach of this paper, though the relation of this method (for buggy data generation) to the methods actually used for bug detection means that proper evaluation is a bit subtle. Overall the final feeling was that this was sufficiently demonstrated to make the interesting techniques worth publishing. I would urge the authors though to pay careful attention to the points made by reviewer aZJ8 in revising the paper.
Decision: Accept (Poster)