Self-Supervised Bug Detection and Repair

Abstract: Machine learning-based program analyses have recently shown the promise of integrating formal and probabilistic reasoning towards aiding software development. However, in the absence of large annotated corpora, training these analyses is challenging. Towards addressing this, we present BugLab, an approach for self-supervised learning of bug detection and repair.  BugLab co-trains two models: (1) a detector model that learns to detect and repair bugs in code, (2) a selector model that learns to create buggy code for the detector to use as training data. A Python implementation of BugLab improves by 30% upon baseline methods on a test dataset of 2374 real-life bugs and finds 19 previously unknown bugs in open-source software.


Reviews and author responses:

This paper describes a self-supervised ML-based bug fixer that uses a combination of a bug localization and repair model, with a bug injection model. The two models are identical in architecture: they select a program location to mutate, and then one of a few possible mutations and its parameters (e.g., swapping a binary operator with another). Training is done concurrently from a dynamic corpus of buggy and bug-free examples. The bug fixer is trained on detecting bug-free examples, or fixing buggy ones. The bug injector is trained on selecting the location and type of bug to inject to maximize the bug fixer's loss.

The initial corpus is also augmented via a number of "semantics preserving" rewrites of examples (e.g., variable renaming).

The approach is shown to outperform bug fixers that don't have the benefit of a bug injector, as well as those that don't use data augmentation. Furthermore, the approach was shown to find previously-undiscovered bugs in open-source code, as well as perform better than previous SOTA models on a manually curated test dataset of bugs.  # Overall

Originality: The idea of using a localization+repair model as both a bug injector and a bug fixer is neat and unusual (except for adversarial related work). Augmentation with buggy or "semantics-preserving" rewrites is otherwise not that new (e.g., the Ramakrishnan et al., and Vasic et al. papers cited used semantics-preserving rewrites as well, but also "Generating Adversarial Computer Programs using Optimized Obfuscations" from ICLR21).

Quality: The submission is well described, but the actual benefit of the learned bug selector is not adequately evaluated in my opinion. Baselines are well studied. 

Clarity: The submission is reasonably well written, but it leaves important information in the appendix. The training regime is very very briefly described in the main paper. The evaluation is very well presented.

Significance: The idea of a form of "curriculum training" where hard-to-fix bugs are injected into the training dataset is neat and seems important for this (and many) code-understanding model. The results seem positive and advance the state of the art. A new dataset of simple Python bugs from actual open-source projects is a valuable contribution and is likely to be used by many future papers in this space. New previously-undiscovered bugs were found in open-source projects!


# Comments

This was a fun paper to read! The teacher/learner formulation was interesting, reminiscent of adversarial training, and not often used in this space (there are a handful of adversarial robustness papers for code, but that's about it). As such, this was an intriguing approach.

What's more, results matter: the authors found a few real bugs in open-source software, some of which were confirmed. To my knowledge, similar neural bug-finding systems haven't revealed many real bugs, and tend to focus on synthetic datasets, so this is a great new result from this work.

My main concern is that, although the experimental results seem positive and encouraging, why exactly they look like that is not very clear from the experimental evaluation or the design. In part, that's because the bug fixer and the bug injector are operating asynchronously, rather than in controlled fixed-length generations. I'm guessing this was to improve efficiency? The authors don't provide justification for this design choice. However, this gives little understanding of how the two components evolve, and how to compare them to baselines and ablations.

For example, it's not clear how the bug injector's performance changes over time, or how diverse the samples it generates are. Are they harder over time? Or is this just a matter of adding more noise to the training data for the bug fixer? For example, the difference between the Random Selector and PyBugLab on table 1 is quite small. If we could compare the two on the same number of generations of the same length for the fixer and injector, perhaps we could draw some solid conclusions. But as they are, maybe the neural injector run faster or got lucky or who knows? Or perhaps the random selector shot itself in the foot by generating many more rewritten examples? Given how close the results are, and given that we see no confidence intervals from multiple experiment repetitions, it's very hard to believe that this is a repeatable, stable result.

While we're at it, what exactly is the difference between the Random Selector and Supervised rows of Table 1? Isn't that just changing the initial data generator to create more buggy examples than bug-free examples? Does this serve any other purpose? 

I'd recommend that the authors make the experiment a bit more controllable, and show how the bug fixer's performance on the test dataset evolves over time with the learned injector, the random injector, and the original dataset.

Another related question is why we don't see a line in that table for the semantic rewrites. This seems to be a standard tool, and appears to be working well, so it would be useful to understand how the no-injector model, or the random-injector model would work with the static, ahead-of-time data augmentation.

So overall, it's not obvious to me that the learned bug injector actually does something, and if it does, whether it's big. Maybe it's the data augmentation (semantics preserving or synthetic bug injection) that does the trick. I'd like to understand better the the contribution of the learned bug injector is significant.

A related concern is sample efficiency. Because the training mechanism is rather ambiguous due to its asynchronous execution, we don't really know what the training dataset size is, or how many actual samples the bug detector saw before converging to its best behavior. Would training it longer but with less fancy an augmentation mechanism have achieved similar results?

A smaller concern is about the actual model architecture. Having to manually define scoring functions per concrete rewrite seems super awkward and unscalable. This is most glaring in the case of literal repair, where to limit the number of candidate rewrites that must be scored, the possible repairs is limited to just a few (-2, -1, ...). I wonder if a generative rewrite approach (perhaps like AnyCodeGen from ICML 20 or "Generative code modelling with graphs" from ICLR19) might be a more scalable approach, perhaps harder to train of course.

I liked the augmentation approach used, and it seems to help. However, calling it "semantics preserving" is a bit of a misnomer. Python has very intrusive reflection capabilities. Changing docstrings and renaming variables changes the semantics of the program. For example, if you rename variables in this program, you will cause an assertion error:

```
MY_VARIABLE = 42
def f():
  assert globals()['MY_VARIABLE'] == 42
```

Certainly there are good reasons not to write such a Python program, but the fact remains that your rewrites are not semantics preserving. Please don't call them that.

I was wondering, why isn't your "semantics-preserving rewrite" also an operation that the bug selector can train on and introduce in the same way as the bug injections? Arguably, such a rewrite might also throw off the detector, so trying to find such hard rewrites would be valuable for the selector. Why are the two (bug injections, benign rewrites) treated differently?

I found the comparisons to strong baselines well done. I was particularly impressed with the validation against the original datasets for the GREAT re-implementation (in the appendix). I was originally concerned with your finding that the GNN does better than GREAT, despite the findings in that paper. It was good to read the steps you took to ensure you weren't misrepresenting GREAT (although of course, it's still a mystery). A brief mention in the main paper that you did all that would have been nice, since without at least a pointer, the reader is left suspicious.

The comparison to the CuBERT baseline was also helpful in contextualizing the benefits of your approach. I was curious why you didn't also show the performance of your model on the CuBERT benchmarks (for the same tasks you chose).


# Questions

1. For the same number of total samples seen by the bug detector (ignoring the selector), how do the various approaches compare?
2. Is augmentation with semantics-preserving rewrites by itself able to give better results?
3. Why are rewrites and bug injections treated differently? Aren't they both types of augmentation?
 No issues found.

 Thank you for the detailed responses.

# Point by point

> Indeed, this allows to split the work across different processes (and hosts). This is in particular necessary because we need to re-run program analysis tools to featurise rewritten programs (to be able to do things such as resolving calls), which in turn requires installing Python dependencies etc. The computational effort of this is high, and hence we opted for a solution where we can hide it by running it in parallel, asynchronously, on other worker hosts.

Wouldn't running in lock-step, where you train the bug finder for a fixed number of steps, stop, train the selector for a few steps, stop, repeat, etc., fix this issue? Admittedly it would roughly double your training time, and it probably wouldn't be necessary in a production deployment, but for the purposes of a research study, having a controlled experiment is rather essential, no?

For example, what would happen if the machines or processes you chose to run the selector got bogged down with other co-tenant jobs, were scheduled on the wrong socket, slowed down, started defragmenting, or whatever it is machines and sockets in clusters do? Wouldn't that affect the benefits obtained via the selector in an unmeasurable, uncontrolled way?

I wonder if a way to mitigate for this asynchrony without blowing up wall-clock time for experiments would be to do something mildly speculative. You have both sides train at the same time, but you hold out new examples generated by the selector, until the detector is ready for a new generation -- essentially a barrier -- and then put those examples into the pool. Admittedly, that would potentially waste some parallelism, but at least you could argue that, say, the selector injected 100 new samples after every 1 detector epoch, or something like that.


> It’s not clear how to best automatically measure the performance of the selector model in an interpretable way – we are happy to take suggestions here. What we have done is started an experiment in which we store the selector regularly during training. We then generate smallish sets of buggy samples from each saved selector model, and then plot the performance of a fixed model (such as the “Supervised” model) for each generated dataset. If the selector indeed learns to select “harder” bugs over time, we expect the accuracy of the model to go down over time.

That seems reasonable. Do you have those numbers somewhere or was this a proposal? If the latter, yes, that sounds great!

> As discussed above, the number of training steps are identical for the “Random Selector” and “PyBugLab” baselines. The fact that result trends are comparable across the GNN and GREAT models provides some evidence that our results are meaningful (and we can report that the results are entirely in sync with additional experiments run before the final ones used in the paper). However, your concern regarding error bars is perfectly reasonable, and we have now kicked off additional training runs for the PyBugLab model whose results we will post once they are done.

Perhaps I wasn't clear. My concern is not that you used a different number of training steps for the bug detector. My concern is that you don't know what a fixed number of training steps for the bug detector corresponds to for the bug selector. In two identically configured experiments, if the bug selector happens to run twice as fast, or half as fast, you might get wildly different results, or (worse yet) you might not, and you wouldn't know the difference.

I imagine error bars, in conjunction with your proposed experiment of evaluating how selector's bugs get harder for a fixed detector might alleviate these concerns.

> Indeed, you're right that in principle, programs could rely on these (though apart from docopt, we are not aware of any regular usage of such features). However, we don't know what a better name for these rewrites, indicating that they are most likely not influencing program semantics. It would be great to hear your thoughts on alternatives names for these rewrites.

Why not just call them "rewrites"? All of this is augmentation, ultimately, and defining an artificial (from a formal standpoint) boundary between the two kinds seems rather meaningless, since you're training on everything equally -- well you are not, but you might as you admit in your response. Compilers are tested on "skeletal program enumerations" (PLDI 17) that blow up the semantics of a program, but help find compiler bugs. Why do we have to find a "benign" subset of rewrites for neural networks?

> Our main objection to this is that we have not found a principled way to avoid a pathological scenario in which the bug selector collapses to choosing only benign rewrites.

Have you found a principled way to avoid a pathological scenario in which the bug selector collapses to choosing only literal swaps? (My impression is that you haven't, but please correct me if you are guaranteeing diversity of the bug selector's output) Isn't this the same issue?

# Overall

1. It would help if you showed how the bug selector produces harder examples as it trains (with respect to the random bug finder, as you suggested). Although it wouldn't provide a controlled interaction between the selector and the detector, it would help quantify how the selector learns to be challenging.

2. I still find the concept of "semantics preserving" rewrites a misnomer you don't need. It's not a deal breaker for me, but it's a pity to put an indefensible term in an otherwise great paper.

3. I would find it very informative to characterize the diversity (in terms of rewrites) of the selector's output. I don't see any evidence that the selector keeps producing bugs of all allowed rewrites. Although the results indicate that the selector is helpful, perhaps there's a simpler selector one could train (with fewer rewrites in its arsenal) that gives the same benefit?

Thanks for your great work!

 Thanks for the response.

Regarding the comparison with the pre-trained models, in Section Comparison to CuBERT (line 282-line 302), you have already formulated the bug detector as a classification problem i.e., detecting if a bug exists, and constructed a test set for comparison. In my opinion, comparing with CodeBERT is easy and you just need to feed the code text to CodeBERT to obtain the final representation for the classification. Kindly suggest adding this part to experiments at the next iteration of your paper.


Thank you.



 Thank you for your response.

Regarding additional comparisons, CodeBERT and GraphCodeBERT mentioned by another reviewer are good candidates. There might be additional tools in this survey https://www.mdpi.com/2227-7390/9/11/1180/pdf for example. I understand though that setting up and trying additional tools might be time consuming.

Line 382 - You already mention the performance gap in lines 383-384. Are there other previously unknown limitations that you identified? If not, perhaps keep lines 383-384 and remove the general claim in line 382.

Thank you.

 Thank you for your review and kind comments. We aim to respond to the main comments you raised in your review here.

> How to ensure the raw samples have no bugs and the operations in Section 4.1 can indeed
> introduce bugs

This really relies on the assumption that most code released on GitHub is largely correct
(at least w.r.t. the kinds of bugs we are discussing in this paper). This assumption is
made in most papers in the area, and cursory analysis bears this out (in that it is very
unusual to just find bugs in this code).

While it is true that not all of our rewrites always introduce bugs (e.g., due to aliasing
of variables), this is also a relatively rare occurrence. For now, we feel it is sufficient
to treat such cases as noise in the training data, though more precise rewriting tools would
potentially improve overall results.

> I cannot agree that all the related approaches rely on datasets that cannot represent the
> characteristics of bugs in real code mentioned in the Introduction section, there are also
> some works that focus on vulnerability detection with the real vulnerability data such as
> Devign [1].

Sorry, this is really mostly due to excessive squeezing into the page limit. The Devign dataset
(as well as for example the Defects4J dataset) are realistic, and representative of a certain
class of bugs, but not sufficiently large to support training very high-capacity models, which
easily require hundreds of thousands of samples during training.

> In the experiments, the bug selector selects k=5 bugs to introduce, is this hyperparameter
> has a significant effect on the final performance, it is better to provide some analysis.

This is indeed a good question that we studied since submission time. Concretely, the parameter k determines the balance of buggy/non-buggy samples in our dataset (i.e., there are k buggy samples for every non-buggy sample in our training data), and hence particularly influences the localization performance (as localization includes the buggy/non-buggy decision).
While we did not experiment with k (which was chosen in some early experiments), we have experimented with a weight schedule which slowly decays the loss contribution of buggy examples during training of the detector. This is based on the observation that starting training with a realistic buggy/non-buggy dataset (e.g, 1:20 or even more extreme) would inhibit learning anything, as the model will just get stuck on always predicting NoBug. Instead, by decaying the weight of buggy samples, we can encourage the model to learn generalize bug-finding rules, and then later calibrate towards a more realistic distribution. The results of this experiment are largely what we expected:
* “repair (given location)” does not change substantially
* localization results on PyPiBugs (which has no NoBug examples) are substantially worse
* localization results on RandomBugs (which has 15% NoBug examples) are slightly worse overall; but “NoBug” localization accuracy jumps from ~50% to ~80%.
We will add more details and analysis on this in an appendix of the next iteration of our paper.

> if I understand correctly, comment deletion will be removed automatically during the graph
> construction, so is it necessary for a strategy of semantics-preserving rewrites?

Comments are represented as tokens (e.g. in Python’s `tokenize`) and thus appear in the graph.

> Some latest pretrained baselines should be included such as CodeBERT[2] and GraphCodeBERT[3].

We believe CodeBERT to be roughly comparable to the concurrently developed cuBERT, and the authors
did not investigate its use for bug detection/repair (though we could add something similar to
cuBERT). As this would mean substantial additional work, we have decided to not consider CodeBERT.

The case for GraphCodeBERT is similar, though it differs more substantially from cuBERT. While we agree that this would clearly be an interesting data point for further comparison, it would still require setting up an entirely new dataset to fine-tune a bug detector on top of GraphCodeBERT to compare to our work, involving a number of design decisions that could greatly change the results. We feel that this extension is beyond the scope of a simple comparison to existing baselines.

> The paper mentioned that PYPIBUGS has published at an anonymous location, I want to further
> confirm that the code and the related dataset will be public?

We will release both the source code for BugLab as well as snapshots of trained models, along
with the new PyPiBugs dataset. The latter is already included in the supplementary material.


 Thank you for your careful and insightful review of our submission!

> In part, that's because the bug fixer and the bug injector are operating asynchronously, rather 
> than in controlled fixed-length generations. I'm guessing this was to improve efficiency?
> The authors don't provide justification for this design choice.

Indeed, this allows to split the work across different processes (and hosts). This is in particular
necessary because we need to re-run program analysis tools to featurise rewritten programs (to be
able to do things such as resolving calls), which in turn requires installing Python dependencies
etc. The computational effort of this is high, and hence we opted for a solution where we can hide
it by running it in parallel, asynchronously, on other worker hosts.

> For example, it's not clear how the bug injector's performance changes over time, or how diverse
> the samples it generates are. Are they harder over time? Or is this just a matter of adding more
> noise to the training data for the bug fixer?

It’s not clear how to best automatically measure the performance of the selector model in an interpretable way – we are happy to take suggestions here.
What we have done is started an experiment in which we store the selector regularly during training. We then generate smallish sets of buggy samples from each saved selector model, and then plot the performance of a fixed model (such as the “Supervised” model) for each generated dataset. If the selector indeed learns to select “harder” bugs over time, we expect the accuracy of the model to go down over time.

> For example, the difference between the Random Selector and PyBugLab on table 1 is quite small.

While this is true for the randomly selected bugs, the differences become more stark for the PyPiBugs dataset – which is exactly what we were expecting here: our claim is that the distribution of bugs in heuristically, synthetically generated datasets (which is what the Random Selector does) differs substantially from the distribution of bugs encountered in real source code.
The results in Table 1 show that by training the PyBugLab models using our methodology (which focuses training on examples that are hard for the models to handle), the models generalize better to bugs in real source code (and even become better on synthetic data as a bonus).

> If we could compare the two on the same number of generations of the same length for the fixer
> and injector, perhaps we could draw some solid conclusions.
[...]
> A related concern is sample efficiency. Because the training mechanism is rather ambiguous due
> to its asynchronous execution, we don't really know what the training dataset size is, or how many
> actual samples the bug detector saw before converging to its best behavior. Would training it
> longer but with less fancy an augmentation mechanism have achieved similar results?

This is discussed in l241-l243: the “supervised” model was trained until results on a validation subset did not improve any more, whereas the “Random Selector” model was trained in the same regime as PyBugLab(+SemPr); namely for 300*200k (60M) training samples. Hence, the compute resources invested into training of the detector model are identical for the “Random Selector” and “PyBugLab” models, but results improve for the latter. We will make this point more explicitly in a future version of our paper.

> But as they are, maybe the neural injector run faster or got lucky or who knows? Or perhaps the
> random selector shot itself in the foot by generating many more rewritten examples? Given how
> close the results are, and given that we see no confidence intervals from multiple experiment
> repetitions, it's very hard to believe that this is a repeatable, stable result.

As discussed above, the number of training steps are identical for the “Random Selector” and “PyBugLab” baselines. The fact that result trends are comparable across the GNN and GREAT models provides some evidence that our results are meaningful (and we can report that the results are entirely in sync with additional experiments run before the final ones used in the paper).
However, your concern regarding error bars is perfectly reasonable, and we have now kicked off additional training runs for the PyBugLab model whose results we will post once they are done.

> While we're at it, what exactly is the difference between the Random Selector and Supervised rows
> of Table 1? Isn't that just changing the initial data generator to create more buggy examples than
> bug-free examples? Does this serve any other purpose?

Yes, the “Random Selector” is essentially using an infinite data generator rather than a fixed training dataset. Concretely, the “Supervised” model has a fixed training dataset of \~1M datapoints (l237) that it iterates over repeatedly during training (~20 times before early stopping terminates training), while the “Random Selector” model is trained on ~60M dynamically generated datapoints. As expected, the latter shows a substantial improvement in performance on the the similarly generated “RandomBugs” testset, but substantially less improvement on the more realistic PyPiBugs dataset.

> Another related question is why we don't see a line in that table for the semantic rewrites. This
> seems to be a standard tool, and appears to be working well, so it would be useful to understand how
> the no-injector model, or the random-injector model would work with the static, ahead-of-time data
> augmentation.

This is indeed an oversight in our paper. We don’t expect this experiment to yield any unexpected results (i.e., it will give a small boost to all model variants), and hence would like to first focus on other experiments (e.g., additional runs to provide error bars).

> A smaller concern is about the actual model architecture. Having to manually define scoring
> functions per concrete rewrite seems super awkward and unscalable. This is most glaring in the
> case of literal repair, where to limit the number of candidate rewrites that must be scored, the
> possible repairs is limited to just a few (-2, -1, ...). I wonder if a generative rewrite approach
> (perhaps like AnyCodeGen from ICML 20 or "Generative code modelling with graphs" from ICLR19) might
> be a more scalable approach, perhaps harder to train of course.

We briefly comment on this at the beginning of Sect. 4: while more general rewrites would be
desirable to catch more complex bugs, even the limited set considered in this paper is too hard
for current models to tackle in a meaningful way.

We note that our currently considered set of rewrites really falls into three classes (and
are implemented as such):
1. replace token by constant (wrong operator / literal)
2. replace token by other token (variable misuse)
3. swap two tokens (argument swapping)
It's not clear that there is a unified architecture that could handle all of these well -
a model that just ``edits'' the input sequence would in principle be powerful enough
(assuming a functionality to copy tokens outside the vocab), but would also be lacking any
meaningful model biases and hence we expect it to do extremely poorly.

> I liked the augmentation approach used, and it seems to help. However, calling it "semantics
> preserving" is a bit of a misnomer. Python has very intrusive reflection capabilities.

Indeed, you're right that in principle, programs could rely on these (though apart from docopt,
we are not aware of any regular usage of such features). However, we don't know what a _better_
name for these rewrites, indicating that they are most likely not influencing program semantics. It would be great to hear your thoughts on alternatives names for these rewrites.

> I was wondering, why isn't your "semantics-preserving rewrite" also an operation that the bug
> selector can train on and introduce in the same way as the bug injections? Arguably, such a
> rewrite might also throw off the detector, so trying to find such hard rewrites would be
> valuable for the selector. Why are the two (bug injections, benign rewrites) treated differently?

This is indeed a good suggestion that we've been meaning to evaluate. Our main objection to this
is that we have not found a principled way to avoid a pathological scenario in which the bug
selector collapses to choosing _only_ benign rewrites.

> I found the comparisons to strong baselines well done. I was particularly impressed with the
> validation against the original datasets for the GREAT re-implementation (in the appendix).

Thank you - these were significant work, and we're happy that this is appreciated.

> A brief mention in the main paper that you did all that would have been nice, since without
> at least a pointer, the reader is left suspicious.

This is briefly alluded to in line281, but we can extend this to refer to the reproduction
effort explicitly.


 Thank you for your comments and feedback!

> The paper should compare their model not just with a baseline, but also with other bug
> detection and repair models.

We found this near-impossible due to the range of different supported programming languages
(as the appropriate processing of source code is a substantial engineering effort, where
minor differences can have huge consequences in results) and the differing classes of considered bugs (e.g., reaching from single-purpose models focused on variable misuse to fully generic rewrites).
Do you have specific models in mind that we should compare with, where you expect the comparison
to be particularly informative?

> Line 30 - what do you mean by "shows improved performance over training with randomly inserted bugs"?

As discussed in Sect. 5.1, the "Supervised" model variant was trained without learning a
bug selection model, but instead uses a fixed training set of randomly inserted bugs.
(This is a common strategy to generate synthetic training data, e.g. used by our references
[3, 13, 20, 27]).

> Line 33 - Please state what are the false positive rates.

As discussed in Sect. 5.1, the FPR is 98.1%. We will update the text to include this number
here.

> Line 382 - what limitations did you identify? It is not clear what you mean.

We refer to the fact that we quantify how poorly good results on synthetic datasets (such as our RandomBugs dataset) transfer to more realistic datasets (such as our PyPiBugs dataset), as discussed in line 257. While this was already known informally in the research community, our paper provides clearer evidence on the scale of this effect. We will clarify the text.


Authors propose self-supervised learning approach that trains two models: one to detect and repair bugs in code and another one to generate code with difficult bugs. They use their trained model to improve results on baseline test dataset and to find bugs in real code.  Authors propose self-supervised learning approach that trains two models: one to detect and repair bugs in code and another one to generate code with difficult bugs. They use their trained model to improve results on baseline test dataset and to find bugs in real code. I think that the self-supervised learning approach and the resulting models it produces is valuable for automated bug detection and repair. The evaluation of the models on the real-life bugs is also valuable. The evidence that the tool can find bugs in open-source packages is also positive.

The paper should compare their model not just with a baseline, but also with other bug detection and repair models.

When run on the real-life software, the tool suffers high false positive rate. For the tool to be useful, this needs to be addressed in future work.

Line 30 - what do you mean by "shows improved performance over training with randomly inserted bugs"? 
Line 33 - Please state what are the false positive rates. 
Line 127 - "other" is repeated 
Line 274 - should be "seems to be" 
Line 382 - what limitations did you identify? It is not clear what you mean. Authors address the limitations of their work in section5.2. I think this covers the limitations well. 
I agree with authors that the work does not have obvious negative societal impact.

This paper proposes an approach for supervised learning of bug detection and repair called BUGLAB and it includes a detector model to detect and repair bugs and a selector model to create buggy code for detection. By conducting experiments on two different datasets RANDOMBUGS and PYPIBUGS, BUGLAB shows great improvement compared with the baselines.  This paper is well-organized and the analysis of the experimental results seems very sufficient. Furthermore, it provides a novel approach with self-supervised techniques to detect and repair bugs. Compared with current approaches that mostly based on a supervised learning approach for vulnerability-related tasks, the approach is interesting.  Specifically, it contains two different models, the first one is the detector model to detect buggy code, and the second one is the bug selector model to introduce some bugs in the original code by the self-defined rewrite rules.  It further contains some operations for data augmentation. - How to ensure the raw samples have no bugs and the operations in Section 4.1 can indeed introduce bugs, for example in Variable Misuse, variable-swapping rewrites cannot assure the bugs will be introduced if a,b,c are the same value in the extreme conditions? Section 5.2 also mentioned this problem.
- I cannot agree that all the related approaches rely on datasets that cannot represent the characteristics of bugs in real code mentioned in the Introduction section, there are also some works that focus on vulnerability detection with the real vulnerability data such as Devign [1].
- In the experiments,  the bug selector selects k=5 bugs to introduce, is this hyperparameter has a significant effect on the final performance, it is better to provide some analysis.
- if I understand correctly, comment deletion will be removed automatically during the graph construction, so is it necessary for a strategy of semantics-preserving rewrites?
- Some latest pretrained baselines should be included such as CodeBERT[2] and GraphCodeBERT[3].
- The paper mentioned that PYPIBUGS has published at an anonymous location, I want to further confirm that the code and the related dataset will be public?



[1] Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks.
[2] CodeBERT: A Pre-Trained Model for Programming and Natural Languages.
[3] GraphCodeBERT: Pre-training Code Representations with Data Flow.

Meta-review: The paper provides a novel approach to train a bug detector by co-training a bug injection procedure together with the bug detector. This is an interesting idea, and while the resulting bug detector has a high number of false positives, it was able to find new bugs in PyPI packages. (19 of 1000 reported bugs turned out to be real bugs). 

The bug injection procedure is based on transformations that are meant to introduce bugs in the code at a known location; the transformations are hand-crafted, but the model learns where to apply them to make the bugs hard to find. The system also relies on semantics preserving transformations to introduce additional variety to the set of non-buggy programs. One limitation of the approach is that the bug introducing transformations may not actually be introducing bugs in all cases (for example, if they swap two variables that happen to be aliases of each other), and the semantics preserving transformation may not be semantics preserving (as one of the reviewers pointed out). 

Overall, this is a strong paper; it presents a novel and interesting idea, and while there was a desire from some reviewers for additional baselines, the evaluation is still fairly convincing. One of the reviewers surfaced a paper that I think is relevant and should be cited ("Generating Adversarial Computer Programs using Optimized Obfuscations"), but that paper is quite different does not detract from the novelty of this work.


Decision: accept