Self-Supervised Bug Detection and Repair
Authors: Miltiadis Allamanis
Henry Richard Jackson-Flux
Marc Brockschmidt
Venue: NeurIPS 2021 Poster (2021)
Keywords: ml4code
bug detection
gnn

Abstract: Machine learning-based program analyses have recently shown the promise of integrating formal and probabilistic reasoning towards aiding software development. However, in the absence of large annotated corpora, training these analyses is challenging. Towards addressing this, we present BugLab, an approach for self-supervised learning of bug detection and repair.  BugLab co-trains two models: (1) a detector model that learns to detect and repair bugs in code, (2) a selector model that learns to create buggy code for the detector to use as training data. A Python implementation of BugLab improves by 30% upon baseline methods on a test dataset of 2374 real-life bugs and finds 19 previously unknown bugs in open-source software.

Self-Supervised Bug Detection and Repair
Miltiadis Allamanis, Henry Jackson-Flux, Marc Brockschmidt Microsoft Research, Cambridge, UK (miallama, mabrocks (oaicckssmmiiiccce (0microoott.com
Abstract
Machine learning-based program analyses have recently shown the promise of integrating formal and probabilistic reasoning towards aiding software develop- ment. However, in the absence of large annotated corpora, training these analyses is challenging. Towards addressing this, we present BUGLAB, an approach for self- supervised learning of bug detection and repair. BUGLAB co-trains two models: (1) detector model that learns to detect and repair bugs in code, (2) a selector model that learns to create buggy code for the detector to use as training data. A Python implementation of B BUGLAB improves by up to 30% upon baseline meth- ods on a test dataset of 2374 real-life bugs and finds 19 previously unknown bugs in open-source software.
Introduction
Detecting and repairing bugs in source code requires strong reasoning skills over formal structures (e.g. data and control flow) and ambiguous information (e.g, identifier names, coding idioms, and comments). Traditional program analyses are able to detect critical bugs through formal reasoning and combinatorial search, but need be manually coded by experts. That is lengthy and costly process, which misses the opportunity to use ambiguous information pervasive within code.
Towards broadening the applicability of such methods, and utilizing ambiguous information, deep learning-based bug detection methods are being investigated [22 potential to further improve the engineering of software we rely on every day. However, many challenges in the area remain open, such as creating robust bug detection and repair methods that cover wide range of common bugs the absence of large supervised training corpora. Existing work focuses on randomly inserted bugs [22 [3]. Cloze test proxy tasks [3], corpora of small code edits that may contain bugs [9] build errors (28). All these approaches rely on datasets of very limited size or ones known not to be representative of the characteristies of bugs found in real code.
3 13 These methods have the
In this work, we propose BUGLAB, self-supervised approach that trains robust bug detectors by co-training a bug selector that learns to create hard-to-detect bugs (Sec. 2}. For example, for a given code snippet with well-named variables, variable misuse bug may be easy detect and repair, whereas an incorrect comparison operator might be significantly harder to identify. We propose neural architecture for BUGLAB (Sec. 3) and implement it for Python (Sec. 4). Our implementation considers four broad classes of seemingly simple, yet hard-to-detect bugs and shows improved performance over training with randomly-inserted bugs on PYPIBUGS, a new, manually curated test set of 2374 real-life bugs (Sec. 5). Furthermore, we tested our trained models on popular open-source Python packages and identified 19 previously unreported bugs, though false positive rates of ~ 98% remain impractical. We hope that creating machine learning methods that can detect these bugs early and assist developers will speed up software development and allow engineers to deliver more robust software. We release PyPIBugs and our code at https: htt::/gittttt com/ mi miccosoft//nurris221 self- selff-upervvieed-bu-- www-bbeuctteiorr....me and-repair
"Work done while Microsoft Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2 Self-Supervised Bug Detection
In this section, we first introduce the concept of code rewriting, and then use it to define BUGLAB as a framework for self-supervised learning of bug detection and repair.
Code Rewriting Rewriting is common within compilers and their optimizations, test-driven search- based bug repair tools, mutation testing, and refactoring tools. Rewrites can be semantics preserving (e.g. renamings of local variables), or semantics-alteriin (e.g. replacing by !=).
pemantics-ressrriigg
Let S denote the set of all syntax trees (not necessarily rooted in the start symbol of the language grammar). Syntax tree locations & E {c} U in a syntax tree s S are recursively defined, where Bic = and Siy for l oi is the i-th child Bjr (i.e. Si(2,3) denotes the third child of the second child of s). We define rewrite rule p (mp. tp) as a pair of matching function mp : {true, false) and a transformation function Ep S - S. The matching function m, yields true iff the rule p applicable at the root of a subtree s. The transformation function can be applied obtain transformed syntax tree. For convenience, we define (s)
false. We then write ps)) to indicate the modification of a syntax tree s using P when possible, and otherwise the identity function. For reversible rewrite rules p. we denote the inverse rule as p such that (p(s)) holds. We discuss concrete rewrite rules p in Sec. 4
iff m ms(s)
Given a set of rewrite rules R we define the set of "potential rewrites" in syntax tree s as Rg (E, p) / E R, location in 8, (Sje)) = (rue); ). For each tuple (e, E R. we use = s(p) e to denote the syntax tree obtained by applying p at location É of s. In BUGLAB, train models that use rewrites from R to insert and repair bugs. We will discuss such neural models Sec.
BUGLAB In BUGLAB, we are interested in self- supervised training of robust bug detector model Dg with parameters 0 on an unannotated codebase C. Let be set of rewrite rules" that allows to insert and repair bugs. We train Dg to be able recognize the "hardest" possible rewrites that could be applied on our codebase C. For this, we consider the loss LD, of Dg on rewritten code snippet s/p]e. for which the model needs to predict the repairing rewrite (e, Formally, we want to min- imize the objective
Original
Bug Selector S.
Code w/ introduced bugs
Learning
Bug Detector Do
Ex-o max
LnZ (slples (e, (s (C,p (C.p)ERS
Detected Bug
Figure 1: BUGLAB overview: se- lector model So. decides which (bug- introducing) rewrite to apply to input code snippet. Then a bug detector Dg tries to locate and repair the inserted bug (if one was inserted).
However, for any useful detector the set rewrites R is commonly very large or unbounded and computing the maximum over all (E, p) E RR is practically intractable. To address this, BUGLAB introduces bug selector model Sg (with parameters Φ). whose goal is to approximate the intractable (ax(E,,) LDD (-). We maximum. We denote this as (E, p) ~ Sa(s) and the overall BUGLAB training objective can be written as a min-max optimization problem:
then sample rewrites from Sg instead of computing the
maxmin Exg [E(e,))-6(() [Cn, (s(ple. (c. (e.p- 9
The two models S and D in BUGLAB are "symmetric" in the sense that they both predict rewrites on code snippets, and only differ in their objectives one aiming to introduce bugs and one aiming to repair them. In practice, we can and do use the same architecture to model both S and D, which we will discuss in the next section. At test time, we discard S and only use the trained detector D to locate and repair bugs.
In this work we assume that R contains special "identity" rewrite rule Pe that does change the code.

---

3
Neural Models
In this section, we discuss how we represent code in BUGLAB and the neural models we use to learn how to rewrite code in the selector and detector models.
Code Representation We consider source code as set of entities vi- V which relate to each other with a set of typed relations 6 E E, where a relation c₁ = (v,, 7; U2) denotes a relationship between entities vy and vj with type r. The entities and relations can be thought as a heterogeneous graph G = (V: E). The choice of code entities and their relationships is a form of high-level feature extraction. We discuss concrete entities and relationships for Python in [See. We also define projection function too that accepts V and E and returns a sequence Viok of the token entities in V with the nodes appearing in relations in E deterministically mapped elements of Viok. i.e. Erok = {p((x)) r, p.,(), where p maps the entities in V to Viok- Ptok will be used for relational transformer models.
To learn a neural representation of the code entities v, first we define an embedding function e(v;) which maps the content of each entity to an initial D D-dimensional representation. Throughout this work - similar to Allamanis et al. 4 and other previous work - we deterministically split the string representation of each node into subtokens (e.g., fooBar is split into foo and bar), embed them through a learned embedding matrix, and use max pooling to get a single vector. We then "contextualize" the entity representations within G using one of two models: MI.P-based GNN model with max message aggregation and the GREAT relational transformer of Hellendoorn et al. [3] over the token sequence and relations Viok, Brok IP tok (V, E). GREAT uses both positional encodings and the projected relations in Erok. See Appx. A for detailed architecture descriptions. Other models to compute entity representations can be used, but were not explored in this work.
We use Te to denote the computed vector representation of the entity at location l, independent of the model used to produce it. We use these representations to define our code rewriting models.
Probabilistic Code Rewriting Models Both bug selection and bug detection require to model the probability of applying a specific rewrite at location in a code snippet s, either to introduce or repair a bug. For this, we factorize this task into localization and rewrite- rewrite-given- given-location models, í.e.
p ((e, p) |s R = Proc(e 8, RB Prew (p f, 8, R2)
We model Pioe as a probability distribution over the relevant locations {E (E, p) E R JU (NOBUG), where NoBuG a special location used indicate that the code is not buggy. In practice, we imple- ment this similar to a pointer net 199 using the representations re (see Appx. A for details).
To select rewrites, This function maps a vector representation of an entity Tę and potential additional metadata onto scalar score. The rule-specific metadata Ms,s, e) is defined for some rewrites, e-g. containing representations of other entities that could be used in the location f. We will discuss three concrete rule score functions in [Sec. 4 The rewrite probability distribution Prew is then modeled by softmax over the scores of all applicable rewrites at a target location f, i.e.
rewrite type-specific learnable rule score functions wp (re.Mp(s, f)).
Prew (p L,S, RR)
= softmax (ww (re,M (s, E))
(ER
4 A Python Implementation
This section presents an implementation of BUGLAB for Python called PYBUGLAB. PYBUGLAB currently tackles a large subset of "stupid simple bugs" [16]. Fixing these bugs requires small changes to the code, but commonly has significant impact code correctness. Such bugs may be thought as a form of typographical mistake or a copy-paste error, and are often relatively hard to locate by humans but obvious after the fact. They are also quite common, observed in the empirical statistics of Karampatsis and Sutton [16 and Just et al. [14]. Future work may focus on a broader set of rewrite rules or even learnable rewrites, but as we will observe in Sec. 5) more work is needed towards this. Almost all ideas in PYBUGLAB transfer straightforwardly to other programming languages other than Python, but would require some engineering effort to implement.
PYBUGLAB Code Entities and Relations In this work, we follow related literature (see Sec. for more) and extract entities and relationships that are readily available by tokenizers, parsers,
3

---

def foo (a,
b, c=0):
c: NoBuG
Ig:
1 not
if a
in
b
[1: b,
1g:t- //-, X-
117: False
c bar
(b
12: not in
110 n,
118 a. b, c_1a__eg
13: c
111 <o,
l19: n. b, c_ia__eg
c_ _e_nng
if c is neg
[3
or
is
int:
4 m.
12:-22-1,.1
120: >m, <, <5, mm,
(::
(13 a, by not c_is_neg
2211 .-
return
True
(e: (ar(c,
414 and
122 8, by c__ie_eg
return
[2]]
17: e//o
(15 by e_de__eg
Figure 2: Code snippet and rewrites available to PYBUGLAB.
existing simple program analyses, or other Python-specific program analysis tools. The complete list of entities and relationships can be found in Appx. Baadd include syntactic entities and relations, relations about the intraprocedural data and control flow, types, and documentation, Some notable entities include SyntaxNodes, Tokens, and Symbols (references to variables and functions). Fig. in Appx. B) shows a graph of the entities and relationships of the snippet in Fig. 2
4.1 Bug-Iinu PYBUGLAB Rewrite Rules
PYBUGLAB focuses on four common kinds of bugs. Fig. 2 shows code snippet and the rewrites allowed for each location, which number 63 even for this small example.
Variable Misuse Originally defined by Allamanis et al. 3 as a Cloze test for source code, Vasic et al. 30 and Hellendoomn et al. [33 reformulated the task localizing a variable misuse bug (if any) within a snippet and repairing it. PYBUGLAB uses the latter representation. Variable misuse bugs are common, with 12.8- 14.8% found in the ManySStuBs4J corpus [16] and about 6% of them caught during Java compilation in the Google build system [28). To insert and repair variable misuse bugs, PYBUGLAB supports variable-swapping rewrites, such as in locations 11, 4a and 44 (amongst others) in Fig. 2] To score variable-swapping rewrite, we use the representation of the rewrite location re along with the representation r, of variable Symbol a that could replace the current variable, i.e. is in-scope and has been defined before L. The rule score function wp for replacing the variable at E with the symbol is then computed as the inner product r. Pa.
Argument Swapping (or Argument Selection) First coined by Rice al. 26 it refers swapping
the arguments of a function invocation, e.g. in 4e of Fig. 2 Rice et al. 26 and DeepBugs [22] tackled this problem when all arguments are single identifiers. PYBUGLAB extends this swapping arbitrary argument expressions. The rule score function wp for an argument swapping rewrite is two-layer MLP applied to the concatenation of the output representations of the representation of the parameter and the to- be-bbbe-wwpp bwaspe arguments arg1, and arg2: MLP [[pparame: Targi, Targa)).
Wrong Operator Corrupting operators has long history in mutation testing [4] Detecting incor- rect operators with deep learning was first tackled by DeepBugs (22) by using learnable embeddings of operators, operands and literals for arithmetic and comparison operators. DeepBugs focused only on binary operators. In PYBUGL and comparison operators and two unary operators: logical and arithmetic negation. Locations (1 414. 416. and (20 in Fig. 2 are rewrites related to wrong operators. The rule score function Wp for an operator rewrite again uses an inner product, r. Top. where rop is a learned embedding for operator op. Note that we rewrite operators only to compatible operators (e.g. to > but not +).
we tackle all binary operators, including Boolean, arithmetic
Wrong Literal Corrupting operands, and specifically, literals appearing in the source code, is also a common strategy in mutation testing. As in mutation testing, PYBUGLAB handles limited number of commonly used literals, allowing rewrites to replace integer literals within the set of -2, -1, 0, 1, 2 and swapping the Boolean literal True with False and vice versa. The scoring function identical to the operator rewrite, using a learnable embedding Fist for each literal lit.
4.2 PYBUGLAB Rewrite Rules for Data Augmentation
We additionally consider more rewrite rules that are not meant to change the program semantics, using them as form of data augmentation. This in spirit similar to ideas in computer vision where images are transformed (e.g. rotated, cropped) but maintain their original content. Such rewrites

---

Algorithm Sequential Training Procedure for Selector and Detector models
Require:
Code dataset C, initial detector/selector model parameters ((0), gi(0)
1: for meta-epoch i = 0 to I do
2: I/ Create dataset of buggy programs:
3: CD + (sspll.. pl. I C, samples (c, Saro
(li+1) & update ((6) by training D on C
4:
5:
I/ Create dataset of hard-to-detect bugs:
{ (s,, leaarmaaxx (r,)e (Lngte (slple, (c, (6, |SEC]
6:
S
((++1) + update (5(+) by training S on C(
7:
have been shown to yield adversarially robust models of code (23] Although our goal is not to provide adversarial robustness, believe that such rewrites can help generalization. PYBUGLAB implements the following rewrites for this purpose:
Variable Renaming renames a local variable to a random name not already in scope.
Comment Deletion removes code comments, including docstrings and inline comments. Such comments commonly contain natural language information that is useful for code comprehension, but usually do not affect program semantics.
Comparison Expression Mirroring swaps the two sides of a comparison operator and changes appropriately. For example, a<b is transformed to b>a. Note that in cases such as foo() bar O) this will change the order of execution of foo and bar, possibly altering program semantics.
If-Else Branch Swapping negates the test condition of an if-else statement or a ternary expres- sions using DeMorgan's 's law and swaps the then body with the else body.
4.3 Implementation Details
To make the training computationally tractable we approximate Eq. 1 A simplified, sequential version of training procedure shown in Alg. Intuitively, we alternate between training the two models, as the (discrete) sampling of rewrite rules in the selector models precludes direct end- to-end training. We first use the current state of the selector model to generate "hard" samples and train the detector model on these samples (we always include the unmodified (i.e., NoBuG case) as sample). Then, we use the loss of the detector model to identify those generated samples that were hardest to detect and train the selector model to produce such samples.
In practice, implemented the training procedure cating processes, and all of the described steps happen in parallel. We do not use "generations" (0) ((1) ... of datasets, but instead use two constantly updated "pools" of training data, one D/S: for the detector and one for the selector. Each training step samples minibatch from the current state of the corresponding data pool. We remove samples from the data pool once they have been sampled y times for use in training, in spirit similar to replay buffers in reinforcement learning. In our experiments, y was set to 4. We regularly (in separate, concurrent processes) take snapshots of the the current state of the D and S models to generate new elements that are updated to the data pools, matching the procedure described in Alg. [ We approximate the arg max in line 6 by only considering the samples chosen in line for each input program. During training of S, we then mask out the unobserved choices before computing the loss.
system of asynchronously communi-
5 Evaluation
We now discuss our new dataset and evaluate PYBUGLAB, We highlight key results.
Datasets train PYBUGLAB we retrieve the 4k most downloaded packages in the Python package index (PyPI) and take 3.4k of them training packages, using the rest for test purposes. During training, PYBUGLAB installs each package along with all its dependencies. Installing all the depen- dencies is important for extracting the entities and the relations beyond local syntactic ones (e.g. type
5

---

inference, method resolution). For each file, PYBUGLAB checks if it is duplicate of a file that has already been seen in the training following the method of Allamanis [ and runs all the relevant program analyses to extract the entities and relationships in each function. When we use additional rewrites for data augmentation, these are applied at the input of the PYBUGLAB pipeline as form of pre-processing. Following Alg. [ the bug selector S selects k = 5 bugs to introduce, rewrites the source code text, and then the program analyses extract the new entities and relationships for the rewritten code snippets. The initial and rewritten code snippets are then used to create the training data for the detector and selector models.
We use two testsets to measure performance. First, we create RANDOMBUGS, a testset of 761 445 snippets derived from functions from the 600 PyPI test packages (not seen during training). For each function we find within these packages we add it to the dataset along with 9 rewritten functions with a randomly inserted bug. On average graphs have 260 nodes, 601 edges, 25 rewrite locations, and 130 possible rewrites. We also collect a testset of real bugs. Although we conjecture that, in practice, the vast majority of bugs like those discussed Seec. 4.1 are fixed when developers locally test their software, few of those slip and then are fixed across different revisions checked into a version control systems. We have crawled the accessible repositories of all 285k packages in the Python Package Index (PyPI), collected and manually filtered bugs captured by the rewrites from 4.1 This new dataset, PYPIBUGS, contains 2374 real-world, small bugs. We describe the data collection process in (* :. se (aadd Soperceee ..........
in Appx. DJ in addition, we c.....7 PYPBUGS-PosFFii the examples from PYPIBUGS after bug was fixed we believe these samples are very likely to not contain any bugs anymore. We publish the dataset at https: //ww://wwwww.. www.wwff..ooo.tt.ommmmmmmm...... com/ it in the supplementary material.
and include
5.1 Quantitative Evaluation
Our first experiment aims to evaluate whether the BUGLAB training framework yields more precise bug detectors. We consider two model architectures, using either GNNs or the GREAT transformer to compute embeddings of code entities (architecture details and hyperparameter choices can be found in Appx. A). We use four different training strategies: "supervised" is training only bug detector on a fixed dataset of million functions from the 3.4k training packages with randomly inserted bugs. "Random Selector" refers to a variant of PYBUGLAB using a bug selector model that uniformly at random picks a rewrite to insert bugs. Finally, PYBUGLAB and PYBUGLAD +Aug use our framework from [ecc. 2, with the latter also using additional rewrites to augment our code corpus. For the fully supervised model, we train with early stopping over a validation set; the other models trained for a fixed number of 300 epochs (with 200k training samples per epoch) for the bug detecto and the last detector model is used for evaluation.
Effectiveness of BUGLAB Training We first consider the performance of different models on the synthetic RANDOMBUGS dataset. Tbl. I shows the accuracy of predicting full bug repair correctly ("Joint") and analo- gous Eq. break this up into local- ization accuracy ("Loc") of predicting the correct location (or NoBuG for cor- rect examples) and a repair accuracy ("Repair") for selecting the correct rewrite given the buggy location.
Table 1: Accuracies (%) for different training strategies and model architectures RANDOMBUGS.
RANDOMBUGE BUGS
GNN
GREAT
Joint
Tac
Repair
Joint
Loc
Repair
Supervised Random Selector
62.4
73.6
81.2
51.0
61.9
76.3
69.4
79.6
84.0
63.9
73.6
82.0
PYBUGLAB PYBUGLAB +Aug
69.6
80.4
84.2
64.0
74.3
82.3
70.3
81.1
84.5
65.3
75.3
82.5
We observe that PyBUGLAB ABrtaannng,,, leads, to more robust bug. detectors, compared to. other ... -e..............................
.o... mo... poore.rouust serccores for both GNNs and GREAT. Random selector models - a form of data augmentation im- prove performance over supervised methods but mostly on in-distribution BANDOMBUGS samples. helps generalization, but does not make a substantial ger se.t crreee
As expected. augmenting, the code ...... ter..... difference. Expanding the kinds of rewrites used to augment the data and learning to select them may improve performance in the future.
3 Thii amounts to about 1.5 weeks for the GNN models and about I week for the GREAT models on single P100 GPU.

---

Table 2: Results for different training strategies and model architectures PYPIBUGS.
PYPIBUGS
PYPIBUGS-PostEr
GNN
GREAT
GNN
GREAT
Joint
Loc
Repair
Joint
Loc
Repair
Loc
Joint AUC
Loc
Joint AUC
Supervised
20.0
28.4
61.8
16.8
25.8
58.6
17.8
0.087
20.7
0.044
Random Selector
21.2
27.0
69.2
20.6
26.8
67.2
47.5
0.108
52.5
0.117
PYBUGLAB
24.2
31.3
70.7
24.0
32.8
67.9
32.9
0.160
28.6
0.140
PYBUGLAB +Aug
26.4
33.5
72.0
23.2
29.7
32.6
0.187
48.2
0.129
Table Localization and Repair Accuracy (%) per bug kind for the PYBUGLAB +Aug model.
RANDOMBUGS
PYPIBUGS
Bug Type
GNN
GREAT
GNN
GREAT
Loc
Repair
Loc
Repair
Loc
Repair
Loc
Repair
Argument Swapping
85.0
57.3
65.5
57.2
33.2
73.9
24.3
72.7
Wrong Assign Op
96.1
99.1
94.5
98.6
20.0
68.9
14.0
58.1
Wrong Binary Op
83.0
85.2
77.3
81.4
27.2
54.3
36.6
43.7
Wrong Boolean Op
71.8
99.5
43.6
99.5
27.6
96.9
15.7
97.2
Wrong Comparison Op
83.9
79.3
80.0
76.4
33.7
66.1
31.1
53.5
Wrong Literal
71.7
74.7
66.6
71.6
21.6
78.4
17.9
79.5
Variable Misuse
84.9
88.4
78.2
86.3
35.3
70.5
34.0
69.4
NoBuG
53.8
62.5
Furthermore, bug. localization is much harder than repair at a given location. This is somewhat ................................ expected: there are many more candidate locations compared to potential repairs at given location. However, this suggests that research should focus on the localization problem rather than repair.
We turn the results PYPIBUGS, shown in Tbl. which also includes the accuracy of choosing the special NoBuG location on the SYPIBUGS-PosttH dataset, as well as the area under the precision recall curve for the results on both PYPIBUGS and PYPlBUGGS-Pottix
find that detecting and repairing real-life bugs is significantly harder, than handling randomly inserted bugs. As PYBUGLAB models trained using learned bug selector outperform those using "Random Selector", we speculate that the learned selector avoids generating easy-to-detect bugs, focusing the detector model on recognizing deeper semantic patterns. Despite this, improvements in RANDOMBUGS often correlate with improvements in PYPIBUGS. This is encouraging: collecting PYPIBUGS-liK datasets is costly; corpora with random bugs can help measure relative improve- ments some extent. Finally, we find that * recognizing non-buggy samples is very. hard, and in particular, does not always profit from training in PYBUGLAB
observed that the models raised some confident but incorrect warnings at very "odd" locations. However, these warnings were different across models. We have tested ensembling strategy averaging the output probabilities of five separately trained GNN models. This results in localization and repair accuracies of 83.0% and 85,4% on RANDOMBUGS (vs. 81.1% and 84.5%) and 34.4% and 72.2% on PYPIBUGS (vs. 33.5% and 72.0%). As discuss in Sec. 5.2 finding the cause of the "spurious" warnings is important future work.
In qualitative analysis (Sec. 5.2).
Per- Fer-Bug Evaluation To better understand which bugs are hard detect, break down the results the best-performing PYBUGLAB +Aug models on RANDOMBUGS by type bug in Tbl. 3] We observe that incorrect literals some of the hardest bugs detect. Incorrect assignment operators (e.g. = and +=) are easy to detect in RANDOMBUGS, but significantly harder in PYPlBUGS. This may be attributed class imbalance, with simple assignment (=) being the majority class. Detecting if snippet has bug or not seems to be the hardest task; no model achieves accuracy beyond 63%.
.... ...
We note that our experiments, GNNs-based models seem to often outperform GREAT, somewhat contradicting the results of Hellendoorn et al. [13]. We have performed substantial additional experi- ments to investigate and verify these results, cf. Sec. A.2 This may have to do with the performance of these models on long sequences or that the GNN has access to more fine-grained information, instead of relations over the projected token sequences. For example, this could attributed to the lack of syntax and symbol nodes in the representation used in GREAT. Nevertheless, GREAT
7

---

Table 5: Bug distribution (%) in different datasets
Bug Kind
PYPIBUGS
RANDOMBUGs
Selector Samples
Argument Swapping
11.9
8.4
23.8
Wrong Assignment
1.9
8.5
5.3
Wrong Binary Operator
3,4
2.4
2.3
Wrong Boolean Operator
8.1
2.2
6,4
Wrong Comparison Operator
17.1
8.2
7.4
Wrong Literal
3.7
11.6
12.4
Variable Misuse
53.8
58.6
42.5
noticeably better (62.5% vs. 53.8%) at detecting NoBuG and locating wrong binary operators in PYPIBUGS.
Bug Selector Performance To understand how training of the bug selector proceeds, we perform two experiments. In our first experiment, take snapshot of the selector model during training of the PYBUGLAB +Aug (GNN) model every 24 hours, after initial burn-in phase of 12 hours. We then generate 10000 buggy samples using each of these snapshots and then test fixed model on each of these snapshots. The results of this are shown in Tbl. 4 us-
Table 4: Development of Performance on Bug Selector Samples
Training Time
Joint
Loc
Repair
0.5 days
64.2
83.8
72.1
1.5 days
62.5
80.7
72.9
2.5 days
62.0
83.0
69.8
3.5 days
61.7
82.5
69.8
4.5 days
61.9
83.0
69.5
ing a fully trained PYBUGLAB +Aug (GNN) model from
5.5 days
61.1
83.0
68.6
another training run as a fixed model. We conclude that
6.5 days
60.5
78.7
72.4
PyBuLAAB succeeds learning to generate "............................... to observe the selector model trading off "harder-to-localle localizze and "harder-to-
.... .........
find bugs, though
a* fix" properties. Tests on other models show similar trends, confirming the robustness of this result.
In second experiment, compare the distribution of different bug kinds in PYPIBUGS and RANDOMBUGS with the distribution of bugs sampled from the final snapshot of our selector model from above. The results are shown in Tbl. where swapping, use of wrong literals and of assignment operators) are substantially over-represented, whereas mistakes in comparison operators and variable misuse inder-representee This indicates that PYBUGLAB generates hard to find, but not necessarily realistic bugs.
see that number of bugs (argument
Comparison CuBERT Finally, we compare our models to CuBERT [15), which uses a masked language modeling objective to treit BERT-like model and then learns bug detectors specific class of bugs (e.g., wrong binary operators) on top of this pre-trained model. Note that CuBERT detects if a bug exists but does not localize it. For the comparison, we create two sub-datasets of PYP. IBUGS: PYPlBUGS- BUGS-oo--o contains the 501 samples that involve the binary operators supported by CuBERT, and PYPIBUGS- -YPIBuGss-VrMMisssee, which contains the 1278 bugs that involve variable misuses. We complete both of these datasets with 501 (resp. 1278) random NoBuG code samples from our distribution used in CuBERT's s training. Since
RANDOMBUGS, to match the 1:1 buggy/non-buug CuBERT classification models focus on a single bug type, to compare to PYBUGLAB we mask out all code locations that do not correspond to bug that could be detected by the corresponding CuBERT model. We then treat the prediction of the NoBuG location as a "non-buggy" prediction and all other locations "buggy" prediction. For example, for the snippet in Fig. 2 only the locations l2, l11, las 116. and 120 and their corresponding rewrites are considered by PY BUBGAA for the comparison on PYPIBUGS-WroogOP
Tbl. shows the results of comparing the released Cu- BERT snapshots with the
Table 6: Comparison with CuBERT
[5]
CuBERT
PYBUGLAB (GNN)
Prec
Recall
FI
Prec
Recall
FI
PYBUGLAB +Aug GNN model. We observe that PyPIBuGs-VarMinn
PyPIBuGs-
0.764
0.251
0.378
0.730
0.764
0.746
0.632
0.403
0.493
0.740
0.840
0.787
the PYBUGLAB models
have substantially better recall than CuBERT-based models, even though they were trained to de- .x. tect more bug types. When calibrating CuBERT ....e.. to have recall equal to PYBUGLAB, their precision drops substantially, In particular, on PYPIBUGS PYBSSSS-Oooo it is reduced to 0.609, and

---

1 def update roomId,
1 def make_id (name))
title, =ereueest_parans" 3 check_ topck_typeeee (roomId, (rooomd, basestring)
get_ rand_ string (12)
if len (name) <= 22:
4 check_ type FrooIdd basestring)
4 name name [ :2]
5 [...]
5 return name
+
(a) A wrong comparison operator bug (red box) in PYPIBUGS detected and repaired by the GNN
(b) A variable misuse (red box) caught in open- project. GNN PYBUGLAB +Aug suggests to rewrite roomId title. The fixing pull request is found here
PyBuGLAB +Aug models.
Figure 3: Bugs found by PYBUGLAB. Snippets reformatted and abbreviated to fit figure.
on MarMuuss,,,,,s,e, it reduced to 0.613; in both cases, PYBUGLAB outperforms CuBERT substantially.
5.2 Qualitative Inspection of Raised Warnings
We now take a qualitative look at the raised warnings raised by PYBUGLAB. As example, Fig. 3a shows sample of PYPIBUGS where the developer used an incorrect comparison operator. Once pointed to it, it is clear to a human that the truncation statement line has no effect (under the reasonable assumption that name is string), and that a different comparison operator (>) is necessary.
To gain an understanding of the performance of PYBUGLAB on realistic data, we performed an in- depth analysis of the cases flagged as bugs by our best-performing model on the code found within the 4k top PyPI packages. We observed a mixture of false positives with few previously unseen real-life bugs, matching the quantitative results in Tbl. 3 First, find that the majority of the false positives are "incorrect literal" detections. This suggests that learning to detect such bugs is a hard problem. Furthermore, many literals serve default "configurations" (e.g. the number of retries for a network request) and different values are not bugs. We posit that a large percentage of literal replacements the selector learns make fall in this category.
We also found that some repairs suggested by the model actually produce semantically equivalent code. For example, the model lacks knowledge that two variables refer to the same object in memory (aliasing), and so attempts to "repair" variable misuse bugs by switching between these. Other examples includes checking the return values of standard functions such Python's 's str. find, which returns -1 if the query string is not found. In such cases, PYBUGLAB often suggested to rewrite if x find (y) <= -1 to if x. x.find(y) (y) == -1, which makes no difference in practice. These false negatives can be attributed to the fact that the bug selector model considers such changes introducing bugs, even though they
actually changing behavior. This suggests that for better results, the rewrite rules need to ensure that the rewrites are not semantics-preserving and represent bugs.
Finally, some reported issues were sufficiently complex that it took us (the human authors) a couple of minutes of thought to conclude that warning is spurious. Simultaneously, there some warnings that are "obviously" incorrect to us, but the reasons why the neural models raise them is unclear. This highlights the importance of research on explainability techniques along with better ways calibrate model confidence. The fact that selectors may introduce spurious "bugs" may also be affecting how the detector model learns. Ideas that have appeared in reinforcement learning, such the one of Dennis et al. [8), may allow models to improve their performance in spite of spurious bugs.
Overall, only 19 of the 1000) reported warnings, were found to be real-life bugs, Of these 19, we reported 11 on GitHub (6 already merged, 5 pending approval). See Appx. Gfor details. 3 other bugs had already been fixed between the version PYBUGLAB processed and the current version or the project was deprecated, whereas another 5 bugs are minor and we decided not to report them. One of the detected bugs is shown in Fig. 3b) Overall, most of the detected bugs appear within unit tests, logging, or exception handling, possibly because bugs there do not impact the core functionality of project. However, given the number of such bugs we collected in PYPIBUGS, we believe that such bugs arise equally often in other code, but that they are detected and fixed more quickly.

---

Although our analysis only forms a lower bound on the precision of PYBUGLAB and related methods, it suggests that there is still ample room for future improvements towards making machine learning- based bug detection and repair practically useful.
6 Related Work
Detecting bugs in source code has been researched since the early days of computing. Traditionally, bug detection tackled as a formal task, where any code that cannot be proved to satisfy some correctness property may contain bug. This is essential for security- and safety-critical bugs, but not for other - -err-equie equally common - bugs. In the last decade, software engineering and programming language research have increasingly realized ambiguous information within code (e.g. variable names, comments) contains valuable information and using this information can yield valuable results (2]. The main premise is that patterns in source code, such patterns in names, control, and data flow can be informative. This information can also be exploited to detect some bugs. For example, Ray et al.
24 noted that even simple language models tend to assign lower probability to buggy code.
Multiple static analysis methods have been researched that combine some form of data-oriented bug detection. This ranges from language model-based tools, such as the early work of Wang et al. 32 to specification-miniin
tools such as the work of Eberhardt et al. [0O. BUGLAB is related to DeepBugs [22] which uses an MLP over a limited window of code tokens and train separate models to detect wrong operators, operands, and argument swappings. BUGLAB opts for a more structured representation of code and single model. Allamanis et al. B Vasic al. 300 Hellendoomn et al. [3] tackle variable misuse bugs (one of the kinds of bugs included in PY PYBUGLAB) but either by randomly introducing the bugs in code or using Cloze-like test. Instead, BUGLAB opts for self- supervised approach and tackles a broader range of bugs. Concurrently to this work, Patra and Pradel 21 showed an alternative method for learning to generate realistic bugs. Dinella et al. 9 learn supervised sequential model that performs graph transformations that replicate small edits in code (refactoring, introducing functionality, bug fixing, etc.). Their model Hoppity could serve learnable rewrite operation in BUGLAB in future work. Dynamic analysis methods have also been researched with promising results (31. but collecting representative dynamic traces over a diverse set of programs at scale (e.g. from the top Python packages used in this work) is practically impossible.
BUGLAB is related to ideas around supervupe learning recently explored in deep learning, computer vision, and NLP. In our case, we aim to train a bug detection model without using training data from real-life bugs. BUGLAB resembles ELECTRA (6), with the important difference that the rewrites to the input code go beyond single token replacement that need to respect strict constraints of programming languages (syntax, variable scopes) and the model is directly used for bug detection, rather than for pre- pre-training. The main BUGLAB objective [Eq. I also resembles GANs 12] with the exception that the objective is non-differentiable (introducing a bug alters the discrete data representation). the selector is a structured probabilistic code rewriting model, and that we are mainly interested in the bug detector (analogous to the discriminator) rather than the selector.
Discussion and Conclusions
Learned program analyses offer the promise to improve how develop software. They also offer great opportunity to study machine learning models that combine formal and probabilistic reasoning. Towards achieving these we presented BUGLAB, a self-supervised approach for learning program analyses, that improves upon bascline methods and detects bugs in real-life code. We also empirically show the limitations of existing bug-detecting machine learning methods, which suffer from imprac- tical false- possitvvv rates. Importantly, we show the large gap of performance of existing methods on corpora randomly inserted bugs commonly used in prior work and real-life bugs.
Acknowledgements
We want to thank Sebastian Nowozin and Marwin Segler for helpful discussions, Marwin Segler for comments on a draft of this work, and the anonymous reviewers for useful questions and suggestions. Finally, we would like to thank the contributors to the following open-source tools used: PyTorch 20)
PyDriller (27), MessagePack
LibCST Jedi
Kubernetes
Helm
10

---

References
[1] M. Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pages 143-153, 2019.
[2] M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4) 2018.
[3] M. Allamanis, M. Brockschmidt, and M. Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations (ICLR), 2018.
[4] M. Allamanis, E. Barr, S. Ducousso, and 7. Gao. Typilus: Neural type hints. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 91-105. Association for Computing Machinery (ACM), 2020.
[5] Bachlechner, B. P. Majumder, H. H. Mao, W. Cottrell, and J. McAuley. Rezero is all you need: Fast convergence at large depth. arXiv preprint arXiv:2003 04887, 2020.
[6] K. Clark, M.-T. Luong, Q V. Le, and C. D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv preprint vrXiv20033.05555 2020.
[7] M. Cvitkovic, B. Singh, and A. Anandkumar. Deep learning code with unbounded vocabulary. In Machine Learning 4 Programming, 2018
[8] M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine. Emer- gent complexity and zero-shot transfer via unsupervised environment design. arXiv preprint arXiv:20/2.02096 2020.
[9] E. Dinella, H. Dai, Z. Li, M. Naik, L. Song, and K. Wang. Hoppity: Learning graph transfor- mations detect and fix bugs in programs. In International Conference on Learning Represen- tations, 2019.
[10] J. Eberhardt,
S. Steffen, V. Raychev, and M. Vechev. Unsupervised learning of API aliasing specifications. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Lan- guage Design and Implementation, pages 745-759, 2019.
[11] Gilmer, S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International Conference Machine Learning, pages 1263-1272, 2017.
[12] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint 1406. 2014.
[13] V. J. Hellendoorn, C. Sutton, R. Singh, P. Maniatis, and D. Bieber. Global relational models of source code. In International Conference on Learning Representations, 2020.
[14] R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 654-665, 2014.
[15] A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning (ICML), 2020.
[16] R.-M. Karampatsis and C. Sutton. How often do single-statement bugs occur? the ManySStuBss4 dataset. In Proceedings of the 17th International Conference on Mining Soft- ware Repositories, pages 573-577, 2020.
[17] Katz. Librarics.io Open Source Repository and Dependency Metadata, Jan. 2020. URL https://dois org/10..//.0 5281/zenoo 3626071
[18] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated graph sequence neural networks. In International Conference Learning Representations (ICLR), 2016.
11

---

[19] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In Interna- tional Conference on Learning Representations (ICLR), 2017.
[20] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. 2017.
[21] J. Patra and M. Pradel. Semantic bug seeding: learning-base based approach for creating realistic bugs. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 906-918, 2021.
[22] M. Pradel and K. Sen. DeepBugs: A learning approach to name-based bug detection. Proceed- ings of the ACM on Programming Languages, 2(00PSLLA)1-25, 2018.
[23] G. Ramakrishnan, J. Henkel, Z. Wang, A. Albarghouthi. S. Jha, and T. Reps. Semantic robust- ness of models of source code. arXiv preprint arXiv:2002.0:04: 2020.
[24] B. Ray, V. Hellendoorn, S. Godhane, Z. Tu, A. Bacchelli, and Devanbu. On the "naturalness" of buggy code. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pages 428 4398433..M0 IEEE, 2016.
[25] V. Raychev, M. Vechev, and A. Krause. Predicting program properties from Big Code. In Principles of Programming Languages (POPL), 2015.
[26] A. Rice, E. Aftandilian, C. Jaspan, E. Johnston, M. Pradel, and Y. Arroyo-Paredes, Detecting argument selection defects. Proceedings of the ACM on Programming Languages, ((OOPSLA): 1-22, 2017.
[27] D. Spadini, M. Aniche, and A. Bacchelli. PyDriller: Python framework for mining soft- ware repositories. In Proceedings of the 2018 26th ACM Joint Meeting on European Soft- ware Engineering Conference and Symposium on the Foundations of Software Engineering ESEC/FSE 2018, pages 908-911, New York, New York, USA, 2018. ACM Press. ISBN 9781450355735. doi: 00.1145/22602243266558. cfm7doid-3236024 3264598
URL http://dl. acm orggcitatioon
[28] D. Tarlow, S. Moitra, A. Rice, Z. Chen, P.-A. Manzagol, C. Sutton, and E. Aftandilian. Learning to fix build errors with Graph2Diff neural networks. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, pages 19-20, 2020.
[29] H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.172:: 2021.
[30] M. Vasic, A. Kanade, P. Maniatis, D. Bieber, and R. Singh. Neural program repair by jointly learning to localize and repair. arXiv preprint arXiv: 1XXiv:9004.00 000///94.0777 17720,0 2019.
[31] K. Wang and Z. Su. Blended, precise semantic program embeddings. In Proceedings of the 41stACM SIGPLAN Conference on Programming Language Design and Implementation, pages 121-134, 2020.
[32] S. Wang, D. Chollak, D. Movshovitz-Attia and L. Tan. Bugram: bug detection with n-gram lan- guage models. In Proceedings of the 31 IEEE/ACM International Conference on Automated Software Engineering, pages 708-719, 2016
[33] J. Wei, M. Goyal, G. Durrett, and I. Dillig. LambdaNet: Probabilistic type inference using graph neural networks. arXiv preprint arXiv:2005.02161, 2020.
12


Reviews:
[{"review_id": "uokunvYoEee", "reviewer": "Reviewer_XqDo", "summary": "Authors propose self-supervised learning approach that trains two models: one to detect and repair bugs in code and another one to generate code with difficult bugs. They use their trained model to improve results on baseline test dataset and to find bugs in real code.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors propose self-supervised learning approach that trains two models: one to detect and repair bugs in code and another one to generate code with difficult bugs. They use their trained model to improve results on baseline test dataset and to find bugs in real code. I think that the self-supervised learning approach and the resulting models it produces is valuable for automated bug detection and repair. The evaluation of the models on the real-life bugs is also valuable. The evidence that the tool can find bugs in open-source packages is also positive.\n\nThe paper should compare their model not just with a baseline, but also with other bug detection and repair models.\n\nWhen run on the real-life software, the tool suffers high false positive rate. For the tool to be useful, this needs to be addressed in future work.\n\nLine 30 - what do you mean by \"shows improved performance over training with randomly inserted bugs\"? \nLine 33 - Please state what are the false positive rates. \nLine 127 - \"other\" is repeated \nLine 274 - should be \"seems to be\" \nLine 382 - what limitations did you identify? It is not clear what you mean.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors address the limitations of their work in section5.2. I think this covers the limitations well. \nI agree with authors that the work does not have obvious negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "txs8Uob9F3k", "reviewer": "Reviewer_3FwM", "summary": "This paper proposes an approach for supervised learning of bug detection and repair called BUGLAB and it includes a detector model to detect and repair bugs and a selector model to create buggy code for detection. By conducting experiments on two different datasets RANDOMBUGS and PYPIBUGS, BUGLAB shows great improvement compared with the baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-organized and the analysis of the experimental results seems very sufficient. Furthermore, it provides a novel approach with self-supervised techniques to detect and repair bugs. Compared with current approaches that mostly based on a supervised learning approach for vulnerability-related tasks, the approach is interesting.  Specifically, it contains two different models, the first one is the detector model to detect buggy code, and the second one is the bug selector model to introduce some bugs in the original code by the self-defined rewrite rules.  It further contains some operations for data augmentation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- How to ensure the raw samples have no bugs and the operations in Section 4.1 can indeed introduce bugs, for example in Variable Misuse, variable-swapping rewrites cannot assure the bugs will be introduced if a,b,c are the same value in the extreme conditions? Section 5.2 also mentioned this problem.\n- I cannot agree that all the related approaches rely on datasets that cannot represent the characteristics of bugs in real code mentioned in the Introduction section, there are also some works that focus on vulnerability detection with the real vulnerability data such as Devign [1].\n- In the experiments,  the bug selector selects k=5 bugs to introduce, is this hyperparameter has a significant effect on the final performance, it is better to provide some analysis.\n- if I understand correctly, comment deletion will be removed automatically during the graph construction, so is it necessary for a strategy of semantics-preserving rewrites?\n- Some latest pretrained baselines should be included such as CodeBERT[2] and GraphCodeBERT[3].\n- The paper mentioned that PYPIBUGS has published at an anonymous location, I want to further confirm that the code and the related dataset will be public?\n\n\n\n[1] Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks.\n[2] CodeBERT: A Pre-Trained Model for Programming and Natural Languages.\n[3] GraphCodeBERT: Pre-training Code Representations with Data Flow.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "T2lxXLtZ_--", "reviewer": "Reviewer_6URn", "summary": "This paper describes a self-supervised ML-based bug fixer that uses a combination of a bug localization and repair model, with a bug injection model. The two models are identical in architecture: they select a program location to mutate, and then one of a few possible mutations and its parameters (e.g., swapping a binary operator with another). Training is done concurrently from a dynamic corpus of buggy and bug-free examples. The bug fixer is trained on detecting bug-free examples, or fixing buggy ones. The bug injector is trained on selecting the location and type of bug to inject to maximize the bug fixer's loss.\n\nThe initial corpus is also augmented via a number of \"semantics preserving\" rewrites of examples (e.g., variable renaming).\n\nThe approach is shown to outperform bug fixers that don't have the benefit of a bug injector, as well as those that don't use data augmentation. Furthermore, the approach was shown to find previously-undiscovered bugs in open-source code, as well as perform better than previous SOTA models on a manually curated test dataset of bugs.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overall\n\nOriginality: The idea of using a localization+repair model as both a bug injector and a bug fixer is neat and unusual (except for adversarial related work). Augmentation with buggy or \"semantics-preserving\" rewrites is otherwise not that new (e.g., the Ramakrishnan et al., and Vasic et al. papers cited used semantics-preserving rewrites as well, but also \"Generating Adversarial Computer Programs using Optimized Obfuscations\" from ICLR21).\n\nQuality: The submission is well described, but the actual benefit of the learned bug selector is not adequately evaluated in my opinion. Baselines are well studied. \n\nClarity: The submission is reasonably well written, but it leaves important information in the appendix. The training regime is very very briefly described in the main paper. The evaluation is very well presented.\n\nSignificance: The idea of a form of \"curriculum training\" where hard-to-fix bugs are injected into the training dataset is neat and seems important for this (and many) code-understanding model. The results seem positive and advance the state of the art. A new dataset of simple Python bugs from actual open-source projects is a valuable contribution and is likely to be used by many future papers in this space. New previously-undiscovered bugs were found in open-source projects!\n\n\n# Comments\n\nThis was a fun paper to read! The teacher/learner formulation was interesting, reminiscent of adversarial training, and not often used in this space (there are a handful of adversarial robustness papers for code, but that's about it). As such, this was an intriguing approach.\n\nWhat's more, results matter: the authors found a few real bugs in open-source software, some of which were confirmed. To my knowledge, similar neural bug-finding systems haven't revealed many real bugs, and tend to focus on synthetic datasets, so this is a great new result from this work.\n\nMy main concern is that, although the experimental results seem positive and encouraging, why exactly they look like that is not very clear from the experimental evaluation or the design. In part, that's because the bug fixer and the bug injector are operating asynchronously, rather than in controlled fixed-length generations. I'm guessing this was to improve efficiency? The authors don't provide justification for this design choice. However, this gives little understanding of how the two components evolve, and how to compare them to baselines and ablations.\n\nFor example, it's not clear how the bug injector's performance changes over time, or how diverse the samples it generates are. Are they harder over time? Or is this just a matter of adding more noise to the training data for the bug fixer? For example, the difference between the Random Selector and PyBugLab on table 1 is quite small. If we could compare the two on the same number of generations of the same length for the fixer and injector, perhaps we could draw some solid conclusions. But as they are, maybe the neural injector run faster or got lucky or who knows? Or perhaps the random selector shot itself in the foot by generating many more rewritten examples? Given how close the results are, and given that we see no confidence intervals from multiple experiment repetitions, it's very hard to believe that this is a repeatable, stable result.\n\nWhile we're at it, what exactly is the difference between the Random Selector and Supervised rows of Table 1? Isn't that just changing the initial data generator to create more buggy examples than bug-free examples? Does this serve any other purpose? \n\nI'd recommend that the authors make the experiment a bit more controllable, and show how the bug fixer's performance on the test dataset evolves over time with the learned injector, the random injector, and the original dataset.\n\nAnother related question is why we don't see a line in that table for the semantic rewrites. This seems to be a standard tool, and appears to be working well, so it would be useful to understand how the no-injector model, or the random-injector model would work with the static, ahead-of-time data augmentation.\n\nSo overall, it's not obvious to me that the learned bug injector actually does something, and if it does, whether it's big. Maybe it's the data augmentation (semantics preserving or synthetic bug injection) that does the trick. I'd like to understand better the the contribution of the learned bug injector is significant.\n\nA related concern is sample efficiency. Because the training mechanism is rather ambiguous due to its asynchronous execution, we don't really know what the training dataset size is, or how many actual samples the bug detector saw before converging to its best behavior. Would training it longer but with less fancy an augmentation mechanism have achieved similar results?\n\nA smaller concern is about the actual model architecture. Having to manually define scoring functions per concrete rewrite seems super awkward and unscalable. This is most glaring in the case of literal repair, where to limit the number of candidate rewrites that must be scored, the possible repairs is limited to just a few (-2, -1, ...). I wonder if a generative rewrite approach (perhaps like AnyCodeGen from ICML 20 or \"Generative code modelling with graphs\" from ICLR19) might be a more scalable approach, perhaps harder to train of course.\n\nI liked the augmentation approach used, and it seems to help. However, calling it \"semantics preserving\" is a bit of a misnomer. Python has very intrusive reflection capabilities. Changing docstrings and renaming variables changes the semantics of the program. For example, if you rename variables in this program, you will cause an assertion error:\n\n```\nMY_VARIABLE = 42\ndef f():\n  assert globals()['MY_VARIABLE'] == 42\n```\n\nCertainly there are good reasons not to write such a Python program, but the fact remains that your rewrites are not semantics preserving. Please don't call them that.\n\nI was wondering, why isn't your \"semantics-preserving rewrite\" also an operation that the bug selector can train on and introduce in the same way as the bug injections? Arguably, such a rewrite might also throw off the detector, so trying to find such hard rewrites would be valuable for the selector. Why are the two (bug injections, benign rewrites) treated differently?\n\nI found the comparisons to strong baselines well done. I was particularly impressed with the validation against the original datasets for the GREAT re-implementation (in the appendix). I was originally concerned with your finding that the GNN does better than GREAT, despite the findings in that paper. It was good to read the steps you took to ensure you weren't misrepresenting GREAT (although of course, it's still a mystery). A brief mention in the main paper that you did all that would have been nice, since without at least a pointer, the reader is left suspicious.\n\nThe comparison to the CuBERT baseline was also helpful in contextualizing the benefits of your approach. I was curious why you didn't also show the performance of your model on the CuBERT benchmarks (for the same tasks you chose).\n\n\n# Questions\n\n1. For the same number of total samples seen by the bug detector (ignoring the selector), how do the various approaches compare?\n2. Is augmentation with semantics-preserving rewrites by itself able to give better results?\n3. Why are rewrites and bug injections treated differently? Aren't they both types of augmentation?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No issues found.", "review_text": "", "extra_scores": {"ethical_concerns": "No issues found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a novel approach to train a bug detector by co-training a bug injection procedure together with the bug detector. This is an interesting idea, and while the resulting bug detector has a high number of false positives, it was able to find new bugs in PyPI packages. (19 of 1000 reported bugs turned out to be real bugs). 

The bug injection procedure is based on transformations that are meant to introduce bugs in the code at a known location; the transformations are hand-crafted, but the model learns where to apply them to make the bugs hard to find. The system also relies on semantics preserving transformations to introduce additional variety to the set of non-buggy programs. One limitation of the approach is that the bug introducing transformations may not actually be introducing bugs in all cases (for example, if they swap two variables that happen to be aliases of each other), and the semantics preserving transformation may not be semantics preserving (as one of the reviewers pointed out). 

Overall, this is a strong paper; it presents a novel and interesting idea, and while there was a desire from some reviewers for additional baselines, the evaluation is still fairly convincing. One of the reviewers surfaced a paper that I think is relevant and should be cited ("Generating Adversarial Computer Programs using Optimized Obfuscations"), but that paper is quite different does not detract from the novelty of this work.

Decision: Accept (Poster)