Self-Supervised Bug Detection and Repair

1 Introduction . Detecting and repairing bugs in source code requires strong reasoning skills over formal structures ( e.g . data and control flow ) and ambiguous information ( e.g . identifier names , coding idioms , and comments ) . Traditional program analyses are able to detect critical bugs through formal reasoning and combinatorial search , but need to be manually coded by experts . That is a lengthy and costly process , which misses the opportunity to use ambiguous information pervasive within code . Towards broadening the applicability of such methods , and utilizing ambiguous information , deep learning-based bug detection methods are being investigated [ 22 , 3 , 13 ] . These methods have the potential to further improve the engineering of software we rely on every day . However , many challenges in the area remain open , such as creating robust bug detection and repair methods that cover a wide range of common bugs in the absence of large supervised training corpora . Existing work focuses on randomly inserted bugs [ 22 , 13 ] , Cloze test proxy tasks [ 3 ] , corpora of small code edits that may contain bugs [ 9 ] or build errors [ 28 ] . All these approaches rely on datasets of very limited size or ones known not to be representative of the characteristics of bugs found in real code . In this work , we propose BUGLAB , a self-supervised approach that trains robust bug detectors by co-training a bug selector that learns to create hard-to-detect bugs ( Sec . 2 ) . For example , for a given code snippet with two well-named variables , a variable misuse bug may be easy to detect and repair , whereas an incorrect comparison operator might be significantly harder to identify . We propose a neural architecture for BUGLAB ( Sec . 3 ) and implement it for Python ( Sec . 4 ) . Our implementation considers four broad classes of seemingly simple , yet hard-to-detect bugs and shows improved performance over training with randomly-inserted bugs on PYPIBUGS , a new , manually curated test set of 2374 real-life bugs ( Sec . 5 ) . Furthermore , we tested our trained models on popular open-source Python packages and identified 19 previously unreported bugs , though false positive rates of ∼ 98 % remain impractical . We hope that creating machine learning methods that can detect these bugs early and assist developers will speed up software development and allow engineers to deliver more robust software . We release PyPIBugs and our code at https : //github.com/ microsoft/neurips21-self-supervised-bug-detection-and-repair . ∗Work done while at Microsoft Research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 2 Self-Supervised Bug Detection . In this section , we first introduce the concept of code rewriting , and then use it to define BUGLAB as a framework for self-supervised learning of bug detection and repair . Code Rewriting Rewriting is common within compilers and their optimizations , test-driven searchbased bug repair tools , mutation testing , and refactoring tools . Rewrites can be semantics-preserving ( e.g . renamings of local variables ) , or semantics-altering ( e.g . replacing > = by ! = ) . Let S denote the set of all syntax trees ( not necessarily rooted in the start symbol of the language grammar ) . Syntax tree locations ` ∈ { } ∪ N∗ in a syntax tree s ∈ S are recursively defined , where s| = s and s| ` for ` = ` ′◦i is the i-th child of s| ` ′ ( i.e . s| ( 2,3 ) denotes the third child of the second child of s ) . We define a rewrite rule ρ = ( mρ , tρ ) as a pair of a matching function mρ : S → { true , false } and a transformation function tρ : S → S. The matching function mρ ( s ) yields true iff the rule ρ is applicable at the root of a subtree s. The transformation function can be applied to obtain a transformed syntax tree . For convenience , we define tρ ( s ) = s iff mρ ( s ) = false . We then write ρ ( s ) to indicate the modification of a syntax tree s using ρ when possible , and otherwise the identity function . For reversible rewrite rules ρ , we denote the inverse rule as ρ−1 such that ρ−1 ( ρ ( s ) ) = s holds . We discuss concrete rewrite rules ρ in Sec . 4 . Given a set of rewrite rules R we define the set of “ potential rewrites ” in a syntax tree s as RRs = { 〈 ` , ρ〉 | ρ ∈ R , ` location in s , mρ ( s| ` ) = true } . For each tuple 〈 ` , ρ〉 ∈ RRs , we use s′ = s [ ρ ] ` to denote the new syntax tree obtained by applying ρ at location ` of s. In BUGLAB , we train models that use rewrites from RRs to insert and repair bugs . We will discuss such neural models in Sec . 3 . BUGLAB In BUGLAB , we are interested in selfsupervised training of a robust bug detector model Dθ with parameters θ on an unannotated codebase C. Let R be a set of rewrite rules2 that allows to insert and repair bugs . We train Dθ to be able to recognize the “ hardest ” possible rewrites that could be applied on our codebase C. For this , we consider the loss LDθ of Dθ on a rewritten code snippet s [ ρ ] ` , for which the model needs to predict the repairing rewrite 〈 ` , ρ−1 〉 . Formally , we want to minimize the objective Es∼C [ max 〈 ` , ρ〉∈RRs LDθ ( s [ ρ ] ` , 〈 ` , ρ−1 〉 ) ] . However , for any useful detector the set of rewrites RRs is commonly very large or unbounded and computing the maximum over all 〈 ` , ρ〉 ∈ RRs is practically intractable . To address this , BUGLAB introduces a bug selector model Sφ ( with parameters φ ) , whose goal is to approximate the intractable max〈 ` , ρ〉∈RRs LDθ ( · ) . We can then sample rewrites from Sφ instead of computing the maximum . We denote this as 〈 ` , ρ〉 ∼ Sφ ( s ) and the overall BUGLAB training objective can be written as a min-max optimization problem : max φ min θ Es∼C [ E〈 ` , ρ〉∼Sφ ( s ) [ LDθ ( s [ ρ ] ` , 〈 ` , ρ−1 〉 ) ] ] . ( 1 ) The two models S and D in BUGLAB are “ symmetric ” in the sense that they both predict rewrites on code snippets , and only differ in their objectives — one aiming to introduce bugs and one aiming to repair them . In practice , we can and do use the same architecture to model both S and D , which we will discuss in the next section . At test time , we discard S and only use the trained detector D to locate and repair bugs . 2In this work we assume that R contains a special “ identity ” rewrite rule ρ∅ that does not change the code . 3 Neural Models . In this section , we discuss how we represent code in BUGLAB and the neural models we use to learn how to rewrite code in the selector and detector models . Code Representation We consider source code as a set of entities vi ∈ V which relate to each other with a set of typed relations ek ∈ E , where a relation ek = ( vi , r , vj ) denotes a relationship between entities vi and vj with type r. The entities and relations can be thought as a heterogeneous graph G = ( V , E ) . The choice of code entities and their relationships is a form of high-level feature extraction . We discuss concrete entities and relationships for Python in Sec . 4 . We also define a projection function Ptok that accepts V and E and returns a sequence Vtok of the token entities in V with the nodes appearing in relations in E deterministically mapped to elements of Vtok , i.e . Etok = { ( p ( vi ) , r , p ( vj ) ) } , where p maps the entities in V to Vtok . Ptok will be used for relational transformer models . To learn a neural representation of the code entities vi , first we define an embedding function e ( vi ) which maps the content of each entity to an initial D-dimensional representation . Throughout this work — similar to Allamanis et al . [ 4 ] and other previous work — we deterministically split the string representation of each node into subtokens ( e.g. , fooBar is split into foo and bar ) , embed them through a learned embedding matrix , and use max pooling to get a single vector . We then “ contextualize ” the entity representations within G using one of two models : a MLP-based GNN model with max message aggregation and the GREAT relational transformer of Hellendoorn et al . [ 13 ] over the token sequence and relations Vtok , Etok = Ptok ( V , E ) . GREAT uses both positional encodings and the projected relations in Etok . See Appx . A for detailed architecture descriptions . Other models to compute entity representations can be used , but were not explored in this work . We use r ` to denote the computed vector representation of the entity at location ` , independent of the model used to produce it . We use these representations to define our code rewriting models . Probabilistic Code Rewriting Models Both bug selection and bug detection require to model the probability of applying a specific rewrite at a location in a code snippet s , either to introduce or repair a bug . For this , we factorize this task into localization and rewrite-given-location models , i.e . p ( 〈 ` , ρ〉 | s , RRs ) = ploc ( ` | s , RRs ) prew ( ρ | ` , s , RRs ) . ( 2 ) We model ploc as a probability distribution over the relevant locations { ` | 〈 ` , ρ〉 ∈ RRs } ∪ { NoBug } , where NoBug is a special location used to indicate that the code is not buggy . In practice , we implement this similar to a pointer net [ 19 ] using the representations r ` ( see Appx . A for details ) . To select rewrites , we use rewrite type-specific learnable rule score functions wρ ( r ` , Mρ ( s , ` ) ) . This function maps a vector representation of an entity r ` and potential additional metadata onto a scalar score . The rule-specific metadata Mρ ( s , ` ) is defined for some rewrites , e.g . containing representations of other entities that could be used in the location ` . We will discuss three concrete rule score functions in Sec . 4 . The rewrite probability distribution prew is then modeled by a softmax over the scores of all applicable rewrites at a target location ` , i.e . prew ( ρ | ` , s , RRs ) = softmax 〈 ` , ρ′〉∈RRs ( wρ′ ( r ` , Mρ′ ( s , ` ) ) ) .

Summary: This paper describes an adversarial approach towards detecting and repairing local bugs in human-written software. The approach involves training in tandem two neural networks, the detector, which discovers and repairs bugs in a given piece of code, and the selector, which produces bugs that the detector is unable to detect. At test time, only the detector is used.  The architecture used for each of the networks is similar: first the program is represented by a graph with labeled edges, then each node is embedded using a language-like model (to take into account semantic information in variable names), then embeds the graph using either a GREAT relational transformer or a GNN. The code rewriting model then predicts a distribution over locations to rewrite using a pointer network, and a distribution of rewrites at that given location using a softmax over potential rewrites.  The types of bugs considered are local and shallow bugs such as variable misnamings or use of the wrong operator or literal in an expression. The authors create a corpus of code by crawling pypi and create a corpus of known shallow bugs by looking for version updates that include edits that fit their definition of a shallow bug. They additionally include a corpus of random generated bugs. In general, their approach works better on the random bug corpus than on the real bug corpus, and also works much better at repairing bugs given localization than it does at localization. They are able to achieve better performance than the special purpose CuBERT model using a F1 metric. Additionally, they were able to detect 19 bugs in open source projects, but with a TPR of 0.19% this is probably not yet a productionizable tool in this context.