Self-Supervised Bug Detection and Repair

1 Introduction . Detecting and repairing bugs in source code requires strong reasoning skills over formal structures ( e.g . data and control flow ) and ambiguous information ( e.g . identifier names , coding idioms , and comments ) . Traditional program analyses are able to detect critical bugs through formal reasoning and combinatorial search , but need to be manually coded by experts . That is a lengthy and costly process , which misses the opportunity to use ambiguous information pervasive within code . Towards broadening the applicability of such methods , and utilizing ambiguous information , deep learning-based bug detection methods are being investigated [ 22 , 3 , 13 ] . These methods have the potential to further improve the engineering of software we rely on every day . However , many challenges in the area remain open , such as creating robust bug detection and repair methods that cover a wide range of common bugs in the absence of large supervised training corpora . Existing work focuses on randomly inserted bugs [ 22 , 13 ] , Cloze test proxy tasks [ 3 ] , corpora of small code edits that may contain bugs [ 9 ] or build errors [ 28 ] . All these approaches rely on datasets of very limited size or ones known not to be representative of the characteristics of bugs found in real code . In this work , we propose BUGLAB , a self-supervised approach that trains robust bug detectors by co-training a bug selector that learns to create hard-to-detect bugs ( Sec . 2 ) . For example , for a given code snippet with two well-named variables , a variable misuse bug may be easy to detect and repair , whereas an incorrect comparison operator might be significantly harder to identify . We propose a neural architecture for BUGLAB ( Sec . 3 ) and implement it for Python ( Sec . 4 ) . Our implementation considers four broad classes of seemingly simple , yet hard-to-detect bugs and shows improved performance over training with randomly-inserted bugs on PYPIBUGS , a new , manually curated test set of 2374 real-life bugs ( Sec . 5 ) . Furthermore , we tested our trained models on popular open-source Python packages and identified 19 previously unreported bugs , though false positive rates of ∼ 98 % remain impractical . We hope that creating machine learning methods that can detect these bugs early and assist developers will speed up software development and allow engineers to deliver more robust software . We release PyPIBugs and our code at https : //github.com/ microsoft/neurips21-self-supervised-bug-detection-and-repair . ∗Work done while at Microsoft Research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 2 Self-Supervised Bug Detection . In this section , we first introduce the concept of code rewriting , and then use it to define BUGLAB as a framework for self-supervised learning of bug detection and repair . Code Rewriting Rewriting is common within compilers and their optimizations , test-driven searchbased bug repair tools , mutation testing , and refactoring tools . Rewrites can be semantics-preserving ( e.g . renamings of local variables ) , or semantics-altering ( e.g . replacing > = by ! = ) . Let S denote the set of all syntax trees ( not necessarily rooted in the start symbol of the language grammar ) . Syntax tree locations ` ∈ { } ∪ N∗ in a syntax tree s ∈ S are recursively defined , where s| = s and s| ` for ` = ` ′◦i is the i-th child of s| ` ′ ( i.e . s| ( 2,3 ) denotes the third child of the second child of s ) . We define a rewrite rule ρ = ( mρ , tρ ) as a pair of a matching function mρ : S → { true , false } and a transformation function tρ : S → S. The matching function mρ ( s ) yields true iff the rule ρ is applicable at the root of a subtree s. The transformation function can be applied to obtain a transformed syntax tree . For convenience , we define tρ ( s ) = s iff mρ ( s ) = false . We then write ρ ( s ) to indicate the modification of a syntax tree s using ρ when possible , and otherwise the identity function . For reversible rewrite rules ρ , we denote the inverse rule as ρ−1 such that ρ−1 ( ρ ( s ) ) = s holds . We discuss concrete rewrite rules ρ in Sec . 4 . Given a set of rewrite rules R we define the set of “ potential rewrites ” in a syntax tree s as RRs = { 〈 ` , ρ〉 | ρ ∈ R , ` location in s , mρ ( s| ` ) = true } . For each tuple 〈 ` , ρ〉 ∈ RRs , we use s′ = s [ ρ ] ` to denote the new syntax tree obtained by applying ρ at location ` of s. In BUGLAB , we train models that use rewrites from RRs to insert and repair bugs . We will discuss such neural models in Sec . 3 . BUGLAB In BUGLAB , we are interested in selfsupervised training of a robust bug detector model Dθ with parameters θ on an unannotated codebase C. Let R be a set of rewrite rules2 that allows to insert and repair bugs . We train Dθ to be able to recognize the “ hardest ” possible rewrites that could be applied on our codebase C. For this , we consider the loss LDθ of Dθ on a rewritten code snippet s [ ρ ] ` , for which the model needs to predict the repairing rewrite 〈 ` , ρ−1 〉 . Formally , we want to minimize the objective Es∼C [ max 〈 ` , ρ〉∈RRs LDθ ( s [ ρ ] ` , 〈 ` , ρ−1 〉 ) ] . However , for any useful detector the set of rewrites RRs is commonly very large or unbounded and computing the maximum over all 〈 ` , ρ〉 ∈ RRs is practically intractable . To address this , BUGLAB introduces a bug selector model Sφ ( with parameters φ ) , whose goal is to approximate the intractable max〈 ` , ρ〉∈RRs LDθ ( · ) . We can then sample rewrites from Sφ instead of computing the maximum . We denote this as 〈 ` , ρ〉 ∼ Sφ ( s ) and the overall BUGLAB training objective can be written as a min-max optimization problem : max φ min θ Es∼C [ E〈 ` , ρ〉∼Sφ ( s ) [ LDθ ( s [ ρ ] ` , 〈 ` , ρ−1 〉 ) ] ] . ( 1 ) The two models S and D in BUGLAB are “ symmetric ” in the sense that they both predict rewrites on code snippets , and only differ in their objectives — one aiming to introduce bugs and one aiming to repair them . In practice , we can and do use the same architecture to model both S and D , which we will discuss in the next section . At test time , we discard S and only use the trained detector D to locate and repair bugs . 2In this work we assume that R contains a special “ identity ” rewrite rule ρ∅ that does not change the code . 3 Neural Models . In this section , we discuss how we represent code in BUGLAB and the neural models we use to learn how to rewrite code in the selector and detector models . Code Representation We consider source code as a set of entities vi ∈ V which relate to each other with a set of typed relations ek ∈ E , where a relation ek = ( vi , r , vj ) denotes a relationship between entities vi and vj with type r. The entities and relations can be thought as a heterogeneous graph G = ( V , E ) . The choice of code entities and their relationships is a form of high-level feature extraction . We discuss concrete entities and relationships for Python in Sec . 4 . We also define a projection function Ptok that accepts V and E and returns a sequence Vtok of the token entities in V with the nodes appearing in relations in E deterministically mapped to elements of Vtok , i.e . Etok = { ( p ( vi ) , r , p ( vj ) ) } , where p maps the entities in V to Vtok . Ptok will be used for relational transformer models . To learn a neural representation of the code entities vi , first we define an embedding function e ( vi ) which maps the content of each entity to an initial D-dimensional representation . Throughout this work — similar to Allamanis et al . [ 4 ] and other previous work — we deterministically split the string representation of each node into subtokens ( e.g. , fooBar is split into foo and bar ) , embed them through a learned embedding matrix , and use max pooling to get a single vector . We then “ contextualize ” the entity representations within G using one of two models : a MLP-based GNN model with max message aggregation and the GREAT relational transformer of Hellendoorn et al . [ 13 ] over the token sequence and relations Vtok , Etok = Ptok ( V , E ) . GREAT uses both positional encodings and the projected relations in Etok . See Appx . A for detailed architecture descriptions . Other models to compute entity representations can be used , but were not explored in this work . We use r ` to denote the computed vector representation of the entity at location ` , independent of the model used to produce it . We use these representations to define our code rewriting models . Probabilistic Code Rewriting Models Both bug selection and bug detection require to model the probability of applying a specific rewrite at a location in a code snippet s , either to introduce or repair a bug . For this , we factorize this task into localization and rewrite-given-location models , i.e . p ( 〈 ` , ρ〉 | s , RRs ) = ploc ( ` | s , RRs ) prew ( ρ | ` , s , RRs ) . ( 2 ) We model ploc as a probability distribution over the relevant locations { ` | 〈 ` , ρ〉 ∈ RRs } ∪ { NoBug } , where NoBug is a special location used to indicate that the code is not buggy . In practice , we implement this similar to a pointer net [ 19 ] using the representations r ` ( see Appx . A for details ) . To select rewrites , we use rewrite type-specific learnable rule score functions wρ ( r ` , Mρ ( s , ` ) ) . This function maps a vector representation of an entity r ` and potential additional metadata onto a scalar score . The rule-specific metadata Mρ ( s , ` ) is defined for some rewrites , e.g . containing representations of other entities that could be used in the location ` . We will discuss three concrete rule score functions in Sec . 4 . The rewrite probability distribution prew is then modeled by a softmax over the scores of all applicable rewrites at a target location ` , i.e . prew ( ρ | ` , s , RRs ) = softmax 〈 ` , ρ′〉∈RRs ( wρ′ ( r ` , Mρ′ ( s , ` ) ) ) .

Summary: This paper proposes a self-supervised approach that trains bug detectors by co-training a bug selector that learns to create bugs. A high-level framework is formulated, and then a python implementation using graph neural networks and relational transformers is provided. Experiments on a manually curated bug dataset demonstrate its effectiveness.