Neural Program Repair by Jointly Learning to Localize and Repair

1 INTRODUCTION . Advances in machine learning and the availability of large corpora of source code have led to growing interest in the development of neural representations of programs for performing program analyses . In particular , different representations based on token sequences ( Gupta et al. , 2017 ; Bhatia et al. , 2018 ) , program parse trees ( Piech et al. , 2015 ; Mou et al. , 2016 ) , program traces ( Reed & de Freitas , 2015 ; Cai et al. , 2017 ; Wang et al. , 2018 ) , and graphs ( Allamanis et al. , 2018 ) have been proposed for a variety of tasks including repair ( Devlin et al. , 2017b ; Allamanis et al. , 2018 ) , optimization ( Bunel et al. , 2017 ) , and synthesis ( Parisotto et al. , 2017 ; Devlin et al. , 2017a ) . In recent work , Allamanis et al . ( 2018 ) proposed the problem of variable misuse ( VARMISUSE ) : given a program , find program locations where variables are used , and predict the correct variables that should be in those locations . A VARMISUSE bug exists when the correct variable differs from the current one at a location . Allamanis et al . ( 2018 ) show that variable misuses occur in practice , e.g. , when a programmer copies some code into a new context , but forgets to rename a variable from the older context , or when two variable names within the same scope are easily confused . Figure 1a shows an example derived from a real bug . The programmer copied line 5 to line 6 , but forgot to rename object name to subject name . Figure 1b shows the correct version . Allamanis et al . ( 2018 ) proposed an enumerative solution to the VARMISUSE problem . They train a model based on graph neural networks that learns to predict a correct variable ( among all type- correct variables available in the scope ) for each slot in a program . A slot is a placeholder in a program where a variable is used . The model is trained on a synthetic dataset containing a training example for each slot in the programs from a corpus of correct source files , and teaching the model to predict the correct , existing variable for each slot . At inference time , a program of unknown correctness is turned into n prediction tasks , one for each of its n slots . Each prediction task is then performed by the trained model and predictions of high probability that differ from the existing variable in the corresponding slot are provided to the programmer as likely VARMISUSE bugs . Unfortunately , this enumerative strategy has some key technical drawbacks . First , it approximates the repair process for a given program by enumerating over a number of independent prediction problems , where important shared context among the dependent predictions is lost . Second , in the training process , the synthetic bug is always only at the position of the slot . If for example , the program in Figure 1b were used for training , then five training examples , one corresponding to each identifier in a blue box ( a variable read , in this case ) , would be generated . In each of them , the synthetic bug is exactly at the slot position . However , during inference , the model generates one prediction problem for each variable use in the program . In only one of these prediction problems does the slot coincide with the bug location ; in the rest , the model now encounters a situation where there is a bug somewhere else , at a location other than the slot . This differs from the cases it has been trained on . For example , in Figure 1a , the prediction problem corresponding to the slot on line 5 contains a bug elsewhere ( at line 6 ) and not in the slot . Only the problem corresponding to the slot on line 6 would match how the model was trained . This mismatch between training and test distributions hampers the prediction accuracy of the model . In our experiments , it leads to an accuracy drop of 4 % to 14 % , even in the non-enumerative setting , i.e. , when the exact location of the bug is provided . Since the enumerative approach uses the prediction of the same variable as the original variable for declaring no bugs at that location , this phenomenon contributes to its worse performance . Another drawback of the enumerative approach is that it produces one prediction per slot in a program , rather than one prediction per program . Allamanis et al . ( 2018 ) deal with this by manually selecting a numerical threshold and reporting a bug ( and its repair ) only if the predicted probability for a repair is higher than that threshold . Setting a suitable threshold is difficult : too low a threshold can increase false positives and too high a threshold can cause false negatives . In order to deal with these drawbacks , we present a model that jointly learns to perform : 1 ) classification of the program as either faulty or correct ( with respect to VARMISUSE bugs ) , 2 ) localization of the bug when the program is classified as faulty , and 3 ) repair of the localized bug . One of the key insights of our joint model is the observation that , in a program containing a single VARMISUSE bug , a variable token can only be one of the following : 1 ) a buggy variable ( the faulty location ) , 2 ) some occurrence of the correct variable that should be copied over the incorrect variable into the faulty location ( a repair location ) , or 3 ) neither the faulty location nor a repair location . This arises from the fact that the variable in the fault location can not contribute to the repair of any other variable – there is only one fault location – and a variable in a repair location can not be buggy at the same time . This observation leads us to a pointer model that can point at locations in the input ( Vinyals et al. , 2015 ) by learning distributions over input tokens . The hypothesis that a program that contains a bug at a location likely contains ingredients of the repair elsewhere in the program ( Engler et al. , 2001 ) has been used quite effectively in practice ( Le Goues et al. , 2012 ) . Mechanisms based on pointer networks can play a useful role to exploit this observation for repairing programs . We formulate the problem of classification as pointing to a special no-fault location in the program . To solve the joint prediction problem of classification , localization , and repair , we lift the usual pointer-network architecture to multi-headed pointer networks , where one pointer head points to the faulty location ( including the no-fault location when the program is predicted to be non-faulty ) and another to the repair location . We compare our joint prediction model to an enumerative approach for repair . Our results show that the joint model not only achieves a higher classification , localization , and repair accuracy , but also results in high true positive score . Furthermore , we study how a pointer network on top of a recurrent neural network compares to the graph neural network used previously by Allamanis et al . ( 2018 ) . The comparison is performed for program repair given an a priori known bug location , the very same task used by that work . Limited to only syntactic inputs , our model outperforms the graph-based one by 7 percentage points . Although encouraging , this comparison is only limited to syntactic inputs ; in contrast , the graph model uses both syntax and semantics to achieve state-of-the-art repair accuracy . In future work we plan to study how jointly predicting bug location and repair might improve the graph model when bug location is unknown , as well as how our pointer-network-based model compares to the graphbased one when given semantics , in addition to syntax ; the latter is particularly interesting , given the relatively simpler model architecture compared to message-passing networks ( Gilmer et al. , 2017 ) . In summary , this paper makes the following key contributions : 1 ) it presents a solution to the general variable-misuse problem in which enumerative search is replaced by a neural network that jointly localizes and repairs faults ; 2 ) it shows that pointer networks over program tokens provide a suitable framework for solving the VARMISUSE problem ; and 3 ) it presents extensive experimental evaluation over multiple large datasets of programs to empirically validate the claims . 2 RELATED WORK . Allamanis et al . ( 2018 ) proposed an enumerative approach for solving the VARMISUSE problem by making individual predictions for each variable use in a program and reporting back all variable discrepancies above a threshold , using a graph neural network on syntactic and semantic information . We contrast this paper to that work at length in the previous section . Devlin et al . ( 2017b ) propose a neural model for semantic code repair where one of the classes of bugs they consider is VARREPLACE , which is similar to the VARMISUSE problem . This model also performs an enumerative search as it predicts repairs for all program locations and then computes a scoring of the repairs to select the best one . As a result , it also suffers from a similar training/test data mismatch issue as Allamanis et al . ( 2018 ) . Similar to us , they use a pooled pointer model to perform the repair task . However , our model uses multi-headed pointers to perform classification , localization , and repair jointly . DeepFix ( Gupta et al. , 2017 ) and SynFix ( Bhatia et al. , 2018 ) repair syntax errors in programs using neural program representations . DeepFix uses an attention-based sequence-to-sequence model to first localize the syntax errors in a C program , and then generates a replacement line as the repair . SynFix uses a Python compiler to identify error locations and then performs a constraint-based search to compute repairs . In contrast , we use pointer networks to perform a fine-grained localization to a particular variable use , and to compute the repair . Additionally , we tackle variable misuses , which are semantic bugs , whereas those systems fix only syntax errors in programs . The DeepBugs ( Pradel & Sen , 2018 ) paper presents a learning-based approach to identifying namebased bugs . The main idea is to represent program expressions using a small set of features ( e.g. , identifier names , types , operators ) and then compute their vector representations by concatenating the individual feature embeddings . By injecting synthetic bugs , the classifier is trained to predict program expressions as buggy or not for three classes of bugs : swapped function arguments , wrong binary operator , and wrong operand in a binary operation . Similar to previous approaches , it is also an instance of an enumerative approach . Unlike DeepBugs , which embeds a single expression , our model embeds the full input program ( up to a maximum prefix length ) and performs both localization and repair of the VARMISUSE bugs in addition to the classification task . Moreover , our model implements a pointer mechanism for representing repairs that often requires pointing to variable uses in other parts of the program that are not present in the same buggy expression . Sk p ( Pu et al. , 2016 ) is another enumerative neural program repair approach to repair student programs using an encoder-decoder architecture . Given a program , for each program statement si , the decoder generates a statement s′i conditioned on an encoding of the preceding statement si−1 and the following statement si+1 . Unlike our approach , which can generate VARMISUSE repairs using a pointer mechanism , the Sk p model would need to predict full program statements for repairing such bugs . Moreover , similar to the DeepBugs approach , it would be difficult for the model to predict repairs that include variables defined two or more lines above the buggy variable location . Automated program repair ( APR ) has been an area of active research in software engineering . The traditional APR approaches ( Gazzola et al. , 2018 ; Monperrus , 2018 ; Motwani et al. , 2018 ) differ from our work in the following ways : 1 ) They require a form of specification of correctness to repair a buggy program , usually as a logical formula/assertion , a set of tests , or a reference implementation ; 2 ) They depend on hand-designed search techniques for localization and repair ; 3 ) The techniques are applied to programs that violate their specifications ( e.g. , a program that fails some tests ) , which means that the programs are already known to contain bugs . In contrast , a recent line of research in APR is based on end-to-end learning , of which ours is an instance . Our solution ( like some other learning-based repair solutions ) has the following contrasting features : 1 ) It does not require any specification of correctness , but learns instead to fix a common class of errors directly from sourcecode examples ; 2 ) It does not perform enumerative search for localization or repair—we train a neural network to perform localization and repair directly ; 3 ) It is capable of first classifying whether a program has the specific type of bug or not , and subsequently localizing and repairing it . The APR community has also designed some repair bechmarks , such as ManyBugs and IntroClass ( Goues et al. , 2015 ) , and Defects4J ( Just et al. , 2014 ) , for test-based program repair techniques . The bugs in these benchmarks relate to the expected specification of individual programs ( captured through test cases ) and the nature of bugs vary from program to program . These benchmarks are therefore suitable to evaluate repair techniques guided by test executions . Learning-based solutions like ours focus on common error types , so it is possible for a model to generalize across programs , and work directly on embeddings of source code .

Summary: This paper considers the problem of VarMisuse, a kind of software bug where a variable has been misused. Existing approaches to the problem create a complex model, followed by enumerating all possible variable replacements at all possible positions, in order to identify where the bug may exist. This can be problematic for training which is performed using synthetic replacements; enumeration on non-buggy positions does not reflect the test case. Also, at test time, enumerating is expensive, and does not accurately capture the various dependencies of the task. This paper instead proposes a LSTM based model with pointers to break the problem down into multiple steps: (1) is the program buggy, (2) where is the bug, and (3) what is the repair. They evaluate on two datasets, and achieve substantial gains over previous approaches, showing that the idea of localizing and repairing and effective.