HOPPITY: LEARNING GRAPH TRANSFORMATIONS TO DETECT AND FIX BUGS IN PROGRAMS

1 INTRODUCTION . The sheer size and complexity of modern codebases makes it impossible for them to be bug-free . As a result , a more reasonable and effective strategy has emerged , which aims to prevent bugs in production by applying automated tools to detect and even fix them early in the development process . This trend has gained increasing popularity in recent years . Examples include Google ’ s Tricorder ( Sadowski et al. , 2015 ) , Facebook ’ s Getafix ( Scott et al. , 2019 ) and Zoncolan , and Microsoft ’ s Visual Studio IntelliCode . The techniques underlying these tools can be classified into broadly two categories : logical , rule-based techniques ( Sadowski et al. , 2015 ) and statistical , data-driven techniques ( Allamanis et al. , 2018 ; Pradel & Sen , 2018 ; Vasic et al. , 2019 ) . The former uses manually written rules capturing undesirable code patterns and scans the entire codebase for these classes of bugs . The latter learns to detect abnormal code from a large code corpus using deep neural networks . Despite great strides , however , both kinds of tools are limited in generality because they target error patterns in specific codebases or they target specific bug types . For instance , Zoncolan ’ s rules are designed to be specifically applicable to Facebook ’ s codebases , and deep learning models target specialized bugs in variable naming ( Allamanis et al. , 2018 ) or binary expressions ( Pradel & Sen , 2018 ) . Moreover , the patterns are relatively syntactic , allowing them to be specified by human experts using logical rules or learnt from a corpus of programs . In this paper , we propose a novel learning-based approach for finding and fixing bugs in Javascript programs automatically . Javascript is a scripting language designed for web application development . It has been the most popular programming language on GitHub since 2014 ( Github , 2019 ) . Repairing Javascript code presents a unique challenge as bugs manifest in diverse forms due to unusual language features and the lack of tooling support . Therefore , the primary goal of our approach is generality since it must be effective against a board spectrum of programming errors , such as using wrong operators or identifiers , accessing undefined properties , mishandling variable scopes , triggering type incompatibilites , among many others . Another important novel aspect concerns our approach ’ s ability to deal with bugs that are more complex and semantic in nature , namely , bugs that require adding or removing statements from a program , which are not considered by prior works . Finally , compared to automated program repair techniques ( Le Goues et al. , 2019 ; Scott et al. , 2019 ; Hua et al. , 2018 ; Chen ∗The first two authors contributed equally to this work . et al. , 2018 ) which require knowledge of bug location , this paper presents an end-to-end approach including localizing bugs , predicting the types of fixes , and generating patches . We design our model architecture in a similar vein as a Neural Turing Machine ( NTM ) ( Graves et al. , 2014 ) . It consists of an external memory ( a Graph Neural Network ) for embedding a buggy program and a central controller ( an LSTM ) that makes a sequence of primitive actions ( e.g. , predicting type , generating patch , etc . ) to perform a fix . The multi-step decision process is implemented by an autoregressive model . Crucially , our model differs from the standard NTM in how the memory is manipulated : apart from the common read and write operations , the controller can also expand or shrink the memory when adding or deleting nodes in the original graph . We have realized our approach in a tool called HOPPITY . By training on 290,715 Javascript code change commits collected from Github , HOPPITY correctly detects and fixes bugs in 9,490 out of 36,361 programs using a beam size of three . 2 MOTIVATING EXAMPLES . Javascript is quite different compared to traditional object-oriented languages ( e.g . C++ , Java , or C # ) . In addition to the weak , dynamic typing discipline of scripting languages , Javascript supports many peculiar features that do not exist in other languages . For example , it allows a property ( i.e. , a field ) to be added to or removed from an object at runtime . As another example , Javascript did not support block-level scoping until recently , allowing a variable defined in a block structure such as a for loop to be exposed to the entire function in which the loop occurs . While the latest ES6 language standard incorporates block-level scoping , developers have been programming without it for decades , resulting in a large body of legacy code . Finally , Javascript ’ s eval function , which interprets and executes a string as a code fragment , is widely regarded as a major source of bugs and vulnerabilities . All of these aspects make programming in Javascript a frustrating and error-prone experience . function clearEmployeeListOnLinkClick ( ) { document.querySelector ( `` a '' ) .addEventListener ( `` click '' , function ( event ) { document.querySelector ( `` ul '' ) .InnerHTML = `` '' ; } ) ; } ( a ) InnerHTML should have been innerHTML . if ( matches ) { return { episode : Number ( matches.groups.episode ) , hosts : matches.groups.hosts.split ( / ( [ , & ] +|\sand\s ) / ) . map ( el = > S ( el ) .trim ( ) .s ) } ; } ( b ) Highlighted parentheses should have been removed . module.exports = function ( grunt ) { grunt.initConfig ( { execute : { ... } , copy : { ... } , checktextdomain : { ... } wp_readme_to_markdown : { ... } , makepot : { ... } } ) ... grunt.registerTask ( ’ default ’ , [ ’ wp_readme_to_markdown ’ , ’ makepot ’ , ’ execute ’ , ’ checktextdomain ’ ] ) } ; ( c ) copy function should have also been included in the highlighted list . export default { computed : { level ( ) { return dictMap.skillLevel [ parseInt ( ( this.value === 0 ? 1 : this.value ) /20 ) ] ; } } , ... } ( d ) parseInt should have been removed because === implies this.value is an integer . Figure 1 : Example programs that illustrate limitations of existing approaches inculding both rulebased static analyzers and neural-based bug predictors . Static analyzers aim to detect common coding errors in Javascript programs by applying logical rulebased reasoning on source code . TAJS ( Jensen et al. , 2009 ) and ESLint ( Zakas , 2013 ) are prominent examples . These tools face important challenges to be effective . We present several examples in Figure 1 to illustrate their limitations . Due to the complex nature of client-side web APIs , TAJS and ESLint choose to ignore analyzing built-in libraries for the sake of scalability . As depicted in Figure 1a , when developers mistakenly capitalize the first letter of innerHTML , a property of class Element in DOM ( Document Object Model ) , both analyzers fail to catch the error . Javascript will then silently allow developers to set the previously non-existent property InnerHTML to the empty string . Later , when developers attempt to access the intended property , innerHTML , the program will crash and potentially cause a security vulnerability or incur a costly debugging experience . Additionally , static analyzers can never deal with functional bugs ( i.e. , errors that violate the program specification and yet conform to the coding rules ) . Figure 1b shows one such example . The goal is to split a string using regular expressions . However , the program incorrectly splits the input ' and ' into [ `` , ' and ' , `` ] instead of [ `` , `` ] , which is what the developer intended . Since the error is simply a mismatch between the developer ’ s implicit specification and implementation , static analyzers are incapable of catching it . The bugs that static analyzers missed in both cases are in hindsight quite obvious to human programmers . The criteria they use is very simple : any code snippet that seems to deviate from common code patterns is likely to be buggy . This is precisely the observation that our approach seeks to mimic . In particular , if a model observes a property or an unusual way of splitting strings that never appeared in the training data , it is likely to recognize those abnormal code fragments as potential bugs . The main advantage of our approach over existing neural-based bug detectors ( Allamanis et al. , 2018 ; Pradel & Sen , 2018 ; Vasic et al. , 2019 ) is its generality . Unlike prior works that target specific classes of bugs ( e.g. , variable naming issues or binary expression bugs ) , we train a single model to deal with a wide range of bug types , encompassing all previously proposed ones . Compared to past approaches that leverage a graph-based neural network model ( Allamanis et al. , 2018 ) , our model is capable of more sophisticated transformations such as adding or removing nodes , as shown in Figure 1c and 1d . Finally , our model not only locates but also fixes bugs , whereas program repair ( Le Goues et al. , 2019 ; Scott et al. , 2019 ; Hua et al. , 2018 ; Chen et al. , 2018 ) or bug localization ( Ball et al. , 2003 ; Jose & Majumdar , 2011 ; Wang et al. , 2019 ) techniques only solve a single task . 3 MODEL . We model the problem of detecting and repairing bugs in programs as a structured prediction problem on a graph-based representation of programs . Given a graph gbug that represents a buggy program , we wish to predict a graph gfix that represents the fixed program . Our model aims to capture the structured prediction by a sequence of up to T steps of graph transformations : p ( gfix|gbug ; θ ) = p ( g1|gbug ; θ ) p ( g2|g1 ; θ ) . . . p ( gfix|gT−1 ; θ ) ( 1 ) The high-level overview of the graph sequence transformation is shown in Figure 2 . Different programs may need a different number of steps T ( gbug ) which is also determined by the model . We first introduce our representation module for programs in Sec 3.1 . We then elucidate each step of the above transformation in Sec 3.2 . Finally , we summarize and present the full model in Sec 3.3 . 3.1 PROGRAM REPRESENTATION . Programs written in a high-level language have rich structure . Researchers have proposed graph-based representations to capture this structure ( Allamanis et al. , 2018 ) . We start with this approach of representing programs using graphs with certain modifications for our task . As shown in the left part of Figure 2 , we first parse the program ’ s source code into an abstract syntax tree ( AST ) form that captures the program ’ s syntactic structure . We then connect the leaf nodes with SuccToken edges . Unlike previous approaches , we additionally add value nodes that store the actual content of the leaf nodes , with special ValueLink edges connecting them together . The purpose of introducing this additional set of nodes is to provide a name-independent strategy for code representation and modification , which we elucidate in the next section . Hereafter , we use gfix , gbug or g in general to represent either the source code or the corresponding graph structure . After representing the program as a graph , we use a Graph Neural Network ( GNN ) ( Scarselli et al. , 2008 ) to map the graph into a representation in a fixed dimensional vector space . Specifically , given a graph g = ( V , E ) with set of nodes V and edges E , we need a function f ( g ) 7→ ( Rd , R|V |×d ) to obtain the d-dimensional representation of graph g ( denoted as ~g ) , as well as representations of individual nodes v ∈ V ( denoted as ~v ) . To parameterize f ( · ) , we employ the form in GIN ( Xu et al. , 2018 ) , with our adaptation to our multigraph for program representation in the following manner : h ( l+1 ) , k v = σ ( ∑ u∈Nk ( v ) W l , k 1 h ( l ) u ) , ∀k ∈ { 1 , 2 , . . . , K } h ( l+1 ) v = σ ( W l 2 [ h ( l+1 ) ,1 v , h ( l+1 ) ,2 v , . . . , h ( l+1 ) , K v ] + h ( l ) v ) ( 2 ) where Wl , k1 ∈ Rd×d , Wl2 ∈ RdK×d are model parameters and σ ( · ) is tanh in this implementation . K is the total number of edge types in this multi-graph representation . In the end , the node embedding is ~v = h ( L ) v , where L is the total number of propagations in the GNN . N k ( v ) is the set of neighbors of node v that are connected by edge with type k. Following GIN , the graph representation ~g is the aggregation of hlv , ∀l ∈ 0 , 1 , . . . , L. We use max pooling to aggregate hlv for each l , and then take the average of these L+ 1 vectors to obtain ~g . Initially , we use the node type as one-hot features as a starting value for h ( 0 ) v , where the types are either obtained from the AST representation , or from the local value table as shown in Figure 2 . Note that we don ’ t use features like variable names or function names in this graph representation , as different programs may follow different naming conventions . Instead , we focus on the syntactic structure of the source code , so as to enable naming-agnostic representation across different programs .

Summary: The paper proposes to learn from bugfixing commits to fix errors in other code. There are several good contributions of the paper, but the main one is to design a language of instructions that fix a program and to formulate the prediction to be a sequence of such instructions. The model, however, is insightful in other ways as well. The lack of naming features or generation of names makes it to point to other identifiers in a program - a major problem for most models for code. Instead, the proposed model only builds embeddings from the structure around the variables.