Neural Program Generation Modulo Static Analysis

1 Introduction . Neural models of source code have received much attention in the recent past [ 38 , 9 , 26 , 23 , 30 , 36 , 16 , 24 ] . However , these models have a basic weakness : while they frequently excel at generating individual expressions or lines of code , they do not do so well when tasked with synthesizing larger code blocks . For example , as we show later in this paper , state-of-the-art transformer models [ 8 , 6 , 24 ] can generate code with elementary semantic errors , such as uninitialized variables and type-incorrect expressions , when asked to generate method bodies , as opposed to single lines . Even in terms of syntactic accuracy measures , the quality of the code that transformers produce on such “ long-horizon ” tasks can be far removed from the ground truth . The root cause of these issues , we believe , is that current neural models of code treat programs as text rather than artifacts that are constructed following a semantics . In principle , a model could learn semantics from syntax given enough data . In practice , such learning is difficult for complex , general-purpose languages . In this paper , we propose to address this challenge through an alternative neurosymbolic approach . Our main observation is that symbolic methods—specifically , static program analysis—can extract deep semantic relationships between far-removed parts of a program . However , these relationships are not apparent at the level of syntax , and it is difficult for even large neural networks to learn them automatically . Driven by this observation , we use a static-analysis tool as a weak supervisor for a deep model of code . During generation , our model invokes this static analyzer to compute a set of semantic facts about the code generated so far . The distribution over the model ’ s next generation actions is conditioned on these facts . We concretely develop our approach by extending the classic formalism of attribute grammars [ 20 ] . Attribute grammars are like context-free grammars but allow rules to carry symbolic attributes of the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . context in which a rule is fired . In our model , called Neurosymbolic Attribute Grammars ( NSGs ) , the context is an incomplete program , and rules are fired to replace a nonterminal ( a stand-in for unknown code ) in this program . The attributes are semantic relationships ( for example , symbol tables ) computed using static analysis . The neural part of the model represents a probability distribution over the rules of the grammar conditioned on the attributes . During generation , the model repeatedly samples from this distribution while simultaneously computing the attributes of the generated code . We evaluate our approach in the task of generating the entire body of a Java method given the rest of the class in which the method occurs . We consider a large corpus of curated Java programs , over a large vocabulary of API methods and types.Using this corpus , we train an NSG whose attributes , among other things , track the state of the symbol table and the types of arguments and return values of invoked methods at various points of a program , and whose neural component is a basic tree LSTM . We compare this model against several recent models : fine-tuned versions of two GPT-NEO [ 6 ] transformers and the CODEGPT [ 24 ] transformer , OpenAI ’ s CODEX system [ 8 ] ( used in a zero-shot manner ) , and a GNN-based method for program encoding [ 7 ] . Some of these models are multiple orders of magnitude larger than our NSG model . Our experiments show that the NSG model reliably outperforms all of the baselines on our task , both in terms of producing programs free of semantic errors and in terms of matching the ground truth syntactically . In summary , this paper makes three contributions : • We present a new approach to the generative modeling of source code that uses a static-analysis tool as a weak supervisor . • We embody this approach in the specific form of neurosymbolic attribute grammars ( NSGs ) . • We evaluate the NSG approach on the long-horizon task of generating entire Java method bodies , and show that it significantly outperforms several larger , state-of-the-art transformer models . 2 Conditional Program Generation We start by stating our problem , known as conditional program generation ( CPG ) [ 26 ] . We imagine a joint distribution D ( X , Y ) , where X ranges over specifications of program-generation problems and Y ranges over programs . The probability D ( X = X , Y = Y ) is high when Y is a solution to X . Also , we consider a family of distributions P✓ ( Y |X = X ) , parameterized by ✓ , that we might want to learn . Learning to conditionally generate programs amounts to finding parameters ✓ that minimize the prediction error E ( X , Y ) ⇠D [ ( P✓ ( X|Y ) , Y ) ] , where is a suitable distance function between programs . Specifications and distances between programs can be defined in many ways . In our experiments , the goal is to generate Java method bodies . A specification is an evidence set that contains information—e.g. , method names , types of variables and methods—about the class in which the method lies . We define ( Y1 , Y2 ) to be a large number if Y1 or Y2 violates one of several language-level invariants ( e.g. , type-safety , initialization of variables before use ) that we require programs to satisfy . When both programs satisfy the invariants , ( Y1 , Y2 ) measures the textual dissimilarity between the two programs . Note that CPG is a much more challenging task than the well-studied next-token-prediction task [ 24 , 7 ] . The goal is to predict long sequences of tokens ( e.g. , an entire method body ) . Also , X is a ( possibly imprecise ) specification of the code to generate , not just a sequence of tokens we are trying to complete by , say , choosing the correct method to call for a variable . Example . Fig . 1- ( a ) illustrates the kind of task that we target . Here , we are given a class with a missing write method . The specification X includes : ( i ) the class name FileUtil ; ( ii ) the type String of the class variable err ; ( iii ) information about complete methods within the class ( including the methods ’ return types and formal-parameter types and names , and sequences of API calls made within such methods ) ; ( iv ) information about the method with missing code ( write ) , including its name , formal parameters , and JavaDoc comments for the method with missing code ( e.g. , “ write lines to file ” ) . Our objective on this input is to generate automatically a non-buggy , natural completion of write , without any provided , partial implementation of the method . To understand the challenges in this task , consider a completion that starts by : ( i ) declaring a local variable var_0 ; and ( ii ) invoking the constructor for FileWriter and storing the result in var_0 . A proper implementation of these two steps must ensure that var_0 is of type FileWriter . Also , the first argument to the constructor of FileWriter must be of type File ( or a subtype of File ) . As we show in Sec . 5 , it is hard for state-of-the-art neural models to learn to satisfy these rules . In contrast , in our approach , the generator has access to a set of semantic attributes computed via static analysis . These attributes include a symbol table mapping in-scope variables to their types . Suppose that during training we are given the following line of code : “ var_0 = new FileWriter ( f , true ) ” . Our model ’ s symbol table includes the names var_0 and f and their types . The grammar is also able to compute the type of the first argument in the invoked constructor for FileWriter . Consequently , the model can observe that the type of f is listed as File in the symbol table , and that f is the first argument to the FileWriter constructor . With a few observations like these , the model can learn that the first argument of “ new FileWriter ” tends to be of type File ( or a subtype ) . During generation , the model uses this knowledge , locating a variable of the correct type in the symbol table each time it constructs a FileWriter . Fig . 1- ( b ) shows a top completion of write generated by our NSG implementation . Note that all variables in this code are initialized before use , and that all operations are type-safe . Also , the name var_0 is reused between the try and the catch blocks . Such reuse is possible because the symbol table carries information about the scopes to which different names belong . Finally , as we will see in Sec . 5 , the extra information provided by the static analyzer can also help with accuracy in terms of syntactic matches with the ground truth . 3 Static Analysis with Attribute Grammars . As mentioned in Sec . 1 , we develop our approach as an extension of the classic attribute grammar ( AG ) framework [ 20 ] . Now we give some background on static analysis using AGs . In the next section , we show how to use AGs to weakly supervise a neural program generator . An AG extends a traditional context-free grammar ( CFG ) [ 18 ] by attaching a set of attributes to each terminal or nonterminal symbol of the grammar and by using a set of attribute equations to propagate attribute values through syntax trees . The attributes of a symbol S can be divided into inherited attributes and synthesized attributes , which we suffix by # and `` , respectively . Inherited attributes transfer information from parent to child , or from a node to itself . Synthesized attributes transfer information from child to parent , from a node to a sibling , or from a node to itself . We assume that the terminal symbols of the grammar have no synthesized attributes and that the root symbol of the grammar has a special set of inherited attributes , known as the initial attributes . The output attributes of a production S ! S1 , . . . , Sk consist of the synthesized-attribute occurrences of the nonterminal S , plus the inherited-attribute occurrences of all of the Si ’ s . The input attributes are the inherited-attribute occurrences of S , plus the synthesized-attribute occurrences of the Si ’ s . The grammar ’ s attribute equations relate the input and output attributes of a node in terms of the attributes of its parent , children , and left sibling in the syntax tree that the grammar generates . Example . Consider the simple CFG in Fig . 2- ( a ) . The nonterminal Stmt stands for program statements . The grammar says that a statement can either be a sequential composition of statements , a method call , or a variable declaration . A natural AG extension of this CFG tracks symbol tables , which allow easy lookup of all variables in scope . Specifically , the grammar associates two symbol-table-valued attributes , symTab # and symTabOut `` , with Stmt ( Fig . 2- ( b ) ) . The attributes are propagated following the equations in Fig . 2- ( c ) . In these equations , we distinguish between the three different occurrences of nonterminal “ Stmt ” via the symbols “ Stmt $ 0 , ” “ Stmt $ 1 , ” and “ Stmt $ 2. ” where the numbers denote the leftmost occurrence , the next-to-leftmost occurrence , etc . In this case , the leftmost occurrence is the left-hand-side occurrence . For concreteness , let us consider the attribute equations for the production for sequential composition in the grammar . Here , the inherited attribute of Stmt $ 0 gets passed “ down ” the syntax tree as an inherited attribute of Stmt $ 1 . The synthesized attribute received at Stmt $ 1 is passed to Stmt $ 2 as an inherited attribute . More generally , the attribute equations define a left-to-right information flow through the syntax tree , as illustrated in Fig . 2- ( d ) .

Summary: This paper presents a new model for automatic code completion of full methods (rather than simply single lines).  The main innovation is the use of a static analysis tool which can analyze partially generated programs.  The program is generated hierarchically from an attribute grammar using a model they call a Neurosymbolic Attribute Grammer (NSG).  At each step the model has access to: - The context code surrounding the code to be generated - All the tree nodes which have been generated so far - The attributes generated by running the static analyzer over the partially completed program  Their hypothesis is that the kind of information that the static analyzer generates is difficult for a neural network model to learn by simply training on a large dataset of programs.  They show that their model outperforms both a smallish transformer baseline (~100M parameters) and a Graph Neural Network baseline.