Neural Program Generation Modulo Static Analysis

1 Introduction . Neural models of source code have received much attention in the recent past [ 38 , 9 , 26 , 23 , 30 , 36 , 16 , 24 ] . However , these models have a basic weakness : while they frequently excel at generating individual expressions or lines of code , they do not do so well when tasked with synthesizing larger code blocks . For example , as we show later in this paper , state-of-the-art transformer models [ 8 , 6 , 24 ] can generate code with elementary semantic errors , such as uninitialized variables and type-incorrect expressions , when asked to generate method bodies , as opposed to single lines . Even in terms of syntactic accuracy measures , the quality of the code that transformers produce on such “ long-horizon ” tasks can be far removed from the ground truth . The root cause of these issues , we believe , is that current neural models of code treat programs as text rather than artifacts that are constructed following a semantics . In principle , a model could learn semantics from syntax given enough data . In practice , such learning is difficult for complex , general-purpose languages . In this paper , we propose to address this challenge through an alternative neurosymbolic approach . Our main observation is that symbolic methods—specifically , static program analysis—can extract deep semantic relationships between far-removed parts of a program . However , these relationships are not apparent at the level of syntax , and it is difficult for even large neural networks to learn them automatically . Driven by this observation , we use a static-analysis tool as a weak supervisor for a deep model of code . During generation , our model invokes this static analyzer to compute a set of semantic facts about the code generated so far . The distribution over the model ’ s next generation actions is conditioned on these facts . We concretely develop our approach by extending the classic formalism of attribute grammars [ 20 ] . Attribute grammars are like context-free grammars but allow rules to carry symbolic attributes of the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . context in which a rule is fired . In our model , called Neurosymbolic Attribute Grammars ( NSGs ) , the context is an incomplete program , and rules are fired to replace a nonterminal ( a stand-in for unknown code ) in this program . The attributes are semantic relationships ( for example , symbol tables ) computed using static analysis . The neural part of the model represents a probability distribution over the rules of the grammar conditioned on the attributes . During generation , the model repeatedly samples from this distribution while simultaneously computing the attributes of the generated code . We evaluate our approach in the task of generating the entire body of a Java method given the rest of the class in which the method occurs . We consider a large corpus of curated Java programs , over a large vocabulary of API methods and types.Using this corpus , we train an NSG whose attributes , among other things , track the state of the symbol table and the types of arguments and return values of invoked methods at various points of a program , and whose neural component is a basic tree LSTM . We compare this model against several recent models : fine-tuned versions of two GPT-NEO [ 6 ] transformers and the CODEGPT [ 24 ] transformer , OpenAI ’ s CODEX system [ 8 ] ( used in a zero-shot manner ) , and a GNN-based method for program encoding [ 7 ] . Some of these models are multiple orders of magnitude larger than our NSG model . Our experiments show that the NSG model reliably outperforms all of the baselines on our task , both in terms of producing programs free of semantic errors and in terms of matching the ground truth syntactically . In summary , this paper makes three contributions : • We present a new approach to the generative modeling of source code that uses a static-analysis tool as a weak supervisor . • We embody this approach in the specific form of neurosymbolic attribute grammars ( NSGs ) . • We evaluate the NSG approach on the long-horizon task of generating entire Java method bodies , and show that it significantly outperforms several larger , state-of-the-art transformer models . 2 Conditional Program Generation We start by stating our problem , known as conditional program generation ( CPG ) [ 26 ] . We imagine a joint distribution D ( X , Y ) , where X ranges over specifications of program-generation problems and Y ranges over programs . The probability D ( X = X , Y = Y ) is high when Y is a solution to X . Also , we consider a family of distributions P✓ ( Y |X = X ) , parameterized by ✓ , that we might want to learn . Learning to conditionally generate programs amounts to finding parameters ✓ that minimize the prediction error E ( X , Y ) ⇠D [ ( P✓ ( X|Y ) , Y ) ] , where is a suitable distance function between programs . Specifications and distances between programs can be defined in many ways . In our experiments , the goal is to generate Java method bodies . A specification is an evidence set that contains information—e.g. , method names , types of variables and methods—about the class in which the method lies . We define ( Y1 , Y2 ) to be a large number if Y1 or Y2 violates one of several language-level invariants ( e.g. , type-safety , initialization of variables before use ) that we require programs to satisfy . When both programs satisfy the invariants , ( Y1 , Y2 ) measures the textual dissimilarity between the two programs . Note that CPG is a much more challenging task than the well-studied next-token-prediction task [ 24 , 7 ] . The goal is to predict long sequences of tokens ( e.g. , an entire method body ) . Also , X is a ( possibly imprecise ) specification of the code to generate , not just a sequence of tokens we are trying to complete by , say , choosing the correct method to call for a variable . Example . Fig . 1- ( a ) illustrates the kind of task that we target . Here , we are given a class with a missing write method . The specification X includes : ( i ) the class name FileUtil ; ( ii ) the type String of the class variable err ; ( iii ) information about complete methods within the class ( including the methods ’ return types and formal-parameter types and names , and sequences of API calls made within such methods ) ; ( iv ) information about the method with missing code ( write ) , including its name , formal parameters , and JavaDoc comments for the method with missing code ( e.g. , “ write lines to file ” ) . Our objective on this input is to generate automatically a non-buggy , natural completion of write , without any provided , partial implementation of the method . To understand the challenges in this task , consider a completion that starts by : ( i ) declaring a local variable var_0 ; and ( ii ) invoking the constructor for FileWriter and storing the result in var_0 . A proper implementation of these two steps must ensure that var_0 is of type FileWriter . Also , the first argument to the constructor of FileWriter must be of type File ( or a subtype of File ) . As we show in Sec . 5 , it is hard for state-of-the-art neural models to learn to satisfy these rules . In contrast , in our approach , the generator has access to a set of semantic attributes computed via static analysis . These attributes include a symbol table mapping in-scope variables to their types . Suppose that during training we are given the following line of code : “ var_0 = new FileWriter ( f , true ) ” . Our model ’ s symbol table includes the names var_0 and f and their types . The grammar is also able to compute the type of the first argument in the invoked constructor for FileWriter . Consequently , the model can observe that the type of f is listed as File in the symbol table , and that f is the first argument to the FileWriter constructor . With a few observations like these , the model can learn that the first argument of “ new FileWriter ” tends to be of type File ( or a subtype ) . During generation , the model uses this knowledge , locating a variable of the correct type in the symbol table each time it constructs a FileWriter . Fig . 1- ( b ) shows a top completion of write generated by our NSG implementation . Note that all variables in this code are initialized before use , and that all operations are type-safe . Also , the name var_0 is reused between the try and the catch blocks . Such reuse is possible because the symbol table carries information about the scopes to which different names belong . Finally , as we will see in Sec . 5 , the extra information provided by the static analyzer can also help with accuracy in terms of syntactic matches with the ground truth . 3 Static Analysis with Attribute Grammars . As mentioned in Sec . 1 , we develop our approach as an extension of the classic attribute grammar ( AG ) framework [ 20 ] . Now we give some background on static analysis using AGs . In the next section , we show how to use AGs to weakly supervise a neural program generator . An AG extends a traditional context-free grammar ( CFG ) [ 18 ] by attaching a set of attributes to each terminal or nonterminal symbol of the grammar and by using a set of attribute equations to propagate attribute values through syntax trees . The attributes of a symbol S can be divided into inherited attributes and synthesized attributes , which we suffix by # and `` , respectively . Inherited attributes transfer information from parent to child , or from a node to itself . Synthesized attributes transfer information from child to parent , from a node to a sibling , or from a node to itself . We assume that the terminal symbols of the grammar have no synthesized attributes and that the root symbol of the grammar has a special set of inherited attributes , known as the initial attributes . The output attributes of a production S ! S1 , . . . , Sk consist of the synthesized-attribute occurrences of the nonterminal S , plus the inherited-attribute occurrences of all of the Si ’ s . The input attributes are the inherited-attribute occurrences of S , plus the synthesized-attribute occurrences of the Si ’ s . The grammar ’ s attribute equations relate the input and output attributes of a node in terms of the attributes of its parent , children , and left sibling in the syntax tree that the grammar generates . Example . Consider the simple CFG in Fig . 2- ( a ) . The nonterminal Stmt stands for program statements . The grammar says that a statement can either be a sequential composition of statements , a method call , or a variable declaration . A natural AG extension of this CFG tracks symbol tables , which allow easy lookup of all variables in scope . Specifically , the grammar associates two symbol-table-valued attributes , symTab # and symTabOut `` , with Stmt ( Fig . 2- ( b ) ) . The attributes are propagated following the equations in Fig . 2- ( c ) . In these equations , we distinguish between the three different occurrences of nonterminal “ Stmt ” via the symbols “ Stmt $ 0 , ” “ Stmt $ 1 , ” and “ Stmt $ 2. ” where the numbers denote the leftmost occurrence , the next-to-leftmost occurrence , etc . In this case , the leftmost occurrence is the left-hand-side occurrence . For concreteness , let us consider the attribute equations for the production for sequential composition in the grammar . Here , the inherited attribute of Stmt $ 0 gets passed “ down ” the syntax tree as an inherited attribute of Stmt $ 1 . The synthesized attribute received at Stmt $ 1 is passed to Stmt $ 2 as an inherited attribute . More generally , the attribute equations define a left-to-right information flow through the syntax tree , as illustrated in Fig . 2- ( d ) .

Summary: The paper proposes a novel approach to conditional program generation that leverages supervision from a static analysis tool during generation. This serves to address a common difficulty in program synthesis using language models, that of generating code without semantic errors. The program generation and static analysis both follow the linear information flow through a program defined by an attribute grammar, with the program generation model having access to the attributes from the analysis. The paper applies this technique to a dataset of 1 million Java programs, comparing primarily with baselines CodeGPT and GNN2NAG. Evaluating both on static checks, overall fidelity metrics, and exact match, the paper finds the proposed NSG method outperforms the baselines considered on nearly all metrics.