Latent Execution for Neural Program Synthesis Beyond Domain-Specific Languages

1 Introduction . Program synthesis from input-output ( IO ) pairs , also called programming by example ( PBE ) , requires high-level reasoning and remains a challenging problem for deep models . Unlike Natural Language Processing ( NLP ) [ 5 , 16 ] and perceptual tasks such as Computer Vision ( CV ) [ 14 , 22 ] , the mapping from IO pairs to the program itself is hard to model . Many works attempt to learn a direct mapping from training samples , but often found that it is already difficult to achieve a low training error , and generalization to new problems is even harder . Alternatively , one might choose to formulate program synthesis as a search problem : to find the program that satisfies IO pairs . Unfortunately , the search space of programs is often vast and highly non-smooth , i.e. , a small perturbation of the program often leads to a complete change of the output . While there are many previous works on programming by example tasks [ 6 , 17 , 9 ] , they mainly focus on Domain Specific Languages ( DSLs ) , and can not be easily applied to popular general-purpose programming languages . For example , to synthesize C programs , we need to deal with both high-level 1The code is available at https : //github.com/Jungyhuk/latent-execution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . control flows ( e.g. , branching and loop ) and low-level operations ( e.g. , which variable is the target of assignment ) . Moreover , unlike DSLs ( e.g. , Karel ) for which it is feasible to implement a per-line interpreter , C programs need compilation and a partial C program can not execute . On the other hand , some recent works investigate natural language descriptions as the auxiliary information of the program specification , and they evaluate neural program synthesis models on constrained or simplified competitive programming problems [ 27 , 3 , 23 , 11 , 4 ] . Although some of these works demonstrate promising results for synthesizing Python or C code , they require manual annotations of natural language specifications [ 27 ] or large-scale pre-training on human-written programs [ 11 , 4 ] , and the performance significantly degrades when only input-output examples are fed into the synthesizer [ 3 ] . To synthesize C programs from input-output examples only , we propose LaSynth , which generates the program in a recurrent and token-by-token manner . As the first contribution on model architectures for program synthesis , we propose to use two latent parallel representations in the recurrent model . One representation is learned from regular recurrent models as in autoregressive language models [ 24 ] , with the double attention mechanism over IO pairs proposed in RobustFill [ 17 ] and an operation predictor that models the arithmetic relationship between the program input and output . The second representation , named Latent Execution Trace ( LaET ) , models the hypothetical input signal for the remaining partial program to execute to get to the desired output . Motivated by the line of work on execution-guided program synthesis [ 47 , 18 , 57 , 12 ] , we learn a latent representation for C programs which are not executed via interpreters , and train the model given only IO pairs without the intermediate program execution states . The two parallel representations are trained end-to-end . As the second contribution on dataset construction , we demonstrate that it is possible to automatically construct a C codebase that is of high quality , controllable and concise through our proposed program synthesis procedure . Specifically , starting from randomly generated C programs that might contain a lot of redundant statements , we show that via iterative retraining , the subsequent generated code from our learned model becomes more concise and similar to human-written ones . Moreover , learning directly from the generated code leads to better performance given the same amount of samples , and improves the sample efficiency . We observe similar results when applying our iterative retraining technique to Karel [ 9 ] , another programming by example benchmark consisting of randomly generated programs . Although the initial Karel dataset includes a large proportion of complicated programs with different control flow constructs , we demonstrate that nearly half of the problems can be solved by straight-line programs , which again confirms that randomly generated programs tend to be unnecessarily complicated . We envision that the iterative retraining procedure could greatly reduce laborious efforts in human codebase collection in future research . As the third contribution , we show for the first time that short C code in a restricted domain ( tens of tokens , no library call ) with sequential , branching , loop and simple arithmetic operations can be effectively synthesized from IO pairs only . In particular , while LaSynth tends to generate more concise programs ( and does not have exact token match with random generated ground truth code ) , when measuring whether the program execution outputs match the IO pairs , LaSynth achieves 55.2 % accuracy , and outperforms existing neural program synthesis models by around 20 % . These results demonstrate the effectiveness of learning latent execution traces . 2 Neural Program Synthesis from Input-Output Examples . In programming by example tasks , the program specification is a set of input-output examples [ 17 , 9 ] . Specifically , we provide the synthesizer with a set of K input-output pairs { ( I ( k ) , O ( k ) ) } Kk=1 ( { IO } K in short ) . These input-output pairs are annotated with a ground truth program P ? , so that P ? ( I ( k ) ) = O ( k ) for any k ∈ { 1 , 2 , ... , K } . To measure the program correctness , we include another set of heldout test cases { IO } Ktesttest that differs from { IO } K . The goal of the program synthesizer is to predict a program P from { IO } K , so that P ( I ) = P ? ( I ) = O for any ( I , O ) ∈ { IO } K + { IO } Ktesttest . C Program Synthesis . In this work , we make the first attempt of synthesizing C code in a restricted domain from input-output examples only , and we focus on programs for list processing . List processing tasks have been studied in some prior works on input-output program synthesis , but they synthesize programs in restricted domain-specific languages instead of full-fledged popular programming languages [ 6 , 39 , 38 ] . Our C code synthesis problem brings new challenges for programming by example . Compared to domain-specific languages , the syntax and semantics of C are much more complicated , which significantly enlarges the program search space . Meanwhile , learning good representations for partially ℎ '' # $ 𝐼 ' '' # $ Latent Executor ℎ '' 𝐼 ' '' Program Decoder 𝑝 '' Latent Executor Program Decoder 𝑝 '' # $ ( a ) Model Overview 𝐿𝑆𝑇𝑀- 𝑂 IO Encoder 𝑝/ , 𝑝 $ , … , 𝑝 '' # $ Program Context ℎ '' # $ 𝑠 '' 1 𝑠 '' 2 𝑜𝑝4 '' ℎ '' Max Pool 𝑚 '' 𝑑 '' Softmax 𝑝 '' ( b ) Program Decoder 𝐿𝑆𝑇𝑀7 Training Target Softmax 𝐸𝑋𝐸𝐶 '' ( c ) Latent Executor I O Op 2 4 O=2+I 2 0 O=2-I … …𝑠 '' 2 ( $ ) … 𝑠 '' 2 ( = ) 𝑜𝑝4 '' ( $ ) … 𝑜𝑝4 '' ( = ) 𝐼 ' '' # $ ℎ '' 𝐼 ' '' # $ 𝐼 ' '' 𝐼 ' > 𝑂 ( d ) 𝑠 '' 1 ( $ ) … 𝑠 '' 1 ( = ) Operation Predictor Attention Retrieve ℎ '' ? $ 𝐼 ' '' ? $ 𝑝 '' ? $ Figure 2 : ( a ) An overview of LaSynth model architecture . ( b ) , ( c ) , and ( d ) present the details of the program decoder , latent executor , and the operation predictor . Note that the operation predictor is specialized for numerical calculation , and thus is not used for the Karel domain . decoded programs also becomes more difficult . In particular , prior neural program synthesizers that utilize per-line interpreters for the programming language to guide the synthesis and representation learning [ 12 , 44 , 37 , 18 , 38 ] are not directly applicable to C. Although it is possible to dump some intermediate variable states during C code execution [ 10 ] , since partial C programs are not executable , we are able to obtain all the execution states only until a full C code is generated , which is too late to include them in the program decoding process . In particular , the intermediate execution state is not available when the partial program is syntactically invalid , and this happens more frequently for C due to its syntax design . 3 Program Synthesis with Learned Execution . In this section , we present LaSynth which learns to represent the execution of partial programs to guide the synthesis process . Fig . 2 ( a ) provides an overview of LaSynth model architecture which consists of two components , the program decoder and the latent executor . We present the core design below , and defer more details to Appendix B and Appendix C . 3.1 Model Overview . At a high level , the program decoder ( Fig . 2 ( b ) ) takes a latent vector ht−1 that represents the generated partial program , the previous ( generated ) program token pt−1 , and outputs the latent vector ht and the next program token pt to be generated at time step t : ( ht , pt ) = ProgramDecoder ( ht−1 , pt−1 ; IOt−1 ) ( 1 ) Here the recurrent model is conditioned on the IO pair IOt−1 . When IOt = IO : = ( I , O ) for every t , i.e. , IOt remains constant over the entire recurrent generation process , Eqn . 1 represents the standard recurrent architecture used in most autoregressive natural language models [ 24 , 49 ] , and is also used in prior works on program synthesis from input-output examples [ 17 , 9 ] . For program decoding , the decoder first takes two attention vectors sIt and s O t computed from IO pairs and latent vector ht−1 via double attention [ 17 ] , and utilizes a max pooling layer to compute an aggregated embedding mt for all IO pairs ( Fig . 2 ( b ) ) : mt = MaxPoolj∈ { 1,2 , ... , K } ( tanh ( W [ s I ( j ) t ; s O ( j ) t ] ) ) ( 2 ) Here the superscript ( j ) indicates that the representation is for the j-th IO pair , [ a ; b ] is vector concatenation of a and b , and W is a trainable matrix . To facilitate the prediction of long programs , we compute an attention vector dt over previously generated program tokens using the standard attention mechanism [ 5 , 30 ] : dt = Attention ( mt , { p0 , ... , pt−1 } ) ( 3 ) Finally , the next token pt is sampled from P [ pt ] = Softmax ( V dt ) pt where V is a trainable matrix . 3.2 Latent Executor Design . As shown in our experiments ( Sec . 5 ) , the standard program decoder architecture may not be able to achieve strong performance in program synthesis when the program complexity increases . One main reason is that the standard program decoder only takes the initial IO pairs as the input without considering the program execution , thus the learned representation for the partial program does not effectively guide the synthesis process . Motivated by prior works that utilize execution traces for Karel program synthesis [ 12 , 44 , 47 ] , in this paper , we introduce latent executor ( Fig . 2 ( c ) ) which maintains a second representation Ît during program decoding . Intuitively , Ît−1 models the hypothetical input of the partial program pt ... T so that its output becomes O . Given the estimated input Ît−1 and the latent vector ht , the latent executor returns Ît at the next time step t : Ît = LatentExecutor ( Ît−1 , ht ) ( 4 ) The collection of { Ît } Tt=0 is the latent execution trace ( LaET ) . With the help of latent executor , we now use the IO pairs IOt−1 : = ( Ît−1 , O ) instead of ( I , O ) for the program decoder ( Eqn . 1 ) .

Summary: This paper presents a neural program synthesis system, called LaSynth, that aims to generate programs in non-domain specific languages (DSLs) unlike much of the prior work. In particular, this paper focuses on using LaSynth for the C programming language. A core reason prior work in program synthesis has been generally restricted to DSLs is, as the authors note (and I agree with), because of the linguistic complexity of general-purpose languages and the computational intractability that is often associated with synthesizing programs in such languages.  A core novelty of LaSynth is that it uses two forms of conjoined/parallel representations, one of which uses a hypothetical input to help determine the remaining partial program to execute to get the desired output. This somewhat resembles BUSTLE’s approach of breaking down components into small pieces and then assembling them for a larger program (ICLR ’21), but the actual system-level approach in LaSynth, in my mind, is notably different from BUSTLE and other prior works that I’m familiar (some of which the authors seem to have missed in the prior work – NetSyn (Mandal et al., MLSys ’21), REPL (Ellis et al., NeurIPS ’20), etc.), but we'll get to that later.