Execution-Guided Neural Program Synthesis

1 INTRODUCTION . Program synthesis is a traditional challenging problem . Such a problem typically takes a specification as the input , and the goal is to generate a program within a target domain-specific language ( DSL ) . One of the most interesting forms of the specifications is input-output examples , and there have been several applications , such as FlashFill ( Gulwani , 2011 ; Gulwani et al. , 2012 ) . Recently , there is an increasing interest of applying neural network approaches to tackle the program synthesis problem . For example , Devlin et al . have demonstrated that using an encoder-decoderstyle neural network , their neural program synthesis algorithm called RobustFill can outperform the performance of the traditional non-neural program synthesis approach by a large margin on the FlashFill task ( Devlin et al. , 2017b ) . Despite their promising performance , we identify several inefficiencies of such encoder-decoder-style neural program synthesis approaches . In particular , such a neural network considers program synthesis as a sequence generation problem ; although some recent work take the syntactical information into consideration during program generation ( Bunel et al. , 2018 ; Rabinovich et al. , 2017 ; Yin & Neubig , 2017 ; Parisotto et al. , 2017 ; Xu et al. , 2017 ) , the semantic information , which is typically well-defined in the target DSL , is not effectively leveraged by existing work . In light of this observation , in this work , we develop simple yet principled techniques that can be combined with any existing encoder-decoder-style neural program synthesizers . The main novel technique is called execution-guided synthesis . The basic idea is to view the program execution as a sequence of manipulations to transform each input state into the corresponding output state . In such a view , executing a partial program can result in intermediate states ; thus , synthesizing the rest of the program can be conditioned on these intermediate states , so that the synthesizer can take the state changes into account in the followup program generation process . Therefore , we can leverage this idea to combine with any existing encoder-decoder-style neural synthesizer , and we observe that it can significantly improve the performance of the underlying synthesizers . In addition , we also propose a simple yet effective technique called synthesizer ensemble , which leverages the semantic information to ensemble multiple neural program synthesizers . In particular , for the input-output program synthesis problem , we can easily verify if a synthesized program satisfies the input-output specification , which allows us to remove invalid predictions from the ensemble during inference time . Albeit its simplicity , to the best of our knowledge , we are not aware of any previous neural program synthesis work applying this approach . We observe that this technique further boosts the performance substantially . We evaluate our techniques on the Karel task ( Bunel et al. , 2018 ; Devlin et al. , 2017a ) , the largest publicly available benchmark for input-output program synthesis , on which the most performant model in the past can achieve only an accuracy of around 77 % ( Bunel et al. , 2018 ) . We observe that our proposed techniques can gain better performance than the previous state-of-the-art results . In particular , by combining both of our techniques , we can achieve an accuracy of 92 % , which is around 15 percentage points better than the state-of-the-art results . This shows that our approach is effective in boosting the performance of algorithms for neural program synthesis from input-output examples . 2 NEURAL PROGRAM SYNTHESIS FROM INPUT-OUPUT EXAMPLES . In this section , we first introduce the input-output program synthesis problem and existing encoderdecoder-style neural program synthesis approaches , then present an overview of our approaches . 2.1 PROBLEM DEFINITION . We follow the literature ( Devlin et al. , 2017b ; Bunel et al. , 2018 ; Chen et al. , 2018 ) to formally define the input-output program synthesis problem below . Problem Definition 1 ( Program emulation ) LetL be the space of all valid programs in the domainspecific language ( DSL ) . Given a set of input-output pairs { ( Ik , Ok ) } Kk=1 ( or { IOK } in short ) , where there exists a program P ∈ L , such that P ( Ik ) = Ok , ∀k ∈ { 1 , ... , K } . Our goal is to compute the output Otest for a new test input Itest , so that Otest = P ( Itest ) . Although the problem definition only requires to compute the output for a test input , a typical method is to synthesize a program P ′ ∈ L such that P ′ is consistent with all input-output pairs { IOK } , and then use P ′ to compute the output . In this case , we say program P ′ emulates the program P corresponding to { IOK } . In particular , in this work , we are mainly interested in the following formulation of the problem . Problem Definition 2 ( Program synthesis ) Let L be the space of all valid programs in the domainspecific language ( DSL ) . Given a training dataset of { IOK } i for i = 1 , ... , N , where N is the size of the training data , compute a synthesizer Γ , so that given a test input-output example set { IOK } test , the synthesizer Γ ( { IOK } test ) = P produces a program P , which emulates the program corresponding to { IOK } test . 2.2 ENCODER-DECODER-STYLE NEURAL PROGRAM SYNTHESIS APPROACHES . There have been many approaches proposed for different neural program synthesis tasks , and most of them follow an encoder-decoder-style neural network architecture ( Bunel et al. , 2018 ; Devlin et al. , 2017b ; Parisotto et al. , 2017 ) . Figure 1 shows a general neural network architecture for input-output program synthesis . First , an encoder converts input-output examples { IOK } into an embedding . For example , RobustFill ( Devlin et al. , 2017b ) deals with the string transformation tasks , thus it uses LSTMs as the encoder . For the Karel task , both inputs and outputs are 2D grids ( see Figure 2 ) ; therefore , Bunel et al . ( 2018 ) employ a CNN as the encoder . Once the IO embeddings are computed , these approaches employ an LSTM decoder to generate the programs conditioned on the embeddings . For program synthesis , one unique property is that the generated program should satisfy the synthax of L. Therefore , a commonly used refinement is to filter syntactically invalid program prefixes during generation ( Parisotto et al. , 2017 ; Devlin et al. , 2017b ; Bunel et al. , 2018 ) . In the above approaches , only the syntax information is leveraged ; the semantics of L is not utilized . In particular , standard supervised training procedure could suffer from program aliasing : for the same input-output examples , there are multiple semantically equivalent programs , but all except the one provided in the training data will be penalized as wrong programs . To mitigate this issue , Bunel et al . propose to train with reinforcement learning , so that it rewards all semantically correct programs once they are fully generated ( Bunel et al. , 2018 ) . In our work , we demonstrate that we can leverage the semantic information in an effective way that provides a better performance . 2.3 AN OVERVIEW OF OUR APPROACHES . In this work , we propose two general and principled techniques that can improve the performance over existing work , which are execution-guided synthesis ( Section 3 ) and synthesizer ensemble ( Section 4 ) . The main idea of our techniques is to better leverage the semantics of the language L during synthesis . Meanwhile , our techniques are compatible with any existing encoder-decoder-style neural program synthesis architecture . We will describe these techniques in detail in the following sections . 3 EXECUTION-GUIDED SYNTHESIS . Existing approaches generate the entire program only based on the input-output examples before execution . However , this is an inefficient use of the semantics of L. For example , when a program consists of a sequence of statements , we can view the output to be a result by continuously executing each statement in the sequence to convert the input state into a sequence of intermediate states . Figure 2 illustrates such an example . From this perspective , instead of generating the whole program at once , we can generate one statement at a time based on the intermediate/output state pairs . However , most interesting programs are not just sequential . In this work , we explore this idea using a general control-flow framework . In particular , given any language L , we extend it with three classical types of control-flow : sequential , branching , and looping . The extended language is call Lext . Then , we develop our above idea based on Lext , called execution-guided synthesis . In the following , to make our discussion concise , we first formalize Lext ( Section 3.1 ) , and then present the idea of execution-guided synthesis ( Section 3.2 ) . 3.1 THE FORMAL SPECIFICATION OF THE EXTENDED LANGUAGE Lext In this work , we assume some additional control-flow syntax on top of L. We define the extended language Lext in Table 1 . In particular , we assume that a code block B can be composed by a sequence of statements S ∈ L or sub code blocks , and each code block can also be an if-statement or a while-statement . We use C to indicate a condition expression in L , and ⊥ to indicate the termination of a program execution . The semantics of Lext is specified in Table 2 . These rules are largely standard following the convention in programming language literature . In particular , each rule ’ s name starts with S- indicating that they are semantics rules ; the suffixes indicate the constructors each rule specifies , ( e.g. , Stmt for statements , Seq for sequences , etc. ) . The judgment 〈B , s〉 → 〈B′ , s′〉 indicates a small-step execution of program B over state s to result in a new program B′ and a new state s′ . The judgments 〈S , s〉 ⇓ s′ and 〈C , s〉 ⇓ b capture the big-step execution in L that statement S evaluates to a new state s′ from s , and condition C evaluates to a boolean value b from s. Following the semantics of Lext , we can formally define a program execution . Definition 1 ( Program execution ) Given a program P ∈ Lext and an input I , the execution is a sequence s0 ... sT , such that ( 1 ) s0 = I ; ( 2 ) B0 = P ; ( 3 ) 〈Bi , si〉 → 〈Bi+1 , si+1〉 for i = 0 , ... , T−1 ; and ( 4 ) BT = ⊥ . The output of the program is O = sT . 3.2 EXECUTION-GUIDED SYNTHESIS ALGORITHM . In Definition 1 , we can observe that the initial and final states are simply two special states provided as the input-output examples of the synthesis problem . Thus , a synthesizer Γ for input-output pairs should also be able to take any state-pairs as inputs . Our execution-guided synthesis algorithm takes advantage of this fact to improve the performance of the synthesizer . In the following , we discuss three cases from the easiest to the hardest to present our approach . Sequential programs . We now consider the simplest case , where the program is in the form of S1 ; ... ; ST , to illustrate the basic idea of execution-guided synthesis algorithm . We present the algorithm in Algorithm 1 . Assuming the input-output examples are { IOK } , we can treat them as K state-pairs { ( ski , sko ) } Kk=1 , where ski = Ik , sko = Ok . The Exec algorithm takes the synthesizer Γ and the input-output pairs { IOK } as its input . It also takes an additional input ∆ , which is the ending token to be synthesized . For the top-level program , ∆ will be the ⊥ token . Later we will see that when synthesizing the sub-program for If- and While-blocks , different ending tokens will be used . The synthesized program is initially empty ( line 3 ) . Then the algorithm iteratively generates one statement S at a time ( line 4-10 and 17 ) , and appends it to the end of P ( line 16 ) . Importantly , if S is not an if-statement or a while-statement , for which we handle separately , the algorithm executes the newly generated statement S to transit ski into sknew ( line 11-14 ) . Therefore , in the subsequent iteration , the synthesizer can start from the new states sknew after executing the partial program P Algorithm 1 Execution-guided synthesis ( sequential case ) 1 : function EXEC ( Γ , { ( ski , sko ) } Kk=1 , ∆ ) 2 : // The main algorithm is called using Exec ( Γ , { IOK } , ⊥ ) 3 : P ← ⊥ 4 : S ← Γ ( { ( ski , sko ) } Kk=1 ) 5 : while S 6= ∆ do 6 : if S = if -token then // If-statement synthesis 7 : S , { ( ski , sko ) } Kk=1 ← ExecIf ( Γ , { ( ski , sko ) } Kk=1 ) 8 : else 9 : if S = while-token then // While-statement synthesis 10 : S , { ( ski , sko ) } Kk=1 ← ExecWhile ( Γ , { ( ski , sko ) } Kk=1 ) 11 : else // Execution of S 12 : 〈S , ski 〉 → 〈⊥ , sknew〉 for k = 1 , ... , K 13 : ski ← sknew for k = 1 , ... , K 14 : end if 15 : end if 16 : P ← P ; S 17 : S ← Γ ( { ( ski , sko ) } Kk=1 ) 18 : end while 19 : return P 20 : end function Algorithm 2 Execution-guided synthesis ( if-statement ) 1 : function EXECIF ( Γ , I ) 2 : C ← Γ ( I ) 3 : It ← { ( si , so ) ∈ I|〈C , si〉 ⇓ true } 4 : If ← { ( si , so ) ∈ I|〈C , si〉 ⇓ false } 5 : Bt ← Exec ( Γ , It , else-token ) 6 : Bf ← Exec ( Γ , If , fi-token ) 7 : I ′t ← { ( snew , so ) | ( si , so ) ∈ It ∧ 〈Bt , si〉 ⇓ snew } 8 : I ′f ← { ( snew , so ) | ( si , so ) ∈ If ∧ 〈Bf , si〉 ⇓ snew } 9 : I ← I ′t ∪ I ′f 10 : S ← if C then Bt else Bf fi 11 : return S , I 12 : end function generated so far . In doing so , the synthesizer can see all intermediate states to better adjust the followup synthesis strategies to improve the overall synthesis performance . Branching programs . Dealing with if-statements is slightly more complicated than sequential programs , since in an if-statement , not all statements will be executed on all inputs . Following the execution in Algorithm 1 naively , we have to use Γ to synthesize the entire if-statement before being able to execute the partially generated program to derive intermediate states . Therefore , in Algorithm 2 , we extend the above idea to handle if-statements . When the next predicted token is an if -token , our execution-guided synthesizer first predicts the condition of the if-statement C ( line 2 ) . Then , we evaluate C over all state-pairs . Based on the evaluation results , we can divide the IO pairs into two sets It and If ( line 3-4 ) , so that all states in the former meet the branching condition to go to the true branch , and all states in the latter go to the false branch . Therefore , in the followup synthesis , we do not need to consider If ( or It ) when synthesizing the true branch ( or the false branch ) ( line 5-6 ) . Note that in line 5-6 , synthesizing both true-branch and false-branch employ execution-guided synthesis algorithm to leverage intermediate states , and different ending tokens ( i.e. , else and fi ) are supplied respectively . Once we have done the synthesis of both branches , we can execute the generated branches to get the new states I ( line 7-9 ) , and return the newly generated if-statement and updated states to the caller of this algorithm . In Algorithm 2 , we use 〈B , s〉 ⇓ s′ to indicate a big-step execution of code block B over state s to get s′ . In particular , this means that 〈B , s〉 → 〈B1 , s1〉 → ... → 〈⊥ , s′〉 . Looping programs . The remaining problem is to handle while-statements . Due to the rule S-While ( see Table 2 ) , a while-statement while C do B end ( 1 ) is equivalent to if C then ( B ; while C do B end ) else ⊥ fi ( 2 ) Therefore , we can employ a procedure similar to Algorithm 2 once a while-token is predicted . However , there are two differences . First , in ( 2 ) , the false-branch is empty , thus we do not need to deal with the false-branch . Second , although the true-branch is B ; while C do B end , once we have generated B , we do not need to generate the rest of the true-branch , since both C and B have been generated . The detailed algorithm can be found in Appendix B . Remarks . The final algorithm is called by Exec ( Γ , { IOK } , ⊥ ) . Note that our execution-guided synthesis algorithm can be applied to any neural synthesizer Γ , and we can train the synthesizer Γ using any supervised or reinforcement learning algorithms that have been proposed before ( Devlin et al. , 2017b ; Bunel et al. , 2018 ) . In our evaluation , we demonstrate that our execution-guided synthesis technique helps boost the performance of both supervised and reinforcement learning algorithms proposed in existing work ( Bunel et al. , 2018 ) .

Summary: This paper presents two new ideas on leveraging program semantics to improve the current neural program synthesis approaches. The first idea uses execution based semantic information of a partial program to guide the future decoding of the remaining program. The second idea proposes using an ensembling approach to train multiple synthesizers and then select a program based on a majority vote or shortest length criterion. The ideas are evaluated in the context of the Karel synthesis domain, and the evaluation shows a significant improvement of over 13% (from 77% to 90%).