Neural Program Synthesis with Query

1 INTRODUCTION . Program synthesis is the task of automatically finding a program that satisfies the user intent expressed in the form of some specifications like input-output examples ( Gulwani et al. , 2017 ) . Recently , there has been an increasing interest in tackling it using neural networks in various domains , including string manipulation ( Gulwani , 2011 ; Gulwani et al. , 2012 ; Devlin et al. , 2017b ) , list processing ( Balog et al. , 2017 ; Zohar & Wolf , 2018 ) and graphic applications ( Ellis et al. , 2018 ; 2019 ) . Despite their promising performance , most of their success relies on the well-designed input-output examples , without which the performance of the program synthesis model drops heavily . Traditionally , the input-output examples are selected elaborately . For example , in Karel ( Devlin et al. , 2017b ; Bunel et al. , 2018 ) , the given input-output examples are required to have a high branch coverage ratio on the test program ; In list processing , the given input-output examples are guided by constraint propagation ( Balog et al. , 2017 ) to ensure their effectiveness on the unknown program . However , providing such input-output examples is unrealistic because it requires the users to be experienced programmers to describe the underlying program with several input-output examples . Worse , the users must be familiar with the distribution of the training dataset to avoid input-output examples that are out-of-distribution ( Shin et al. , 2019 ) . Some researchers have tried to solve this problem by search or random selection to find counter examples iteratively ( Jha et al. , 2010 ; Laich et al. , 2020 ; Hajipour et al. , 2020 ) . However , they do not model the similarity between the input-output examples and programs accurately , and thus the quality of the examples can not be guaranteed . In summary , how to generate informative input-output examples without expert experience is still an important problem that remains a challenge . In this paper , we propose a query-based framework that can tackle the above-mentioned problems efficiently . Inspired by the interactive program synthesis , the query-based framework trains a query neural network to generate informative input-output examples by querying the underlying program iteratively . Specifically , we firstly train a query network with programs and then use it to generate informative input-output examples . After that , we use these generated examples to train the program synthesis network . This framework has three advantages : ( 1 ) There is no need for well-designed input-output examples in both training and testing , which leads to a good generalization . ( 2 ) The query network works in a generative manner , which can essentially reduce the computation cost Predicted programIOs Generate SynthesizeInteract Embed Query network Synthesis networkUser intent ( program ) Figure 1 : The query-based framework . The query network generates informative input-output examples by interacting with the user , and then the generated examples are sent to the synthesis network to synthesize the underlying program . while facing problems with large input-output space . ( 3 ) The query network serves as a plug-andplay module with high scalability , for it is separate from the program synthesis process entirely . To train the query network , we model the similarity between input-output examples and programs by introducing the functional space ( F-space ) and defining the distance between programs in it . Each program can be projected into a vector in F-space . The distance between two vectors indicates the functional difference between these two programs , so if two programs are projected into the same vector in the F-space , they are supposed to be functionally equivalent . Each input-output example corresponds to a set of programs that satisfy it , and thus it can be represented by a set of vectors in F-space . Furthermore , we project this set of vectors to a distribution ( Sun & Nielsen , 2019 ) to make the representation continuous and differentiable , and thus can be optimized by the back-propagation algorithm . With the concept of information theory , we train the query network to generate queried input-output examples that maximize the probability of the corresponding program and minimize the probability of the others , resulting in the maximization of the mutual information between the queries and the corresponding programs . In this way , the query network can generate informative queried input-output examples efficiently . We evaluate our method on the Karel task and the list processing task . Without utilizing the welldesigned input-output examples both in training and testing , we achieve and even outperform the state-of-the-art performance on the Karel task and list processing task , which shows the effectiveness and generalization of our query-based framework . 2 OVERVIEW . In this section , we give the statement of the query problem and then formalize it . 2.1 PROBLEM STATEMENT . To give an intuition of the query problem , consider an oracle that contains some kind of symbolic rules ( e.g.programs ) and our goal is to discover the symbolic rules inside this oracle . To do this , the traditional program synthesis assumes that the oracle can provide some informative signals automatically , based on which a well-designed synthesizer can find the rules . However , this is a strong assumption on the oracle and can not cover every situation . In this work , we consider a more general problem called the query problem , that the informative signals are gained actively by querying the oracle , and the only requirement for the oracle is that it can respond to all kinds of queries ( including meaningless ones ) . The crux of this query problem is , how to generate queries that can help us to identify the underlying rules in the oracle efficiently ? For the sake of illustration , consider the program synthesis task , the oracle corresponds to the underlying program p∗ that satisfies the user intent , and the ( query , response ) pairs correspond to the input-output examples JeK = { ( xk , yk ) } K . To measure different queries , we borrow the terminology ” acquisition function ” from Bayesian optimization . The acquisition function scores all possible queries and the query algorithm chooses the one with the highest score as the next query . We set the acquisition function to be the mutual information between the queries and the oracle I ( JeK ; p∗ ) , maximizing which brings more information gain . Specifically , given input-output examples queried so far JeKt−1 = { ( x1 , y1 ) , ( x2 , y2 ) , · · · ( xt−1 , yt−1 ) } , we train the query network to pick the query xt which obtains the response yt and can maximize I ( JeK ∪ { ( xt , yt ) } ; p∗ ) . The mutual information is hard to calculate analytically . Thus , to approximate the I ( JeK ∪ { ( xt , yt ) } ; p∗ ) , we adopt the InfoNCE ( van den Oord et al. , 2018 ) and propose a novel way to model the relationship between programs and input-output examples called functional space ( F-space ) . Before dive into the specific method , we will give a problem formulation to make our problem description more accurate . 2.2 PROBLEM FORMULATION . To lay a theoretical foundation of our query method , we give out the problem formulation as follows . First , we define the functional equivalence , which is consistent with the definition of equivalence of functions in mathematics . Definition 2.1 ( Functional equivalence ) . Let I be the input example domain containing all valid input examples , O be the output example domain containing all possible output examples , and P be the program domain containing all valid programs under the domain specific language ( DSL ) , each program p ∈ P can be seen as a function p : I → O . For two programs pi ∈ P and pj ∈ P , pi and pj are functional equivalent if and only if ∀x ∈ I , pi ( x ) = pj ( x ) . Using the concept of functional equivalence , we can formulate the program synthesis task : Definition 2.2 ( Program synthesis ) . Suppose there is an underlying program p ∈ P and K inputoutput examples JeK = { ( xk , yk ) | ( xk , yk ) ∈ I × O } K generated by it , the program synthesis task aims to find a program p̂ which is functional equivalent to p using JeK . Following the definitions above , we can define the task of query . Definition 2.3 ( Query ) . The process of query is a map fq . Given a program p ∈ P and a set of history K input-output examples JeK = { ( xk , yk ) | ( xk , yk ) ∈ I × O } K , the process of query generates a query xk+1 = fq ( JeK ) ∈ I , and then get a response yk+1 = p ( xk+1 ) ∈ O . The query and response are added to history input-output examples : JeK ← JeK ∪ { ( xk+1 , yk+1 ) } and this process repeats . The query process aims to distinguish the underlying program with as few queries as possible , which is consistent with the acquisition function that each query should contain as much information as possible . 3 METHODS . The query network and the synthesis network are trained separately . The query network is trained to find the query contains that is most informative , and after that , the synthesis network is trained to predict the corresponding program using the queried input-output examples . In the following , first , we introduce the F-space to model the similarity between the input-output examples and programs , and then , we illustrate how to train the query network with F-space . 3.1 F-SPACE . To model the query process with a neural network , we propose the functional space ( F-space ) . Definition 3.1 ( F-space and functional distance ) . F-space is a |I| dimensional space which consists of all valid programs that can be implemented by program domain P. Each program is represented by |I| different output examples v = ( y1 , y2 , . . . , y|I| ) . The distance in F-space can be measured by the number of different output examples : d ( vi , vj ) = |diff ( vi , vj ) | . Intuitively , F-space ( P , d ) measures the functional differences between programs . Each program can be represented by a vector in F-space , and if different programs are represented by the same vector v = ( y1 , y2 , . . . , y|I| ) in F-space , it indicates that these two programs get the same result for all inputs , and they are considered functional equivalent which is consistent with Definition 2.1 . In practice , a space with dimension |I| is too large to compute . To solve this problem , we utilize the sparsity of F-space and learn an approximate space with dimension reduction by neural networks . Regarding input-output examples , representing them by vectors is not appropriate . Consider a set of input-output examples JeK = { ( xk , yk ) } K , we can not find a vector that represents it when K < |I| because there are more than one programs which satisfies these K input-output examples . For example , when there is only one input-output example JeK = { ( 2 , 4 ) } , we can not distinguish the underlying program y = 2x from y = x2 . Thus , when K < |I| , the input-output examples JeK = { ( xk , yk ) } K should be modeled as a set of vectors in F-space , projected by the set of programs that satisfy them . We summarize the properties of the representation of input-output examples in F-space as follows : • Each set of input-output examples JeK = { ( xk , yk ) } K should be represented by a set of F-space vectors JrK = { vn } N . • For two sets of input-output examples JeKi and JeKj , if JeKi ⊆ JeKj , then their F-space representations JrKi and JrKj have the relationship JrKi ⊇ JrKj . • For two sets of input-output examples JeKi and JeKj , suppose JeK′ = JeKi ∩ JeKj , then in F-space , their corresponding representations have the relationship JrK′ = JrKi ∪ JrKj • Following the property above , if JeK′ = JeKi ∪ JeKj , then in F-space , their corresponding representations have the relationship JrK′ = JrKi ∩ JrKj Representing input-output examples by sets satisfies all these properties already . However , to optimize the representation using neural networks , we need to make it continuous and differentiable . Thus , we model the representation of JeK in F-space as a Normal distribution ( Ren & Leskovec , 2020 ; Sun & Nielsen , 2019 ) , and the probability of the distribution indicates the possibility that it is the underlying program to be synthesized given input-output examples JeK . We illustrate the projection , the subset relationship , and the union/intersection operation of distributions in Figure 2 . Under this representation , the query process becomes the process of reducing the uncertainty of the distribution . As described in Section 2 , the query process aims to distinguish the underlying program with as few queries as possible . In F-space , the equivalent description is , the training of the query network aims to find F-space using as few queries as possible . To train the query network , we define several neural operators for the neural network training as follows . Program projection . To project a program p to a vector v , we traditionally use a sequence model as our encoder : v = Encoderp ( p ) . ( 1 ) Note that the program synthesis model differs largely on different datasets , so we choose to vary our program encoder according to the specific program synthesis models on different datasets . Mostly , our program encoder is the reverse of the program synthesis decoder . Input-output example projection . Given one input-output example JeK = { ( x , y ) } , we can project it into a Normal distribution using the same architecture of the corresponding program synthesis model , except that we add a MLP to output the two parameters of the Normal distribution : µ and log ( σ2 ) . [ µ , log ( σ2 ) ] = MLPe ( Encodere ( JeK ) ) , ( 2 ) where MLP means a multi-layer perceptron . Using this projection , we can model JeK as N ( µ , log ( σ2 ) ) in F-space . Input-output examples intersection . Given K input-output examples JeK = { ( xk , yk ) } K , each example { ( xk , yk ) } is represented by a Normal distribution N ( µk , σk ) using the projector above . The purpose of the intersection is to aggregate these Normal distributions into a new one , which represents JeK in F-space . Under the assumption of independence , the probability of the intersection distribution should be : Printersection = K∏ i=1 Pri . ( 3 ) Fortunately , the product of independent Normal distributions is still a Normal distribution , which means that we can represent it in [ µ′ , log ( σ′2 ) ] . In practice , we utilize the attention mechanism with another MLP to let the neural network learn the new Normal distribution ( Ren & Leskovec , 2020 ) : [ µ′ , log ( σ′2 ) ] = K∑ i=1 wi [ µi , log ( σ 2 i ) ] , ( 4 ) wi = exp ( MLPattention ( [ µi , log ( σ 2 i ) ] ) ) ∑K j=1 exp ( MLPattention ( µj , log ( σ 2 j ) ) ) . ( 5 ) This formulation not only keeps the form of distributions closed but also approximates the mode of the effective support of Normal distributions , which satisfies our requirement on the intersection of distribution ( Sun & Nielsen , 2019 ) . Inverse projection to query . Finally , we need to generate the query from the representation of input-output examples in F-space . Using the projection and the intersection operation mentioned above , we can project the K input-output examples to the representation [ µ , log ( σ2 ) ] . To generate query xnew from this representation , we introduce a decoder which has a similar architecture to Encodere but reversed : xnew = Decoder ( [ µ , log ( σ 2 ) ] ) . ( 6 ) Next , we will introduce how to train the neural network .

Summary: The authors propose a framework to generate unit tests by generating candidate inputs for which labels are queried from an oracle. The idea is to have a vector space where each point represents a function (different programs that are functionally equivalent map to the same point) and a unit test is represented as a set of points that correspond to functions that would pass that unit test.   They project a unit test onto a normal distribution on the function space where the probability of the distribution is whether a point is the underlying function to be synthesized given the unit test.   They merge multiple unit tests using an attention mechanism into a single point in the function space and then train the unit test generator such that the mutual information between the probability distribution on the merged unit test and the probability distribution on the programs is maximized using InfoNCE loss.  A synthesizer network is used to synthesize the program from generated unit tests. They test on Karel and list processing task.