Synthetic Datasets for Neural Program Synthesis

1 INTRODUCTION . Program synthesis is one of the central problems in Artificial Intelligence studied from the early days ( Manna & Waldinger , 1971 ; Waldinger & Lee , 1969 ) and has seen a lot of recent interest in the machine learning and programming languages community ( Alur et al. , 2013 ; Gulwani et al. , 2012 ; 2015 ; Muggleton , 1991 ; Lin et al. , 2014 ; Solar-Lezama , 2013 ) . The recent neural approaches can be broadly classified into two categories : program induction and program synthesis . Both approaches share the objective of learning program semantics but do so in different ways . Program induction aims to embed the semantics of a particular algorithm into a differentiable model trained end-to-end , whereas the goal of program synthesis is for a model to learn the semantics of a domain-specific language ( DSL ) and produce programs defined by corresponding specifications . Both problems necessitate large datasets , either of I/O pairs in the case of program induction , or programs with corresponding I/O pairs in the case of program synthesis . However , since large datasets for program induction and synthesis tasks do not exist , these approaches train models on large synthetically generated datasets . Presumably , if a model can accurately predict arbitrary program outputs ( for induction ) or programs in the DSL ( for synthesis ) then it has likely learnt the correct algorithm or DSL semantics . Although this approach has led to some impressive synthesis results in many domains , synthetically generating datasets that cover all DSL programs and the corresponding input space can be problematic , especially for more complex DSLs like Karel the Robot ( Pattis , 1981 ) which includes complex control-flow primitives ( while loops and if conditionals ) and operators . Likewise , for induction tasks , the sampling procedure for program specifications may lead to undesirable biases in the training distribution that inhibit strong generalization . In this paper , we consider two problem settings . The first is the Karel domain and the recently proposed Karel synthesis model ( Bunel et al. , 2018 ) . We identify many distributions of input examples and DSL programs for which the Karel synthesis model performs poorly . The second problem setting is a program induction problem in which a model is trained to execute and predict the output of simple arithmetic expressions , which we denote the Calculator domain ; for this domain , ∗ricshin @ cs.berkeley.edu †Machine Learning at Berkeley we considered common synthetic data generation strategies including one from tensor2tensor ( Vaswani et al. , 2018 ) , an open-source deep learning library . Upon analysis , we find evidence of undesirable artifacts resulting from certain biases in the generation algorithm . Our results indicate that models trained with common methodologies for synthesizing datasets fail to learn the full semantics of the DSL , even when they perform well on a test set , and suggest the need of a more principled way to generate synthetic datasets . For some program and input distributions , the state-of-the-art neural synthesis models perform quite poorly , often achieving less than 5 % generalization accuracy . In our paper , we develop a new methodology for creating training distributions over programs in the DSL to mitigate some of these issues . Moreover , unlike previous works that have ignored considering the distributions over input space , we show that input distributions also play a significant role in determining the synthesizer performance . Our methodology involves defining the distribution over DSL programs and input space using a set of random variables to encode much of the valuable features which describe the data , e.g . in the Karel domain , the amount of control flow nesting in programs or the number of markers present in the inputs . Our methodology allows us to identify several specialized distributions over the input space and Karel programs on which the current state of the art synthesis models ( Bunel et al. , 2018 ) perform poorly when trained on traditional program and input distributions , and tested on our new distributions . From this , we design new training distributions by ensuring greater uniformity over the random variables in our methodology . By retraining the same architecture on these new training data distributions , we observe a greater ability to generalize , with significant improvements when evaluated on the aforementioned test sets . We also observe similar improvements in the Calculator domain as well . This paper makes the following key contributions : • We propose a new methodology to generate different desirable distributions over the space of datasets for program induction and synthesis tasks . • We instantiate the methodology for the Karel and Calculator domains and show that model generalization is worse on datasets generated by our technique . • We then retrain models in both domains and demonstrate that models achieve greater overall generalization performance when trained on datasets generated with our methodology . 2 RELATED WORK . 2.1 TRAINING MODELS WITH SYNTHETIC DATA . In certain domains like computer vision and robotics , collecting high-quality real-world training data incurs significant cost , and so many researchers have investigated the use of large amounts of synthetic data . For example , Christiano et al . ( 2016 ) ; Peng et al . ( 2017 ) ; Pinto et al . ( 2017 ) ; Bousmalis et al . ( 2017 ) aim to learn robotics policies that compensate for differences between the real world and the simulation . Within computer vision , Shrivastava et al . ( 2016 ) demonstrate learning from entirely synthetic images for gaze and pose estimation . 2.2 NEURAL PROGRAM INDUCTION AND SYNTHESIS .. Program induction methods learn differentiable modules such as stack ( Joulin & Mikolov , 2015 ) , RAM ( Kurach et al. , 2016 ) , GPU ( Kaiser & Sutskever , 2015 ) , and read-write external memory ( Graves et al. , 2014 ) to represent algorithms . Other methods attempt to learn differentiable control flow operations ( Gaunt et al. , 2016 ; Neelakantan et al. , 2015 ) . These approaches reconstruct outputs given inputs , inferring the underlying algorithms . Other methods instead learn neural modules from program traces rather than from I/O examples ( Reed & de Freitas , 2015 ; Xiao et al. , 2018 ; Cai et al. , 2017 ) . Devlin et al . ( 2017 ) ; Parisotto et al . ( 2017 ) use neural program synthesis techniques for learning string editing programs in RobustFill . Similarly , Balog et al . ( 2016 ) learn array programs in DeepCoder , and Bhupatiraju et al . ( 2017 ) learn to compose API calls . Bunel et al . ( 2018 ) apply neural program synthesis to the Karel domain we consider in our paper ; we use their architecture and dataset . Many of these approaches report high test performance , but good performance on synthetically-specificed programs may not indicate the model ’ s ability to generalize to arbitrary user-desired programs . In fact , our results show that under certain distributions , these models perform quite poorly . To verify these models appropriately generalize to reasonably complex arbitrary programs , the test set should sufficiently represent the universe of these programs and their specifications . Likewise , to avoid biasing the result , the test set should also draw uniformly from these programs and specifications . For example , for RobustFill , the data generation methodology only sampled programs uniformly from the string DSL , but did not take into account the distribution over input strings such as their length , frequencies of occurrence of regular expressions and their nesting , common words and constants etc . 3 DATA GENERATION METHODOLOGY . Currently , automated data generation focuses in large part on a constructive process , whose parameters can be tuned . We propose a complementary approach in which we perform a subsequent filtering step on this process to ensure that the resulting distribution has certain properties . We define a salient random variable as one whose distribution in the final dataset is of interest . In the case of program synthesis , we consider two kinds of salient variables : variables denoting important features of a program in the given DSL , such as its length and degree of nesting ; and variables denoting features for the input space . In many cases , we can modify our sampling procedure to ensure a desirable distribution of a particular salient variable . However , for some salient variables , it is infeasible to tune the parameters of a given sampling procedure in order to obtain a desired distribution for that salient variable . For example , if we sample programs directly from a context-free grammar , it is difficult to control the distribution of various salient variables such as program length , degree of nesting , etc . This is a notable problem in both the Karel and Calculator domains . Furthermore , within the context of program synthesis specifically , there is often an additional challenge : not all inputs are valid for all programs . For example , in the Karel domain , an input for a given program would be invalid if the program attempts to perform illegal actions for that input ( such as move into walls or pickMarker in a cell containing no markers ) . The requirement that the program/input pairs suit each other itself acts as an unpredictable filter that makes it difficult to ensure uniformity of salient variables by tuning generation parameters . As such , we propose a methodology for randomly sampling a dataset D ( consisting of elements of S ) while ensuring that a given salient variable ν : S→ X ( where X is finite and discrete ) , denoted as a random variable X = ν ( s ) , has a uniform distribution throughout D. To do this , we first sample an example s ∼ q ( · ) from an original distribution q . We then add s to D with probability g ( s ) , where g ( s ) = ( Pq [ X = ν ( s ) ] + ε ) −1 ( min x∈X Pq [ X = x ] + ε ) , with Pq [ X ] , the probabilities induced over X via q , calculated empirically based on counts computed with past samples drawn from q . We repeat the above until D is of a desired size . For full pseudocode see Section B.1 in the appendix . We use ε ∈ R+ as a hyperparameter to trade off the runtime of the above procedure with the level of X ’ s uniformity in D. In Section B.2 ( in the Appendix ) , we provide a probabilistic bound on the uniformity of X in the resulting distribution , for the case where ε = 0 . Also , for when ε > 0 , we can show that drawing a single sample is possible in O ( 1ε ) calls to the original sampler ( proof in Section B.3 , with empirical experiments in Section B.4 and B.5 ) . Increasing ε increases the algorithm ’ s speed , at the cost of allowing the distribution of X in D to further diverge from uniform . 4 APPLICATION TO KAREL : EXPERIMENTS WITH NEW TEST DISTRIBUTIONS . Karel is an educational programming language ( Pattis , 1981 ) where the programmer writes imperative programs with conditionals and loops to produce a sequence of actions for an agent ( a robot named Karel ) which lives in a rectangular m × n grid world . For a detailed description of the particular instantiation of the Karel language and input grid specification that we consider , see Section A . The program synthesis task that we consider is as follows : given a set of pairs of input and output grids { ( i1 , o1 ) , · · · , ( in , on ) } , find a Karel program π such that executing π on i1 results in o1 , i2 results in o2 , and so on . In the paper , n = 5 unless otherwise specified . An example Karel synthesis task with the I/O examples and the corresponding Karel program to be synthesized is shown in Figure 1 . In this section , we employ the Karel instantiation of our abstract data generation methodology in Section 4.1 to generate different test datasets . By imposing a more uniform distribution over the salient random variables when generating the I/O specifications and target programs which make up the test set , we observe much lower accuracies of the previous Karel synthesis models ( Bunel et al. , 2018 ) compared to the original test set .

Summary: This paper provides a good presentation of a serious problem in evaluating (as well as training!) performance of machine learning models for program synthesis / program induction: considering specifically the problem of learning a program which corresponds to given input/output pairs, since large datasets of "real-world" programs typically do not exist it necessary to construct a synthetic dataset for training and testing; this requires both (a) generating programs, and (b) generating input/output examples for these programs. Enumerating either all possible programs or examples is typically impossible, and so a sampling scheme is used to simulate "reasonable" programs and examples. This may hinder generalization to other data not often produced by the sampling scheme.