Adversarial Synthetic Datasets for Neural Program Synthesis

1 INTRODUCTION . Program synthesis has long been a key goal of AI research . In particular , researchers have become increasingly interested in the task of programming by example ( PBE ) , where the goal is to generate a program consistent with a given set of input-output ( I/O ) pairs . Recent studies have achieved impressive results , capable of solving PBE problems that humans would find difficult ( e.g. , Sharma et al . ( 2017 ) ; Zohar & Wolf ( 2018 ) ; Ellis et al . ( 2019 ) ) . However these studies have a concerning weakness : since large , naturally occurring datasets of program synthesis problems do not exist , these studies train and test their models on synthetic datasets of randomly generated programs and I/O pairs . The justification for using these synthetic datasets is that if a model can correctly predict programs for arbitrary PBE problems , then it has likely learned the semantics of the programming language and can generalize to problems outside the synthetic data distribution ( Devlin et al. , 2017 ) . While this justification is plausible , a model might also perform well because it has learned specific aspects of the synthetic data distribution , and recent studies have found this to be the case for several state-of-the-art models ( Shin et al. , 2019 ; Clymo et al. , 2019 ) . These studies find that current PBE models often perform poorly on distributions different from that of the training data , and they propose methods to mitigate this issue by generating synthetic data with more varied distributions . The idea behind these methods is that a model trained on more varied synthetic data should generalize to a wider variety of distributions , hopefully including those of real-world PBE problems . Nevertheless , we find that these methods are often insufficient . Previous studies differ on what constitutes a “ varied distribution ” of synthetic data , creating definitions based on problem-specific heuristics . While generating training data based on these heuristics does help models generalize to certain distributions , we find that models trained using these methods still fail to generalize to many other distributions , including those resembling distributions of real-world problems . Moreover , different methods fail to generalize to different distributions , raising the question of how one should construct test sets to evaluate these methods . While previous studies have arbitrarily picked test sets that they believe present a reasonable challenge for state-of-the-art methods , this approach may lead to overly optimistic evaluations . A study may report that a method performed well because the researchers failed to find those distributions on which the method performs poorly . In this paper , we propose an adversarial method to generate a training set . Our adversarial approach builds a training set iteratively , finding data distributions on which a given model performs poorly and adding data drawn from those distributions to the training set on each iteration . We test this method by using it to generate training data for the PCCoder model from Zohar & Wolf ( 2018 ) , and we show that models trained using our method generalize to a variety of distributions better than previously proposed methods . Moreover , we propose using a variation of our adversarial approach to generate test sets to evaluate PBE methods . We create test sets for different versions of PCCoder using this approach and show that these test sets reveal weaknesses in models that are not obvious when using other test sets . This paper makes the following key contributions : 1 . We propose a new , adversarial method to generate desirable distributions on which to train models for PBE . 2 . We show that models trained using our method generalize to a variety of datasets better than models trained using previously proposed methods . 3 . We show that our adversarial approach may also be used to generate test sets that are less likely to overestimate the performance of a model . 2 RELATED WORK . Most studies on PBE methods generate I/O pairs using random sampling schemes , filtering out invalid I/O pairs for each program by constraining the sample space and rejecting sets of I/O pairs that do not meet these constraints . Balog et al . ( 2016 ) construct a dataset of PBE problems for DeepCoder by enumerating programs up to a given length and removing programs with easily detectable issues ( e.g. , redundant variables ) . For each program generated , they then create I/O pairs by sampling inputs uniformly from a restricted range of values guaranteed to yield valid outputs for the given program . Feng et al . ( 2018 ) and Zohar & Wolf ( 2018 ) also create datasets using the DeepCoder DSL ( domain-specific language ) in a similar manner . Bunel et al . ( 2018 ) generate PBE problems for Karel ( Pattis , 1981 ) by randomly sampling programs from the Karel DSL and removing programs with obvious problems , similar to Balog et al . They then generate I/O pairs for each program by sampling random inputs and running the program to obtain the corresponding outputs . However , Bunel et al . do not specify what sampling distributions are used for the programs and I/O pairs . Parisotto et al . Parisotto et al . ( 2016 ) create a dataset for the Flashfill domain ( Gulwani et al. , 2012 ) by enumerating programs up to 13 expressions long and then randomly sampling inputs to create I/O pairs . They report that while their model achieves 97 % accuracy on their synthetic data , they achieve only 38 % accuracy on a small dataset of 238 real-world problems . Devlin et al . ( 2017 ) use a data generation approach similar to Parisotto et al . but with an improved model and are more successful , achieving 92 % accuracy on the same real-world dataset used by Parisotto et al . All of the papers above focus on advancing models for PBE , but they do so largely using synthetic data to train those models . Shin et al . ( 2019 ) report that even minor differences between the synthetic data distributions used for training and evaluation can drastically decrease a model ’ s performance . To solve this problem , they propose a data generation method to improve a model ’ s ability to generalize to other data distributions . They first choose a set of “ salient variables ” for the domain , defined as a mapping from I/O pairs in the synthetic dataset to a finite , discrete set . They then sample I/O pairs such that the salient variables will be approximately uniformly distributed in the resulting dataset . Shin et al . find that the model proposed by Bunel et al . ( 2018 ) better generalizes to a variety of distributions when trained with data generated with this method . However , this method has two major disadvantages . First , it requires the user to determine the correct salient variables , which may be difficult for complex domains . Second , if the domain of valid I/O pairs is highly dependent on the program , it is often prohibitively complex to enforce uniformity across salient variables . Recently , Clymo et al . ( 2019 ) proposed a method to generate PBE problems using a SMT solver . They impose constraints on the I/O pairs to ensure that pairs selected for the dataset are not too similar to each other and then select I/O pairs that satisfy these constraints using a SMT solver . However , when testing an implementation of this method on the DeepCoder domain , the reported improvement of the constraint-based methods over simpler sampling methods is marginal , with the best constraint-based method performing only 2.4 % better than the best sampling method . Moreover , many of their constraints are highly specific to the DeepCoder domain , and Clymo et al . do not offer a way to adapt their method to other problem spaces . Nevertheless , they present a method that does not require the salient variables of Shin et al . ( 2019 ) , and they show that their approach can be applied to domains such as DeepCoder , for which it is difficult to enforce uniformity across salient variables . Our adversarial method is closely related to the literature on adversarial training . Most similar to our work is that of Volpi et al . ( 2018 ) , who propose an adversarial training procedure to learn models resistant to covariate shift . Given a model M trained on data from a given domain X , Volpi et al . generate synthetic training data by sampling data points for which M performs poorly , under the constraint that the sampled data points must be within a certain Wassertein distance ρ from X . By retraining M on the synthetic data , Volpi et al . create a new model resistant to covariate shifts of magnitude ρ . However , a key difficulty in applying their method is that the user must predict the magnitude of a covariate shift before it actually occurs in order to select ρ . Moreover , our approach differs from that of Volpi et al . in that our goal is to train a model to generalize to virtually any distribution of data valid for a given programming problem , where as Volpi et al . try to train a model to be resistant to a single , anticipated covariate shift . Our method also does not require the user to predict a hyperparameter such as ρ to train a model . Other studies have also proposed adversarial methods to generate synthetic training data ( e.g. , Goodfellow et al . ( 2015 ) ; Sinha et al . ( 2020 ) ) . However , these methods try to train a model to be resistant to imperceptible adversarial attacks , which make small perturbations to training examples such that the perturbed inputs cause the target model to output incorrect answers with high confidence . Our method tries to train a model to generalize to all valid data distributions , not just the small perturbations found in adversarial attacks . Some studies have proposed iteratively constructing training sets based on the concept of “ surprise. ” In the field of program synthesis , Pu et al . ( 2017 ) generate I/O pairs to specify a program by first generating a large set of candidate I/O pairs and then iteratively selecting the I/O pairs from this set that are most surprising , where surprise is estimated using a neural network trained to predict which I/O pairs will be selected ( I/O pairs assigned lower probabilities by the neural network are more surprising ) . Pu et al . use this approach to ensure that their I/O pairs are sufficient to specify desired programs unambiguously . In contrast , our approach tries to find and generalize to out-ofdistribution data , and therefore generates I/O pairs to maximize the difficulty of the resulting PBE problems rather than maximizing surprise . Our adversarial method uses an evolutionary algorithm to find desirable training distributions . This approach is similar to that of studies on competitive co-evolution ( e.g. , Rosin & Belew ( 1997 ) ; Arcuri & Yao ( 2007 ) ) . For example , Arcuri & Yao ( 2014 ) co-evolve populations of programs and populations of unit tests given a formal specification of a desired program . The fitness value of a candidate program depends on how many unit tests it passes , while the fitness value of a candidate unit test depends on how many programs it causes to fail . However , the goal of Arcuri & Yao ( 2014 ) and similar works is to simply generate a vanilla program synthesis algorithm capable of generating a program given its formal specification . They do not try to ensure that their algorithms generalize to different data distributions as we do ( e.g. , their algorithm might only work for certain distributions of formal specifications ) . Furthermore , in order to compute fitness values , co-evolutionary methods require the user to provide a formal specification of the desired program ( e.g. , using Z notation ( Spivey & Abrial , 1992 ) ) . Our approach only requires the user to provide I/O pairs describing the behavior of a desired program , which tend to be easier to provide than a formal specification ( Palshikar , 2001 ) .

Summary: Synthesis models trained on synthetic datasets of randomly generated programs and corresponding IO pairs often fail to generalize to real-world PBE problems. The models often learn specific aspects of the synthetic data distribution rather than the semantics of the programming language. This work proposes a more principled adversarial approach to generate synthetic data. The training set is built iteratively by finding data distributions on which a given model performs poorly and adding data drawn from these distributions to the training set. The model and the training dataset are built in tandem. The candidate distributions are generated by mutating the current population of distributions. Models trained using this method are shown to generalize to a variety of distributions.