Guiding Program Synthesis by Learning to Generate Examples

1 INTRODUCTION . Over the years , program synthesis has been applied to a wide variety of different tasks including string , number or date transformations ( Gulwani , 2011 ; Singh & Gulwani , 2012 ; 2016 ; Ellis et al. , 2019 ; Menon et al. , 2013 ; Ellis & Gulwani , 2017 ) , layout and graphic program generation ( Bielik et al. , 2018 ; Hempel & Chugh , 2016 ; Ellis et al. , 2019 ; 2018 ) , data extraction ( Barowy et al. , 2014 ; Le & Gulwani , 2014 ; Iyer et al. , 2019 ) , superoptimization ( Phothilimthana et al. , 2016 ; Schkufza et al. , 2016 ) , code repair ( Singh et al. , 2013 ; Nguyen et al. , 2013 ; D ’ Antoni et al. , 2016 ) , language modelling ( Bielik et al. , 2017 ) , synthesis of data processing programs ( Polosukhin & Skidanov , 2018 ; Nye et al. , 2019 ) or semantic parsing Shin et al . ( 2019a ) . To capture user intent in an easy and intuitive way , many program synthesizers let its users provide a set of input-output examples I which the synthesized program should satisfy . Generalization challenge A natural expectation of the end user in this setting is that the synthesized program works well even when I is severely limited ( e.g. , to one or two examples ) . Because of this small number of examples and the big search space of possible programs , there are often millions of programs consistent with I . However , only a small number of them generalizes well to unseen examples which makes the synthesis problem difficult . Existing methods Several approaches have provided ways to address the above challenge , including using an external model that learns to rank candidate programs returned by the synthesizer , modifying the search procedure by learning to guide the synthesizer such that it returns more likely programs directly , or neural program induction methods that replace the synthesizer with a neural network to generate outputs directly using a latent program representation . However , regardless of what other features these approaches use , such as conditioning on program traces ( Shin et al. , 2018 ; Ellis & Gulwani , 2017 ; Chen et al. , 2019 ) or pre-training on the input data ( Singh , 2016 ) , they are limited by the fact that their models are conditioned on the initial , limited user specification . This work We present a new approach for program synthesis from examples which addresses the above challenge . The key idea is to resolve ambiguity by iteratively strengthening the initial specification I with new examples . To achieve this , we start by using an existing synthesizer to find a candidate program p1 that satisfies all examples in I . Instead of returning p1 , we use it to find a distinguishing input x∗ that leads to ambiguities , i.e. , other programs pi that satisfy I but produce different outputs p1 ( x∗ ) 6= pi ( x∗ ) . To resolve this ambiguity , we first generate a set of candidate outputs for x∗ , then use a neural model ( which we train beforehand ) that acts as an oracle and selects the most likely output , and finally , add x∗ and its predicted output to the input specification I . The whole process is then repeated . These steps are similar to those used in Oracle Guided Inductive Synthesis ( Jha et al. , 2010 ) with two main differences : ( i ) we automate the entire process by learning the oracle from data instead of using a human oracle , and ( ii ) as we do not use a human oracle to produce a correct output , we need to ensure that the set of candidate outputs contains the correct one . Augmenting an existing Android layout synthesizer In this work we apply our approach to a state-of-the-art synthesizer , called InferUI ( Bielik et al. , 2018 ) , that creates an Android layout program which represents the implementation of a user interface . Given an application design consisting of a set of views ( e.g. , buttons , images , text fields , etc . ) and their location on the device screen , InferUI synthesizes a layout program that when rendered , places the views at that same location . Concretely , each input-output example ( x , y ) consists of a device screen x ∈ R4 and a set of n views y ∈ Rn×4 , all of which are represented using their coordinates in a two dimensional euclidean space . As an example , the input specification I shown in Figure 1 contains a single example with absolute view positions for a Nexus 4 device and the InferUI synthesizer easily finds multiple programs that satisfy it ( dashed box ) . To apply our method and resolve the ambiguity , we find a distinguishing input x∗ , in this case a narrower P4 Pro device , on which some of the candidate programs produce different outputs . Then , instead of asking the user to manually produce the correct output , we generate additional candidate outputs ( to ensure that the correct one is included ) and our learned neural oracle automatically selects one of these outputs ( the one it believes is correct ) and adds it to the input specification I . In this case , the oracle selects output p2 ( x∗ ) as both buttons are correctly resized and the distance between them was reduced to match the smaller device width . In contrast , p1 ( x∗ ) contains overlapping buttons while in pn ( x∗ ) , only the left button was resized . Automatically obtaining real-world datasets An important advantage of our approach is that we reduce the problem of selecting which program generalizes well to the simpler task of deciding which output is correct . This is especially useful for domains , such as the Android layout synthesis , for which the output correctness depends mostly on properties of the output and not on the program used to generate it . As a result , obtaining a suitable training dataset can be easier as we do not require the hard to obtain ground-truth programs , for which currently no large real-world datasets exist ( Shin et al. , 2019b ) . In fact , it is possible to train the oracle using unsupervised learning only , with a dataset consisting of correct input-output examples DU . For example , in layout synthesis an autoencoder can be trained over a large number of views and their positions extracted from running real-world applications . However , instead of training such an unsupervised model , in our work we useDU to automatically construct a supervised datasetDS by labelling the samples inDU as positive and generating a set of negative samples by adding suitable noise to the samples in DU . Finally , we also obtain the dataset DS+ that additionally includes the input specification I . In the domain of Android layouts , although it is more difficult , such a dataset can also be collected automatically by running the same application on devices with different screen sizes . Our contributions We present a new approach to address the ambiguity in the existing Android layout program synthesizer InferUI by iteratively extending the user provided specification with new input-output examples . The key component of our method is a learned neural oracle used to generate new examples trained with datasets that do not require human annotations or ground-truth programs . To improve generalization , InferUI already contains a probabilistic model that scores programs q ( p | I ) as well as handcrafted robustness properties , achieving 35 % generalization accuracy on a dataset of Google Play Store applications . In contrast , our method significantly improves the accuracy to 71 % while using a dataset containing only correct and incorrect program outputs . We make our implementation and datasets available online at : https : //github.com/eth-sri/guiding-synthesizers 2 RELATED WORK . In this section we discuss the work most closely related to ours . Guiding program synthesis To improve scalability and generalization of program synthesizers several techniques have been proposed that guide the synthesizer towards good programs . The most widely used approach is to implement a statistical search procedure which explores candidate programs based on some type of learned probabilistic model – log-linear models ( Menon et al. , 2013 ; Long & Rinard , 2016 ) , hierarchical Bayesian prior ( Liang et al. , 2010 ) , probabilistic higher order grammar ( Lee et al. , 2018 ) or neural network ( Balog et al. , 2017 ; Sun et al. , 2018 ) . Kalyan et al . ( 2018 ) also takes advantage of probabilistic models but instead of implementing a custom search procedure , they use the learned model to guide an existing symbolic search engine . In addition to approaches that search for a good program directly ( conditioned on the input specification ) , a number of works guide the search by first selecting a high-level sketch of the program and then filling in the holes using symbolic ( Ellis et al. , 2018 ; Murali et al. , 2017 ; Nye et al. , 2019 ) , enumerative or neural search ( Bosnjak et al. , 2017 ; Gaunt et al. , 2016 ) . A similar idea is also used by ( Shin et al. , 2018 ) , but instead of generating a program sketch the authors first infer execution traces ( or condition on partial traces obtained as the program is being generated ( Chen et al. , 2019 ) ) , which are then used to guide the synthesis of the actual program . In comparison to prior work , a key aspect of our approach is to guide the synthesis by generating additional input-output examples that resolve the ambiguities in the input specification . Guiding the synthesizer in this way has several advantages – ( i ) it is interpretable and the user can inspect the generated examples , ( ii ) it can be used to extend any existing synthesizer by introducing a refinement loop around it , ( iii ) the learned model is independent of the actual synthesizer ( and its domain specific language ) and instead is focused only on learning the relation between likely and unlikely input-output examples , and ( iv ) often it is easier to obtain a dataset containing program outputs instead of a dataset consisting of the actual programs . Further , our approach is complementary to prior works as it treats the synthesizer as a black-box that can generate candidate programs . We also note that several prior works explore the design of sophisticated neural architectures that encode input-output examples ( Sun et al. , 2018 ; Devlin et al. , 2017 ; Parisotto et al. , 2017 ) and incorporating some of their ideas might lead to further improvements to our models presented in Section 4 . Learning to rank To choose among all programs that satisfy the input specification , existing program synthesizers select the syntactically shortest program ( Liang et al. , 2010 ; Polozov & Gulwani , 2015 ; Raychev et al. , 2016 ) , the semantically closest program to a reference program ( D ’ Antoni et al. , 2016 ) or a program based on a learned scoring function ( Liang et al. , 2010 ; Mandelin et al. , 2005 ; Singh & Gulwani , 2015 ; Ellis & Gulwani , 2017 ; Singh , 2016 ) . Although the scoring function usually extracts features only from the synthesized program , some approaches also take advantage of additional information – Ellis & Gulwani ( 2017 ) trains a log-linear model using a set of handcrafted features defined over program traces and program outputs while Singh ( 2016 ) leverages unlabelled data by learning common substring expressions shared across the input data . Similar to prior work , our work explores various representations over which the model is learned . Because we applied our work to a domain where outputs can be represented as images ( rather than strings or numbers ) , to achieve good performance we explore different types of models ( i.e. , convolutional neural networks ) . Further , we do not assume that the synthesizer can efficiently enumerate all programs that satisfy the input specification as in Ellis & Gulwani ( 2017 ) ; Singh ( 2016 ) . For such synthesizers , applying a ranking of the returned candidates will often fail since the correct program is simply not included in the set of synthesized programs . Therefore , the neural oracle is defined over program outputs instead of actual programs . This reduces the search space for the synthesizer as well as the complexity of the machine learning models . Neural program induction Devlin et al . ( 2017 ) and Parisotto et al . ( 2017 ) , as well as related line of work on neural machines ( Graves et al. , 2016 ; Reed & de Freitas , 2016 ; Bošnjak et al. , 2017 ; Chen et al. , 2018 ) , explore the design of end-to-end neural approaches that generate the program output for a new input without the need for an explicit search . In this case the goal of the neural network is not to find the correct program explicitly , but rather to generate the most likely output for a given input based on the input specification . These approaches can be integrated in our work as one technique for generating a set of candidate outputs for a given distinguishing input instead of obtaining them using a symbolic synthesizer . However , the model requirements in our work are much weaker – it is enough if the correct output is among the top n most likely candidates rather than requiring 100 % precision for all possible inputs as in program induction .

Summary: A method for a refinement loop for program synthesizers operating on input/ouput specifications is presented. The core idea is to generate several candidate solutions, execute them on several inputs, and then use a learned component to judge which of the resulting input/output pairs are most likely to be correct. This avoids having to judge the correctness of the generated programs and instead focuses on the easier task of judging the correctness of outputs. An implementation of the idea in a tool for synthesizing programs generating UIs is evaluated, showing impressive improvements over the baseline.