Title: GUIDING PROGRAM SYNTHESIS BY LEARNING TO GENERATE EXAMPLES

Abstract: A key challenge of existing program synthesizers is ensuring that the synthesized program generalizes well. This can be difficult to achieve as the specification provided by the end user is often limited, containing as few as one or two inputoutput examples. In this paper we address this challenge via an iterative approach that finds ambiguities in the provided specification and learns to resolve these by generating additional input-output examples. The main insight is to reduce the problem of selecting which program generalizes well to the simpler task of deciding which output is correct. As a result, to train our probabilistic models, we can take advantage of the large amounts of data in the form of program outputs, which are often much easier to obtain than the corresponding ground-truth programs.

1. INTRODUCTION:

Over the years, program synthesis has been applied to a wide variety of different tasks including string, number or date transformations (Gulwani, 2011; Singh & Gulwani, 2012; 2016; Ellis et al., 2019; Menon et al., 2013; Ellis & Gulwani, 2017) , layout and graphic program generation (Bielik et al., 2018; Hempel & Chugh, 2016; Ellis et al., 2019; 2018) , data extraction (Barowy et al., 2014; Le & Gulwani, 2014; Iyer et al., 2019) , superoptimization (Phothilimthana et al., 2016; Schkufza et al., 2016) , code repair (Singh et al., 2013; Nguyen et al., 2013; D'Antoni et al., 2016) , language modelling (Bielik et al., 2017) , synthesis of data processing programs (Polosukhin & Skidanov, 2018; Nye et al., 2019) or semantic parsing Shin et al. (2019a) . To capture user intent in an easy and intuitive way, many program synthesizers let its users provide a set of input-output examples I which the synthesized program should satisfy.

Generalization challenge A natural expectation of the end user in this setting is that the synthesized program works well even when I is severely limited (e.g., to one or two examples). Because of this small number of examples and the big search space of possible programs, there are often millions of programs consistent with I. However, only a small number of them generalizes well to unseen examples which makes the synthesis problem difficult.

Existing methods Several approaches have provided ways to address the above challenge, including using an external model that learns to rank candidate programs returned by the synthesizer, modifying the search procedure by learning to guide the synthesizer such that it returns more likely programs directly, or neural program induction methods that replace the synthesizer with a neural network to generate outputs directly using a latent program representation. However, regardless of what other features these approaches use, such as conditioning on program traces (Shin et al., 2018; Ellis & Gulwani, 2017; Chen et al., 2019) or pre-training on the input data (Singh, 2016) , they are limited by the fact that their models are conditioned on the initial, limited user specification.

This work We present a new approach for program synthesis from examples which addresses the above challenge. The key idea is to resolve ambiguity by iteratively strengthening the initial specification I with new examples. To achieve this, we start by using an existing synthesizer to find a candidate program p 1 that satisfies all examples in I. Instead of returning p 1 , we use it to find a distinguishing input x * that leads to ambiguities, i.e., other programs p i that satisfy I but produce different outputs p 1 (x * ) = p i (x * ). To resolve this ambiguity, we first generate a set of candidate outputs for x * , then use a neural model (which we train beforehand) that acts as an oracle and selects Selected Output Extend the Input Specification I ← I ∪ {(341dp (P4 Pro),

)}

Figure 1 : An overview of our approach which introduces a refinement loop around a black-box synthesizer that incrementally extends the input specification I = {(x 1 , y 1 )}, which in this case contains a single example, with additional input-output examples until all ambiguities are resolved.

the most likely output, and finally, add x * and its predicted output to the input specification I. The whole process is then repeated. These steps are similar to those used in Oracle Guided Inductive Synthesis (Jha et al., 2010) with two main differences: (i) we automate the entire process by learning the oracle from data instead of using a human oracle, and (ii) as we do not use a human oracle to produce a correct output, we need to ensure that the set of candidate outputs contains the correct one.

Augmenting an existing Android layout synthesizer In this work we apply our approach to a state-of-the-art synthesizer, called InferUI (Bielik et al., 2018) , that creates an Android layout program which represents the implementation of a user interface. Given an application design consisting of a set of views (e.g., buttons, images, text fields, etc.) and their location on the device screen, InferUI synthesizes a layout program that when rendered, places the views at that same location. Concretely, each input-output example (x, y) consists of a device screen x ∈ R 4 and a set of n views y ∈ R n×4 , all of which are represented using their coordinates in a two dimensional euclidean space. As an example, the input specification I shown in Figure 1 contains a single example with absolute view positions for a Nexus 4 device and the InferUI synthesizer easily finds multiple programs that satisfy it (dashed box). To apply our method and resolve the ambiguity, we find a distinguishing input x * , in this case a narrower P4 Pro device, on which some of the candidate programs produce different outputs. Then, instead of asking the user to manually produce the correct output, we generate additional candidate outputs (to ensure that the correct one is included) and our learned neural oracle automatically selects one of these outputs (the one it believes is correct) and adds it to the input specification I. In this case, the oracle selects output p 2 (x * ) as both buttons are correctly resized and the distance between them was reduced to match the smaller device width. In contrast, p 1 (x * ) contains overlapping buttons while in p n (x * ), only the left button was resized.

Automatically obtaining real-world datasets An important advantage of our approach is that we reduce the problem of selecting which program generalizes well to the simpler task of deciding which output is correct. This is especially useful for domains, such as the Android layout synthesis, for which the output correctness depends mostly on properties of the output and not on the program used to generate it. As a result, obtaining a suitable training dataset can be easier as we do not require the hard to obtain ground-truth programs, for which currently no large real-world datasets exist (Shin et al., 2019b) . In fact, it is possible to train the oracle using unsupervised learning only, with a dataset consisting of correct input-output examples D U . For example, in layout synthesis an autoencoder can be trained over a large number of views and their positions extracted from running real-world applications. However, instead of training such an unsupervised model, in our work we use D U to automatically construct a supervised dataset D S by labelling the samples in D U as positive and generating a set of negative samples by adding suitable noise to the samples in D U . Finally, we also obtain the dataset D S+ that additionally includes the input specification I. In the domain of Android layouts, although it is more difficult, such a dataset can also be collected automatically by running the same application on devices with different screen sizes.

Our contributions:

We present a new approach to address the ambiguity in the existing Android layout program synthesizer InferUI by iteratively extending the user provided specification with new input-output examples. The key component of our method is a learned neural oracle used to generate new examples trained with datasets that do not require human annotations or ground-truth programs. To improve generalization, InferUI already contains a probabilistic model that scores programs q(p | I) as well as handcrafted robustness properties, achieving 35% generalization accuracy on a dataset of Google Play Store applications. In contrast, our method significantly improves the accuracy to 71% while using a dataset containing only correct and incorrect program outputs. We make our implementation and datasets available online at:

https://github.com/eth-sri/guiding-synthesizers

2. RELATED WORK:

In this section we discuss the work most closely related to ours.

Guiding program synthesis:

To improve scalability and generalization of program synthesizers several techniques have been proposed that guide the synthesizer towards good programs. The most widely used approach is to implement a statistical search procedure which explores candidate programs based on some type of learned probabilistic model -log-linear models (Menon et al., 2013; Long & Rinard, 2016) , hierarchical Bayesian prior (Liang et al., 2010) , probabilistic higher order grammar (Lee et al., 2018) or neural network (Balog et al., 2017; Sun et al., 2018) . Kalyan et al. (2018) also takes advantage of probabilistic models but instead of implementing a custom search procedure, they use the learned model to guide an existing symbolic search engine. In addition to approaches that search for a good program directly (conditioned on the input specification), a number of works guide the search by first selecting a high-level sketch of the program and then filling in the holes using symbolic (Ellis et al., 2018; Murali et al., 2017; Nye et al., 2019) , enumerative or neural search (Bosnjak et al., 2017; Gaunt et al., 2016) . A similar idea is also used by (Shin et al., 2018) , but instead of generating a program sketch the authors first infer execution traces (or condition on partial traces obtained as the program is being generated (Chen et al., 2019) ), which are then used to guide the synthesis of the actual program.

In comparison to prior work, a key aspect of our approach is to guide the synthesis by generating additional input-output examples that resolve the ambiguities in the input specification. Guiding the synthesizer in this way has several advantages -(i) it is interpretable and the user can inspect the generated examples, (ii) it can be used to extend any existing synthesizer by introducing a refinement loop around it, (iii) the learned model is independent of the actual synthesizer (and its domain specific language) and instead is focused only on learning the relation between likely and unlikely input-output examples, and (iv) often it is easier to obtain a dataset containing program outputs instead of a dataset consisting of the actual programs. Further, our approach is complementary to prior works as it treats the synthesizer as a black-box that can generate candidate programs. We also note that several prior works explore the design of sophisticated neural architectures that encode input-output examples (Sun et al., 2018; Devlin et al., 2017; Parisotto et al., 2017) and incorporating some of their ideas might lead to further improvements to our models presented in Section 4.

Learning to rank To choose among all programs that satisfy the input specification, existing program synthesizers select the syntactically shortest program (Liang et al., 2010; Polozov & Gulwani, 2015; Raychev et al., 2016) , the semantically closest program to a reference program (D 'Antoni et al., 2016) or a program based on a learned scoring function (Liang et al., 2010; Mandelin et al., 2005; Singh & Gulwani, 2015; Ellis & Gulwani, 2017; Singh, 2016) . Although the scoring function usually extracts features only from the synthesized program, some approaches also take advantage of additional information -Ellis & Gulwani (2017) trains a log-linear model using a set of handcrafted features defined over program traces and program outputs while Singh (2016) leverages unlabelled data by learning common substring expressions shared across the input data.

Similar to prior work, our work explores various representations over which the model is learned.

Because we applied our work to a domain where outputs can be represented as images (rather than strings or numbers), to achieve good performance we explore different types of models (i.e., convolutional neural networks). Further, we do not assume that the synthesizer can efficiently enumerate all programs that satisfy the input specification as in Ellis & Gulwani (2017) ; Singh (2016) . For such synthesizers, applying a ranking of the returned candidates will often fail since the correct program is simply not included in the set of synthesized programs. Therefore, the neural oracle is defined over program outputs instead of actual programs. This reduces the search space for the synthesizer as well as the complexity of the machine learning models.

Neural program induction Devlin et al. (2017) and Parisotto et al. (2017) , as well as related line of work on neural machines (Graves et al., 2016; Reed & de Freitas, 2016; Bošnjak et al., 2017; Chen et al., 2018) , explore the design of end-to-end neural approaches that generate the program output for a new input without the need for an explicit search. In this case the goal of the neural network is not to find the correct program explicitly, but rather to generate the most likely output for a given input based on the input specification. These approaches can be integrated in our work as one technique for generating a set of candidate outputs for a given distinguishing input instead of obtaining them using a symbolic synthesizer. However, the model requirements in our work are much weaker -it is enough if the correct output is among the top n most likely candidates rather than requiring 100% precision for all possible inputs as in program induction.

3. LEARNING TO GENERATE NEW INPUT-OUTPUT EXAMPLES:

Let I = {(x i , y i )} N i=1 denote the input specification consisting of user provided input-output examples. Further, assume we are given an existing synthesizer which can find a program p satisfying all examples in I, i.e., ∃p ∈ L, ∀(x i , y i ) ∈ I. p(x i ) = y i , where p(x i ) is the output obtained by running program p on input x i and L is a hypothesis space of valid programs. To reduce clutter, we use the notation p |= I to denote that p satisfies all examples in I. We extend the synthesizer such that a program p not only satisfies all examples in I but also generalizes to unseen examples as follows:

1. Generate a candidate program p 1 |= I that satisfies the input specification I.

2. Find a distinguishing input x * , a set of programs p 2 , . . . , p n that satisfy the input specification I but produce different outputs when evaluated on x * , and define candidate outputs as

y = {p 1 (x * ), p 2 (x * ), • • • , p n (x * )}. If no distinguishing input x * exists, return program p 1 .

3. Query an oracle to determine the correct output y * ∈ y for the input x * .

4. Extend the input specification with the distinguishing input and its corresponding output I ← I ∪ {(x * , y * )} and continue with the first step.

Finding a distinguishing input To find the distinguishing input x * we take advantage of existing symbolic synthesizers by asking the synthesizer to solve ∃x * ∈ X , p 2 ∈ L.

p 2 |= I ∧p 2 (x * ) = p 1 (x * ),

where X denotes a set of valid inputs. The result is both a distinguishing input x * as well as a program p 2 that produces a different output than p 1 . Programs p 1 and p 2 form the initial sequence of candidate outputs y = [p 1 (x * ); p 2 (x * )] which is extended until the oracle is confident enough that y contains the correct output (described later in this section).

To make our approach applicable to any existing synthesizer, including those that can not solve the above satisfiability query directly (e.g., statistical synthesizers), we note that the following sampling approach can also be applied: first, use the synthesizer to generate the top n most likely programs, then randomly sample a valid input x * not in the input specification, and finally check if that input leads to ambiguities by computing the output of all candidate programs.

Finding candidate outputs To extend y with additional candidate outputs once the distinguishing input x * is found, three techniques can be applied: (i) querying the synthesizer for another program with a different output: ∃p ∈ L. p |= I ∧ ∀ yi∈y p(x * ) = y i , (ii) sampling a program induction model P (y | x * , I) and using the synthesizer to check whether each sampled output is valid, or (iii) simply sampling more candidate programs, running them on x * and keeping the unique outputs. It is possible to use the second approach as we are only interested in the set of different outputs, rather than the actual programs. The advantage of (i) is that it is simple to implement for symbolic synthesizers and is guaranteed to find different outputs if they exist. In contrast, (ii) has the potential to be faster as it avoids calling the synthesizer and works for both statistical and symbolic synthesizers. Finally, (iii) is least effective, but it does not require pretraining and can be applied to any synthesizer.

Neural oracle:

The key component of our approach is a neural oracle which selects the correct program output from a set of candidate outputs y. Formally, the neural oracle is defined as arg max y * ∈y f θ ( | x * , y * , I), where f is a function with learnable parameters θ (in our case a neural network) that returns the probability of the output y * being correct given the input x * and the input specification I. We train the parameters θ using a supervised dataset

D S+ = {(, x i , y i , I i )} N i=1 ∪ {(, x j , y j , I j )} M j=1

which, for a given distinguishing input x * and input specification I, contains both the correct () as well as the incorrect () outputs. Because it might difficult to obtain such a dataset in practice, we also define a simpler model f θ ( | x * , y * ) that is trained using a supervised dataset D S = {(, x i , y i )} N i=1 ∪ {(, x j , y j )} M j=1 which does not include the input specification. In the extreme case, where the dataset contains only the correct inputoutput examples D U = {(x i , y i )} N i=1 , we define the oracle as f θ (y * | x * ). Even though the dataset does not contain any labels, we can still train f in an unsupervised manner. This can be achieved for example by splitting the output into smaller parts y i = y 1 i • • • y t i (such as splitting a word into characters) and training f as an unsupervised language model. Alternatively, we could also train an autoencoder that first compresses the output into a lower dimensional representation, with the loss corresponding to how well it can be reconstructed. To achieve good performance, the architecture used to represent f is tailored to the synthesis domain at hand, as discussed in the next section.

Dynamically controlling the number of generated candidate outputs Since generating a large number of candidate outputs is time consuming, we allow our models to dynamically control the number of sampled candidates for each distinguishing input x * . That is, instead of generating all candidate outputs y first and only then querying the neural oracle to select the most likely one, we query the oracle after each generated candidate output and let the model decide whether more candidates should be generated. Concretely, we define a threshold hyperparameter t ∈ R [0,1] which is used to return the first candidate output y * for which the probability of the output being correct is above this threshold. Then, only if there are no candidate outputs with probability higher that t, we return arg max of all the candidates. Note for t = 1 this formulation is equivalent to returning the arg max of all the candidates while for t = 0, it corresponds to always returning the first candidate, regardless of its probability of being correct. We show the effect of different threshold values as well as the number of generated candidate outputs in Section 5.

4. INSTANTIATION OF OUR APPROACH TO ANDROID LAYOUT SYNTHESIS:

In this section we describe how to apply our approach to the existing Android layout program synthesizer InferUI (Bielik et al., 2018) . Here, the input x ∈ R 4 defines the absolute positions of the top left and bottom right coordinates of a given device screen while the output y ∈ R n×4 consists of n views and their absolute positions.

Finding a distinguishing input and candidate outputs Because InferUI uses symbolic search, finding a distinguishing input and candidate outputs is encoded as a logical query solved by the synthesizer as described in Section 3. However, instead of synthesizing the layout program containing correct outputs of all the views at once (as done in InferUI), we run the synthesizer n times, each time predicting the correct output for only a single view (starting from the largest view) which is then added as an additional input-output example (we provide a concrete example in Appendix C.4). This is necessary since there are exponentially many combinations of the view positions when considering all the views at once and the InferUI synthesizer is not powerful enough to include the correct one in the set of candidate outputs (e.g., for samples with more than 10 views in less than 4%). The advantage of allowing the position of only a single view to change, while fixing the position of all prior views, is that the correct candidate output is much easier to include. The disadvantage is that the neural oracle only has access to partial information (consisting of the prior view positions) and therefore performs a sequence of greedy predictions rather than optimizing all view positions jointly.

Neural oracle Because the input x has the same dimensions as each view, we encode it as an additional view in all our network architectures. In Figure 2 we show three different neural architectures that implement the oracle function f , each of which uses a different way to encode the input-output example into a hidden representation followed by a fully-connected ReLU layer and a softmax that computes the probability that the output is correct. In the following, we describe the architecture of all the models. The formal feature definitions are included in Appendix A. In (CNN), the output is converted to an image (each view is drawn as a rectangle with a 1px black border on a white background) and used as an input to a convolutional neural network (CNN) with 3 convolutional layers with 5×5 filters of size 64, 32 and 16 and max pooling with kernel size 2 and stride 2. To support outputs computed for different screen sizes, the image dimensions are slightly larger than the largest device size. We regularize the network during training by positioning the outputs with a random offset such that they are still fully inside the image, instead of placing them in the center. This is possible as the image used as input to the CNN is larger than the device size.

• • • intersect       2 3 • • • 0       ReLU (x,

We provide visualization of this regularization in Appendix C.2.

In (MLP), the output is transformed to a normalized feature vector and then fed into a feedforward neural network with 3 hidden layers of size 512 with ReLU activations. To encode the properties of a candidate output y, we use high-level handcrafted features adapted from InferUI such as the number of view intersections or whether the views are aligned or centered. By instantiating the features for both horizontal and vertical orientation we obtain a vector of size 30, denoted as ϕ MLP (x * , y * ), which is used as input to the neural network. For the model f ( | x * , y * , I) the network input is a concatenation of output features (as before) ϕ MLP (x * , y * ), features of each sample in the input specification [ϕ MLP (x, y)] (x,y)∈I , and their difference [ϕ MLP (x * , y * )-ϕ MLP (x, y)] (x,y)∈I . This difference captures how the views have been resized or moved between the devices with different screen dimensions. This allows the model to distinguish between outputs that are all likely when considered in isolation but not when also compared to examples in I (as illustrated in Appendix C.3).

In (RNN), we use the fact that each output consists of a set of views y * = [v 1 ; . . . ; v n ] by using an encoder to first compute a hidden representation of each view. These are then combined with a LSTM to compute the representation of the whole output. To encode a view v i , we extract pairwise feature vectors with all other views (including the input) ϕ RNN (v i , x * , y * ) = [φ(v i , v j )] vj ∈{x * }∪y * \vi and combine them using a LSTM. Here, φ : R 4 × R 4 → R n is a function that extracts n real valued features from a pair of views. For each pair of views, we apply 11 simple transformations of the view coordinates, capturing their distance (4 vertical, 4 horizontal), the size difference (in width and height) and the ratio of the aspect ratios. For the model f ( | x * , y * ) we additionally use 17 highlevel features computed for each view that are adapted from InferUI. When using f ( | x * , y * , I), these additional high-level features are not required and instead we only use the 11 simple transformations combined in the same way as for the MLP model. That is, by concatenating

ϕ RNN (v i , x * , y * ), [ϕ RNN (v i , x, y)] (x,y)∈I and their difference [ϕ RNN (v i , x * , y * ) -ϕ RNN (v i , x, y)] (x,y)∈I .

Datasets To train our models we obtained three datasets D U , D S and D + S , each containing an increasing amount of information at the expense of being harder to collect.

The unsupervised D U = {(x i , y i )} N i=1 is the simplest dataset and contains only positive input-output samples obtained by sampling ≈ 22, 000 unique screenshots (including the associated metadata of all the absolute view positions) of Google Play Store applications taken from the Rico dataset (Deka et al., 2017) . Since the screenshots always consist of multiple layout programs combined together, we approximate the individual programs by sorting the views in decreasing order of their size and taking a prefix of random length (of up to 30 views). For all of the datasets, we deduplicate the views that have the same coordinates and filter out views with a negative width or height.

The supervised

D S = {(, x i , y i )} (xi,yi)∈D U ∪ ≤15 j=1 {(, x i , y i + ij )} (xi,yi)∈D U

contains both correct and incorrect input-output examples. In our work this dataset is produced synthetically from D U by extending it with incorrect samples. Concretely, the positive samples correspond to those in the dataset D U and for each positive sample we generate up to 15 negative samples by applying a transformation ij to the correct output. The transformations considered in our work are sampled from the common mistakes the synthesizer can make -resizing a view, shifting a view horizontally, shifting a view vertically or any combination of the above.

The supervised dataset is:

D S+ = {(, x i , y i , I i )} N i=1 ∪ {(, x j , y j , I j )} M j=1

, where each I i contains the same application rendered on multiple devices. We downloaded the same applications as used in the Rico dataset from the Google Play Store and executed them on three Android emulators with different device sizes. The number of valid samples is ≈ 600 since not all applications could be downloaded, executed or produced the same set of views (or screen content) when executed on three different devices. The negative examples are generated by running the synthesizer with the input specification I containing a single sample and selecting up to 16 outputs that are inconsistent with the ground-truth output for the other devices.

5. EVALUATION:

We evaluate our approach by applying it to an existing Android layout synthesizer called InferUI (Bielik et al., 2018) as described in Section 4. InferUI is a symbolic synthesizer which encodes the synthesis problem as a set of logical constraints that are solved using the state-of-the-art SMT solver Z3 (De Moura & Bjørner, 2008) . To improve generalization, InferUI already implements two techniques -a probabilistic model that selects the most likely program among those that satisfy the input specification, and a set of handcrafted robustness constraints φ(p) that prevent synthesizing layouts which violate good design practices. We show that even if we disable these two optimizations and instead guide the synthesizer purely by extending the input specification with additional input-output examples, we can still achieve an accuracy increase from 35% to 71%.

In all our experiments, we evaluate our models and InferUI on a test subset of the D S+ dataset which contains 85 Google Play Store applications, each of which contains the ground truth of the absolute view positions on three different screen dimensions -1400×2520, 1440×2560 and 1480×2600. We use one screen dimension as the input specification I, the second as the distinguishing input and the third one only to compute the generalization accuracy. The generalization accuracy of a synthesized program p |= I is defined as the percentage of views which the program p renders at the correct position.

InferUI Baseline To establish a baseline, we run InferUI in three modes as shown in Table 1 . The baseline mode returns the first program that satisfies the input specification, denoted as p |= I, and achieves only 15.5% generalization accuracy. In the second mode the synthesizer returns the most likely program according to a probabilistic model q(p | I) which leads to an improved accuracy of 24.7%. The third mode additionally defines a set of robustness properties φ(p) that the synthesized program needs to satisfy, which together with the probabilistic model achieve 35.2% accuracy. The generalization accuracy of all InferUI models is relatively low as we are using a challenging dataset where each sample contains on average 12 views. Note however, that this is expected since increasing the number of views leads to an exponentially larger hypothesis space. Further, to establish an upper bound on how effective a candidate ranking approach can be, we query the synthesizer for up to 100 different candidate programs (each producing a unique output) and check how often the correct program is included. While for small samples (with up to 4 views) the correct program is almost always included, for samples with 6 views it is among the synthesized candidates in only 30% of the cases and for samples with more than 10 views in less than 4%. Sampling more outputs will help only slightly as increasing the number of views would require generating exponentially more candidates.

Our Work We apply our approach to the InferUI synthesizer by iteratively generating additional input-output examples that strengthen the input specification. The specification initially contains absolute positions of all the views for one device and we extend the specification by adding one view at a time (rendered on a different device) as described in Section 4. In the experiments presented here we focus on evaluating the overall improvement of the InferUI synthesizer extended with our approach. We provide additional experiments that evaluate the effectiveness of the neural oracle as well as an ablation study in Appendix B.

Generalization Accuracy:

The results of our approach instantiated with various neural oracle models are shown in Table 2 . The best model trained on the dataset D S has almost the same accuracy as InferUI with all its optimizations enabled. This means that it is possible to replace existing optimizations and handcrafted robustness constraints by training on an easy to obtain dataset consisting of correct outputs and their perturbations. More importantly, when training on the harder to obtain dataset D S+ , the generalization accuracy more than doubles to 71% since the model can also condition on the input specification I. However, the results also show that the design of the model using the neural oracle is important for achieving good results. In particular, both MLP models achieve poor accuracy since the high level handcrafted features adapted from the InferUI synthesizer are not expressive enough to distinguish between correct and incorrect outputs. The CNN models achieve better accuracy but are limited for the opposite reason, they try to learn all the relevant features from the raw pixels which is challenging as many features require pixel level accuracy across large distances (e.g., whether two views in the opposite parts of the screen are aligned or centered). The RNN model performs the best, especially when also having access to the input specification. Even though it also processes low level information, such as distance or size difference between the views, it uses a more structured representation that first computes individual view representations that are combined to capture the whole output.

Number of Candidate Outputs:

We show the effect of different threshold values t used by the neural oracle to dynamically control whether to search for more candidate outputs as well as the maximum number of candidate outputs in Table 3 . We can see that using the threshold both slightly improves the accuracy (+0.3%) but more importantly, significantly reduces the average number of generated candidate outputs from 14.7 to 6.0 using the same number of maximum generated outputs |y| = 16.

Incorporating User Feedback Even though our approach significantly improves over the InferUI synthesizer, it does not achieve perfect generalization accuracy. This is because for many synthesizers the perfect generalization is usually not achievable -the correct program and its outputs depends on a user preference, which is only expressed as severely underspecified set of input-output examples. For example, for a given input specification there are often multiple good layout programs that do not violate any design guidelines and which one is chosen depends on a particular user. To achieve 100% in practice, we perform an experiment where the user can inspect the input-output examples generated by our approach and correct them if needed. Then, we simply count how many corrections were required. The applications in our dataset have on average 12 views and for our best model, no user corrections are required in 30% of the cases and in 27%, 15%, 12%, 5% of the cases the user needs to provide 1, 2, 3 or 4 corrections, respectively. In contrast, InferUI with all optimizations enabled requires on average twice as many user corrections and achieves perfect generalization (i.e., zero user corrections) in only 3.5% of the cases.

6. CONCLUSION:

In this work we present a new approach to improve the generalization accuracy of existing program synthesizers. The main components of our method are: (i) an existing program synthesizer, (ii) a refinement loop around that synthesizer, which uses a neural oracle to iteratively extend the input specification with new input-output examples, and (iii) a neural oracle trained using an easy to obtain dataset consisting of program outputs. To show the practical usefulness of our approach we apply it to an existing Android layout synthesizer called InferUI (Bielik et al., 2018) and improve its generalization accuracy by 2×, from 35% to 71%, when evaluated on a challenging dataset of real-world Google Play Store applications.

APPENDIX:

We provide three appendices. Appendix A contains an in-depth description of the feature transformations for the MLP and RNN models. Appendix B contains an additional ablation experiment as well as results from the oracle evaluation. Appendix C provides visualizations of positive and negative candidate outputs, the CNN regularization as well as outputs from the end-to-end synthesis.

A FEATURE DEFINITIONS:

In this section we describe in detail the feature transformations used in Section 4. As mentioned in Section 4 each view consists of the 4 coordinates: x l (x lef t ), y t (y top ), x r (x right ), y b (y bottom ). We use v.w for the width, v.h for the height and v.r for the aspect ratio (v.w/v.h) of view v.

A.1 MLP:

In Table 4 we define the 7 feature types that lead to a vector of size 30 used in the MLP model. All the features are normalized (divided) by the factor in the normalization column. The same dimension is defined as:

v 1 .w = v 2 .w ∧ v 1 .h = v 2 .h |V | 2 1 vertical + 1 horizontal

Number of view alignments:

Pairwise comparison if two views align (e.g. for vertical alignment: 

v 1 .x l = v 2 .x l ∨ v 1 .x l = v 2 .x r ∨v 1 .x r = v 2 .x l ∨ v 1 .x r = v 2 .x

A.2 RNN:

We formally define the 11 transformations used as the pairwise view feature vector φ : R 4 ×R 4 → R n in the RNN model. These features capture properties like the view's distance to the other view or the size difference. The feature vectors extracted for each pair of views are combined to a fixed length representation by passing them through the LSTM, in the decreasing order of view size. In Figure 3 all the properties are listed and visualized. 

B EXPERIMENTS:

In this appendix, we evaluate the oracle performance and include an ablation study of our models.

B.1 ORACLE EVALUATION:

We evaluate the oracle's performance (shown in Table 5 ) using two metrics: The pairwise accuracy and one vs. many accuracy. The pairwise accuracy is the binary ranking problem -for each pair consisting of a positive and negative candidate, it is considered to be correct, if the score of the positive example is higher than of the negative example (the pair was ranked correctly). To capture if the network can't distinguish between the samples, we also report a second value in square brackets in which we count as correct if the score of the positive example is higher or equal to the score of the negative example. In addition, we measure the one vs. many accuracy, in which the oracle is correct if the score of the correct candidate was higher than the scores of all incorrect candidates (since selecting the correct candidate out of many candidates represents what we are interested in during the end-to-end synthesis experiment).

Usually, models with a higher pairwise accuracy also have a higher one vs. many accuracy since more pairs are ranked correctly. The one vs. many accuracy might be lower than expected if the mistakes of the pairwise accuracy spread over a wide number of different samples (instead of having many mistakes in a small set of samples).

In general, the results of the one vs. many accuracy correlate with the results of the end-to-end synthesis. For example, RNN+CNN and RNN trained on D S+ achieve the highest scores in both of the experiments. One exception is the MLP model trained on D S+ dataset which performs worse on the end-to-end synthesis experiment even though the oracle performs better. A reason for this is that the MLP often makes mistakes when predicting the early views which then affect the positioning of the subsequent views. For the same reason, the accuracies of the oracle experiment are in general higher than the accuracy we observe in the synthesis experiment. Another interesting insight is that the MLP often scores the positive and the negative examples the same since there is a large difference in the pairwise accuracy depending on if we count the same score as correct or not. This shows that the MLP's feature functions are not expressive enough.

B.2 ABLATION STUDY:

To investigate which high-level properties are learned by our models, we generate an ablation dataset from the test dataset of D S+ . The results of the study are collected in Table 6 and Table 7 .

Recall from Section 4 that ϕ MLP (x, y) computes a vector of size 30 with the handcrafted functions which are described in detail in Appendix A.1. For each pair of positive (x * , y pos [ϕ MLP (x * , y neg ) -ϕ MLP (x, y)] (x,y)∈I . We add a pair to the ablation dataset for the violated property if d = 0, that is, if the property on the positive sample is different from the negative sample. For example, if the number of view intersections of the positive sample and input specification are the same, but different for the negative sample, we add this pair of positive and negative samples to the intersections-ablation dataset.

The MLP models perform very well on all the ablation properties. The reason is that MLP's features are the handcrafted features which are designed to capture the properties in the ablation dataset. The MLP's performance on the whole test dataset is worse, which indicates that some properties in the dataset are not captured by the handcrafted features.

The CNN fails at preserving the aspect ratios due to the pooling and the downsampling operations. Alignments and intersection properties can be learned with local filters detecting overlapping views. Off-screen views can be detected by checking the downsampled representation after the convolutional layers. On these three properties, the CNN performs best. The RNN performs better on the ablation dataset than the CNN since it processes the exact numeric values of the view coordinates.

The overall performance on the ablation dataset is better than the one on the whole test dataset for all models, indicating that there are some features not expressed in the handcrafted functions. Furthermore, the results of the ablation study should be considered carefully: the ablation study shows tendencies, but features could be correlated or have common hidden features.

Published as a conference paper at ICLR 2020 

C VISUALIZATION:

In this section, we provide visualizations of two concepts introduced in Section 4 -the rendered output candidates and how the CNN input is shifted for regularization. Further, we visualize the steps of the end-to-end synthesis and render outputs of synthesized programs. 

MODELS -INPUT/OUTPUT:



C.4 END-TO-END SYNTHESIS PROCEDURE:

In Figure 7 we present an example of the end-to-end synthesis described in Section 3 and Section 4. The input of the synthesizer is on the top left of the Figure 7 and consists of an input specification I with a single input-output pair (x, y). The synthesis loop starts by the synthesizer generating a candidate program p 1 |= I (not shown) and checking whether ambiguities exists. To do this, it finds a distinguishing input x * , in our example a smaller device x * = [0, 0, 1400, 2520] and a set of programs p 2 , . . . , p n that satisfy the input specification I but produce different outputs when evaluated on x * . Three possible candidate outputs p 1 (x * ), p 2 (x * ), p 3 (x * ) are visualized in Figure 7 (column Iteration 1). Note that as described in Section 4, the synthesis proceeds by synthesizing the views iteratively therefore, the first iteration contains only the first view v 1 . The oracle (in this example the CNN+RNN model trained on D S+ ) selects the second candidate (the oracle prediction is indicated by the dashed rectangle) since its score is the largest with 0.88. The position of v 1 on the device x * is added to the specification and the iterative synthesis proceeds by selecting the correct output for the second view. In the second iteration we perform the same steps, except now we synthesize different candidate outputs for the second view v 2 on the device x * , given that the position of the first view v 1 is fixed from the previous iteration. The same process is repeated until the absolute positions of all the views are predicted for the distinguishing input x * . The new input specification, shown at the bottom in Figure 7 , contains the absolute view positions for both the original input x as well as the distinguishing input x * . After that, the whole process repeats by querying for another distinguishing device and extending the input specification even further.

Note that in our evaluation we restrict the distinguishing input to be among those device sizes for which we trained a model. Since our datasets consist of three different screen sizes the choice is small and all the experiments in Section 5 are evaluated with a single distinguishing input x * . However, we could generate more than three dimensions for the training dataset and use them to train additional models. This was however not needed as three different devices are typically enough to resolve majority of ambiguities (as long as the oracle does not make mistakes).

C.4.1 LIMITATIONS OF THE MLP FEATURES:

Figure 8 visualizes eight candidate outputs in the first synthesis iteration for the same application as in Figure 7 . The MLP oracle assigns a score of 0 to all the output candidates in which view v 1 is off-screen. However, the handcrafted features are not expressive enough to distinguish between the candidate outputs in the bottom row which all have a score of 0.76. In general, the MLP performs bad in selecting the first views (earlier iterations) since there are only few views from which properties e.g. margins can be extracted. On the bottom right, the synthesis result is compared to the correct output. Since the first view was not selected correctly in the synthesis, all the subsequent inner views are also slightly off in comparison to the correct output.

C.5 EXAMPLES OF SYNTHESIS OUTPUTS:

Figure 9 visualizes the rendered output of the synthesizer for the distinguishing device x * . The correct output is on the left while the synthesis output of the different oracles is on the right. At first glance, the outputs of the MLP look visually appealing since the handcrafted features contain properties like intersections or off-screen and the model learns to avoid them. However, it is often not expressive enough, since the view sizes or positions are often different to the correct output. This happens in particular if there are only few views which is why the second view is often misplaced which leads to consecutive errors. This also explains the bad performance in the synthesis experiment in Table 1 in comparison to the oracle experiment in Table 5 . The CNN outputs look much worse and often contain views which are misaligned or not centered. The RNN performs quite well one the first and third application but fails on the second application. The RNN+CNN predicts the correct output for the first application but fails on five views (out of 22) in the second and on seven (out of 17) views in the third application. 

:



Figures and Tables:

Figure 2: Illustration of three different models used to implement the neural oracle -CNN, RNN and MLP. All models include a fully-connected ReLU layer and a softmax layer (not shown in the image) which computes the probability that the output y is correct. Note, that the features in the MLP model are shown unnormalized, that is, centered = 3 denotes that three views are centered.

) and negative examples (x * , y neg ) and their corresponding sample in the input specification (x, y) we compute the difference of their handcrafted feature functions d = [ϕ MLP (x * , y pos ) -ϕ MLP (x, y)] (x,y)∈I -Published as a conference paper at ICLR 2020

Figure4shows four different rendered output candidates. The correct candidate is on the left and the three incorrect ones are on the right. The candidates differ in their 6th view which is moved around and overlaps in all of the three incorrect candidates. The three negative examples are generated by the synthesizer as described in Section 4 (for dataset D S+ ) or applying perturbations sampled from the synthesizer mistakes (for dataset D S ).

Figure 4: One positive example (left) and 3 negative examples (right).

Figure 6: The input specification provides an important indication that candidate 2 is the correct one (unlike candidate 4).

Figure 9: Rendered outputs on device x * of the synthesizer for different oracles which were trained on D S+ .

Generalization accuracy of the existing InferUI synthesizer.

Table content: <table><tr><td>InferUI</td><td>p |= I</td><td>arg max p|=I q(p | I)</td><td>arg max p|=I∧φ(p) q(p | I)</td></tr><tr><td>(Bielik et al., 2018)</td><td>baseline</td><td>+ probabilistic model</td><td>+ probabilistic &amp; robustness model</td></tr><tr><td>Accuracy</td><td>15.5%</td><td>24.7%</td><td>35.2%</td></tr></table>

Generalization accuracy of different models used as neural oracle in our approach.

Table content: <table><tr><td>Accuracy</td></tr></table>

Effect of the threshold t and the maximum number of generated candidate outputs |y| on the accuracy and the average number of generated candidate outputs per view (shown in brackets).

Table content: <table><tr><td>t = 0.9</td><td>t = 1</td></tr></table>

Feature definitions for the MLP.

Table content: <table><tr><td>Size</td><td>Name</td><td colspan="2">Description</td><td>Normalization</td></tr><tr><td>1</td><td>Number of off-screen views</td><td>-</td><td/><td>|V |</td></tr><tr><td>3</td><td>Number of views with a spe-</td><td colspan="3">Instantiated for the ratios 1,</td><td>|V |</td></tr><tr><td/><td>cific aspect ratio</td><td colspan="3">[3/4, 4/3] and [9/16, 9/16]</td></tr><tr><td>1</td><td>Number of view intersections</td><td colspan="3">Pairwise comparison if two</td><td>|V | 2</td></tr><tr><td/><td/><td>views</td><td>intersect</td><td>(without</td></tr><tr><td/><td/><td colspan="3">counting fully contained</td></tr><tr><td/><td/><td>views.)</td><td/></tr><tr><td>1</td><td>Number of views which have</td><td/><td/></tr><tr><td/><td>the same dimension</td><td/><td/></tr></table>

Accuracy of different models (used as neural oracle) evaluated on the test dataset of D S+ . For the pairwise accuracy we count as correct if the score of the positive sample is smaller than the score of the negative sample. In square brackets, we report the model performance if we additionally consider equal scores of the positive and the negative sample as correct (in case it differs).

Table content: <table><tr><td>Pairwise Accuracy</td></tr></table>

Ablation study based on different subsets of the D S+ test dataset to check which general properties the model has learned. The f ( | x * , y * , I) models are trained on D S+ .

Table content: <table><tr><td>Pairwise Accuracy</td></tr></table>

Ablation study based on different subsets of the D S+ test dataset to check which general properties the model has learned. The f ( | x * , y * ) models are trained on D S .

Table content: <table><tr><td>Pairwise Accuracy</td></tr></table>

Peer review (iclr2020):
[Summary] This paper aims to improve the generalization of program synthesis, ensuring that the synthesized programs not only work on observed input/output (I/O) examples but also generalize well to assessment examples (i.e. model the real intent of the end-user). To this end, the paper proposes a framework that iteratively alternates between producing programs with an existing program synthesizer and augmenting examples to disambiguate possible programs with a neural oracle that learns to select correct outputs. Several architectures and the design of input of the neural oracle have been investigated. The experiments on Andriod layout program synthesis with an InferUI synthesizer show that the proposed framework can improve the generalization of synthesized programs. However, I find it is difficult to evaluate the effectiveness without sufficient qualitative results and the intermediate outputs (e.g. a distinguishing input and candidate outputs) of the proposed framework (see details below). Significance: are the results significant? 4/5 Novelty: are the problems or approaches novel? 4/5 Evaluation: are claims well-supported by theoretical analysis or experimental results? 3/5 Clarity: is the paper well-organized and clearly written? 4/5 [Strengths] *motivation* - The motivation for improving the generalization of program synthesis by augmenting examples is convincing. *novelty* - The idea of utilizing a neural network to select correct outputs to augment examples for disambiguating the possible programs is intuitive and convincing. This paper presents an effective way to implement this idea. *technical contribution* - The paper investigates a set of network architectures and ways to specify the network input for learning the neural oracle. The RNN+CNN model that leverages both rendered views and features seems effective. *clarity* - The overall writing is clear. The authors utilize figures well to illustrate the ideas. Figure 1 clearly shows the proposed framework. *experimental results* - The presentations of the results are clear. The results demonstrate that the proposed framework can improve generalization accuracy. *reproducibility* - Given the clear description in the main paper and the details provided in the appendix, I believe reproducing the results is possible if the dataset is available. [Weaknesses] *related work* The descriptions of the related work are not comprehensive. Some neural program synthesis works explore a variety of mechanisms to encode examples and fuse their features, which are not mentioned in the paper. [Devlin et al. in ICML 2017] investigates different attention mechanisms to sequentially encode a set of I/O examples and performs pooling to merge them. [Sun et al. in ICML 2018] proposes a doubly encoding method to capture more details of examples and merge the features using a relation network. I believe it would be interesting to see if these methods could further improve the performance of the neural oracle. *experiment setup* - The experiments are not sufficient. While the claims look promising, the proposed method is only evaluated in only one dataset, which is not sufficiently convincing. I suggest the authors to also experiment the FlashFillTest dataset where string transformation programs are synthesized. - A more comprehensive description of the dataset is lacking. *experiment results* - I find it hard to judge the effectiveness of the proposed framework without seeing sufficient qualitative results. I suggest the authors randomly sample some synthesized programs (both success and failure) and present them in the paper. - I believe it is important to present some examples of the given I/O pairs, initially synthesized programs (p_1), found distinguishing input (x*), candidate outputs (y), the prediction of the neural oracle (i.e. selected outputs), the augmented examples (I cup {(x*, y*)}), and finally the next synthesized program. Without this, it is very difficult to understand the performance of the proposed framework and what could go wrong. *ablation study: the neural oracle* Only the final performance (i.e. the program synthesis performance) is shown in the paper. I believe it would be helpful if the performance of the neural oracle was also presented. As the whole framework depends on how accurate the neural oracle can select the correct output, it is important to evaluate this. One way to show this is to simply show the performance of all the neural oracles (with different architectures) trained on D_S (the positive samples and the incorrect samples) or even D_{S+}. Devlin et al. *RobustFill: Neural Program Learning under Noisy I/O* in ICML 2017 Sun et al. *Neural Program Synthesis from Diverse Demonstration Videos* in ICML 2018