Title: Neuro-Symbolic Program Synthesis
Authors: Emilio Parisotto
Abdel-rahman Mohamed
Rishabh Singh
Lihong Li
Dengyong Zhou
Pushmeet Kohli
Venue: ICLR 2017
Keywords: Deep learning
Structured prediction

Abstract: Recent years have seen the proposal of a number of neural architectures for the problem of Program Induction. Given a set of input-output examples, these architectures are able to learn mappings that generalize to new test inputs. While achieving impressive results, these approaches have a number of important limitations: (a) they are computationally expensive and hard to train, (b) a model has to be trained for each task (program) separately, and (c) it is hard to interpret or verify the correctness of the learnt mapping (as it is defined by a neural network). In this paper, we propose a novel technique, Neuro-Symbolic Program Synthesis, to overcome the above-mentioned problems. Once trained, our approach can automatically construct computer programs in a domain-specific language that are consistent with a set of input-output examples provided at test time. Our method is based on two novel neural modules. The first module, called the cross correlation I/O network, given a set of input-output examples, produces a continuous representation of the set of I/O examples. The second module, the Recursive-Reverse-Recursive Neural Network (R3NN), given the continuous representation of the examples, synthesizes a program by incrementally expanding partial programs. We demonstrate the effectiveness of our approach by applying it to the rich and complex domain of regular expression based string transformations. Experiments show that the R3NN model is not only able to construct programs from new input-output examples, but it is also able to construct new programs for tasks that it had never observed before during training.

TL;DR: A neural architecture for learning programs in a domain-specific language that are consistent with a given set of input-output examples

Review discussion:
This paper proposes a model that is able to infer a program from input/output example pairs, focusing on a restricted domain-specific language that captures a fairly wide variety of string transformations, similar to that used by Flash Fill in Excel.  The approach is to model successive “extensions” of a program tree conditioned on some embedding of the input/output pairs.  Extension probabilities are computed as a function of leaf and production rule embeddings — one of the main contributions is the so-called “Recursive-Reverse-Recursive Neural Net” which computes a globally aware embedding of a leaf by doing something that looks like belief propagation on a tree (but training this operation in an end-to-end differentiable way).
There are many strong points about this paper.
In contrast with some of the related work in the deep learning community, I can imagine this being used in an actual application in the near future.  The R3NN idea is a good one and the authors motivate it quite well.
Moreover, the authors have explored many variants of this model to understand what works well and what does not.
Finally, the exposition is clear (even if it is a long paper), which made this paper a pleasure to read.  Some weaknesses of this paper: the results are still not super accurate, perhaps because the model has only been trained on small programs but is being asked to infer programs that should be much longer.  And it’s unclear why the authors did not simply train on longer programs…  It also seems that the number of I/O pairs is fixed?
So if I had more I/O pairs, the model might not be able to use those additional pairs (and based on the experiments, more pairs can hurt…).
Overall however, I would certainly like to see this paper accepted at ICLR.
Other miscellaneous comments:
* Too many e’s in the expansion probability expression — might be better just to write “Softmax”.
* There is a comment about adding a bidirectional LSTM to process the global leaf representations before calculating scores, but no details are given on how this is done (as far as I can see).
* The authors claim that using hyperbolic tangent activation functions is important — I’d be interested in some more discussion on this and why something like ReLU would not be good.
* It’s unclear to me how batching was done in this setting since each program has a different tree topology.  More discussion on this would be appreciated.  Related to this, it would be good to add details on optimization algorithm (SGD?  Adagrad?  Adam?), learning rate schedules and how weights were initialized.  At the moment, the results are not particularly reproducible.
* In Figure 6 (unsolved benchmarks), it would be great to add the program sizes for these harder examples (i.e., did the approach fail because these benchmarks require long programs?  Or was it some other reason?)
* There is a missing related work by Piech et al (Learning Program Embeddings…) where the authors trained a recursive neural network (that matched abstract syntax trees for programs submitted to an online course) to predict program output (but did not synthesize programs).
Thanks for your comments and feedback.
Q. ..it’s unclear why the authors did not simply train on longer programs…
Restriction to smaller program sizes was mainly due to the following two computational considerations. As mentioned in the batching question, since each program has a different tree structure, to compute batch gradients we needed to sequentially accumulate gradients over N program trees and then perform the learning update with the accumulated gradients. The computational cost of this operation increases with the size of the program trees.
A further issue is that valid input/output strings for programs often grow with the program length, in the sense that for programs of length 40 a minimal valid input/output string will typically be much longer than a minimal valid input/output string for length 20 programs.
For example, if the program was something like (Concat (ConstStr "longstring") (Concat (ConstStr "longstring") (Concat (ConstStr "longstring") ...))), the valid output string would be "longstringlongstringlongstring..." which could be many hundreds of characters long. Because of limited GPU memory, the I/O encoder models can quickly run out of memory.
Despite these two stated issues, we think that these are mainly engineering challenges and not definite weaknesses of our approach.
Therefore for future work we hope to write a more efficient implementation that can handle programs up to length 40-60.
Q. It also seems that the number of I/O pairs is fixed?
So if I had more I/O pairs, the model might not be able to use those additional pairs (and based on the experiments, more pairs can hurt…).
Yes, our current I/O encoder assumes fixed-size I/O pair set.
That said, if we had access to more I/O pairs we can sample multiple sets of cardinality 5. Each of these sets could then be used to generate a candidate program from our neural architecture, and the final program can be chosen among these candidates by evaluating consistency over the full I/O set. The design of an I/O encoder that can handle I/O sets of variable cardinality is a good direction for future work.
Q. * There is a comment about adding a bidirectional LSTM to process the global leaf representations before calculating scores, but no details are given on how this is done (as far as I can see).
Given the global leaf representations, we ordered them sequentially from left-most leaf node to right-mode leaf node. We then treated each leaf node as a time step for a BLSTM to process. This provided a sort of skip connection between leaf nodes, which potentially reduces the path length information needs to travel between leaf nodes in the tree. The hidden states of the BLSTM are then used in place of the global tree representations during the score calculations. We will add this description to the paper in a new revision.
Q. The authors claim that using hyperbolic tangent activation functions is important — I’d be interested in some more discussion on this and why something like ReLU would not be good.
The results of our extensive experimentation showed that the ReLU activations could sometimes reach the same level as the tanh activations but it was more variable in performance (some ReLU-based seeds did not converge to a good accuracy) and also were far more unstable (some ReLU-based seeds diverged later in training after reaching a certain performance). We will add these details to the paper.
Q. It’s unclear to me how batching was done in this setting since each program has a different tree topology. More discussion on this would be appreciated. Related to this, it would be good to add details on optimization algorithm (SGD? Adagrad? Adam?), learning rate schedules and how weights were initialized. At the moment, the results are not particularly reproducible.
Because of the difficulty of batching different tree topologies, batching was done sequentially, with each batch sample processed one at a time.
Therefore for each minibatch of size N, we accumulated the gradients for each sample.
After all N sample gradients were accumulated, we updated the parameters and reset the accumulated gradients. Due to this sequential processing, in order to get results in a reasonable time, we limited our batch sizes to between 8-12.
Despite the computational inefficiency, batching was critical to sucessfully train an R3NN as online learning often caused the network to diverge. We think a parallelized batching implementation could potentially allow more reasonable batch sizes of 32-100, and allow the R3NN to train faster and potentially reach better results.
Therefore for future work we aim to write a parallelized implementation.  As you suggest, we will add more details on how the R3NN was trained in the revision of the paper.
Q. In Figure 6 (unsolved benchmarks), it would be great to add the program sizes for these harder examples (i.e., did the approach fail because these benchmarks require long programs? Or was it some other reason?)
Yes, the system couldn't learn programs for the benchmarks in Figure 6 because of the longer size of the desired programs.
The task in Figure 6(a) requires 6 Concat arguments, whereas the task in Figure 6(b) requires 5 Concat arguments.
Q. There is a missing related work by Piech et al (Learning Program Embeddings…) where the authors trained a recursive neural network (that matched abstract syntax trees for programs submitted to an online course) to predict program output (but did not synthesize programs).
Thanks for the citation. Piech et al. use NPM-RNN model to embed program ASTs, where a subtree of the AST rooted at a node n is represented by a matrix obtained by combining representations of the children of node n and the embedding matrix of the node n itself (corresponding to its functional behavior). The forward pass in our R3NN architecture from leaf nodes to the root node is, at a high-level, similar, but we use a distributed representation for each grammar symbol that leads to a different root representation.
Moreover, R3NN also performs a reverse-recursive pass to ensure all nodes in the tree encode global information about other nodes in the tree.
Finally, the R3NN network is then used to incrementally build a tree to learn a program. We will add this discussion to the paper and cite the work of Piech et al. appropriately.
This paper sets out to tackle the program synthesis problem: given a set of input/output pairs discover the program that generated them. The authors propose a bipartite model, with one component that is a generative model of tree-structured programs and the other component an input/output pair encoder for conditioning. They consider applying many variants of this basic model to a FlashFill DSL. The experiments explore a practical dataset and achieve fine numbers. The range of models considered, carefulness of the exposition, and basic experimental setup make this a valuable paper for an important area of research. I have a few questions, which I think would strengthen the paper, but think it's worth accepting as is.
Questions/Comments:
- The dataset is a good choice, because it is simple and easy to understand. What is the effect of the "rule based strategy" for computing well formed input strings?
- Clarify what "backtracking search" is? I assume it is the same as trying to generate the latent function?
- In general describing the accuracy as you increase the sample size could be summarize simply by reporting the log-probability of the latent function. Perhaps it's worth reporting that? Not sure if I missed something.
Thanks for your comments and suggestions.
Question: What is the effect of the "rule based strategy" for computing well formed input strings?
Response: Given a program P (sampled from the DSL), the rule-based strategy generates input strings for the program P ensuring that the pre-conditions of P are met (i.e. P doesn't throw an exception on the input strings). It collects the pre-conditions of all Substring expressions present in the sampled program P and then generates inputs conforming to them.
For example, let's assume the sampled program is (SubStr(v,(CAPS,2,Start), (" ",3,Start))), which extracts the substring between the start of 2nd capital letter and start of 3rd whitespace. The rule-based strategy would ensure that all the generated input strings consist of at least 2 capital letters and 3 whitespaces in addition to other randomly generated characters.
Question: Clarify what "backtracking search" is? I assume it is the same as trying to generate the latent function?
Response: During program generation, we use the R3NN to generate a tree incrementally in the DSL that is conditioned using the I/O encoder. The tree generative model takes a partial program tree (PPT), and predicts: i) which non-terminal node to expand in the tree, and ii) which production rule to use in the grammar. The model starts with the start symbol e of the grammar, and builds the tree until getting the program tree (PT), where all leaves of the tree are terminal symbols. At test time, we either i) use production rules with the maximum score (1-best case) or ii) sample production rules from the distribution induced by the model over possible expansions. The backtracking search corresponds to the second case of sampling production rules from the learnt distribution. In hindsight, backtracking search is perhaps not the most suitable name for this approach. We will rename it to make the exposition clearer.
Question: In general describing the accuracy as you increase the sample size could be summarized simply by reporting the log-probability of the latent function. Perhaps it's worth reporting that? Not sure if I missed something.
Response: Given a set of input-output examples, there can be multiple possible programs (functions) in the DSL that are consistent because of: i) there are many equivalent programs in the DSL since it is quite rich, and ii) input-output examples are an under-specification. We can present the log-probability of the latent functions, but that might not correspond directly to the metric of learning a consistent program as there might be alternate programs in the DSL that are functionally equivalent with respect to the input-output examples.
The paper presents a method to synthesize string manipulation programs based on a set of input output pairs. The paper focuses on a restricted class of programs based on a simple context free grammar sufficient to solve string manipulation tasks from the FlashFill benchmark. A probabilistic generative model called Recursive-Reverse-Recursive Neural Network (R3NN) is presented that assigns a probability to each program's parse tree after a bottom-up and a top-down pass. Results are presented on a synthetic dataset and a Microsoft Excel benchmark called FlashFill.
The problem of program synthesis is important with a lot of recent interest from the deep learning community. The approach taken in the paper based on parse trees and recursive neural networks seems interesting and promising.
However, the model seems too complicated and unclear at several places (details below). On the negative side, the experiments are particularly weak, and the paper does not seem ready for publication based on its experimental results. I was positive about the paper until I realized that the method obtains an accuracy of 38% on FlashFill benchmark when presented with only 5 input-output examples but the performance degrades to 29% when 10 input-output examples are used. This was surprising to the authors too, and they came up with some hypothesis to explain this phenomenon. To me, this is a big problem indicating either a bug in the code or a severe shortcoming of the model. Any model useful for program synthesis needs to be applicable to many input-output examples because most complicated programs require many examples to disambiguate the details of the program.
Given the shortcoming of the experiments, I am not convinced that the paper is ready for publication.
Thus, I recommend weak reject. I encourage the authors to address the comments below and resubmit as the general idea seems promising.
More comments:
I am unclear about the model at several places:
- How is the probability distribution normalized? Given the nature of bottom-up top-down evaluation of the potentials, should one enumerate over different completions of a program and the compare their exponentiated potentials? If so, does this restrict the applicability of the model to long programs as the enumeration of the completions gets prohibitively slow?
- What if you only use 1 input-output pair for each program instead of 5? Do the results get better?
- Section 5.1.2 is not clear to me. Can you elaborate by potentially including some examples? Does your input-output representation pre-supposes a fixed number of input-output examples across tasks (e.g. 5 or 10 for all of the tasks)?
Regarding the experiments,
- Could you present some baseline results on FlashFill benchmark based on previous work?
- Is your method only applicable to short programs? (based on the choice of 13 for the number of instructions)
- Does a program considered correct when it is identical to a test program, or is it considered correct when it succeeds on a set of held-out input-output pairs?
- When using 100 or more program samples, do you report the accuracy of the best program out of 100 (i.e. recall) or do you first filter the programs based on training input-output pairs and then evaluate a program that is selected?
Your paper is well beyond the recommended limit of 8 pages. please consider making it shorter.
Thanks for the questions and feedback.
We would like to start by clarifying the experimental protocol that was used for the FlashFill benchmark evaluation, which has caused the reviewer to think that there was a bug in our model. The unfortunate confusion caused the reviewer to believe that our model performs worse with more I/O examples - this is incorrect. As expected, on programs uniformly generated from our DSL, the model with 10 I/O examples performed better than the model trained on 5 input/output examples. This was the reason we show our results on the synthetic dataset with 10 I/O examples (Table 1 and Table 3).
In order to check the real-world applicability of our models, we evaluated the trained models on an auxiliary task – i.e. on the FlashFill benchmarks. The FlashFill benchmark set consist of 238 tasks created by the Excel team, which only presents a subset of all possible programs representable in the DSL. These FlashFill benchmarks are never seen during training. Since the  FlashFill benchmarks contain only 5 I/O examples for each task, to run the model that took 10 I/O examples as input, we decided to duplicate the I/O examples.
Therefore, there was no additional information given to the model with 10 I/O examples compared to the 5 I/O examples.
So, we should not expect the model with 10 input/outputs to perform better than the one with 5 I/o outputs.
But why did it perform worse?  Our models are trained on the synthetic training dataset that is generated uniformly from the DSL. Because of the discrepancy between the training data distribution (uniform) and auxiliary task data distribution, the model with 10 input/output examples might not perform the best on the FlashFill task distribution, even though it performs better on the synthetic data distribution (on which it is trained).
We will clarify this in our writeup. We will also be happy to add an analysis where we train models that take x number of input/output examples where x = 1,2,...,10 and test the learnt program on the 10 input-output examples out of which only x are provided to the model.
Q: Compare with baseline results on FlashFill benchmark based on previous work?
The most comparable baselines to our's synthesis system are the current state-of-the-art general SyGuS (syntax-guided synthesis) solvers that are based on enumeration, stochastic solving, and SMT (Satisfiability Modulo Theories). These solvers take a DSL (Context-free grammar) and a specification (input-output examples) as input, and learn a derivation in the grammar that is consistent with a specification. These solvers do not scale to FlashFill benchmarks (even with a much-restricted set of operators), with the best solver performing worse than our baseline solver of DSL enumeration with increasing program size. This baseline solver achieves an accuracy of 1% with 10-sample and an accuracy of 12% with 300-samples.
Q: Is your method only applicable to short programs?
No, our framework is applicable to programs of any size. Restriction to smaller program sizes was mainly due to the following two computational considerations. Since each program has a different tree structure, to compute batch gradients we needed to sequentially accumulate gradients over N program trees and then perform the learning update with the accumulated gradients. The computational cost of this operation increases with the size of the program trees.
A further issue is that valid input/output strings for programs often grow with the program length, in the sense that for programs of length 40 a minimal valid input/output string will typically be much longer than a minimal valid input/output string for length 20 programs.
For example, if the program was something like (Concat (ConstStr "longstring") (Concat (ConstStr "longstring") (Concat (ConstStr "longstring") ...))), the valid output string would be "longstringlongstringlongstring..." which could be many hundreds of characters long. Because of limited GPU memory, the I/O encoder models can quickly run out of memory.
Despite these two stated issues, we think that these are mainly engineering challenges and not definite weaknesses of our approach.
Therefore, for future work we hope to write a more efficient implementation that can handle programs up to length 40-60.
Q: When is a program considered correct?
In our current setting, we consider a learnt program to be correct if it succeeds on the set of input-output pairs. Note that different programs might have the same functional behavior on input-output examples.
Q: When using 100 samples, how do you report accuracy?
We sample 100 programs using the learnt distribution, and return any program amongst them that is consistent with the I/O examples if one exists.
Q: What if you use only 1 input-output pair instead of 5?
To test our input-output encoders, we first started with testing encoders by encoding only 1 input-output pair. As soon as we trained the model with programs with ASTs of size greater than 7, this model resulted in considerably worse test accuracy compared to models with 5 and 10 input/output examples.
Q: Section 5.1.2 is not clear. Can you elaborate with an example? Does I/O representation assume fixed number of I/O?
We have added an illustration of the simple cross correlation encoder over a single input-output example. https://drive.google.com/file/d/0B1c9_fRwzM4yc1gwVVBTS3dGZ0U/view?usp=sharing
The encodings of each example pair is concatenated to obtain a representation of the set of examples. We will add this to the appendix of the paper.
Q: How is the probability distribution normalized? Given the nature of bottom-up top-down evaluation of the potentials, should one enumerate over different completions of a program and the compare their exponentiated potentials? If so, does this restrict the applicability of the model to long programs as the enumeration of the completions gets prohibitively slow?
Our current approach does not normalize the learnt program distribution over their size. Since we sample for grammar expansions to incrementally generate the program, the larger programs do get penalized more and are less likely to be enumerated. As part of future work, we are enriching the models to also predict likely program size, which we plan to use to normalize the program distribution.
Dear Reviewer,
We have uploaded a revised paper with an additional experiment to clarify the confusion regarding models trained with additional input-output examples, the FlashFill results, and added a figure for cross correlation encoder. Please find the experiment results and description in Section 6.5 (Figure 4). As expected, the train and test accuracies for the learnt models  increase with increasing number of input-output examples. The 1-best accuracies for different models (each trained for 74 epochs) are presented in the table below. Please also find the corresponding graph here: https://drive.google.com/file/d/0B1c9_fRwzM4yZFVUc2w3c0hSd1k/view?usp=sharing
I/Os    1     2     3     4     5     6     7     8     9     10
train   43   43   47   49   49   51   50   50   53   57
test    43   44   45   46   49   52   51   49   51   55
Please let us know if there are any other questions.
Thanks!

Meta-review:
role: AC
data: comment: Dear reviewers, do you have any reactions after the authors responded to your reviews?
role: PC
data: comment: There is a bit of a spread in the reviewer scores and unfortunately it wasn't possible to entice the reviewers to participate in a discussion. The area chair therefore discounts the late review of reviewer3, who seems to have had a misunderstanding that was successfully rebutted by the authors. The other reviewers are supportive of the paper.

Decision: Accept (Poster)