Title: DeepCoder: Learning to Write Programs
Authors: Matej Balog
Alexander L. Gaunt
Marc Brockschmidt
Sebastian Nowozin
Daniel Tarlow
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning
Applications
Structured prediction

Abstract: We develop a first line of attack for solving programming competition-style problems from input-output examples using deep learning. The approach is to train a neural network to predict properties of the program that generated the outputs from the inputs. We use the neural network's predictions to augment search techniques from the programming languages community, including enumerative search and an SMT-based solver. Empirically, we show that our approach leads to an order of magnitude speedup over the strong non-augmented baselines and a Recurrent Neural Network approach, and that we are able to solve problems of difficulty comparable to the simplest problems on programming competition websites.


Review discussion:
This paper presents an approach to learn to generate programs.
Instead of directly trying to generate the program, the authors propose to train a neural net to estimate a fix set of attributes, which then condition a search procedure. This is an interesting approach, which make sense, as building a generative model of programs is a very complex task.
Faster computation times are shown in the experimental section with respect to baselines including DFS, Enumeration, etc. in a setup with very small programs of length up to 5 instructions have to be found.
It is not clear to me how the proposed approach scales to larger programs, where perhaps many attributes will be on. Is there still an advantage?
The authors use as metric the time to find a single program, whose execution will result in the set of 5 input-output pairs given as input.
However, as mentioned in the paper, one is not after a generic program but after the best program, or a rank list of all programs (or top-k programs) that result in a correct execution.
Could the authors show experiments in this setting? would still be useful to have the proposed approach? what would the challenges be in this more realistic scenario?
In the second experiment the authors show results where the length of the program at training time is different than the length at test time.
However, the results are shown when only 20% of the programs are finished. Could you show results for finding all programs?
The paper is missing an analysis of the results. What type of programs are difficult? how often is the NNet wrong? how does this affect speed? what are the failure modes of the proposed method?
The authors proposed to have a fix-length representation of the each input-output pair, and then use average pooling to get the final representation.
However, why would average pooling make sense here? would it make more sense to combine the predictions at the decoder, not the encoder?
Learning from only 5 executions seems very difficult to me. For programs so small it might be ok, but going to more difficult and longer programs this setting does not seem reasonable.
In summary an interesting paper. This paper tackles a problem that is outside my area of expertise so I might have miss something important.
Thank you for your review!
In our general response posted on 29 December we have commented on your first and last questions regarding the length of considered programs (our programs are solving much more complex problems compared to prior work) and the use of 5 I/O examples only (we agree a stronger specification will be needed, and it can be naturally incorporated into DeepCoder).
We agree on the importance of finding a "best" program. To solve this problem, we would need to develop a ranking function or -- our favoured solution -- a data generation process that generates training programs according to an intelligent prior distribution over programs. After training the neural net on programs generated from this prior, there will be a natural bias towards searching over program space in best-first order. In this case, the method presented in the paper can be used un-modified.
Because of this, we felt it was reasonable to leave the ranking problem for future work.
In the second experiment we considered showing the speedups for solving a given proportion of test problems (e.g. the chosen 20%) more informative than showing the speedup for solving all problems. The reason for this is that the latter metric would only compare the running times of the most difficult problem for the baseline and for DeepCoder respectively, ignoring the speedups achieved on all other (perhaps more typical) problems.
In the 3rd revision we have added an analysis of failure modes in a new Appendix G. We have looked at a generalisation of confusion matrices to the multilabel classification setting: for each attribute in a ground truth program (rows) measure how often each other attribute (columns) is chosen as a false positive. We then re-ordered the confusion matrix to try to expose block structure in the false positives. This reveals groups of instructions that tend to be difficult to distinguish.
While the results are somewhat noisy, we observe a few general things:
- There is increased confusion amongst instructions that select out a single element from an array: HEAD, LAST, ACCESS, MINIMUM, MAXIMUM.
- FILTER and ZIPWITH are predicted more often regardless of the ground truth and most of the lambdas associated with them are also predicted often.
- There are some groups of lambdas that are more difficult for the network to distinguish within: + vs -; +1 vs -1; /2 vs /3 vs /4.
- When a program uses **2, the network often thinks it's using *, presumably because both can lead to large values in the output.
The NNet is never making hard decisions, so there isn't a clear concept of it being wrong. It can only rank the needed operations far down in its ordering, which causes the search to take longer.
However, in the Sort and add enumeration experiment reported in Section 5.1, it only happened in 3 instances that the neural network slowed down the search: the respective slowdowns were by factors of 3.1x, 1.5x and 1.3x.
Regarding your question on pooling information from individual I/O examples, we have chosen arithmetic averaging as it is one of the simplest operations that combines information from all the examples and is independent of their order. Other pooling operations could be tried, and whether they are better would be an empirical question. We thought that having each I/O example predict the output independently and then combine the predicted attributes would be inferior, because then there is no opportunity for the model to integrate different pieces of evidence from different I/O examples in a non-trivial way.
N/A
The paper presents a technique to combine deep learning style input-output training with search techniques to match the input of a program to the provided output. Orders of magnitude speedup over non-augmented baselines are presented.
Summary:
The proposed search for source code implementations based on a rather small domain specific language (DSL) is compelling but also expected to some degree
Quality: The paper is well written.
Clarity: Some of the derivations and intuitions could be explained in more detail but the main story is well described.
Originality: The suggested idea to speed up search based techniques using neural nets is perfectly plausible.
Significance: The experimental setup is restricted to smaller scales but the illustrated improvements are clearly apparent.
Details:
1. The employed test set of 100 programs seems rather small.
in addition the authors ensure that the test set programs are semantically disjoint from the training set programs. Could the authors provide additional details about the small size of the test set and how to the disjoint property is enforced?
2. The length of the programs is rather small at this point in time. A more detailed ablation regarding the runtime seems useful. The search based procedure is probably still the computationally most expensive part.
Hence the neural net provides some additional prior information rather than tackling the real task.
> 1. The employed test set of 100 programs seems rather small.
> in addition the authors ensure that the test set programs are semantically disjoint
> from the training set programs. Could the authors provide additional details about
> the small size of the test set and how to the disjoint property is enforced?
To enforce disjointness, we computed the outputs of all programs on a small set of example inputs, and only chose programs that showed output different from all other programs considered before.
Thus, every pair of programs has at least one input on which they differ. We did this as a preprocessing step for all programs, before randomly splitting off test and validation sets.
For the experiment, we subsampled our test set down to only 100 programs. We have now sampled a second set of 500 programs of length 3 and run experiments for the faster/more promising methods on this set, using the trained models we used in the paper (i.e., these results are comparable to Table. 1 from the paper, but on a larger set of examples). The results are as follows:
Back-end solver "DFS":
Time to solve 20%:
Baseline:   0.04078s
DeepCoder:  0.00268s
Speedup: 15.2x
Time to solve 40%:
Baseline:   0.12561s
DeepCoder:  0.03261s
Speedup: 3.9x
Time to solve 60%:
Baseline:   0.31413s
DeepCoder:  0.11019s
Speedup: 2.9x
Back-end solver "Enumeration":
Time to solve 20%:
Baseline:   0.08002s
DeepCoder:  0.00129s
Speedup: 62.2x
Time to solve 40%:
Baseline:   0.33539s
DeepCoder:  0.00615s
Speedup: 54.6x
Time to solve 60%:
Baseline:   0.86050s
DeepCoder:  0.02728s
Speedup: 31.5x
Back-end solver \lambda^2:
Time to solve 20%:
Baseline:   18.8710s
DeepCoder:  0.23483s
Speedup: 80.4x
Time to solve 40%:
Baseline:   49.6019s
DeepCoder:  0.52448s
Speedup: 94.6x
Time to solve 60%:
Baseline:   84.2487s
DeepCoder:  13.5468s
Speedup: 6.2x
Our plan is to complete these experiments for the remaining methods and longer programs before uploading a new revision.
This is a good paper, well written, that presents a simple but effective approach to predict code properties from input output pairs.
The experiments show superiority to the baseline, with speedup factors between one to two orders of magnitude. This is a solid gain!
The domain of programs is limited, so there is more work to do in trying such ideas on more difficult tasks. Using neural nets to augment the search is a good starting point and a right approach, instead of generating full complex code.
I see this paper as being above the threshold for acceptance.
Thank you to all the reviewers for their time and comments.
To summarize, the reviews say this is an interesting and well-written paper. The biggest issue raised in the reviews is that the scale of programs is said to be small (we disagree), and there are questions about how the approach will scale to more complicated programs (this is a big open problem, which--as discussed in Sec 7--we agree we don't solve, but we believe that DeepCoder lays a solid foundation upon which to scale up from.)
We would like to respond to these issues in a bit more detail:
1. DeepCoder is solving problems that are significantly more complex than those tackled in machine learning research (e.g., as can be found in [1-6]), and it can significantly speed up strong methods from the programming languages community. DeepCoder scales up better than any method we are aware of for solving simple programming competition style problems from I/O examples.
2. Inducing large programs from weak supervision (in this case input-output examples) is a major open problem in computer science. We don't claim to solve this problem, and it would be a major breakthrough if we did. There clearly need to be many significant research contributions to get there. DeepCoder develops two ideas that are generally useful: 1. learning bottom-up cues to guide search over program space, and 2. building a system that sits on top of and improves the strongest search-based techniques. The benefit is that as new search techniques are developed, the DeepCoder framework can be added on top, and it will likely provide significant additional improvements (we have now shown large improvements over 3 different search techniques: 2 from the original paper plus the \lambda^2 system requested in reviewer comments).
3. There is a question of whether there is enough information in input-output examples to induce large programs, and we agree that there is probably not enough information using strictly the formulation in this paper (see Discussion).
However, it is straightforward to extend DeepCoder: First, by using natural language as an additional input to the encoder, we can learn to extract more information about what the program is expected to do, and how it is expected to do it.
Second, generating data from a trained generative model of source code will cause the system to favor generating programs that are likely under the trained model, which will help with the ranking problem.
Third, the definition of attributes can be expanded to be richer (e.g., ranking the different instructions per position in the program, or combinations of instructions), to enable the neural network component to solve more of the problem (and thus rely less on the search). Together, we believe these directions represent a clear path towards scaling up program synthesis even further by leveraging the key ideas in DeepCoder.
[1] Graves, A., Wayne, G. and Danihelka, I., 2014. Neural turing machines. arXiv preprint arXiv:1410.5401.
[2] Zaremba, W., Mikolov, T., Joulin, A. and Fergus, R., 2015. Learning simple algorithms from examples. arXiv preprint arXiv:1511.07275.
[3] Andrychowicz, M. and Kurach, K., 2016. Learning Efficient Algorithms with Hierarchical Attentive Memory. arXiv preprint arXiv:1602.03218.
[4] Riedel, S., Bošnjak, M. and Rocktäschel, T., 2016. Programming with a Differentiable Forth Interpreter. arXiv preprint arXiv:1605.06640.
[5] Gong, Q, Tian, Y, Zitnick, C. L., Unsupervised Program Induction with Hierarchical Generative Convolutional Neural Networks. ICLR 2017 Submission.
[6] Kurach, K., Andrychowicz, M. and Sutskever, I., 2015. Neural random-access machines. arXiv preprint arXiv:1511.06392.
We have now uploaded two revisions of our submission, taking the discussion and reviewer feedback here into account. Most notably, we have extended the experimental evaluation to also consider the Lambda2 program synthesis system as back-end solver, and furthermore extended the considered test set of programs to synthesize. We also updated the text in several places to include the clarifications we provided in the comments here, and to cite more related work.
Finally, we are working towards another revision that will include a analysis of failure cases and hope to publish it before the end of the week.

Meta-review:
role: PC
data: comment: This is a well written paper that attempts to craft a practical program synthesis approach by training a neural net to predict code attributes and exploit these predicted attributes to efficiently search through DSL constructs (using methods developed in programming languages community). The method is sensible and appears to give consistent speedups over baselines, though its viability for longer programs remains to be seen. There is potential to improve the paper. One of the reviewers would have liked more analysis on what type of programs are difficult and how often the method fails, and how performance depends on training set size etc. The authors should improve the paper based on reviewer comments.

Decision: Accept (Poster)