Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a technique to combine deep learning style input-output training with search techniques to match the input of a program to the provided output. Orders of magnitude speedup over non-augmented baselines are presented.

Summary:
———
The proposed search for source code implementations based on a rather small domain specific language (DSL) is compelling but also expected to some degree

Quality: The paper is well written.
Clarity: Some of the derivations and intuitions could be explained in more detail but the main story is well described.
Originality: The suggested idea to speed up search based techniques using neural nets is perfectly plausible.
Significance: The experimental setup is restricted to smaller scales but the illustrated improvements are clearly apparent.

Details:
————
1. The employed test set of 100 programs seems rather small. in addition the authors ensure that the test set programs are semantically disjoint from the training set programs. Could the authors provide additional details about the small size of the test set and how to the disjoint property is enforced?

2. The length of the programs is rather small at this point in time. A more detailed ablation regarding the runtime seems useful. The search based procedure is probably still the computationally most expensive part. Hence the neural net provides some additional prior information rather than tackling the real task.
review_title: My thoughts

final_score: rating: null
review_content: N/A
review_title: N/A

final_score: rating: 7: Good paper, accept
review_content: This paper presents an approach to learn to generate programs. Instead of directly trying to generate the program, the authors propose to train a neural net to estimate a fix set of attributes, which then condition a search procedure. This is an interesting approach, which make sense, as building a generative model of programs is a very complex task.

Faster computation times are shown in the experimental section with respect to baselines including DFS, Enumeration, etc. in a setup with very small programs of length up to 5 instructions have to be found. 
It is not clear to me how the proposed approach scales to larger programs, where perhaps many attributes will be on. Is there still an advantage?

The authors use as metric the time to find a single program, whose execution will result in the set of 5 input-output pairs given as input. However, as mentioned in the paper, one is not after a generic program but after the best program, or a rank list of all programs (or top-k programs) that result in a correct execution.
Could the authors show experiments in this setting? would still be useful to have the proposed approach? what would the challenges be in this more realistic scenario?

In the second experiment the authors show results where the length of the program at training time is different than the length at test time. However, the results are shown when only 20% of the programs are finished. Could you show results for finding all programs? 

The paper is missing an analysis of the results. What type of programs are difficult? how often is the NNet wrong? how does this affect speed? what are the failure modes of the proposed method?

The authors proposed to have a fix-length representation of the each input-output pair, and then use average pooling to get the final representation. However, why would average pooling make sense here? would it make more sense to combine the predictions at the decoder, not the encoder?

Learning from only 5 executions seems very difficult to me. For programs so small it might be ok, but going to more difficult and longer programs this setting does not seem reasonable. 

In summary an interesting paper. This paper tackles a problem that is outside my area of expertise so I might have miss something important. 

review_title: An interesting approach. I have several questions that would like the authors to address

final_score: rating: 6: Marginally above acceptance threshold
review_content: This is a good paper, well written, that presents a simple but effective approach to predict code properties from input output pairs. 

The experiments show superiority to the baseline, with speedup factors between one to two orders of magnitude. This is a solid gain!

The domain of programs is limited, so there is more work to do in trying such ideas on more difficult tasks. Using neural nets to augment the search is a good starting point and a right approach, instead of generating full complex code.

I see this paper as being above the threshold for acceptance.


review_title: A relatively simple and effective idea for predicting programs from input output pairs

Meta-review: This is a well written paper that attempts to craft a practical program synthesis approach by training a neural net to predict code attributes and exploit these predicted attributes to efficiently search through DSL constructs (using methods developed in programming languages community). The method is sensible and appears to give consistent speedups over baselines, though its viability for longer programs remains to be seen. There is potential to improve the paper. One of the reviewers would have liked more analysis on what type of programs are difficult and how often the method fails, and how performance depends on training set size etc. The authors should improve the paper based on reviewer comments.
Decision: Accept (Poster)