Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposed the concept of "property signatures" , which are learned to represent programs. The property signatures are essentially some key attributes that one may summarize from a given set of input-output pairs, which the target function has. Then a program can be generated by evaluating these property signatures vectors (which is simply a bag-of-word representation with only 0 or 1 as each element). Much discussions have been given to discuss why and how these properties may be useful and very little real experiments are conducted quantitatively compared with existing works. Although this paper is quite interesting, I think this paper is in its very early stage and there are a lot of serious concerns I have for using this approach to synthesize the real complex programs. 

1) First of all, the notion of property signatures are easy to understand and is very natural. Just like human beings, when we write a program, we first think about the possible attributes of this program may have given a set of input-output pairs for both correctness and completeness. However, this is also the hard part of this idea. Potentially it could have an exponential number of possible properties as the program goes more complex and complex. It will quickly become computationally intractable problem. 

2) When I read the middle of paper, I would eager to know how authors can effectively find a good set of properties of a target program from a given input-output pairs. However, when I eventually reached the Section 4, I was kindly disappointed since I did not see any effective and principle way to get them. All I saw are "randomly sample and generate". This may be Ok for a very simple program given a set of simple input-output pairs. But it is definitely not feasible for any complex function, not to mention project. I think this is the key for the proposed idea since how to construct a good set of property signatures is crucial to treat them as the inputs for any program synthesis task later. 

3) There are very little baselines to compare against even though authors listed "substantial prior work on program synthesis". I understand the existing works may have their limitation in both what they can do and how well they can do. But it is still important to compare with directly on the same set of benchmarks. Otherwise, it is hard to be convincing that this approach is indeed superior compared to existing ones.  
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: ** Summary
The paper studies the problem of program synthesis from examples and it proposes the notion of property signature as a set of "features" describing the program that can be used to simplify the synthesis by combining together sub-programs with similar features.

** Evaluation
The paper is outside my scope of research, so my review is an educated guess.

The use of properties to summarize some features about the program and the possibility to evaluate them directly from samples seems a very interesting idea. As the authors mentioned, this may help in creating useful features and useful architectures to simplify the learning task. The concept of property signatures is very well introduced and explained. The authors also provide an extensive comparison to related work. The empirical results seem promising in showing how property signatures make the synthesis much faster and better.

The downsides of the paper are:
- While it is clear how to build property signatures, it is quite unclear to me how they simplify the generation of programs that combine smaller/simpler programs.
- Sect 3.2 on how to learn useful properties is rather vague and it would need a much more detail explanation.
- Although the authors release code for the paper, the description of the experiments seem rather shallow and it requires a much higher level of detail on how the learning process is set up and executed.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: I would like to be able to recommend accepting this paper, but I can't.  It describes two contributions to the community that could be valuable:

1: searcho, a programming language designed for studying program synthesis via search over programs
2: A method of automatically choosing interesting features for guided program search. 

The paper does not give enough evidence to ascertain the value of either contribution.  There are now a large number of program synthesis works using ML, and a huge literature on program synthesis without ML.   While many of ML works use a DSL as the testbed, surely the authors' feature selection method can be applied in some of these DSLs, allowing comparisons with current art?   On the other hand, many of the ML methods don't require the DSL used in the works that introduce them.  Can searcho + your set of training and test programs distinguish between these methods, and establish a benchmark?  Are there some more realistic tasks/applications that searcho makes reasonable, not readily approached with previous DSLs used for testing these algorithms?   I think this paper could be valuable if it could demonstrate either in what ways the new feature selection algorithm is  an improvement over prior ML methods, or that searcho is valuable as a benchmark or for approaching interesting applications.


######################################################################################
edit after author response:  raising to 6, I think the deepcoder experiments are useful in contextualizing the contribution of the authors' feature selection method.   


review_title: Official Blind Review #3

Meta-review: The authors propose improved techniques for program synthesis by introducing the idea of property signatures. Property signatures help capture the specifications of the program and the authors show that using such property signatures they can synthesise programs more efficiently.

I think it is an interesting work. Unfortunately, one of the reviewers has strong reservations about the work. However, after reading the reviewer's comments and the author's rebuttal to these comments I am convinced that the initial reservations of R1 have been adequately addressed. Similarly, the authors have done a great job of addressing the concerns of the other reviewers and have significantly updated their paper (including more experiments to address some of the concerns). Unfortunately R1 did not participate in subsequent discussions and it is not clear whether he/she read the rebuttal. Given the efforts put in by the authors to address different concerns of all the reviewers and considering the positive ratings given by the other two reviewers I recommend that this paper be accepted. 

Authors,
Please include all the modifications done during the rebuttal period in your final version. Also move the comparison with DeepCoder to the main body of the paper.
Decision: Accept (Poster)