Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of paper:

This paper tackles the problem of inferring graphics programs from hand-drawn images by splitting it into two separate tasks:
(1) inferring trace sets (functions to use in the program) and
(2) program synthesis, using the results from (1).
The usefulness of this split is referred to as the trace hypothesis.

(1) is done by training a neural network on data [input = rendered image; output = trace sets] which is generated synthetically. During test time, a trace set is generated using a population-based method which samples and assigns weights to the guesses made by the neural network based on a similarity metric. Generalization to hand-drawn images is ensured by by learning the similarity metric.

(2) is done by feeding the trace set into a program synthesis tool of Solar Lezama. Since this is too slow, the authors design a search policy which proposes a restriction on the program search space, making it faster. The final loss for (2) in equation 3 takes into consideration the time taken to synthesize images in a search space. 

---

Quality: The experiments are thorough and it seems to work. The potential limitation is generalization to non-synthetic data.
Clarity: The high level idea is clear however some of the details are not clear.
Originality: This work is one of the first that tackles the problem described.
Significance: There are many ad-hoc choices made in the paper, making it hard to extract an underlying insight that makes things work. Is it the trace hypothesis? Or is it just that trying enough things made this work?

---

Some questions/comments:
- Regarding the trace set inference, the loss function during training and the subsequent use of SMC during test time is pretty unconventional. The use of the likelihood P_{\theta}[T | I] as a proposal, as the paper also acknowledges, is also unconventional. One way to look at this which could make it less unconventional is to pose the training phase as learning the proposal distribution in an amortized way (instead of maximizing likelihood) as, for example, in [1, 2].
- In section 2.1., the paper talks about learning the surrogate likelihood function L_{learned} in order to work well for actual hand drawings. This presumably stems from the problem of mismatch between the distribution of the synthetic data used for training and the actual hand drawings. But then L_{learned} is also learned from synthetic data. What makes this translate to non-synthetic data? Does this translate to non-synthetic data?
- What does "Intersection over Union" in Figure 8 mean?
- The details for 3.1 are not clear. In particular, what does t(\sigma | T) in equation 3 refer to? Time to synthesize all images in \sigma? Why is the concept of Bias-optimality important?
- It seems from Table 4 that by design, the learned policy for the program search space already limits the search space to programs with maximum depth of the abstract syntax tree of 3. What is the usual depth of an AST when using Sketch?

---

Minor Comments:
- In page 4, section 2.1: "But pixel-wise distance fares poorly... match the model's renders." and "Pixel-wise distance metrics are sensitive... search space over traces." seem to be saying the same thing
- End of page 5: \citep Polozov & Gulwani (2015)
- Page 6: \citep Solar Lezama (2008)

---

References

[1] Paige, B., & Wood, F. (2016). Inference Networks for Sequential Monte Carlo in Graphical Models. In Proceedings of the 33rd International Conference on Machine Learning, JMLR W&CP 48: 3040-3049.
[2] Le, T. A., Baydin, A. G., & Wood, F. (2017). Inference Compilation and Universal Probabilistic Programming. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (Vol. 54, pp. 1338–1348). Fort Lauderdale, FL, USA: PMLR.
review_title: Nice experiments but a lot of ad-hoc choices

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a method to infer lines of code that produces a given image. The method consists of two components. One is to generate traces, which are primitive commands of a graphic program, given an image. The other is to infer lines of code given traces. The first component uses a deep neural network for the conversion and a novel architecture is used for the network. The second component uses a learnt search polity to speed up the inference. Experimental results on a small dataset show that the proposed method can generate lines of code of a graphics program for the images reasonably well. It also discusses possible applications of the method.

Overall, the paper is interesting and the proposed method seems reasonable. Also, it is well contrasted with related work. However, the paper contains too many contents and it is hard to understand the important details without reading supplement and the references. It might be even worth considering to split the paper into two ones and each paper proposes one idea (component) at a time with more details.

That said, I understood the basic ideas of the paper and I liked them. My concern is only around how to write.
review_title: Interesting algorithms and results, but too many contents in a single paper

final_score: rating: 4: Ok but not good enough - rejection
review_content: I think the idea of inferring programmatic descriptions of handwritten diagrams is really cool, and that the combination of SMC-based inference with constraint-based synthesis is nice. I also think the application is clearly useful – one could imagine that this type of technology would eventually become part of drawing / note-taking applications.

That said, based on the current state of the manuscript, I find it difficult to recommend acceptance. I understand that the ICLR does not strictly have a page limit, but I think submitting a manuscript of over 11 pages is taking things a bit too far. The manuscript would greatly benefit from a thorough editing pass and some judicious reconsideration of space allocated to figures. Moreover, despite its relative verbosity, or perhaps because of it, I found it surprisingly difficult to extract simple implementation details from the text (for example I had to dig up the size of the synthetic training corpus from the 44-page appendix). 

Presentation issues aside, I think this is great work. There is a lot here, and I am sympathetic to the challenges of explaining everything clearly in a single (short) paper. That said, I do think that the authors need to take another stab at this to get the manuscript to a point where it can be impactful. 

Minor Comments 

- I don't understand what the "hypothesis" is in the trace hypothesis. Breaking down the problem into an AIR-style sequential detection task and a program induction is certainly a reasonable thing to do. However, the word "hypothesis" is generally used to refer to a testable explanation of a phenomenon, which is not really applicable here. 

- How is the edit distance defined? In particular, are we treating the drawing commands as a set or a sequence when we calculate "the number of drawing commands by which two trace sets differ"?

- I took me a while to understand that the authors first consider the case of SMC for synthetic images with a pixel-based likelihood, and then move on to SMC with and edit-distance based surrogate likelihood for hand-drawn pictures. The text seems to suggest that only 100 of such hand drawn images were actually used, is that correct?
 
- What does the (+) operator do in Figure 3?

- I am not sure that "correcting errors made by the neural network" is the most accurate way to describe a reranking of the top-k samples returned by the SMC sweep.

- Table 3 is very nice, but does not need to be a full page. 

- I would recommend that the authors consolidate wrap-around figures into full-width figures. 

review_title: A great paper that needs an editing pass

Meta-review: The paper addresses an interesting problem, is novel and works.
While the paper improved through reviews + rebuttal, the reviewers still find the presentation lacking. 
Decision: Reject