Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper proposes an approach to infer the attribute values of an input image representing a user interface. The model first infers the most likely initial attribute values, and iteratively refine them to improve the similarity between the input image and the interface generated from the newly inferred attributes. The model is trained on synthetic datasets generated by a black box rendering engine, and generalizes well to real-world datasets. To address the issues of pixel based metrics and mean squared error, the authors instead uses the probability that two images are equal in the attribute space to define the cost between these two images.

Although I'm not familiar with the problem addressed by the paper, I found the paper very clear and well written. Overall, the method is sensible and elegant, and could easily be applied to other domains. My only disappointment is maybe the fact that only the Android Button was considered, and it is not clear how the model would perform with other and more sophisticated Android components.

A few questions for the authors:
- How many steps do you perform in the refinement loop? This is an important information, but I couldn't find it in the paper. Typically, I was surprised to see in the first row of Table 2 that the model with a random attribute initialization can reach such a high performance. But I imagine that you need many more iterations to converge if you start from random attributes than from the best prediction initialization?
- Also, what is the stopping criterion? Do you decide to stop when none of the proposed attribute changes improve the pixel-level accuracy?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes an approach for reverse-engineering webpages using Siamese networks and imitation learning. While the idea of using synthetic data (which can be easily procedurally generated) to do this reverse-engineer training is very clever, prior work has exploited it also. Novel elements include the attribute refinement using imitation learning, and the authors show the effect of this step, but the improvement is small. Thus, the limited novelty and not very convincing results make the question the potential impact of this paper.

Some questions:
a) The authors mention they cannot use a GAN-style method because all generated images are by definition true/real; how about learning whether a *pair* is real or fake? (where the pair consists of the design specification and the rendered version). 
b) Are the baselines strong enough? None of them seem to be from recent prior work. How about a direct comparison to some of the work listed in the second para on page 2?
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: Authors proposed an algorithm to predict the attribute of GUI elements from rasterized design images. The problem is separated into two steps. The first step is to predict initial values of the attributes (border width, color, padding etc) from the image where the type of UI element and set of attributes are already known. Authors designed a typical convolutional DNN for each of the attributes. The second step is to learn a policy \pi to iteratively adjust one attribute a time until the final rendering matches input image pixel-perfect. 

Authors conducted the experiment with large synthetic data set of Android buttons, and evaluate the performance with a held-out synthetic set as well as a 110 hand crafted real world buttons set sampled from apps in Google Play App Store. Several variations of the same model were compared. The result showed that network structure, padding strategy (this is a bit unexpected), background type and color selection strategy all affect the accuracy significantly. 

Reviewer has concern about the scope and application value of the problem as a research paper. A number of key prior assumptions have to be made to let the algorithm work: the type of the UI element need to be known; the list of attributes and their value ranges need to be fixed beforehand and each of the attribute demands a DNN; the refinement iteration has the actual rendering in the loop which could be costly on current App development platforms. 

Feedback questions.

1) The abstract mentioned vector image as input but the main body only discussed rasterized images.

2) Since the rendering process could be costly, it's useful to discuss the speed of convergence in the attribute value adjustment iterations.

3) Reviewer is interested in the nature of the error (7.5% loss) but it's not discussed.

4) In related work, authors mentioned the REINFORCE algorithm by Williams et al 1992 is expensive. It could help the reader if a brief explanation of why it's expensive is provided.

5) In Section 3 Background, authors mentioned that the attributes are not independent of each other, which is a major challenge. Reviewer would like to see some discussion or experiment data on how this affects the process and how did the current algorithm address it.

6) It's a bit surprise that color clipping method has a big impact on accuracy. Some examples could have helped the reviewer understand it.

7) In Section 4.2 first paragraph, it seems that the user of the algorithm need to set the [-c, c] clipping values manually per feature. This sounds like quite some prior knowledge and hand-tuning.

8) In Section 4.2. Second paragraph, Reviewer can understand the nessesity of additive and subtractive operations, but why multiplication?

9) In the equation to the end of page 5, do we need an extra outer bracket for the denominator? By the way, the equations should be numbered for easier reference.

10) The task of predicting Android button attributes, while practical, seems over-simplified. Reviewer suggests at least experiment with a set of common UI elements to proof the horizontal performance.

11) In Section 5.1, Reviewer respect the experiment results but doesn't understand why solid color background provides the best variety but screenshots don't. May need more analysis and explanation.

12) In Table 1, the first line for variant (C) also looks pretty good, or even better than core on the Android app store dataset.

13) In Section 5.1, the effect of color clipping selection seems very specific to applications with a fixed color palette. While this is indeed the majority, this prior knowledge need to be speciifed clearly by saying it's tailored towards such applications (or use more examples to proof that's not the case).

14) In Table 2: Pixel Sim's performance on Best Prediction Initialization seems pretty good, and Reviewer believes this is the more practical scenario. Is a more complicated Siamese Network justified?


review_title: Official Blind Review #3

Meta-review: The majority of reviewers suggest rejection, pointing to concerns about design and novelty. Perhaps the most concerning part to me was the consistent lack of expertise in the applied area. This could be random bad luck draw of reviewers, but more likely the paper is not positioned well in the ICLR literature. This means that either it was submitted to the wrong venue, or that the exposition needs to be improved so that the paper is approachable by a larger part of the ICLR community. Since this is not currently true, I suggest that the authors work on a revision.
Decision: Reject