Learning to Infer User Interface Attributes from Images
Authors: Philippe Schlattner
Pavol Bielik
Martin Vechev
Venue:  (2020)

Abstract: We present a new approach that helps developers automate the process of user interface implementation. Concretely, given an input image created by a designer (e.g, using a vector graphics editor), we learn to infer its implementation which when rendered (e.g., on the Android platform), looks visually the same as the input image. To achieve this, we take a black box rendering engine and a set of attributes it supports (e.g., colors, border radius, shadow or text properties), use it to generate a suitable synthetic training dataset, and then train specialized neural models to predict each of the attribute values. To improve pixel-level accuracy, we also use imitation learning to train a neural policy that refines the predicted attribute values by learning to compute the similarity of the original and rendered images in their attribute space, rather than based on the difference of pixel values.


Under review as a conference paper at ICLR 2020
LEARNING TO INFER USER INTERFACE ATTRIBUTES FROM IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
We explore a new domain of learning to infer user interface attributes that helps developers automate the process of user interface implementation. Concretely, given an input image created by designer, we learn to infer its implementation which when rendered, looks visually the same as the ipput image. To achieve this, we take a black box rendering engine and set of attributes it supports (e.g., colors, border radius, shadow or text properties), use it to generate a suitable synthetic training dataset, and then train specialized neural models to predict each of the attribute values. To improve pixel-level accuracy, we also use imitation learning to train neural policy that refines the predicted attribute values by learning compute the similarity of the original and rendered images in their attribute space, rather than based the difference of pixel values. We instantiate our approach the task of inferring Android Button attribute values and achieve 92.5% 5% accuracy dataset consisting of real-world Google Play Store applications.
INTRODUCTION
With over 5 million applications in Google Play Store and Apple App Store and over a billion webpages, significant amount of time be saved by automating even small parts of their de- velopment. To achieve this, several tools have been recently developed that help user interface designers explore and quickly prototype different ideas, including Sketch2Code (Microsoft) 2018) and InkToCode (Corrado et al. 2018). which generate user interface sketches from hand drawn images, Swire (Huang et al., 2019) and Rico (Deka et al., (017). which allow retrieving designs similar to the one supplied by the user and Rewire (Swearngin et al.] 2018), which transforms im- ages into vector representations consisting of rectangles, circles and lines. the same time, to help developers implement the design, a number of approaches have been proposed that generate layout code that places the user interface components at the desired position (e.g.. when resizing the application). These include both symbolic synthesis approaches such as InferUI (Bielik et al. 2018). which encodes the problem as a satisfiability query of a first-order logic formula, as well as statistical approaches (Beltramelli 2018 Chen al. 2018). which use encoder-decoder neural networks to process the input image and output the corresponding implementation.
In this work, we explore new domain of inferring implementation of an user interface compo- nent from an image which when rendered, looks visually the same as the input image. Going from an image a concrete implementation a time consuming, yet necessary task, which is often out- sourced to a company for a high fee (replia 2019 [sd2android] 2019 psd2mobii (009).. Compared prior work, we focus on the pixel-accurate implementation, rather than on producing sketches or the complementary task of synthesizing layouts that place the components at the desired positions.
Coneretely, given a black box rendering engine that defines set of categorical and numerical at- tributes of component, we design a two step process which predicts the attribute values from an input image - (i) first, we train a neural model to predict the most likely initial attribute values, and then (fi) we use imitation learning to iteratively refine the attribute values to achieve pixel-level accu- racy. Crucially, all our models are trained using synthetic datasets that are obtained by sampling the black box rendering engine, which makes it easy to train models for other attributes in the future. We instantiate our approach to the task of inferring the implementation of Android Button attributes and show that it generalizes well to real-world dataset consisting of buttons found in existing Google Play Store applications. In particular, our approach successfully infers the correct attribute values in 94.8% and 92.5% of the cases for the synthetic and the world datasets, respectively.

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
As an application, our work is related to number of recently developed tools in the domain of user interface design and implementation with the goal of making developers more productive, as discussed in Section Here we give overview of the related research from a technical perspective.
Inverting rendering engines to interpret images The most closely related work to ours phrases the task of inferring attributes from images as the more general task of learning to invert the render- ing engines used to produce the images. For example, [Wu et al. (017)) use reinforcement learning to train a neural pipeline that given an image of a cartoon scene or Minecraft screenshot, identifies objects and small number of high level features (e.g., whether the object is oriented left right). Ganin et al. (2018) also use reinforcement learning. but with an adversarially learned reward signal, to generate a program executed by a graphies engine that draws simple CAD programs or handwrit- ten symbols and digits. Johnson et al. (2018) and Ellis et al. (2018) design a neural architecture that generates a program that when rendered, produces that same 2D 3D shape as in the input image. JhhisoJoooonnnooo et al. (2018) train the network using a combination of supervised pretraining and re- inforcement learning with a custom reward function (using Chamfer distance to measure similarity of two objects), Eilli et al. (2018) use two step process that first uses supervised learning to predict a set of objects in the image and then synthesizes a program (e.g., containing loops) that draws them.
In comparison to these prior works, our approach differs in three key aspects. First, the main chal- lenge in prior works is predicting the set of objects contained in the image and how to compose them. Instead, the focus of our work is in predicting set of object properties after the objects in the image were already identified. Second, instead of using the expensive REINFORCE (Williams, 1992) algorithm (or its variation) to train our models, we the network to make an initial prediction and then uses imitation Icarning to refine it. This is pos- sible because, in our setting, there is fixed set of attributes known in advance for which we can generate a suitable synthetic dataset used by both of these steps. Finally, because our goal is to learn pixel-accurate attribute values, the refinement loop takes as input both the original image, as well as the rendered image of the current attribute predictions. As a result, we do not require our models to predict aixel-accurate rendering of an attribute value but instead, only predict whether the attribute values in two images are the same or in which direction they should be adjusted.
two step process that first pretrains
Attribute prediction Optical character recognition (Jaderberg et al. 2016 Lyu et al. 2018 1 Jader-
berg et al. 2014: Gupta et al. 2016) is a well studied example of predicting an attribute from an image with a large number real-world applications. Other examples include predicting text fonts (Zhao et al. 2018 WWan et al. 2015: Chen et al.) (2014), predicting eye gaze (Shrivastaval jet al. 2017), face pose and lighting (Kulkarni et al. 2015). chair pose and content (Wu et al. 2018) or 3D object shapes and pose (Kundu et al. 2018). to name just a few. The attribute prediction network used our work to predict the initial attribute value is similar to these existing approaches, except that it is applied to new domain of inferring user interface attributes. As result, while some of the challenges remain the same (c.g.. how to effectively generate synthetic datasets), our main challenge is designing a pipeline, together with a network architecture capable of achieving pixel-level accuracies on a range of diverse attributes.
BACKGROUND: USER INTERFACE ATTRIBUTES
Visual design of user interface components can be specified in many different ways - by defining a program that draws on a canvas, by defining a program that instantiates components at runtime and manipulates their properties, declaratively by defining attribute values in a configuration file (e.g.. using CSS), or by using a bitmap image that is rendered in place of the components. In our work, we follow the best practices and consider the setting where the visual design is defined declaratively, thus allowing separating the design from the logic that controls the application behaviour.
Formally. let C denote component with a set of attributes A. The domain of possible values of each attribute a; E AA is denoted as O; As all the attributes are rendered on physical device, their domains are finite sets containing measurements in pixels or a set of categorical values. For example, the domain for the text color attribute three RGB channels N3x (0,256), the domain for text gravity is (top, left, center, right, bottom) and the domain for border width is N|0.20], We distinguish
2

---

Under review as a conference paper at ICLR 2020
two domain types: (i) comparable (e.g., colors, shadows sizes) for which a valid distance metric d: Ox0->No0 N10,00 exists, and (ūi) uncomparable (e.g., font types or text gravity) for which the distance between any two attribute values equal to one. We use θ C θ₂ x x...... x Gr to denote the space of all possible attribute configurations, and use the function render: θ + gxx to denote image with width w, height h and three color channels obtained by rendering the attribute configuration y E @. Furthermore, we the notation y y~Q to denote a random sample of attribute values from the space of all valid attribute configurations. Finally, we note that attributes often affect the same parts of the rendered image (e.g.. the shadow is overlayed on top of the background) and they are in general not independent of each other (e.g, changing the border width affects the border radius).
4 LEARNING TO INFER USER INTERFACE ATTRIBUTES FROM IMAGES
We now present our approach for learning to infer user interface component attributes from images.
Problem statement Given input image I E _3xxxx our goal is to find attribute configura- tion y E 2 which when rendered, produces an image most visually similar to I:
arg min yee cost render(y)
where cost I X I - (0,00) is function that computes the visual similarity of a given user interface component in two images. returns zero if the component looks visually the same in both images or positive real value denoting the degree to which the attributes are dissimilar.
The first challenge that arises from the problem statement above is how to define the cost function. Pixel based metrics, such as mean squared error of pixel differences, are not suitable and instead of producing images with similar attribute values, produce images that have on average similar colors. Training discriminator also does not work, as all the generated images are produced by rendering a set of attributes and are true images by definition. Finally, the cost can be computed not over the rendered image but by comparing the predicted attributes y with the ground-truth labels. Unfortunately, even if we would spend the effort and annotated a large number of images with their ground-truth attributes, using manually annotated dataset restricts the space of models that can be used to infer to only those that do supervised learning. In what follows we address this challenge by showing how to define the cost function over attributes (used for supervise learning) as well as over images (used for reinforcement learning), both by using synthetically generated dataset.
Our approach To address the task of inferring user interface attributes from images, we propose a two step process that - (i) first selects the most likely initial attribute values arg max p(y I) by learning probability distribution of attribute values conditioned on the input image, and then (ii) iteratively refines the attribute values by learning a policy π (Ay(() [I, render the probability distribution of how each attribute should be changed, conditioned on both the original image, well as the rendered image of the attribute configuration y iteration i. We use the policy π to define the cost between two images as cost I') == 1- (Ay=0 I,I'). That is, the cost is defined as the probability that the two images are not equal in the attribute space.
that represents
We illustrate both steps in Figure with an example that predicts attributes of a Button component. set of convolutional neural networks, each of which
In Figure (a), the input image is passed is trained to predict a single attribute value. In our example, the most likely value predicted for the border width is 2dp while the most likely color of the border is #4a4a4a. Then, instead of returning the most likely attribute configuration y. we take advantage of the fact that it can be rendered and compared to the original input image. This give us additional information that is used to refine the predictions as shown in Figure (b). Here, we use pair of siamese networks (pre- trained on the prediction task) to learn the probability distribution over changes required to make the component attributes in both images the same. In our example, the network predicts that the border color and the text gravity attributes have already the correct values but the border width should be decreased by 2dp and the shadow should be increased by 4dp. Then, due to the large number of different attributes that affect each other, instead of applying all the changes at once, we select and apply a single attribute change. In our example, the Ay corresponds to adjusting the value of border width by - 2dp. Since the change is supposed to correct a mispredicted attribute value, we accept it only if it indeed makes the model more confident that the prediction is correct.
3

---

Under review as a conference paper at ICLR 2020
(a) Attribute Prediction (Section 4.1
Input Image
y I)
Predicted Attributes
render
y
p
I)
border width
2dp
0.77
SIGN
CNN
border color
0.99
SIGN IN
shadow
Odp
0.91
Training
text gravity center
0.96
D (render(s (
(b) Attribute Refinement Loop (Section 4.2
Input Image
(Da II, render(())))) Predicted Changes
Select & Apply Change
a
Ay~ (Ay *0 render (y ()) y = y Ay
Ay (9)
CNN
border width
2dp
0.67
Siamese Networks
border color
0
0.99
shadow
+ 4dp
0.61
text gravity
0.98
Accept or Reject Change
SIGN IN
CNN
(++1)
min (y (0,))/))) cost render (y))
render (0)
Training
DA = ((render
re A(y - )
Figure (a) Illustration of the attribute prediction network which takes an input an image with component (a Button) and predicts all the component attribute values. (b) Refinement loop which renders the attribute values obtained from (a) and iteratively refines them to match the input image.
Synthetic datasets We instantiate our approach by training it purely using synthetic datasets, while ensuring that it generalizes well real-world images. This allows us to avoid the expen- sive task of collecting and annotating real-world datasets and more importantly, makes our approach easily applicable new domains and attributes. In particular, given space of possible attribute configurations @ and a rendering function render, we generate two different datasets D and DA used to train the attribute prediction network and the policy æ,, respectively. The dataset D (renderr( (gyler)))))))))) (?), (f) @) is constructed by sampling valid attribute configuration (+)) O and rendering to produce the input image. To generate Da, we sample two attribute configurations (0) ( ~ O that used to render two input images and train the network to predict the difference of their attributes, that is, DA ((render(g)
θ) render θ)) A(y -y())) ) ii==1
For both datasets, to avoid overfitting when training models for attributes with large domain of possible values, we sample only subset of attributes, while setting the remaining attributes to values from the previous example ( As result, every consecutive samples are similar each other, since a subset of their attributes is the same. Further, because the real-world images do not contain components in isolation but together with other components that fill the rest of the screen, we introduce three additional attributes Tos E N, 3/pos E N and background. We use Tpos and Ypoe to denote the horizontal and vertical position of the component in the image, respectively. This allows the network to learn robust predictions regardless of the component position in the image. We use background to select the background on which the component is rendered. We experimented with three different choices of backgrounds only while color, random solid color and overlaying the component on top of an existing application, all of which are evaluated in Section 5
4.1 ATTRIBUTE PREDICTION
The attribute prediction network architecture is multilayer convolutional neural network (CNN) followed by a set of fully connected layers. The multilayer convolutional part consists of 6 repeti- tive sequences of convolutional layers with ReLU activations, followed by batch normalization and a max- max-maxoooo pooling layer of size 2 and stride 2. For the convolutional layers we use 3x3 filters of size 32, 32, 64, 64, 128 and 128, respectively. The result of the convolutional part is then flattened and
4

---

Under review as a conference paper at ICLR 2020
connected to fully connected layer of size 256 with ReLU activation followed by final softmax layer (or a single neuron for regression). We note that this is not a fixed architecture and instead, is adapted to given attribute by performing an architecture search, as shown in Section [
Supporting multiple input sizes To support user interface components of different sizes, we select the input image dimension such that it is large enough contain them. This is necessary as our goal is to infer pixel-accurate attribute values and scaling down or resizing the image to fixed dimension not an option, as it leads to severe performance degradation. However, this is problematie, as most of the input images are smaller or cropped in order to remove other components. As a result, before feeding the image to the network we need to increase the dimension of the input without resizing or scaling. To achieve this, we pad the missing pixels with the values of edge pixels, which improves the generalization to real-world images as shown Section 5.1 and illustrated in Appendix C
To improve the accuracy and reduce the variance of the attribute prediction network, we perform the following two optimizations. First, we compute the most likely attribute value by combining multiple predictions and selecting the most likely among them. This is achieved by generating several perturbations of the input image, each of which shifts the image randomly, horizontally by f ~ () and vertically by Ey U(-1,1), (), ensemble models but instead of training multiple models we generate and evaluate multiple inputs.
Optimizations
where E N. This is similar to
Second, to improve the accuracy of the color attributes, we perform color clipping by picking the closest color to one of those present in the input image. To reduce the set of all possible colors, use saliency maps (Simonyan et al. 2013) to select subset of the pixels most relevant for the prediction. In our experiments we keep only the pixels with the normalized saliency value above 0.8. Then, we clip the predicted color the closest color from the top five most common colors among the pixels selected by the saliency map. We provide illustration of the color clipping in Appendix D
4.2 ATTRIBUTE REFINEMENT LOOP
We now describe how to learn a function x <(Ay [I, render(y)) that represents probability distri- bution of how each attribute should be changed, conditioned on both the original image as well as the rendered image of the current attribute configuration y. We can think of π as a policy, where the actions correspond to changing an attribute value and the state is tuple of the original and the currently rendered image. We can then apply imitation learning to train the policy on a synthetic dataset D = {(render((y(() )), render
Because the range of possible values (z - yy())) can be large and sparse, we limit the range by clipping it to an interval [-c,d c] (where e is hyperparameter set c= for all the attributes in our evaluation). To fix a change larger than c, we perform sequence of small changes. For comparable attributes, the
))) Aly( - - -
delta between two attribute values is defined their distance Aly (t) yk = d( d(y))) yk ) For uncomparable attributes, the delta is binary and determines whether the value is correct or not.
The model architecture used to represent consists of two convolutional neural networks with shared weights 6, also called siamese neural networks, each of which computes a latent represen- tation of the input image h fo [I) and h, = fo (render(y)). The function fo has the same architecture as the attribute prediction network, except that we replace the fully connected layer with one that has a bigger size and remove the last softmax layer. Then, we combine the latent features h and h, into a single vector h = izz: h2: h + hr: hr h,; h2 @ hyl, where C denotes element-wiie wwie multiplication. Finally, the vector h is passed to fully connected layer of size 256 with ReLU activations, followed by a final softmax layer. Once the models are trained, we perform the refinement loop as follows:
Select attribute to change. As in general attributes interfere with each other, in each refinement iter- ation we adjust only a single attribute, which is chosen by sampling from the following distribution:
1- Ay, = 0 I, render(y))
P[A a:]
Σ m Ayy 0 I. render(y))
where T(Ay: 0 I. re)der(y)) denotes the probability that the k-th attribute should not be changed, that is, the predicted change is zero. Since we train a separate model for each attribute, the probability that the given attribute should be changed is n (AAy = 0 I, render(y)).
5

---

Under review as a conference paper at ICLR 2020
Select attribute's new value. For comparable attributes, we adjust their value by sampling from the probability distribution computed by the policy म. which contains changes in range |-c, cl. For uncomparable attributes another approach has to be chosen, since the delta prediction network computes only whether the attribute is correct and not how to change it. Instead, we select the new value by sampling from the probability distribution computed by the corresponding attribute prediction network.
Accept or reject the change. In typical setting, we would accept the proposed changes as long as the model predicts that an attribute should be changed. However, in our domain we can render the proposed changes and check whether the result is consistent with the model. Concretely, we accept the change y' if it reduces the cost, that is, cost vender(y')) < cost(I, render (y)). Note that this optimization possible only if the change was supposed to fix the attribute value, that is, the change was in the range (-c, c) or the attribute is uncomparable.
5 EVALUATION
To evaluate the effectiveness of our approach, we apply it to the task of generating Android Button implementations. Concretely, we predict the following 12 attributes border color, border width, border radius, height, width, padding, shadow, main color, text color, text font type, text gravity and text size. We do not predict the text content for which specialized models already exist (Jaderberg et al. 2016 Lyu et al. 2018 Jaderberg et al. 2014: Gupta et al. 2016). We provide domains and the visualization of all these attributes in Appendix A In what follows we first describe our datasets and evaluation metrics, then we present detailed evaluation of our approach consisting of the attribute prediction network (Section 4.T) and the refinement loop (Section 4.2).
Datasets To train the attribute prediction network we use synthetic dataset D containing 220,000 images and their corresponding attributes as described in Section 4 To train the refinement loop second synthetic dataset DA also containing 220,000 image pairs. During training perform two iterations of DAgger (Ross et al., 2011). each of which generates =20,0000 samples obtained by running the policy the initial training dataset. To evaluate our models datasets - (i) synthetic Daym generated in the same way as for training, and (äi) real-world Daplay we obtained by manually implementing 1 110 buttons in existing Google Play Store applications. The il- lustration of samples and our inferred implementations for both datasets are provided in Appendix
two
Evaluation metries To remove clutter, we introduce an uniform unit to measure attribute similar- ity called perceivable difference. We say that two attributes have the same (=) perceivable difference if their values are the same or almost indistinguishable. For example, the text size is perceivably the same, if the distance of the predicted y and the ground-truth y" value is d(y, d(y,y") v²) S. 1, while the
border width is perceivably the same only if it is predicted perfectly, i.c., d(y. = The formal definition of perceivable difference with visualizations of all attributes is provided in Appendix B
5.1 ATTRIBUTE PREDICTION
A detailed summary of the variations of our attribute prediction models and their effect on perfor- mance is shown in Table [ To enable easy comparison, we selected good performing instantiation of our models, denoted as core, against which all variations in the rows (A)-(E) can be directly compared. Based on our experiments, we then select the best configuration that achieves accuracy 93.6% and 91.4% on the synthetic and the real-world datasets, respectively. All models were trained for 500 epochs, with early-stopping of epochs, using a batch size of 128 and initial learning rate of 0.01. In what follows we provide a short discussion of each of the variations from Table [
Image background (A) We trained our models on synthetic datasets with three different component backgrounds of increasing complexity white color, random solid color and user interface screen- shot. Unsurprisingly, the models trained with white background fail to generalize to real-world datasets and achieve only 56.7% accuracy. Perhaps surprisingly, although the models trained with the screenshot background improve significantly, they also fail to generalize and achieve only 75.6% accuracy. Upon closer inspection, this is because overlaying components over existing images often introduces small visual artefacts around the component. On the other hand, random color back- grounds generalize well to real-world dataset they have enough variety and no visual artefacts.

---

Under review as a conference paper at ICLR 2020
Table 1: Variations of the attribute prediction network and their effect on the model accuracy.
Network Architecture
Dataset & Input Preprocessing
Accuracy
arch
Iryd
color
inputsise
tr
background
D
Doploy
core
Co,Ro
saliencytops
x30
tra
rand
92.7%
90.1%
white
88.9%
56.7%
(4)
screenshot
88 88.6%
75.6%
cender
89.4%
82.4%
(B)
tri 92.3%
90.3%
135 310
92.5%
91.2%
(C)
180 x 350
92.5%
88.7%
imagerop5
89.2%
85.9%
(D)
imageall
87.1%
83.0%
none
74.1%
72.0%
Cs
83.2%
83.4%
Cs
88.6%
88.2%
(E)
Rh
62.6%
64.3%
Ra
67.7%
65.3%
best
Cay R2
0.1
saliencytopó
x35x310
tr
rand
93.6 %
91.4%
arch. model architecture
fr input transformation
Irrd reduced learning plateau
Affine transformations (B) Since the components can appear in any part of the input image, we use three methods generate the training datasets tr places the component randomly at any position with a margin of at least 20 pixels of the image border, fr2 places the component in the middle of the image with a horizontal offset 6, ~ 13, 13) and vertical offset c. (- 19), and center always places the component exactly in the center. We can see that using either tr or tra leads to significantly more robust model and increases the real- world accuracy by 8%.
Input image size & padding (C) As our goal is to perform rixel-accurate predictions, we do not scale down resize the input images. However, since large images include additional noise (e.g.. parts of the application unrelated to the predicted component), we measure how robust our model is to such noise by training models for three different input sizes - 135 x310, 150 x 330 and 180 350, While the accuracy on the synthetic dataset is not affected, the real-world accuracy shows a slight decrease for larger sizes that contain more noise. However, note that the decrease is so small because of our padding technique, which extends the component to larger size by padding the missing pixels with edge pixel values. When using no padding, the accuracy of the real-world dataset drops to 72% and when padding with white color the accuracy drops even further to 71% (not shown in Table I
Color clipping (D) We experimented with different color clipping techniques - saliencyrops that considers the top colors in the saliency map of given attribute, imagetops that considers the top colors in the image, and image,n that considers all the colors in the image. The color clipping using saliency maps performs the best and leads to more than 3% and 16% improvements over other types of clipping or using clipping, respectively. While the other types of color clipping also perform reasonably well, they typically fail for images that include many colors, where the saliency map helps focusing only the colors relevant to the prediction. We note that the color clipping works best for components with solid colors and the improvement for gradient colors limited.
Network architecture (E) We adapt the architecture presented in Section 4.1 for each attribute, by performing a small scale architecture search. Concretely, we choose between using classification (C) or regression (R), the kernel sizes, the number of output channels, whether use pooling layer and whether we use additional fully connected layer before the softmax layer. Although the results in Table I are not directly comparable, they provide only the aggregate accuracy over all attributes (additionally for regression experiments we consider only numerical attributes), they do show that such architectural choices have a significant impact the network's performance.

---

Under review as a conference paper at ICLR 2020
Table Accuracy of the attribute refinement loop instantiated with different similarity metries. The accuracy shown in brackets denotes the improvement compared to the initial attribute values.
Random Attribute Initialization
Best Prediction Initialization
Metric
D
Daploy
D
Donlay
Our Work
Learned Dst.
94.4% (+57.3%)
91.3%
(+53.3%)
94.8% (+1.2%)
92.5% (+1.1%)
Baselines
Pixel Sim.
59.6%
(+22.5%)
65.0% (+27.0%)
93.6% ( 0.0%)
91.1% (-0.3%
Structural Sim.
81.1%
(+-41.0%)
71.9%
(+33.9%%
93.4% (-0.2%)
89.3% (-2.1%)
Wasserstein Dst.
63.4% (+26.3%)
61.8% (+23.8%)
91.8% (-18%)
89.6% (-18%%
5.2 ATTRIBUTE REFINEMENT
To evaluate our attribute refinement loop, we perform two experiments that refine the attribute val- ues: (i) starting from random initial values, and (ii) starting from values computed by the attribute prediction network. For both experiments, we show that the refinement loop improves the accuracy of the predicted attribute values, as well as significantly outperforms other similarity metries used as a baseline. We trained all our models for 500 epochs, with early-stopping of 15 epochs, using batch size of 64, learning rate of 0.01 and gradient clipping of 3, which is necessary to make the training stable. Further, we initialize the siamese networks with the pretrained weights of the best attribute prediction network, which leads to both improved accuracy of 4%, as well as faster conver- gence when compared to training from scratch. Finally, we introduce hyperparameter that controls which attributes are refined. This is useful as it allows the refinement loop to improve the overall accuracy even if only subset of the attributes values can be refined successfully.
Attribute refinement improves accuracy The top row in Table (right) shows the accuracy of our refinement loop when applied starting from values predicted by the best attribute prediction network. Based on our hyperparameters, we refined the following six attributes - text size, text gravity, text font, shadow, width and height. The refined attributes are additionally set to random initial values for the experiment in Table [2 (left). The overall improvement for both synthetic and real-world dataset is s 1.1% when starting from the values predicted by the attribute prediction network. When starting from random values the refinement loop can still recover predictions of almost the same quality, although with 12x more refinement iterations. The reason why the improvement is not higher is mainly because = 5% of the errors are performed when predicting the text color and the text padding, for which both the attribute prediction networks and the refinement loop work poorly. This suggest that better network architecture is needed to improve the accuracy of these attributes.
Effectiveness of our learned similarity metric To evaluate the quality of our learned cost function, which computes image similarity in the attribute space, we use the following similarity metrics as a baseline pairwise pixel difference, structural similarity (Wang et al. 2004), and Wasserstein distance. As the baseline metrics depend heavily the fact that the input components are aligned in the two images (e.g., when computing pairwise pixel difference). for a fair comparison we add manual preprocessing step that centers the components in the input image. The results from Table show that all of these metrics are significantly worse compared to our learned cost function. Even though they provide some improvement when starting from random attributes, the improvement is limited and all of them result in accuracy decrease when used starting from good attributes.
6 CONCLUSION
We present an approach for learning to infer user interface attributes from images. We instanti- ate it the task of learning the implementation of the Android Button component and achieve 92.5% accuracy on dataset consisting of Google Play Store applications. We show that this can be achieved by training purely using suitable datasets generated synthetically. This result indicates that our method is a promising step towards automating the process of user interface implementation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Tony Beltramelli. Pix2code: Generating code from graphical user interface screenshot. In Pro- ceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS '18, pp. 3:1-3:6, New York, NY, USA, 2018. ACM ISBN 978- 978-1-4503--587-2. doi: 10.1145/ 3220134.322013: htth://///dde acm. org 1ttp://111111//11
3220115
Pavol Bielik, Mare Fischer, and Martin Vechev. Robust relational layout synthesis from examples for android. Proc. ACM Program. Lang,, 20t.:2000000011111111111-2220220 October 2018. ISSN 2475-
1421. doi: 10.1145/3276526. URL
Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. From ui design image to gui skeleton: A neural machine translator to bootstrap mobile gui implementation. Proceedings of the 40th International Conference Software Engineering, ICSE '18, pp. 665-676, New York, NY, USA, 2018. ACM. ISBN 17781-45003-6333 doi: 10. 10.1145511005553318822222 URL
http: :ttp://do3 acm. org /100 145 318011511551 1111880555.8888822 3180240
Guang Chen, Jianchao Yang, Hailin Jin, Jonathan Brandt, Eli Shechtman, Aseem Agarwala, and Tony X. Han. Large-scale visual font recognition. In Proceedings of the 2014 IEEE Conference on Computer Vision and Patter Recognition, CVPR 14, pp. 3598-3605, Washington, DC, USA, 2014. IEEE Computer Society. ISBN 978 978-1-4799-51188-3 doi: 10.1109/CVPR.2044.4444 URL
https://doi. org.:/0.....00... 1109///PR..... 2ttp///VVR.22...... 2014.460,
Alex Corrado, Avery Lamp, Brendan Walsh, Edward Aryee, Erica Yuen, George Matthews, Jen Madiedo, Jeremie Laval, Luis Torres, Maddy Leger, Paris Hsu, Patrick Chen, Tim Rait, Seth Chong, Wjdan Alharthi, and Xiao Tu. Ink to code, 2018. URL https: htta://wwwr /www.
microsoft. com en gawgeearaagaaaaaaa waarraagepprofiao pwo.froffffio werrofiles//ee www.eleeeeiiieoeeeeeeeeeee code/]
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design ap- plications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST 17, pp. 845-854, New York, NY. USA, 2017. ACM. ISBN 978- 978-1-4503-4981- URL http: acm. wwg./o.g///0..
9. doi: (0.1145/3126594.30
3126651
Kevin Ellis, Daniel Ritchie, Armando Solar- and Josh Tenenbaum. Learning to infer graph- ics programs from hand-drawn images. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 6059-6000 6068. Curran Associates, Inc., 2018. URL http: http://aapppraaae wtpperssnippii nips cc/ppperr prrasr-rgrammma hand-m-aand-daaaa images.
7845- 1055-leannnn lor-eeeannn---oonnoeee to--egttttttt---eeeeeeeeeeee pdf
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami, and Oriol Vinyals. Synthesizing programs for images using reinforced adversarial learning. CoRR, Abs//880.00118 2018. URL
http:://arsti..og//bbb////
Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Forrest Huang, John F. Canny, and Jeffrey Nichols. Swire: Sketch-based user interface retrieval. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pp. 104:1-104:10, New York, NY, USA. 2019, ACM. ISBN 178---403--5977-- 3290605.3300334. URL
doi: 10.1 145/ acm. wtt.://orro..0.............. http:/1111112322222 /tt.a/..rrrg..1113333//...00 3300334
M Jaderberg, K Simonyan, Vedaldi, and A Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. Neural Information Processing Systems, 2014.
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Reading text in the wild
with convolutional neural networks. Int. J. Comput. Vision, 116(1):1-20, January 2016. ISSN 0920-5691. doi: 10.1007/s1 1263-015-0822-32 011263-015-823--
URL http: /xxx dx. doi ftt..org/1..100/

---

Under review as a conference paper at ICLR 2020
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Tejas D Kulkarni, William F. Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 2539-2547. Curran Associates, Inc., 2015. URL http::
nips. ctpps.ccc/aper/// 3851-deeps convo coonvolutions inall--olll--a wwweoonalniiiveeee inverse graphics grephics
Abhijit Kundu, Yin Li, and James M. Rebg. 3d-renn: Instance-level 3d object reconstruction via -endrr-and-oompar. In CVPR, 2018.
Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai. Mask TextSpotter: An end- to-end trainable neural network for spotting text with arbitrary shapes. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision - ECCV 2018, pp. 71-88, Cham, 2018. Springer International Publishing. ISBN 97-3-0030-001,
Al Microsoft. Sketch 2 code, 2018. UtK.:httpee//bhett///eee....eeee
net/
psd2android, 2019. URL
psd2mobi, 2019.
andoooo
android ui
replia, 2019. URL http:://ww/wwwreee//
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 627-635, 2011.
János Schanda. Colorimetry: understanding the CIE system. John Wiley & Sons, 2007.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312 311.6044, 2013.
Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon, and Andrew J. Ko. Rewire: Interface design assistance from examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '1, pp. 504:1-504:12 New York, NY, USA, doi: 10. 10.1455317337443174073, URL http://doo.
2018. ACM. ISBN 978- 978-1-4503-5620- acm.org/10.cm 110
Zhangyang Wang, Jianchao Yang, Hailin Jin, Eli Shechtman, Aseem Agarwala, Jonathan Brandt, and Thomas S. Huang. Deepfont: Identify your font from an image. In Proceedings of the 23rd ACM International Conference on Multimedia, MM '15, pp. 451-459, New York, NY, USA, 2015. ACM. ISBN 978-1-4503-334544 doi: 0..114///733333.206622 acm.
URL http://doii
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE TRANSACTIONS ON IMAGE PROCESSING, 1304):600-612, 2004.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 233) May 1992 ISSN 1573-0565. doi: 10.1007/ BF00992696. htttts::///oii.... wttp://///11.100////0002222000
Jiajun Wu, Joshua Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2017.
Jiajun WU, Tianfan Xue, Joseph Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, and William T Freeman. 3d interpreter networks for viewer-centered wireframe modeling. Inter- national Journal of Computer Vision (IJC)), 12099):009-11066, 2018.
Nanxuan Zhao, Ying Cao, and Rynson W.H. Lau. Modeling fonts in context: Font prediction on web designs. Computer Graphics Forum (Proc. Pucific Graphics 2018), 37, 2018.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
We provide six appendices. In Appendix we define domains of all the attributes considered in our work and include their visualizations. In Appendix B we include details of the stopping criterion used the refinement loop and provide accuracy breakdown the individual attribute values. In Appendix [ we illustrate three different techniques pad images larger size. In Appendix [ we show an example of using color clipping with saliency maps. In Appendix [ formally define the perceivable different metric used compute accuracy in evaluation. Finally, in Appendix we provide examples of images and the inferred implementations for samples in both the synthetic and real-world datasets.
A ANDROID BUTTON ATTRIBUTES
We provide definition of all the attributes considered in work well as their visualization in For border radius we use a special value oo to denote round buttons.
Table 3
Table 3: Button attribute domains and illustration of their visual appearance.
Attribute
Domain
Example
Border Color
Nax [0,25]
gray
purple
Border Radius
N 0.00
Uo
dp
15 dp
Border Width
N0.12]
dp
dp
Main Color
Nax [0,255
blue
purple
Padding
N
right
Shadow
N
dp
Text Color
[0255]
purple
Text Font Family
{
(thin, light, regular, medium um, bolt]
light
bold
Text Gravity
(top, left, center, center, right, bottom)
left
right
Text Size
N 10,300 UO
14pt
Height
N 20,60
30dp
50dp
Width
N 25,275
70dp
110dp
B
REFINEMENT LOOP
Stopping criterion For a given similarity metric (e.g., our learned attribute distance, pixel simi- larity, etc.) the stopping criterion of the refinement loop is defined using two hyperparameters:
early stopping: stop if the similarity metric has not improved in the last n iterations. In our experiments we use n
maximum number of iterations: stop if the maximum number of iterations reached. In our experiments we set maximum number of iterations to 8 when starting from the best prediction and 100 if starting from random attribute values.

---

Under review as a conference paper at ICLR 2020
Table Per attribute accuracy before and after applying the refinement loop when starting from the best predictions of the attribute prediction network for the real-world Daplay dataset.
Accuracy Donlay
Attribute
Before Refinement
After Refinement
Refined Attributes
Text Size
96.2%
99.1 9
Text Gravity
94.4%
96.3%
Text Font Family
84.0%
85.8%
Shadow
93.7%
97.3%
Width
99.1%
99.1% 99.1%
Height
93.6%
97.3%
Non-Refined Attributes
Main Color
99.1%
99.1 1
Text Color
69.2%
69.2%
Border Color
92.6%
92.6%
Padding
81.1%
81.1%
Border Width
94.5%
94.5%
Border Radius
99.1%
99.1%
All Attributes
91.4%
92.5%
Per-attribute accuracy We provide per-attribute accuracy of the refinement loop in Table 4 As
described in Section 5.2 we refined the following six attributes text size, text gravity, text font, shadow, width and height. When considering only these six attributes, the refinement loop improves the accuracy by 2.3%, from 93.5% to 95.8%.
The remaining errors are mainly due to text color, padding and text font attributes. The padding achieves accuracy 81 % and is difficult to learn as it often interferes with the text alignment and the same position of the text can be achieved with different attribute values. The text font attribute accu- racy is 85% which is slightly improved by using the refinement loop but could be improved further. The worst attribute overall is the text color that achieves 69% accuracy. Upon closer inspection, the predictions of the text color challenging due the fact that the text typically narrow and when rendered, it does not have a solid background. Instead, the color is an interpolation of the text color and background color computed by the anti-aliasing in the rendering engine.
Attribute dependencies All the results presented in Table computed starting from the initial of attribute values that typically contains mistakes. Since different attributes can (and do) dependent on each other, mispredicting one attribute can negatively affect the predictions of other attributes. As concrete example, refining text font while assuming that all the other attribute values are correct leads to an improvement of 12% which is significantly higher than 2% from Table 4
well as in our evaluation, are
To partially address this issue, the refinement loop have different values when used as input to the refinement loop. Concretely, two attributes would have the same value, of the values is temporarily changed random valid value and returned to the original value at the end of each refinement iteration.
heuristic which ensures that all attributes
C IMAGE PADDING
To improve robustness of our models on real-world images we experimented with three techniques of image padding shown in Figure 2 In (a) the image is padded with the edge values, in (b) the image is padded with a constant solid color and in (c) the image is simply extended to the required input size.
12

---

Under review as a conference paper at ICLR 2020
(a) Edge pixel padding
(b) Constant color padding
(c) Expanding bounding-box
START
START
START
PRIVACY POLICY
Figure 2: Illustration of different padding methods to resize the image to the network input size.
D COLOR CLIPPING USING SALIENCY MAPS
To improve color clipping results we are limiting the colors which the predicted colors can be clipped by only considering the top 5 colors within the thresholded saliency map of the input image. An illustration of this process is shown in Figure where (a) shows initial input image, (b) the saliency map the prediction, and (c) and (d) the thresholded saliency map (we use threshold 0.8) and the colors it contains.
(a) Input
(b) Saliency map
(c) Thresholded map
(d) Masked colors
GOT IT
OO SX
eorls
GOT !!
Figure 3: Restricting colors for color clipping.
E PERCEIVABLE ATTRIBUTE DIFFERENCE
We use to denote the distance between two attribute values. For all numerical attributes except colors, the distance is defined the Vj. To better capture the difference between colors,
We define the perceivable difference for each attribute in Table 5
attribute value difference, i.e., d(ys. Uj) we define their distance using the CIE76 formula (Schanda 2007), denoted dE. Furthermore, we provide illustration of the worse case perceivable difference for each attribute in Table
Table 5: Perceivable difference definition for all attributes used in our work.
Attribute
same (=)
similar (z))
different (+)
Border Color
< 5dE
5dE < c S 10dE
10dE <
Border Radius
1dp
1dp E S 3dp
3dp
Border Width
S Odp
Odp < S
1dp
1dp
Main Color
5dE
5dE < < 10dE
10dE
Padding
1dp
< 1dp
3dp
3dp
Shadow
< Odp
Odp < S 2åp
2dp < 6
Text Color
5dE
5dE e 10dE
10dE <
Text Font Family
same font
different font
Text Gravity
same gravity
different gravity
Text Size
1sp
1dp <
2dp
2sp
Height
< 1dp
1dp < < 3dp
3dp <
Width
2dp
2dp < 4dp
4dp
13

---

Under review as a conference paper at ICLR 2020
Table 6: Examples of perceivable difference between two attribute values. For the same (=) and the
similar (-) perceivable difference, we include worst case examples.
Attribute
Ground-trute
Examples of Perceivable Difference
same
similar (
different (+)
Border Color
Ok
Ok
Ok
Ok
Border Radius
Ok
Border Width
Ok
Ok
Ok
Ok
Main Color
Ok
Padding
Ok
Ok
Ok
Ok
Shadow
Ok
Ok
Text Color
Ok
Ok
Ok
Ok
Text Font
Ok
Ok
0k
Text Gravity
Ok
Ok
Ok
Text Size
Ok
Ok
Ok
Ok
Height
Ok
Ok Ok
Ok
Width
Ok
Ok
Ok Ok
F DATASETS AND INFERRED IMPLEMENTATION VISUALIZATIONS
We provide illustrations of our approach for inferring Android Button implementations from im- ages. Concretely, we include examples of images for which our approach works well, as well as examples where our models make mistakes. The visualizations for the synthetic Daym and real- world Dsplay dataset of buttons found in Google Play Store applications are shown in Table 7 and Table &, respectively. Each table row is divided into 4 parts: an image of the input, the preprocessed input image, a rendering of the predicted Button and rendering of the refined Button.
14

---

Under review as a conference paper at ICLR 2020
Table 7: Visualization of the attribute predictions for the synthetic buttons in the Daym dataset.
Input
Preprocessed
Predicted
Refined
Good predictions
Poor predictions
15

---

Under review as a conference paper at ICLR 2020
Table 8: Visualization of the attribute predictions for the real- world buttons the Doplay dataset.
Input
Preprocessed
Predicted
Refined
Good predictions
SIGN
Hewway' Docins
Login with aevlookk
Lugin wil Fanebook
Login with Facebook
Login with Facebook
Login with Emaiil
Not Regstered! Sign Up
Sige Google
bosk
Twitier
Twilier
Twitter
Twitte
Timill
Poor predictions
SEAAL
RETALL
Gre-stop lagin with the. GO Sans sacurity
passeord
one scree'
The Aril
OL: Sotiit
Erter your User
LDG
16


Reviews:
[{"review_id": "SJlsL5NoFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach for reverse-engineering webpages using Siamese networks and imitation learning. While the idea of using synthetic data (which can be easily procedurally generated) to do this reverse-engineer training is very clever, prior work has exploited it also. Novel elements include the attribute refinement using imitation learning, and the authors show the effect of this step, but the improvement is small. Thus, the limited novelty and not very convincing results make the question the potential impact of this paper.\n\nSome questions:\na) The authors mention they cannot use a GAN-style method because all generated images are by definition true/real; how about learning whether a *pair* is real or fake? (where the pair consists of the design specification and the rendered version). \nb) Are the baselines strong enough? None of them seem to be from recent prior work. How about a direct comparison to some of the work listed in the second para on page 2?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HkeXm6OKKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors proposed an algorithm to predict the attribute of GUI elements from rasterized design images. The problem is separated into two steps. The first step is to predict initial values of the attributes (border width, color, padding etc) from the image where the type of UI element and set of attributes are already known. Authors designed a typical convolutional DNN for each of the attributes. The second step is to learn a policy \\pi to iteratively adjust one attribute a time until the final rendering matches input image pixel-perfect. \n\nAuthors conducted the experiment with large synthetic data set of Android buttons, and evaluate the performance with a held-out synthetic set as well as a 110 hand crafted real world buttons set sampled from apps in Google Play App Store. Several variations of the same model were compared. The result showed that network structure, padding strategy (this is a bit unexpected), background type and color selection strategy all affect the accuracy significantly. \n\nReviewer has concern about the scope and application value of the problem as a research paper. A number of key prior assumptions have to be made to let the algorithm work: the type of the UI element need to be known; the list of attributes and their value ranges need to be fixed beforehand and each of the attribute demands a DNN; the refinement iteration has the actual rendering in the loop which could be costly on current App development platforms. \n\nFeedback questions.\n\n1) The abstract mentioned vector image as input but the main body only discussed rasterized images.\n\n2) Since the rendering process could be costly, it's useful to discuss the speed of convergence in the attribute value adjustment iterations.\n\n3) Reviewer is interested in the nature of the error (7.5% loss) but it's not discussed.\n\n4) In related work, authors mentioned the REINFORCE algorithm by Williams et al 1992 is expensive. It could help the reader if a brief explanation of why it's expensive is provided.\n\n5) In Section 3 Background, authors mentioned that the attributes are not independent of each other, which is a major challenge. Reviewer would like to see some discussion or experiment data on how this affects the process and how did the current algorithm address it.\n\n6) It's a bit surprise that color clipping method has a big impact on accuracy. Some examples could have helped the reviewer understand it.\n\n7) In Section 4.2 first paragraph, it seems that the user of the algorithm need to set the [-c, c] clipping values manually per feature. This sounds like quite some prior knowledge and hand-tuning.\n\n8) In Section 4.2. Second paragraph, Reviewer can understand the nessesity of additive and subtractive operations, but why multiplication?\n\n9) In the equation to the end of page 5, do we need an extra outer bracket for the denominator? By the way, the equations should be numbered for easier reference.\n\n10) The task of predicting Android button attributes, while practical, seems over-simplified. Reviewer suggests at least experiment with a set of common UI elements to proof the horizontal performance.\n\n11) In Section 5.1, Reviewer respect the experiment results but doesn't understand why solid color background provides the best variety but screenshots don't. May need more analysis and explanation.\n\n12) In Table 1, the first line for variant (C) also looks pretty good, or even better than core on the Android app store dataset.\n\n13) In Section 5.1, the effect of color clipping selection seems very specific to applications with a fixed color palette. While this is indeed the majority, this prior knowledge need to be speciifed clearly by saying it's tailored towards such applications (or use more examples to proof that's not the case).\n\n14) In Table 2: Pixel Sim's performance on Best Prediction Initialization seems pretty good, and Reviewer believes this is the more practical scenario. Is a more complicated Siamese Network justified?\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bkgi2uRR9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to infer the attribute values of an input image representing a user interface. The model first infers the most likely initial attribute values, and iteratively refine them to improve the similarity between the input image and the interface generated from the newly inferred attributes. The model is trained on synthetic datasets generated by a black box rendering engine, and generalizes well to real-world datasets. To address the issues of pixel based metrics and mean squared error, the authors instead uses the probability that two images are equal in the attribute space to define the cost between these two images.\n\nAlthough I'm not familiar with the problem addressed by the paper, I found the paper very clear and well written. Overall, the method is sensible and elegant, and could easily be applied to other domains. My only disappointment is maybe the fact that only the Android Button was considered, and it is not clear how the model would perform with other and more sophisticated Android components.\n\nA few questions for the authors:\n- How many steps do you perform in the refinement loop? This is an important information, but I couldn't find it in the paper. Typically, I was surprised to see in the first row of Table 2 that the model with a random attribute initialization can reach such a high performance. But I imagine that you need many more iterations to converge if you start from random attributes than from the best prediction initialization?\n- Also, what is the stopping criterion? Do you decide to stop when none of the proposed attribute changes improve the pixel-level accuracy?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The majority of reviewers suggest rejection, pointing to concerns about design and novelty. Perhaps the most concerning part to me was the consistent lack of expertise in the applied area. This could be random bad luck draw of reviewers, but more likely the paper is not positioned well in the ICLR literature. This means that either it was submitted to the wrong venue, or that the exposition needs to be improved so that the paper is approachable by a larger part of the ICLR community. Since this is not currently true, I suggest that the authors work on a revision.
Decision: Reject