Learning to Infer User Interface Attributes from Images

1 INTRODUCTION . With over 5 million applications in Google Play Store and Apple App Store and over a billion webpages , a significant amount of time can be saved by automating even small parts of their development . To achieve this , several tools have been recently developed that help user interface designers explore and quickly prototype different ideas , including Sketch2Code ( Microsoft , 2018 ) and InkToCode ( Corrado et al. , 2018 ) , which generate user interface sketches from hand-drawn images , Swire ( Huang et al. , 2019 ) and Rico ( Deka et al. , 2017 ) , which allow retrieving designs similar to the one supplied by the user and Rewire ( Swearngin et al. , 2018 ) , which transforms images into vector representations consisting of rectangles , circles and lines . At the same time , to help developers implement the design , a number of approaches have been proposed that generate layout code that places the user interface components at the desired position ( e.g. , when resizing the application ) . These include both symbolic synthesis approaches such as InferUI ( Bielik et al. , 2018 ) , which encodes the problem as a satisfiability query of a first-order logic formula , as well as statistical approaches ( Beltramelli , 2018 ; Chen et al. , 2018 ) , which use encoder-decoder neural networks to process the input image and output the corresponding implementation . In this work , we explore a new domain of inferring an implementation of an user interface component from an image which when rendered , looks visually the same as the input image . Going from an image to a concrete implementation is a time consuming , yet necessary task , which is often outsourced to a company for a high fee ( replia , 2019 ; psd2android , 2019 ; psd2mobi , 2019 ) . Compared to prior work , we focus on the pixel-accurate implementation , rather than on producing sketches or the complementary task of synthesizing layouts that place the components at the desired positions . Concretely , given a black box rendering engine that defines a set of categorical and numerical attributes of a component , we design a two step process which predicts the attribute values from an input image – ( i ) first , we train a neural model to predict the most likely initial attribute values , and then ( ii ) we use imitation learning to iteratively refine the attribute values to achieve pixel-level accuracy . Crucially , all our models are trained using synthetic datasets that are obtained by sampling the black box rendering engine , which makes it easy to train models for other attributes in the future . We instantiate our approach to the task of inferring the implementation of Android Button attributes and show that it generalizes well to a real-world dataset consisting of buttons found in existing Google Play Store applications . In particular , our approach successfully infers the correct attribute values in 94.8 % and 92.5 % of the cases for the synthetic and the real-world datasets , respectively . 2 RELATED WORK . As an application , our work is related to a number of recently developed tools in the domain of user interface design and implementation with the goal of making developers more productive , as discussed in Section 1 . Here we give overview of the related research from a technical perspective . Inverting rendering engines to interpret images The most closely related work to ours phrases the task of inferring attributes from images as the more general task of learning to invert the rendering engines used to produce the images . For example , Wu et al . ( 2017 ) use reinforcement learning to train a neural pipeline that given an image of a cartoon scene or a Minecraft screenshot , identifies objects and a small number of high level features ( e.g. , whether the object is oriented left or right ) . Ganin et al . ( 2018 ) also use reinforcement learning , but with an adversarially learned reward signal , to generate a program executed by a graphics engine that draws simple CAD programs or handwritten symbols and digits . Johnson et al . ( 2018 ) and Ellis et al . ( 2018 ) design a neural architecture that generates a program that when rendered , produces that same 2D or 3D shape as in the input image . While Johnson et al . ( 2018 ) train the network using a combination of supervised pretraining and reinforcement learning with a custom reward function ( using Chamfer distance to measure similarity of two objects ) , Ellis et al . ( 2018 ) use a two step process that first uses supervised learning to predict a set of objects in the image and then synthesizes a program ( e.g. , containing loops ) that draws them . In comparison to these prior works , our approach differs in three key aspects . First , the main challenge in prior works is predicting the set of objects contained in the image and how to compose them . Instead , the focus of our work is in predicting a set of object properties after the objects in the image were already identified . Second , instead of using the expensive REINFORCE ( Williams , 1992 ) algorithm ( or its variation ) to train our models , we use a two step process that first pretrains the network to make an initial prediction and then uses imitation learning to refine it . This is possible because , in our setting , there is a fixed set of attributes known in advance for which we can generate a suitable synthetic dataset used by both of these steps . Finally , because our goal is to learn pixel-accurate attribute values , the refinement loop takes as input both the original image , as well as the rendered image of the current attribute predictions . As a result , we do not require our models to predict pixel-accurate rendering of an attribute value but instead , to only predict whether the attribute values in two images are the same or in which direction they should be adjusted . Attribute prediction Optical character recognition ( Jaderberg et al. , 2016 ; Lyu et al. , 2018 ; Jaderberg et al. , 2014 ; Gupta et al. , 2016 ) is a well studied example of predicting an attribute from an image with a large number of real-world applications . Other examples include predicting text fonts ( Zhao et al. , 2018 ; Wang et al. , 2015 ; Chen et al. , 2014 ) , predicting eye gaze ( Shrivastava et al. , 2017 ) , face pose and lighting ( Kulkarni et al. , 2015 ) , chair pose and content ( Wu et al. , 2018 ) or 3D object shapes and pose ( Kundu et al. , 2018 ) , to name just a few . The attribute prediction network used in our work to predict the initial attribute value is similar to these existing approaches , except that it is applied to a new domain of inferring user interface attributes . As a result , while some of the challenges remain the same ( e.g. , how to effectively generate synthetic datasets ) , our main challenge is designing a pipeline , together with a network architecture capable of achieving pixel-level accuracies on a range of diverse attributes . 3 BACKGROUND : USER INTERFACE ATTRIBUTES . Visual design of user interface components can be specified in many different ways – by defining a program that draws on a canvas , by defining a program that instantiates components at runtime and manipulates their properties , declaratively by defining attribute values in a configuration file ( e.g. , using CSS ) , or by using a bitmap image that is rendered in place of the components . In our work , we follow the best practices and consider the setting where the visual design is defined declaratively , thus allowing separating the design from the logic that controls the application behaviour . Formally , let C denote a component with a set of attributes A . The domain of possible values of each attribute ai ∈A is denoted as Θi . As all the attributes are rendered on a physical device , their domains are finite sets containing measurements in pixels or a set of categorical values . For example , the domain for the text color attribute is three RGB channels N3× [ 0,255 ] , the domain for text gravity is { top , left , center , right , bottom } and the domain for border width is N [ 0,20 ] . We distinguish two domain types : ( i ) comparable ( e.g. , colors , shadows or sizes ) for which a valid distance metric d : Θ×Θ→N [ 0 , ∞ ) exists , and ( ii ) uncomparable ( e.g. , font types or text gravity ) for which the distance between any two attribute values is equal to one . We use Θ ⊆ Θ1×· · ·×Θn to denote the space of all possible attribute configurations , and use the function render : Θ→R3×h×w to denote an image with width w , height h and three color channels obtained by rendering the attribute configuration y∈Θ . Furthermore , we use the notation y∼Θ to denote a random sample of attribute values from the space of all valid attribute configurations . Finally , we note that attributes often affect the same parts of the rendered image ( e.g. , the shadow is overlayed on top of the background ) and they are in general not independent of each other ( e.g , changing the border width affects the border radius ) . 4 LEARNING TO INFER USER INTERFACE ATTRIBUTES FROM IMAGES . We now present our approach for learning to infer user interface component attributes from images . Problem statement Given an input image I ∈ R3×h×w , our goal is to find an attribute configuration y ∈ Θ which when rendered , produces an image most visually similar to I : arg miny∈Θ cost ( I , render ( y ) ) where cost : I × I → R [ 0 , ∞ ) is a function that computes the visual similarity of a given user interface component in two images . It returns zero if the component looks visually the same in both images or a positive real value denoting the degree to which the attributes are dissimilar . The first challenge that arises from the problem statement above is how to define the cost function . Pixel based metrics , such as mean squared error of pixel differences , are not suitable and instead of producing images with similar attribute values , produce images that have on average similar colors . Training a discriminator also does not work , as all the generated images are produced by rendering a set of attributes and are true images by definition . Finally , the cost can be computed not over the rendered image but by comparing the predicted attributes y with the ground-truth labels . Unfortunately , even if we would spend the effort and annotated a large number of images with their ground-truth attributes , using a manually annotated dataset restricts the space of models that can be used to infer y to only those that do supervised learning . In what follows we address this challenge by showing how to define the cost function over attributes ( used for supervise learning ) as well as over images ( used for reinforcement learning ) , both by using a synthetically generated dataset . Our approach To address the task of inferring user interface attributes from images , we propose a two step process that – ( i ) first selects the most likely initial attribute values arg maxy∈Θ p ( y | I ) by learning a probability distribution of attribute values conditioned on the input image , and then ( ii ) iteratively refines the attribute values by learning a policy π ( ∆y ( i ) |I , render ( y ( i ) ) ) that represents the probability distribution of how each attribute should be changed , conditioned on both the original image , as well as the rendered image of the attribute configuration y ( i ) at iteration i . We use the policy π to define the cost between two images as cost ( I , I ′ ) : = 1−π ( ∆y=0 | I , I ′ ) . That is , the cost is defined as the probability that the two images are not equal in the attribute space . We illustrate both steps in Figure 1 with an example that predicts attributes of a Button component . In Figure 1 ( a ) , the input image is passed to a set of convolutional neural networks , each of which is trained to predict a single attribute value . In our example , the most likely value predicted for the border width is 2dp while the most likely color of the border is # 4a4a4a . Then , instead of returning the most likely attribute configuration y , we take advantage of the fact that it can be rendered and compared to the original input image . This give us additional information that is used to refine the predictions as shown in Figure 1 ( b ) . Here , we use a pair of siamese networks ( pretrained on the prediction task ) to learn the probability distribution over changes required to make the component attributes in both images the same . In our example , the network predicts that the border color and the text gravity attributes have already the correct values but the border width should be decreased by 2dp and the shadow should be increased by 4dp . Then , due to the large number of different attributes that affect each other , instead of applying all the changes at once , we select and apply a single attribute change . In our example , the ∆y corresponds to adjusting the value of border width by −2dp . Since the change is supposed to correct a mispredicted attribute value , we accept it only if it indeed makes the model more confident that the prediction is correct . ( a ) Attribute Prediction ( Section 4.1 ) CNNCNN y border width 2dp border color shadow 0dp text gravity center . . . p ( y | I ) 0.77 0.99 0.91 0.96 D = { ( render ( y ( i ) ) , y ( i ) ∼ Θ ) } N i=1 Training Input Image I p ( y | I ) Predicted Attributes render ( y ) Synthetic datasets We instantiate our approach by training it purely using synthetic datasets , while ensuring that it generalizes well to real-world images . This allows us to avoid the expensive task of collecting and annotating real-world datasets and more importantly , makes our approach easily applicable to new domains and attributes . In particular , given a space of possible attribute configurations Θ and a rendering function render , we generate two different datasets D and D∆ used to train the attribute prediction network and the policy π , respectively . The dataset D = { ( render ( y ( i ) ) , y ( i ) ∼ Θ ) } Ni=1 is constructed by sampling a valid attribute configuration y ( i ) ∼ Θ and rendering it to produce the input image . To generateD∆ , we sample two attribute configurations y ( i ) , y ( j ) ∼Θ that are used to render two input images and train the network to predict the difference of their attributes , that is , D∆ = { ( 〈render ( y ( i ) ∼ Θ ) , render ( y ( j ) ∼ Θ ) 〉 , ∆ ( y ( i ) −y ( j ) ) ) } Mi , j=1 . For both datasets , to avoid overfitting when training models for attributes with large domain of possible values , we sample only a subset of attributes , while setting the remaining attributes to values from the previous example y ( i−1 ) . As a result , every two consecutive samples are similar to each other , since a subset of their attributes is the same . Further , because the real-world images do not contain components in isolation but together with other components that fill the rest of the screen , we introduce three additional attributes xpos∈N , ypos∈N and background . We use xpos and ypos to denote the horizontal and vertical position of the component in the image , respectively . This allows the network to learn robust predictions regardless of the component position in the image . We use background to select the background on which the component is rendered . We experimented with three different choices of backgrounds – only while color , random solid color and overlaying the component on top of an existing application , all of which are evaluated in Section 5 .

Summary: This paper proposes an approach for reverse-engineering webpages using Siamese networks and imitation learning. While the idea of using synthetic data (which can be easily procedurally generated) to do this reverse-engineer training is very clever, prior work has exploited it also. Novel elements include the attribute refinement using imitation learning, and the authors show the effect of this step, but the improvement is small. Thus, the limited novelty and not very convincing results make the question the potential impact of this paper.