Pairwise GUI Dataset Construction Between Android Phones and Tablets
Authors: Han Hu
Haolan Zhan
Yujin Huang
Di Liu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Datasets
Tablet
GUI Retrieval
GUI Generation
GUI Recommendation
TL;DR: A Pairwise Dataset for GUI Conversion and Retrieval between Android Phones and Tablets

Abstract: In the current landscape of pervasive smartphones and tablets, apps frequently exist across both platforms.
Although apps share most graphic user interfaces (GUIs) and functionalities across phones and tablets, developers often rebuild from scratch for tablet versions, escalating costs and squandering existing design resources.
Researchers are attempting to collect data and employ deep learning in automated GUIs development to enhance developers' productivity.
There are currently several publicly accessible GUI page datasets for phones, but none for pairwise GUIs between phones and tablets.
This poses a significant barrier to the employment of deep learning in automated GUI development.
In this paper, we introduce the Papt dataset, a pioneering pairwise GUI dataset tailored for Android phones and tablets, encompassing 10,035 phone-tablet GUI page pairs sourced from 5,593 unique app pairs.
We propose novel pairwise GUI collection approaches for constructing this dataset and delineate its advantages over currently prevailing datasets in the field.
Through preliminary experiments on this dataset, we analyze the present challenges of utilizing deep learning in automated GUI development.

Pairwise GUI Dataset Construction Between Android Phones and Tablets
Han Hu, Haolan Zhan; Yujin Huang, Di Liu
Monash University Melbourne, Australia
(han.hu haolan. zhan, yujin. pujin.huang. wwwoaah.oossoe..eeeeeee.o.aaaa.. edu, .1iuu0240studentt.ooo wwwpantt.oaasheemm.cm edu hwwngaamonasshh.....mmmmm
Abstract
In the current landscape of pervasive smartphones and tablets, apps frequently exist across both platforms. Although apps share most graphic user interfaces (GUIs) and functionalities across phones and tablets, developers often rebuild from scratch for tablet versions, escalating costs and squandering existing design resources. Researchers are attempting to collect data and employ deep learning in automated GUIs development to enhance developers' productivity. There are eurrently several publicly accessible GUI page datasets for phones, but none for pairwise GUIs between phones and tablets. This poses a significant barrier the employment of deep learning in automated GUI development. In this paper, we introduce the Papt dataset, a pioneering pairwise GUI dataset tailored for Android phones and tablets, encompassing 10,035 phone-tablet GUI page pairs sourced from 5,593 unique app pairs. We propose novel pairwise GUI collection approaches for constructing this dataset and delineate its advantages over currently prevailing datasets in the field. Through preliminary experiments on this dataset, we analyze the present challenges of utilizing deep learning in automated GUI development.
Introduction
Mobile apps ubiquitous in our daily life for supporting different tasks such as reading, chatting, and banking. Smartphones and tablets are the two types of portable devices with the most available apps [49] To conquer the market, one app is often available on both smartphones and tablets [41). Due to comparable functionalities, the smartphone and tablet versions of the same app have highly similar Graphical User Interface (GUT). Popular apps always share a similar GUI design between phone apps and tablet apps, for example, YouTube (55] and Spotify [48]. If a tool can automatically recommend GUI design for the appropriate tablet platform based on existing mobile GUIs, it can significantly minimise the developer's engineering effort and accelerate the development process. From the user side, a comparable design could provide data or study on user preferences for consistency across different devices. It reduces the need for them to learn new navigation and interaction patterns 44 Therefore, automated GUT development tasks, such as the cross-plas ploss-platfomm conversion of GUI designs, GUI recommendations. etc., are gradually gaining attention from industry However, the field of automatic GUI development is still in a research bottleneck, lacking breakthroughs and widely recognized tools or methods. If present developer needs to develop a cablet-compatible version of their app. they usually start from scratch, resulting in needless cost increases and wasted existing design resources.
and academia (57 37 [2]
According our observations, the growth of automated GUI development is hindered by two reasons. First, as deep learning approaches, particularly generative models [50), become more widespread in the field of automated GUI development, researchers increasingly need pairwise, high-quality GUT
corresponding author
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Figure I: An example of a phone-tablet GUI pair of the app 'BBC News'. The GUI on the left is from the phone, while the GUI on the right is from the tablet.
dataset for crucial purposes such as training models, and analyzing patterns. The current datasets, for example, Rico 15, ReDraw [K] and Clay (37], only include single GUI pages with UI metadata and UI screenshots, and there are no pairwise corresponding GUI page pairs. Current datasets are only suitable for GUI component identification, GUI information summarising, and GUI completion. The lack of valid GUI pairs in current datasets has severely hindered the growth of GUI automated development.
Second, the collecting of GUI pairs between phones and tablets more labor-intensive Individual GUI pages can be automatically collected and labeled by current GUI testing and exploration tools [25] 42 to speed up the collection process. However, due to the disparity in screen size and GUI design between phones and tablets, it is challenging to automatically align the content on both GUI pages. Figure [ shows an example of a phone-tablet GUI pair the app 'BBC News' [ To accommodate tablet devices, the UI component group in the phone GUI is converted to the GUI parts marked as in the tablet GUI. We can find that these components not only change their positions and sizes but also the contents and types of GUIs. Another UI component group, which is marked as 2 in the tablet GUI, is not present in the phone GUI at all. One tablet GUI page may correspond to the contents of multiple phone GUI pages due to the different screen sizes and design styles. The UI contents in group of the tablet GUI correspond to other phone GUIs.
In this paper, we introduce our novel pairwise GUI data collection methodologies and present the newly curated Papt dataset. Papt is the first PAirwise GUI dataset between Android Phones and Tablets, encompassing 10,035 GUI pairs from 5,593 app pairs. The initiative commences with an articulate delineation of the data collection methods, coupled with comprehensive statistical analysis of the dataset. Thereafter, expound upon the specific structuring of the data, highlighting the intrinsic advantages conferred by this unique organization. Progressing into the experimental phase, undertake preliminary investigations, expressly tailored for the GUI conversion task. we engage in a discussion of the challenges currently faced by existing models in the realm GUI conversion, shedding light the complexities inherent to this domain.
Finally,
novel dataset, featuring pairwise GUI representations spanning both mobile and tablet platforms, presents itself as an indispensable resource in the evolving realm of automated GUI design research. Specifically, this dataset is instrumental in cross-platform and adaptive design initiatives 27 32
serving as foundation for model training, verification, and empirical analysis. It further delivers tangible data samples grounded real-world contexts, accelerating advancements in user experience augmentation [31 and bolstering accessibility features [39] (5). Moreover, it can establish a unified benchmark for stringent testing and validation in related domains 20 29). Beyond its tangible utility in practical developmental domains, the dataset also enriches the theoretical domain, pushing the boundaries of contemporary interface design research.
In summary, the novel contributions of this paper are the following:
We contribute the first pairwise GUT dataset between Android phones and tablets

---

We demonstrate our pair collection approaches and open source our data collection tool with the intention foster further research and facilitate the collection of customized GUI data by future researchers
We perform preliminary experiments on the dataset and report the experimental results. We discuss the challenges faced by the current models the GUT conversion task.
2 Related Work
Android GUT Dataset Numerous datasets have been curated to support deep learning applications
in mobile app design. Notably, Rico [1],, with 72,219 screenshots from 9,772 apps, serves as the cornerstone for GUI modeling. Despite its prominence, Rico's limitations, such as noise and erroncous labeling [16 35 have spurred further dataset enhancements. Enrico [36] is the first enhanced dataset drawn from Rico, which is used for mobile layout design categorization and consists of 1460 high-quality screenshots produced by human revision, covering 20 different GUI design topics. The VINS dataset (9). developed specifically for detecting GUI elements, comprises 2,740 Android screenshots manually collected from various sources including Rico and Google Play. Since the data cleaning process for both the Enrico and VINS datasets involves humans, adopting such approaches to improve existing Android GUT datasets at scale is expensive and time-consuming. In contrast, CLAY 37 leverages deep learning for dataset noise reduction, producing 59,555 screenshots based on Rico. Beyond Rico derivatives, several works TTO 12 13
26 also build their datasets for various GUI-related tasks such as Skeleton Generation, search and component prediction.
53
Yet, contemporary Android GUI datasets grapple with obsolescence due to infrequent updates. More critically, the absence of pairwise GUI pages between distinct mobile devices, such as phones and tablets, severely constrains the progress of end-to-end tasks, inclusive but not limited GUI generation and GUI search. So, we introduce the first pairwise dataset between Android phones and tablets to fill the current gaps.
Layout Generation Facilitating GUT conversion between Android and tablet apps underpins our dataset's 's intent. We briefly overview contemporary layout generation methods. LayoutGAN (38) pioneering generative layouts, employs GANs with self-attention and introduces a differentiable wireframe rendering layer for CNN-based discrimination. LayoutVAE VAE (33) is an autoregressive model, utilizing LSTM (24) to process UI elements and VAEs [34, (6) for layout creation. VTN [6] builds on the VAE structure, replacing encoder-decoder with Transformers 5 for layout learning framework, proposed for layout generation. It captures co-occurrences and implicit relationships among elements in layouts and uses such captured features to produce layouts with bounding boxes as units. Based on our experiment results, we find that current layout generation models have limited capacity for GUT conversion between Android and tablet layouts, suggesting the potential research direction in automated GUT development.
sans annotations. At the same time, LayourTransformer [22 purely Transformer-basee
Methodologies of Data Collection
In the process of data collection, our methodology is bifurcated into two phases: the algorithmic automatic pairing of phone-tablet GUI pages, followed by manual verification of these pairings. In this section, we first introduce the source app pairs of our dataset in subsection 3.1 Then, we demonstrate two employed distinct techniques for matching GUI pairs: by dynamically adjusting the device resolution (as detailed in subsection 3.2), and by computing similarity metrics (elaborated upon in subsection 3.3 Subsequent to the automated pairing, pairs are rigorously scrutinized by a panel of three experts. each possessing minimum of one year of GUI development expertise, to discard any inaccuracies or mismatches, which is described in subsection 3.4
3.1 Source App Pairs
We first crawl 6,456 tablet apps from Google Play. Then we match their corresponding phone apps by their app names and app developers. Finally, we collect 5,593 valid phone-tablet app pairs from 22 app categories as the data source for this dataset. The three most common categories of apps

---

in the data source are: Entertainment (8.87%), Social (7.04%), and Communication (5.83%). The categories of apps in our data source exhibit a balanced and dispersed distribution. The majority of these categories account for approximately 4% to 6% of the entire dataset, reinforcing the dataset's robustness in terms of generalizability and heterogeneity. exhaustive breakdown of app category distribution is available in the supplementary materials.
3.2 Dynamically Adjusting Resolution for GUI Pairing
In the Android ecosystem, adaptive layouts enable tailored user experience that caters to spectrum of screen dimensions, ensuring app compatibility across phones, tablets, foldable and ChromeOS devices, well supporting varied orientations and resizable configurations [.]. Consequently, certain apps instantiate device-specific layout files, contingent upon the resolution of the host device. Notably, such apps deploy identical APK files across both mobile and tablet platforms. Our initial focus is directed towards this category of apps.
decompile
install and explore
4. change resolution
2. detection
5. collect
tablet layout file
dalaset
phone-tablet GUT pair
Figure 2: Pipeline of dynamically adjusting the resolution for GUI pairing
According to Android's official guidelines for supporting different screen sizes 18 Android de- velopers can provide alternate layouts for displays with minimum width measured in density- independent pixels (dp or dip) by using the smallest width screen size qualifiers. Illustratively, for MainActivity, one might encounter layouts like httplayyoo/mmiinnnttiiiiii... _activitt..mll for smartphones and for tablets exhibiting width of 600 Jensity-independet pixels. It is pivotal to understand that the qualifier (sw600dp) references the more diminutive dimension among the screen's two axes, unaffected by device orientation. Such dedicated layouts indicate that the respective app's MainActivity is tailored for tablet-centric interface. Notably, two prevalent smallest width qualifiers, 600dp and 720dp, are meticulously designed to align with the prevalent 7" and 10" tablets [18).
Figure shows the pipeline of matching GUI pairs by dynamically adjusting device resolutions. Drawing from our initial analyses, the initial steps involve decompiling the acquired app pairs and probing for layout files that cater to the 600dp and 720dp tablet configurations within the source files (as depicted in steps 1 and 2 of Figure 2) Out of the assessed 5,593 app pairs, layout files characteristic to the smallest width qualifiers for tablet devices are identifiable 1,214 pairs, all of which utilized the same app across phone and tablet platforms. It becomes evident that such pairs employ Responsive/Adaptic layouts, which modulate the GUI layout contingent on the device they hosted on. For the identified app pairs, the Windows Manager Command of ADB (adb shell wm size) [4] is employed to dynamically alter the device's 's resolution, thereby revealing their intended smartphone and tablet GUI layouts. By sequentially emulating both the tablet and phone resolutions on a single tablet device, we facilitate the automatic pairing of GUIs captured pre and post these simulations, illustrated in steps 4 and 5 of Figure 2
3.3 Similarity-Band GUI Pairing
Excluding 1,214 applications with platform-speciffe XML definitions, we apply UI similarity- based GUI pairing to other 4,379 phone-tablet pairs, accommodating those developing mirror apps or utilizing Java/Kotlin code for dynamic screen adaptation.
As depicted in 3 the process for matching GUT pairs is primarily hinged on the comparison of UI similarities. Notably, since each app comprises numerous GUI pages, direct pairwise

---

1. install
explore activities
capture
5. threshold exceeded,
collection
install
2. explore activities
capture
3. activity comparison
4. GUT similarity comparison
Figure Pipeline of similarity-based GUI pairing
comparison of all pages across mobile and tablet apps becomes computationally intensive. In the Android ecosystem, an activity serves as the basic window for the app to render its UI [3]. Initially, corresponding Android activity pairs for each app pair are discerned. This is followed by comparison of individual GUI pages rooted their associated Android activity pairs. The methodology employed to ascertain Android activity pairs involves the extraction of activity names from each GUI, which are subsequently encoded into semantic vectors via a pre-trained BERT model [17]. Pairs are then matched based the proximity of their semantic vectors, delincated in Step 3 of Figure 3 For instance, GUTs in activities titled homeActivity and mainActivity can be correlated based on the similarity of their vectors. It's noteworthy that single Android activity can encompass multiple fragments ( and GUI pages 40 [1], each distinguished by its unique set of UI components and layout structures in prevalent applications. To refine the pairing process at a lower granularity, the attributes of GUI components in activity pairs are contrasted between mobile and tablet devices, process elucidated in Step 4 of Figure Herein, UI components are primarily distinguished based on their typologies and inherent properties. Components from both mobile and tablet platforms, exhibiting analogous types and attributes, are subsequently deemed as the same views. As an illustrative point, two coinciding TexvViews, Image are recognized as paired entities. more than half of the UI components in two GUI pages are paired, they are considered a phone-table GUI pair (Step 5 in Figure 3).
or Buttons with congruent texts or images
3.4 Manual GUI Pair Verification
We collect 12,331 GUI pairs in the automated collection. Due to the inherent constraints of ADB, contemporary data collection mechanisms frequently falter in retrieving metadata for the UI type Web View as well as certain bespoke third-party UI components. Concurrently, for certain obscured Android fragments and UIs, only the metadata for the foreground UI is of relevance, given that the overlaid UT remains undisplayed on the prevailing screen. Yet, extant data collection tools [1 might inadvertently accumulate data from both the visible and concealed UT, leading to potential data inaccuracies.
To address these challenges, an additional round of manual data validation is undertaken by a panel of three specialists, each boasting a minimum of one year's expertise in Android development. These individuals meticulously scrutinize each pair against two pivotal metrics: data integrity and pair coherence. This manual validation encompass a meticulous verification of harvested pairs, identification and elimination of pairs with erroneous metadata, and an assessment of the logical consistency of the matched pairs. Throughout this rigorous verification phase, the evaluators also actively engage in the manual matching of certain phone-tablet GUI pairs. In this round of manual verification, we remove 2,296 pairs of logical inconsistency and obvious visual mismatches between screenshots and their metadata.
4 Characteristics of the Papt Dataset
Using the methodologies described in Section 3 pairs. this section, we elucidate key features of the Papt dataset encompassing UI View types (claborated in subsection 4.1). data format (detailed in subsection 4.2 and comparative strengths
we have gathered 10,035 valid phone-tabler GUI
5

---

relative to other datasets (expanded upon in subsection 4.3 Comprehensive statistics of the dataset are available in the supplementary material.
4.1 Distribution of UI View Types
Within Android development, UI views serve fundamental elements for user interface construction, responsible for both rendering and handling user interactions within a designated screen segment [5]. For instance, diverse elements such as buttons, textual content. images, and lists all fall under the category of views. Notably, in our dataset, TexvView and Image View types and their derivatives, like AppCompatTextView
and AppCompatInage View, emerge the most prominent UI view categories. Their respective counts, 73,349 and 68, 19,4,6 overshadow other view types. This dominance can be attributed to the GUI's inherent focus information dissemination predominantly through text and images. Additionally, ImageButton and Button, tallying 10,366 and 10,235 respectively, rank as the third and fourth most prevalent UI views. Given the pivotal role of click operations in user-GUI interactions, the ubiquity of button-related views in our dataset is logical.
Sereenshol
GUI Metndala
duolingo
The Bus' you pyyist likk willd beighn
0.9 PORDONMNN
"heeareff Af "comuderie www.tttwwweeeeeeeeeeee Deettaaee Toar" STARTED Buch udd (egae)14) "laaNaas"" Tactiau" A.
plane Iivesissi Maaoaatiiyy
Figure 4: A screenshot of GUI and its part UI metadata
Figure 5: An example GUI pair in Spotify
4.2
Data Format
In this section, we first show an example screenshot of a GUI and its corresponding UI metadata. Then, introduce the format of each GUI pair in the dataset.
GUI Screenshot and its Metadata We install and run phone-tablet app pairs on the Pixel6 and Samsung Galaxy tab S8, respectively. We use uiautomator2 [51] to collect screenshots and GUI metadata of the dynamically running apps. Figure shows an example of a collected GUT screenshot and metadata of some UI components inside the GUI. This example is from the app 'Duolingo [I9] The metadata is a documentary object model (DOM) tree of current GUIs, which includes the hierarchy and properties (e.g.. class, bounding box, layout) of UI components. We can infer the GUI hierarchy from the DOM tree hierarchy in metadata.
GUI Pair Figure 5 shows an example of pairwise GUI pages of the app 'Spotify' in our dataset. All GUI pairs in one phone-tablet app pair are placed in the same direc- tory. Each pair consists of four elements: screenshot of the GUI running on the phone pphone_1676185 1
the metadata data corresponding to the GUI screenshot on the phone ( (phone_ deeee_16666111 (656189565.00 MaiinMMMiiiittnnnnnn ), screenshot of the GUI running on the tablet (obblee_66661956666 MainActiviry. ), and the metadata data corresponding to the GUI sereenshot on the tablet is Timestamp_Actiio Name. As shown in Figure 5] The filename tablet [ablet_16761800 1676189565_ MainActivity.xm! indicates that this file was obtained by the tablet and was collected with the timestamp 1676189565, this GUI belongs to MainActiviry and this file is a metadata file in XML format. We use timestamps and activity names distinguish phone-tabled GUI pairs.
MainActiviry.xm/ ). The naming format for all files in the dataset
Pairs in the Dataset The pairs the dataset are contained in separate folders according to the app. Most of the app folders are named after the package name of the app's APK, for example, air.com. maccommmmyeriaagg... mobile, and a few are named after the app's name, for example, Spotify. In each app folder, described in Section 4.2 each pair contains four elements: the phone GUI screenshot,

---

the XML file of the phone GUI metadata, the corresponding tablet GUI screenshot, and the XML file of the tablet GUI metadata. We also shared the script for loading all GUI pairs in the open source repository.
4.3 Comparison with Available Datasets
Compared to other current datasets, our dataset has advantages in terms of applicable tasks and data accuracy.
Table 1: Comparison between our dataset and other GUI datasets
Dntased
GLT Platform
RGU
#tara Source App
Latest Lddates
Mainly supported facks
Rico [T
Phome 72,000
9,710
Sep. 2017
UT Component Recognation, GUT completion
UtDeode [0
Phone
185,277
5,043
2018
Ul Skeleson Generation
Gallery D.C. 0T
Phone
68,702
5,04
Nov. 2019
UI Sourch
LrhiiDroid
Phase
394,289
15,007
20120
UI Search
Prediction
U1SK
54,987
7,748
20.00
kanzo
Phone
9,700
Layout Design Caagorization
Phone
Search
WarkenWWorkk
Phose
22,417
6,269
ictoza sumatarization
Clay
Phone
59.555
0,700
May: 2020
Component Recugnition, GUI completion
Papt
Phone, Tablee
20,070
11,186
Jan
UT Component Rocagnition, GUI completion GUT conversion Ul search
Broader Applicable Tasks Table I shows a summary of our and other GUT datasets. First, since our data consist of phone-tablet pairs, must locate the corresponding GUI pages between phones and tablets, resulting in a lesser number of pages than comparable datasets. However, we now have broader data source (including tablet GUIs), more supported tasks, and newer data. Notably, it the only available GUI dataset that contains phone-tablet pairwise GUIs, which provides effective data support for the application of deep learning techniques GUI generation, recommendation, testing, vulnerability detection, and other domains 22
29
28
30
14
Data Accuracy Specifically, our data eliminates large number of GUI visual mismatches that are frequent in current datasets like as Rico and Enrico. Due to the limitations of the previous data collection tools, some GUIs have visual mismatches in the metadata and screenshots. Figure shows typical visual mismatch examples between hierarchy metadata and screenshots in current datasets. Based the bounding box coordinates of Android views provided in the metadata, depict the location of the views in the metadata as black dashed line in the screenshot. We mark the obvious visual mismatches with a solid red line box, which do not correspond any of the views in the rendered screenshot. The metadata provides information on the UI elements behind the current layer, but these elements cannot be interacted with on the current screenshot. Visual mismatches between the UI data in the metadata and the screenshot would result in the UI data in the metadata and the screenshot not corresponding one the other. Too many mismatch cases would have a negative impact on the efficiency of model generation and search. The selected UI collecting tool, Ulautomator2, has optimised the GUI caption technique to avoid metadata and screenshots from containing inconsistent UI information [51]. During manual pair reviews, our volunteers also eliminated GUI pages with mismatched. Compared to other datasets, such as rico, fewer mismatches give us a higher accuracy of our data.
Home
Zealand
World
Politics
Sport
visuel misuaces
Managed rstreat: The issue that comee with a polentially eyewstaring.. proeriag.
Travel
Lifestyle
Money
Figure 6: Examples of visual mismatches in current GUI datasets.
7

---

Preliminary Experiments
To demonstrate the usability of our dataset, we perform preliminary experiments on the dataset. Owing content limitations, we primarily focus on presenting the results derived from the GUI conversion experiment within the main body of the paper. We select current state-of art approaches for this task and use the automatic metrics for evaluation. We will discuss the qualitative and quantitative performance as well as the limitations of selected approaches on our proposed dataset.
In our supplementary appendix, have included more experimental results of diverse tasks. Addi- tionally, we provide a more detailed introduction of the employed methodologies, the metrics, and case studies for enhanced comprehension.
5.1 Selected Approaches
The GUI conversion task is to automatically convert an existing GUI to a new layout of GUI [ For example, given the present phone layout, generate tablet layout automatically. As far as we know, no researcher has proposed method for generating tablet GUI from a phone GUI, so we will use some existing relevant GUI generation methods to apply to this task. Three approaches. which are all widely used in GUT generation and introduced in Section 2 are selected in this task: Trnoorrrrn
[2
LayoutVAE
33 and VTN
5.2 Evaluation Metrics
For GUI conversion task, it's important to evaluate layouts in terms of two perspectives: perceptual quality and diversity. follow with the similar evaluation protocol in 66] 22 and utilize a set of metrics to evaluate the quality and diversity aspects. Specifically, we use the following metrics:
also known as the Jaceard index 233 is a method to quantify the percent overlap between the ground-trute and generated output.
Mean Intersection over Union (mIoU) 47
measures the overlap ratio between the ground-truth and our generated output. The overlap metric use the total overlapping area among any two bounding boxes inside the whole page and the average IoU between elements.
Overlap [54] 22]:
Wasserstein (W) distance 45) is the distance of the classes and bounding boxes to the real data. contains W class and W bbox metrics. Wasserstein distance is to evaluate the diversity between the real and generated data distributions.
Unique matches [46]: is the number of unique matchies according to the DocSim 46) It measures the matching overlap between real sets of layouts and generated layouts. designed for diversity evaluation.
Matched rate: To more directly show how many UI components in the ground truth's 's tablet GUI are successfully and automatically converted by the model, we select the metric: the matched rate. Suppose there total of m UI components in the ground truth, and there the generated tablet GUI that match the components in the ground truth, then the matched rate is calculated n/m.
components in
5.3 Experimental Setup
Since we are comparing the structure of the rendered GUT as opposed to its pixels, we do not consider the contents of images and texts inside the GUI. Therefore, we convert GUI pages to wireframes by converting each category of UI component to box of the specified color, which has been widely adopted in recently related works 22 38 35
In our GUI conversion work, unlike typical GUI generation tasks, we request the model to learn to generate a tablet GUI comparable to the input phone GUI, rather than another phone GUI. As input to the training model, we encode all UI components of phone GUI as component sequence from top to bottom and left to right. Each UI component in the GUT encoded as a quaternion (ar, y: w. lt). Where x and y denote the and y coordinates the upper left corner of this UI component, and w and ht denote the length and width of the component. Similarly, the tablet GUTs the pair are encoded into a sequence that serves as the ground truth for training and validation.

---

Following the setup of experiments in SayoutTransooorne 1000 pairs (10%) in the total dataset the test set and the rest of the data as the training set. All the results of the metrics are calculated on the test set. All of the results are based on 5-fold cross validation on the test set.
and LayoutVAE, we randomly select
5.4 Quantitative Evaluation
Table 2:
Automatic evaluation results on the test set.
Model
mloU 4
Overlap |
W class 4
W bbox 4
# Unique matchoes f
Matched rate
LayourVAF
0.10
0.23
0.29
0.012
356
0.13
LayourTrnnsfrmme
0.12
0.32
0.31
0.024
445
0.15
0.35
0.37
0,026
0.19
We present the quantitative evaluation results in Table 2 As expected, we observe that VTN model achieves the best performances in terms of all metrics. It yields large improvement regarding perceptual quality, such as W class, and Unique matches. Besides, obviously outperforms the LayoutVAE model across all metrics and is slightly better than LayoutTransforme model. The reason heavily rely on the mutual enhancement between self-attention and VAE mechanisms. However, according to the final metric Matched rate, only less than 20% of the UI components can be matched between the generated tablet GUT layouts and ground truth. Put succinetly, equivalent results can be attained by directly transferring certain phone GUI patterns the synthesized tablet GUI. The results suggest that further work is needed to design and train a more effective model in the GUT conversion task. We hope that our open-source dataset will enable more researchers to participate in automated UI development and contribute more effective methods.
5.5 Qualitative Evaluation
Inpet
Ground truth
Input
Ground truth
LayonrVAE
LayonrTransformn
LayourVAR
Layour Transformer
VIN
Generated GUIs
Generated GUls
Figure 7: Two examples of generated GUIs by selected approaches
To better understand the performance of different models on our task, we present qualitative results of selected models and their generated outputs in Figure 7 In alignment with the quantitative results in T we observe that the VTN model outperform other selected models. demonstrates the efficiency of self-attention mechanism on the layout generation task. However, we can observe that the best selected model VTN still fall short of generating precise margins and positions towards the ground-truth. There is still much room for improvement in learning the relationship between phone and tablet GUIs, and the current results are far from helpful to GUI developers. The GUI conversion in the crossing platform poses huge challenges for the existing of-th-----h--rrt model. Therefore, simply utilizing the previous layout generation model cannot tackle the challenges of GUI conversion in both Android phones and tablets.
5.6 Research Questions and Observations from Experiments
In the evaluations delineated Sections 5.4 and 5.5 becomes evident that contemporary models
fall short in facilitating effective automated GUI development. As illustrated in Table 2 even the

---

best-performing model, VTN, manages to accurately reproduce mere 20% of the UI components present the ground truth of the corresponding table. Notably, the left example in Figure 7 depicts scenario where our proposed method exhibits a relatively superior GUI generation. However, the instances on the right are notably subpar. We hypothesize that this disparity arises due to the intricate containment and nesting relationships among UI components in the right example, resulting in more convoluted metadata structure in comparison to the left. Consequently, discerning the relationships between UI components proves challenging for the model.
From our preliminary experiments and insights, we highlight key research queries that currently permeate the domain, aiming to advance the field:
GUI Encoding and Representation: Our paper delves into the intricate nature of GUL, complex data structure encompassing a wide variety of structural information and metadata. The methods of representing and encoding these GUT data instrumental in facilitating tasks such data dimensionality reduction and multimodal data fusion, potentially leading to significant enhancement in the generation effect. This complex encoding presents ripe area for further exploration and potential innovation.
Furthermore, the logical relationship between UI components, such as those images arranged side by side or images with texts below, provides an exciting avenue for research. Defining and modeling these relationships could pave the way for a new understanding of the interplay between UI components and could have broader applications in automated GUI development.
Modeling UI relationship:
Addressing Non-correspondence Contents: The fragmentation of Android screens poses challenges, particularly when tablet GUI in a phone-tablet pair houses more content than its phone counterpart. Navigating these content discrepancies presents a valuable research trajectory, with implications for enhancing cross-device GUI adaptability.
Conclusion
In this paper, we introduce the first pairwise GUT dataset Papt between Android phones and tablets and aims to bridge phone and tablet GUIs. We introduce the characteristics and preliminary experiments in this dataset. We discuss some valuable research problems based current experimental results and hope our dataset can facilitate the development of deep learning in automated GUI development.
References
[1] Adaptive Layout, 2023. http://seeeesssaddddoo//o//ooooe
[2] Andriod Fragment, 2022. http://deeebbprsraaoodoco////gnr
[3] Android Activity 2023. http://leebbarraaooooooooooomont
[4] Android Adb, 2023. https///eebbooc..aotoooo////oocr
[5] Android View, 2023. http://bhhhaiiaraainnggggggaaais
[6] Diego Martin Arroyo, Janis Postels, and Federico Tombari. Variational transformer networks for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13642-13652, 2021.
[7] BBC News, 2023. https///www.bbc.coomnes
[8] Miroslav Behan and Ondrej Krejcar. Adaptive graphical user interface solution for modern user devices. In Intelligent Information and Database Systems: 4th Asian Conference, ACIDS 2012, Kaohsiung, Taiwan, March 19-21, 2012, Proceedings, Part I1 4, pages 41 12-0 Springer, 2012.
[9] Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif EI-Nasr. Vins: Visual search for mobile user interface design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-14, 2021.
10

---

[10] Chunyang Chen, Ting Su, Guozhu Meng. Zhenchang Xing, and Yang Liu. From ui design image to gui skeleton: neural machine translator to bootstrap mobile gui implementation. In Proceedings of the 40th International Conference on Software Engineering, pages 665-676, 2018.
[11] Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. Gallery de: Design search and knowledge discovery through auto-created gui component gallery. Proceedings of the ACM on Human- Crmman-Computer Interaction, 3(CSCW):1-22, 2019.
[12] Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John Grundy, and Jinshui Wang. Wireframe- pereframe-based ui design search through image autoencoder. ACM Transactions on Software Engineering and Methodology (TOSEM), 29(3):1 2020.
[13] Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. Unblind your apps: Predicting natural-language labels for mobile gui components by deep learning. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 322-334, 2020.
[14] Qiuyuan Chen, Han Hu, and Zhaoyi Liu. Code summarization with abstract syntax tree. In Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part V 26, pages 652 660. Springer, 2019.
[15] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology, pages 845-854, 2017.
[16] Biplab Deka, Bardia Doosti, Forrest Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Ranjitha Kumar, Tao Dong, and Jeffrey Nichols. An early rico retrospective: Three years of uses for mobile app dataset. Artificial Intelligence for Human Computer Interaction: A Modern Approach, pages 229-256, 2021.
[17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019.
[18] Different Screen Support, 2023.
sureen//spoor--ifffeeettssreeoe
[19] Duolingo, 2023. http://www..duoingo.comm
[20] Sidong Feng, Suyu Ma, Jinzhong Yu, Chunyang Chen, Tingting Zhou, and Yankun Zhen. Auto-icon: An automated code generation tool for icon designs assisting in ui development. In 26th International Conference on Intelligent User Interfaces, pages 59 99, 2021.
[21] GUI State, 2023. http://dlelleegeeetooooooo///o/e
[22] Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layouttranffommer Layout generation and completion with self-attention. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1004- 1014, 2021.
[23] Lieve Hamers et al. Similarity measures in scientometric research: The jaccard index versus salton's cosine formula. Information Processing and Managemeni, 25(3):315-18, 1989.
[24] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98): 1735-1780, 1997.
[25] Cuixiong Hu and Iulian Neamtiu. Automating gui testing for android applications. In Pro- ceedings of the 6th International Workshop on Automation of Software Test, pages 77-83, 2011.
[26] Han Hu, Qiuyuan Chen, and Zhaoyi Liu. Code generation from supervised code embeddings. In Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part IV 26, pages 388-396. Springer, 2019.
11

---

[27] Han Hu, Ruiqi Dong, John Grundy, Thai Minh Nguyen, Huaxiao Liu, and Chunyang Chen. Automated mapping of adaptive upp guis from phones to tvs. ACM Trans. Softw. Eng. Methodol. 33(2), dec 2023. ISSN 1049-331X. doi: 10.1 14.4143663.... URI. https://doii org/10
1145/3631968
[28) Han Hu, Yujin Huang, Qiuyuan Chen, Terry Tue Zhuo, and Chunyang Chen. A first look at on-device models in ios apps. arXiv preprint arXiv:2307. 12322 2023.
[29] Han Hu, Yujin Huang, Qiuyuan Chen, Terry Yue Zhuo, and Chunyang Chen. A first look at on-device models in ios apps. ACM Trans. Softw: Eng. Methodol, 33(1), nov 2023. ISSN 1049-331X. doi: 10. 145/3617777 URt.tttts:///aoi org/10. 10. 1145/3617177
[30] Yujin Huang, Han Hu, and Chunyang Chen. Robustness of on-device models: Adversarial attack to deep learning models on android apps. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 101-110. IEEE, 2021.
[31] Yue Jiang, Wolfgang Stuerzlinger, Matthias Zwicker, and Christof Lutteroth. Oresolver: efficient solver for adaptive gui layout with or-constraints. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-14, 2020.
[32] Yue Jiang. Wolfgang Stuerzlinger, and Christof Lutteroth. Reverseorc: Reverse engineering of resizable user interface layouts with or-constraints. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-18, 2021.
[33] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from label set. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 9895-9904, 2019.
[34] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 131261131 2013.
[35] Chunggi Lee, Sanghoon Kim, Dongyun Han, Hongjun Yang, Young- Woo Park, Bum Chul Kwon, and Sungahn Ko. Guicomp: gui design assistant with real-time, multi-faceted feedback. In Proceedings of the 2020 CHll conference on human factors in computing systems, pages 1-13, factoractors
2020.
[36] Luis A Leiva, Asutosh Hota, and Antti Oulasvirta. Enrico: A dataset for topic modeling of mobile ui designs. 22nd International Conference on Human- Human-Computer Interaction with Mobile Devices and Services, pages 1-4, 2020.
[37] Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. Learning to denoise raw mobile ui layouts for improving datasets at scale. In Proceedings of the 2022 CHI Conference Human Factors in Computing Systems, pages 1-13, 2022.
[38] Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Gener- ating graphic layouts with wireframe discriminators. arXiv preprint 1000:v:1001.0600 2019.
[39] Suyu Ma, Chunyang Chen, Hourieh Khalajzadeh, and John Grundy. A first look at dark mode in real-world android apps. ACM Transactions on Software Engineering and Methodology, 2022.
[40] Aravind Machiry, Rohan Tahiliani, and Mayur Naik. Dynodroid: An input generation system for android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pages 224-234, 2013.
[41] Rabiya Majeed-Ariss, Eileen Baildam, Malcolm Campbell, Alice Chieng, Debbie Fallon, Andrew Hall, Janet E McDonagh, Simon R Stones, Wendy Thomson, and Veronica Swallow. Apps and adolescents: systematic review of adolescents' use of mobile phone and tablet apps that support personal management of their chronic long-term physical conditions. Journal of medical Internet research, 17(1))x287,, 2015.
[42] Atif M. Memon. Gui testing: Pitfalls and process. Computer, 35(08))87-88, 2002.
12

---

[43] Kevin Moran, C Ceraal-Crrdenss M Curcio. R Bonett, and D Poshyvanyk. The redraw dataset: set of android screenshots, gui metadata, and labeled images of gui components, 2019.
[44] Antti Oulasvirta, Niraj Ramesh Dayama, Morteza Shiripour, Maximilian John, and Andreas Karrenbauer. Combinatorial optimization of graphical user interface designs. Proceedings of the IEEE, 108(3):434 464, 2020.
[45] Victor M Panaretos and Yoav Zemel. Statistical aspects wasserstein distances. Annual review of statistics and its application. 6:405 431,, 2019.
[46] Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar Averbuch-Elor. Read: Recursive autoencoders for document layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 544-545, 2020.
[47] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, lan Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 658-666, 2019.
[48] Spotify, 2023. http://eewwiiipppiiioggwwwiiisst
[49] Tablet Share, 2023.
[50] Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models, arXiv preprint arXiv:151100184. 2015.
[51] Uiautomator2, 2023. https:/ppay.google..omm/.tt
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017.
[53] Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium User Interface Software and Technology, pages 498-510, 2021.
[54] Varduhi Yeghiazaryan and Irina Voiculescu. Family of boundary overlap metries for the evaluation of medical image segmentation. Journal of Medical Imaging, 5(1)0050000015006 2018.
[55] You'll YouTube, 2023.
[56] Haolan Zhan, Lei Shen, Hongshen Chen, and Hainan Zhang. CoLV: A collaborative latent variable model for knowledge- grounded dialogue generation. In EMNLP, pages 2250-2261 Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
[57] Yanjie Zhao, Li Li, Xiaoyu Sun, Pei Liu, and John Grundy. Code implementation recom- mendation for android gui components. In Proceedingy of the ACM/IEEF 44th International Conference on Software Engineering: Companion Proceedings, pages 31-35, 2022.
13


Reviews:
[{"review_id": "ABjXFuqFDE", "reviewer": "Reviewer_abm3", "summary": "", "questions": "", "limitations": "The authors are direct about the their trained models being insufficient to be practical.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper has a clear value proposition: a dataset to train models that automate generation of a table UI from phone UI, and vice versa. This is a practical and valuable problem. \n\nThe dataset construction is clever and reasonably rigorous, including manual review. The dataset is sensibly structured, large, and diverse (across types of app).\n\nThe paper is well structure: problem, related work, author's dataset, analysis of dataset, training of models on dataset, analysis of results on models. It flows and reads well.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "OnRfNLw0R5", "reviewer": "Reviewer_s1Us", "summary": "", "questions": "", "limitations": "There is no limitation section.\n\nThe authors are encouraged to consider my comments in the \"Opportunities for Improvement\".", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This is the first GUI dataset that pairs the UI between the phone and tablet app.\n\n2. The dataset is carefully cleansed to avoid errors.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "DyhBiMX8Tg", "reviewer": "Reviewer_BuL6", "summary": "", "questions": "", "limitations": "1. The previous dataset was for only one platform and the authors collected it for two platforms. Apart from that they got it validated via a set of application developers. Though they don't report how many samples were discarded after validation", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The dataset is large\n2. The authors have manually verified this dataset by a set of application developers\n3. They discuss why prior dataset might have erroneous data", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "BfPrN3cNEq", "reviewer": "Reviewer_ZrJu", "summary": "", "questions": "", "limitations": "1. The proposed dataset can somehow address the limitations in the GUI related problems, but it only contains a small number of samples under limited scope and system.\n2. The experimental results are expected to provide more insights, for example, what is the major research question in this research field?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The pairwise GUI dataset between phones and tablets is interesting and critical.\n2. The authors also provide the way to collect more data.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All reviewers are excited about this dataset, while have concerns about the size of the 
data set and that it only focuses on Android applications. Balancing pros and cons, we
recommend accept of this work, while encouraging the authors to continue enrich both
the size and the diversity of this dataset.
Decision: Accept (Poster)