SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data Accuracy Specifically, our data eliminates a large number of GUI visual mismatches that are frequent in current datasets like as Rico and Enrico. Due to the limitations of the previous data collection tools, some GUIs have visual mismatches in the metadata and screenshots. Figure 6 shows typical visual mismatch examples between hierarchy metadata and screenshots in current datasets. Based on the bounding box coordinates of Android views provided in the metadata, we depict the location of the views in the metadata as a black dashed line in the screenshot. We mark the obvious visual mismatches with a solid red line box, which do not correspond to any of the views in the rendered screenshot. The metadata provides information on the UI elements behind the current layer, but these elements cannot be interacted with on the current screenshot. Visual mismatches between the UI data in the metadata and the screenshot would result in the UI data in the metadata and the screenshot not corresponding one to the other. Too many mismatch cases would have a negative impact on the efficiency of model generation and search. The selected UI collecting tool, UIautomator2, has optimised the GUI caption technique to avoid metadata and screenshots from containing inconsistent UI information [51]. During manual pair reviews, our volunteers also eliminated GUI pages with mismatched. Compared to other datasets, such as rico, fewer mismatches give us a higher accuracy of our data.
Figure 6: Examples of visual mismatches in current GUI datasets.
5 Preliminary Experiments
To demonstrate the usability of our dataset, we perform preliminary experiments on the dataset. Owing to content limitations, we primarily focus on presenting the results derived from the GUI conversion experiment within the main body of the paper. We select current state-of-art approaches for this task and use the automatic metrics for evaluation. We will discuss the qualitative and quantitative performance as well as the limitations of selected approaches on our proposed dataset.
In our supplementary appendix, we have included more experimental results of diverse tasks. Additionally, we provide a more detailed introduction of the employed methodologies, the metrics, and case studies for enhanced comprehension.
5.1 Selected Approaches
The GUI conversion task is to automatically convert an existing GUI to a new layout of GUI [8]. For example, given the present phone layout, generate a tablet layout automatically. As far as we know, no researcher has proposed a method for generating a tablet GUI from a phone GUI, so we will use some existing relevant GUI generation methods to apply to this task. Three approaches, which are all widely used in GUI generation and introduced in Section 2, are selected in this task: LayoutTransformer [22], LayoutVAE [33] and VTN [6].
For GUI conversion task, it's important to evaluate layouts in terms of two perspectives: perceptual quality and diversity. We follow with the similar evaluation protocol in [6, 22, 9] and utilize a set of metrics to evaluate the quality and diversity aspects. Specifically, we use the following metrics:
Mean Intersection over Union (mIoU) [47]: also known as the Jaccard index [23], is a method to quantify the percent overlap between the ground-truth and generated output.
Overlap [54, 22]: measures the overlap ratio between the ground-truth and our generated output. The overlap metric use the total overlapping area among any two bounding boxes inside the whole page and the average IoU between elements.
Wasserstein (W) distance [45]: is the distance of the classes and bounding boxes to the real data. It contains W class and W bbox metrics. Wasserstein distance is to evaluate the diversity between the real and generated data distributions.
Unique matches [46]: is the number of unique matchies according to the DocSim [46]. It measures the matching overlap between real sets of layouts and generated layouts. It is designed for diversity evaluation.
Matched rate: To more directly show how many UI components in the ground truth's tablet GUI are successfully and automatically converted by the model, we select the metric: the matched rate. Suppose there are a total of  $m$  UI components in the ground truth, and there are  $n$  components in the generated tablet GUI that match the components in the ground truth, then the matched rate is calculated as  $n / m$ .
Since we are comparing the structure of the rendered GUI as opposed to its pixels, we do not consider the contents of images and texts inside the GUI. Therefore, we convert GUI pages to wireframes by converting each category of UI component to a box of the specified color, which has been widely adopted in recently related works [22, 38, 33]. In our GUI conversion work, unlike typical GUI generation tasks, we request the model to learn to generate a tablet GUI comparable to the input phone GUI, rather than another phone GUI. As input to the training model, we encode all UI components of a phone GUI as a component sequence from top to bottom and left to right. Each UI component in the GUI is encoded as a quaternion  $(x,y,w,h)$ . Where  $x$  and  $y$  denote the  $x$  and  $y$  coordinates of the upper left corner of this UI component, and  $w$  and  $h$  denote the length and width of the component. Similarly, the tablet GUIs in the pair are encoded into a sequence that serves as the ground truth for training and validation.
Following the setup of experiments in LayoutTransformer and LayoutVAE, we randomly select 1000 pairs  $(10\%)$  in the total dataset as the test set and the rest of the data as the training set. All the results of the metrics are calculated on the test set. All of the results are based on 5-fold cross validation on the test set.
5.4 Quantitative Evaluation
Table 2: Automatic evaluation results on the test set.
Model	mIoU ↓	Overlap ↓	W class ↓	Wbbox ↓	# Unique matchces ↑	Matched rate ↓
LayoutVAE	0.10	0.23	0.29	0.012	356	0.13
LayoutTransformer	0.12	0.32	0.31	0.024	445	0.15
VTN	0.13	0.35	0.37	0.026	541	0.19
We present the quantitative evaluation results in Table 2. As expected, we observe that VTN model achieves the best performances in terms of all metrics. It yields large improvement regarding perceptual quality, such as W class, and Unique matches. Besides, it obviously outperforms the LayoutVAE model across all metrics and is slightly better than LayoutTransformer model. The reason heavily relies on the mutual enhancement between self-attention and VAE mechanisms. However, according to the final metric Matched rate, only less than  $20\%$  of the UI components can be matched between the generated tablet GUI layouts and ground truth. Put succinctly, equivalent results can be attained by directly transferring certain phone GUI patterns to the synthesized tablet GUI. The results suggest that further work is needed to design and train a more effective model in the GUI conversion task. We hope that our open-source dataset will enable more researchers to participate in automated UI development and contribute more effective methods.
5.5 Qualitative Evaluation
Figure 7: Two examples of generated GUIs by selected approaches
To better understand the performance of different models on our task, we present qualitative results of selected models and their generated outputs in Figure 7. In alignment with the quantitative results in Table 2, we observe that the VTN model outperform other selected models. It demonstrates the efficiency of self-attention mechanism on the layout generation task. However, we can observe that the best selected model VTN still fall short of generating precise margins and positions towards the ground-truth. There is still much room for improvement in learning the relationship between phone and tablet GUIs, and the current results are far from helpful to GUI developers. The GUI conversion in the crossing platform poses huge challenges for the existing state-of-the-art model. Therefore, simply utilizing the previous layout generation model cannot tackle the challenges of GUI conversion in both Android phones and tablets.
5.6 Research Questions and Observations from Experiments
In the evaluations delineated in Sections 5.4 and 5.5, it becomes evident that contemporary models fall short in facilitating effective automated GUI development. As illustrated in Table 2, even the
best-performing model, VTN, manages to accurately reproduce a mere  $20\%$  of the UI components present in the ground truth of the corresponding table. Notably, the left example in Figure 7 depicts a scenario where our proposed method exhibits a relatively superior GUI generation. However, the instances on the right are notably subpar. We hypothesize that this disparity arises due to the intricate containment and nesting relationships among UI components in the right example, resulting in a more convoluted metadata structure in comparison to the left. Consequently, discerning the relationships between UI components proves challenging for the model.
From our preliminary experiments and insights, we highlight key research queries that currently permeate the domain, aiming to advance the field:
GUI Encoding and Representation: Our paper delves into the intricate nature of GUI, a complex data structure encompassing a wide variety of structural information and metadata. The methods of representing and encoding these GUI data are instrumental in facilitating tasks such as data dimensionality reduction and multimodal data fusion, potentially leading to a significant enhancement in the generation effect. This complex encoding presents a ripe area for further exploration and potential innovation.
Modeling UI relationship: Furthermore, the logical relationship between UI components, such as those images arranged side by side or images with texts below, provides an exciting avenue for research. Defining and modeling these relationships could pave the way for a new understanding of the interplay between UI components and could have broader applications in automated GUI development.
Addressing Non-correspondence Contents: The fragmentation of Android screens poses challenges, particularly when a tablet GUI in a phone-tablet pair houses more content than its phone counterpart. Navigating these content discrepancies presents a valuable research trajectory, with implications for enhancing cross-device GUI adaptability.
In this paper, we introduce the first pairwise GUI dataset Papt between Android phones and tablets and aims to bridge phone and tablet GUIs. We introduce the characteristics and preliminary experiments in this dataset. We discuss some valuable research problems based on current experimental results and hope our dataset can facilitate the development of deep learning in automated GUI development.
[1] Adaptive Layout, 2023. https://developer.android.com/jetpack/compose/layout/adaptive.
[2] Android Fragment, 2022. https://developer.android.com/guide/fragments.
[3] Android Activity, 2023. https://developer.android.com/reference/android/app/Activity.
[4] Android Adb, 2023. https://developer.android.com/studio/command-line/adb.
[5] Android View, 2023. https://data-flair_training/blogs/android layout-and-views/.
[6] Diego Martin Arroyo, Janis Postels, and Federico Tombari. Variational transformer networks for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13642-13652, 2021.
[7] BBC News, 2023. https://www.bbc.com/news.
[8] Miroslav Behan and Ondrej Krejcar. Adaptive graphical user interface solution for modern user devices. In Intelligent Information and Database Systems: 4th Asian Conference, ACIDS 2012, Kaohsiung, Taiwan, March 19-21, 2012, Proceedings, Part II 4, pages 411-420. Springer, 2012.
[9] Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr. Vins: Visual search for mobile user interface design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-14, 2021.
[43] Kevin Moran, C Bernal-Cardenas, M Curcio, R Bonett, and D Poshyvanyk. The redraw dataset: A set of android screenshots, gui metadata, and labeled images of gui components, 2019.
[44] Antti Oulasvirta, Niraj Ramesh Dayama, Morteza Shiripour, Maximilian John, and Andreas Karrenbauer. Combinatorial optimization of graphical user interface designs. Proceedings of the IEEE, 108(3):434-464, 2020.
[45] Victor M Panaretos and Yoav Zemel. Statistical aspects of Wasserstein distances. Annual review of statistics and its application, 6:405-431, 2019.
[46] Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar Averbuch-Elor. Read: Recursive autoencoders for document layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 544–545, 2020.
[47] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658-666, 2019.
[48] Spotify, 2023. https://en.wikipedia.org/wiki/Spotify.
[49] Tablet Share, 2023. https://gs.statcounter.com/platform-market-share/Desktop-mobile-tablet.
[50] Lucas Theis, Aaron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844, 2015.
[51] Ui automator2, 2023. https://play.google.com/store.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[53] Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, pages 498-510, 2021.
[54] Varduhi Yeghiazaryan and Irina Voiculescu. Family of boundary overlap metrics for the evaluation of medical image segmentation. Journal of Medical Imaging, 5(1):015006-015006, 2018.
[55] YouTube, 2023. https://en.wikipedia.org/wiki/YouTube.
[56] Haolan Zhan, Lei Shen, Hongshen Chen, and Hainan Zhang. CoLV: A collaborative latent variable model for knowledge-grounded dialogue generation. In EMNLP, pages 2250–2261, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
[57] Yanjie Zhao, Li Li, Xiaoyu Sun, Pei Liu, and John Grundy. Code implementation recommendation for android gui components. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pages 31-35, 2022.