SlowGuess/ACL_2020_OCR huggingface dataset:
Contextual Representation of UI Objects. To compute latent representations of each candidate object,  $c_{j,k}^{\prime}$ , we use both the object's properties and its context, i.e., the structural relationship with other objects on the screen. There are different ways for encoding a variable-sized collection of items that are structurally related to each other, including Graph Convolutional Networks (GCN) (Niepert et al., 2016) and Transformers (Vaswani et al., 2017). GCNs use an adjacency matrix predetermined by the UI structure to regulate how the latent representation of an object should be affected by its neighbors. Transformers allow each object to carry its own positional encoding, and the relationship between objects can be learned instead.
The input to the Transformer encoder is a combination of the content embedding and the positional encoding of each object. The content properties of an object include its name and type. We compute the content embedding of by concatenating the name embedding, which is the average embedding of the bag of tokens in the object name, and the
8203
type embedding. The positional properties of an object include both its spatial position and structural position. The spatial positions include the top, left, right and bottom screen coordinates of the object. We treat each of these coordinates as a discrete value and represent it via an embedding. Such a feature representation for coordinates was used in ImageTransformer to represent pixel positions in an image (Parmar et al., 2018). The spatial embedding of the object is the sum of these four coordinate embeddings. To encode structural information, we use the index positions of the object in the preorder and the postorder traversal of the view hierarchy tree, and represent these index positions as embeddings in a similar way as representing coordinates. The content embedding is then summed with positional encodings to form the embedding of each object. We then feed these object embeddings into a Transformer encoder model to compute the latent representation of each object,  $c_{j,k}^{\prime}$ .
The grounding model is trained by minimizing the cross entropy loss between the predicted and ground-truth object and the loss between the predicted and ground-truth operation.
Our goal is to develop models and datasets to map multi-step instructions into automatically executable actions given the screen information. As such, we use PILXELHELP's paired natural instructions and action-screen sequences solely for testing. In addition, we investigate the model quality on phrase tuple extraction tasks, which is a crucial building block for the overall grounding quality<sup>4</sup>.
We use two metrics that measure how a predicted tuple sequence matches the ground-truth sequence.
- Complete Match: The score is 1 if two sequences have the same length and have the identical tuple  $[\hat{r}_j, \hat{o}_j, \hat{u}_j]$  at each step, otherwise 0.
- Partial Match: The number of steps of the predicted sequence that match the ground-truth sequence divided by the length of the ground-truth sequence (ranging between 0 and 1).
We train and validate using ANDROIDHOWTo and RICOSCA, and evaluate on PixelHELP. During training, single-step synthetic command-action
Span Rep. hb:d	Partial	Complete
SumPooling ∑k=b dh	92.80	85.56
StartEnd [hb; hd]	91.94	84.56
[hb; hd,ˆb:d,φ(d-b)]	91.11	84.33
Table 1: ANDROIDHOWTo phrase tuple extraction test results using different span representations  $h^{b:d}$  in (8).  $\hat{e}^{b:d} = \sum_{k=b}^{d} w(h_k)e(t_k)$ , where  $w(\cdot)$  is a learned weight function for each token embedding (Lee et al., 2017). See the pseudocode for fast computation of these in the appendix.
examples are dynamically stitched to form sequence examples with a certain length distribution. To evaluate the full task, we use Complete and Partial Match on grounded action sequences  $a_{1:m}$  where  $a_{j} = [r_{j},o_{j},u_{j}]$ .
The token vocabulary size is 59K, which is compiled from both the instruction corpus and the UI name corpus. There are 15 UI types, including 14 common UI object types, and a type to catch all less common ones. The output vocabulary for operations include CLICK, TEXT, SWIPE and EOS.
5.2 Model Configurations and Results
Tuple Extraction. For the action-tuple extraction task, we use a 6-layer Transformer for both the encoder and the decoder. We evaluate three different span representations. Area Attention (Li et al., 2019) provides a parameter-free representation of each possible span (one-dimensional area), by summing up the encoding of each token in the subsequence:  $h^{b:d} = \sum_{k = b}^{d}h_{k}$ . The representation of each span can be computed in constant time invariant to the length of the span, using a summed area table. Previous work concatenated the encoding of the start and end tokens as the span representation,  $h^{b:d} = [h_b;h_d]$  (Lee et al., 2016) and a generalized version of it (Lee et al., 2017). We evaluated these three options and implemented the representation in Lee et al. (2017) using a summed area table similar to the approach in area attention for fast computation. For hyperparameter tuning and training details, refer to the appendix.
Table 1 gives results on ANDROIDHOWTo's test set. All the span representations perform well. Encodings of each token from a Transformer already capture sufficient information about the entire sequence, so even only using the start and end encodings yields strong results. Nonetheless, area attention provides a small boost over the others. As a new dataset, there is also considerable headroom remaining, particularly for complete match.
4Our model code is released at https : // github . com / google-research / google-research / tree/master/seq2act.
8204
Screen Encoder	Partial	Complete
Heuristic	62.44	42.25
Filter-1 GCN	76.44	52.41
Distance GCN	82.50	59.36
Transformer	89.21	70.59
Table 2: PixelHelp grounding accuracy. The differences are statistically significant based on t-test over 5 runs  $(p < 0.05)$ .
Grounding. For the grounding task, we compare Transformer-based screen encoder for generating object representations  $h^{b:d}$  with two baseline methods based on graph convolutional networks. The Heuristic baseline matches extracted phrases against object names directly using BLEU scores. Filter-1 GCN performs graph convolution without using adjacent nodes (objects), so the representation of each object is computed only based on its own properties. Distance GCN uses the distance between objects in the view hierarchy, i.e., the number of edges to traverse from one object to another following the tree structure. This contrasts with the traditional GCN definition based on adjacency, but is needed because UI objects are often leaves in the tree; as such, they are not adjacent to each other structurally but instead are connected through non-terminal (container) nodes. Both Filter-1 GCN and Distance GCN use the same number of parameters (see the appendix for details).
To train the grounding model, we first train the Tuple Extraction sub-model on AndroidHOWTo and RICOsCA. For the latter, only language related features (commands and tuple positions in the command) are used in this stage, so screen and action features are not involved. We then freeze the Tuple Extraction sub-model and train the grounding sub-model on RICOsCA using both the command and screen-action related features. The screen token embeddings of the grounding sub-model share weights with the Tuple Extraction sub-model.
Table 2 gives full task performance on PixelHELP. The Transformer screen encoder achieves the best result with  $70.59\%$  accuracy on Complete Match and  $89.21\%$  on Partial Match, which sets a strong baseline result for this new dataset while leaving considerable headroom. The GCN-based methods perform poorly, which shows the importance of contextual encodings of the information from other UI objects on the screen. Distance GCN does attempt to capture context for UI objects that
are structurally close; however, we suspect that the distance information that is derived from the view hierarchy tree is noisy because UI developers can construct the structure differently for the same UI. As a result, the strong bias introduced by the structure distance does not always help. Nevertheless, these models still outperformed the heuristic baseline that achieved  $62.44\%$  for partial match and  $42.25\%$  for complete match.
To explore how the model grounds an instruction on a screen, we analyze the relationship between words in the instruction language that refer to specific locations on the screen, and actual positions on the UI screen. We first extract the embedding weights from the trained phrase extraction model for words such as top, bottom, left and right. These words occur in object descriptions such as the check box at the top of the screen. We also extract the embedding weights of object screen positions, which are used to create object positional encoding. We then calculate the correlation between word embedding and screen position embedding using cosine similarity. Figure 5 visualizes the correlation as a heatmap, where brighter colors indicate higher correlation. The word top is strongly correlated with the top of the screen, but the trend for other location words is less clear. While left is strongly correlated with the left side of the screen, other regions on the screen also show high correlation. This is likely because left and right are not only used for referring to absolute locations on the screen, but also for relative spatial relationships, such as the icon to the left of the button. For bottom, the strongest correlation does not occur at the very bottom of the screen because many UI objects in our dataset do not fall in that region. The region is often reserved for system actions and the on-screen keyboard, which are not covered in our dataset.
The phrase extraction model passes phrase tuples to the grounding model. When phrase extraction is incorrect, it can be difficult for the grounding model to predict a correct action. One way to mitigate such cascading errors is using the hidden state of the phrase decoding model at each step,  $q_{j}$ . Intuitively,  $q_{j}$  is computed with the access to the encoding of each token in the instruction via the Transformer encoder-decoder attention, which can
While it is possible to directly use screen visual data for grounding, detecting UI objects from raw pixels is nontrivial. It would be ideal to use both structural and visual data.
8205
Figure 5: Correlation between location-related words in instructions and object screen position embedding.
potentially be a more robust span representation. However, in our early exploration, we found that grounding with  $q_{j}$  performs stunningly well for grounding RicosCA validation examples, but performs poorly on PixelHelp. The learned hidden state likely captures characteristics in the synthetic instructions and action sequences that do not manifest in PixelHelp. As such, using the hidden state to ground remains a challenge when learning from unpaired instruction-action data.
The phrase model failed to extract correct steps for 14 tasks in PILxELHELP. In particular, it resulted in extra steps for 11 tasks and extracted incorrect steps for 3 tasks, but did not skip steps for any tasks. These errors could be caused by different language styles manifested by the three datasets. Synthesized commands in RICOscA tend to be brief. Instructions in ANDROIDHOWTo seem to give more contextual description and involve diverse language styles, while PILxELHELP often has a more consistent language style and gives concise description for each step.
Previous work (Branavan et al., 2009, 2010; Liu et al., 2018; Gur et al., 2019) investigated approaches for grounding natural language on desktop or web interfaces. Manuvinakurike et al. (2018) contributed a dataset for mapping natural language instructions to actionable image editing commands in Adobe Photoshop. Our work focuses on a new domain of grounding natural language instructions into executable actions on mobile user interfaces. This requires addressing modeling challenges due to the lack of paired natural language and action data, which we supply by harvesting rich instruction data from the web and synthesizing UI commands based on a large scale Android corpus.
Our work is related to semantic parsing, particularly efforts for generating executable outputs such
as SQL queries (Suhr et al., 2018). It is also broadly related to language grounding in the human-robot interaction literature where human dialog results in robot actions (Khayrallah et al., 2015).
Our task setting is closely related to work on language-conditioned navigation, where an agent executes an instruction as a sequence of movements (Chen and Mooney, 2011; Mei et al., 2016; Misra et al., 2017; Anderson et al., 2018; Chen et al., 2019). Operating user interfaces is similar to navigating the physical world in many ways. A mobile platform consists of millions of apps that each is implemented by different developers independently. Though platforms such as Android strive to achieve interoperability (e.g., using Intent or AIDL mechanisms), apps are more often than not built by convention and do not expose programmatic ways for communication. As such, each app is opaque to the outside world and the only way to manipulate it is through its GUIs. These hurdles while working with a vast array of existing apps are like physical obstacles that cannot be ignored and must be negotiated contextually in their given environment.
Our work provides an important first step on the challenging problem of grounding natural language instructions to mobile UI actions. Our decomposition of the problem means that progress on either can improve full task performance. For example, action span extraction is related to both semantic role labeling (He et al., 2018) and extraction of multiple facts from text (Jiang et al., 2019) and could benefit from innovations in span identification and multitask learning. Reinforcement learning that has been applied in previous grounding work may help improve out-of-sample prediction for grounding in UIs and improve direct grounding from hidden state representations. Lastly, our work provides a technical foundation for investigating user experiences in language-based human computer interaction.
We would like to thank our anonymous reviewers for their insightful comments that improved the paper. Many thanks to the Google Data Compute team, especially Ashwin Kakarla and Muqthar Mohammad for their help with the annotations, and Song Wang, Justin Cui and Christina Ou for their help on early data preprocessing.
8206
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
S. R. K. Branavan, Harr Chen, Luke S. Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1, ACL '09, pages 82-90, Stroudsburg, PA, USA. Association for Computational Linguistics.
S.R.K. Branavan, Luke Zettlemoyer, and Regina Barzilay. 2010. Reading between the lines: Learning to map high-level instructions to commands. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1268-1277, Uppsala, Sweden. Association for Computational Linguistics.
David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI'11, pages 859-865. AAAI Press.
Howard Chen, Alane Suhr, Dipendra Misra, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Conference on Computer Vision and Pattern Recognition.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual Symposium on User Interface Software and Technology, UIST '17.
Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur. 2019. Learning to navigate the web. In International Conference on Learning Representations.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 364-369, Melbourne, Australia. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Tianwen Jiang, Tong Zhao, Bing Qin, Ting Liu, Nitesh Chawla, and Meng Jiang. 2019. Multi-input multi-output sequence labeling for joint extraction of fact and condition tuples from scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 302-312, Hong Kong, China. Association for Computational Linguistics.
Huda Khayrallah, Sean Trott, and Jerome Feldman. 2015. Natural language for human robot interaction.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Kenton Lee, Tom Kwiatkowski, Ankur P. Parikh, and Dipanjan Das. 2016. Learning recurrent span representations for extractive question answering. CoRR, abs/1611.01436.
Yang Li, Lukasz Kaiser, Samy Bengio, and Si Si. 2019. Area attention. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3846-3855, Long Beach, California, USA. PMLR.
E. Z. Liu, K. Guu, P. Pasupat, T. Shi, and P. Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR).
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Ramesh Manuvinakurike, Jacqueline Brixey, Trung Bui, Walter Chang, Doo Soon Kim, Ron Artstein, and Kallirroi Georgila. 2018. Edit me: A corpus and a framework for understanding natural language image editing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA).
Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pages 2772-2778. AAAI Press.
8207
Dipendra Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1004-1015.
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. 2016. Learning convolutional neural networks for graphs. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2014-2023, New York, New York, USA. PMLR.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4055-4064, Stockholm, Sweden. PMLR.
Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora.
Zhanna Sarsenbayeva. 2018. Situational impairments during mobile interaction. In Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, pages 498-503.
Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to executable formal queries. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2238-2249, New Orleans, Louisiana. Association for Computational Linguistics.
Richard Szeliski. 2010. Computer Vision: Algorithms and Applications, 1st edition. Springer-Verlag, Berlin, Heidelberg.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692-2700. Curran Associates, Inc.
We present the additional details and analysis of the datasets. To label action phrase spans for the ANDROIDHOWTo dataset, 21 annotators (9 males and 12 females, 23 to 28 years old) were employed as contractors. They were paid hourly wages that
are competitive for their locale. They have standard rights as contractors. They were native English speakers, and rated themselves 4 out of 5 regarding their familiarity with Android (1: not familiar and 5: very familiar).
Each annotator is presented a web interface, depicted in Figure 6. The instruction to be labeled is shown on the left of the interface. From the instruction, the annotator is asked to extract a sequence of action phrase tuples on the right, providing one tuple per row. Before a labeling session, an annotator is asked to go through the annotation guidelines, which are also accessible throughout the session.
To label each tuple, the annotator first indicates the type of operation (Action Type) the step is about by selecting from Click, Swipe, Input and Others (the catch-all category). The annotator then uses a mouse to select the phrase in the instruction for "Action Verb" (i.e., operation description) and for "object description". A selected phrase span is automatically shown in the corresponding box and the span positions in the instruction are recorded. If the step involves an additional argument, the annotator clicks on "Content Input" and then marks a phrase span in the instruction (see the second row). Once finished with creating a tuple, the annotator moves onto the next tuple by clicking the "+" button on the far right of the interface along the row, which inserts an empty tuple after the row. The annotator can delete a tuple (row) by clicking the "-" button on the row. Finally, the annotator clicks on the "Submit" button at the bottom of the screen to finish a session.
The lengths of the instructions range from 19 to 85 tokens, with median of 59, and they describe a sequence of actions from 1 to 19 steps, with a median of 5. Although the description for operations tend to be short (most of them are one to two words), the description for objects can vary dramatically in length, ranging from 1 to 19. The large range of description span lengths requires an efficient algorithm to compute its representation.
B Computing Span Representations