SlowGuess/ACL_2020_OCR huggingface dataset:
Yang Li Jiacong He Xin Zhou Yuan Zhang Jason Baldridge
Google Research, Mountain View, CA, 94043
{liyang,zhouxin,zhangyua,jasonbaldridge}@google.com
We present a new problem: grounding natural language instructions to mobile user interface actions, and create three new datasets for it. For full task evaluation, we create PixelHELP, a corpus that pairs English instructions with actions performed by people on a mobile UI emulator. To scale training, we decouple the language and action data by (a) annotating action phrase spans in HowTo instructions and (b) synthesizing grounded descriptions of actions for mobile user interfaces. We use a Transformer to extract action phrase tuples from long-range natural language instructions. A grounding Transformer then contextually represents UI objects using both their content and screen position and connects them to object descriptions. Given a starting screen and instruction, our model achieves  $70.59\%$  accuracy on predicting complete ground-truth action sequences in PixelHELP.
Language helps us work together to get things done. People instruct one another to coordinate joint efforts and accomplish tasks involving complex sequences of actions. This takes advantage of the abilities of different members of a speech community, e.g. a child asking a parent for a cup she cannot reach, or a visually impaired individual asking for assistance from a friend. Building computational agents able to help in such interactions is an important goal that requires true language grounding in environments where action matters.
An important area of language grounding involves tasks like completion of multi-step actions in a graphical user interface conditioned on language instructions (Branavan et al., 2009, 2010; Liu et al., 2018; Gur et al., 2019). These domains matter for accessibility, where language interfaces could help visually impaired individuals perform tasks with
Figure 1: Our model extracts the phrase tuple that describe each action, including its operation, object and additional arguments, and grounds these tuples as executable action sequences in the UI.
interfaces that are predicated on sight. This also matters for situational impairment (Sarsenbayeva, 2018) when one cannot access a device easily while encumbered by other factors, such as cooking.
We focus on a new domain of task automation in which natural language instructions must be interpreted as a sequence of actions on a mobile touch-screen UI. Existing web search is quite capable of retrieving multi-step natural language instructions for user queries, such as "How to turn on flight mode on Android." Crucially, the missing piece for fulfilling the task automatically is to map the returned instruction to a sequence of actions that can be automatically executed on the device with
8198
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8198-8210 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
little user intervention; this our goal in this paper. This task automation scenario does not require a user to maneuver through UI details, which is useful for average users and is especially valuable for visually or situationally impaired users. The ability to execute an instruction can also be useful for other scenarios such as automatically examining the quality of an instruction.
Our approach (Figure 1) decomposes the problem into an action phrase-extraction step and a grounding step. The former extracts operation, object and argument descriptions from multi-step instructions; for this, we use Transformers (Vaswani et al., 2017) and test three span representations. The latter matches extracted operation and object descriptions with a UI object on a screen; for this, we use a Transformer that contextually represents UI objects and grounds object descriptions to them.
We construct three new datasets<sup>1</sup>. To assess full task performance on naturally occurring instructions, we create a dataset of 187 multi-step English instructions for operating Pixel Phones and produce their corresponding action-screen sequences using annotators. For action phrase extraction training and evaluation, we obtain English How-To instructions from the web and annotate action description spans. A Transformer with spans represented by sum pooling (Li et al., 2019) obtains  $85.56\%$  accuracy for predicting span sequences that completely match the ground truth. To train the grounding model, we synthetically generate 295k single-step commands to UI actions, covering 178K different UI objects across 25K mobile UI screens.
Our phrase extractor and grounding model together obtain  $89.21\%$  partial and  $70.59\%$  complete accuracy for matching ground-truth action sequences on this challenging task. We also evaluate alternative methods and representations of objects and spans and present qualitative analyses to provide insights into the problem and models.
Given an instruction of a multi-step task,  $I = t_{1:n} = (t_1, t_2, \dots, t_n)$ , where  $t_i$  is the  $i$ th token in instruction  $I$ , we want to generate a sequence of automatically executable actions,  $a_{1:m}$ , over a sequence of user interface screens  $S$ , with initial screen  $s_1$
and screen transition function  $s_j = \tau(a_{j-1}, s_{j-1})$ :
$$
p \left(a _ {1: m} \mid s _ {1}, \tau , t _ {1: n}\right) = \prod_ {j = 1} ^ {m} p \left(a _ {j} \mid a _ {<   j}, s _ {1}, \tau , t _ {1: n}\right) \tag {1}
$$
An action  $a_{j} = [r_{j},o_{j},u_{j}]$  consists of an operation  $r_j$  (e.g. Tap or Text), the UI object  $o_j$  that  $r_j$  is performed on (e.g., a button or an icon), and an additional argument  $u_{j}$  needed for  $o_j$  (e.g. the message entered in the chat box for Text or null for operations such as Tap). Starting from  $s_1$ , executing a sequence of actions  $a_{< j}$  arrives at screen  $s_j$  that represents the screen at the  $j$ th step:  $s_j = \tau (a_{j - 1},\tau (\dots \tau (a_1,s_1)))$
$$
p \left(a _ {1: m} \mid s _ {1}, \tau , t _ {1: n}\right) = \prod_ {j = 1} ^ {m} p \left(a _ {j} \mid s _ {j}, t _ {1: n}\right) \tag {2}
$$
Each screen  $s_j = [c_{j,1:|s_j|}, \lambda_j]$  contains a set of UI objects and their structural relationships.  $c_{j,1:|s_j|} = \{c_{j,k} \mid 1 \leq k \leq |s_j|\}$ , where  $|s_j|$  is the number of objects in  $s_j$ , from which  $o_j$  is chosen.  $\lambda_j$  defines the structural relationship between the objects. This is often a tree structure such as the View hierarchy for an Android interface (similar to a DOM tree for web pages).
An instruction  $I$  describes (possibly multiple) actions. Let  $\bar{a}_j$  denote the phrases in  $I$  that describes action  $a_j$ .  $\bar{a}_j = [\bar{r}_j, \bar{o}_j, \bar{u}_j]$  represents a tuple of descriptions with each corresponding to a span—a subsequence of tokens—in  $I$ . Accordingly,  $\bar{a}_{1:m}$  represents the description tuple sequence that we refer to as  $\bar{a}$  for brevity. We also define  $\bar{A}$  as all possible description tuple sequences of  $I$ , thus  $\bar{a} \in \bar{A}$ .
$$
p \left(a _ {j} \mid s _ {j}, t _ {1: n}\right) = \sum_ {\bar {A}} p \left(a _ {j} \mid \bar {a}, s _ {j}, t _ {1: n}\right) p (\bar {a} \mid s _ {j}, t _ {1: n}) \tag {3}
$$
Because  $a_{j}$  is independent of the rest of the instruction given its current screen  $s_{j}$  and description  $\bar{a}_{j}$ , and  $\bar{a}$  is only related to the instruction  $t_{1:n}$ , we can simplify (3) as (4).
$$
p \left(a _ {j} \mid s _ {j}, t _ {1: n}\right) = \sum_ {\bar {A}} p \left(a _ {j} \mid \bar {a} _ {j}, s _ {j}\right) p (\bar {a} \mid t _ {1: n}) \tag {4}
$$
1Our data pipeline is available at https://github.com/google-research/google-research/tree/master/seq2act.
2https://developer.android.com/reference/android/view(View.html
8199
We define  $\hat{a}$  as the most likely description of actions for  $t_{1:n}$ .
$$
\begin{array}{l} \hat {a} = \underset {\bar {a}} {\arg \max } p (\bar {a} | t _ {1: n}) \\ = \arg \max  _ {\bar {a} _ {1: m}} \prod_ {j = 1} ^ {m} p \left(\bar {a} _ {j} \mid \bar {a} _ {<   j}, t _ {1: n}\right) \tag {5} \\ \end{array}
$$
This defines the action phrase-extraction model, which is then used by the grounding model:
$$
p \left(a _ {j} \mid s _ {j}, t _ {1: n}\right) \approx p \left(a _ {j} \mid \hat {a} _ {j}, s _ {j}\right) p \left(\hat {a} _ {j} \mid \hat {a} _ {<   j}, t _ {1: n}\right) \tag {6}
$$
$$
p \left(a _ {1: m} \mid t _ {1: n}, S\right) \approx \prod_ {j = 1} ^ {m} p \left(a _ {j} \mid \hat {a} _ {j}, s _ {j}\right) p \left(\hat {a} _ {j} \mid \hat {a} _ {<   j}, t _ {1: n}\right) \tag {7}
$$
$p(\hat{a}_j|\hat{a}_{< j},t_{1:n})$  identifies the description tuples for each action.  $p(a_j|\hat{a}_j,s_j)$  grounds each description to an executable action given the screen.
The ideal dataset would have natural instructions that have been executed by people using the UI. Such data can be collected by having annotators perform tasks according to instructions on a mobile platform, but this is difficult to scale. It requires significant investment to instrument: different versions of apps have different presentation and behaviors, and apps must be installed and configured for each task. Due to this, we create a small dataset of this form, PILXELHELP, for full task evaluation. For model training at scale, we create two other datasets: ANDROIDHOWTo for action phrase extraction and RICOscA for grounding. Our datasets are targeted for English. We hope that starting with a high-resource language will pave the way to creating similar capabilities for other languages.
3.1 PILXELHELP Dataset
Pixel Phone Help pages<sup>3</sup> provide instructions for performing common tasks on Google Pixel phones such as switch Wi-Fi settings (Fig. 2) or check emails. Help pages can contain multiple tasks, with each task consisting of a sequence of steps. We pulled instructions from the help pages and kept ones that can be automatically executed. Instructions that require additional user input such as Tap the app you want to uninstall are discarded.
Figure 2: PixelHelp example: Open your device's Settings app. Tap Network & internet. Click Wi-Fi. Turn on Wi-Fi.. The instruction is paired with actions, each of which is shown as a red dot on a specific screen.
Also, instructions that involve actions on a physical button such as Press the Power button for a few seconds are excluded because these events cannot be executed on mobile platform emulators.
We instrumented a logging mechanism on a Pixel Phone emulator and had human annotators perform each task on the emulator by following the full instruction. The logger records every user action, including the type of touch events that are triggered, each object being manipulated, and screen information such as view hierarchies. Each item thus includes the instruction input,  $t_{1:n}$ , the screen for each step of task,  $s_{1:m}$ , and the target action performed on each screen,  $a_{1:m}$ .
In total, PILXELHELP includes 187 multi-step instructions of 4 task categories: 88 general tasks, such as configuring accounts, 38 Gmail tasks, 31 Chrome tasks, and 30 Photos related tasks. The number of steps ranges from two to eight, with a median of four. Because it has both natural instructions and grounded actions, we reserve PILXELHELP for evaluating full task performance.
3.2 AndroidHOWTo Dataset
No datasets exist that support learning the action phrase extraction model,  $p(\hat{a}_j|\hat{a}_{< j},t_{1:n})$ , for mobile UIs. To address this, we extracted English instructions for operating Android devices by processing web pages to identify candidate instructions for how-to questions such as how to change the input method for Android. A web crawling service scrapes instruction-like content from various websites. We then filter the web contents using both heuristics and manual screening by annotators.
Annotators identified phrases in each instruction that describe executable actions. They were given a tutorial on the task and were instructed to skip instructions that are difficult to understand or label. For each component in an action description, they
<sup>3</sup>https://support.google.com/pixelphone
select the span of words that describes the component using a web annotation interface (details are provided in the appendix). The interface records the start and end positions of each marked span. Each instruction was labeled by three annotators: three annotators agreed on  $31\%$  of full instructions and at least two agreed on  $84\%$ . For the consistency at the tuple level, the agreement across all the annotators is  $83.6\%$  for operation phrases,  $72.07\%$  for object phrases, and  $83.43\%$  for input phrases. The discrepancies are usually small, e.g., a description marked as your Gmail address or Gmail address.
The final dataset includes 32,436 data points from 9,893 unique How-To instructions and split into training (8K), validation (1K) and test (900). All test examples have perfect agreement across all three annotators for the entire sequence. In total, there are 190K operation spans, 172K object spans, and 321 input spans labeled. The lengths of the instructions range from 19 to 85 tokens, with median of 59. They describe a sequence of actions from one to 19 steps, with a median of 5.
3.3 RICOscA Dataset
Training the grounding model,  $p(a_{j}|\hat{a}_{j}, s_{j})$  involves pairing action tuples  $a_{j}$  along screens  $s_{j}$  with action description  $\hat{a}_{j}$ . It is very difficult to collect such data at scale. To get past the bottleneck, we exploit two properties of the task to generate a synthetic command-action dataset, RICO SCA. First, we have precise structured and visual knowledge of the UI layout, so we can spatially relate UI elements to each other and the overall screen. Second, a grammar grounded in the UI can cover many of the commands and kinds of reference needed for the problem. This does not capture all manners of interacting conversationally with a UI, but it proves effective for training the grounding model.
Rico is a public UI corpus with 72K Android UI screens mined from 9.7K Android apps (Deka et al., 2017). Each screen in Rico comes with a screenshot image and a view hierarchy of a collection of UI objects. Each individual object,  $c_{j,k}$ , has a set of properties, including its name (often an English phrase such as Send), type (e.g., Button, Image or checkbox), and bounding box position on the screen. We manually removed screens whose view hierarchies do not match their screenshots by asking annotators to visually verify whether the bounding boxes of view hierarchy leaves match each UI object on the corresponding screenshot image. This
filtering results in 25K unique screens.
For each screen, we randomly select UI elements as target objects and synthesize commands for operating them. We generate multiple commands to capture different expressions describing the operation  $\hat{r}_j$  and the target object  $\hat{o}_j$ . For example, the Tap operation can be referred to as tap, click, or press. The template for referring to a target object has slots Name, Type, and Location, which are instantiated using the following strategies:
- Name-Type: the target's name and/or type (the OK button or OK).
- Absolute-Location: the target's screen location (the menu at the top right corner).
- Relative-Location: the target's relative location to other objects (the icon to the right of Send).
Because all commands are synthesized, the span that describes each part of an action,  $\hat{a}_j$  with respect to  $t_{1:n}$ , is known. Meanwhile,  $a_j$  and  $s_j$ , the actual action and the associated screen, are present because the constituents of the action are synthesized. In total, RICOSCA contains 295,476 single-step synthetic commands for operating 177,962 different target objects across 25,677 Android screens.
4 Model Architectures
Equation 7 has two parts.  $p(\hat{a}_j|\hat{a}_{< j},t_{1:n})$  finds the best phrase tuple that describes the action at the  $j$ th step given the instruction token sequence.  $p(a_j|\hat{a}_j,s_j)$  computes the probability of an executable action  $a_j$  given the best description of the action,  $\hat{a}_j$ , and the screen  $s_j$  for the  $j$ th step.
4.1 Phrase Tuple Extraction Model
A common choice for modeling the conditional probability  $p(\bar{a}_j|\bar{a}_{< j},t_{1:n})$  (see Equation 5) are encoder-decoders such as LSTMs (Hochreiter and Schmidhuber, 1997) and Transformers (Vaswani et al., 2017). The output of our model corresponds to positions in the input sequence, so our architecture is closely related to Pointer Networks (Vinyals et al., 2015).
Figure 3 depicts our model. An encoder  $g$  computes a latent representation  $h_{1:n} \in R^{n \times |h|}$  of the tokens from their embeddings:  $h_{1:n} = g(e(t_{1:n}))$ . A decoder  $f$  then generates the hidden state  $q_j = f(q_{<j}, \bar{a}_{<j}, h_{1:n})$  which is used to compute a query vector that locates each phrase of a tuple  $(\bar{r}_j, \bar{o}_j, \bar{u}_j)$  at each step.  $\bar{a}_j = [\bar{r}_j, \bar{o}_j, \bar{u}_j]$  and they
8201
Figure 3: The Phrase Tuple Extraction model encodes the instruction's token sequence and then outputs a tuple sequence by querying into all possible spans of the encoded sequence. Each tuple contains the span positions of three phrases in the instruction that describe the action's operation, object and optional arguments, respectively, at each step.  $\varnothing$  indicates the phrase is missing in the instruction and is represented by a special span encoding.
are assumed conditionally independent given previously extracted phrase tuples and the instruction, so  $p(\bar{a}_j|\bar{a}_{< j},t_{1:n}) = \prod_{\bar{y}\in \{\bar{r},\bar{o},\bar{u}\}}p(\bar{y}_j|\bar{a}_{< j},t_{1:n}).$
Note that  $\bar{y}_j \in \{\bar{r}_j, \bar{o}_j, \bar{u}_j\}$  denotes a specific span for  $y \in \{r, o, u\}$  in the action tuple at step  $j$ . We therefore rewrite  $\bar{y}_j$  as  $y_j^{b:d}$  to explicitly indicate that it corresponds to the span for  $r, o$  or  $u$ , starting at the  $b$ th position and ending at the  $d$ th position in the instruction,  $1 \leq b < d \leq n$ . We now parameterize the conditional probability as:
$$
\begin{array}{l} p \left(y _ {j} ^ {b: d} \mid \bar {a} _ {<   j}, t _ {1: n}\right) = \operatorname {s o f t m a x} \left(\alpha \left(q _ {j} ^ {y}, h ^ {b: d}\right)\right) \tag {8} \\ y \in \{r, o, u \} \\ \end{array}
$$
As shown in Figure 3,  $q_{j}^{y}$  indicates task-specific query vectors for  $y \in \{r, o, u\}$ . They are computed as  $q_{j}^{y} = \phi(q_{j}, \theta_{y}) W_{y}$ , a multi-layer perceptron followed by a linear transformation.  $\theta_{y}$  and  $W_{y}$  are trainable parameters. We use separate parameters for each of  $r, o$  and  $u$ .  $W_{y} \in R^{|\phi_{y}| \times |h|}$  where  $|\phi_{y}|$  is the output dimension of the multi-layer perceptron. The alignment function  $\alpha(\cdot)$  scores how a query vector  $q_{j}^{y}$  matches a span whose vector representation  $h^{b:d}$  is computed from encodings  $h_{b:d}$ .
Span Representation. There are a quadratic number of possible spans given a token sequence (Lee et al., 2017), so it is important to design a fixed-length representation  $h^{b:d}$  of a variable-length token span that can be quickly computed. Beginning-Inside-Outside (BIO) (Ramshaw and Marcus, 1995)-commonly used to indicate spans in tasks such as named entity recognition-marks
whether each token is beginning, inside, or outside a span. However, BIO is not ideal for our task because subsequences for describing different actions can overlap, e.g., in click  $X$  and  $Y$ , click participates in both actions click  $X$  and click  $Y$ . In our experiments we consider several recent, more flexible span representations (Lee et al., 2016, 2017; Li et al., 2019) and show their impact in Section 5.2.
With fixed-length span representations, we can use common alignment techniques in neural networks (Bahdanau et al., 2014; Luong et al., 2015). We use the dot product between the query vector and the span representation:  $\alpha(q_j^y, h^{b:d}) = q_j^y \cdot h^{b:d}$ . At each step of decoding, we feed the previously decoded phrase tuples,  $\bar{a}_{<j}$  into the decoder. We can use the concatenation of the vector representations of the three elements in a phrase tuple or the sum their vector representations as the input for each decoding step. The entire phrase tuple extraction model is trained by minimizing the softmax cross entropy loss between the predicted and ground-truth spans of a sequence of phrase tuples.
4.2 Grounding Model
Having computed the sequence of tuples that best describe each action, we connect them to executable actions based on the screen at each step with our grounding model (Fig. 4). In step-by-step instructions, each part of an action is often clearly stated. Thus, we assume the probabilities of the operation  $r_j$ , object  $o_j$ , and argument  $u_j$  are
8202
Figure 4: The Grounding model grounds each phrase tuple extracted by the Phrase Extraction model as an operation type, a screen-specific object ID, and an argument if present, based on a contextual representation of UI objects for the given screen. A grounded action tuple can be automatically executed.
independent given their description and the screen.
$$
\begin{array}{l} p (a _ {j} | \hat {a} _ {j}, s _ {j}) = p ([ r _ {j}, o _ {j}, u _ {j} ] | [ \hat {r} _ {j}, \hat {o} _ {j}, \hat {u} _ {j} ], s _ {j}) \\ = p (r _ {j} | \hat {r} _ {j}, s _ {j}) p (o _ {j} | \hat {o} _ {j}, s _ {j}) p (u _ {j} | \hat {u} _ {j}, s _ {j}) \\ = p \left(r _ {j} \mid \hat {r} _ {j}\right) p \left(o _ {j} \mid \hat {o} _ {j}, s _ {j}\right) \tag {9} \\ \end{array}
$$
We simplify with two assumptions: (1) an operation is often fully described by its instruction without relying on the screen information and (2) in mobile interaction tasks, an argument is only present for the Text operation, so  $u_{j} = \hat{u}_{j}$ . We parameterize  $p(r_{j}|\hat{r}_{j})$  as a feedforward neural network:
$$
p \left(r _ {j} \mid \hat {r} _ {j}\right) = \operatorname {s o f t m a x} \left(\phi \left(\hat {r} _ {j} ^ {\prime}, \theta_ {r}\right) W _ {r}\right) \tag {10}
$$
$\phi (\cdot)$  is a multi-layer perceptron with trainable parameters  $\theta_r$ .  $W^{r}\in R^{|\phi_{r}|\times |r|}$  is also trainable, where  $|\phi_r|$  is the output dimension of the  $\phi (\cdot ,\theta_r)$  and  $|r|$  is the vocabulary size of the operations.  $\phi (\cdot)$  takes the sum of the embedding vectors of each token in the operation description  $\hat{r}_j$  as the input:  $\hat{r}_{j}^{\prime} = \sum_{k = b}^{d}e(t_{k})$  where  $b$  and  $d$  are the start and end positions of  $\hat{r}_j$  in the instruction.
Determining  $o_j$  is to select a UI object from a variable-number of objects on the screen,  $c_{j,k} \in s_j$  where  $1 \leq k \leq |s_j|$ , based on the given object description,  $\hat{o}_j$ . We parameterize the conditional probability as a deep neural network with a softmax output layer taking logits from an alignment function:
$$
\begin{array}{r l} p \left(o _ {j} \mid \hat {o} _ {j}, s _ {j}\right) & = p \left(o _ {j} = c _ {j, k} \mid \hat {o} _ {j}, c _ {j, 1: \left| s _ {j} \right|}, \lambda_ {j}\right) \\ & = \operatorname {s o f t m a x} \left(\alpha \left(\hat {o} _ {j} ^ {\prime}, c _ {j, k} ^ {\prime}\right)\right) \end{array} \tag {11}
$$
The alignment function  $\alpha (\cdot)$  scores how the object description vector  $\hat{o}_j^{\prime}$  matches the latent representation of each UI object,  $c_{j,k}^{\prime}$ . This can be as simple as the dot product of the two vectors. The latent representation  $\hat{o}_j^{\prime}$  is acquired with a multi-layer perceptron followed by a linear projection:
$$
\hat {o} _ {j} ^ {\prime} = \phi \left(\sum_ {k = b} ^ {d} e \left(t _ {k}\right), \theta_ {o}\right) W _ {o} \tag {12}
$$
$b$  and  $d$  are the start and end index of the object description  $\hat{o}_j$ .  $\theta_o$  and  $W_{o}$  are trainable parameters with  $W_{o} \in R^{|\phi_{o}| \times |o|}$ , where  $|\phi_{o}|$  is the output dimension of  $\phi(\cdot, \theta_{o})$  and  $|o|$  is the dimension of the latent representation of the object description.