SlowGuess/NAACL_2021_OCR huggingface dataset:
Nancy Xu
Computer Science Dept.
xnancy@stanford.edu
Sam Masling
smasling@stanford.edu
Michael Du
mdu7@stanford.edu
Giovanni Campagna
Computer Science Dept. Stanford University
Larry Heck
Viv Labs
Samsung Research
gcampaign@stanford.edu
larry.heck@ieee.org
Grounding natural language instructions on the web to perform previously unseen tasks enables accessibility and automation. We introduce a task and dataset to train AI agents from open-domain, step-by-step instructions originally written for people. We build RUSS (Rapid Universal Support Service) to tackle this problem. RUSS consists of two models: First, a BERT-LSTM with pointers parses instructions to ThingTalk, a domain-specific language we design for grounding natural language on the web. Then, a grounding model retrieves the unique IDs of any webpage elements requested in ThingTalk. RUSS may interact with the user through a dialogue (e.g. ask for an address) or execute a web operation (e.g. click a button) inside the web runtime. To augment training, we synthesize natural language instructions mapped to ThingTalk. Our dataset consists of 80 different customer service problems from help websites, with a total of 741 step-by-step instructions and their corresponding actions. RUSS achieves  $76.7\%$  end-to-end accuracy predicting agent actions from single instructions. It outperforms state-of-the-art models that directly map instructions to actions without ThingTalk. Our user study shows that RUSS is preferred by actual users over web navigation.
Grounding natural language is a key to building robots and AI agents (Chen and Mooney, 2011) that interact seamlessly with people. Besides grounding tasks visually (Mirowski et al., 2018; Venugopalan et al., 2015), future AI agents must be able to ground language and execute actions on the web.
We build a general-purpose, interactive agent to master tasks from open-domain natural language instructions on websites. We focus on the service domain for tasks such as redeeming a gift card,
James Landay
Computer Science Dept. Computer Science Dept. Stanford University Stanford University
landay@stanford.edu
lam@stanford.edu
Expert Instructions
1. Go to https://www.amazon.com/wishlist
2. Ask the user for their email
3. Then enter the user's email in the text field under email address or username
ThingTalk Parses
Semantic Parser
1. @goto (website =
"https://www.amazon.com/wishlist"
2. @ask (dict_key = "email")
3. @retrieve (description = "email address or username") => @retrieve (type = enum:input, below = id) => @enter (dict_key = "email", element_id = id)
Grounded Actions
Grounding Model
1. @goto(website =
https://www.amazon.com/wishlist)
2. @ask_dict_key = "email"
3. @enter(dict_key = "email", element_id = 37)
Figure 1: RUSS's semantic parser maps natural language instructions into ThingTalk (our DSL) and uses a grounding model to resolve elements in ThingTalk for grounded actions. The runtime executes the actions.
logging out of all your accounts, or resetting a password.
Conversational agents capable of providing universal access to the web through a language interface are an important step towards achieving information equity. These agents empower those who are visually impaired or situationally preoccupied (e.g. driving) to obtain web-based knowledge and services for which they would otherwise require a laptop or mobile device for (Sarsenbayeva, 2018). Already, virtual assistants and call centers demonstrate a large number of scenarios where language interfaces backed by web overlays are required by
Association for Computational Linguistics: Human Language Technologies, pages 1022-1032
companies and users. However, unlike virtual assistants, web agents like RUSS are universal, navigating the Web, interacting with users, and bypassing the need for domain-specific APIs.
On average over  $60\%$  of Americans have contacted customer service in a month (Statista Research Department, 2019). A call center manager might instruct its agents to do the following to help a customer through a password reset: "go to passwordreset.com; ask the user for their desired new password; click the reset button". As the agent performs the instructions on the web behind-the-scenes, the user is read information or asked questions periodically over a conversational interface (such as a phone).
Our approach, RUSS (Figure 1), trains an agent that masters any web task specified from open-domain instructions. To do so, we design a domain-specific language (DSL) for grounding on the web and implement it as a subset of the ThingTalk programming language (Campagna et al., 2019). Each natural language instruction maps to one of six agent actions that interact with users or operate on webpages. Actions that operate on the web are passed element IDs that are retrieved from high-level user language by grounding its corresponding ThingTalk on the active webpage. In the following, we use ThingTalk to refer to our subset taylored to web operations, where not ambiguous. We break down the problem into two components: (1) a semantic parser that takes single-step natural language instructions and maps to ThingTalk statements using a BERT-LSTM pointer network, and (2) a grounding model that takes ThingTalk and retrieves an element ID on the active webpage where needed.
1. Task: The new problem of building an interactive web agent capable of mastering tasks from open-domain natural language instructions.
2. RUSS: A fully functioning agent that services user support requests from natural language instructions. RUSS consists of a semantic parser, a grounding model, and a runtime. We release RUSS as an open-source repository
3. ThingTalk: A typed DSL that grounds natural language instructions on the web. ThingTalk is designed to be an expressive
target for natural language semantic parsing, and amenable to training data synthesis.
4. RUSS Dataset: a) Evaluation: a collection of 741 real-world step-by-step natural language instructions (raw and annotated) from the open web, and for each: its corresponding webpage DOM, ground-truth ThingTalk, and ground-truth actions; and b) Synthetic: a synthetic dataset of 1.5M natural language instructions mapped to ThingTalk.
5. Evaluation of RUSS:  $76.7\%$  accuracy on our RUSS evaluation dataset. Our semantic parser maps natural language instructions to ThingTalk at  $85\%$  accuracy and our grounding model achieves  $75\%$  accuracy in resolving web element descriptions. A user study of RUSS shows preference of the natural language interface over existing Web UIs.
Grounding in the Visual and Physical Worlds (Robotics). Grounding language in both the physical world (Chen and Mooney, 2011) and in images and videos ((Venugopalan et al., 2015), (Hendricks et al., 2018)) through systems like visual question-answering (Antol et al., 2015) have been extensively explored. For example, Thomason et al. (2016) describe the game "I Spy" where human and robot take turns describing one object among several in a physical environment, requiring grounding of natural language to the physical world, and robot-human dialogues are explored in (Thomason et al., 2019). Previous work has proposed adaptive language interfaces for robots in dynamic settings such as (Liu et al., 2018), (Ito et al., 2020; Liu et al., 2018), (Karamcheti et al., 2020), and (Kim et al., 2020). Other work builds physical world agents that operate through sequential actions (Chen and Mooney, 2011; Misra et al., 2017; Mirowski et al., 2018).
Natural Language Digital Interfaces. An intelligent automated software assistant that collaborates with humans to complete tasks was first introduced in (Allen et al., 2007). Since then, identifying UI components from natural language commands has been an important area of research in grounding, with prior work investigating approaches to map natural language instructions to mobile interfaces such as Android (Li et al., 2020) and Adobe photo editing GUIs (Manuvinakurike et al., 2018). Earlier work mapped natural lan
<sup>1</sup>https://github.com/xnancy/russ
Agent Action	Description
@goto(url)	Navigate to the given URL
@enter.element_id, dict_key)	Find the closest match to the given dictionary key and enter its value in the given input element
@click.element_id)	Click on the given element
@read.element_id)	Read the content of the given element to the user
@say(message)	Read the given message to the user
@ask_dict_key)	Ask the user for the value of a dictionary key
Grounding Function	Description
@retrieve(desr_type, loc, above, below, right_of, left_of): element_id	Retrieves the elements matching the descriptors, returns an element_id.
Table 1: WebLang Agent Actions and a Grounding Function
guage commands to web elements on a TV screen through a combination of lexical and gesture intent (Heck et al., 2013). More recently, Pasupat et al. (2018) attempted to map natural language commands written by Amazon Mechanical Turkers to web elements (without actions). Unlike prior research, our work focuses on a new domain of parsing natural language instructions into executable actions on the web, where instead of mapping directly to elements using a neural model, we semantically parse natural language instructions to formal actions that support web navigation as well as user interactivity.
Dialogue Agents for The Web. Other web-based dialogue agents are developed through single-use heuristics and more recently through programming-by-demonstration (PBD) tools. This approach allows users and developers to author programs that operate on the web and invoke those programs in natural language (Li et al., 2017; Li and Riva, 2018; Fischer et al., 2020; Sarmah et al., 2020). CoScripter (Leshed et al., 2008) additionally allows the user to edit the demonstration in natural language, and parses a limited natural language into executable form. While related in end goal, our work does not require user demonstration and can operate using existing real-world instructions. We note though that the WebLang intermediate representation and our grounding model can be used to improve the robustness of PBD systems as well.
3 Task and Model
Given a set of natural language instructions  $S = (i_1, \ldots, i_n)$  and a starting web page, our task is to construct an agent that follows the instructions through a series of action  $A = (a_1, \ldots, a_n)$ . Actions include web navigation and end-user inter
action in order to obtain necessary information. Surveying online customer service tasks, 6 action operations were identified as necessary for agents: open a URL page, enter text, click on buttons, say something to the user, read the results to the user, and ask user for some information. Details are described in Table 1, where elements on a web page are assumed to be given unique element IDs.
RUSS is trained to execute tasks by grounding natural language instructions on the web. The modular design of RUSS, with separate semantic parser and grounding model, is motivated by the high cost of training data acquisition, and the ability to improve each component independently.
We first describe ThingTalk, then the three components of Russ: the semantic parser model, the grounding model, and the runtime.
3.1 ThingTalk
ThingTalk is designed to be (1) robust to open-domain natural language, (2) a suitable target for semantic parsing from natural language, and (3) trainable with only synthetic data.
The primitives in ThingTalk include all the agent actions and a grounding function @retrieve (Table 1). The latter is informed by the descriptions in the instructions we found in the wild. The input features accepted by @retrieve are:
- descr: textual description of the element
- type: type of element (button, input box, paragraph, header, etc.)
- loc: absolute position of the element on the page
- above/below/...: position of the element relative to another; above, below, right, and left.
To support element descriptions involving multiple features or other elements, ThingTalk is com
Figure 2: The RUSS semantic parser, using the BERT-LSTM architecture
Figure 3: Representation of an instruction in RUSS
positional in design. A ThingTalk program is a sequence of statements with syntax  $[r\Rightarrow ]^* a$  , where  $r$  is the retrieve operation and  $a$  is an agent action. @retrieve returns an element_id that is passed to @click (to click on the element), @read (to read the text in the element to the user), or @enter (to enter text in the element). For agent actions that require an element id, we call the sequence of @retrieve functions used to obtain the final element id used in the agent action the query. See Figure 1 for sample ThingTalk parses from natural language instructions. The orange ThingTalk parse demonstrates a query with 2 @retrieve functions.
3.2 Semantic Parser Model
To translate natural language instructions into ThingTalk, we use the previously proposed BERT-LSTM model (Xu et al., 2020). BERT-LSTM is an encoder-decoder network that uses a pretrained BERT encoder (Devlin et al., 2019) and LSTM (Hochreiter and Schmidhuber, 1997) decoder with a pointer-generator (See et al., 2017; Paulus et al., 2018). The architecture is shown in Fig. 2. The model is trained to encode natural language utterances and produce the ThingTalk code token-by-token. The pointer network in the decoder allows the model to predict out-of-vocabulary words by copying from the input utterances.
We preprocess the natural language by performing entity extraction, where entity strings are mapped to placeholder tokens (URL, LOC, TYPE), and the strings are substituted back into the ThingTalk code after parsing with the placeholder tokens. This resolves errors related to long URLs being broken into tokens that are not always copied to ThingTalk together and helps disambiguate important input features. For example: "Click the button on the top of the amazon.com
Instruction: "Enter the user's order number in the text field that says order number"
DOM:  
element_id: 1, type = "body"  
element_id: 2, type = "h1", text = "Your Orders"  
element_id: 3, type = "form"  
...  
element_id: 48, type = "label", text = "order number"  
element_id: 49, type = "input"  
...  
ThingTalk:  
@retrieve(description = "order number", type = input)  
 $\Rightarrow$  @enter(text = order_number, element = id)  
Action: @enter(text = order_number, element = 49)
page" maps to "Click the TYPE on the LOC of the URL page". We use a simple set of heuristics to identify the entity strings for each placeholder token, such as the presence of a 'www', '.com', 'http' substring to indicate a URL entity.
3.3 Grounding Model
The webpage is modeled using the Document Object Model (DOM), which is a hierarchical representation of all elements in the page. Our DOM representation records element features such as the following for each element:
- inner text content of the element
- HTML id, tag, class
- hidden state (True/False if element is visible on the webpage)
height/width of the element
- left/right/top/bottom coords of the element
- list of child elements in the DOM.
An example is shown in Fig. 3.
RUSS's grounding model grounds a ThingTalk @retrieve function by mapping it to an element ID. The input features in the @retrieve function are mapped against scores derived from the element features in the DOM to identify the best match.
The grounding model consists of the following steps. It filters elements by their type and absolute location. Next it handles relative positioning by identifying those elements with the right relational context to, and not too far away from, the given element's coordinates. It passes the text of the remaining candidates through a SentenceBERT (Reimers and Gurevych, 2019) neural network and computes the cosine similarities of their embeddings with the embedding of the input text
description. The elements with the highest score are returned.
3.4 The Run-Time
To execute the grounded ThingTalk program, RUSS starts a new automated Chrome session for each task and uses Puppeteer to automate web actions in the browser. RUSS uses a Google Voice API to implement actions involving user interactions (@say, @ask, or @read). For @ask actions, RUSS uses a preprogrammed dialogue to ask the user for a dictionary key (such as "name"), verifies the dictionary key is a valid string, and stores the value given by the user in a user's dictionary under that key. In @enter actions, we retrieve information to be entered by finding its closest match among the user's dictionary keys.
This paper contributes two datasets, the RUSS Evaluation Dataset with real-world instructions and the RUSS Synthetic Dataset for training semantic parsers.
4.1 RUSS Evaluation Dataset
The RUSS Evaluation Dataset consists of real-world tasks from customer service help centers of popular online companies. To make our task-open domain, the online help centers we use span a diverse range of domains including music, email, online retail, software applications, and more. For each instruction in a task, the dataset includes:
- the English instruction in natural language as it appears in the original website, and the human-edited version of the instruction
- the DOM of the web page where the instruction can be executed, with the element features associated with each element
- the ThingTalk code corresponding to the instruction
- the grounded action of the instruction
To collect the RUSS Evaluation dataset, we acquire a list of "Top 100 visited websites" and locate tasks that offer line-by-line help instructions from those. An author of the paper walked through each task, performed the actions as instructed, scraped the webpage in the browser, and annotated the instruction with the corresponding ThingTalk code. Steps found missing from the instructions were inserted. If an instruction mapped to several actions, the text was broken into individual
Figure 4: Lengths of instructions in the RUSS Evaluation Dataset
Figure 5: Distribution of actions in the RUSS Evaluation Dataset. @click, @enter, and @read require a webpage element.
instructions. Note that the human worker did not participate in the design of ThingTalk; they were asked to write instructions as if they were teaching another human step-by-step.
We collected a total of 80 tasks and 741 lines of instructions from 22 different online help centers. The dataset is split into a dev set and a test set, with 304 instructions from 30 tasks in the dev set and 437 instructions from 50 tasks in the test set. The RUSS Evaluation dataset is not used for training. On average, instructions in RUSS contain 9.6 tokens (Fig. 4), significantly longer than the crowdsourced web instructions in PhraseNode which average 4.1 tokens. The three most common actions in the dataset are "click", "ask" and "enter" (Fig. 5).  $61.4\%$  of the natural-language instructions require retrieving an element from the webpage (click, enter, read). Table 2 illustrates different types of reasoning supported by the @retrieve descriptors and their frequency in the RUSS Evaluation Dataset. Lastly, 76 of the 455 element queries use two @retrieve functions, with the rest all just using one, and  $53.7\%$ ,  $42.7\%$ , and  $3.6\%$  of the @retrieve functions have 1, 2, and 3 descriptors, respectively (Fig. 6).
While the language has just 7 core actions, the combinatorial space of possible actions and web elements is much larger - on the order of 1000s
ThingTalk Includes: (@retrieve feature)	Description	Frequency
Type reasoning (type)	Requires specific HTML type (e.g. button, checkbox)	29.0%
Input target (type = input)	Requires target element is a text input	25.0%
Relational reasoning (below/above/left of...)	References neighboring features of the element	10.3%
Spatial reasoning (location)	References element location on the webpage	4.6%
No web element (No @retrieve)	No element (operation is @ask / @goto / @say)	38.6%
Figure 6: # @retrieve functions in each RUSS instruction and # descriptors in each @retrieve.
of possible combinations per instruction. On average the DOMs of the webpages contain 689 web elements each.
The total vocabulary size of the Evaluation Dataset found in the wild is 684 words. We find that at least one of the most frequent 300 words in the Evaluation vocabulary is present in  $>50\%$  of the Evaluation Dataset instructions. There are also many domain-specific words throughout the instructions.
Labeling large numbers of instructions in ThingTalk for training is time consuming and demands expertise. To address this, we use a typed template-based synthesis method to generate our training data. We write templates for each ThingTalk primitive and common combinations thereof. We also scrape a large dataset of naturally occurring DOM element text, webpage URLs, and phrases that are likely to be variable names to use for each parameter. The synthesizer compositionally expands the templates and sample values from the scraped dataset to construct a large training set of instructions mapped to ThingTalk automatically. We generate hundreds of different types of natural language templates which are combined to create a Synthetic Dataset with 1.5M training samples. This composition method creates roughly 840 distinct templates. To promote generalizability of our model, the total vocabulary size of the Synthetic corpus is large compared to the evaluation vocabulary size at
Table 2: Subset of reasoning types (with the @retrieve input feature used to indicate it) supported by ThingTalk and their frequency in the RUSS dataset. Some statements require multiple reasoning types.
Model	Accuracy (test)
RUSS (1.5M training parses)	87.0%
Ablations	Accuracy (dev)
RUSS (1.5M training parses)	88.2%
— entity extraction	77.6%
— 1M training parses, entity extraction	70.0%
Table 3: Evaluation of Semantic Parsing Model (trained on 1.5M parses) on RUSS Evaluation test set. Ablations are performed on the dev set. “-” in Ablations subtracts a feature from the RUSS model, the second ablation is trained on 500k training parses.
9305 words.
An example of a simple template is:
"At the loc of the page,
@click the button that says desc
which is mapped to the ThingTalk:
$$
\begin{array}{l} @ \operatorname {r e t r i e v e} (\operatorname {d e s c r} = \operatorname {d e s c r}, \operatorname {l o c} = \operatorname {l o c}) \longrightarrow \\ @ \text {c l i c k} (\text {e l e m e n t} = \text {i d}) \\ \end{array}
$$
RUSS achieves  $76.7\%$  overall accuracy on the Evaluation Dataset, even though all of RUSS, including the semantic parser is trained with only synthetic data.
We perform 3 experiments to evaluate the individual components and the system as a whole: 1) Accuracy evaluation of RUSS's Parsing Model with ablation studies. 2) Accuracy evaluation and baseline comparisons of RUSS's Grounding Model. 3) User study evaluating RUSS's ability to master 5 tasks on-the-job. We test usability and efficacy of RUSS compared with existing customer service help websites.
5.1 Semantic Parsing Accuracy
Our first experiment evaluates the accuracy of our semantic parser on the RUSS Evaluation dataset. We measure Exact Match Accuracy: a parse is considered correct only if it matches the gold annotation token by token.
Model	Grounding Acc (test)
RUSS	63.6%
End-to-End Baseline	51.1%
PhraseNode	46.5%
The results are shown in Table 3. The parser obtains  $87.0\%$  accuracy on the test set. Despite using no real-world training data, the semantic parser achieves high accuracy on the challenging evaluation set. It achieves an accuracy of  $81.4\%$  for instructions involving web elements, and  $94.6\%$  for the rest. This suggests the semantic parser can handle both types of instructions with high accuracy, especially instructions that parse to user interactions (no web element).
We perform an ablation study on the RUSS Evaluation dev set as seen in Table 3. RUSS achieves  $88.2\%$  accuracy on the dev set. The entity extraction technique where string entities are replaced with placeholders during training, as discussed in Section 3.2, contributes  $10.6\%$  improvement in accuracy. Training without this pre-processing step and with only 500K parses will reduce the accuracy further by  $7.6\%$ . This suggests that it is important to have a large synthetic training data set.
5.2 Grounding Evaluation