SlowGuess/ACL_2020_OCR huggingface dataset:
Shashank Srivastava $^{1*}$  Oleksandr Polozov $^{2}$
<sup>1</sup>University of North Carolina, Chapel Hill ssrivastava@cs.unc.edu {polozov,
Nebojsa Jojic<sup>2</sup> Christopher Meek<sup>2</sup>
2Microsoft Research, Redmond jojic, meek}@microsoft.com
We explore learning web-based tasks from a human teacher through natural language explanations and a single demonstration. Our approach investigates a new direction for semantic parsing that models explaining a demonstration in a context, rather than mapping explanations to demonstrations. By leveraging the idea of inverse semantics from program synthesis to reason backwards from observed demonstrations, we ensure that all considered interpretations are consistent with executable actions in any context, thus simplifying the problem of search over logical forms. We present a dataset of explanations paired with demonstrations for web-based tasks. Our methods show better task completion rates than a supervised semantic parsing baseline (40% relative improvement on average), and are competitive with simple exploration-and-demonstration based methods, while requiring no exploration of the environment. In learning to align explanations with demonstrations, basic properties of natural language syntax emerge as learned behavior. This is an interesting example of pragmatic language acquisition without any linguistic annotation.
People routinely perform repetitive web-based tasks, involving sequences of clicking and typing actions. These include activities such as forwarding emails, booking flight tickets, ordering pizza, etc. These activities largely consist of small sequences of actions in an environment with restricted semantics, and are potentially amenable to automation. In this work, we explore whether an AI agent can be taught such tasks through natural language explanations and a single demonstration by a user (as one might teach such a task to a human assistant).
Figure 1: AI assistants that can be taught web-based procedures by their users can have diverse practical applications. Here, we explore learning very simple tasks from the Mini World-of-Bits framework using natural language explanations and a single demonstration of the task
From the perspective of language understanding, this involves challenges such as converting instructional language to actions, resolving ambiguities through pragmatics, and learning script-like behavior. The web domain is rich in textual, structural and spatial features, allowing for exploration of multiple types of grounding behavior including spatial and visual language understanding, as well as reasoning over semi-structured data. Also, despite its richness, the tasks involved usually do not require much background knowledge.
From a practical perspective, teachable AI assistants can change the way people interact with computers. Today's conversational assistants such as Alexa or Cortana act on a small number of preprogrammed language commands (e.g., "What is the weather going to be like?"). However, they cannot be taught new functionalities important to a user (as in Figure 1). Enabling users to teach computers personalized procedures through explained demonstrations can make conversational AI systems fundamentally more useful.
In Section 2, we situate our work in the broader body of work on grounded semantic parsing and
*Work done while the first author was at Microsoft Research.
7652
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7652-7662 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
learning from language. Section 3 summarizes our framework and dataset. In Section 4, we describe our approach in detail. Here, we investigate a new paradigm for interpreting language in grounded contexts. Instead of mapping statements to logical forms that then execute in a context as in traditional semantic parsing, the method considers the set of possible typing and clicking actions in a context, identifies features of corresponding web elements and their relationships with other elements on the webpage, and aligns these to natural language explanations through a generative model. Section 5 describes the empirical evaluation. Our contributions are:
- An approach towards learning web-based tasks from a single explained demonstration.
- A dataset of explanations and demonstrations for tasks from the MiniWoB framework.
- Empirical results showing that explained demonstrations can be an effective mode of supervision for learning such tasks. Language can significantly reduce the number of samples needed compared to learning from demonstrations alone.
Semantic Parsing: Supervised models for converting statements to logical forms have long been studied in a wide range of settings (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Wong and Mooney, 2007; Kwiatkowski et al., 2010; Yin and Neubig, 2017). More recent approaches focused on using weaker forms of supervision such as denotations or observations of world state (Berant et al., 2013; Clarke et al., 2010; Krishnamurthy and Mitchell, 2012) and semi-supervised methods aimed at efficient prototyping (Pasupat and Liang, 2015; Wang et al., 2015). These methods require more readily available supervision, such as question/answer pairs for model training, rather than annotations of logical forms. (Artzi and Zettlemoyer, 2013) learn to follow instructions in the context of robot navigation by conditioning parsing on environmental context. Artzi and Zettlemoyer (2011) use conversational feedback as a signal to induce logical forms for individual utterances from transcripts of conversations in a dialog-based setup. Some other recent approaches (Long et al., 2016; Guu et al., 2017) explore learning language from sequences of utterances and interactions in simple environments, which is conceptually similar to our
work. Muhlgay et al. (2019) and Guu et al. (2017) explore better strategies to search the space of logical forms. While all of these methods are related to multiple facets of work, our method diverges from them in that the space of candidate logical forms is driven by the constraints of possible actions in an environment rather than the natural language utterance. This guarantees that all of the considered logical forms during search are consistent with executable actions in any novel context. Finally, some recent methods (Andreas et al., 2016) marginalize over latent interpretations of language in context of downstream tasks. We use a similar Bayesian approach, where actions are chosen by marginalizing logical forms (rather than choosing a single interpretation of an explanation).
Interactive Learning from Language: Several frameworks have leveraged natural language supervision to learn new tasks, starting with early work on the SHRLDU system (Winograd, 1972) and Interactive Task Learning (Laird et al., 2017). In particular, several reinforcement learning approaches have been explored in text-based environments for learning strategies, following instruction manuals, game playing, etc. (Branavan et al., 2009; Goldwasser and Roth, 2014; Misra et al., 2018; Narasimhan et al., 2015). These approaches leverage the ability to explore and interact with the environment to learning policies that lead to favourable outcomes. This is different from our goal here, where the agent needs to learn from a single explained demonstration of a task, and no interactivity with the environment is assumed. Some recent approaches have shown language explanations to be effective for learning realistic tasks including relation extraction, concept learning and question answering (Hancock et al., 2018; Srivastava et al., 2017, 2018; Andreas et al., 2018).
In terms of the goal and problem formulation, our approach extends multiples lines of previous work. Quirk et al. (2015)'s work is similar to ours in motivation in learning user-specified recipes, but has no aspects of grounding or demonstrations.(Wang et al., 2016) explore interactive parser training through language games in context of block-world environments. Pasupat et al. (2018) explore mapping natural language to specific elements on complex and realistic web-pages, although not in context of learning from demonstrations. Our framework directly extends previous work on learning web-based tasks from the Mini
7653
Figure 2: Crowd-worker interface used for collecting natural language explanations and demonstrations
Figure 3: Examples of collected explanations
Word-of-Bits framework using multiple demonstrations and exploration of the environment (Shi et al., 2017; Liu et al., 2018). In particular, our DSL extends the constraint language defined in Liu et al. (2018) to explore learning from explained demonstrations instead.
3 Framework and dataset
We build on the Mini World-of-bits (MiniWoB) framework (Shi et al., 2017), a collection of web-based tasks initially proposed as a testbed for reinforcement learning agents. The tasks vary in difficulty in terms of the number of actions required, variability between instances of the task, and types of reasoning involved (including clicking specified buttons, forwarding emails and playing tic-tac-toe). See the top half of Figure 2 for an example of a task. Each task consists of a task description (yellow box), and an interactive web interface.
While previous methods have focused on learning sequential decision making to complete these tasks through a mixture of exploration (the framework provides simulators, where correctly completing a task yields a reward) and behavior cloning (by observing multiple demonstrations from human users); our focus is on learning to complete these tasks in a one-shot sense (without any exploration). This is because the one-shot case is a much more realistic scenario for learning web-based procedures from a teacher. In practical situations (where there are no simulators), it would not be feasible for an AI agent to learn to book flights by booking multiple incorrect tickets, or manage a user's email by sending multiple incorrect emails. On the other hand, a paradigm where the agent attempts to generalize from a single demonstration and explanations can be feasible for many more of such scenarios.
Task: Forward an email  
Click on the segment that mentions Maureen  
Click on the button name "Forward" at the bottom of the page  
Type in the word 'Amata' in front of the row tagged 'to'  
Click on the arrow button at the top of the page
Task: Select a radio button Focus on the word sequence after Select Click on the radio button to the left of the word sequence Press submit
We created a dataset of natural language explanations paired with demonstrations by human users for tasks from the MiniWoB framework. For this, crowdsourced workers on Amazon Mechanical Turk were asked to demonstrate how to complete these tasks and provide stepwise explanations to an AI assistant on how to complete the task. Since users would be unfamiliar with most of the tasks, for each task they were allowed to experiment with the interface as many times as they liked, and only the final demonstration was logged.
In all, we collect 520 demonstrations (each consisting of a sequence of click/type actions in the context of a MiniWoB task) paired with stepwise explanation sequences. Figure 3 shows samples of collected explanations. On average, each explanation sequence contains 3.3 explanations. The dataset contains 1719 explanations in total (individual steps), averaging 8.4 words per explanation. The size of the vocabulary of the explanations is 995. In general, workers found the teaching process to be engaging, with an average rating of 8.3 on a 1-10 scale on how they enjoyed the HIT in a post-completion survey. The dataset is available at https://aka.ms/Web-D-E.
Data characteristics: From a manual analysis of 100 randomly selected explanation sequences and task demonstrations, we find that in almost all cases (97%), the sequence of actions described in the explanations corresponds to the sequence of actions in the demonstration. More than  $85\%$  of explanations mention a clicking or typing action, while around  $10\%$  identify an entity/string on the webpage that is used in an action in the next step (e.g., the first explanation for the second task in Figure 3). Around  $3\%$  of the explanations correspond to conditionals and hypotheticals, which go beyond the scope of our approach. Roughly  $15\%$  of the explanations mention multiple entities on the webpage – usually specifying one element in relation to the other (e.g., "the radio button to the right of the text-box").
7654
Return Type	Operator	Type	Example invocation/description
Action	Action Click (element)	click the icon ... enter the destination ...
Action TypeString (element, string)
Identify
Web Element(s)	HasTag (tag)	Semantic	find the button ... ...that says 'submit' ... the email that mentions Jeanette ...
HasText (string)
HasTextIncluding (string)
HasPosnHigh()	Spatial	the button at the top ...
HasPosnLow()
RelnNear (element)
RelnSameRow (element)	the icon next to ...
RelnSameCol (element)
RelnBelow (element)
RelnAbove (element)
RelnRightOf (element)	the link below the icon ...
RelnLeftOf (element)
HasNumericIndex (int)	Count	the last option in the list ...
Identify
String(s)	IndexedWord(int)	From task description	the last word ...
BeforeWord(string)	the city after "from:"
AfterWord(string)
FindMatchingContext (string, context)	enter "Seattle" as the source city ...
Table 1: Major operators in DSL for learning of web-based procedures
3.2 DSL for semantic parsing
We define a domain specific language (DSL) for describing web-based procedures in terms of DOM elements by expanding on the constraint language in Liu et al. (2018). The DSL operators correspond to actions on DOM elements, element features and relations between them. The DSL defines the vocabulary of logical forms for parsing of user explanations, and grounds sensors and effectors in the web environment. Table 1 summarizes the DSL. There are three types of operations: (1) click and type actions on specified web elements (with a specified string, in case of a type action), (2) operations that filter elements on a page that satisfy a criterion, and (3) operations that filter strings based on a criterion. We include a special operator FindMatchingContext to accommodate cases in which the users provide explanations for an instance of a task with specific arguments mentioned in the task description (e.g., see the last row in Table 1). In this case, the operator can pick out the corresponding argument for the new instance by looking at the surrounding context in the new task description. The evaluation of logical forms in the DSL in the context of a webpage consists of set operations over all DOM elements on the webpage (and text-spans of up to two tokens for string operators). For example, the logical form HasTag (type=button) will evaluate to the set of elements on a page that have a HTML tag type with value button.
4 Learning from explanations and demonstrations
Our approach for learning web-based tasks, which we call LED – for Learning from Explained Demon
Figure 4: Modeling principle for Learning from Explained Demonstrations (LED). We prefer logical forms  $(l)$  that are both consistent with the user demonstration  $(d)$  in the context  $(c)$ , and relevant to the user's explanations  $(x)$ .
strations, models the process of explaining a demonstration of a task in a grounded context. We assume that the reasoning behind each action in a demonstration can be described by a logical form,  $l$ , in the DSL.  $^1LED$ 's essential idea is that preferred logical forms are both (1) consistent with the user demonstration,  $d$ , in the observed context,  $c$ , and (2) relevant to the user's language explanations,  $x$ .
Figure 4 illustrates this for a toy-example, where the context consists of a web-page with three elements, the demonstration consists of a single action, and a corresponding explanation is provided. Based on the observed demonstration (that elem3 was clicked), it is hard to infer the reason behind clicking it. Multiple logical forms in the DSL can be consistent with clicking elem3 in this context. e.g., it is at the top of the page, its color is blue, etc. However, these interpretations would not justify the provided explanation as those logical forms are not relevant to the explanation. Modeling relevance between logical forms and explanations can help identify the reasoning behind user demonstrations.
This framing diverges from traditional semantic parsing, where statements  $x$  are mapped to logi-
<sup>1</sup>We do not infer individual logical forms corresponding to an explanation, since we marginalize over all logical forms that resolve to the same action in a context.
cal forms  $l$  (e.g., database queries), which are then are executed against a context  $c$  (e.g., a knowledge base) to get a denotation (corresponds here with a demonstration)  $d$ . i.e.,  $d = \llbracket l(x)\rrbracket_{c}$ . In this model-theoretic view of semantics, parsed logical forms are not informed by the environmental context until execution. In comparison, LED roots logical forms in the observed context, and thus pragmatic consistency is ensured by design. We maximize the log-likelihood of observing the explanations given the demonstration in a grounded context:
$$
\mathcal {L} (\theta) = \log p (x | d, c) = \log \sum_ {l} \underbrace {p (x | l)} _ {\text {r e l e v a n c e}} \underbrace {p (l | d , c)} _ {\text {c o n s i s t e n c y}} \tag {1}
$$
Here, the first term corresponds to scoring relevance between logical forms and explanations (modeled using a semantic parsing model). The second term enforces consistency between candidate logical forms and the demonstration in the context, and can be deterministically evaluated. As we see in Section 4.2, consistency is enforced by temperature-based annealing during training.
4.1 Grounded Logical forms as latent variables
Eqn 1 marginalizes over latent logical forms. To make this tractable, we represent a logical form in a grounded context as an assignment of a tuple of discrete variables,  $l \coloneqq (e_0, f_0, r, e_1, f_1, a, t, f_t)$ . These variables indicate things such as which DOM element is acted upon ( $e_0$ ), if its relation ( $r$ ) with another element on the page ( $e_1$ ) is relevant, and so on. These are defined below.
-  $e_0 \in \mathrm{domElements}(c)$  denotes the DOM-element on which an action is performed. (e.g.,  $e_0 = \mathrm{elem3}$  in Fig 4) This is observed from the demonstration, thus  $p(e_0) = \mathbb{I}_{e_0 = e_{\mathrm{observed}}}$ .<sup>3</sup>
-  $f_0 = (f_{01} \dots f_{0n_F})$  is a set of selector variables, where  $f_{0i}$  denotes if feature  $i$  of element  $e_0$  is relevant for choosing it. Its domain is  $\{\phi \cup \mathcal{F}_i\}$ , where  $\mathcal{F}_i$  is the range of values feature  $i$  can take.  $f_{0i} = \phi$  denotes that the feature was not relevant for choosing  $e_0$  (e.g.,  $f_{0\text{color}} = \phi$  in Fig 4). If  $f_{0i} \neq \phi$ , it can only take the observed value of the feature for  $e_0$  in the context (e.g.,  $f_{0\text{tag}} = \text{square}$  in Fig 4). In Table 1, these correspond to operators that return web-elements and have names with prefix Has.
-  $r$  denotes if relation  $r$  between  $e_0$  and another element on the webpage is relevant for choosing it. Its domain is  $\{\phi \cup \mathcal{R}\}$ , where  $\mathcal{R}$  is the set of (binary) relations between elements in the DSL. In Table 1, these are operators that have names with prefix  $\operatorname{Reln}$ .  $r = \phi$  denotes that the no relation was relevant for choosing  $e_0$ . If  $r \neq \phi$ , it can only take the value of a relation that exists between  $e_0$  and another element. (e.g., in Fig 4,  $r$  can't take the value LeftOf, since elem3 is the rightmost element in the context). Our choice of having a single variable for  $r$  disallows logical forms with multiple or nested relations. This was guided by an analysis of our dataset, where none of the collected explanations show such behavior.
-  $e_1$  denotes that relation  $r$  between elements  $e_0$  and  $e_1$  is relevant for choosing  $e_0$ . Its domain is  $\{\phi \cup \text{domElements}(c)\}$ .  $e_1 = \phi$  if and only if  $r = \phi$ , i.e. if no relation is relevant for choosing  $e_0$ . If  $r = \text{reln}$ ,  $e_1$  can only take values of elements such that  $\text{reln}(e_0, e_1)$  is true in the context.
-  $f_{1} = (f_{11}\dots f_{1n_{F}})$  is a set of selector variables, where  $f_{1i}$  denotes if feature  $i$  of element  $e_1$  is relevant. e.g., for 'click the checkbox next to the button that says submit', the HasText feature of the button is relevant).  $f_{1i} = \phi$  denotes that feature  $i$  was not relevant. If  $f_{1i} \neq \phi$ , it can only take the observed value of the feature for  $e_1$ .
-  $a$  denotes the action performed on  $e_0$  (click or type). This is observed from the demonstration.
-  $t$  denotes the string to type, if  $a = type$ . This is observed from the demonstration (and is a substring of the task description text).
-  $f_{t} = (f_{t1}\dots f_{tn_{T}})$  is a set of selector variables, where  $f_{tj}$  denotes if the text feature  $j$  of  $t$  is relevant for choosing it (In Table 1, operators with a string return type correspond to text features).
Inverse Semantics: Assignments of values to these variables represents a search in the DSL space, since given any context, there is a mapping a from logical forms to an assignment of these variables. A key idea here is that, borrowing from program synthesis, we can leverage the inverse semantics of operators in the DSL (Polozov and Gulwani, 2015) to guarantee consistency of logical forms with the grounded context. i.e., at any step, the space of candidate logical forms we consider is consistent with the observed demonstration. This is possible because in our case, computing the inverse semantics for all operators in the DSL is feasible. $^4$
4Since there is only a relatively small number of candidates
2For example, in Figure 4, click (tag  $=$  triangle & rightOf(square)) won't be considered for the provided utterance, as it is inconsistent with the context.
$^3\mathbb{I}_{condition}$  denotes an indicator function for condition.
7656
As just described, our approach will use the context of the webpage leverage DSL inverse semantics to maintain an implicit set of candidate logical forms that are consistent with the observed demonstration. We will use variational inference to infer the logical forms that are most relevant to the seen explanations, and choose the action to take based on the inferred distribution over logical forms.
4.2 Model Description
In Eqn 1, the second term corresponds to a prior probability over logical forms given a demonstration and context (webpage). Our representation of logical forms as latent variable assignments (from Section 4.1) enables us to decompose this probability into local factor distributions. We choose these local priors to correspond to distributions that are uniform over assignments that are consistent, and has zero support otherwise, similar to previous work on pragmatic reasoning (Frank and Goodman, 2012; Monroe et al., 2017). In other words, these distributions are proportional to indicator function over valid assignments of variables in each factor. As seen below, these define a prior over  $l$  that is also proportional to a simple indicator function over values of  $l$  that are consistent with the observed demonstration and context.
$$
\begin{array}{l} p (l \mid d, c) = p \left(e _ {0}, f _ {0}, r, e _ {1}, f _ {1}, a, t, f _ {t} \mid d, c\right) \\ = p \left(e _ {0} | d\right) p \left(f _ {0} | e _ {0}, c\right) p \left(e _ {1}, r | e _ {0}, c\right) \\ \times p (f _ {1} | e _ {1}, c) p (a, t | d) p (f _ {t} | t, c) \\ \propto \mathbb {I} _ {\text {V a l i d} (e _ {0}, d)} \mathbb {I} _ {\text {V a l i d} (f _ {0}, e _ {0}, c)} \mathbb {I} _ {\text {V a l i d} (e _ {1}, r, e _ {0}, c)} \\ \times \mathbb {I} _ {\text {V a l i d} (f _ {1}, e _ {1}, c)} \mathbb {I} _ {\text {V a l i d} (a, t, d)} \mathbb {I} _ {\text {V a l i d} (f _ {t}, t, c)} \\ = \mathbb {I} _ {\text {V a l i d} (l, d, c)} \tag {2} \\ \end{array}
$$