SlowGuess/NAACL_2021_OCR huggingface dataset:
Sahisnu Mazumder*
Department of Computer Science  
University of Illinois at Chicago, USA  
sahisnumazumder@gmail.com
Oriana Riva
oriana.riva@microsoft.com
AI assistants can now carry out tasks for users by directly interacting with website UIs. Current semantic parsing and slot-filling techniques cannot flexibly adapt to many different websites without being constantly re-trained. We propose FLIN, a natural language interface for web navigation that maps user commands to concept-level actions (rather than low-level UI actions), thus being able to flexibly adapt to different websites and handle their transient nature. We frame this as a ranking problem: given a user command and a webpage, FLIN learns to score the most relevant navigation instruction (involving action and parameter values). To train and evaluate FLIN, we collect a dataset using nine popular websites from three domains. Our results show that FLIN was able to adapt to new websites in a given domain.
AI personal assistants, such as Google Assistant, can now interact directly with the UI of websites to carry out human tasks (Tech Crunch, 2019). Users issue commands to the assistant, and this executes them by typing, selecting items, clicking buttons, and navigating to different pages in the website. Such an approach is appealing as it can reduce the dependency on third-party APIs and expand an assistant's capabilities. This paper focuses on a key component of such systems: a natural language (NL) interface capable of mapping user commands (e.g., "find an Italian restaurant for  $7pm$ ") into navigation instructions that a web browser can execute.
One way to implement such an NL interface is to map user commands directly into low-level UI actions (button clicks, text inputs, etc.). The UI elements appearing in a webpage are embedded by concatenating their DOM attributes (tag, classes, text, etc.). Then, a scoring function (Pasupat et al., 2018) or a neural policy (Liu et al., 2018a) are
trained to identify which UI element best supports a given command. Learning at the level of UI elements is effective, but only in controlled (UI elements do not change over time (Shi et al., 2017a)) or restricted (single applications (Branavan et al., 2009)) environments. This is not the case in the "real" web, where (i) websites are constantly updated, and (ii) a user may ask an assistant to execute the same task in any website of their choice (e.g., ordering pizza with dominos.com or pizzahut.com). The transient nature and diversity of the web call for an NL interface that can flexibly adapt to environments with a variable and unknown set of actions, without being constantly re-trained.
To achieve this goal, we take two steps. First, we conceptualize a new way of designing NL interfaces for web navigation. Instead of mapping user commands into low-level UI actions, we map them into meaningful "concept-level" actions. Concept-level actions are meant to express what a user perceives when glancing at a website UI. In the example shown in Figure 1, the homepage of OpenTable has a concept-level action "Let's go" (where "Let's go" is the label of a search button), which represents the concept of searching something which can be specified using various parameters (a date, a time, a number of people and a search term). Intuitively, websites in a given domain (say, all restaurant websites) share semantically-similar concept-level actions and the semantics of a human task tend to be time invariant. Hence, learning at the level of concept-level actions can lead to a more flexible NL interface.
However, while concept-level actions vary less than raw UI elements, they still manifest with different representations and parameter sets across websites. Searching a restaurant in opentable.com, for example, corresponds to an action "Let's go" which supports up to four parameters; in yelp.com, the same action is called "search" and supports two parameters (search term and location). Websites in
*Work done while interning at Microsoft Research.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2777-2788 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Web task execution driven by NL commands in the OpenTable website. The user command is mapped to the concept-level action "Let's go" whose execution causes the transition from the home to the search results page.
one domain may also have different action types (e.g., making a table reservation vs. ordering food).
Our second insight to tackle this problem is to leverage semantic parsing methods in a novel way. Traditional semantic parsing methods (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2007; Branavan et al., 2009; Lau et al., 2009; Thummalapenta et al., 2012) deal with environments that have a fixed and known set of actions, hence cannot be directly applied. Instead, we propose FLIN, a new semantic parsing approach where instead of learning how to map NL commands to executable logical forms (as in traditional semantic parsing), we leverage the semantics of the symbols (name of actions/parameters and parameter values) contained in the logical form (the navigation instruction) to learn how to match it against a given command. Specifically, we model the semantic parsing task as a ranking problem. Given an NL command  $c$  and the set of actions  $A$  available in the current webpage, FLIN scores the actions with respect to  $c$ . Simultaneously, for each parameter  $p$  of an action, it extracts a phrase  $m$  from  $c$  that expresses a value of  $p$ , and then scores  $p$ 's values with respect to  $m$  to find the best value assignment for  $p$ . Each action with its associated list of parameter value assignments represents a candidate navigation instruction to be ranked. FLIN learns a net score for each instruction based on corresponding action and parameter value assignment scores, and outputs the highest-scored instruction as the predicted navigation instruction.
To collect a dataset for training and testing FLIN,
we built a simple rule-based Action Extractor tool that extracts concept-level actions along with their parameters (names and values, if available) from webpages. The implementation and evaluation of this tool is out of scope for this paper. $^{1}$  In a complete system, illustrated in Figure 1, we envision the Action Extractor to extract and pass the concept-level actions present in the current webpage to FLIN, which computes a candidate navigation instruction  $\mathcal{N}$  to be executed by an Action Executor (e.g., a web automation tool such as Selenium (2020) or Ringer (Barman et al., 2016)).
Overall, we make the following contributions: (1) we conceptualize a new design approach for NL interfaces for web navigation based on concept-level actions; (2) we build a match-based semantic parser to map NL commands to navigation instructions; and (3) we collect a new dataset based on nine websites (from restaurant, hotel and shopping domains) and provide empirical results that verify the generalizability of our approach. Code and dataset are available at https://github.com/microsoft/flin-nl2web.
Semantic parsing has long been studied in NLP with applications to databases (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2007; Ferre, 2017),
<sup>1</sup>The tool processes a webpage's DOM tree (structure and attributes) and visual appearance (using computer vision techniques) to extract actions along with names and values of their parameters (if any). It is an active area of research (Nguyen and Csallner, 2015; Liu et al., 2018b; Chen et al., 2020b,a).
knowledge-based question answering (Berant et al., 2013; Yih et al., 2015), data exploration and visual analysis (Setlur et al., 2016; Utama et al., 2018; Lawrence and Riezler, 2016; Gao et al., 2015), robot navigation (Artzi and Zettlemoyer, 2013; Tellex et al., 2011; Janner et al., 2018; Guu et al., 2017; Fried et al., 2018; Garcia et al., 2018), object manipulation (Frank and Goodman, 2012), object selection in commands (Golland et al., 2010; Smith et al., 2013), language game learning (Wang et al., 2016), and UI automation (Branavan et al., 2009; Fazzini et al., 2018; Zhao et al., 2019). This work assumes environments with a fixed and known set of actions, while we deal with variable and unknown sets of actions.
Work on NL-guided web task execution includes learning from demonstrations (Allen et al., 2007), building reinforcement learning agents (Shi et al., 2017b; Liu et al., 2018a), training sequence to sequence models to map natural language commands into web APIs (Su et al., 2017, 2018), and generating task flows from APIs (Williams et al., 2019). These techniques assume different problem settings (e.g., reward functions) or deal with low-level web actions or API calls. Unlike FLIN, they do not generalize across websites.
Pasupat et al. (2018) propose an embedding-based matching model to map natural language commands to low-level UI actions such as hyperlinks, buttons, menus, etc. Unlike FLIN, this work does not deal with predicting parameter values (i.e., actions are un-parametrized).
Models that jointly perform intent detection and slot filling (Guo et al., 2014; Liu and Lane, 2016; Chen et al., 2019) are not applicable to our problem for three reasons. First, they are trained on a per-application basis using application-specific intent and slot labels, and thus cannot generalize across websites. Second, they semantically label words in an utterance, but do not do value assignment, hence they cannot output executable navigation paths. Third, they perform multi-class classification (i.e., they assume only one intent to be true for a user query) and have no notion of state (e.g., current webpage). They do not deal with intents with overlapping semantics which may occur across pages of the same website (e.g., in the example shown in Figure 1, the same user query may map to the action "Let's go" in the OpenTable's home page or to the action "Find a Table" in the search results page).
Let  $A_w = \{a_1, a_2, \dots, a_n\}$  be the set of concept-level actions available in a webpage  $w$ . Each action  $a \in A_w$  is defined by an action name  $n_a$  and a set of  $K$  parameters  $P_a = \{p_1, p_2, \dots, p_K\}$ . Each parameter  $p \in P_a$  is defined by a name and a domain  $\text{dom}(p)$  (i.e., a set of values that can be assigned to parameter  $p$ ), and can be either closed domain or open domain.
For closed-domain parameters, the domain is bounded and consists of a finite set of values that  $p$  can take; the set is imposed by the website UI, such as the available colors and sizes for a product item or the available reservation times for a restaurant.
For open-domain parameters, the domain is, in principle, unbounded, but, in practice, it consists of all words/phrases which can be extracted from an NL command  $c$ . With reference to Figure 1, the "let's go" (search) action has  $n_a =$  "let's go" and  $P_a = \{ \text{"time", "date", "people", "location, restaurant, or cuisine"}\}$ . The first three parameters are closed domain and the last one (the search term) is open domain. The Action Extractor module (Figure 1) names actions and parameters after labels and texts appearing in the UI (or, if absent, using DOM attributes); it also automatically scrapes values of closed-domain parameters (from drop-down menus or select lists).
Given the above setting, our goal is to map an NL command  $c$  issued in  $w$  into a navigation instruction  $\mathcal{N}$ , consisting of a correct action name  $n_{a^*}$  corresponding to action  $a^* \in A_w$  and an associated list of  $m \leq |P_{a^*}|$  correct parameter-value assignments, given by  $\{(p_i = v_j') \mid p_i \in P_{a^*}, v_j' \in \text{dom}(p_i), 0 \leq i \leq K, 1 \leq j \leq |\text{dom}(p_i)|\}$ .
4 The FLIN Model
The task of solving the above semantic parsing problem can be decomposed into two sub-tasks: (i) action recognition, i.e., recognizing the action  $a \in A_w$  intended by  $c$ , and (ii) parameter recognition and value assignment, i.e., deciding whether a parameter of an action is expressed in  $c$  and, if so, assigning the value to that parameter. A parameter is expressed in  $c$  by a mention (word or phrase). For example, in Figure 1, "me and my friend" is a mention of parameter "people" in  $c$  and a correct parsing should map it to the domain value "2 people". Thus, the second sub-task involves first
2We use the same notation  $p$  to refer to the parameter and the parameter name interchangeably.
Figure 2: Architecture of FLIN including Action Scoring, Parameter Mention Extraction and Parameter Value Scoring components [BERT block diagram courtesy: (Devlin et al., 2019)].
extracting a mention of a given parameter from  $c$ , and then matching it against a set of domain values to find the correct value assignment. For an open-domain parameter, the extracted mention becomes the value of the parameter and no matching is needed, e.g. in Figure 1, the mention "Italian" should be assigned to the parameter "location, restaurant, or cuisine".
With reference to Figure 2, FLIN consists of four components, designed to solve the aforementioned two sub-tasks: (1) Action Scoring, which scores each available action with respect to the given command ( $\S 4.1$ ); (2) Parameter Mention Extraction, which extracts the mention (phrase) from the command for a given parameter ( $\S 4.2$ ); (3) Parameter Value Scoring, which scores a given mention with a closed-domain parameter value or rejects it if no domain values can be mapped to the mention ( $\S 4.3$ ); and (4) Inference (not shown in the figure), which uses the scores of actions and parameter values to infer the action-parameter-value assignment with the highest score as the predicted navigation instruction ( $\S 4.4$ ).
4.1 Action Scoring
Given a command  $c$ , we score each action  $a \in A_w$  to measure the similarity of  $a$  and  $c$ 's intent. We loop over the actions in  $A_w$  and their parameters to obtain a list of action name and parameters pairs
$(n_{a},P_{a})$  , and then score them with respect to  $c$
To score each pair  $(n_a, P_a)$ , we learn a neural network based scoring function  $S_{a}(.)$  that computes its similarity with  $c$ . We represent  $c$  as a sequence  $\{w_1, w_2, \dots, w_R\}$  of  $R$  words. To learn a vector representation of  $c$ , we first convert each  $w_i$  into corresponding one-hot vectors  $x_i$ , and then learn embeddings of each word using an embedding matrix  $E_w \in \mathbb{R}^{d \times |V|}$  as  $\mathbf{v}_i = E_w.x_i$ , where  $V$  is the word vocabulary. Next, given the word embedding vectors  $\{\mathbf{v}_i \mid 1 \leq i \leq |R|\}$ , we learn the forward and backward representation using a BiLSTM network (Schuster and Paliwal, 1997). Let the final hidden state for forward LSTM and backward LSTM, after consuming  $\{\mathbf{v}_i \mid 1 \leq i \leq R\}$ , be  $\overrightarrow{\mathbf{h}_R}$  and  $\overleftarrow{\mathbf{h}_1}$  respectively, we learn a joint representation of  $c$  as  $\mathbf{v}_c = [\overrightarrow{\mathbf{h}_R}; \overleftarrow{\mathbf{h}_1}] \in \mathbb{R}^{2d}$ , where  $[\cdot]$  denotes concatenation.
Next, we learn a vector representation of  $(n_a, P_a)$ . We use the same word embedding matrix  $E_w$  and Bi-LSTM layer to encode the action name  $n_a$  into a vector  $\mathbf{v}_a = B_iLSTM(n_a)$ . Similarly, we encode each parameter  $p \in P_a$  into a vector  $\mathbf{v}_p = B_iLSTM(p)$ , and compute the net parameter semantics of action  $a$  as the mean of the parameter vectors  $(\overline{\mathbf{v}}_p = \text{mean}\{\mathbf{v}_p \mid p \in P_a\})$ . Finally, to learn the overall semantic representation of  $(n_a, P_a)$ , we concatenate  $\mathbf{v}_a$  and  $\overline{\mathbf{v}}_p$  and learn a combined representation using a feed-forward (FF)
layer as
$$
\mathbf {v} _ {a p} = \tanh (W _ {a}. [ \mathbf {v} _ {a}; \overline {{\mathbf {v} _ {p}}} ] + b _ {a}) \qquad (1)
$$
where  $W_{a}\in \mathbb{R}^{4d\times 2d}$  and  $b_{a}\in \mathbb{R}^{2d}$  are weights and biases of the FF layer, respectively.
Given  $\mathbf{v}_c$  and  $\mathbf{v}_{ap}$ , we compute the intent similarity between  $c$  and  $(n_a, P_a)$  using cosine similarity:
$$
\begin{array}{l} S _ {a} (c, n _ {a}, P _ {a}) = \frac {1}{2} [ \operatorname {c o s i n e} (\mathbf {v} _ {c}, \mathbf {v} _ {a p}) + 1 ] \tag {2} \\ = \frac {1}{2} \left(\frac {\mathbf {v} _ {c} . \mathbf {v} _ {a p}}{\| \mathbf {v} _ {c} \| \| \mathbf {v} _ {a p} \|} + 1\right) \\ \end{array}
$$
where  $||\cdot ||$  denotes euclidean norm of a vector.  $S_{a}(\cdot)\in [0,1]$  is computed for each  $a\in A_w$  (and is used in inference,  $\S 4.4$ ).
The parameters of  $S_{a}(.)$  are learned by minimizing a margin-based ranking objective  $\mathcal{L}_a$ , which encourages the scores of each positive  $(n_a, P_a)$  pair to be higher than those of negative pairs in  $w$ :
$$
\mathcal {L} _ {a} = \sum_ {q \in Q ^ {+}} \sum_ {q ^ {\prime} \in Q ^ {-}} \max  \left\{S \left(q ^ {\prime}\right) - S (q) + 0. 5, 0 \right\} \tag {3}
$$
where  $Q^{+}$  is a set of positive  $(n_{a},P_{a})$  pairs in  $w$  and  $Q^{-}$  is a set of negative  $(n_a,P_a)$  pairs obtained by randomly sampling action name and parameter pairs (not in  $Q^{+}$ ) in  $w$ .
4.2 Parameter Mention Extraction
Given a command  $c$  and a parameter  $p$ , the goal of this step is to extract the correct mention  $m_p$  of  $p$  from  $c$ . In particular, we aim to predict the text span in  $c$  that represents  $m_p$ . We formulate this task as a question-answering problem, where we treat  $p$  as a question,  $c$  as a paragraph, and  $m_p$  as the answer. We fine-tune a pre-trained BERT (Devlin et al., 2019) model<sup>3</sup> to solve this problem.
As shown in Figure 2 (bottom-right), we represent  $p$  and  $c$  as a pair of sentences packed together into a single input sequence of the form [CLS]  $p$  [SEP]  $c$ , where [CLS], [SEP] are special BERT tokens. For tokenization, we use the standard Word-Piece Tokenizer (Wu et al., 2016). From BERT, we obtain  $T_{i}$  as output token embedding for each token  $i$  in the packed sequence. We only introduce a mention start vector  $S \in \mathbb{R}^{H}$  and a mention end vector  $E \in \mathbb{R}^{H}$  during fine-tuning. The probability of word  $i$  being the start of the mention is computed as a dot product between  $T_{i}$  and  $S$  followed by a
softmax over all of the words in  $c$ :  $P_{i} = \frac{e^{S\cdot T_{i}}}{\sum_{j}e^{S\cdot T_{j}}}$ .
The analogous formula is used to compute the end probability  $P_{j}$ . The position  $i$  (position  $j > i$ ) with highest start (end) probability  $P_{i}(P_{j})$  is predicted as start (end) index of the mention and the corresponding tokens in  $c$  are combined into a word sequence to extract  $m_{p}$ . We fine-tune BERT by minimizing the training objective as the sum of the log-likelihoods of the correct start and end positions. We train BERT to output [CLS] as  $m_{p}$ , if  $p$  is not expressed in  $c$  (no mention is identified, hence  $p$  is discarded from being predicted).
4.3 Parameter Value Scoring
Once the mention  $m_p$  is extracted for a closed-domain parameter  $p$ , we learn a neural network based scoring function  $S_p(\cdot)$  to score each  $p$ 's value  $v' \in \text{dom}(p)$  with respect to  $m_p$ . If  $p$  is open-domain, parameter value scoring is not needed.
The process is similar to that of action scoring, but, in addition to word-level semantic similarity, we also compute character-level and lexical-level similarity between  $v'$  and  $m_p$ . In fact,  $v'$  and  $m_p$  often have partial lexical matching. For example, given the domain value "7:00 PM" for the parameter "time", possible mentions may be "7 in the evening", "19:00 hrs", "at 7 pm", etc., where partial lexical-level similarity is observed. However, learning word-level and character-level semantic similarities is also important as "PM" and "evening" as well as "7:00 PM" and "19:00" are lexically distant to each other, but semantically closer.
Word-level semantic similarity. We use the same word embedding matrix  $E_{w}$  used in action scoring to learn the word vectors for both  $m_{p}$  and  $v'$ . We use a Bi-LSTM layer (not shared with Action Scoring) to encode mention (value) into a word-level representation vectors  $\mathbf{v}_{m}^{wd}(\mathbf{v}_{v'}^{wd})$ . We compute the word-level similarity between  $m_{p}$  and  $v'$  as
$$
S _ {p} ^ {w d} \left(m _ {p}, v ^ {\prime}\right) = \frac {1}{2} [ \operatorname {c o s i n e} \left(\mathbf {v} _ {m} ^ {w d}, \mathbf {v} _ {v ^ {\prime}} ^ {w d}\right) + 1 ] \tag {4}
$$
Character-level semantic similarity. We use a character embedding matrix  $E_{c}$  to learn the character vectors for each character composing the words in  $m_p$  and  $v'$ . To learn the character-level vector representation  $\mathbf{v}_{m}^{char}$  of  $m_p$ , we first learn the word vector for each word in  $m_p$  by composing the character vectors in sequence using an LSTM network (Hochreiter and Schmidhuber,
3https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1
1997), and then compose the word vectors for all mention words using a BiLSTM layer to obtain  $\mathbf{v}_m^{char}$ . Similarly, the character-level vector representation  $\mathbf{v}_{v'}^{char}$  of  $v'$  is obtained. Next, we compute the character-level similarity between  $m_p$  and  $v'$  as
$$
S _ {p} ^ {c h a r} (m _ {p}, v ^ {\prime}) = \frac {1}{2} \left[ c o s i n e (\mathbf {v} _ {m} ^ {c h a r}, \mathbf {v} _ {v ^ {\prime}} ^ {c h a r}) + 1 \right] (5)
$$
Lexical-level similarity. We use a fuzzy string matching score (using the Levenshtein distance to calculate the differences between sequences) and a custom value matching score which is computed as the fraction of words in  $v'$  that appear in  $m_p$ ; then we compute a linear combination of these similarity scores (each score  $\in [0,1]$ ) as the net lexical-level similarity score, denoted as  $S_p^{lex}(m_p,v') \in [0,1]$ .
Net value-mention similarity score. It is the mean of the three scores above:  $S_{p}(m_{p}, v') = \text{mean}\{S_{p}^{wd}(m_{p}, v'), S_{p}^{\text{char}}(m_{p}, v'), S_{p}^{\text{lex}}(m_{p}, v')\}$ .
The parameters of  $S_{p}(\cdot)$  are learned by minimizing a margin-based ranking objective  $\mathcal{L}_p$ , which encourages the scores  $S_{p}(\cdot)$  of each mention and positive value pair to be higher than those of mention and negative value pairs for a given  $p$ , and can be defined following the previously defined  $\mathcal{L}_a$  (See eq. 3).
The inference module takes the outputs of Action Scoring, Parameter Mention Extraction and Parameter Value Scoring to compute a net score  $S_{ap}(.)$  for each action  $a \in A_w$  and associated list of parameter value assignment combinations. Then, it uses  $S_{ap}(.)$  to predict the navigation instruction.
Parameter value assignment. We first infer the value to be assigned to each  $p \in P_a$ , where the predicted value  $\hat{v_p}$  for a closed-domain  $p$  is given by  $\hat{v_p} = \arg \max_{v' \in \text{dom}(p)} S_p(m_p, v')$  provided  $S_p(m_p, v') \geq \rho$ . Here,  $\rho$  is a threshold score (tuned empirically) for parameter value prediction.
While performing the value assignment for  $p$ , we consider  $S_{p}(m_{p},\hat{v_{p}})$  as the confidence score for  $p$ 's assignment. If  $S_{p}(m_{p},v^{\prime}) < \rho$  for all  $v^{\prime}\in dom(p)$ , we consider the confidence score for  $p$  as 0, implying  $m_p$  refers to a value which does not exist in  $dom(p)$ ;  $p$  is discarded and no value assignment is done. If  $p$  is an open-domain parameter,  $m_{p}$  is inferred as  $\hat{v}_{p}$  with a confidence score of 1.
If all  $p \in P_a$  are discarded from the prediction for a parametrized action  $a \in A_w$ , we discard  $a$ , as  $a$  no longer becomes executable in  $w$ .
Once we get all confidence scores for all value assignments for all  $p \in P_a$ , we compute the average confidence score  $\overline{S_p} (P_a)$ , and consider it to be the net parameter value assignment score for  $a$ .