Venue: NAACL 2021

# FLIN: A Flexible Natural Language Interface for Web Navigation

 Sahisnu Mazumder\({}^{*}\)

Department of Computer Science

University of Illinois at Chicago, USA

sahisnunazumder@gmail.com &Oriana Riva

Microsoft Research

Redmond, USA

oriana.riva@microsoft.com

Work done while interning at Microsoft Research.

###### Abstract

AI assistants can now carry out tasks for users by directly interacting with website UIs. Current semantic parsing and slot-filling techniques cannot flexibly adapt to many different websites without being constantly re-trained. We propose _FLIN_, a natural language interface for web navigation that maps user commands to concept-level actions (rather than low-level UI actions), thus being able to flexibly adapt to different websites and handle their transient nature. We frame this as a ranking problem: given a user command and a webpage, FLIN learns to score the most relevant navigation instruction (involving action and parameter values). To train and evaluate FLIN, we collect a dataset using nine popular websites from three domains. Our results show that FLIN was able to adapt to new websites in a given domain.

## 1 Introduction

AI personal assistants, such as Google Assistant, can now interact directly with the UI of websites to carry out human tasks Tech Crunch (2019). Users issue commands to the assistant, and this executes them by typing, selecting items, clicking buttons, and navigating to different pages in the website. Such an approach is appealing as it can reduce the dependency on third-party APIs and expand an assistant's capabilities. This paper focuses on a key component of such systems: a natural language (NL) interface capable of mapping user commands (e.g., "_find an Italian restaurant for 7pm_") into navigation instructions that a web browser can execute.

One way to implement such an NL interface is to map user commands directly into low-level UI actions (button clicks, text inputs, etc.). The UI elements appearing in a webpage are embedded by concatenating their DOM attributes (tag, classes, text, etc.). Then, a scoring function Pasupat et al. (2018) or a neural policy Liu et al. (2018a) are trained to identify which UI element best supports a given command. Learning at the level of UI elements is effective, but only in controlled (UI elements do not change over time Shi et al. (2017)) or restricted (single applications Branavan et al. (2009)) environments. This is not the case in the "real" web, where (i) websites are constantly updated, and (ii) a user may ask an assistant to execute the _same_ task in _any_ website of their choice (e.g., ordering pizza with dominos.com or piezzahut.com). The transient nature and diversity of the web call for an NL interface that can _flexibly adapt_ to environments with a _variable_ and _unknown_ set of actions, without being constantly re-trained.

To achieve this goal, we take two steps. First, we conceptualize a new way of designing NL interfaces for web navigation. Instead of mapping user commands into low-level UI actions, we map them into meaningful "_concept-level_" actions. Concept-level actions are meant to _express what a user perceives when glancing at a website UI_. In the example shown in Figure 1, the homepage of OpenTable has a concept-level action "_Let's go_" (where "Let's go" is the label of a search button), which represents the concept of searching something which can be specified using various parameters (a date, a time, a number of people and a search term). Intuitively, websites in a given domain (say, all restaurant websites) share semantically-similar concept-level actions and the semantics of a human task tend to be time invariant. Hence, learning at the level of concept-level actions can lead to a more flexible NL interface.

However, while concept-level actions vary less than raw UI elements, they still manifest with different representations and parameter sets across websites. Searching a restaurant in opertable.com, for example, corresponds to an action "_Let's go_" which supports up to four parameters; in yelp.com, the same action is called "_search_" and supports two parameters (search term and location). Websites in

one domain may also have different action types (e.g., making a table reservation vs. ordering food).

Our second insight to tackle this problem is to leverage semantic parsing methods in a novel way. Traditional semantic parsing methods Zelle and Mooney (1996); Zettlemoyer and Collins (2007); Branavan et al. (2009); Lau et al. (2009); Thummalapenta et al. (2012) deal with environments that have a _fixed_ and _known_ set of actions, hence cannot be directly applied. Instead, we propose _FLIN_, a new semantic parsing approach where instead of learning how to map NL commands to executable logical forms (as in traditional semantic parsing), we leverage the semantics of the symbols (name of actions/parameters and parameter values) contained in the logical form (the navigation instruction) to _learn how to match it_ against a given command. Specifically, we model the semantic parsing task as a _ranking problem_. Given an NL command \(c\) and the set of actions \(A\) available in the current webpage, FLIN scores the actions with respect to \(c\). Simultaneously, for each parameter \(p\) of an action, it extracts a phrase \(m\) from \(c\) that expresses a value of \(p\), and then scores \(p\)'s values with respect to \(m\) to find the best value assignment for \(p\). Each action with its associated list of parameter value assignments represents a candidate navigation instruction to be ranked. FLIN learns a net score for each instruction based on corresponding action and parameter value assignment scores, and outputs the highest-scored instruction as the predicted navigation instruction.

To collect a dataset for training and testing FLIN, we built a simple rule-based _Action Extractor_ tool that extracts concept-level actions along with their parameters (names and values, if available) from webpages. The implementation and evaluation of this tool is out of scope for this paper.1 In a complete system, illustrated in Figure 1, we envision the Action Extractor to extract and pass the concept-level actions present in the current webpage to FLIN, which computes a candidate navigation instruction \(\mathcal{N}\) to be executed by an _Action Executor_ (e.g., a web automation tool such as Selenium (2020) or Ringer Barman et al. (2016)).

Footnote 1: The tool processes a webpage’s DOM tree (structure and attributes) and visual appearance (using computer vision techniques) to extract actions along with names and values of their parameters (if any). It is an active area of research Nguyen and Csallner (2015); Liu et al. (2018); Chen et al. (2020, 2020).

Overall, we make the following contributions: **(1)** we conceptualize a new design approach for NL interfaces for web navigation based on concept-level actions; **(2)** we build a match-based semantic parser to map NL commands to navigation instructions; and **(3)** we collect a new dataset based on nine websites (from restaurant, hotel and shopping domains) and provide empirical results that verify the generalizability of our approach. Code and dataset are available at [https://github.com/microsoft/flin-nl2web](https://github.com/microsoft/flin-nl2web).

## 2 Related Work

Semantic parsing has long been studied in NLP with applications to databases Zelle and Mooney (1996); Zettlemoyer and Collins (2007); Ferre (2017),

Figure 1: Web task execution driven by NL commands in the OpenTable website. The user command is mapped to the concept-level action “Let” s go” whose execution causes the transition from the home to the search results page.

 

knowledge-based question answering [1, 23], data exploration and visual analysis [25, 40, 69, 78], language game learning [26], and UI automation [1, 13, 14, 15, 16, 17, 18, 19], object manipulation [14], object selection in commands [15, 16], language game learning [27], and UI automation [1, 17, 18, 19]. This work assumes environments with a fixed and known set of actions, while we deal with _variable_ and _unknown_ sets of actions.

Work on NL-guided web task execution includes learning from demonstrations [15], building reinforcement learning agents [13, 14], training sequence to sequence models to map natural language commands into web APIs [26, 27], and generating task flows from APIs [28]. These techniques assume different problem settings (e.g., reward functions) or deal with low-level web actions or API calls. Unlike FLIN, they do not generalize across websites.

Pasupat et al. pasupat:2018 propose an embedding-based matching model to map natural language commands to low-level UI actions such as hyperlinks, buttons, menus, etc. Unlike FLIN, this work does not deal with predicting parameter values (i.e., actions are un-parametrized).

Models that jointly perform intent detection and slot filling [19, 10, 15] are not applicable to our problem for three reasons. First, they are trained on a per-application basis using application-specific intent and slot labels, and thus cannot generalize across websites. Second, they semantically label words in an utterance, but do not do value assignment, hence they cannot output executable navigation paths. Third, they perform multi-class classification (i.e., they assume only one intent to be true for a user query) and have no notion of state (e.g., current webpage). They do not deal with intents with overlapping semantics which may occur across pages of the same website (e.g., in the example shown in Figure 1, the same user query may map to the action "_Let's go_" in the OpenTable's home page or to the action "_Find a Table_" in the search results page).

## 3 Problem Formulation

Let \(A_{w}=~{}\{a_{1},a_{2},...,a_{n}\}\) be the set of concept-level actions available in a webpage \(w\). Each action \(a\in A_{w}\) is defined by an _action name_\(n_{a}\) and a set of \(K\)_parameters_\(P_{a}=~{}\{p_{1},\,p_{2},\,...,\,p_{K}\}\). Each parameter \(p\in P_{a}\) is defined by a name2 and a domain \(dom(p)\) (i.e., a set of values that can be assigned to parameter \(p\)), and can be either _closed domain_ or _open domain_.

Footnote 2: We use the same notation \(p\) to refer to the parameter and the parameter name interchangeably.

For closed-domain parameters, the domain is bounded and consists of a finite set of values that \(p\) can take; the set is imposed by the website UI, such as the available colors and sizes for a product item or the available reservation times for a restaurant.

For open-domain parameters, the domain is, in principle, unbounded, but, in practice, it consists of all words/phrases which can be extracted from an NL command \(c\). With reference to Figure 1, the "_let's go_" (search) action has \(n_{a}=\)"_let's go_" and \(P_{a}=\{\)"_time_", "_date_", "_people_", "_location, restaurant, or cuisine_"\(\}\). The first three parameters are closed domain and the last one (the search term) is open domain. The Action Extractor module (Figure 1) names actions and parameters after labels and texts appearing in the UI (or, if absent, using DOM attributes); it also automatically scrapes values of closed-domain parameters (from drop-down menus or select lists).

Given the above setting, our goal is to map an NL command \(c\) issued in \(w\) into a navigation instruction \(\mathcal{N}\), consisting of a correct _action name_\(n_{a^{*}}\) corresponding to action \(a^{*}\!\in A_{w}\) and an associated list of \(m\!\leq\!|P_{a^{*}}|\) correct _parameter-value assignments_, given by \(\{(p_{i}=v^{\prime}_{j})~{}|~{}p_{i}\in P_{a^{*}},\,v^{\prime}_{j}\in dom(p_{ i}),\,0\leq i\leq K,\,1\leq j\leq|dom(p_{i})|~{}\}\).

## 4 The FLIN Model

The task of solving the above semantic parsing problem can be decomposed into two sub-tasks: **(i) action recognition**, i.e., recognizing the action \(a\in A_{w}\) intended by \(c\), and **(ii) parameter recognition and value assignment**, i.e., deciding whether a parameter of an action is expressed in \(c\) and, if so, assigning the value to that parameter. A parameter _is expressed_ in \(c\) by a mention (word or phrase). For example, in Figure 1, "_me and my friend_" is a mention of parameter "_people_" in \(c\) and a correct parsing should map it to the domain value "_2 people_". Thus, the second sub-task involves first 

extracting a mention of a given parameter from \(c\), and then matching it against a set of domain values to find the correct value assignment. For an open-domain parameter, the extracted mention becomes the value of the parameter and no matching is needed, e.g. in Figure 1, the mention "_Italian_" should be assigned to the parameter "_location, restaurant, or cuisine_".

With reference to Figure 2, FLIN consists of four components, designed to solve the aforementioned two sub-tasks: **(1) Action Scoring**, which scores each available action with respect to the given command (SS4.1); **(2) Parameter Mention Extraction**, which extracts the mention (phrase) from the command for a given parameter (SS4.2); **(3) Parameter Value Scoring**, which scores a given mention with a closed-domain parameter value or rejects it if no domain values can be mapped to the mention (SS4.3); and **(4) Inference** (not shown in the figure), which uses the scores of actions and parameter values to infer the action-parameter-value assignment with the highest score as the predicted navigation instruction (SS4.4).

### Action Scoring

Given a command \(c\), we score each action \(a\in A_{w}\) to measure the similarity of \(a\) and \(c\)'s intent. We loop over the actions in \(A_{w}\) and their parameters to obtain a list of action name and parameters pairs (\(n_{a}\), \(P_{a}\)), and then score them with respect to \(c\).

To score each pair (\(n_{a}\), \(P_{a}\)), we learn a neural network based scoring function \(S_{a}(.)\) that computes its similarity with \(c\). We represent \(c\) as a sequence {\(w_{1},w_{2},...,w_{R}\)} of \(R\) words. To learn a vector representation of \(c\), we first convert each \(w_{i}\) into corresponding one-hot vectors \(x_{i}\), and then learn embeddings of each word using an embedding matrix \(E_{w}\in\mathbb{R}^{d\times|V|}\) as \(\mathbf{v}_{i}{=}E_{w}.x_{i}\), where \(V\) is the word vocabulary. Next, given the word embedding vectors {\(\mathbf{v}_{i}~{}|~{}1\leq i\leq|R|\)}, we learn the forward and backward representation using a BiLSTM network Schuster and Paliwal (1997). Let the final hidden state for forward LSTM and backward LSTM, after consuming {\(\mathbf{v}_{i}~{}|~{}1\leq i\leq R\)}, be \(\overrightarrow{\mathbf{h}_{R}}\) and \(\overleftarrow{\mathbf{h}_{1}}\) respectively, we learn a joint representation of \(c\) as \(\mathbf{v}_{c}=[\overrightarrow{\mathbf{h}_{R}};\overleftarrow{\mathbf{h}_{1 }}]\in\mathbb{R}^{2d}\), where [:] denotes concatenation.

Next, we learn a vector representation of (\(n_{a}\), \(P_{a}\)). We use the same word embedding matrix \(E_{w}\) and Bi-LSTM layer to encode the action name \(n_{a}\) into a vector \(\mathbf{v}_{a}{=}BiLSTM(n_{a})\). Similarly, we encode each parameter \(p\in P_{a}\) into a vector \(\mathbf{v}_{p}{=}BiLSTM(p)\), and compute the _net parameter semantics_ of action \(a\) as the mean of the parameter vectors (\(\overline{\mathbf{v}_{p}}{=}mean\{\mathbf{v}_{p}~{}|~{}p\in P_{a}\}\)). Finally, to learn the overall semantic representation of (\(n_{a}\), \(P_{a}\)), we concatenate \(\mathbf{v}_{a}\) and \(\overline{\mathbf{v}_{p}}\) and learn a combined representation using a feed-forward (FF)

Figure 2: Architecture of FLIN including Action Scoring, Parameter Mention Extraction and Parameter Value Scoring components [BERT block diagram courtesy: Devlin et al. (2019)].

 

layer as

\[{\bf v}_{ap}=tanh(W_{a}.[{\bf v}_{a};\overline{v}_{p}]+b_{a}) \tag{1}\]

where \(W_{a}\in\mathbb{R}^{4d\times 2d}\) and \(b_{a}\in\mathbb{R}^{2d}\) are weights and biases of the FF layer, respectively.

Given \({\bf v}_{c}\) and \({\bf v}_{ap}\), we compute the intent similarity between \(c\) and (\(n_{a}\), \(P_{a}\)) using cosine similarity:

\[S_{a}(c,n_{a},P_{a}) =\frac{1}{2}~{}[~{}cosine({\bf v}_{c},{\bf v}_{ap})+1] \tag{2}\] \[=\frac{1}{2}~{}(\frac{{\bf v}_{c}.{\bf v}_{ap}}{||{\bf v}_{c}|||| {\bf v}_{ap}||}+1)\]

where \(||\cdot||\) denotes euclidean norm of a vector. \(S_{a}(\cdot)\in[0,1]\) is computed for each \(a\in A_{w}\) (and is used in inference, SS4.4).

The parameters of \(S_{a}(\cdot)\) are learned by minimizing a margin-based ranking objective \(\mathcal{L}_{a}\), which encourages the scores of each positive (\(n_{a}\), \(P_{a}\)) pair to be higher than those of negative pairs in \(w\):

\[\mathcal{L}_{a}=\sum_{q\in Q^{+}}\sum_{q^{\prime}\in Q^{-}}max\{S(q^{\prime} )-S(q)+0.5,~{}0\} \tag{3}\]

where \(Q^{+}\) is a set of positive (\(n_{a}\), \(P_{a}\)) pairs in \(w\) and \(Q^{-}\) is a set of negative (\(n_{a}\), \(P_{a}\)) pairs obtained by randomly sampling action name and parameter pairs (not in \(Q^{+}\)) in \(w\).

### Parameter Mention Extraction

Given a command \(c\) and a parameter \(p\), the goal of this step is to extract the correct mention \(m_{p}\) of \(p\) from \(c\). In particular, we aim to predict the text span in \(c\) that represents \(m_{p}\). We formulate this task as a question-answering problem, where we treat \(p\) as a question, \(c\) as a paragraph, and \(m_{p}\) as the answer. We fine-tune a pre-trained BERT Devlin et al. (2019) model3 to solve this problem.

Footnote 3: [https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1)

As shown in Figure 2 (bottom-right), we represent \(p\) and \(c\) as a pair of sentences packed together into a single input sequence of the form [CLS] \(p\) [SEP] \(c\), where [CLS], [SEP] are special BERT tokens. For tokenization, we use the standard WordPiece Tokenizer Wu et al. (2016). From BERT, we obtain \(T_{i}\) as output token embedding for each token \(i\) in the packed sequence. We only introduce a mention start vector \(S\in\mathbb{R}^{H}\) and a mention end vector \(E\in\mathbb{R}^{H}\) during fine-tuning. The probability of word \(i\) being the start of the mention is computed as a dot product between \(T_{i}\) and \(S\) followed by a softmax over all of the words in \(c\): \(P_{i}=\frac{e^{S\cdot T_{i}}}{\sum_{j}e^{S\cdot T_{j}}}\). The analogous formula is used to compute the end probability \(P_{j}\). The position \(i\) (position \(j>i\)) with highest start (end) probability \(P_{i}\) (\(P_{j}\)) is predicted as start (end) index of the mention and the corresponding tokens in \(c\) are combined into a word sequence to extract \(m_{p}\). We fine-tune BERT by minimizing the training objective as the sum of the log-likelihoods of the correct start and end positions. We train BERT to output [CLS] as \(m_{p}\), if \(p\) is not expressed in \(c\) (no mention is identified, hence \(p\) is discarded from being predicted).

### Parameter Value Scoring

Once the mention \(m_{p}\) is extracted for a closed-domain parameter \(p\), we learn a neural network based scoring function \(S_{p}(\cdot)\) to score each \(p\)'s value \(v^{\prime}\in dom(p)\) with respect to \(m_{p}\). If \(p\) is open-domain, parameter value scoring is not needed.

The process is similar to that of action scoring, but, in addition to _word-level semantic similarity_, we also compute _character-level_ and _lexical-level_ similarity between \(v^{\prime}\) and \(m_{p}\). In fact, \(v^{\prime}\) and \(m_{p}\) often have partial lexical matching. For example, given the domain value "?:00 PM" for the parameter "_time_", possible mentions may be "**7**_in the evening_", "_19**_0** hrs_", "_at **7 pm_", etc., where partial lexical-level similarity is observed. However, learning word-level and character-level semantic similarities is also important as "PM" and "evening" as well as "?:00 PM" and "19:00" are lexically distant to each other, but semantically closer.

**Word-level semantic similarity.** We use the same word embedding matrix \(E_{w}\) used in action scoring to learn the word vectors for both \(m_{p}\) and \(v^{\prime}\). We use a Bi-LSTM layer (not shared with Action Scoring) to encode mention (value) into a word-level representation vectors \({\bf v}_{m}^{wd}\) (\({\bf v}_{v^{\prime}}^{wd}\)). We compute the word-level similarity between \(m_{p}\) and \(v^{\prime}\) as

\[S_{p}^{wd}(m_{p},v^{\prime})=\tfrac{1}{2}~{}[~{}cosine({\bf v}_{m}^{wd},{\bf v }_{v^{\prime}}^{wd})+1] \tag{4}\]

**Character-level semantic similarity.** We use a character embedding matrix \(E_{c}\) to learn the character vectors for each character composing the words in \(m_{p}\) and \(v^{\prime}\). To learn the character-level vector representation \({\bf v}_{m}^{char}\) of \(m_{p}\), we first learn the word vector for each word in \(m_{p}\) by composing the character vectors in sequence using an LSTM network Hochreiter and Schmidhuber ( 

1997), and then compose the word vectors for all mention words using a BiLSTM layer to obtain \({\bf v}_{m}^{char}\). Similarly, the character-level vector representation \({\bf v}_{v^{\prime}}^{char}\) of \(v^{\prime}\) is obtained. Next, we compute the character-level similarity between \(m_{p}\) and \(v^{\prime}\) as

\[S_{p}^{char}(m_{p},v^{\prime})=\tfrac{1}{2}~{}[cosine({\bf v}_{ m}^{char},{\bf v}_{v^{\prime}}^{char})+1] \tag{5}\]

**Lexical-level similarity.** We use a _fuzzy string matching score_ (using the Levenshtein distance to calculate the differences between sequences)4 and a custom _value matching score_ which is computed as the fraction of words in \(v^{\prime}\) that appear in \(m_{p}\); then we compute a linear combination of these similarity scores (each score \(\in[0,1]\)) as the net lexical-level similarity score, denoted as \(S_{p}^{lex}(m_{p},v^{\prime})\in[0,1]\).

Footnote 4: pypi.org/project/fuzzywuzzy/

**Net value-mention similarity score.** It is the mean of the three scores above: \(S_{p}(m_{p},v^{\prime})=mean\{S_{p}^{wd}(m_{p},v^{\prime}),S_{p}^{char}(m_{ p},v^{\prime}),S_{p}^{lex}(m_{p},v^{\prime})\}\).

The parameters of \(S_{p}(\cdot)\) are learned by minimizing a margin-based ranking objective \(\mathcal{L}_{p}\), which encourages the scores \(S_{p}(.)\) of each mention and positive value pair to be higher than those of mention and negative value pairs for a given \(p\), and can be defined following the previously defined \(\mathcal{L}_{a}\) (See eq. 3).

### Inference

The inference module takes the outputs of Action Scoring, Parameter Mention Extraction and Parameter Value Scoring to compute a net score \(S_{ap}(.)\) for each action \(a\in A_{w}\) and associated list of parameter value assignment combinations. Then, it uses \(S_{ap}(.)\) to predict the navigation instruction.

**Parameter value assignment.** We first infer the value to be assigned to each \(p\in P_{a}\), where the predicted value \(\hat{v_{p}}\) for a closed-domain \(p\) is given by \(\hat{v_{p}}=\arg\max_{v^{\prime}\in dom(p)}S_{p}(m_{p},v^{\prime})\)_provided_\(S_{p}(m_{p},v^{\prime})\geq\rho\). Here, \(\rho\) is a threshold score (tuned empirically) for parameter value prediction.

While performing the value assignment for \(p\), we consider \(S_{p}(m_{p},\hat{v_{p}})\) as the confidence score for \(p\)'s assignment. If \(S_{p}(m_{p},v^{\prime})<\rho\) for all \(v^{\prime}\in dom(p)\), we consider the confidence score for \(p\) as \(0\), implying \(m_{p}\) refers to a value which does not exist in \(dom(p)\); \(p\) is discarded and no value assignment is done. If \(p\) is an open-domain parameter, \(m_{p}\) is inferred as \(\hat{v_{p}}\) with a confidence score of \(1\).

If _all_\(p\in P_{a}\) are discarded from the prediction for a _parametrized_ action \(a\in A_{w}\), we discard \(a\), as \(a\) no longer becomes executable in \(w\).

Once we get all confidence scores for all value assignments for all \(p\in P_{a}\), we compute the average confidence score \(\overline{S_{p}}(P_{a})\), and consider it to be the _net parameter value assignment score_ for \(a\).

**Navigation instruction prediction.** Finally, we compute the overall score for a given action \(a\) and associated list of parameter value assignments as \(S_{ap}=\alpha*S_{a}(c,n_{a},P_{a})+(1-\alpha)*\overline{S_{p}}(P_{a})\), where \(\alpha\) is a linear combination coefficient empirically tuned. The predicted navigation instruction for command \(c\) is the action and associated parameter value assignments with the highest \(S_{ap}\) score.

## 5 Evaluation

We evaluate FLIN on nine popular websites from three representative domains: _(i) Restaurants (R), (ii) Hotels (H)_, and _(iii) Shopping (S)_. We collect labelled datasets for each website (SS5.1) and perform _in-domain cross-website evaluation_. Specifically, we train one FLIN model for each domain using one website, and test on the other (two) websites in the same domain. Ideally, a single model could be trained by using the training data of _all_ three domains and applied to all test websites, but we opt for _domain-specific_ training/evaluation to better analyze how FLIN leverages the semantic overlap of concept-level actions (that exists across in-domain websites) to generalize to new websites. We discard _cross-domain_ evaluation because the semantics of actions and parameters do not significantly overlap across our three domains.

### Experimental Setup

To train and evaluate FLIN, we collect two datasets: (i) **WebNav** consists of (English) command and navigation instruction pairs, and (ii) **DialQueries** consists of (English) user utterances extracted from existing dialogue datasets paired with navigation instructions.

To collect WebNav, given a website and a task it supports, we first identity which pages are related to the task. For example, in OpenTable we find 8 pages related to the task "making a restaurant reservation": the page for searching restaurants, for browsing search results, for viewing a restaurant's profile, for submitting a reservation, etc. Then, using our Action Extractor tool, we enumerate all actions present in each task-related page.

 

For each action the extractor provides a name, parameters and parameter values, if any. The action names are inferred from various DOM attributes (aria-label, value, placeholder, etc.) and text associated with the relevant DOM element. The goal of the Action Extractor is to label UI elements as humans see them. For example, the search box in the OpenTable website, (instead of being called "search input") is called "Location, Restaurant or Cuisine", which is in fact the placeholder text associated with that input, and what users see in the UI. Parameter values are scraped automatically from DOM select elements (e.g., option value tag). We manually inspect the output of the Action Extractor and correct possible errors (e.g., missing actions). However, for every website, we obtain a different action/parameter scheme. There is no generalized mapping between similar actions/parameters across websites as building such mapping would require significant manual effort. Table 1 reports number of pages, actions and parameters extracted for all websites used in our experiments.

With this data, we construct \(<\)page_name, action_name, [parameter_name]\(>\) triplets for all actions across all websites, and we ask two annotators to write multiple command templates corresponding to each triplet with parameter names as placeholders. A command template may be "_Book a table for \(<\)time\(>\)_". For closed-domain parameters, the Action Extractor automatically scrapes their values from webpages (e.g., {12:00 pm, 12:15 pm, etc.} for the _time_ parameter), and we ask annotators to provide paraphrases for them (e.g., "at noon"). For open-domain parameters, we ask annotators to provide example values (e.g., "pizza" for a restaurant search term). We assemble the final dataset by instantiating command templates with randomlychosen parameter value paraphrases, and then split it into train, validation and test datasets. Overall, we generate a total of 53,520 command and navigation instruction pairs. We use train and validation splits for _opentable.com_, _hotels.com_ and _rei.com_ for model training. Table 1 summarizes the sizes of the train, validation and test splits for all websites.

The second dataset, DialQueries, consists of real user queries extracted from the SGD dialogue dataset (Datasets, 2020) and from Restaurants, Hotels and Shopping "pre-built agents" of Dialogflow (dialogflow.com). We extract queries that are map-pable to our website's tasks and adapt them by replacing out-of-vocabulary mentions of restaurants, hotels, cities, etc. with equivalent entities from our vocabulary. We manually map 421, 155 and 63 dialogue queries into navigation instructions for opentable.com, hotels.com, rei.com, respectively. We use this dataset only for evaluation purposes.

**Training details.** All hyper-parameters are tuned on the validation set. Batch-size is 50. Number of training epochs for action scoring is 7, for parameter mention extraction is 3, and for parameter value scoring is 22. One negative example is sampled for \(Q^{-}\) (in Eq. 3) in every epoch. Dropout is 0.1. Hidden units and embedding size are 300. Learning rate is 1e-4. Regularization parameter is 0.001, \(\rho=0.67\) and \(\alpha=0.4\) (SS4.4). The Adam optimizer (Kingma and Ba, 2014) is used for optimization. We use a Tesla P100 GPU and TensorFlow for implementation.

**Compared models.** There is no direct baseline for this work as related approaches differ in the type of output or problem settings. As discussed in SS2, Pasupat et al. (2018) do not perform parameter recognition and value assignments. Liu et al. (2018a) require a reward function for neural policy learning. Joint intent detection and slot filling models perform multi-class classification and do not consider the current state (current webpage), thus not being able to deal with similar intents in different webpages; further, they perform slot filling (equivalent to parameter mention extraction) but do not perform parameter value assignments, thus being unable to output executable paths for web navigation.

Nonetheless, we compare FLIN against two of its variants that use its match-based semantic parsing approach but with the following differences: (i) **FLIN-sem** uses only word-level and character-level semantic similarity for parameter value scoring (no lexical similarity); and (ii) **FLIN-lex** uses

\begin{table}
\begin{tabular}{l r r r r} \hline \hline
* / / 993 \\ bookatable.co.uk (R) & 7 & 14 & 19 &
- / 587 \\ \hline hotels.com (H) & 7 & 25 & 46 & 15693 / 31377 / 1240 \\ hyrat.com (H) & 8 & 17 & 48 &
- / / 1150 \\ radissonhotels.com (H) & 7 & 20 & 42 &
- / / 1104 \\ \hline rei.com (S) & 11 & 25 & 40 & 7001 / 1399 / 933 \\ ebay.com (S) & 9 & 24 & 38 &
- / / 556 \\ macros.com (S) & 11 & 26 & 37 &
- / / 619 \\ \hline \hline \end{tabular}
\end{table}
Table 1: WebNav dataset statistics. In the last column, -/-/x denotes the website is only used for evaluation, not for training, and X is the number of commands used.
 

only lexical similarity in parameter value scoring.

**Evaluation metrics.** We use accuracy (**A-acc**) to evaluate action prediction and average F1 score (**P-F1**) to evaluate parameter prediction performance. P-F1 is computed using the average _parameter precision_ and _parameter recall_ over test commands5. Given a command, parameter precision is computed as the fraction of parameters in the predicted instruction which are correct and parameter recall as the fraction of parameters in the gold instruction which are predicted correctly. If the predicted action is incorrect or no action has been predicted for a given test command, we consider both parameter precision and recall to be 0 for the command.

Footnote 5: The P-F1 (parameter F1 score) is computed as the “harmonic mean” of average parameter precision and average parameter recall (averaged over all test queries).

We also compute (i) _Exact Match Accuracy_ (EMA), defined as the percentage of test commands where the predicted instruction exactly matches the gold navigation instruction, and (ii) 100% Precision Accuracy (**PA-100**), defined as the percentage of test commands for which the parameter precision is 1.0 and the predicted action is correct, but parameter recall \(\leq\) 1.0. 6 Similar to parameter precision and recall, while computing EMA and PA-100 for test commands, if the predicted action is incorrect or no action has been predicted for a test command, we consider both the exact match and PA-100 value to be 0 for that command.

Footnote 6: Although we formulate the mapping problem as a ranking one, we do not consider standard metrics such as mean average precision (MAP) or normalized discounted cumulative gain (NDCG) because FLIN outputs only one navigation instruction (instead of a ranked list), given that in a real web navigation system only one predicted action can be executed.

### Performance Results

Table 2 reports the performance comparison of FLIN on the WebNav dataset. We evaluate both inwebsite (model trained and tested on the same website, 2\({}^{nd}\)-5\({}^{th}\) columns) and cross-website (model trained on one website and tested on a different one, 6\({}^{th}\)-13\({}^{th}\) columns) performance. FLIN and its two variants adapt relatively well to previously-unseen websites thanks to FLIN's match-based semantic parsing approach. FLIN achieves best overall performance, and is able to adapt to new websites by achieving comparable (or higher) action accuracy (A-acc) and parameter F1 (P-F1) score. Considering PA-100, in the Restaurants domain, 75.6% of commands in the training website (OpenTable), and 60.3% (bookatable) and 82.4% (yelp) of commands in the two test websites are mapped into correct and executable actions (no wrong predictions). PA-100 is generally high also for the other two domains. EMA is lower than PA-100, as it is much harder to predict _all_ parameter value assignments correctly. Regarding the FLIN variants, both FLIN-sem and FLIN-lex generally perform worse than FLIN because by combining both lexical and semantic similarity FLIN can be more accurate in doing parameter value assignments and generalize better.

From a generalizability point of view, the most challenging domain is Hotels. While the performance of action prediction (A-acc) for Hotels is in the 45.5%-93.9% range, EMA is in the 14.6%-64.3% range. The drop is mainly due to commands with many parameters (e.g., check-in date, check-out date, number of rooms, etc.), definitely more than in the queries for the other two domains. Shopping is more challenging than Restaurants because

\begin{table}
\begin{tabular}{l|c c c c|c c c c|c c c c} \hline \hline  & A-acc & P-F1 & EMA & PA-100 & A-acc & P-F1 & EMA & PA-100 & A-acc & P-F1 & EMA & PA-100 \\ \hline  & R: opentable.com (training website) & \multicolumn{4}{c|}{R: yelp.com} & \multicolumn{4}{c}{R: bookatable.co.uk} \\ FLIN-sem & 0.935 & 0.499 & 0.306 & 0.310 & **0.949** & 0.483 & 0.321 & 0.380 & **0.954** & 0.415 & 0.318 & 0.339 \\ FLIN-lex & 0.618 & 0.496 & 0.272 & 0.582 & 0.493 & 0.423 & 0.243 & 0.488 & 0.654 & 0.348 & 0.235 & 0.362 \\ FLIN & **0.937** & **0.815** & **0.679** & **0.756** & 0.926 & **0.836** & **0.703** & **0.824** & 0.732 & **0.639** & **0.543** & **0.603** \\ \hline  & H: hotels.com (training website) & \multicolumn{4}{c|}{H: hyatt.com} & \multicolumn{4}{c}{H: radissonhotlets.com} \\ FLIN-sem & 0.933 & 0.749 & 0.423 & 0.468 & **0.806** & 0.505 & 0.154 & 0.286 & **0.883** & **0.630** & **0.221** & 0.399 \\ FLIN-lex & 0.859 & 0.720 & 0.311 & 0.811 & 0.675 & 0.467 & 0.101 & 0.536 & 0.364 & 0.250 & 0.067 & 0.325 \\ FLIN & **0.939** & **0.874** & **0.643** & **0.869** & 0.740 & **0.551** & **0.187** & **0.570** & 0.455 & 0.353 & 0.146 & **0.413** \\ \hline  & S: rei.com (training website) & \multicolumn{4}{c|}{S: ebay.com} & \multicolumn{4}{c}{S: marcsys.com} \\ FLIN-sem & 0.852 & 0.786 & **0.668** & 0.704 & 0.559 & 0.414 & 0.239 & 0.298 & 0.833 & 0.526 & 0.119 & 0.218 \\ FLIN-lex & 0.875 & 0.781 & 0.530 & **0.828** & **0.902** & 0.721 & 0.248 & 0.437 & 0.873 & **0.604** & 0.166 & **0.411** \\ FLIN & **0.913** & **0.826** & **0.668** & 0.782 & 0.897 & **0.729** & **0.327** & **0.453** & **0.878** & 0.598 & **0.176** & 0.358 \\ \hline \hline \end{tabular}
\end{table}
Table 2: In-website and cross-website performance comparison of FLIN variants. WebNav is used for training and/or testing, as specified. All metric scores are scaled out of 1.0.

 

while in the Restaurants domain, FLIN must deal with actions/parameters that relate to the _same_ entity type (restaurant) with a relatively-contained vocabulary, shopping products can range so widely to be effectively _different entity types_ with a diverse set of actions/parameters that can vary significantly across websites. For a more detailed discussion of both aspects see SS5.3.

We also test FLIN on the real user queries of the DialQueries dataset available for three websites. As Table 3 shows, despite FLIN not being trained on DialQueries, overall its A-acc is above 50% and its PA-100 is above 46% which demonstrate the robustness of FLIN in the face of new commands.

### Error Analysis

We randomly sampled 135 wrongly-predicted WebNav commands (15 for each of the 9 websites), and classified them into 5 error types (see Table 4).

Overall, 13% of the failures were cases in which an action was not predicted (e.g., for the command "_only eight options_" with ground truth "_filter by size_{'size'='8'}", no action was predicted). 29% of the failures were action miss-predictions (the predicted action did not match the gold action) mainly caused by multiple actions in the given webpage having overlapping semantics. E.g., the command "_options for new york for just 6 people and 2 kids_" got mapped to "_select hot destination_{'destination'= 'new york'}" instead of "_find hotel_{'adults'='6'; 'children'='2'; 'destination'='new york'}". Similarly, "_apply the kids' shoes filter_" got mapped to "_filter by gender_{'gender'= 'kids'}" instead of "_filter by category_{'category': 'kids footwear'}".

Together with action miss-predictions, failures in identifying closed-domain parameters (third row in Table 4) were the most common, especially in hotels and shopping websites. This is because these in-domain websites tend to have a more diverse action and parameter space than that for restaurant websites, thus leading to action and parameter types that were not observed in training data. For example, the search action in Hyatt has _special rates_ and _use points_ parameters, not present in that of Hotels.com (training site); or eBay has an action _filter by style_ not present in Rei (training site).

Failures in predicting the value of a correctly-identified closed-domain parameter mention (forth row in the table) were often due to morphological variations in parameter values not frequently observed in training (e.g. "_8:00 in the evening_" got mapped to the value '_18:00_' instead of '_20:00_').

Errors in extracting open-domain parameters were due to parameter names too generic (e.g. "_search keyword_"), extracted mentions partially matching gold mentions (e.g., "_hyatt_" vs. the gold mention "_hydatt regency grand cypress_"), or multiple formats of the parameter values (e.g., various formats for _phone number_ or _zip code_).

## 6 Conclusion

To generalize to many websites, NL-guided web navigation assistants require an NL interface that can work with new website UIs without being re-trained each time. To this end, we proposed FLIN, a matching-based semantic parsing approach that maps user commands to concept-level actions. While various optimizations are possible, FLIN adapted well to new websites and delivered good performance. We have used it in restaurant, shopping and hotels websites, but its design can apply to more domains.

## Acknowledgments

We would like to thank Jason Kace (Microsoft) for many constructive discussions and for designing and implementing the _Action Extractor_ and _Action Executor_ modules used by the NL-guided web navigation system.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Error Type & \%(R) & \%(H) & \%(S) & \%(all) \\ \hline Action not predicted & \(17.7\) & \(8.9\) & \(13.3\) & \(13.3\) \\ \hline Action miss-predicted & \(17.7\) & \(28.8\) & \(40.0\) & \(28.9\) \\ \cline{2-5} Failed to identify closed-domain parameter & \(20.0\) & \(57.7\) & \(31.1\) & \(36.3\) \\ \hline Closed-domain parameter value miss-predicted & \(31.1\) & \(6.7\) & \(4.4\) & \(14.1\) \\ \hline Fail to extract open-domain parameter value & \(15.5\) & \(11.1\) & \(24.4\) & \(17.0\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Error analysis based on 135 test commands from WebNav. Columns do not sum up to 100% as multiple parameters in the same command may be problematic for different reasons.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline  & A-acc & P-F1 & EMA & PA-100 \\ \hline opentable.com (R) & 0.719 & 0.582 & 0.565 & 0.584 \\ hotels.com (H) & 0.730 & 0.514 & 0.381 & 0.500 \\ rein.com (S) & 0.507 & 0.464 & 0.428 & 0.460 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance on the DialQueries dataset (FLIN models trained on the WebNav dataset).

 



## References

* Volume 2_, AAAI'07, pages 1514-1519. AAAI Press.
* Artzi and Zettlemoyer (2013) Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transactions of the Association for Computational Linguistics_, pages 49-62.
* Barman et al. (2016) Shaon Barman, Sarah Chasins, Rastislav Bodik, and Sumit Gulwani. 2016. Ringer: Web Automation by Demonstration. In _Proc. f the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications_, OOPSLA 2016, pages 748-764. ACM.
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1533-1544.
* Volume 1_, ACL. '09, pages 82-90, USA. Association for Computational Linguistics.
* Chen et al. (2020a) Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020a. Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning. In _Proc. of the ACM/IEEE 42nd International Conference on Software Engineering_, ICSE '20, pages 322-334.
* Chen et al. (2020b) Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020b. Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? In _Proc. of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE'20)_, ESEC/FSE 2020, pages 1202-1214, New York, NY. ACM.
* Chen et al. (2019) Qian Chen, Zhu Zhuo, and W. Wang. 2019. BERT for Joint Intent Classification and Slot Filling. _ArXiv_, abs/1902.10909.
* Datasets (2020) Google Research Datasets. 2020. The schema-guided dialogue dataset. [https://github.com/google-research-datasets/dstc8-schema-guided-dialogue](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue).
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1_, pages 4171-4186. Association for Computational Linguistics.
* Fazzini et al. (2018) Mattia Fazzini, Martin Prammer, Marcelo d'Amorim, and Alessandro Orso. 2018. Automatically Translating Bug Reports into Test Cases for Mobile Apps. In _Proc. of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis_, ISSTA 2018, page 141-152, New York, NY, USA. Association for Computing Machinery.
* Ferre (2017) Sebastien Ferre. 2017. Sparklis: An expressive query builder for SPARQL endpoints with guidance in natural language. _Semantic Web_, 8(3):405-418.
* Frank and Goodman (2012) Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. _Science_, 336(6084):998-998.
* Fried et al. (2018) Daniel Fried, Jacob Andreas, and Dan Klein. 2018. Unified Pragmatic Models for Generating and Following Instructions. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1_, pages 1951-1963. Association for Computational Linguistics.
* Gao et al. (2015) Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G. Karahalios. 2015. DataTone: Managing Ambiguity in Natural Language Interfaces for Data Visualization. In _Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology_, UIST '15, pages 489-500. Association for Computing Machinery.
* Garcia et al. (2018) Francisco Javier Chiyah Garcia, David A. Robb, Xingkun Liu, Atanas Lakkov, Pedro Patron, and Helen F. Hastie. 2018. Explain Yourself: A Natural Language Interface for Scrutable Autonomous Robots. _CoRR_, abs/1803.02088.
* Golland et al. (2010) Dave Golland, Percy Liang, and Dan Klein. 2010. A Game-Theoretic Approach to Generating Spatial Descriptions. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_, EMNLP '10, pages 410-419, USA. Association for Computational Linguistics.
* Guo et al. (2014) D. Guo, G. Tur, W. Yih, and G. Zweig. 2014. Joint semantic utterance classification and slot filling with recursive neural networks. In _2014 IEEE Spoken Language Technology Workshop (SLT)_, pages 554-559.
* Guu et al. (2017) Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. 2017. From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1051-1062, Vancouver, Canada. Association for Computational Linguistics

* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Janner et al. (2018) Michael Janner, Karthik Narasimhan, and Regina Barzilay. 2018. Representation learning for grounded spatial reasoning. _Transactions of the Association of Computational Linguistics_, pages 49-61.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Lau et al. (2009) Tessa Lau, Clemens Drews, and Jeffrey Nichols. 2009. Interpreting Written How-to Instructions. In _Proc. of the 21st International Joint Conference on Artificial Intelligence_, IJCAI '09, pages 1433-1438, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
* Lawrence and Riezler (2016) Carolin Lawrence and Stefan Riezler. 2016. NLmaps: A natural language interface to query OpenStreetMap. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations_, pages 6-10.
* Liu and Lane (2016) Bing Liu and Ian Lane. 2016. Joint Online Spoken Language Understanding and Language Modeling With Recurrent Neural Networks. In _Proc. of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 22-30. Association for Computational Linguistics.
* Liu et al. (2018a) Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018a. Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration. In _6th International Conference on Learning Representations, ICLR 2018_.
* Liu et al. (2018b) Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018b. Learning Design Semantics for Mobile Apps. In _Proc. of the 31st Annual ACM Symposium on User Interface Software and Technology_, UIST '18, pages 569-579, New York, NY, USA. Association for Computing Machinery.
* Nguyen and Csallner (2015) Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI. In _Proc. of the 30th IEEE/ACM International Conference on Automated Software Engineering_, ASE '15, pages 248-259. IEEE Press.
* Pasupat et al. (2018) Panupong Pasupat, Tian-Shun Jiang, Evan Liu, Kelvin Guu, and Percy Liang. 2018. Mapping natural language commands to web elements. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4970-4976. Association for Computational Linguistics.
* Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. _IEEE transactions on Signal Processing_, 45(11):2673-2681.
* Selenium (2020) Selenium. 2020. [https://www.selenium.dev](https://www.selenium.dev).
* Setlur et al. (2016) Vidya Setlur, Sarah E. Battersby, Melanie Tony, Rich Gossweiler, and Angel X. Chang. 2016. Eviza: A Natural Language Interface for Visual Analysis. In _Proceedings of the 29th Annual Symposium on User Interface Software and Technology_, UIST '16, page 365-377. Association for Computing Machinery.
* Shi et al. (2017a) Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. 2017a. World of Bits: An Open-Domain Platform for Web-Based Agents. In _Proc. of the 34th International Conference on Machine Learning_, volume 70, pages 3135-3144.
* Shi et al. (2017b) Tianlin Tim Shi, Andrej Karpathy, Linxi Jim Fan, Jonathan Hernandez, and Percy Liang. 2017b. World of bits: An open-domain platform for web-based agents. In _Proc. of the 34th International Conference on Machine Learning-Volume 70_, pages 3135-3144. JMLR. org.
* Smith et al. (2013) Nathaniel J Smith, Noah Goodman, and Michael Frank. 2013. Learning and using language via recursive pragmatic reasoning about other agents. In _Advances in neural information processing systems_, pages 3039-3047.
* Su et al. (2017) Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, Michael Gamon, and Mark Encarnacion. 2017. Building natural language interfaces to web APIs. In _Proc. of the 2017 ACM on Conference on Information and Knowledge Management_, pages 177-186. ACM.
* Su et al. (2018) Yu Su, Ahmed Hassan Awadallah, Maioosen Wang, and Ryen W White. 2018. Natural language interfaces with fine-grained user interaction: A case study on web APIs. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 855-864.
* Crunch (2019) Tech Crunch. 2019. Google is bringing AI assistant Duplex to the web. [https://techcrunch.com/2019/05/07/google-is-bringing-ai-assistant-duplex-to-the-web/](https://techcrunch.com/2019/05/07/google-is-bringing-ai-assistant-duplex-to-the-web/).
* Tellexx et al. (2011) Stefanic Tellexx, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation. In _Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence_, AAAI'11, pages 1507-1514. AAAI Press.
* Thummalapenta et al. (2012) Suresh Thummalapenta, Saurabh Sinha, Nimit Singhana, and Satish Chandra. 2012. Automating Test Automation. In _Proceedings of the 34th International Conference on Software Engineering_, ICSE '12, page 881-891. IEEE Press.
* Utama et al. (2018) Prasetyta Utama, Nathaniel Weir, Fuat Basik, Carsten Binnig, Ugur Cetintemel, Benjamin Hattasch, Amir Ilkhechi, Shekar Ramaswamy, and Arf Usta. 2018. An end-to-end neural natural language interface for databases. _arXiv preprint arXiv:1804.00401_ 

Sida I. Wang, Percy Liang, and Christopher D. Manning (2016)Learning Language Games through Interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2368-2378. External Links: Document Cited by: SS1.
* Kyle Williams et al. (2019)Automatic Task Completion Flows from Web APIs. In Proc. of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Vol. , pp. 1009-1012. External Links: Document Cited by: SS1.
* Yonghui Wu et al. (2016)Google's neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. Cited by: SS1.
* Wen-tau Yih et al. (2015)Semantic parsing via staged query graph generation: question answering with knowledge base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1321-1331. External Links: Link, Document Cited by: SS1.
* J. M. Zelle and R. J. Mooney (1996)Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pp. 1050-1055. External Links: ISBN 978-1-4503-3213-X, Link, Document Cited by: SS1.
* Luke Zettlemoyer and Michael Collins (2007)Online Learning of Relaxed CCG Grammars for Parsing to Logical Form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 678-687. External Links: Link, Document Cited by: SS1.
* Yu Zhao et al. (2019)ReCDroid: automatically Reproducing Android Application Crashes from Bug Reports. In Proceedings of the 41st International Conference on Software Engineering, ICSE '19, Berlin, Germany, pp. 128-139. External Links: ISBN 978-3-319-5033-3, Link, Document Cited by: SS1.