SlowGuess/NAACL_2021_OCR huggingface dataset:
Navigation instruction prediction. Finally, we compute the overall score for a given action  $a$  and associated list of parameter value assignments as  $S_{ap} = \alpha * S_{a}(c, n_{a}, P_{a}) + (1 - \alpha) * \overline{S_{p}}(P_{a})$ , where  $\alpha$  is a linear combination coefficient empirically tuned. The predicted navigation instruction for command  $c$  is the action and associated parameter value assignments with the highest  $S_{ap}$  score.
We evaluate FLIN on nine popular websites from three representative domains: (i) Restaurants  $(R)$ , (ii) Hotels  $(H)$ , and (iii) Shopping  $(S)$ . We collect labelled datasets for each website ( $\S 5.1$ ) and perform in-domain cross-website evaluation. Specifically, we train one FLIN model for each domain using one website, and test on the other (two) websites in the same domain. Ideally, a single model could be trained by using the training data of all three domains and applied to all test websites, but we opt for domain-specific training/evaluation to better analyze how FLIN leverages the semantic overlap of concept-level actions (that exists across in-domain websites) to generalize to new websites. We discard cross-domain evaluation because the semantics of actions and parameters do not significantly overlap across our three domains.
To train and evaluate FLIN, we collect two datasets: (i) WebNav consists of (English) command and navigation instruction pairs, and (ii) DialQueries consists of (English) user utterances extracted from existing dialogue datasets paired with navigation instructions.
To collect WebNav, given a website and a task it supports, we first identity which pages are related to the task. For example, in OpenTable we find 8 pages related to the task "making a restaurant reservation": the page for searching restaurants, for browsing search results, for viewing a restaurant's profile, for submitting a reservation, etc. Then, using our Action Extractor tool, we enumerate all actions present in each task-related page.
4 pypi.org/project/fuzzywuzzy/
For each action the extractor provides a name, parameters and parameter values, if any. The action names are inferred from various DOM attributes (aria-label, value, placeholder, etc.) and text associated with the relevant DOM element. The goal of the Action Extractor is to label UI elements as humans see them. For example, the search box in the OpenTable website, (instead of being called "search input") is called "Location, Restaurant or Cuisine", which is in fact the placeholder text associated with that input, and what users see in the UI. Parameter values are scraped automatically from DOM select elements (e.g., option value tag). We manually inspect the output of the Action Extractor and correct possible errors (e.g., missing actions). However, for every website, we obtain a different action/parameter scheme. There is no generalized mapping between similar actions/parameters across websites as building such mapping would require significant manual effort. Table 1 reports number of pages, actions and parameters extracted for all websites used in our experiments.
With this data, we construct <page_name, action_name, [parameter_name]> triplets for all actions across all websites, and we ask two annotators to write multiple command templates corresponding to each triplet with parameter names as placeholders. A command template may be "Book a table for <time)". For closed-domain parameters, the Action Extractor automatically scrapes their values from webpages (e.g., {12:00 pm, 12:15 pm, etc.} for the time parameter), and we ask annotators to provide paraphrases for them (e.g., "at noon"). For open-domain parameters, we ask annotators to provide example values (e.g., "pizza" for a restaurant search term). We assemble the final dataset by instantiating command templates with randomly-chosen parameter value paraphrases, and then split it into train, validation and test datasets. Overall, we generate a total of 53,520 command and navigation instruction pairs. We use train and validation splits for opentable.com, hotels.com and rei.com for model training. Table 1 summarizes the sizes of the train, validation and test splits for all websites.
The second dataset, DialQueries, consists of real user queries extracted from the SGD dialogue dataset (Datasets, 2020) and from Restaurants, Hotels and Shopping "pre-built agents" of Dialogflow (dialogflow.com). We extract queries that are mapped to our website's tasks and adapt them by replacing out-of-vocabulary mentions of restaurants,
Table 1: WebNav dataset statistics. In the last column, -/x denotes the website is only used for evaluation, not for training, and X is the number of commands used.
Website (Domain)	# Pg	# Act	# Par	Train / Valid / Test
opentable.com (R)	8	26	38	14332 / 2865 / 1911
yelp.com (R)	8	14	25	- / - / 993
bookatable.co.uk (R)	7	14	19	- / - / 587
hotels.com (H)	7	25	46	15693 / 3137 / 1240
hyatt.com (H)	8	17	48	- / - / 1150
radissonhotels.com (H)	7	20	42	- / - / 1104
rei.com (S)	11	25	40	7001 / 1399 / 933
ebay.com (S)	9	24	38	- / - / 556
macys.com (S)	11	26	37	- / - / 619
hotels, cities, etc. with equivalent entities from our vocabulary. We manually map 421, 155 and 63 dialogue queries into navigation instructions for opentable.com, hotels.com, rei.com, respectively. We use this dataset only for evaluation purposes.
Training details. All hyper-parameters are tuned on the validation set. Batch-size is 50. Number of training epochs for action scoring is 7, for parameter mention extraction is 3, and for parameter value scoring is 22. One negative example is sampled for  $Q^{-}$  (in Eq. 3) in every epoch. Dropout is 0.1. Hidden units and embedding size are 300. Learning rate is 1e-4. Regularization parameter is 0.001,  $\rho = 0.67$  and  $\alpha = 0.4$  (\$4.4). The Adam optimizer (Kingma and Ba, 2014) is used for optimization. We use a Tesla P100 GPU and TensorFlow for implementation.
Compared models. There is no direct baseline for this work as related approaches differ in the type of output or problem settings. As discussed in §2, Pasupat et al. (2018) do not perform parameter recognition and value assignments. Liu et al. (2018a) require a reward function for neural policy learning. Joint intent detection and slot filling models perform multi-class classification and do not consider the current state (current webpage), thus not being able to deal with similar intents in different webpages; further, they perform slot filling (equivalent to parameter mention extraction) but do not perform parameter value assignments, thus being unable to output executable paths for web navigation.
Nonetheless, we compare FLIN against two of its variants that use its match-based semantic parsing approach but with the following differences: (i) FLIN-sem uses only word-level and character-level semantic similarity for parameter value scoring (no lexical similarity); and (ii) FLIN-lex uses
Table 2: In-website and cross-website performance comparison of FLIN variants. WebNav is used for training and/or testing, as specified. All metric scores are scaled out of 1.0.
A-acc	P-F1	EMA	PA-100	A-acc	P-F1	EMA	PA-100	A-acc	P-F1	EMA	PA-100
FLIN-sem	R: opentable.com (training website)	R: yelp.com	R: bookatable.co.uk
0.935	0.499	0.306	0.310	0.949	0.483	0.321	0.380	0.954	0.415	0.318	0.339
FLIN-lex	0.618	0.496	0.272	0.582	0.493	0.423	0.243	0.488	0.654	0.348	0.235	0.362
FLIN	0.937	0.815	0.679	0.756	0.926	0.836	0.703	0.824	0.732	0.639	0.543	0.603
FLIN-sem	H: hotels.com (training website)	H: hyatt.com	H: radissonhotels.com
0.933	0.749	0.423	0.468	0.806	0.505	0.154	0.286	0.883	0.630	0.221	0.399
FLIN-lex	0.859	0.720	0.311	0.811	0.675	0.467	0.101	0.536	0.364	0.250	0.067	0.325
FLIN	0.939	0.874	0.643	0.869	0.740	0.551	0.187	0.570	0.455	0.353	0.146	0.413
FLIN-sem	S: rei.com (training website)	S: ebay.com	S: macys.com
0.852	0.786	0.668	0.704	0.559	0.414	0.239	0.298	0.833	0.526	0.119	0.218
FLIN-lex	0.875	0.781	0.530	0.828	0.902	0.721	0.248	0.437	0.873	0.604	0.166	0.411
FLIN	0.913	0.826	0.668	0.782	0.897	0.729	0.327	0.453	0.878	0.598	0.176	0.358
only lexical similarity in parameter value scoring.
Evaluation metrics. We use accuracy (A-acc) to evaluate action prediction and average F1 score (P-F1) to evaluate parameter prediction performance. P-F1 is computed using the average parameter precision and parameter recall over test commands<sup>5</sup>. Given a command, parameter precision is computed as the fraction of parameters in the predicted instruction which are correct and parameter recall as the fraction of parameters in the gold instruction which are predicted correctly. If the predicted action is incorrect or no action has been predicted for a given test command, we consider both parameter precision and recall to be 0 for the command.
We also compute (i) Exact Match Accuracy (EMA), defined as the percentage of test commands where the predicted instruction exactly matches the gold navigation instruction, and (ii)  $100\%$  Precision Accuracy (PA-100), defined as the percentage of test commands for which the parameter precision is 1.0 and the predicted action is correct, but parameter recall  $\leq 1.0$ . Similar to parameter precision and recall, while computing EMA and PA-100 for test commands, if the predicted action is incorrect or no action has been predicted for a test command, we consider both the exact match and PA-100 value to be 0 for that command.
5.2 Performance Results
Table 2 reports the performance comparison of FLIN on the WebNav dataset. We evaluate both in-website (model trained and tested on the same website,  $2^{nd} - 5^{th}$  columns) and cross-website (model trained on one website and tested on a different one,  $6^{th} - 13^{th}$  columns) performance. FLIN and its two variants adapt relatively well to previously-unseen websites thanks to FLIN's match-based semantic parsing approach. FLIN achieves best overall performance, and is able to adapt to new websites by achieving comparable (or higher) action accuracy (A-acc) and parameter F1 (P-F1) score. Considering PA-100, in the Restaurants domain,  $75.6\%$  of commands in the training website (OpenTable), and  $60.3\%$  (bookatable) and  $82.4\%$  (yelp) of commands in the two test websites are mapped into correct and executable actions (no wrong predictions). PA-100 is generally high also for the other two domains. EMA is lower than PA-100, as it is much harder to predict all parameter value assignments correctly. Regarding the FLIN variants, both FLIN-sem and FLIN-lex generally perform worse than FLIN because by combining both lexical and semantic similarity FLIN can be more accurate in doing parameter value assignments and generalize better.
From a generalizability point of view, the most challenging domain is Hotels. While the performance of action prediction (A-acc) for Hotels is in the  $45.5\% - 93.9\%$  range, EMA is in the  $14.6\% - 64.3\%$  range. The drop is mainly due to commands with many parameters (e.g., check-in date, checkout date, number of rooms, etc.), definitely more than in the queries for the other two domains. Shopping is more challenging than Restaurants because
5The P-F1 (parameter F1 score) is computed as the "harmonic mean" of average parameter precision and average parameter recall (averaged over all test queries).
Although we formulate the mapping problem as a ranking one, we do not consider standard metrics such as mean average precision (MAP) or normalized discounted cumulative gain (NDCG) because FLIN outputs only one navigation instruction (instead of a ranked list), given that in a real web navigation system only one predicted action can be executed.
Table 3: Performance on the DialQueries dataset (FLIN models trained on the WebNav dataset).
A-acc	P-F1	EMA	PA-100
opetable.com (R)	0.719	0.582	0.565	0.584
hotels.com (H)	0.730	0.514	0.381	0.500
rei.com (S)	0.507	0.464	0.428	0.460
while in the Restaurants domain, FLIN must deal with actions/parameters that relate to the same entity type (restaurant) with a relatively-contained vocabulary, shopping products can range so widely to be effectively different entity types with a diverse set of actions/parameters that can vary significantly across websites. For a more detailed discussion of both aspects see §5.3.
We also test FLIN on the real user queries of the DialQueries dataset available for three websites. As Table 3 shows, despite FLIN not being trained on DialQueries, overall its A-acc is above  $50\%$  and its PA-100 is above  $46\%$  which demonstrate the robustness of FLIN in the face of new commands.
We randomly sampled 135 wrongly-predicted WebNav commands (15 for each of the 9 websites), and classified them into 5 error types (see Table 4).
Overall,  $13\%$  of the failures were cases in which an action was not predicted (e.g., for the command "only eight options" with ground truth "filter by size{'size'=8'}", no action was predicted).  $29\%$  of the failures were action miss-predictions (the predicted action did not match the gold action) mainly caused by multiple actions in the given webpage having overlapping semantics. E.g., the command "options for new york for just 6 people and 2 kids" got mapped to "select hot destination{'destination'= 'new york'}" instead of "find hotel{'adults'= '6'; 'children'= '2'; 'destination'= 'new york')". Similarly, "apply the kids' shoes filter" got mapped to "filter by gender{'gender'= 'kids'}" instead of "filter by category{'category': 'kids footwear'}".
Together with action miss-predictions, failures in identifying closed-domain parameters (third row in Table 4) were the most common, especially in hotels and shopping websites. This is because these in-domain websites tend to have a more diverse action and parameter space than that for restaurant websites, thus leading to action and parameter types that were not observed in training data. For example, the search action in Hyatt has special rates
Table 4: Error analysis based on 135 test commands from WebNav. Columns do not sum up to  $100\%$  as multiple parameters in the same command may be problematic for different reasons.
Error Type	%(R)	%(H)	%(S)	%(all)
Action not predicted	17.7	8.9	13.3	13.3
Action miss-predicted	17.7	28.8	40.0	28.9
Failed to identify closed-domain parameter	20.0	57.7	31.1	36.3
Closed-domain parameter value miss-predicted	31.1	6.7	4.4	14.1
Fail to extract open-domain parameter value	15.5	11.1	24.4	17.0
and use points parameters, not present in that of Hotels.com (training site); or eBay has an action filter by style not present in Rei (training site).
Failures in predicting the value of a correctly-identified closed-domain parameter mention (forth row in the table) were often due to morphological variations in parameter values not frequently observed in training (e.g. "8:00 in the evening" got mapped to the value '18:00' instead of '20:00').
Errors in extracting open-domain parameters were due to parameter names too generic (e.g. "search keyword"), extracted mentions partially matching gold mentions (e.g., "hyatt" vs. the gold mention "hyatt regency grand cypress"), or multiple formats of the parameter values (e.g., various formats for phone number or zip code).
To generalize to many websites, NL-guided web navigation assistants require an NL interface that can work with new website UIs without being re-trained each time. To this end, we proposed FLIN, a matching-based semantic parsing approach that maps user commands to concept-level actions. While various optimizations are possible, FLIN adapted well to new websites and delivered good performance. We have used it in restaurant, shopping and hotels websites, but its design can apply to more domains.
We would like to thank Jason Kace (Microsoft) for many constructive discussions and for designing and implementing the Action Extractor and Action Executor modules used by the NL-guided web navigation system.
James Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary Swift, and William Taysom. 2007. PLOW: A Collaborative Task Learning Agent. In Proceedings of the 22nd National Conference on Artificial Intelligence - Volume 2, AAAI'07, pages 1514-1519. AAAI Press.
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, pages 49-62.
Shaon Barman, Sarah Chasins, Rastislav Bodik, and Sumit Gulwani. 2016. Ringer: Web Automation by Demonstration. In Proc. f the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, pages 748-764. ACM.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544.
S. R. K. Branavan, Harr Chen, Luke S. Zettlemoyer, and Regina Barzilay. 2009. Reinforcement Learning for Mapping Instructions to Actions. In Proc. of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1, ACL '09, pages 82-90, USA. Association for Computational Linguistics.
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020a. Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning. In Proc. of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE '20, pages 322-334.
Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020b. Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? In Proc. of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE'20), ESEC/FSE 2020, pages 1202-1214, New York, NY. ACM.
Qian Chen, Zhu Zhuo, and W. Wang. 2019. BERT for Joint Intent Classification and Slot Filling. ArXiv, abs/1902.10909.
Google Research Datasets. 2020. The schemaguided dialogue dataset. https://github.com/google-research-datasets/dstc8-schema-guided-dialogue.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 4171-4186. Association for Computational Linguistics.
Mattia Fazzini, Martin Prammer, Marcelo d'Amorim, and Alessandro Orso. 2018. Automatically Translating Bug Reports into Test Cases for Mobile Apps. In Proc. of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, page 141-152, New York, NY, USA. Association for Computing Machinery.
Sebastien Ferre. 2017. Sparklis: An expressive query builder for SPARQL endpoints with guidance in natural language. Semantic Web, 8(3):405-418.
Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998-998.
Daniel Fried, Jacob Andreas, and Dan Klein. 2018. Unified Pragmatic Models for Generating and Following Instructions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 1951-1963. Association for Computational Linguistics.
Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G. Karahalios. 2015. DataTone: Managing Ambiguity in Natural Language Interfaces for Data Visualization. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, UIST '15, pages 489-500. Association for Computing Machinery.
Francisco Javier Chiyah Garcia, David A. Robb, Xingkun Liu, Atanas Laskov, Pedro Patron, and Helen F. Hastie. 2018. Explain Yourself: A Natural Language Interface for Scrutable Autonomous Robots. CoRR, abs/1803.02088.
Dave Golland, Percy Liang, and Dan Klein. 2010. A Game-Theoretic Approach to Generating Spatial Descriptions. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP '10, pages 410-419, USA. Association for Computational Linguistics.
D. Guo, G. Tur, W. Yih, and G. Zweig. 2014. Joint semantic utterance classification and slot filling with recursive neural networks. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 554-559.
Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. 2017. From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1051-1062, Vancouver, Canada. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Michael Janner, Karthik Narasimhan, and Regina Barzilay. 2018. Representation learning for grounded spatial reasoning. Transactions of the Association of Computational Linguistics, pages 49-61.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Tessa Lau, Clemens Drews, and Jeffrey Nichols. 2009. Interpreting Written How-to Instructions. In Proc. of the 21st International Joint Conference on Artificial Intelligence, IJCAI '09, pages 1433-1438, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Carolin Lawrence and Stefan Riezler. 2016. NLmaps: A natural language interface to query OpenStreetMap. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 6-10.
Bing Liu and Ian Lane. 2016. Joint Online Spoken Language Understanding and Language Modeling With Recurrent Neural Networks. In Proc. of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 22-30. Association for Computational Linguistics.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018a. Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration. In 6th International Conference on Learning Representations, ICLR 2018.
Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018b. Learning Design Semantics for Mobile Apps. In Proc. of the 31st Annual ACM Symposium on User Interface Software and Technology, UIST '18, pages 569-579, New York, NY, USA. Association for Computing Machinery.
Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI. In Proc. of the 30th IEEE/ACM International Conference on Automated Software Engineering, ASE '15, pages 248-259. IEEE Press.
Panupong Pasupat, Tian-Shun Jiang, Evan Liu, Kelvin Guu, and Percy Liang. 2018. Mapping natural language commands to web elements. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4970-4976. Association for Computational Linguistics.
Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673-2681.