SlowGuess/NAACL_2021_OCR huggingface dataset:
With an effective semantic parser to ThingTalk, we next measure the grounding accuracy: the percent of correctly identified element_ids from the 252 natural language commands referring to web elements in the RUSS test set. As shown in Table 4, RUSS achieves an accuracy of  $63.6\%$ .  $81.4\%$  of the instructions are parsed correctly, and  $77.9\%$  of the correct parses are grounded accurately. Had the semantic parser been correct  $100\%$  of the time, the Grounding Model would achieve an accuracy of  $73.0\%$ . The semantic parser is more likely to correctly parse simple instructions such as "click sign in", which are also generally easier for the Grounding Model, explaining the delta between  $77.9\%$  and  $73.0\%$ .
We create an End-to-end Baseline model to compare against the 2-step approach of RUSS. Here, we represent web elements using RUSS's feature elements as before. However, we do not parse the natural language sentences into their input features in RUSS, but is left intact as input to
Table 4: RUSS outperforms state-of-the-art PhraseNode in the grounding subtask on the RUSS Evaluation test set.
Reasoning	RUSS	PhraseNode
Type	67.8%	61.5%
Input	75.6%	60.4%
Relational	70.0%	53.5%
Spatial	36.7%	30.3%
Table 5: Grounding Accuracy Comparison of RUSS and PhraseNode by Reasoning type on the RUSS Evaluation test set.
Sentence-Bert to compute its embedding. Like Section 4.3, the element sharing the closest embedding with the input sentence is returned. This end-to-end baseline model performs with  $12.6\%$  less accuracy than RUSS, illustrating the benefits of using a semantic parser.
To compare our grounding model with state-of-the-art results, we also replicate the best performing embedding model from (Pasupat et al., 2018), which we reference as PhraseNode. The webpage features used as inputs in PhraseNode are a subset of our representation. PhraseNode achieves an accuracy of  $46.5\%$ , which is  $4.6\%$  worse than our Baseline and  $17.2\%$  lower than RUSS. We show that the combination of a high-performance semantic parser and a well-tuned grounding model can outperform the best end-to-end neural models for grounding on the web.
The entire one-time process for training RUSS takes approximately 7 hours on an NVIDIA Tesla V100. RUSS can perform a new task on-the-job by running the instructions through the semantic parser in less than 1 minute.
We analyze how well RUSS and PhraseNode perform for sentences in the Evaluation Set requiring different types of reasoning (Table 5). Russ outperforms the state-of-the-art PhraseNode (Pasupat et al., 2018) for all the reasoning types. It performs well on grounding tasks that involve type, input, and relational reasoning. Evaluation of the spatial reasoning instructions revealed that many referenced image features (e.g. "click the hamburger menu icon"), which is not supported by RUSS. The results show that ThingTalk is simple enough to be generated by a neural language model, while comprehensive enough to express the wide range of open-domain natural language instructions for web tasks.
Unlike end-to-end models that struggle with long, complex instructions, we find that RUSS ben
1 Redeem Amazon Gift Card
Get Pinterest Ad Account Number
3 Log out of all Spotify accounts
4 Create new Walmart account
5 Send Google feedback
Table 6: Tasks in RUSS User Study
efits from added reasoning in instructions that constrains the potential set of element candidates (e.g. "the element must be an input"). Webpages commonly have thousands of elements and the probability of matching the right element increases with constraints.
Of the 741 instructions in the RUSS dataset, 6 contain attributes that are not well expressed in ThingTalk. For example, "select the user's birth month in the month drop down" is not parsed correctly because ThingTalk does not have a notion of selecting an element in a menu. This feature will be added in the future.
Another source of errors lies in how webpages are constructed. Important attributes needed for grounding can be hidden behind classes. For example, an element may be labeled as "Click here", but the text is not present in the DOM text attribute and instead obscured behind a site-specific class name such as "next-page-button". Grounding techniques on visual data can be helpful in resolving this class of problems.
The goal of our user study is to evaluate the end-to-end feasibility of RUSS on open-domain instructions from real customer service websites, and evaluate how users respond to RUSS. This is a small-scale study with promising early results, but can benefit from further user studies on larger populations.
We recruited 12 participants who were asked to complete 5 customer-support tasks (Table 6), chosen from popular websites: Amazon, Spotify, pinterest, Google, and Walmart, with both RUSS and the browser. For all tasks, users were given a fake persona (a set of credentials such as email, password, gift card code, etc) to use when interacting with the agent. The study was approved by our IRB and participants were compensated.
The participants in our study ranged from ages 21 to 68 years old, with an average age of 36 years old, a 50/50 male/female ratio, and varied technical sophistication. To reduce learning effects,
Figure 7: Average number of user interactions via utterance or click (left); average time taken to complete tasks in seconds (left)
we used Latin Square Balancing (Bradley, 1958) to ensure that both the web and RUSS trials of each site were performed first half the time. We record users' time to perform each task, number of turns (in RUSS) or clicks (on the web) required to achieve each task, and gave each participant an exit survey containing qualitative assessments.
Participants were able to complete  $85\%$  of the tasks on their own on the web and  $98\%$  of tasks with the help of RUSS. Those who did not finish their task either gave up or failed to complete the task within 5 minutes. The time it took users to accomplish each task was similar for the Web and RUSS (Fig. 7), though RUSS was significantly faster for Task 2, a more complex task users said they were unfamiliar with. This seems to indicate that RUSS is more favorable for unfamiliar, complex tasks.
After trying the 5 tasks,  $69\%$  of users reported they prefer RUSS over navigating online help pages. Reasons cited include ease of use, efficiency, and speed, even though the times of completion were similar. Participants were generally pleased with their RUSS experience, and only one person said that they were unlikely to use RUSS again (Fig. 8). However, many users did report that they wished RUSS was as visually stimulating as the browser. Other users noted that they felt more familiar and comfortable with the browser.
As a final discussion, it is worth noting that while the user study results are extremely promising, this is a small scale study. RUSS's runtime needs stronger error handling for out-of-context conversation. Currently, RUSS gives the user 3 tries to return an expected response before terminating. RUSS also times out if a webpage takes more than  $>60$  seconds to load in Puppeteer. We saw instances of both of these situations in the RUSS user study in the few cases the user failed to complete a task.
Figure 8: Qualitative results from user studies. On a scale 1-5 for satisfaction,  $1 =$  not satisfied at all and 5 = exceeded expectations. For likelihood,  $1 =$  will never use again and  $5 =$  will definitely use again.
RUSS demonstrates how a semantic parser and grounding model can be used to perform unseen web tasks from natural language instructions. By achieving  $76.7\%$  accuracy on the RUSS Evaluation Dataset, we show how a modular semantic parsing approach can outperform end-to-end neural models on this task, and demonstrate how humans interact with RUSS-like systems in the user study. Like many datasets in NLP, we believe extensive research is still required to go from RUSS's  $76.6\%$  overall accuracy on the Evaluation Dataset to  $100\%$ . As seen in Table 4, prior models like PhraseNode achieve only  $46.5\%$  grounding accuracy, which points to additional work necessary in grounding natural language on the web.
The RUSS Evaluation dataset introduces a set of real instructions for grounding language to executable actions on the web to evaluate future research in this direction, including training semantic parsers to new targets using real-world instructions and neural models for grounding formal language representations on the web. Our work provides the task, technical foundation, and user research for developing open-domain web agents like RUSS.
The user study conducted in this paper was submitted to the Institutional Review Board and re
ceived IRB Exempt status. All participants were read an IRB consent form prior to the user study, which detailed the study details. No deception was involved in the study: all participants knew they were evaluating an AI agent in the conversation portion of the user study and were not led to believe otherwise. They study took about 20 minutes. All participants were compensated with $10.
The webpages scraped for the RUSS dataset are all public domain webpages. No individual personal identifying information was used to obtain the webpages. On websites that required accounts to access pages, we created fake user accounts with non-identifying usernames / passwords / emails to navigate the websites in order to limit any privacy risks that may be involved.
In the future, we see web agents like RUSS helping improve accessibility by helping individuals who are visually impaired, less technologically advance, or otherwise preoccupied receive equitable access to information. Before systems like RUSS are put to practice at scale, the authors believe more research must be done in understanding user behavior with web agents to safeguard against downstream consequences of system errors and to better understand how information can be effectively delivered by AI agents that operate in potentially high-stakes transactions such as health or finance. Our user study is the first step in this direction.
We thank Silei Xu for helpful discussions on constructing the Synthetic dataset, and Richard Socher for feedback and review of the final publication.
This work is supported in part by the National Science Foundation under Grant No. 1900638 and the Alfred P. Sloan Foundation under Grant No. G-2020-13938.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements of outside organizations.
James Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary Swift, and William Taysom. 2007. Plow: A collaborative task
learning agent. In 2007 AAAI Conference on Artificial Intelligence (AAAI), pages 1514-1519. AAAI.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
James V. Bradley. 1958. Complete counterbalancing of immediate sequential effects in a latin square design. pages 525-528.
Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S. Lam. 2019. Genie: A generator of natural language semantic parsers for virtual assistant commands. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019, pages 394-410, New York, NY, USA. ACM.
David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI-2011), pages 859-865.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Michael H Fischer, Giovanni Campagna, Euirim Choi, and Monica S Lam. 2020. Multi-modal end-user programming of web-based virtual assistant skills. arXiv preprint arXiv:2008.13510.
Larry Heck, Dilek Hakkani-Tur, Madhu Chinthakunta, Gokhan Tur, Rukmini Iyer, Partha Parthasarathy, Lisa Stifelman, Elizabeth Shriberg, and Ashley Fidler. 2013. Multi-modal conversational search and browse. In First Workshop on Speech, Language and Audio in Multimedia.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. 2018. Grounding visual explanations. In European Conference on Computer Vision, pages 269-286. Springer.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Nobuhiro Ito, Yuya Suzuki, and Akiko Aizawa. 2020. From natural language instructions to complex processes: Issues in chaining trigger action rules. arXiv preprint arXiv:2001.02462.
Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang. 2020. Learning adaptive language interfaces through decomposition. arXiv preprint arXiv:2010.05190.
Hyounghun Kim, Abhay Zala, Graham Burri, Hao Tan, and Mohit Bansal. 2020. Arramon: A joint navigation-assembly instruction interpretation task in dynamic environments. arXiv preprint arXiv:2011.07660.
Gilly Leshed, Eben M. Haber, Tara Matthews, and Tessa Lau. 2008. Coscripter: Automating & sharing how-to knowledge in the enterprise. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, page 1719-1728, New York, NY, USA. Association for Computing Machinery.
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. Sugilite: Creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, page 6038-6049, New York, NY, USA. Association for Computing Machinery.
Toby Jia-Jun Li and Oriana Riva. 2018. Kite: Building conversational bots from mobile apps. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '18, page 96-109, New York, NY, USA. Association for Computing Machinery.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations.
Ramesh Manuvinakurike, Jacqueline Brixey, Trung Bui, Walter Chang, Doo Soon Kim, Ron Artstein, and Kallirroi Georgila. 2018. Edit me: A corpus and a framework for understanding natural language image editing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. 2018. Learning to navigate in cities without a map. In Advances in Neural Information Processing Systems, pages 2419-2430.
Dipendra Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.08795.
Panupong Pasupat, Tian-Shun Jiang, Evan Liu, Kelvin Guu, and Percy Liang. 2018. Mapping natural language commands to web elements. In Proceedings of the 2018 Conference on Empirical Methods
in Natural Language Processing, pages 4970-4976, Brussels, Belgium. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In ICLR.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3973-3983.
Ritam Jyoti Sarmah, Yunpeng Ding, Di Wang, Cheuk Yin Phipson Lee, Toby Jia-Jun Li, and Xiang 'Anthony' Chen. 2020. Geno: A developer tool for authoring multimodal interaction on existing web applications. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, UIST '20, page 1169-1181, New York, NY, USA. Association for Computing Machinery.
Zhanna Sarsenbayeva. 2018. Situational impairments during mobile interaction. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, pages 498-503.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL.
Statista Research Department. 2019. Share of customers in the united states who have contacted customer service for any reason in the past month from 2015 to 2018.
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidzion, Justin Hart, Peter Stone, and Raymond J Mooney. 2019. Improving grounded natural language understanding through human-robot dialog. In 2019 International Conference on Robotics and Automation (ICRA), pages 6934-6941. IEEE.
Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, and Raymond J Mooney. 2016. Learning multi-modal grounded linguistic semantics by playing "i spy". In IJCAI, pages 3477-3483.
Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. 2015. Sequence to sequence - video to text. In Proceedings of the 2015 International Conference on Computer Vision (ICCV-15), Santiago, Chile.
Silei Xu, Giovanni Campagna, Jian Li, and Monica S Lam. 2020. Schema2QA: High-quality and low-cost Q&A agents for the structured web. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1685-1694.