SlowGuess/NAACL_2022_OCR huggingface dataset:
Daniel Jurafsky and James H. Martin. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 1st edition. Prentice Hall PTR, USA.
Yiping Kang, Yunqi Zhang, Jonathan K. Kummerfeld, Lingjia Tang, and Jason Mars. 2018. Data collection for dialogue system: A startup perspective. In Proceedings of NAACL-HLT 2018: Industry Papers, pages 33-40.
Seokhwan Kim, Luis Fernando D'Haro, Rafael E. Banchs, Jason Williams, and Matthew Henderson. 2016. The fourth dialog state tracking challenge. In Proceedings of IWSDS 2016.
Jason Krone, Yi Zhang, and Mona Diab. 2020. Learning to classify intents and slot labels given a handful of examples. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 96-108.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for self-supervised learning of language representations. In Proceedings of ICLR 2020.
Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019a. Outlier detection for improved data quality and diversity in dialog systems. In Proceedings of NAACL-HLT 2019, pages 517-527.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019b. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of EMNLP-IJCNLP 2019, pages 1311-1316.
Esther Levin and Roberto Pieraccini. 1995. Chronus, the next generation. In Proceedings of the ARPA Workshop on Spoken Language Technology.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kuttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them. Transactions of the Association for Computational Linguistics, 9:1098-1115.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pretrain, prompt, and predict: A systematic survey of prompting methods in Natural Language Processing. CoRR, abs/2107.13586.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. In Proceedings of ACL 2019, pages 4487-4496.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019b. Benchmarking natural language understanding services for building conversational agents. In Proceedings of IWSDS 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019c. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In Proceedings of ICLR 2018.
Nurul Lubis, Christian Geishauser, Michael Heck, Hsien-chin Lin, Marco Moresi, Carel van Niekerk, and Milica Gasic. 2020. LAVA: Latent action spaces via variational auto-encoding for dialogue policy optimization. In Proceedings of COLING 2020, pages 465-479.
Shikib Mehri and Mihail Eric. 2021. Example-driven intent prediction with observers. In Proceedings of NAACL-HLT 2021, pages 2979-2992.
Mahdi Namazifar, Alexandros Papangelis, Gokhan Tur, and Dilek Hakkani-Tur. 2021. Language model is all you need: Natural language understanding as question answering. In Proceedings of ICASSP 2021, pages 7803-7807.
Denis Peskov, Nancy Clarke, Jason Krone, Brigi Fodor, Yi Zhang, Adel Youssef, and Mona Diab. 2019. Multi-domain goal-oriented dialogues (multidogo): Strategies toward curating and annotating large scale dialogue data. In Proceedings of EMNLP-IJCNLP 2019, pages 4526-4536.
Libo Qin, Xiao Xu, Wanxiang Che, and Ting Liu. 2020. AGIF: An adaptive graph-interactive framework for joint multiple intent detection and slot filling. In *Findings of EMNLP* 2020, pages 1807–1816.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of ACL 2018, pages 784-789.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for Machine Comprehension of Text. In Proceedings of EMNLP 2016, page 2383-2392.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. CoRR, abs/1909.05855.
Antoine Raux, Brian Langner, Alan W. Black, and Maxine Eskenazi. 2003. LET's GO: Improving spoken dialog systems for the elderly and non-natives. In Proceedings of EUROSPEECH 2003.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of EMNLP 2019, pages 3982-3992.
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In In Proceedings of NAACL-HLT 2021, pages 296-310.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2Tensor for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas, pages 193-199.
Kai Wang, Junfeng Tian, Rui Wang, Xiaojun Quan, and Jianxing Yu. 2020a. Multi-domain dialogue acts and response co-generation. In Proceedings of ACL 2020, pages 7125-7134.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020b. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained Transformers. In Proceedings of NeurIPS 2020.
Zhuoran Wang and Oliver Lemon. 2013. A simple and generic belief tracking mechanism for the Dialog State Tracking Challenge: On the believability of observed information. In Proceedings of SIGDIAL 2013, pages 423-432.
Wei Wei, Quoc Le, Andrew Dai, and Jia Li. 2018. Airdialogue: An environment for goal-oriented dialogue research. In Proceedings of EMNLP 2018, pages 3844-3854.
Jason D. Williams, Antoine Raux, Deepak Ramachandran, and Alan W. Black. 2013. The Dialogue State Tracking Challenge. In Proceedings of SIGDIAL 2013, pages 404-413.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers:
State-of-the-art natural language processing. In Proceedings of EMNLP 2020: System Demonstrations, pages 38-45.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020a. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In Proceedings of EMNLP 2020, pages 917-929.
Chien-Sheng Wu and Caiming Xiong. 2020. Probing task-oriented dialogue representation from language models. In Proceedings of EMNLP 2020, pages 5036-5051.
Di Wu, Liang Ding, Fan Lu, and Jian Xie. 2020b. SlotRefine: A fast non-autoregressive model for joint intent detection and slot filling. In Proceedings of EMNLP 2020, pages 1932-1937.
Weijia Xu, Batool Haider, and Saab Mansour. 2020. End-to-end slot alignment and recognition for crosslingual NLU. In Proceedings of EMNLP 2020, pages 5052-5063.
Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. In Proceedings of IJCAI 2019, pages 5370-5378.
Steve Young. 2010. Cognitive user interfaces. IEEE Signal Processing Magazine.
Steve Young, Gunnar Evermann, Mark Gales, Thomas Hain, Dan Kershaw, Xunying Liu, Gareth Moore, Julian Odell, Dave Ollason, Dan Povey, Valtcho Valtchev, and Phil Woodland. 2002. The HTK book. Cambridge University Engineering Department.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117.
Haode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen, Guangyuan Shi, Xiao-Ming Wu, and Albert Y.S. Lam. 2021a. Effectiveness of pre-training for few-shot intent classification. In Findings of EMNLP 2021, pages 1114-1120.
Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021b. Few-shot intent detection via contrastive pre-training and fine-tuning. In Proceedings of EMNLP 2021, pages 1906-1912.
Jinghan Zhang, Yuxiao Ye, Yue Zhang, Likun Qiu, Bin Fu, Yang Li, Zhenglu Yang, and Jian Sun. 2020. Multi-point semantic representation for intent classification. In Proceedings of AAAI 2020, pages 9531-9538.
A Appendix: Ontology
The complete ontology of NLU++ is provided in Table 9 and Table 10.
B Appendix: Sentence Encoders in Intent Detection Experiments
CONVERT (Henderson et al., 2020) is trained with the conversational response selection objective (Henderson et al., 2019b) on large Reddit data (Al-Rfou et al., 2016; Henderson et al., 2019a), spanning more than 700M (context, response) sentence pairs. Thanks to its naturally conversational pretraining objective, it has been shown to be especially well-suited for conversational tasks such as intent detection (Casanueva et al., 2020) and slot labelling (Coope et al., 2020). It outputs 1,024-dim sentence encodings.
- github.com/davidalami/ConveRT
LABSE. Language-agnostic BERT Sentence Embedding (LaBSE) (Feng et al., 2020) adapts pretrained multilingual BERT (mBERT) (Devlin et al., 2019) using a dual-encoder framework (Yang et al., 2019) with larger embedding capacity (i.e., a shared multilingual vocabulary of 500k subwords). While LaBSE is the current state-of-the-art multilingual encoder, it also displays very strong monolingual English performance (Feng et al., 2020). It produces 768-dim sentence encodings.
huggingface.co/sentence-transformers/LaBSE
ROBL-1B and LM12-1B (Reimers and Gurevych, 2019; Thakur et al., 2021) are sentence encoders which fine-tune the pretrained RobertaLarge (ROBL) language model (Liu et al., 2019a) and the 12-layer MiniLM (Wang et al., 2020b), respectively, again using a contrastive dual-encoder framework (Reimers and Gurevych, 2019). The models are fine-tuned on a set of more than 1B sentence pairs: this set comprises various data such as Reddit 2015-2018 comments (Henderson et al., 2019a), Natural Questions (Kwiatkowski et al., 2019), PAQ (question, answer) pairs (Lewis et al., 2021), to name only a few.19 ROBL-1B outputs
1,024-dim encodings, while LM12-1B produces 384-dim encodings.
We opted for those two models in particular as one represents a class of large sentence encoders (ROBL-1B), and the other is lightweight (LM12-1B), while both display very strong performance in a myriad of sentence similarity and semantic search tasks, see www.sbert.net/docs/pretrained_models.html.
huggingface.co/sentence-transformers/all-roberta-large-v1
huggingface.co/sentence-transformers/all-MiniLM-L12-v1
C Appendix: QA-Pretrained Models
We rely on the same SQuAD-tuned language models as Namazifar et al. (2021). ROBB-QA can be found online at: https://huggingface.co/deepset/roberta-base-squad2; ALB-QA is available at: https://huggingface.co/twmkn9/albert-base-v2-squad2
D Appendix: Slot Labeling Baselines
CONVEX (Henderson and Vulić, 2021) demonstrates strong SL performance, especially in few-shot settings. It is pretrained on a pairwise cloze task extracted from the Reddit examples (Henderson et al., 2019a), and the majority of the pretrained model's parameters in CONVEX are kept frozen during fine-tuning, making it an extremely efficient model. We adopt the suggested hyper-parameters from Henderson and Vulić (2021).
QA-Based: Namazifar et al. (2021) train an extractive QA-based model to extract the spans of the slots from the input user utterance as answers to manually defined natural language questions (one per slot). It follows the same idea as QA-based ID models. We also provide such questions for each slot along with NLU++ for model training and inference: see the questions in Table 10.
In a nutshell, the contrastive fine-tuning task which combines all the heterogeneous datasets is as follows: given a 'query' sentence from each sentence pair, and a set of  $R$  randomly sampled negatives plus 1 true positive (the sentence from the same pair), the model should predict which sentence from the set of  $R + 1$  sentences is actually paired with the query sentence in the dataset. The full list of all datasets along with the exact model specifications is at:
huggingface.co/sentence-transformers/all-roberta-large-v1.
Table 9: Intents ontology
SLOT	DESCRIPTION-QUESTION	DOMAIN
date	What is the specific date mentioned in this sentence?	general
date_period	What is the time period in days, months or years mentioned in this sentence?	general
date_from	What is the start date of some period mentioned in this sentence?	general
date_to	What is the end date of some period mentioned in this sentence?	general
time	What is the specific time in the day mentioned in this sentence?	general
time_from	What is the start time of some time period mentioned in this sentence?	general
time_to	What is the end time of some time period mentioned in this sentence?	general
time_period	What is the time period in hours or minutes mentioned in this sentence?	general
person_name	What is the name of a person mentioned in this sentence?	general
number	What is the number without context mentioned in this sentence?	general
amount_of-money	What is the specific amount of money mentioned in this sentence?	banking
company_name	What is the name of some sort of company mentioned in this sentence?	banking
shopping_category	What is the category of some expense mentioned in this sentence?	banking
kids	what is the number of kids mentioned in this sentence?	hotels
adults	what is the number of adults mentioned in this sentence?	hotels
people	What is the number of people mentioned in this sentence?	hotels
rooms	What is the number of rooms mentioned in this sentence?	hotels
Table 10: Slots ontology