SlowGuess/NAACL_2022_OCR huggingface dataset:
Our ultimate goal is to expand end-to-end task oriented dialog systems with the disambiguation skill. Therefore, it is required not only to enable the dialog model to resolve DSR-ambiguity, but also to maintain the model's original ability for generating responses or dialog state tracking. To verify that, we first analyze the performance on the original test set ("Origin" columns in Table 2). The models trained with original data (0.676 on MultiWOZ) or the original one mixed with  $5\%$  synthesized data (0.575 on SGD) commonly achieves the best performance, which is reasonable since training data and test data share almost the same distribution. On the other hand, the performance on the original test set of the models trained with the augmented data is comparable with the original training data, which means these models maintain the ability to predict entity name. As for the results over the whole test set in Table 7, the augmented model even achieves better accuracy (0.877) than the original one (0.871) on the SGD test set. Therefore, the augmentation does not hurt the model's ability to predict named entities without disambiguation cases.
Beyond named entities, the augmentation hardly affects the model's ability to predict other dialog slots for the non-disambiguation cases. The results are listed in the "Origin" columns in the Table 8 and Table 3 correspondingly. For both test sets, the models trained with augmented data achieve comparable results with the models trained with original data, which means our augmentation also maintains the distribution of other slot types in the
Test Data Train Data	SGD	MultiWOZ
Origin	AutoAug	HumanAug	Origin	AutoAug	HumanAug
Origin	48.9±0.7	47.7±0.7	47.7±0.7	53.5±0.1	52.2±0.5	52.3±0.4
Origin+Syn2%	50.0±0.3	48.9±0.4	49.0±0.4	53.0±0.1	50.0±0.6	50.1±0.6
Origin+Syn100%	49.5±0.6	48.7±0.5	48.7±0.5	52.8±0.3	50.4±0.5	50.4±0.4
AutoAug	50.2±1.0	49.9±1.0	49.7±1.0	52.4±0.4	53.5±0.3	53.5±0.3
AutoAug+Syn2%	49.8±0.4	49.6±0.4	49.4±0.4	52.5±0.2	54.5±0.1	54.5±0.1
AutoAug+Syn100%	51.0±0.4	50.9±0.4	50.6±0.4	53.2±0.2	55.2±0.4	55.2±0.4
original data. In conclusion, our augmentation does not impede the model from learning the original data distribution. And the model trained with the augmented data perform well no matter whether the disambiguation case exists.
5.3 Leveraging Augmented Turns
To find the most efficient method to leverage our dataset, we explore the following experiment settings. Since SGD and MultiWOZ are both task-oriented dialog datasets and share some common domains, pre-training on one dataset might help learn the other one. Therefore, for MultiWOZ model, we first pre-finetune the model with the original SGD and then fine-tune it on the origin MultiWOZ. We also conduct the experiment that uses the augmented SGD training data for the first step of fine-tuning, with or without mixing synthesized single-turn dialogs. All these three experiment settings do not involve augmentation on the MultiWOZ dataset. In addition, Since the augmented turns only take up  $2\%$  of the whole training data, the model rarely sees the disambiguation cases in each epoch. To emphasize those turns, we up-sample those disambiguation turns to the same amount as the original training data.
Table 4 show results for these settings on Multi-WOZ dataset (The joint goal accuracy results can be found in Table 6). For the named entity accuracy, the setting "Upsample+Syn" achieves the best result, because the more disambiguation turns the models see, the better the model learns the skill to solve the ambiguity. As for the joint goal accuracy, setting "Aug+Syn" performs better than "Upsample+Syn" because too much disambiguation turns inevitably introduce bias and affect learning the original task. Therefore, if we need to solve DSR-ambiguity in a new dataset, the best option is to conduct augmentation with our framework and train models together with synthesized single-turn data. Although not as good as setting "Aug+Syn", the setting "PreFineTuneAug+Syn" performs better than
Table 3: Joint goal accuracy evaluated on the whole test set.
Name Entity Accuracy
Origin	HumanAug
Origin	67.6±0.7	48.8±0.1
Origin+Syn	67.0±0.7	55.6±2.9
Aug	63.3±1.6	73.9±2.9
Aug+Syn	63.3±1.3	87.4±0.4
PreFineTuneOrigin	67.8±0.4	44.1±1.3
PreFineTuneAug	68.4±0.3	49.5±1.1
PreFineTuneAug+Syn	68.5±0.9	65.8±0.6
Upsample	63.5±1.0	83.7±3.2
Upsample+Syn	63.3±0.5	88.3±0.8
Table 4: Results for more training setting based on the MultiWOZ dataset, in terms of the name entity accuracy over only augmented turns. The amount of synthesized data “+Syn” is the same as the amount of original test set in this table. “PreFineTuneOrigin” means first prefinetuning model with original SGD training data and then fine-tuning on MultiWOZ training data, while “PreFineTuneAug” means first pre-finetuning model with augmented SGD training data. The setting “Upsample” means up-sampling augmented turns to the same amount of training data.
the model trained on original data in terms of both JGA and named entity accuracy. Please note that this setting does not require any augmentation on MultiWOZ. Hence, to solve disambiguation cases in a new dataset, the cheapest choice is to fine-tune a model on our augmented dataset (MultiWOZ and SGD) first, and then fine-tune it on the original data, mixed with the synthesized single-turn dataset. The above experiments are conducted and evaluated on the MultiWOZ dataset. We also apply the same settings on the SGD dataset and the results can be found in the Table 5 and Table 6.
5.4 Impact of Entity Addressing Methods
To explore the impact of different addressing methods, we conduct the ablation study by finetuning GPT2 with the synthesized single-turn dialog datasets of each individual addressing method (results shown in Table 9). For each addressing method, we generate  $100K / 10K / 10K$  single-turn dialogs as the train/dev/test set, which is comparable to the MultiWOZ or the SGD datasets. We
find that when focusing only on the disambiguation task with a simple context structure like single-turn dialog, the model can easily learn all kinds of addressing methods, except for "Multiple Addressing". The model accuracy drops by  $\approx 33\%$  in that case. Even if we combine multiple addressing methods together except "Multiple Addressing", the model can still understand the addressing target. However, when the user chose multiple entities, it is hard for models to accurately predict how many entities the user selected.
6.1 Task-Oriented Dialog Datasets
MultiWOZ (Budzianowski et al., 2018) is one of the most popular task-oriented dialog dataset. It covers multiple domains, consists of a large amount of dialogs, and has been chosen as benchmark for many dialog tasks, e.g. dialog state tracking (Zhang et al., 2019, 2020a; Heck et al., 2020), dialog policy optimization (yang Wu et al., 2019; Wang et al., 2020a,b) and end-to-end dialog modeling (Zhang et al., 2020b; Hosseini-Asl et al., 2020; Peng et al., 2020; Huang et al., 2021). And to polish it up to be a better benchmark, many works pay effort to improve and correct dataset (Eric et al., 2020; Zang et al., 2020; Qian et al., 2021; Han et al., 2021; Ye et al., 2021). In this paper, we choose MultiWOZ 2.2 version to conduct augmentation. Schema-Guided Dataset (SGD) (Rastogi et al., 2019) is the largest public task-oriented dialog dataset, containing  $18\mathrm{K}+$  dialogs. It covers in total 20 domains and 45 services. The dataset is constructed by generating dialog outlines from interactions between two dialog simulators, and then being paraphrased by crowd-workers. SIMMC 2.0 (Kottur et al., 2021) is a newly-released multi-modal task-oriented dialog dataset around situated interactive multi-modal conversations (Moon et al., 2020). It focuses on dialogs with multi-modal context, which can be in the form of either co-observed image or virtual reality environment. The dataset contains  $11\mathrm{K}+$  dialogs and covers two shopping domains.
As for the disambiguation problem, neither MultiWOZ nor SGD has related cases or annotations. SIMMC 2.0 is well-annotated for disambiguation, but it only covers two domains, and addresses entity mostly with multi-modal knowledge. Therefore, we augment MultiWOZ and SGD with the disambiguation templates from the SIMMC 2.0.
6.2 Ambiguity & Clarification Questions
Ambiguity is a common phenomenon across many conversation-involved NLP tasks, e.g. conversational search (Rosset et al., 2020), Question-Answering (White et al., 2021), open-domain dialog (Aliannejadi et al., 2021) and intent classification (Bihani and Rayz, 2021; Dhole, 2020). The problem mainly results from two aspects: 1. user's ambiguous keyword (e.g. "orange" can be either color or fruit (Coden et al., 2015)) and 2. lacking of enough constraints for accurate searching, leading to multiple results (e.g. "I want to book a cheap hotel" where there might be multiple "cheap" hotels). Previous work proposes to incorporate clarification questions to solve the ambiguity problem (Purver et al., 2001; Schlangen, 2004; Radlinski and Craswell, 2017), including both model-wise (Li et al., 2017; Rao and Daumé III, 2019; Yu et al., 2020) and dataset-wise (Aliannejadi et al., 2019; Xu et al., 2019; Min et al., 2020; Zamani et al., 2020b). Our work it the first to point out the ambiguity within the database-searching of task-oriented dialog systems and introduce clarification questions to help solve this problem.
In addition, most of the work focus on when and how to generate clarification questions (Kumar and Black, 2020). Typical clarification question generation is based on the context with a Seq2Seq model (Zamani et al., 2020a). Rao and Daumé III (2019) propose to utilize the generative adversarial network to learn generating relevant clarification question based on corresponding answers. Sekulic et al. (2021) takes user engagement into consideration to generate high-quality clarification questions. In this work, instead of focusing on question generation, we put our attention on understanding the user's answer to clarification questions.
In this paper, we proposed a new task, dataset result disambiguation, which is ignored in most popular public task-oriented dialog datasets such as MultiWOZ and SGD. We showed that models trained on these two datasets can not deal with entity ambiguities. We proposed to address this issue by augmenting existing datasets with relevant disambiguation turns. We extract templates of the disambiguation turns from the SIMMC2.0 dataset and jointly generate new turns with the databases from MultiWOZ and SGD for augmentation. To ensure the quality and correctness of the augmentation, we
recruit crowd-workers to paraphrase the generated sentences. We benchmark our augmented dataset with the GPT2 model. We observe that the augmentations empower dialog models with a new skill to solve disambiguation tasks without performance drop on the original task. In the future, we plan to incorporate state-of-the-art and realistic entity referencing techniques cases to improve the datasets, which further enhances the dialog system. We hope that our work stimulates further research in identifying and incorporating such universal dialog skills in dialog systems avoiding exploding data-costs.
To ensure that the dataset does not have any sensitive topics, we ask crowd-workers to make comments if the dialog content involves any of following: 1. offensive, racist, biased and non-tolerant behavior; 2. violence and self-harm; 3. sexual or flirtatious behavior; 4. controversial and polarizing topics. Since the database of both MultiWOZ and SGD are sampled from real world, annotators also comment if there are real names included in the slot values, which can be personally identifiable information (PII). Considering both of these two datasets are public dataset, we do not replace those named entities with placeholders. The detailed description of sensitive topics is included in the Fig. 7 in the appendix.
Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue corpora with clarifying questions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4473-4484, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mohammad Aliannejadi, Hamed Zamani, Fabio A. Crestani, and William Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Geetanjali Bihani and Julia Taylor Rayz. 2021. Fuzzy classification of multi-intent utterances. In NAFIPS.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the
2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Anni Coden, Daniel F. Gruhl, Neal Lewis, and Pablo N. Mendes. 2015. Did you mean a or b? supporting clarification dialog for entity disambiguation. In SumPre-HSWI@ESWC.
Kaustubh D. Dhole. 2020. Resolving intent ambiguities by retrieving discriminative clarifying questions. ArXiv, abs/2008.07559.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 422-428, Marseille, France. European Language Resources Association.
Ting Han, Ximing Liu, Ryuichi Takanobu, Yixin Lian, Chongxuan Huang, Dazhen Wan, Wei Peng, and Minlie Huang. 2021. Multiwoz 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation. In NLPCC.
M. Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, M. Moresi, and Milica Gavsi'c. 2020. Trippy: A triple copy strategy for value independent neural dialog state tracking. In SIGdial.
Ehsan Hosseini-Asl, B. McCann, Chien-Sheng Wu, Semih Yavuz, and R. Socher. 2020. A simple language model for task-oriented dialogue. *ArXiv*, abs/2005.00796.
Tianjian Huang, Shaunak Halbe, Chinnadhurai Sankar, Pooyan Amini, Satwik Kottur, Alborz Geramifard, Meisam Razaviyayn, and Ahmad Beirami. 2021. Dair: Data augmented invariant regularization. arXiv preprint arXiv:2110.11205.
Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. Simmc 2.0: A task-oriented dialog dataset for immersive multimodal conversations. ArXiv, abs/2104.08667.
Vaibhav Kumar and Alan W Black. 2020. ClarQ: A large-scale and diverse dataset for clarification question generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7296-7301, Online. Association for Computational Linguistics.
Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2017. Learning through dialogue interactions by asking questions. In ICLR.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of
the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783-5797, Online. Association for Computational Linguistics.
Seungwhan Moon, Satwik Kottur, Paul A Crook, Ankita De, Shivani Poddar, Theodore Levin, David Whitney, Daniel Difranco, Ahmad Beirami, Eunjoon Cho, et al. 2020. Situated and interactive multimodal conversations. arXiv preprint arXiv:2006.01460.
Baolin Peng, C. Li, Jin chao Li, Shahin Shayandeh, L. Liden, and Jianfeng Gao. 2020. Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model. ArXiv, abs/2005.05298.
Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2001. On the means for clarification in dialogue. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue.
Kun Qian, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. 2021. Annotation inconsistency and entity bias in MultiWOZ. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 326-337, Singapore and Online. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval.
Sudha Rao and Hal Daumé. 2019. Answer-based adversarial training for generating clarification questions. ArXiv, abs/1904.02281.
Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2737-2746, Melbourne, Australia. Association for Computational Linguistics.
Sudha Rao and Hal Daumé III. 2019. Answer-based Adversarial Training for Generating Clarification Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 143-155, Minneapolis, Minnesota. Association for Computational Linguistics.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. arXiv preprint arXiv:1909.05855.
Corby Rosset, Chenyan Xiong, Xia Song, Daniel Fernando Campos, Nick Craswell, Saurabh Tiwary, and Paul N. Bennett. 2020. Leading conversational search by suggesting useful questions. Proceedings of The Web Conference 2020.
David Schlangen. 2004. Causes and strategies for requesting clarification in dialogue. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 136-143, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Ivan Sekulic, Mohammad Aliannejadi, and Fabio A. Crestani. 2021. User engagement prediction for clarification in search. In ECIR.
Jianhong Wang, Yeliang Zhang, Tae-Kyun Kim, and Yunjie Gu. 2020a. Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system. ArXiv, abs/2006.06814.
Kai Wang, Jun-Feng Tian, Rui Wang, Xiaojun Quan, and J. Yu. 2020b. Multi-domain dialogue acts and response co-generation. ACL 2020.
Julia White, Gabriel Poesia, Robert Hawkins, Dorsa Sadigh, and Noah Goodman. 2021. Open-domain clarification question generation without question examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 563-570, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jingjing Xu, Yuechen Wang, Duyu Tang, Nan Duan, Pengcheng Yang, Qi Zeng, Ming Zhou, and Xu Sun. 2019. Asking clarification questions in knowledge-based question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1618-1629, Hong Kong, China. Association for Computational Linguistics.
Qing yang Wu, Yichi Zhang, Yu Li, and Z. Yu. 2019. Alternating recurrent dialog model with large-scale pretrained language models. ArXiv, abs/1910.03756.
Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. 2021. Multiwoz 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. ArXiv, abs/2104.00773.
Lili Yu, Howard Chen, Sida I. Wang, Tao Lei, and Yoav Artzi. 2020. Interactive classification by asking informative questions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2664-2680, Online. Association for Computational Linguistics.
Hamed Zamani, Susan T. Dumais, Nick Craswell, Paul N. Bennett, and Gord Lueck. 2020a. Generating clarifying questions for information retrieval. Proceedings of The Web Conference 2020.
Hamed Zamani, Gord Lueck, Everest Chen, Rodolfo Quispe, Flint Luu, and Nick Craswell. 2020b. Mimics: A large-scale data collection for search clarification. Proceedings of the 29th ACM International Conference on Information & Knowledge Management.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online. Association for Computational Linguistics.
Jian'guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wan, Philip S. Yu, R. Socher, and Caiming Xiong. 2019. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. ArXiv, abs/1910.03544.
Yichi Zhang, Zhijian Ou, Huixin Wang, and Junlan Feng. 2020a. A probabilistic end-to-end task-oriented dialog model with latent belief states towards semi-supervised learning. *ArXiv*, abs/2009.08115.
Yichi Zhang, Zhijian Ou, and Z. Yu. 2020b. Task-oriented dialog systems that consider multiple appropriate responses under the same context. ArXiv, abs/1911.10484.
A Supplementary Details for Augmentation
A.1 Involving Domains
- MultiWOZ: "restaurant", "hotel", and "attraction"
- Google SGD: "events_3", "homes_2", "hotels_4", "media_3", "messaging_1", "movies_1", "movies_3", "music_3", "restaurants_2", "services_1", "services_4", "travel_1", "events_1", "homes_1", "hotels_1", "media_2", "movies_2", "music_1", "hotels_3", "media_1", "music_2", "restaurants_1", "services_2", "services_3",
A.2 Context-Free Grammars
Here we list some examples of the context-free grammars that we use for augmentation:
- SIMPLE -> which OBJECT ((do | did) you VERB-2 | would you VERB-2-WOULD | (would you be | (are | were) you) (VERB-2-ING | ADJ))
- VERB-2 -> want [to (know | learn) [about]] | wish to (know | learn) [more] about | have in mind | mean [by that| exactly | precisely] | need [information for |that info for] | refer to
- VERB-2-WOULD -> want [to (know | learn) [about]] | wish to (know | learn) about | care about | like
- VERB-2-WOULD-LIKE
- VERB-2-WOULD-LIKE -> (further | more) information about | me to check | to (hear about | know [more] about)
- VERB-2-ING -> asking [about | for] | inquiring about | looking at | referring to [exactly] | talking about | thinking [about | of] | (requesting | seeking) (further | more) information about
- ADJ -> curious about | interested in [exactly | learning more about]
A.3 Human Paraphrasing
The whole paraphrasing job involved 37 annotators and cost around $26,000 in total. We employed the Appen crowdsourcing platform to collect the data. We plan to release the geographic characteristics of the annotator population along with the data.
B Licenses for Relevant Artifacts
MultiWOZ: Apache License 2.0
- Google Sechma-Guided Dataset: CC BY-NC-SA 4.0
- SIMMC 2.0: CC BY-NC-SA 4.0
GPT2: Modified MIT License
C Supplementary Details for Experiments
We do a hyper-parameter search for the training on both original dataset and augmented dataset and find the following setting: a batch size of 4 and learning rate of 5e-6 is the best one for both. We run at most 20 epochs for each experiment and do validation for every epoch, with an early stop step of 3. For each experiment, we run for three times with different random seeds and report the average value, along with the standard deviation. We run experiments with NVIDIA RTX A4000 GPU for totally 1440 hours.
C.2 Metric
Joint Goal Accuracy evaluates the performance of predicting dialog states. It counts one for each turn if the model successfully generate all slot values, otherwise count zero.
C.3 Supplementary Experiment Results