SlowGuess/NAACL_2022_OCR huggingface dataset:
Kun Qian†, Satwik Kottur‡, Ahmad Beirami‡, Shahin Shayandeh‡, Paul Crook‡, Alborz Geramifard‡, Zhou Yu†, Chinnadhurai Sankar‡
†Columbia University
{kq2157, zy2461}@columbia.edu, beirami@google.com {skottur, shn, pacrook, alborzg, chinnadhurai}@fb.com
As task-oriented dialog systems are becoming increasingly popular in our lives, more realistic tasks have been proposed and explored. However, new practical challenges arise. For instance, current dialog systems cannot effectively handle multiple search results when querying a database, due to the lack of such scenarios in existing public datasets. In this paper, we propose Database Search Result (DSR) Disambiguation, a novel task that focuses on disambiguating database search results, which enhances user experience by allowing them to choose from multiple options instead of just one. To study this task, we augment the popular task-oriented dialog datasets (MultiWOZ and SGD) with turns that resolve ambiguities by (a) synthetically generating turns through a pre-defined grammar, and (b) collecting human paraphrases for a subset. We find that training on our augmented dialog data improves the model's ability to deal with ambiguous scenarios, without sacrificing performance on unmodified turns. Furthermore, pre-fine tuning and multi-task learning help our model to improve performance on DSR-disambiguation even in the absence of in-domain data, suggesting that it can be learned as a universal dialog skill. Our data and code will be made publicly available.
Task-oriented dialog systems have been widely deployed for popular virtual assistants, like Siri and Google Assistant. They help people with tasks such as booking restaurants and looking for a hotel by searching databases with constraints provided by users. After retrieving a result from the database, a system may continue by conducting actions like making a reservation or providing more information about receiving the result. However, there can be multiple results from the database that match
Hotel	Restaurant	Shopping
Book a room for me at Hilton Hotel on Monday. Which one? The Hilton Central City or the Riverside Hilton Hotel. The second one. I have booked one room at the Riverside Hilton Hotel for one night on Monday	Find me a cheap Japanese restaurant. I have two restaurants that fit. Which one are you interested in? The Wagamama or the Sushi Lover. The Wagamama, may I have its phone number. The phone number of wagamama is 01223462354	what size is that shirt, and how much is it? could you clarify which item you're referring to? Yeah, my bad, the shirt. The black and white one above the red shoes. That shirt is a size XL, and is priced at $29.99.
Figure 1: Examples of disambiguation turns over three different domains.
the same constraints. For example, as shown in Fig. 1, the system finds two available hotels at different locations when the user is asking the system to help book a hotel. This kind of ambiguity stops system from proceeding until the system finds out which result the user looks for. Therefore, we need to enhance the system with the ability to resolve such ambiguity brought out by multiple items returned from database search. We call this type of ambiguity as database search result ambiguity (DSR-ambiguity).
Different from semantic ambiguous words (e.g. "orange" can be referred as either color or fruit), the DSR-ambiguity focuses on results from multiple database search results. Solving such disambiguation tasks consists of two steps: asking clarification questions and understanding user's corresponding answers. While there is a relatively larger body of literature focusing on when and how to give out the clarification question (Rao and Daumé III, 2018; Rao and Daumé, 2019; Kumar and Black, 2020), the focus on understanding user's answers/intents has been relatively sparse. Our work mainly focuses on improving model's ability of understanding the answers by augmenting two existing task-oriented dialog datasets: MultiWOZ (Budzianowski et al., 2018) and Schema-Guided Dataset (SGD) (Rastogi et al., 2019).
MultiWOZ and SGD are the most popular large-scale task-oriented dialog datasets, based on which most of the state-of-the-art dialog system models are commonly trained and evaluated. According to
$^{\S}$  Work done during KQ's internship at Meta AI.
Currently at Google Research.
Human Language Technologies, pages 1158 - 1173
our analysis, there are around  $66\%$  dialogs of the dataset contains multiple dataset-searching results, which means the DSR-ambiguity exists.
In this setting, ambiguities are skipped and the model trained based on these datasets can hardly handle the cases where users prefer to make their own choices among all the results satisfies the constraints. Furthermore, users should be given more detailed information about search results. Ideally, dialog models should provide the information and assist users to make choices, rather than picking one from the result list and recommending it to users. It is not necessary to list all the results, but enumerating 2 or 3 options would help increase user's engagement. To strengthen the model with the ability to handle the ambiguity, we propose to augment these two datasets with disambiguation turns, where the system provides all possible matched results and lets the user make their own decision based on the complete information.
Specifically, we first extract templates from the SIMMC 2.0 dataset (Kottur et al., 2021), which is a multi-modal task-oriented dialog dataset containing disambiguation turns but only covering two domains. Based on the extracted templates and database from MultiWOZ and SGD, we synthesize a one-turn dialog dataset, containing only the disambiguation turn, to check whether the model can learn the disambiguation from the data. To be applicable in reality, we expect the model to learn the skill of disambiguation without compromising the performance on other dialog skills. So, we propose to augment the MultiWOZ and SGD with disambiguation turns and train dialog models with the augmented dataset. To ensure naturalness and diversity of the automatically augmented dataset, we additionally recruit crowd-workers to paraphrase the modified turns.
In conclusion, our contribution includes:
1. We propose Database Search Result Disambiguation, a new dialog task focused on understanding the user's needs through clarification questions.
2. We provide a generic framework for augmenting disambiguation turns, and apply this framework to augment the two most popular task-oriented dialog datasets with disambiguation cases. We also conduct human paraphrasing for the augmented utterances in test sets.
3. We create a benchmark for the new task with pre-trained GPT2 model. The results
- Given dialog context (utterance + slots)
- Given optional results (system response + db results)
Given user's utterance of selecting result
- Target: Understand user's choice and go further based on that
Figure 2: For this disambiguation task, we assume the dialog context, system utterance including result list and user's answer are given. The goal is to extract the result that the user select and continue the dialog.
show that our augmented dataset enhances the model's disambiguation ability, while maintaining the performance on the original tasks.
2 Task Formulation
In this paper, we propose a new task called disambiguation in dialog database search. As shown in Fig. 2, the task assumes that we are provided with the dialog context  $c$ , the system response  $s$  which includes all the optional results, and the user's utterance  $u$  that make a choice. To avoid redundant option lists, we limit the number of options to less than five. The target of the task is to extract the entity of the result selected by the user.
The most popular task-oriented dialog datasets (MultiWOZ, SGD) do not contain many cases for the disambiguation task. In order to enable the dialog model to handle this task, we propose to augment these two datasets in three steps described in the following subsections.
3.1 Synthesizing Single-Turn Dialog
We first develop a single-turn dialog dataset. With this single-turn dataset, the fine-tuned dialog model can focus only on the disambiguation turns and learn the skill to solve the ambiguity problem. Fig. 3 shows an example of the dialog turn, which we would use through this section to introduce the dataset. In this dataset, each dialog turn consists of only a system utterance and a user response. The system utterance gives a list of options (marked in blue) and the user response makes a choice from the list (marked in red). The ground truth output is the named entity of the chosen result.
To synthesize the system and user sentences, we extracted templates from disambiguation turns from the SIMMC 2.0 dataset. For example, the system from SIMMC2.0 asks questions like "do you
Figure 3: An example of the synthesized single-turn dialog. The utterance templates are generated based on CFGs. The candidate entities (italicized) are sampled from the database of MultiWOZ or SGD. The selected entity (bolded) is sampled from the candidates.
mind being a bit more precise about which shoes you're curious about, the red one or the blue one" to solve ambiguity. We delexicalize those utterance by removing the all domain-related tokens such as "shoes", "the red one", "the blue one" based on the annotations from the dataset, and keep the rest as a template.
We then extract a list of context-free grammars (CFGs) from those templates, and then generate natural sentences based on the CFGs. For example, from the previous template we can summarize a grammar: "SENT -> do you mind VERBING", where "VERBING" is a non-terminal token for a verb phrase in an "ING" form. More detailed CFG examples are shown in Appendix A.2. The CFG-based generator can potentially generate around 2 million different system questions and  $30\mathrm{K}+$  different user utterances, which ensure the diversity of the generated data. To cover multiple domains, we utilize the database from the MultiWOZ and SGD datasets, which in total covers 27 domains, each containing one named entity type. We randomly sample a certain number of values from the database based on the domain and entity type, and insert them into the system response. The number of candidate values is also randomly sampled. To make the sentence more natural, we limit the candidate number to be between three and five. Then, we randomly sample one from the candidate list as the selected result.
To make the task harder and more realistic, we also explore different entity addressing methods to generate the user utterance:
- Positional Addressing. Instead of directly addressing the named entity (Fig. 3), users use entity's list position, e.g., "the second one".
- Partial Addressing. User use part of the name for simplicity, e.g. "chiquito" instead of "chiquito restaurant bar"
Figure 4: The blue bar represents the number of dialogs which contain multiple database-search results in each service from the SGD dataset. While the red bar represents the total number of dialogs in each service.
- Addressing with Tipo. We add typos in the named entity to make the model more robust.
- Multiple Addressing. User chooses more than one option at a single time and the model is expected to extract all their choices.
- Addressing with Attributes. User describes the selected result with more attributes, e.g. "the restaurant in the north of the city".
3.2 Automatic Augmentation
The single-turn dialog dataset helps enable models to solve the disambiguation task. However, the single-turn is not an entire dialog and the model barely trained with that can hardly conduct a complete dialog. Our goal is to enhance a complete dialog model with the disambiguation skill while keeping the performance of other tasks. Currently, most of the state-of-the-art task-oriented dialog models are trained with MultiWOZ and SGD dataset. Therefore, we propose to augment these two datasets by adding disambiguation turns.
Fig. 4 shows the proportion of the dialogs in each domain that contains multiple results. We find that nearly  $66.7\%$  of dialogs involve multiple results, where ambiguity can occur. Though in both SGD and MultiWOZ, system would always give a suggestion after searching the database, e.g. "I have 10 suitable results, how about ..." and the user side would simply accept it or ask about something else. This avoids the ambiguity in the dataset. However, the system in the reality would still face the ambiguity problem when interacting with real human beings, who would like to know more about other options. Therefore, we want to augment these
SGD	MultiWOZ
train	dev	test	train	dev	test
dialog	4.7k / 16k	0.9k / 2.5k	1.6k / 4.2k	2.7k / 8.4k	0.3k / 1k	0.3k /1k
turn	5.1k / 330k	1.0k / 48.7k	1.8k / 84.6k	3.2k / 105k	0.4k / 13.8k	0.4k / 13.7k
Table 1: The table presents the numbers of dialogs or turns that are modified for disambiguation cases, and the numbers on the right side of slash are the total number of dialogs or turns in each dataset.
Figure 5: An example of the automatic disambiguation augmentation and human paraphrasing. We first replace the original system suggestion with a synthesized utterance, listing all candidate entities and asking user to select. Then, we generate user chosen answer and insert it to the beginning of the original user utterance. For human paraphrasing, we ask crowd-workers to rewrite the user utterance to gain naturalness and diversity.
two popular dataset with disambiguation turns to improve the model's ability.
First, we locate the turns to be modified. In those turns, the system presents the database-searching results, where the ambiguity takes place. We also incorporate relevant annotation and sentence structure to filter out some inappropriate cases, e.g. the user does not make any choices in this turn. Then we generate a new system utterance to replace the original one. The generation is conducted based on the same toolkit and CFGs from Sec. 3.1, and the slot values are extracted from the corresponding database. As shown in Fig. 5 (highlighted in blue), the new system utterance provides a list of specific searching results without giving any suggestion. Following the language naturalness, we uniformly sample two to four candidate searching results and integrate them with the original entity to compose the result list. After the system utterance, a user utterance is also generated to make the choice, which should be consistent with the original suggestion that the user accepts. If the user rejects the original system suggestion, we do not make any modifica
tion. In the end, we concatenate the generated user utterance with the original one. In this way, we ensure the other unchanged turns of the dialog (especially the following turns) will be coherent with the modified turns, in order to eliminate the effects on the unchanged turns of the dialog as much as possible.
We conduct the same progress on both SGD and MultiWOZ dataset. Note that the ambiguity problem occurs only when there is a specific target entity, e.g. hotel name in the "hotel" domain and not every domain includes such an entity (e.g. any car satisfying constraints is acceptable in the "taxi" domain). Therefore, we only augment the "restaurant", "hotel", and "attraction" domains in the MultiWOZ dataset, and 24 out of 45 services in the SGD dataset, which are listed in the Appendix A.1. The statistics of the augmentation is listed in the Table. 1. More than  $30\%$  of dialogs are involved and with disambiguation turns, and around  $2\%$  of the turns are modified.
The newly generated user utterance is simply the concatenation of the template utterance and the original utterance that responds to the system suggestion. Therefore, the connection between them can be unnatural. In addition, the new user utterance is generated by CFG, which means the utterance itself can be unnatural. Therefore, we conduct human paraphrasing to improve the quality of the user utterance.
3.3 Human Paraphrasing
We recruit crowd-workers to paraphrase the disambiguation turns. Before starting the paraphrasing job, each crowd-worker is required to read through a guideline document to get a better understanding of the task, the requirements and the workflow. A screenshot of the paraphrasing interface is shown in the Appendix Fig. 6. For each paraphrasing job, we present a good example of paraphrasing in the same page as the turn to be modified. To keep consistent with task description in the Sec. 2, we provide the crowd-workers with 1) the modified system utterance, which includes a list of options and asks the user to select, 2) the user utterance, which con
catenates the template-generated sentence and the original user utterance. In the interface, the user utterance is highlighted in a different color (green) and marked as "need paraphrase". To avoid changing user's original choice during paraphrasing, we also show crowd-worker the result value that the user should choose, keeping consistent with the dialog state annotation. In addition, to ensure the disambiguation turn is coherent with the dialog context, we also present the previous user utterance and the next system response.
We conduct the paraphrasing job for the test sets from both SGD and MultiWOZ, as well as the training set of SGD. To evaluate the quality of the human paraphrase process, we randomly sample  $5\%$  of the disambiguation turns and ask another group of crowd-workers to judge whether the modification is valid, which means satisfying all the requirements listed in the guideline document (maintaining all essential information, not similar to the original utterance, not natural, etc.). Each turn receives two judgements. In total, we have an  $88\%$  of agreement rate between two judgements and  $92\%$  of the agreements are error free, which means our paraphrasing job is valid. We also ask annotators to point out if there is any ethical violation in the utterance, which is discussed in more details in Sec. 7.
We use GPT2 (Radford et al., 2019) as our backbone model and fine-tune it with the augmented SGD and MultiWOZ datasets separately.
MultiWOZ. MultiWOZ (Budzianowski et al., 2018) is a multi-task task-oriented dialog dataset. It covers seven domains and contains  $10\mathrm{K}+$  dialogs. Our augmentation focuses mainly on three domains: "attraction", "hotel" and "restaurant", involving more than 3K dialogs. We choose to conduct our augmentation based on the MultiWOZ 2.2 (Zang et al., 2020), which is the most widely-accepted version.
Schema-Guided Dataset. SGD (Rastogi et al., 2019) is another popular multi-task dialog dataset. Since the DSR-ambiguity problem requires the service containing a target entity and not every service satisfies that requirement, our augmentation involved totally 10 domains and 24 services.
We directly compute the accuracy on whether the model can successfully predict the correct named
entity as evaluation metric. Since the generation is similar to the dialog state tracking task, we also compute the joint goal accuracy (details in Appendix.C.2) to evaluate whether the augmentation maintain the model's performance of other tasks.
We train GPT2 with both the original and augmented data, and test the fine-tuned models on original/augmented/human paraphrased test sets. The same experiment is conducted for both datasets. In addition to original and augmented training data, we also explore the impact of the synthesized single-turn dialog. Learned from Table 1, the augmented turns only take up  $2\%$  of the whole dataset. In order to achieve a similar amount of augmentation compared to the automatic augmented data, we sample  $5k$  synthesized single-turn dialogs for SGD and  $3k$  for MultiWOZ, which is around  $2\%$  of each training set. Then, we mix those dialogs with the original (or augmented) training data and evaluate on three test data settings. We also increase the sampling amount of the synthesized dialog to be comparable to the whole training set, represented by "Syn $100\%$  in the table, to explore whether the model achieves a better learning of the entity disambiguation skill with access to more disambiguation cases.
In this section, we present our experimental results including key observations and ablation studies. In addition, we also analyze how to leverage our augmented dataset to deal with DSR-ambiguity in new datasets.
5.1 Augmentation Helps Resolve Ambiguity
Table 2 shows the named entity prediction accuracy evaluated only on the turns involved in augmentation, which is around  $2\%$  of the whole test set. The first column states the different training data settings that we use to fine-tune the GPT2 model, and the first row presents three different test sets.
Comparing the "Origin" column and "AutoAug" column, we find that the performance of the model trained with original data drastically drops from 0.556 to 0.242 for SGD and from 0.676 to 0.488 for MultiWOZ. This verifies our hypothesis that the original datasets contain few disambiguation cases. Therefore, the model trained with the original data cannot understand user's answer towards the clarification question and extract the corresponding entity tokens. On the other hand, the models trained with
Test Data Train Data	SGD	MultiWOZ
Origin	AutoAug	HumanAug	Origin	AutoAug	HumanAug
Origin	55.6±0.7	24.2±00.6	21.1±0.8	67.6±0.7	48.8±0.5	48.8±0.1
Origin+Syn2%	57.5±1.4	27.9±2.5	25.2±1.8	65.0±0.3	48.9±1.4	49.4±1.6
Origin+Syn100%	57.1±0.6	34.4±1.8	30.4±1.5	67.0±0.7	55.4±2.6	55.6±2.9
AutoAug	55.1±0.2	49.6±0.5	43.7±0.8	63.3±1.6	74.4±2.5	73.9±2.9
AutoAug+Syn2%	56.9±0.4	54.8±1.0	48.8±1.7	64.2±0.7	83.8±0.2	83.0±0.7
AutoAug+Syn100%	56.7±0.9	58.3±0.1	50.1±0.2	63.3±1.3	84.6±0.2	83.7±0.7
Table 2: The accuracy of the named entity prediction for only the augmented turns. Each number represents the performance of a model trained with a certain training data setting and evaluated on a certain test set. "Origin"/"AutoAug"/"HumanAug" represents evaluation on the original/automatic augmented(Sec. 3.2)/human paraphrased(Sec. 3.3) data. "+"Syn" represents mixed with synthesized data and the percentage following "+"Syn" means the amount of synthesized data compare to the whole test set.
augmented data achieve better performance (from 0.242 to 0.496 for SGD and from 0.488 to 0.744 for MultiWOZ) on the augmented data, which means those models learn the skill to complete the disambiguation task. The results on the human paraphrased test set, which is more diverse and natural, support the same conclusion. We also combine the synthesized single-turn dialog data with the original training data (or the augmented training data). The original data mixed with full-size synthesized data setting achieves the best result on human paraphrased test set for SGD and the augmented data mixed with full-size synthesized data setting achieves the best one for MultiWOZ.
Table 7 shows the overall named entity accuracy of the whole test set. Since the augmentation only modifies  $2\%$  turns of the whole test set, the difference between the performance of on the original and augmented test set is not as apparent as Table 2. However, the model trained with augmented data still performs better than the model trained with original data on both augmented and human paraphrased test set. The model under "Aug+Syn100%" train setting achieves the best results on five out of six test sets, showing that the augmentation and synthesized data jointly enhance the model's ability to extract named entity.
In addition to named entity prediction, we also explore whether the augmentation helps the model to predict other slot types by computing the joint goal accuracy. Table 8 shows the results for only the augmented turns and Table 3 lists the results on the whole test set. In both tables, the setting "Aug+Syn100%" achieves the best or the second best performance for both augmented and human paraphrased test sets. Hence, our augmentation not only enables the model to solve the disambiguation task, but also improves its ability for dialog state tracking task. The improvement mainly results
from the similarity of the disambiguation task and the dialog state tracking, and more augmented data points enhance the model's understanding of the input sequence.
5.2 Augmentation Brings No Harm