Injecting Comparison Skills in Task-Oriented Dialogue Systems for Database Search Results Disambiguation
Year: 2023

Abstract:
In task-oriented dialogue (TOD) systems designed to aid users accomplish specific goals in one or more domains, the agent retrieves entities that satisfy user constraints from the database. However, when multiple database search results exist, an ambiguity occurs regarding which results to select and present to the user. Existing TOD systems handle this ambiguity by randomly selecting one or few results and presenting their names to the user. However, in a real scenario, users do not always accept a randomly recommended entity, and users should have access to more comprehensive information about the search results. To address this limitation, we propose a novel task called Comparison-Based database search Ambiguity handling (CBA), which handles ambiguity in database search results by comparing the properties of multiple entities to enable users to choose according to their preferences. Accordingly, we introduce a new framework for automatically collecting high-quality dialogue data along with the Disambiguating Schemaguided Dialogue (DSD) dataset, an augmented version of the SGD dataset. Experimental studies on the DSD dataset demonstrate that training baseline models with the dataset effectively address the CBA task. Our dataset and code will be publicized.

heading: 1 Introduction
text: Task-Oriented Dialogue (TOD) systems that help users accomplish specific goals in one or multiple domains have become a subject of great interest in both academics and industry (Chen et al., 2017; Hosseini-Asl et al., 2020; He et al., 2022; Wang et al., 2022). The system performs a database (DB) search (Sukhbaatar et al., 2015; Yin et al., 2015) based on user-specified constraints, confirms the target entity, and then helps the user with the de-
*Equal contribution †Corresponding author
sired action; for example, making a reservation or providing the phone number of a restaurant.
In general, DB search ambiguity occurs when two or more DB search results satisfy all the user search criteria. As shown in Figure 1 (top), existing TOD systems (Budzianowski et al., 2018; Li et al., 2018; Rastogi et al., 2020; Li et al., 2021) address this ambiguity by randomly selecting a single object and presenting it to the user. However, in real-world scenarios, users do not always accept the entity that is randomly recommended by the system. Accordingly, we argue that users should
4047
be granted access to more detailed information regarding the search results. Moreover, large DB in real-world scenarios makes effective methods of disambiguation imperative to apply TOD in practice. A recent study (Qian et al., 2022) attempted to solve DB search result ambiguity by selecting a few options, listing their names, and presenting them to the user. However, forcing the user to select one, given only the entity names, means that the subject of random selecting has passed from the agent to the user, which is neither effective nor user-friendly.
In this work, we propose a novel DB search disambiguation task that presents the characteristics of each entity in the form of a comparison with multiple entities: Comparison-Based DB search Ambiguity handling for dialogue (CBA). This task enables users to choose a desired entity based on their preferences by comparing the properties of multiple entities in DB search results. As in Figure 1 (bottom), the two presented options have an additional difference while satisfying the user’s requirement, for example, whether a restaurant allows pets or serves a vegan menu. The agent handles the DB search ambiguity by presenting the contrasts between the two entities as a trade-off for the user, enabling them to pick what they want. The human evaluation results show that the proposed disambiguation method is more effective and helpful than existing methods.
Furthermore, we present a novel framework for modifying datasets to solve CBA. The framework finds the turn that fails to effectively address DB search ambiguity and modifies it as a comparisonbased disambiguating turn. Subsequently, a user utterance and a confirmation utterance are added so that the generated utterances construct a seamless connection with the front and back of the dialogue. All three utterances are generated with InstructGPT (Ouyang et al., 2022) without human engagement, such as human paraphrasing, reducing time and cost consumption. We apply this framework to Schema-Guided Dialogue (SGD) (Rastogi et al., 2020) dataset and introduce a new dataset, Disambiguating Schema-guided Dialogue (DSD) dataset. A human evaluation compared with the SGD dataset is conducted to validate the high quality of the proposed dataset.
The experimental results reveal that the model trained with the DSD dataset achieves high performance in the CBA task, which means that the
model can effectively solve DB search ambiguity. In addition, we conduct zero-shot experiments for each domain using the domain-specific schema in DSD, demonstrating the potential for domain adaptation performance in the CBA task.

heading: 2 Related Works

heading: 2.1 Task-Oriented Dialogue System
text: In general, the dialogue system can be divided into two types: open-domain (Huang et al., 2020) and task-oriented. Open-domain dialogue refers to carrying out conversations in situations where there is no specific topic or user goal. On the other hand, task-oriented dialogue systems assist users in achieving their goals in a specific domain. Existing research on task-oriented dialogue systems can be categorized into pipeline and end-to-end approaches (Liu and Lane, 2018; Hosseini-Asl et al., 2020), which differ in whether they handle user input, determine the system’s next action, and generate responses using multiple modules or a single module. However, both types of task-oriented dialogue systems require the ability to perform relevant database searches and process results to achieve user-specified objectives. In this study, we propose a method to effectively address the ambiguity of search results that may arise during the database search process.

heading: 2.2 Schema-Guided Dialogue (SGD) dataset
text: Compared to most existing TOD datasets (Budzianowski et al., 2018; Shah et al., 2018), the Schema-Guided Dialogue (SGD) dataset is designed to share information between domains or slots. They define natural language descriptions of slots and intents called schema to enable models to obtain a semantic representation of each schema element. The SGD dataset is one of the most extensive publicly available TOD datasets containing over 16,000 dialogues in the training set. Dialogue outlines are created using domain-specific templates and paraphrased into natural language utterances through crowdsourcing.
However, similar to other TOD datasets, the SGD solves database search ambiguity by randomly selecting one of the candidates, despite the fact that about 67% of dialogues have DB ambiguity (Qian et al., 2022). Accordingly, we present a new dataset that handles database search ambiguity based on a comparison by extending the SGD dataset. In addition, one of the purposes of the SGD
dataset is to test the models’ zero-shot performance (Lee et al., 2022); therefore, the test sets contain unseen services. Since this train-test distribution discrepancy is out of the scope in disambiguation studies, we configure the dataset by modifying only the training dataset of SGD.

heading: 2.3 Database Search results disambiguation task
text: Qian et al. (2022) address the fact that existing TOD systems and datasets cannot effectively solve the DB search result ambiguity. Among the two steps for disambiguation, asking clarification questions (Rao and Daumé III, 2019; Aliannejadi et al., 2019; Zamani et al., 2020) and understanding a user’s corresponding answers, they focus on the latter and aim to train a model to better understand users’ answers. In the former aspect, the system utterance after the database search is limited to selecting a few options from possible candidates and simply listing their names to the user. The user must choose one solely based on the entity names without being provided with any additional information to make a decision. We argue that this is not a helpful disambiguation method from the user’s perspective; rather, the subject of random selection moves from the agent to the user. Therefore, we consider that users should be provided with extensive details about the search results, which can be used as a reference when making user-specific decisions.

heading: 3 Comparison-Based database search Ambiguity handling for dialogue

heading: 3.1 Task proposal
text: We propose a new task called Comparison-Based database search Ambiguity handling for dialogue (CBA) to solve ambiguity in multiple candidates from a database search in a user-friendly manner.
Existing TOD systems address the ambiguity by randomly selecting and listing one or a few options among DB search results. The user is forced to accept or select one entity from the list without detailed information about the listed options.
Therefore, we argue that systems should handle the ambiguity in a more informative and userfriendly manner, as illustrated in Figure 1. When offering entities through database search results, the two entities are presented in a trade-off form so that the users can directly compare the characteristics of each entity and select based on their preferences.
Figure 2 shows the results of human evaluation on three criteria comparing the three methods of handling the DS ambiguity task: comparison-based ambiguity handling, randomly selecting one option (extracted from the SGD dataset) and listing the names of a few options (created in a rule-based manner). In each comparison, it can be seen that more people feel that the CBA method provides additional information for the user to achieve a goal and that the utterance of CBA is more helpful and logical than existing methods.

heading: 3.2 Task formulation
text: To solve this task, the model must present the characteristics of the two entities in the form of a tradeoff through the response. Additionally, the dialogue state and entity name included in the user utterance corresponding to the compare-based system response should be extracted.
Therefore, this task has three objectives. First, agent response is generated in the form of a tradeoff when dialogue context and action policy are given. Second, the dialogue state is extracted from the user utterance when dialogue context is given. Finally, the desired entity name in the system response is extracted through the user utterance.

heading: 4 Disambiguating Schema-guided Dialogue (DSD) Dataset
text: We introduce a new dataset, Disambiguating Schema-guided Dialogue (DSD), along with an automatic framework for constructing high-quality dialogue datasets to solve the CBA task. The DSD dataset includes (1) utterances that solve the ambiguity of database search results and (2) corresponding data frames including a new dialogue act COMPARE. We generate comparison-based disambiguating utterances and replace a target turn with them; the target turn represents the turn where the agent recommends the randomly selected entity to the user to resolve DB search ambiguity in the SGD dataset. The generated utterances are designed to connect seamlessly with the existing dialogue. Specifically, as shown in Figure 1, DSD contains one modified utterance and two additional utterances for the CBA task. The first is a modified system utterance that presents two entities along with additional information about the entities in the form of a trade-off. The second is an additional user utterance in which the user mentions the desired entity according to their preference. To make the dataset more challenging, the user does not directly mention the entity name but the user’s preference for the characteristics of the desired entity. Finally,
a confirmation utterance is followed by the system that confirms the selected entity.

heading: 4.1 Dataset Construction
text: We introduce a novel framework for constructing high-quality datasets with minimal human engagement. The framework leverages a Large Language Model (LLM) (Wei et al., 2022; Thoppilan et al., 2022) for the utterance generation part, which is the most expensive portion of dialogue dataset generation. We use InstructGPT as LLM, a model that is additionally trained with human feedback using reinforcement learning on the GPT-3 (Brown et al., 2020) model. Figure 3 represents the overview of proposed framework to automatically create a high-quality dataset, even with advantages in terms of time and cost.
Find Target Entity & Select Candidate Entity The first step is to find the target entity in the dialogue. We set the entity recommended by the agent in the existing dialogue as the target entity to maintain the flow in the unmodified turns after decision making. Afterward, among the multiple database search results, the agent extracts options that can be presented to the user, excluding entities already rejected by the user in the existing
dialogue. Subsequently, one candidate entity to be compared with the target entity is randomly selected.
Find Target Turn The following step is to find a target turn in the existing dialogue where DB search ambiguity occurs but fails to effectively disambiguate the DB results. The utterances presenting a DB search result can appear several times in one dialogue due to the user rejection; however, we set only the turn in which the agent presents the target entity as the target turn.
Slot Value Attribution We design additional slots for each domain so that the target entity and candidate entity can be compared through new slots that do not appear in the existing dialogue. The domain-specific slots, which are collected by investigating various TOD datasets (Quan et al., 2020; Zhu et al., 2020; Zang et al., 2020), are listed in Table 6 in Appendix A. Automatically, two slots are selected from the list, assigned felicitous values, and used as properties for trade-off form recommendations. For example, in Figure 3, nearby_station and has_parking_lot are selected, and the values are given crosswise to the two entities. For the slots that are not selected for this dialogue, random values are assigned.
Utterance Generation Based on the selected entities and slots, an entity recommendation response based on a comparison is generated using InstructGPT. Additionally, to increase the diversity of the DSD dataset, we proceed with automatic paraphrasing with InstructGPT using the prompts shown in Figure 3.
User utterance is then generated through the InstructGPT. To make user utterance grounding more challenging than directly mentioning the desired entity name or position (e.g., "the second one"), it is designed to mention the desired property. Additionally, 50% of user utterances are randomly selected, and a sentence mentioning the dontcare slot is added, which means the slot is not considered a constraint for decision-making. The order of the sentences corresponding to dontcare and the sentences mentioning the target property are mixed back and forth with a probability of 0.5. Finally, contextual slot-value pairs are added to the belief state, as illustrated in Figure 3. Similar to system utterances, user utterances are automatically para-
phrased through InstructGPT. Confirm utterance is generated to mention and confirm the target entity selected by the user for seamless connection with the following dialogues. All prompts for generating system utterances, user utterances, and confirm utterances, and for paraphrasing are shown in Figure 3.
Revising Data frame Subsequently, the data frames are revised according to the generated utterances. In the case of the system utterance, two recommended entities are added to the data frame as an OFFER dialogue act. Moreover, the COMPARE act is added since information about compared slots and values should be provided. It is a new dialogue act that does not exist in the existing SGD, indicating that the two slots and corresponding values are compared in the system utterance. Second, in the case of a user utterance, one or two slots mentioned in the utterance are added to the belief state in the form of a slot-value pair. Finally, the OFFER act is added to the data frame for confirmation utterance.
Postprocessing Finally, through post-processing, minor impurities in the utterance generation process by the machine are removed, and an arbitrary values of the additional slots are given to the target dialogues and other dialogues. We perform an entity name check on the texts generated by InstructGPT for quality assurance.
We separately provide generated sentences and automatically paraphrased sentences through DSD release. The paraphrased version can be used for data bootstrapping or augmentation.

heading: 4.2 Statistics
text: The statistics for the DSD dataset are shown in Figure 4. We proceed with turn modification for six domains: gray is the number of dialogues with multiple database search results in the corresponding domain and blue is the number of dialogues modified through the proposed framework. If the database search result does not exist because the user presents the target entity before the agent, or if there is no entity left after excluding the mentioned entity and the target entity (if only the already rejected entities remain) is an exception, the turn is not modified through our framework. In the case of event and movie domains, even if there are mul-
tiple database search results, only non-categorical slot values, such as time and place are the only differences between each result. Since the comparable two entities do not exist, these cases are excluded, resulting in a high exception rate in the two domains.

heading: 4.3 Human evaluation
text: Figure 5. illustrates the results of human evaluation comparing the quality of target turns from the SGD and DSD datasets on three criteria: overall quality, fluency, and naturalness. SGD is paraphrased by humans after generating an outline based on a template, whereas DSD is generated without human engagement using the automated framework presented above. Nevertheless, the results show that DSD precedes SGD on all three criteria. In other words, the proposed framework has proven to be an automated framework that produces high-quality datasets with cost and time advantages over human paraphrasing methods.

heading: 5 Experiment
text: We present baseline experiments on the DSD dataset. For the three subtasks presented in Section 3.2, we formulate the problems as dialogue generation (DG) (Wen et al., 2015; Peng et al., 2020), dialogue state tracking (DST) (Williams et al., 2016; Jacqmin et al., 2022), and named entity recognition (NER) (Bowden et al., 2018; Liang et al., 2020).
Experiments are conducted using the DSD and SGD datasets for the three subtasks. Because DSD is created by transforming only the training dataset of SGD, the test set is formed by splitting the train-
ing set at a 9:1 ratio. For a fair comparison, we also split the training set of the SGD, use them as training and test set, and split-indices are shared with the DSD.

heading: 5.1 Dialogue Generation (DG)
text: To solve the CBA task, the model should be able to generate the response by presenting the characteristics of the two entities in a trade-off format after a database search. Therefore, we experiment to check whether a disambiguating system response can be well generated when an action policy, including the dialogue context and COMPARE act is given. Because this generation process is similar to DG, we formulate this subtask as DG and measure the N-gram-based (Lin and Hovy, 2003) metrics of NLG: BLEU-4 (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGEL (Lin, 2004), and CIDEr (Vedantam et al., 2015). T5-small (Raffel et al., 2020) is used as the backbone model. A more detailed experimental setup can be found in the Appendix B.
Table 3 summarizes the results of the experiment to measure whether system utterances are well generated for the entire turn of the test set after training with the SGD and DSD. First, the SGD test set results show that the DSD-trained model does not degrade sentence generation compared to the SGDtrained model. Therefore, it is observed that turn modification does not harm the learning of existing dialogue generation.
Based on the results of the DSD test set, when trained with SGD, there is a performance drop in all four metrics; however, when trained with DSD, the generation performance is maintained, as in the existing dialogue dataset. Therefore, it can be seen that ambiguity can be effectively resolved through DSD learning with the turn modified to
suit the CBA task. In particular, as listed in Table 4, from experimenting with a test set by collecting only target turns containing compared-based disambiguation, we confirm that the model trained with the SGD dataset do not generate target sentences at all; however, in the case of DSD, it maintains good performance. Therefore, it can be confirmed through DSD training that the model performs well in compare-based disambiguating response generation, which cannot be achieved using the conventional dataset training.

heading: 5.2 Dialogue State Tracking (DST)
text: DST, a core of dialogue systems, aims to retrieve pertinent information from user-system dialogues. In the CBA task, the model should be able to determine the desired slot in the utterance containing the user’s preference corresponding to the comparebased system response. Therefore, we conduct a DST that extracts the user’s belief state when the dialogue context, system utterance that presents database search results based on the comparison, and user utterance are given.
We measure the joint goal accuracy and required slot F1 from user utterances with settings similar to those in Experiment 5.1. The joint goal accuracy evaluates the performance of predicting the dialogue states. It counts one for each turn if the model successfully generates all slot values; otherwise, it counts zero. The required slot F1 is the F1 score between the prediction and ground truth slot values. The backbone model used is T5-small.
The experimental results are listed in Table 1. The model trained with the SGD/DSD whole turn is evaluated on the test set of the SGD entire turn, DSD entire turn, and test set consisting only of the DSD target turn. Overall, we observe similar tendency with the Experiment 5.1. Learning through DSD does not adversely affect dialogue state tracking in unmodified turns. Contrarily, the model trained with the existing dataset cannot understand the comparison-based disambiguation problem, whereas the model trained with the DSD shows high performance for the DSD test set. Therefore, it can be seen that the grounding ability for the corresponding user answer after the clarification question is granted through DSD learning.

heading: 5.3 Named Entity Prediction (NER)
text: To solve the CBA task, the model should infer the correct entity name from the user utterance. Therefore, we formulate this subtask as an NER task. In this experiment, system utterance recommending two entities with additional information and corresponding user utterance are used as inputs to evaluate whether the model could extract the correct entity from the system response. We conduct NER experiments with two settings: classification and generation.
First, we use the BERT (Devlin et al., 2018) model for the classification task. Because the entity names are different for each instance, the two entity names mentioned by the system are replaced with special tokens for the correct classification
setting. For example, "<system> <entity0> has a shuttle bus service but no nearby restaurant, while <entity1> has a restaurant nearby but no shuttle bus. Which do you prefer? <user> When choosing an event, providing a shuttle is highly preferred!" is used as the input, and the label is 0 since the user is referring to <entity0>.
The experimental results are presented in Table 2. The average accuracy for the entire domain is 0.938, and when dontcare is included in the user utterance, the accuracy is 0.893. Adding dontcare sentences makes the task more challenging because a reasoning process is added to infer which of the two mentioned slots is the desired slot.
The GPT2 (Radford et al., 2019) model is used in the generation setting. During training, the input sentences and target entities are trained together, and during inference, the input sentences are given as prompts. We evaluate whether the target entity could be identified through exact matching. As summarized in Table 2, the accuracy is slightly lower than that of the classification setting. When the dontcare sentence is added, the tendency to be more difficult for the model to predict the entity name is maintained.

heading: 5.4 Domain adaptation ability of CBA
text: In addition to the three subtasks, we conduct dialogue generation experiments in zero-shot settings to evaluate the domain-adaptation ability of the CBA task. Disambiguating turns are collected to configure the training and test sets for this task.
Table 5 lists the experimental results for the zero-
shot setting (Wu et al., 2019) for each domain. In the case of the All setting, the experiment is performed on the target domain after learning with all domains. In the case of the ZS setting, a zero-shot experiment is conducted for the target domain after training with the remaining five domains, except for the target domain. The average value presented at the bottom of the table is the result calculated simultaneously by collecting the sentences generated in each experiment, reflecting the characteristics of the metrics.
Based on the experiment, although there is a slight decrease in performance in the zero-shot setting, it is shown that the model performs well in dialogue generation (e.g. BLEU-4 27.4 → 21.7, ROUGE-L 45.7 → 41.9). This performance degradation arises from the lack of understanding of the unseen slots, although they accompany descriptions of each slot through schema.
Therefore, the extent of performance degradation is large in domains with unique additional slots
not found in other domains, such as restaurant or home. However, although it is a zero-shot setting, the model understands the COMPARE act well and creates a dialogue in the form of comparing two entities. (See the Appendix. E) Therefore, we confirm the potential of the domain adaptation ability in the CBA task and present a baseline for zero-shot experiments of the CBA task for future work.

heading: 6 Conclusion
text: We propose a new task for TOD, comparison-based database search ambiguity handling (CBA). This task aims to handle database search result ambiguity by comparing the properties of multiple entities to allow users to choose according to their preferences, which has proven to be the preferred solution over existing methods. We also introduced a disambiguating schema-guided dialogue (DSD) dataset by augmenting the SGD dataset to solve our task. Experimental studies have demonstrated that the DSD can effectively inject comparison skills into the TOD systems.
Limitations
Increase in the number of dialogue turns
The DSD dataset has a higher average number of turns compared to the SGD dataset. (20.44→21.21) This is a limitation in terms of completing a task with fewer dialogue turns, one of the objectives of the TOD system (Liu et al., 2018; Tiwari et al., 2021). This is because DSD was created by extracting and augmenting the target turns of SGD. However, assuming that the TOD agent trained with the DSD is applied to real-world scenarios, we expect that the agent will play a role in reducing the number of user rejections by expanding the range of choices to users through compare-based disambiguation.
Ethics Statement
To ensure that our dataset does not involve any potential risks, we ask crowd-workers to check if the generated utterances contains any of following: 1. offensive, sexist, or racist comments, 2. toxic words, 3. sexual behavior. Detailed description, payment, and the interface to collect all human evaluations for this work can be found in Section F and Figure 8 in the Appendix.

heading: Acknowledgements
text: This work was supported by LG AI Research. This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University) & NO.2021-0-02068, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University)], the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2023, and the National Research Foundation of Korea (NRF) grant funded by the Korea government (No. 2021R1A2C2008855). K. Jung is with ASRI, Seoul National University, Korea.

heading: A Additional slots

heading: B Experiment details
text: B.1 Reproductability checklists
Dataset and Source code We provide our experiment source code along with configuration code as supplementary materials. We will publicly release our dataset DSD, and the full codes with weight parameters.
Computing Resources AMD Ryzen Threadripper 2950X (3.50 GHz) with GeForce GTX 2080 Ti is used for the experiments. We use four GPUs for our experiment. All codes are implemented on Python 3.7.13 and PyTorch 1.10.1.
B.2 Train/Test set details
We configured the DSD dataset by modifying only the training dataset of SGD as mentioned in Section 2.2. For a fair comparison, we randomly divided the entire dialog of SGD into a 9:1 ratio and stored it as a train set index and a test set index, respectively. Thereafter, DSD was also split into a train/test set using each index. We will also publish the train set index and test set index attached to the code. The total number of dialogs before split of SGD and DSD is 16,142, and the total number of turns is 164,159. The number of data used in the DSD target turn experiment is 6,197.
B.3 Experiment details for each subtasks
Dialog generation We use T5-small model1 as our backbone model for dialog generation experiment. We utilize the pycocoevalcap 2 tool to compute the scores of the NLG evaluation metrics. The number of parameter of our model is about 60M as like as T5-small. The model trains with batch size 32, and takes about 2 hours per epoch.
We use AdamW (Loshchilov and Hutter, 2017) optimizer with β1 = 0.9, β2 = 0.999, ϵ = 1e− 8. The max gradient norm for gradient clipping is 1.0. In order to find the best-performing model, we conducted experiments on 12 hyper-parameter combinations with 3 epoch steps : per_gpu_batch_size : (4, 8), initial_learning_rate : (1e− 4, 5e− 5, 2e − 5), warmup_step : (0, 500), num_epochs : (3, 5, 10). The hyper-parameter was manually tuned, and the best-performing model is with per_gpu_batch_size 8, initial_learning_rate 5e− 5, warmup_step 0, and num_epochs 10.
Dialog state tracking We use the T5-small model as our backbone for the dialog state tracking experiment. The format of the output of the T5 decoder is [DOMAIN_NAME]-[SLOT_NAME][SLOT_VALUE] using hyphen delimiter such as "restaurant-pets_welcome-true", and the ground truth is also transformed into this format to measure JGA by exact matching and F1 score for each slot. A slot-value pair whose value is None is not generated.
We also use AdamW optimizer and performed hyper-parameter tuning as in the dialog generation experiment, the best-performing model is with per_gpu_batch_size 8, initial_learning_rate 1e− 4, warmup_step 0, and num_epochs 10.
Named Entity Recognition We use the BERT model3 for classification setting and the GPT-2 model4 for generation setting. The number of parameters in classification model is about 110M as like as BERT, and the number of parameters in generation model is about 1.5B as like as GPT-2. We both use AdamW optimizer for the NER experiments. The best-performing hyper-parameters for BERT and GPT-2 are: per_gpu_batch_size 8, initial_learning_rate 1e − 4, warmup_step 0,
1https://huggingface.co/t5-small 2https://github.com/salaniz/pycocoevalcap 3https://huggingface.co/docs/transformers/model_doc/bert 4https://huggingface.co/gpt2
and num_epochs 10.

heading: C Example dialogs per domain
text: Table 7 shows the example dialogs of each six domains.

heading: D Examples added to the utterance generation models
text: Table 8 shows the randomly selected examples added to add_example function in InstructGPT models.

heading: E Example of domain adaptation experiment result
text: Table 9 shows an example of domain adaptation experiment result. The agent response is generated in a compare-based form, but the unseen slot (’two_story’ in this example) is not well understood.

heading: F Human Evaluation Details
text: As shown in Figure 2 and Figure 5, we conducted three human evaluations via Amazon Mechanical Turk. For each evaluation, 100 of the DSD dataset target turns were randomly sampled and evaluated by five crowdworkers each. The crowdworkers were paid $0.07 per HIT, and it cost $160 in total. We hired workers from countries where English is a primary language; AU, CA, NZ, GB, and US. The interface to collect human evaluation can be seen in Figure 6 and Figure 7.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We discuss the limitations of the study in the Limitation section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: We discuss the potential risks of the study in the Ethics section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: We present our main arguments well summarized in the Abstract and 1. Introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? SGD dataset, pretrained T5 model, pretrained BERT model, pretrained GPT-2 model, pretrained InstructGPT model are used for this work. (2.1 SGD dataset section, 5 Experiment section)

heading: 3 B1. Did you cite the creators of artifacts you used?
text: All used datasets and models are well cited. (2.1 SGD dataset section, 5 Experiment section)

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: We used publicly available dataset and models, and these are discussed in Section 2.1 SGD Dataset and (Appendix) B Experimental Details.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? A new dataset was augmented using an open dataset, and it is specified in 2.1 SGD dataset section. The pretrained models are also used as intended. (Appendix) B Experiment Details section

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? There is no potential risk as the entity name refers to the widely used SGD dataset, and offensive comments are dealt with in the Ethics section.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? The domain and statistics of the dataset created in this study are covered in detail. (4 Dataset Construction section)
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We deal with the number of examples in section 4.2 Statistics section, and specify the train/test split in section 5 experiment section and section (Appendix) B.2 Train/Test set details section.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? (Appendix) B. Experiment details section describes computational experiments.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? The number of parameters, GPU hours, and used GPU are presented in section (Appendix) B. Experiment details section.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? The experimental setup or hyperparameter tuning is described in (Appendix) B.3 Experimental details for each subtasks section.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Statistics about experimental results can be found in Table 1 of the 5.1 Dialogue generation section, etc.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5. In the experiment section, the models, parameters, and metrics used in each experiments are discussed.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: It is explained in Figure 2 of 3.1 Task proposal section or 4.3 human evaluation section.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? (Appendix) E Human Evaluation Details includes MTurk screen capture and describes human evaluation details.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? (Appendix) E Human Evaluation Details section includes human evaluation details, e.g., paid participants, respondents’ location, payment, etc.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Crowdworkers did not build the dataset but were tasked with evaluating the machine-generated dataset. Instructions for the machine-generated dataset are found in (Appendix) E, Human Evaluation Details

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Ethics section and (Appendix) F Ethics considerations include human evaluation details for potential risk assessment.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: (Appendix) E Human Evaluation Details section mentions how the crowdworkers were assembled.

heading: Future Work
text: 2021-0-02068, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University)], the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2023, and the National Research Foundation of Korea (NRF) grant funded by the Korea government (No. 2021R1A2C2008855). K. Jung is with ASRI, Seoul National University, Korea