Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation (2024)

Abstract: Most task-oriented dialogue (TOD) benchmarks assume users that know exactly how to use the system by constraining the user behaviors within the system’s capabilities via strict user goals, namely “user familiarity” bias. This data bias deepens when it combines with datadriven TOD systems, as it is impossible to fathom the effect of it with existing static evaluations. Hence, we conduct an interactive user study to unveil how vulnerable TOD systems are against realistic scenarios. In particular, we compare users with 1) detailed goal instructions that conform to the system boundaries (closed-goal) and 2) vague goal instructions that are often unsupported but realistic (opengoal). Our study reveals that conversations in open-goal settings lead to catastrophic failures of the system, in which 92% of the dialogues had significant issues. Moreover, we conduct a thorough analysis to identify distinctive features between the two settings through error annotation. From this, we discover a novel “pretending” behavior, in which the system pretends to handle the user requests even though they are beyond the system’s capabilities. We discuss its characteristics and toxicity while showing recent large language models can also suffer from this behavior.

heading: 1 Introduction
text: Task-oriented dialogue (TOD) systems aim to accomplish specific user goals by comprehending their requests and making appropriate API calls or database (DB) searches (Young et al., 2013). TOD systems typically use a pipeline approach, connecting separate modules such as intent detection, dialogue state tracking, policy management, and natural language generation, often requiring complex rules or heuristics. End-to-end (E2E) TOD systems
†Work done as a research intern at NAVER AI Lab. ‡Work done while at NAVER AI Lab.
Low User Familiarity with the System
(I should provide price range, food type, area (north, east, south, west), and booking info)
Hi! Please book a fancy Italian restaurant in the south area. Five people at 13:30, on monday.
ABC restaurant is an Italian restaurant that matches your requests. Do you wish to reserve a table at the restaurant?
have been highlighted as a fully data-driven solution because of their concise implementation (Bordes et al., 2017; Wen et al., 2017). Recently, such TOD systems have significantly improved on top of pre-trained language models (Hosseini-Asl et al., 2020; Ham et al., 2020; Peng et al., 2021; He et al., 2022).
However, despite the numerous studies on TOD systems and the great successes of large language models, we argue that there is a huge gap between the TOD studies and deployable TOD systems. Among the many reasons hindering end-to-end systems from being widely adopted in the industry, the
37
instability of such systems makes it harder to match the robustness of conventional pipeline systems.
We hypothesize that the major source of this instability lies in the naive assumption about the users during TOD data collection. We call this the user familiarity bias, as illustrated in Figure 1. For instance, during Wizard-of-Oz style data collection (Kelley, 1984), the user-role workers are provided with detailed instructions on the goal they need to achieve which conforms with the system capabilities (Budzianowski et al., 2018; Byrne et al., 2019). Hence, as the user behaviors are strictly constrained, this process simulates users who know exactly how to use the system (Larson et al., 2019). Other datasets based on user simulation, such as M2M and SGD (Shah et al., 2018; Rastogi et al., 2020), include the same user familiarity bias, as they simulate users based on predefined user goals and rules specifying how to converse. On the other hand, real users in the wild often have fairly creative or vague goals way beyond the system coverage, and this user-side bias prevents us from modeling such realistic conversations.
In this paper, we conduct an interactive user study on one of the most-used Wizard-of-Oz-styled TOD benchmarks, MultiWOZ (Budzianowski et al., 2018), to investigate the impact of user familiarity. The main objective of this study is to determine whether the familiarity of the user with the TOD system’s capabilities influences the successful completion of a conversational task. To do this, we first divide the users into two groups: closed-goal and open-goal. The former user group is provided with detailed user goal instructions that are from the MultiWOZ, while the latter is given only a portion of the instructions along with some realistic goals that are not supported by the system, thereby simulating users who are not familiar with TOD systems. Based on each goal type they are assigned to, the users converse with a recent E2E TOD system, GALAXY (He et al., 2022), which is trained on diverse TOD datasets including MultiWOZ.
Our user study reveals that 92% of the dialogues in the open-goal setting has significant issues that often lead to failure in achieving the user goals. Moreover, we find that various inconveniences caused by the TOD system force users to quit the conversation regardless of the goal types. We thoroughly analyze the resulting conversations to identify the impact of user familiarity by annotating erroneous turns. In particular, we figure out
six prevalent errors in both goal settings. As expected, open-goal dialogues contain more problematic turns, and the open-goal setup causes more irrelevant and unfaithful responses.
In addition, we identify unfaithful responses as “pretending” behaviors of the system that primarily arises in the open-goal setting. This is a phenomenon similar to hallucination, in which the dialogue system pretends to handle the user’s requests even though they are beyond the system’s boundaries, but more potentially harmful because it is almost impossible for users to verify the reliability of the information during conversation since the hallucinated pieces of information are usually service-specific. We believe this issue is relatively underexplored as we witness most previous works focused on the closed-goal setting, and our qualitative analysis of the open-goal dialogues demonstrates that such pretending behaviors are prevalent and crucial.
Finally, we conduct case studies to check whether recent large language models with strong zero-shot performance can mitigate each conversational error. We show that large language models are proficient to handle errors within given context, but preventing pretending problem highly depends on the system design, not only on language models’ performance.
Our contributions are threefold: (1) interactive user study that breaks away from the closed-goal assumption; (2) examination of the characteristics of erroneous situations in both dialogue- and turnlevels; and (3) demonstration of the “pretending” problem of the TOD systems, especially as observed in an open-goal assumption, where the agent deceives users as if it handled their exceptional requests.

heading: 2 Related Works

heading: 2.1 TOD Benchmarks
text: MultiWOZ is one of the largest TOD benchmarks containing about 10,000 multi-domain dialogues (Budzianowski et al., 2018), following the Wizard-of-Oz protocol (Kelley, 1984). In this setup, human workers participate in a conversation as either a user or system role. To guide the user-side workers to behave within the desired scenarios, the authors provide goal instructions to the user-side worker detailing what to inform and request to the system-side worker. Meanwhile, the Taskmaster-1 dataset is more severe in that each
conversation of the dataset is collected by one human worker playing both user and system roles, namely the self-play method (Byrne et al., 2019). Thus, these datasets naturally contain significant user familiarity bias. Similarly, other datasets constructed by an automatic user simulator also contain the same bias since the simulation is based on predefined goals and rules bound to the system’s coverage (Shah et al., 2018; Rastogi et al., 2020).

heading: 2.2 Benchmark Reality in TOD
text: Recently, there have been studies concerning the reality of the benchmark dataset in the area of TOD. Kim et al. (2020) incorporate an external unstructured knowledge (i.e., FAQ) to complement the language model trained on limited scenarios. Even though the dataset includes knowledge-seeking turn detection to handle out-of-scope requests, it still assumes high user familiarity with the system in that users require information specified in the external knowledge. Qian et al. (2022); Kim et al. (2022); Yang et al. (2022) point out the limited coverage of dialogue expression by modifying the utterances of the user and system. Furthermore, Sun et al. (2021); Li et al. (2022); Young et al. (2022) improve the model’s natural conversation skills in terms of engagingness by combining with open-domain dialogue. However, we believe the combination cannot be a solution for dealing with users who have open-ended goals. On the other hand, Qin et al. (2021) argue inconsistent responses can be a more critical problem in TOD, and propose an accompanying new dataset to mitigate it.

heading: 2.3 TOD in Deployment
text: Potential issues related to interaction or deployment were discussed among communities. For example, Larsson (2017) mainly discussed technical and architectural difficulties in deploying dialogue systems. More similarly, Leuski and Artstein (2017) presented challenges where dialogue systems do not properly handle users’ sub-dialogues of different topics or domains (i.e., lack of affordance). However, these discussions were mainly tested on proprietary products such as Siri and Alexa.

heading: 2.4 Evaluation of TOD System
text: Many recent works evaluate performance using quantitative metrics for predefined slots and responses. Specifically, Budzianowski et al. (2018) define two task-specific metrics, Inform and Success rate, which measure how often the system has
provided the appropriate entities and answered all the requested attributes. In addition, BLEU (Papineni et al., 2002) is used to measure the fluency of the generated responses. However, Nekvinda and Dušek (2021) report inconsistencies in data preprocessing for these metrics in an attempt to make standardized evaluation scripts. Furthermore, Cheng et al. (2022) build a user simulator capable of dynamic evaluation to solve the static evaluation problem for TOD systems. However, the evaluation is still limited to the closed-goal setup.
Apart from the automatic quantitative evaluation, there are consistent works of user evaluation in spoken dialogue research (Walker et al., 1998; Ai et al., 2007; Gašić et al., 2008). Our work is more closely inspired by user studies in the Human-Computer Interaction (HCI) area that investigated live interactions between chatbots and users. In particular, Yeh et al. (2022) investigate the impacts of various guidance types and timing on performance metrics for TOD systems. Li et al. (2020) analyze conversation logs between users and TOD chatbots and claimed that identifying conversational “non-progress” is crucial for improving chatbot usability.

heading: 3 Interactive User Study
text: In this section, we explain the experimental setups of our interactive user studies on the current stateof-the-art data-driven TOD model. Our focus lies on creating realistic scenarios, breaking away from evaluation solely based on TOD benchmarks. In particular, we are curious about the influence of user familiarity on the TOD system. We describe the details of the study in the following sections.

heading: 3.1 User Goal
text: Most TOD systems assume the users have specific goals in a given domain, e.g., restaurant or hotel reservations. Typically, such goals can be represented by sentences in natural language to control user-side human participants when collecting dialogue data (Budzianowski et al., 2018; Byrne et al., 2019). The following is one of the user goal instructions provided in Budzianowski et al. (2018).
You are looking for a place to stay. The hotel should be in the north and should include free parking. The hotel should be in the type of guesthouse. Once you find the hotel, you want to book it for 3 people and 2 nights starting from wednesday. ... Make sure you get the reference number.
However, all user goals in most TOD benchmarks are based on a naive assumption that the users have sufficient knowledge about the dialogue system in advance. Thus, conversations based on such goals are always within expected scenarios from the TOD system’s point of view. On the other hand, we argue that most real users are not familiar with TOD systems, and such users are prone to making exceptional requests beyond the system’s capacity. To investigate the impact of user familiarity, we set up two user groups that have different types of goals considering their familiarity with TOD systems, which we refer to as closed-goal and open-goal, respectively.
Closed Goal Closed goals contain predefined scenarios which TOD systems can accomplish easily. In other words, it does not include any exceptional requests or actions from the perspective of the tested TOD system. As we mentioned, most dialogues in existing TOD datasets constructed based on such predefined user goals fall within the capacity that the system can correspond to. We use these user goals from the restaurant and hotel domains of MultiWOZ (Budzianowski et al., 2018) as our set of closed goals.
Open Goal Contrary to the closed-goal setting, open-goal settings are used to simulate realistic situations for users who have little idea about the TOD systems except for the domain. Real-world users may have a wider range of purposes than the predefined situations because the system capacity cannot include every possible scenario within its boundaries. Thus, we include exceptional requests which are not covered by the original dataset. Specifically, we create an open-goal by inserting the exceptional requests into a subset of closed-goal. By doing so, we are able to cover essential user requirements covered by the system (e.g., time to visit), while also simulating real-life requests that are unsupported. In our experiments, we limit the number of exceptional requests in a single open-goal to a maximum of two.
To construct the set of exceptional requests, we use InstructGPT (Brown et al., 2020; Ouyang et al., 2022) text-davinci-003 by OpenAI API. First, we input high-level task information as a prompt and let InstructGPT come up with the remaining requests to complete the task instruction. Table 1 is an example input prompt and output of the generated exceptional request. Then, we manually vali-
date the generated outputs to filter out noisy outputs and categorize commonly observed requests for the target domains into an ontology, which is shown in Appendix Listing 1.

heading: 3.2 Participants
text: We recruit 20 participants working at a tech company who meet our inclusion criteria1 : (1) having some experience with AI chatbots and (2) feeling comfortable carrying on written conversations in English. In terms of the number of participants, our sample size is congruent with the guidelines and standards for the sample size for HCI studies (Hwang and Salvendy, 2010; Caine, 2016). Following suggestions by Hwang and Salvendy (2010), we aim to recruit around 10 participants per group. We randomly assign participants one of the two conditions: Open-goal (N = 10; referred to as O1– O10; 7 females) or Closed-goal (N = 10; referred to as C1–C10; 5 females). Each group of participants is provided with the corresponding type of goal instructions.

heading: 3.3 Procedure
text: We implement a chat interface on Gradio (Abid et al., 2019) web platform illustrated in Appendix Figure 4. For the system agent, we use one of the most performant2 E2E TOD model (He et al., 2022) trained on diverse benchmarks including MultiWOZ 2.1 (Eric et al., 2020). Please note that
1Refer to Appendix A for exhaustive demographics. 2Based on the official MultiWOZ leaderboard: https:
//github.com/budzianowski/multiwoz.
this TOD model also contains the user familiarity bias in TOD benchmark datasets, and our experiment can be generalized to every model trained with datasets constructed with similar manner (refer to Section 2.1). We give users structured goals instead of the sentence format in order to prevent copying biases brought on by sentences. After completing each conversation, participants are instructed to specify whether they finish the conversation until the end (whether the goal is achieved) and whether they have encountered any significant inconveniences.
If participants mention that they cannot properly complete the conversation or they experience any inconvenience, we prompt a follow-up checkbox field to ask the categories of inconveniences: (1) Repetitions for repeatedly responding with the same text, (2) Unrelated for irrelevant responses to what users request, (3) Not-aligned for responses contradicting with previous context, and (4) Awkward for grammatically wrong or unfluent responses. We also add a free-form answer field where participants can describe the situations that do not fall within the above four categories. For each study session, we invite one or two participants to a Zoom video call, where a moderator briefs the study and instructs participants to complete five conversations with the TOD model. During the group session, participants are not allowed to disclose anything related to their conversations with a chatbot. The moderator supports participants only when they encounter technical issues. When the chatbot provides wrong responses, participants are guided to repeat their original intent up to two times, as we expect the TOD model to recover from its mistakes. Moreover, participants can continue the conversation with their own arbitrary goals if the chatbot cannot provide services related to the given goals because it is possible for the chatbot to fail to search entities satisfying all requests from users (even in closed-goal settings).

heading: 4 Analysis

heading: 4.1 Dataset and Descriptive Statistics
text: We collect 49 open-goal and 50 closed-goal dialogues from 20 participants; due to technical issues, one open-goal participant missed a conversation. The open-goal dialogues consisted of an average 10.53 turns (an adjacent pair of the user and system messages; SD 3 = 4.33), whereas the closed-goal
3Standard deviation.
dialogues had 8.92 turns on average (SD = 3.62).

heading: 4.2 Dialogue Stability
text: Figure 2 shows the proportion of forced termination during our experiment. We find that only 8% (4 out of 49) of the total open-goal dialogues have finished without any inconveniences, while almost half of the closed-goal dialogues (24 out of 50) show normal termination without any inconveniences. Meanwhile, it is important to note that more than half of the dialogues in both goal types had problematic situations for participants. Statistical tests (and Figure 2) reveal that opengoal settings result in significantly more erroneous dialogues. We describe the analysis method below, but, in short, we find interactive conversations in the wild to have a clear difference from static benchmark evaluations for both goal settings and especially for the open-goal setting.
Termination Pattern To assess the difference in termination patterns between the two goal types, we use mixed-effect models. These multi-level linear regression models can model the effect of the independent variables (i.e., fixed effect) while controlling the random variance among subjects (i.e., random effect) where multiple data points came from the same subject (Pinheiro and Bates, 2000). Treating each dialogue as a data point, we fit a
mixed-effect model to the termination type mapped to a numeric scale (0: normal termination, 1: normal termination with inconvenience, and 2: abnormal termination) in increasing order of severity. We put participants as a random effect and the goal type as a fixed effect to see whether the average severity levels of each group are different.
The maximum-likelihood test reveals that there is a significant random effect of participants (p < .0001, t(17.98) = 5.06), and a significant fixed effect of goal type (p = .002, t(18.14) = 3.71). The estimated mean of the severity scale is 0.68 for closed-goal (SE = 0.13) and 1.39 for open-goal (SE = 0.14) with 95% confidence. This indicates that the severity levels of termination of the two groups are significantly different, and the open-goal dialogues tend to fall in either normal termination with inconvenience or abnormal termination.
Learning Effect We also assess whether there is a learning effect on carrying on a successful conversation as participants go through more conversations. We use a mixed-effect model similar to the above analysis but put the order of dialogue per participant (i.e., 0 to 4) as a fixed effect, instead of the goal type. The maximum likelihood test reveals that the order does not have a significant impact on the model (p = .85). This implies that participants do not seem to “learn” the system’s conversation and request capabilities within 5 times of trials.

heading: 4.3 Erroneous Turn Analysis
text: In this section, we thoroughly analyze erroneous dialogue turns that occurred in both goal types by categorizing dominant error types.
We qualitatively annotate the erroneous turns. First, all authors of this paper separately annotate a subset (10 dialogues; ∼10%) of the user study data with candidates for the error category. Then they
iterate several discussions to generate a codebook of the errors. After the codebook is defined and agreed upon, the main author revisits the entire dataset to annotate erroneous turns. As a result, we identify six types of errors that commonly appeared in the conversations (Table 2). We further describe each error category and corresponding example in the following paragraphs. We denote as a user and as a system in the examples.
Irrelevant If the system response is irrelevant to the dialogue context, especially to the previous user utterance, we annotate such responses as Irrelevant. It is the most frequent error type in the user study. Moreover, the error proportion from open-goal dialogues is significantly higher, 8.8% points, than that from closed-goal dialogues. The following quote is an illustrative example by O1:
: Please let me know the cancellation policies of Alexander.
: Yes, I will work on getting this booked for you.
Self-Contradiction Self-Contradiction represents cases that the system response is contradictory with the previous system’s responses. This error is severe because it makes the users confused and also causes doubt over the systems in terms of proceeding given task, as illustrated in the below quote by C9. We find that open-goal dialogues contain 1.6% points more self-contradiction than closed-goal dialogues.
: how about 3 stars?
: Bridge guest house is available, would you like to try that?
· · ·
: I want 3 stars guesthouse which has a parking lot.
: I don’t have anything 3 stars. would you like to try a different area?
· · · : Thank you. Can you repeat my reservation?
Repetition Repetition refers to cases that the system repeats unnecessary responses across multiple turns. This error is also one of the main obstacles to achieving task success as it often annoys the users to quit the conversation. The following quote shows a severe repetition error that happened during our user study (by O5):
: Yes. Can you make a reservation for 11:45?
: · · · Can you tell me the day for the reservation, the number of people in
your party, and the number of people?
: Tomorrow. 2 people.
: I have your table booked for 2 people on Tuesday at 11:45, · · · What day would you like to dine?
: You already said Tuesday! Thank you for your help.
Poor Fluency When the system’s response is awkward from a given dialogue context or itself, we judge the response as a Poor Fluency. Also, grammatical error belongs to this type. Closed-goal dialogues include about 1% more Poor Fluency errors than open-goal dialogues. The following example represents this type of error (It is from a conversation with C8).
: Hmm, that’s not that I want. Can you cancel the current reservation and make one for an African restaurant?
: No, it does not. I can try a new restaurant if you would like.
Pretending Pretending represents system behaviors pretending to handle user’s requests beyond the system’s scope or answer user’s nonverifiable questions. For instance, in the below example by O4, the system provides false information to the users because the presence of a petfriendliness is not verifiable information for the system. Pretending is the most prominent error type of the open-goal setup; meanwhile, it occurs about 10 times more than in the closed-goal setting.
Closed-goal
Number of dialogues
Repetitions
Unrelated
Not-aligned
Awkward
Others
Open-goal
Figure 3: Inconvenience categories collected from participants by two goal types. It can be seen as an implicit or explicit reason to terminate a conversation from the perspective of the user.
This indicates that this Pretending error might have been overlooked in previous works that focused on static closed-goal evaluations. We further describe and discuss more details of this pretending error type in Section 4.5.
: Is Wagamama pet-friendly?
: Yes it is. Would you like to book it for you?
: Yes. Today at 7pm please.
Miscellaneous The Miscellaneous category, which is less frequent, usually includes arbitrary system behaviors like suddenly making a booking request and ignoring the user’s intent. The following example by C10 describes such a situation.
: Then, is the Indian Cuisine you mentioned earlier available for reservation on Wednesday at 15:45?
: Yes it is. I have booked it for you and your reference number is 64o8zpsl.
: Suddenly? The person I’m going with just now asks if we can change it to the same time as Thursday?
: Ok. I will try to book that for you.

heading: 4.4 Participant Feedback
text: We collect negative feedback from participants at a dialogue-level if there are any inconveniences during the conversation. Figure 3 shows the inconvenience categorization results. As expected, open-goal dialogues show a significantly higher proportion in most inconvenience categories than closed-goal dialogues because the model must have difficulty in handling novel requests in the opengoals. In addition to the inconvenience categories
we specify, we also collect feedback from users in a text form (Others category in Figure 3), and the full table after removing duplicated feedback will be made public in the Appendix. Interestingly, we find that most inconveniences align with our error categories of Section 4.3 well except for the Pretending error type. In other words, the pretending behaviors are not identified as inconveniences from the users’ perspective.
4.5 Why does Pretending Matter? In this subsection, we take a deeper look at the “pretending” behaviors of the TOD systems. The pretending behavior is similar to the hallucination problem (Ji et al., 2022) prevalent in generative models in that it provides false information to the users. However, one distinct feature of the pretending behavior is that it is hard to be recognized as an error at the time of occurrence by only its surface form because most TOD systems rely on variable service-specific knowledge that users cannot easily access while using the service. It also differs from the knowledge base inconsistency (KBI) in Qin et al. (2021). While the KBI only regards the wrong responses based on “verifiable” knowledge, Pretending indicates responses over “non-verifiable” knowledge beyond the system’s scope.
It is a severe problem for both agents and users since it interrupts accurate decision-making to achieve users’ goals. For example, other error types such as Irrelevant, Self-Contradiction, and Repitition can be easily recognized as superficial problems by the users. In those cases, the users can avoid unwanted conversation flow by complaining and terminating the conversations. However, when the TOD system naturally responds to users’ exceptional requests and does not take corresponding action behind, users have no way to perceive the fact that the ongoing conversation is wrong from their initial requests. For this reason, the Pretending is not exposed in any user evaluation shown in Figure 2, Figure 3, or Appendix Table 6. In other words, even users who normally terminated dialogue without any inconvenience (i.e., green bar in Figure 2) can suffer from the pretended dialogues.

heading: 4.6 Can Large Language Models Solve TOD Problems?
text: Although previous studies demonstrate the imperfectness of large language models in TOD systems (Jakobovits et al., 2022; Bang et al., 2023),
we conduct case studies to verify large language models’ capacities to resolve aforementioned conversational errors. Since it is impossible to equalize every experimental setup between fine-tuned language models and proprietary large language models4, we proxy intermediate modules of TOD systems with instruction prompts by modifying those of Chung et al. (2023). On top of that, we pinpoint erroneous turns in Section 4.3 and compare generated responses of each model. We describe detailed setup such as prompts and action definition in Section C.1.
Based on the result of our case studies, as illustrated in Table 5, we observe that large language models mitigate most conversational inconvenience, presumably due to their strong zero/fewshot capabilities on unseen domains. However, regarding Pretending, responses of both cases contain significant flaws. Specifically, according to Table 5, both conventional TOD model and large language model do not recognize the fact that petfriendliness does not belong to the service range and provide untrustworthy responses, which can lead to physical harm (e.g., wrong reservation) in real services. One of the expected causes is language models’ overconfidence of unseen scenarios, but we also find that predefined actions given to TOD models are confined to deal with diverse situations in a flexible manner5. Controlling overconfidence in language models (Miao et al., 2021; Mielke et al., 2022) can partially resolve conversational errors, but defining available actions mostly belongs to the range of system design, especially in service-specific scenarios. We further discuss the future direction in Section D.

heading: 5 Conclusion
text: In this work, we demonstrate user familiarity bias in current TOD benchmarks, which the recent TOD research community has overlooked. To effectively unveil the bias, we contrast two user groups with different user goals via an interactive user study. Against the closed-goal within the constrained scenarios, we introduce a control user group by assigning unconstrained scenarios to the participants, namely open-goal. Users in the two groups converse with the academically-discussed TOD chatbot following the given closed or open-goals. Our
4gpt-4-turbo in our case studies. 5Notably, there are roughly-defined actions in conventional
TOD scenarios, such as inform, request, recommend, etc.
study reveals the TOD system exposed to the user familiarity bias significantly fails to converse with the users with open-goals. We identify prevalent error types by analyzing the resulting conversations. Furthermore, we highlight the pretending behaviors of the TOD system with its characteristics and toxicity, which are not easily solved by simply utilizing large language models.

heading: 6 Limitations
text: Regarding the participants of our user study, all of them are internal employees of a giant tech company, the majority of whom are highly educated (60% hold a master’s or doctoral degree). However, they show various experiences with chatbots, not correlated with educational degree. Some of them are bilingual, while others are not native English speakers. Furthermore, since we assume a traveling situation, the conversational scenario was not challenging, even for non-native speakers.

heading: 7 Ethics Statement
text: In our user study, we collected demographic information such as name, age, gender, the highest level of education, occupation, native language, and experiences with the AI chatbot, after informing them that it would be used only for research purposes and acquiring their consent. We clearly introduced the purpose of our study and the usage of collected information before experiments, and all participants consented to our instructions.
Throughout the interaction with the chatbot, we instructed participants to play the role of potential users only, without disclosing any personally identifiable information about themselves. Collected dialogues were de-identified by giving anonymized user IDs. Throughout the annotating process, the authors examined all the gathered conversations, and no offensive content was found. Participants took part in the chat for roughly 30 minutes and were compensated with a 5,000 KRW (equivalent to 3.7 USD) gift card, which was somewhat higher than the Korean minimum wage at that time.

heading: A Participant Information
text: Participants are aged between 24 to 35 (Median = 29.5), and 12 are female. Three participants report that they are native English speakers. Eight participants have used AI chatbots for less than one month. Participants consist of 7 infrastructure engineers, 6 software engineers, 3 AI research scientists, 2 self-employed, a UX designer, and a data scientist. Detailed information can be found in Table 3.

heading: B Model Implementation Details
text: For the TOD system in our experiments, we use the public implementation of GALAXY6 (He et al., 2022). The model specification follows He et al. (2022); initialized with UniLM (Dong et al., 2019), which has a transformer-based architecture with 109M parameter size. We fine-tune this model on MultiWOZ 2.17 (Eric et al., 2020). We follow the default hyper-parameter settings provided by the authors. Training is completed within a few hours using 1 NVIDIA A100. PyTorch8 library is used for model training, and NLTK9 and spaCy10 are for text processing. We implemented a chat interface on Gradio11 (Abid et al., 2019) web platform. At inference time, greedy search is used for output prediction.
6https://github.com/siat-nlp/GALAXY. Apache license 2.0. 7https://github.com/budzianowski/multiwoz. MIT license. 8https://pytorch.org/ 9https://www.nltk.org/
10https://spacy.io/ 11https://gradio.app/

heading: C Case Studies on Large Language Model
text: C.1 Setup With a slight modification of Chung et al. (2023), we build prompts for action decision and response generation, demonstrated in Table 4, in order to simulate TOD systems utilizing large language models. We adopt predefined actions in GALAXY model as available actions. For end-to-end simulation, large language models generate responses by referring to conversation context and selected action (which is also generated).
C.2 Response Comparison We compare responses of GALAXY model with those of proprietary large language model on sampled dialogues, specifically mentioned in Section 4.3. Table 5 demonstrates each example with partial dialogue turns.

heading: D Discussion
text: As the importance of real user study or interactive evaluation gets bigger, we discuss the future direction in two folds: enhancing transparency to let users trust the chatbot, and managing fallback situations to detect users’ exceptional requests.
D.1 Enhancing Transparency Our participants often struggle to confirm their booking options as in Appendix Table 6. As a way to improve the reliability of TOD systems, we suggest enhancing transparency of the system, which has been actively discussed in the HCI community (Amershi et al., 2019). Transparency is a mechanism exposing hidden (i.e., non-obvious) information to users who have difficulty in getting the information directly (Rader et al., 2018). As our findings show that the lack of user familiarity provokes various inconveniences including the pretending problem, TOD systems in natural language processing field should also be designed to display intermediate by-products during the conversation in order to provide explainable rationales for their decisions (Amershi et al., 2019; Liao et al., 2020).
In the era of billion-scale large language models, the necessity of transparency is still valid. Although emerging works on grounded LLMs (e.g., Yao et al. (2023), ChatGPT with plugins (OpenAI, 2023) try to enhance trustworthiness using executable sources, they are still exposed to familiarity bias problem as long as they keep black-boxed service pipeline.
D.2 Managing Fallback Situation Users with low familiarity with the system inevitably make exceptional requests. As we can find in user comments in Appendix Table 6, a large number of users in an open-goal setup go through irrelevant and pretending responses from the bot. We emphasize the need to recognize exceptional requests and manage fallback situations towards robust TOD systems.
Out-of-Scope Detection In the field of intent classification, previous literature has studied detecting out-of-scope intents to prevent generating erroneous responses from the original intent (Larson et al., 2019; Zhang et al., 2022a,b; Cho et al., 2022). Moreover, Shrivastava et al. (2021) try to generate contextualized fallback responses to users’ exceptional requests. However, more datasets for fallback detection are required especially for multi-turn and multi-domain TOD scenarios beyond the single-turn detection scenarios.
Handling Request as Unstructured Form Kim et al. (2020) combine unstructured knowledge, FAQ pairs, with structured knowledge, DBs. The work includes (unstructured) knowledge-seeking turn detection to handle domain-specific requests with FAQs beyond the scope of structured knowledge. However, the work still assumes high user familiarity, i.e., it always contains relevant knowledge for a given request. We believe retrieval-augmented detection leveraging the FAQ pairs can be a promising approach to strengthen the approach towards a low user familiarity setup effectively (Thulke et al., 2021).
On the other hand, typical dialogue state tracking to access structured knowledge is not robust in terms of handling exceptional requests since it works based on predefined slots. Bae et al. (2022) adopts a text-formed dialogue state by summarising the dialogue context for effective memory management in multi-session open-domain dialogue. We believe that dialogue management based on unstructured information can have advantages not only in avoiding exceptional requests but also in leveraging advanced language understanding abilities of recent language models at a scale, as its generalizable text format.

heading: E Subjective User Feedback
text: F Ontology Used for Open-Goal Dialogues
1 { 2 "hotel": { 3 "Requests": [ 4 "Outdoor seating", 5 "Rooms with exceptional views", 6 "24-hour front desk", 7 "Breakfast options", 8 "Check -in/out policies", 9 "Cancellation policies",
10 "Cost around $150 per nights" 11 ], 12 "Facilities": [ 13 "Facilities: Gym", 14 "Facilities: Spa", 15 "Facilities: Swimming pool", 16 "Facilities: Outdoor terrace", 17 "Facilities: Non -smoking room" 18 ], 19 "Amenities": [ 20 "Amenities: Mini -fridge", 21 "Amenities: Tea and coffee facilities", 22 "Amenities: Private bath" 23 ] 24 }, 25 "restaurant": { 26 "Requests": [ 27 "Cost under $50 per person", 28 "Open until late", 29 "Italian wine serving", 30 "List of menus", 31 "Comfortable seating with enough space", 32 "Delivery or take -out service" 33 ], 34 "FoodOptions": [ 35 "Food option: Gluten -free", 36 "Food option: Vegetarian" 37 ] 38 }, 39 "common": { 40 "Requests": [ 41 "Wheelchair access", 42 "Weather forecast", 43 "Website link", 44 "Good customer review", 45 "Located in safe environment", 46 "Accept payment via cards", 47 "Easy access to public transportation", 48 "Near the tourist attraction" 49 ], 50 "Friendly": [ 51 "Pet -friendly", "Kid -friendly", "Family -friendly" 52 ] 53 } 54 }
Listing 1: A postprocessed ontology for exceptional requests generated from GPT-3. We collect raw task descriptions and categorize common scenarios which are not seen during the model training.

heading: G Demo Interface for User Study

heading: Future Work
text: As the importance of real user study or interactive evaluation gets bigger, we discuss the future direction in two folds: enhancing transparency to let users trust the chatbot, and managing fallback situations to detect users’ exceptional requests. D. 1 Enhancing Transparency Our participants often struggle to confirm their booking options as in Appendix Table 6. As a way to improve the reliability of TOD systems, we suggest enhancing transparency of the system, which has been actively discussed in the HCI community (Amershi et al. , 2019). Transparency is a mechanism exposing hidden (i. e. , non-obvious) information to users who have difficulty in getting the information directly (Rader et al. , 2018). As our findings show that the lack of user familiarity provokes various inconveniences including the pretending problem, TOD systems in natural language processing field should also be designed to display intermediate by-products during the conversation in order to provide explainable rationales for their decisions (Amershi et al. , 2019; Liao et al. , 2020). In the era of billion-scale large language models, the necessity of transparency is still valid. Although emerging works on grounded LLMs (e. g. , Yao et al. (2023), ChatGPT with plugins (OpenAI, 2023) try to enhance trustworthiness using executable sources, they are still exposed to familiarity bias problem as long as they keep black-boxed service pipeline. D. 2 Managing Fallback Situation Users with low familiarity with the system inevitably make exceptional requests. As we can find in user comments in Appendix Table 6, a large number of users in an open-goal setup go through irrelevant and pretending responses from the bot. We emphasize the need to recognize exceptional requests and manage fallback situations towards robust TOD systems. Out-of-Scope Detection In the field of intent classification, previous literature has studied detecting out-of-scope intents to prevent generating erroneous responses from the original intent (Larson et al. , 2019; Zhang et al. , 2022a,b; Cho et al. , 2022). Moreover, Shrivastava et al. (2021) try to generate contextualized fallback responses to users’ exceptional requests. However, more datasets for fallback detection are required especially for multi-turn and multi-domain TOD scenarios beyond the single-turn detection scenarios. Handling Request as Unstructured Form Kim et al. (2020) combine unstructured knowledge, FAQ pairs, with structured knowledge, DBs. The work includes (unstructured) knowledge-seeking turn detection to handle domain-specific requests with FAQs beyond the scope of structured knowledge. However, the work still assumes high user familiarity, i. e. , it always contains relevant knowledge for a given request. We believe retrieval-augmented detection leveraging the FAQ pairs can be a promising approach to strengthen the approach towards a low user familiarity setup effectively (Thulke et al. , 2021). On the other hand, typical dialogue state tracking to access structured knowledge is not robust in terms of handling exceptional requests since it works based on predefined slots. Bae et al. (2022) adopts a text-formed dialogue state by summarising the dialogue context for effective memory management in multi-session open-domain dialogue. We believe that dialogue management based on unstructured information can have advantages not only in avoiding exceptional requests but also in leveraging advanced language understanding abilities of recent language models at a scale, as its generalizable text format