Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users
Authors: Yohan Jo
Xinyan Zhao
Arijit Biswas
Nikoletta Basiou
Vincent Auvray
Nikolaos Malandrakis
Angeliki Metallinou
Alexandros Potamianos
Venue: EMNLP 2023 Findings (2023)
Keywords: Multiparty Dialogues
Contextual Query Rewriting
Dialogue Summarization
Dialogue Systems

Abstract: While most task-oriented dialogues assume conversations between the agent and one user at a time, dialogue systems are increasingly expected to communicate with multiple users simultaneously who make decisions collaboratively. To facilitate development of such systems, we release the Multi-User MultiWOZ dataset: task-oriented dialogues among two users and one agent. To collect this dataset, each user utterance from MultiWOZ 2.2 was replaced with a small chat between two users that is semantically and pragmatically consistent with the original user utterance, thus resulting in the same dialogue state and system response. These dialogues reflect interesting dynamics of collaborative decision-making in task-oriented scenarios, e.g., social chatter and deliberation. Supported by this data, we propose the novel task of multi-user contextual query rewriting: to rewrite a task-oriented chat between two users as a concise task-oriented query that retains only task-relevant information and that is directly consumable by the dialogue system. We demonstrate that in multi-user dialogues, using predicted rewrites substantially improves dialogue state tracking without modifying existing dialogue systems that are trained for single-user dialogues. Further, this method surpasses training a medium-sized model directly on multi-user dialogues and generalizes to unseen domains.

Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users
Yohan Jo
Xinyan Zhao?
Arijit Biswas
Nikoletta Basiou²
Vincent Auvray²
Nikolaos Malandrakis
Angeliki Metallinou
Alexandros Potamianos*
'Sooul National
University, Korea
Amazon, USA
yohan. joesnu. kc.
(xinyazha, barijit, nbasiou, vauvray, malandrn, mmmt ott.r//w..oo.........ooooooooooo
Abstract
User1: Shall we take a bus from Saint John's college to
Pizza Hut Fen Ditton?
While most task-oriented dialogues assume conversations between the agent and one user time, dialogue systems are increasingly expected to communicate with multiple users simultancously who make decisions collabora- tively. To facilitate development of such sys- tems, we release the Multi-User MultiWOZ dataset: task-oriented dialogues among two users and agent. To collect this dataset, each user utterance from MultiWOZ 2.2 was re- placed with small chat between two users that is semantically and pragmatically consistent with the original user utterance, thus resulting in the same dialogue state and system response. These dialogues reflect interesting dynamics of collaborative decision moccsion-mmaing in task orieenedd scenarios, e-g., social chatter and deliberation. Supported by this data, we propose the novel task of multi-user contextual query rewrit- ing: to rewrite task-oriented chat between two users as concise lask-oriented query that re- tains only task-relevant information and that is directly consumable by the dialogue sys- tem. We demonstrate that in multi-user dia- logues, using predicted rewrites substantially improves dialogue state tracking without modi- fying existing dialogue systems that are trained for single-user dialogues. Further, this method surpasses training a medium-sized model di- rectly on multi-user dialogues and generalizes unseen domains.'
User2: too many stops. take
User1: Can you book
us?
Rewrite (Original User Utterance]: would like taxi from
Saint John's 's college to Pizza Hut Fen Ditton.
System:: What time do you want to leave and what time do you want arrive
User1: Okay, We'll need table Saturday. What time are we thinking? User2: Let's do 19:45, Steve can't get there till after 19:00.
Usert: Right. will be 3 people at 19:45.
User2: Don't forget the reference number, need
reference number.
Rewrite (Original User Utterance]: Okay can you book me a table for Saturday at 19:45 for 3 people?
would also like the reference number for the booking.
System:
have successfully booked your reservation. Your reference number 7XYDZ38V Can I help you with anything else?
Intent Informed Slot Value Requested Slot Name Slot Elicitation Social Chatter Deliberation
Figure I: Excerpts of dialogues in our dataset. For each example, the sequence of user utterances is called multi- chat. Rewrites refer original user utterances in MultiWOZ 2.2 that were expanded to multi-user chats between Userl and User2. They can be used as ground- truth rewrites for contextual query rewriting. More ex- amples are Appendix A.I.
all and only necessary information for conducting task. By contrast, multi-usee multi-user task-oriented dia- logues are significantly richer, containing delibera- tion and social chatter, and poses additional chal- lenges like separating out task-relevant information from social and sensitive information related to user privacy. The main bottleneck for first step into supporting multi-user task-oriented dialogues is lack of proper datasets, as most (if all) existing datasets for task-oriented dialogues are single-user. To overcome this limitation and facilitate future re- search, we build and release a dataset multi-user task-oriented dialogues and propose the novel task
1 Introduction
Voice assistants like Amazon Alexa and Google Assistant are widespread, and users often interact with them in multiparty settings, such as playing games and making decisions with family members (Porcheron et al., 2018). However, most dialogue systems are designed support only single-user di- alogues, i.e., the agent expects to converse with one user a time via succinet command that contains
'The dataset is available at httpps//ggithh:: thub. com/ wohanjo/multiuser. www.tserrrumtit.....mm..

---

of multi-user contextual query rewriting.
2 Related Work
Our dataset Multi-User MultiWOZ is an ex- tension of MultiWOZ 2.2 (Zang et al., 2020) to
Our work is closely related to three fields of NLP: task-oriented dialogues, contextual query rewriting. and dialogue summarization.
multi-user dialogues (Figure 1. $3). MultiWOZ 2.2 is one of the largest and most popular single-user lask-oriented dialogues. The guiding principle of our data collection is to extend each user utterance in MultiWOZ 2.2 to a chat between two users mak- ing decisions together (multi-user chat henceforth) that leads to the same dialogue state the source utterance. This allows to (1) re-use system acts and responses annotated in the source single-user dialogue and (2) train a query rewriting model that converts a multi-user chat to its source utterance (as the ground-truv truth rewrite) so that output rewrites can be consumed by dialogue systems that expect single-user utterances, Compared to existing re- lated datasets, which are either single-user (An- dreas et al., 2020; Rastogi et al., 2020; Young et al., 2021) or lack dialogue state annotations (Li et al., 2017), our dialogues reflect interesting dynamics of collaborative peeision-maaing in task-oriented con- versations, such as questions to elicit slot values. social chatter, and deliberation.
Dialogues: Existing datasets of task-oriented dialogues (Zang et al., 2020; Andreas et al., 2020; Byrne et al., 2019; Rastogi et al., 2020; Zhu et al., 2020; Young et al., 2021) are, to our knowledge, all single-user conversations. By con- trast, our dataset consists of task-oriented dialogues where two users are making decisions together with the help of an agent. Since input to dialogue system is a chat between users rather than sin- gle user utterance, such dialogues pose challenges to dialogue systems trained via traditional algo- rithms for dialogue state tracking (Hosseini-Asl et al., 2020; Le et al., 2020a,b). While multiparty dialogue datasets exist (Li et al., 2017; Gopalakr- ishnan al., 2019), they are not task-oriented and thus missing important information for training task-oriented dialogue systems. Our multi-user dataset is task-oriented by nature and annotated with relevant information, such as dialogue states and system acts and responses.
Empowered by this dataset, we propose the novel task multi-user contextual query rewriting: to rewrite a multi-user chat as a single request that is concise and contains all and only task-relevant information ($5). This task is important because (I) it can bridge the gap between multi-user chats and dialogue system trained for single-user dia- logues without replacing the entire dialogue sys- tem, and (2) alleviates users' privacy concerns by processing multi-user chats on device and sending the rewrites to the system server with only task- relevant information. We demonstrate the accuracy of baseline models on the rewriting task and dis- cuss the main challenges. Further, we verify that model-pree prediccee rewrites are helpful for dialogue state tracking for dialogue systems that trained on single-user dialogues, by substantially outper- forming baseline that simply concatenates utter- ances in a multi-user chat and treats it as a "single" utterance. This task also benefits unseen domains. Our contributions are twofold:
Contextual Query Rewriting: Contextual query rewriting refers to decontextualizing user query to self-contained one that contains necessary con- textual information for the agent to serve the user's request (Zamani et al., 2022). This mostly involves anaphora resolution and ellipsis resolution, and sequence-to-scqueece models (e.g., GPT) have per- formed well (Vakulenko et al., 2021; Yu et al., 2020). Our work is related in that the source ut- terance of each mutli-user chat can be seen as a decontextualized rewrite of the chat that the agent can process without parsing the whole chat. While existing work and datasets for contextual query rewriting focus on converting one user request a time (Yuan al., 2022; Dalton et al., 2009; Choi et al., 2018), ()18), our task rewrites a chat between two users (hence, multi-user contextual query rewrit- ing). This task is more challenging as it often goes beyond anaphora/ellipsis resolution; for instance, it also involves resolving deliberation between users, and user intents and slots may be spread out across multiple utterances in a chat. Our dataset provides training and evaluation resources for this task.
We release the Multi-User MultiWOZ dataset, task-oriented dialogues between two users and one agent, under the MIT license.
We propose the multi-user contextual query rewriting task that rewrites multi-user chats as concise task-oriented requests.
Dialogue Summarization: Our dataset is also re- lated to dialogue summarization in that each multi-

---

user chat can be seen as a short task-oriented dia- logue between two users and its rewrite sum- mary of the dialogue. Existing datasets for dialogue summarization are not well-suited for summarizing multi-user task-oriented dialogues. One of the main reasons is that they have different goals for summa- rization. For instance, many datasets (Gliwa et al.. 2019; Krishna et al., 2021; Zhang et al., 2021; Song al., 2020; Zhong al., 2021; Chen et al., 2021; Zhu et al., 2021) aim to summarize diverse perspec- tives of speakers rather than succinct taik-triented information (Fabbri et al., 2021; Lin et al., 2022). Our dataset focuses on summaries where delibera- tions are resolved and only task-relevant informa- tion (e.g., user intents and slots) is retained. While some datasets aim to summarize task-oriented dia- logues from customer services (Zhao et al., 2021; Liu et al., 2019; Feigenblat al., 2021; Lin al.. 2021), they are not designed summarize chats be- tween users; rather, they summarize a chat between user and agent, which has different dynamics than collaborative decision-making between users.
3.1 Pilot Study
We first ran a pilot study to learn the characteristics of collaborative decision-making in task-oriented dialogues and make sure they are reflected in our generated chats. Specifically, we recruited Alexa users and asked them to conduct two tasks in pairs using two Alexa skills. The first is GiftFinder where the users navigate gifts to buy, and the sec- ond is NewsFinder where the users find news arti- cles of common interest. The pilot data revealed four notable characteristics of multi-user chats. Users (1) ask questions to each other elicit slot values (e.g., "What time are we thinking?"", (2) have social chatter aside from expressing intents and slot values (e.g., "How are you going to fix my empty stomach?"). (3) have deliberation over multiple options (e.g., "No, too many stops. Let's take taxi."), and (4) exploit common ground, e.g.. mention the names of cach other and friends (c.g.. "Steve can't get there till after 19:00.").
3.2 Data Collection Protocol
3 Data Collection
To collect multi-user dialogues at scale, we use Amazon Mechanical Turk (MTurk). Each task con- sists of the first four user utterances of dialogue from MultiWOZ (Figure 6-8 in Appendix B.2). Each utterance is accompanied by the system re- sponse. The number of utterances in the generated multi-user chat is predefined between and 4, be- ing skewed toward the number of informed or re- quested slots (Appendix B.3). We asked one turker to expand all user utterances in each task. Com- pared to having two turkers do that together, this is faster and still results in high quality, will be discussed in our dialogue quality assessment ($4). Tasking one worker to generate the entire dialogue has been shown in prior research be a simple ap- proach that leads to better dialogue quality (Young et al., 2021; Byrne et al.. 2019; Nakamura et al.. 2022).
In building dataset of multi-user task-oriented dialogues, our principle is to extend an existing dataset of single-user task-oriented dialogues, such that each user utterance in that dataset is expanded to a chat between two users (multi-user chat) that leads to the same dialogue state. Two main bene- fits to this approach are: (1) we can reuse system acts and responses in the original dataset without annotating them anew, and (2) pairs of multi-user chat (in the collected data) and its source utterance (in the original data) can be used to train query rewriting model that rewrites multi-user chat as concise query (and vice versa).
To that end, we use MultiWOZ 2.2 (Zang et al., 2020) as our basis? It consists of task-oriented dialogues between one user and one agent, where the agent helps the user with booking and find- ing information for eight services (hotel, attraction, restaurant, train, taxi, bus, police, hospital). We expand each of the first four user utterances in each dialogue to an imaginary chat between two users making collaborative decisions.
To ensure that generated chat preserves the dialogue state of the original (source) utterance, we constrained that all informed slot values and requested slots mentioned in the generated chat (see Appendix B.4). Note that this makes the gen- erated chats compatible with the system acts and responses in MultiWOZ.
We describe our pilot study, data collection pro- tocol, validation process, and data statistics.
For generated chats to reflect the characteristics revealed in the pilot study, our instructions included example dialogues that contain social chatter and deliberations. Turkers naturally elicited slot values
MultiWOZ 2.2 is distributed under the MIT license. "Wo focus only on four user utterances per dialogue be- cause of limited budget. This decision does not impair the representation of services, intents, or slots (Appendix A.2).

---

and mentioned names.
score for each criterion as follows. The questions have the following option-score mapping: Yes=l, No=0 / Very-much=2, Acceptable=1, Definitely- Not=0. Overall, the final scores are satisfactory: (QI) 0.99/1 (Q2-3) 1.8/2, (Q4-5) 0.98/1. Espe- cially the scores of Q4-5 suggest that dialogue states are preserved between multi-user chats and their source utterances.
For the dev and test sets in MultWWOZ, we cov- ered all dialogues (1,000 each). For the training set, we sampled 2,400 dialogues while including all di- alogues about the bus, hospital, and police services (since these services are highly underrepresented). Turkers did not overlap between the training set and the test set.
Table shows some example dialogues. More details about the data collection task are available in Appendix B.
The five validation criteria we used are impor- tant quality metrics for multi-user task-oriented dia- logues. We see an opportunity to automate some of them; for example, to automatically check whether dialogue participants have relationship of two cus- tomers, we could train classifier that distinguishes among different types of participant relationships using existing dialogue datasets. We leave this future work.
3.3 Validation
We validated generated multi-user chats via sepa- rate MTurk task. Each task contains one dialogue, i.c., at most four multi-user chats. Each chat is ac- companied by the previous system response (except for the first turn) and the source utterance labeled as "summary" (Figure 13-15 in Appendix C.2).
3.4 Data Statistics
For each pair of generated chat and summary, we asked the following questions.
Table 2 shows some statistics of our data. Com- pared to the single-user counterparts in MultiWOZ, our dataset contains 2.7x turns, 1.7x tokens, and 1.3x negations. Based random sample of 300 multi-user chats (100 for each split), we counted the occurrences of three important types of so- cial dynamics that we found to be characteristic of multi-user task-oriented dialogues:
1. Is the users' relationship two customers mak- ing decisions together? Yes or no?
2. Is the chat realistic chat between humans? Very much, acceptable, or definitely not?
3. Is the last utterance in the chat a realistic ut- terance toward the agent? Very much, accept- able, or definitely not?
Slot elicitation: Users ask each other for infor- mation or preferences related intents and slots (e.g. "Let me think, there are how many our group?").
4. Does the summary have any missing informa- tion that is present in the chat? Yes or no? If yes, what information missing?
5. Does the summary contain any information that is not present in the chat? Yes or no? yes, what additional information is present? We curated 13 qualified validators through qualification task that ensured the validators could apply the five criteria correctly (Figure 9-12).
Social chatter: Utterances for social conversa- tion, such as jokes, feelings, and more (e.g.. "I'm not sure, but I've heard good things."). 'Social chatter' is a broad concept that could be further broken down into intents dia- logue acts specific to multiparty dialogues (e.g., suggesting why a certain slot value is preferable in "too many stops" Figure 1). Such a breakdown is beyond the scope of this paper.
Next, every chat-summary pair in our collection was validated by two qualified validators, where it is considered "flagged" if validator chooses "no" "definitely not" for Q1-Q3 "yes" for Q4 Q5. Based on the results, each chat-summary pair is categorized into: poor if flagged by both validators for at least one question, good not flagged by any validators for any questions, and moderate otherwise. Further, a dialogue is categorized as poor if any of its chats is poor, good if all its chats are good, and moderate otherwise. Poor dialogues are discarded from the final dataset. More details about the tasks are available in Appendix C. Based on the validation results, we computed the
Deliberation: Users make a decision over mul- tiple options (e.g., Userl: "Either 4-5 not sure." User2: "Better make it five to be safe.").
More examples are available in (Appendix A.3 and Figure 1). We found that slot elicitation appears in 24%, social chatter in 23%, and deliberation in 2% of multi-user chats.
Our dataset has reasonable size, containing 16,706 multi-user chats, which slightly larger

---

Dialogue
Dialogue
User2:
Should be try the Alpha-Milton Guest House?
User1: We need your help in finding local hospital. Can you find one with cardiology department? Is there local hospital that has cardiology de- partment??
User1:
What do we know about it?
User2:
User2:
Not much. Can you give some info about the
RW:
place?
RW:
Can you please tell me about the Alpha-Milton Guest House?
Sys.: Addenbrookes Hospital located Hills Rd, Cam- bridge has cardiology. you need their phone number?
Sys.:
Absolutely! They are wonderful guesthouse the northern part town. They moderately priced and rated stars.
Userl: Let's waste time.
User2: I'm not. But it's important to get the relevant formation.
User2:
We do not want to pay for parking.
Userl:
Does have free parking?
User1: Sorry, didn't mean that, But we need to hurry.
RW:
Do they have free parking?
User2: know. Can you just give us their address and postcode?
Sys.:
No they unfortunately do not have parking. there anything else can help you with or any other information you would like know about that hotel or area?
RW: just need their address and postoode please
Sys.: Their address is Hills Rd, Cambridge, postoode
CB20QQ
Userl:
No, that is fine. Should we take a train or bus to
Userl: Thanks a lot.
cambridge?
User2: You've 'v been very helpful.
User2:
Train would better.
User1: We gotta go now.
Userl:
Alright. Please get us a train for Tuesday. No, thats ok. I do need a train to Cambridge on Tuesday though.
User2: Goodbye. Have a great day.
RW:
RW: Thank you and goodbye.
Sys.: You're re welcome. am glad could assist you at
Sys.:
Where are you departing from? And what time would you like to arrive?
TownInfo centre. Goodbye.
User2:
We should leave from london kings cross in the afternoon.
Userl:
Yes, we need to get there by 3pm.
User2:
Alright. Please book a train that arrives by 15:30. I need to arrive by 15:30 and am leaving london
RW:
kings cross.
Sys.:
TR3456 leaves London Kings Cross on Tuesday 13:17 and arrives in Cambridge at 14:08. The cost is 23.60 pounds. Will this meet your needs?
Table I: Example dialogues. (RW: rewrite (i.e., source utterance), Sys.: system response)
than a popular dialogue summarization dataset SAMSum (16,369) (Gliwa et al., 2019) and a par- allel corpus for contextual query rewriting TREC CAsT (173) (Dalton et al., 2009).
Dialogue Quality Assessment
We verify that the collected dialogues have high quality and are realistic to happen in the real world. Adapting the dialogue quality assessment in Chen et al. (2023) to multi-user dialogues, we evaluated each dialogue in six aspects:
Note that each multi-user chat has the la- bels of informed slot-value pairs, requested slots, and intents, because the original dialogue state is preserved through text matching of in- formed/requested slots ($3.2) and semantic vali- dation to contain no missing or extra information ($3.3). This allows us to identify the text spans of slots and values via text match.
1. Realistic: How likely would the user chats occur in real-world interactions with agent?
(completely unlikely to occur) to 5 (highly likely to occur)
Scale:
2. Natural: How fluent are the user chats? Scale:
(completely influent) to (as fluent as native English speakers)
Multi-user chats in our dataset contain 2 to utterances, which does not handle cases where a user speaks to the agent without discussing with the other user. Our data collection did not cover such cases because user utterances in the original MultiWOZ are already reflecting such scenarios. Hence, we recommend training dialogue system on a mix of our dataset and the original MultiWOZ. so that the system can process both single user utterances and multi-user chats reliably.
3. Coherent: How coherent is the overall flow of the dialogue? Scale: I (completely incoher- ent) to 5 (as coherent as reasonably intelligent and attentive speakers)
4. Interesting: How interesting are the user chats? Scale: (generic and dull) to 5 (full of content and very engaging)
Consistent: How consistent is each user?
Scale: 1 (always says something that abruptly

---

Train
Test
MuldWOZ
MultUSeerOOZ
MultiWOZ
MultiUserWOZ
MultWOZ
MultiUserWOZZ
of dialogues (good/moderate) multi-user chats of Turns/Dialogue of Tokens/Dialogue of Tokens/Turn of Stopwords
1,589/695
793/202
731/263
8,859
3,936
3,911
4
11 (2.8x)
11 (2.7x)
4
11 (2.7x)
55
104 (1.9x)
56
103 (1.8x)
56
92 (1.7x)
14
10 (0.7x)
14
10 (0.7x)
14
9 (0.6x)
63,128
119,954 (1.9x)
28,117
52,833 (1.9x)
27,869
47,213 (1.7x)
of Entities
6,614
8,771 (1.3x)
3,089
4,237 (1.4x)
3,133
3,716 (1.2x)
of Negations
260
317 (1.2x)
102
194 (1.9x)
88
112 (1.3x)
Table Data statistics,
"MultiWOZ" columns shows the statistics of single-user counterparts MultiWOZ.
Real
Nat
Coh
Int
Cons
Rel
alogues in a pairwise setting is difficult. It can produce less realistic dialogues than our protocol,
Wult2woz MuldWWZ
4,47
4.67
4.48
3.79
4.46
4.59
4.68
4.83
4.68
3.72
4.69
4.71
mainly because laypeople are not good at creat- ing collaborative scenario and holding a relevant dialogue in interactive settings, unlike single-user settings like MultWWOZ and Wizard Of Wikipedia (Dinan et al., 2019).
Pilot
4.20
4.57
4.43
3.55
4.46
4.41
DailyDialog*
4.85
4.51
3.44
4.57
TopicalChat*
4.92
4.39
4.55
4.87
-
Table 3: Dialogue quality assessment. Multi refers to our dataset Multi-Usst MultWWZ. (Realistic, Natural, Coherent, Interesting, Consistent, Relevent) *The scores of these datasets are from Chen et al. (2023).
Regarding the other criteria, original MultiWOZ was rated slightly higher than our data (by 0.2 points) for Naturalness, Coherence, and Consis- tency with p-value 0.05. This is expected be- cause it is naturally difficult for multi-user dia- logues to achieve the same level of Coherence and Consistency single-user dialogues. Compared to the pilot data, our data showed no statistically significant difference for any criteria (except for Realism), suggesting our protocol produces similar quality to pairwise setting. Even compared to well-established multi-party dialogue datasets, our dialogues are scored higher than DailyDialog for Interestingness and TopicalChat for Coherence.
contradicts what they said carlier) to (never says something that abruptly contradicts what they said earlier)
6. Relevant: Are the user chats relevant to the given scenario? Scale: 1 (completely irrele- vant) (highly relevant)
We randomly sampled 90 dialogues from our data for assessment. For baselines, also evalu- ated two reference datasets for comparison: (1) the same 90 dialogues from original MuliWOZ 2.2 (single-user). In this case, the quality of user utter- ances (as opposed to user chats) is assessed; and (2) 62 pairwise dialogues from our pilot study ($3.1) that were generated by participants in pairs. Each dialogue was evaluated by three qualified turkers. Table 3 lists the quality scores of the datasets. The table also reports the scores of two well- established multiparty dialogue datasets: DialyDia- log (Li et al., 2017) and TopicalChat (Gopalakrish- et al., 2019), evaluated by Chen al. (2023). For Realism, our data and original MultiWOZ data showed no statistically significant difference MMnn--hittee U test), meaning evaluators judge our dialogues be as likely occur in reality as the well-established MultiWOZ. Our data was rated higher than the pilot data (p- (0.)a), sug- gesting that collecting multi-user task-oriented di-
Multi-User Contextual Query Rewriting
We propose the novel task of multi-user contextual query rewriting, which rewrites task-oriented chat between users as a concise query. Our main goal in creating this task for this query rewriting serve as the first module of a language understand- ing pipeline. The main advantage of such a module is that can be plugged into an existing dialogue system and convert a multi-user chat to a concise relf-contained utterance, enabling multi-user dia- logues with minimal modifications the dialogue system. Further, this module can filter task- irrelevant information (e.g., personal information and social chatter) on device before sending the rewrites to the system server, thus enhancing user privacy.

---

R-1
R-2
R-L
Inf
Req
Hal
Userl: Ok, and the postoode? You haven't told the postcode yet.
TS-Base
53.8
33.1
49.3
94.7
92.4
18.9
User2: Seems like the fun time secured. But how are you going to fix my empty stomach?
BART-Base
56.0
35.8
52.1
93.0
92.9
16.7
BART-Large
56.0
35.3
51.6
95.4
92.5
16.5
Userl: know you love indian food, and I feel like cating some, too.
GPT-2-Rase
42.2
23.1
38.5
75.6
67.8
28.5
User2: So check for an indian restaurant that is not far from our nightclub of choice.
Table 4: Query rewriting accuracy, (ROUGE, Informed slot values, Requested slot names, Hallucination Rate)
Ground- -Trnh Rewrite (Source Utterance): What is the postcode for that? am also looking for an indian restaurant near the nightclub, are there any?
5.1 Experiment Settings
T3: i'm looking for an indian restaurant that is not far from my nightclub of choice.
We explore four baseline models: T5-base (Raffel et al., 2019), GPT-2-base (Radford et al., 2019), BART-base and BART-large (Lewis et al., 2019), via the HuggingFace library (Wolf et al., 2020). We chose moderate-sizze models (the '-base' models) main baselines, considering limited hard- ware resources in popular smart speakers (e.g.. Amazon Echo Dot). We included BART-large to see if a larger model improves accuracy. Hyperpa- rameters are detailed in Appendix D.I.
BART: what is the postcode?
GPT2: can you help me find an indian restaurant located in a college or hotel room that i am in?
Table 5: Example outputs of the base models.
form by 2 points, the hallucination rate still remains 16.5% indicating the difficulty of this task.
Table 5 shows example outputs of the base mod- cls. The input chat has a lot of distracting social chatter. The models filtered out social chatter as desired, but the outputs omit important information too. This problem is pronounced for GPT-2, which suffers from hallucination. More example outputs are in Table 9.
The input is a concatenation of all utterances in multi-user chat, each of which is prefixed with "<user>", e.g., "cuser>Shall from Saint John's college to Pizza Hut Fen Dit- No, too many stops. Let's take a Cax you book a taxi for us?". We do not add dialogue history because was not helpful. The output is a rewrite of the chat; the source ut- terance of the chat is used as the ground truth. We use all dialogues labeled 'good' or 'moderate" for the train and dev data. Results are reported test dialogues labeled 'good'.
take a bus
We discuss two main types of errors that should be addressed in future work for improved models.
Error Type 1. Confusion over Deliberation: We observe that BART-base tends to make errors when there is deliberation or discussion on slot val- ues of the same type. Example below contains a deliberation about departure dates ("Wednesday" vs. "Thursday"), and the incorrect one is picked up by the model (compare True RW (true rewrite) and Pred RW (predicted rewrite)). Similarly, in Example 2, both "Asian" and "Chinese" refer to food types, and the model takes the incorrect one.
We report ROUGE-1, 2, and scores (Lin, pooular summarization accuracy metrics. Since the coverage of informed and requested slots is key to this task, we also report the recall of in- formed slot values and requested slot names. Lastly, we measure hallucination rates the inverse of en- tity precision between true and predicted rewrites, where entities are extracted using spaCy*.
Example
5.2 Query Rewriting Results
User1:
Can you help us get a train? We want to be leaving on Thursday. No, actually, we need to go on Wednesday, I'll tell you why later. Oh, okay then, Wednesday please. Can you help me find train? I'll be traveling on Wednesday.
Table 4 shows the query rewriting accuracy of the models. Overall, BART models perform best across most metrics, followed by T5. GPT-2 per- forms significantly worse as it tends to make up stories that are absent in the input chats; this results in low recall of informed and requested slots and high hallucination rates. BART-large are par with BART-base; while it has higher recall of In-
User2:
User1:
True RW:
Pred RW:
Can you help me get a train? want to be leaving on Thursday.
hhttps: //spac/spppcccc/e 1o/ soadelss/entee_crrre
Example

---

Training
Testing
Flatten
<user>i would like find French restaurant, please.<system. You ty cote in the Need reservationn it's moder- utely priced, yes please.
<user>Shall take bus from Saint John's 's's college Pizza Hut Fen Ditton? No, many stops. Let's take taxi. Can you honk taxi for us? <user>Book taxi from Saint John's col- lege to Pizza Hut Fon Ditton
Rewrite
<usersi would like find French restaurant, please.<syttem> You try cote in centre. Need reservation it's moder- ately priced, yes please.
Multi
Hello> We're thinking of finding some European food. Did you want something in particular?<userFFreem Frushr>French food, if you can centre. Need
from Saint John's 's college to Pizza Hut Dit- ton? PuuserNN, many stops. Let's take taxi. Cuserrann you book taxi
<user>Shall
find it, please/<systent You can try cote reservation Ohuh that just opened I think, hope not too expensive, <user> can ask, probably. you know moderately priced?
Table 6; Input formats for dialogue state tracking.
Intent
Inform
Request
Example
Userl:
definitely want certain range.
P
R
F1
P
F1
P
R
Fl
User2:
What do you think it should be?
Flatten
73.7 69.9
71.8
66.7
41.8
51.4
10,9
45.8
17.6
Rewrite
81.6
78.2
79.8
66.7
41.6
512
14.8
62.9
24.0
Userl:
Lets go for moderate.
Multi
69.0 64.7
66.8
67.7
40.2
50.4
7.5
31.3
12.1
True RW:
Yes definitely. would like some-
thing moderate.
(a) In-domain accuracy.
Pred RW:
definitely want a certain range.
Intent
Inform
Request
P
R
F1
Fl
P
R
F1
5.3 Dialogue State Tracking
Flatten Rewrite
53.1
53.1
53.1
18.8
11.7
13.3
62.3
62.3
62.3
25.0
19.0
20.6
Dialogue state tracking is key to understanding user intents and relevant slot values. Assuming that dialogue state tracking occurs after each multi-user chat, we ran another experiment to verify that model-predicted rewrites of multi-user chats can benefit dialogue state tracking when dialogue systems are trained for single-user dialogues only. simulate dialogue systems, we trained BART to take a user utterance and dialogue history as input and predict intents, informed slot values, and requested slot names separately (i.e., three separate models). The output is comma--eppartted values, e.g., "(train- d (Hosseini-Asil et al., 2020).
(b) Unseen domain accuracy.
Table 7: Dialogue state tracking accuracy.
User1:
doesnt matter me.
User2:
I think feel like some Asian food.
User1:
Can you find some Chinese food?
True RW:
I'd like some Chinese food, please!
Pred RW:
doesn't 't matter. would like some Asian food.
Error Type 2. Early Memory: We find that the model is less robust informed slot values in latter utterances. The average number of turns per multi- user chat 2.6. The average position of utterances, from which slot values are correctly recalled, is 1.99. However, the average utterance position for missed slot values is 2.6. This indicates that the model carries over slot values from earlier utter- ances better than from latter ones. Furthermore, when input contains dialogue history preceding the current turn, the average utterance positions of suc- cessfully recalled and missed slot values 5.45 and 6.72, respectively, out of 6.54 utterances av- erage. The following example illustrates this error. The last user utterance clearly informs "moderate", yet the model ignores it.
We explored three settings: Flatten and Rewrite simulate traditional dialogue systems and thus are trained on single-user dialogues using rewrites (not multi-user chats) in data. When testing on multi-user dialogues, however, Flatten takes flat- tened multi-user chat if is a "single" utter- ance from one user, whereas Rewrite takes BART- predicted rewrites (from the previous experiment) in place of the actual multi-user chats. Multi simu- lates special dialogue system that is both trained and tested multi-user dialogues. The input for- mat for each setting is shown in Table 6.
Table 7a shows the precision, recall, and F1-

---

score of intents, informed slot values, and requested slot names. For the systems trained on single-user dialogues (Flatten and Rewrite), feeding predicted rewrites as input (Rewrite) substantially outper- forms feeding a flattened multi-user chat (Flatten) in predicting intents (+8 points) and requests (+6.4 points); they perform similarly for inform predic- tion. This demonstrates the benefit of multi-user contextual query rewriting. While handling multi- user dialogues by summarizing them is an intuitive idea, no prior work has verified its effectiveness empirically and no public dialogue systems do this to our knowledge. Our work offers a dataset that enables this verification and verified its feasibility.
Conclusion
We release the Multi- MultiWOZ dataset, con- taining task-oriented dialogues between two users and one agent. The dialogues reflect important characteristics of collaborative decision-making in lrik-torented settings, such slot elicitation, social chatter, and deliberations. This dataset also enables the task of multi-user contextual query rewriting. which aims to rewrite a multi-user chat as a concise query that contains task relevantt information. We demonstrated that this task improves dialogue state tracking for a dialogue system trained for single- user dialogues both in-domain and cross-domain This result is promising because this task can easily be plugged into language understanding pipeline, processing multi-user dialogues with little modifi- cation to the dialogue system. Further, this task can be conducted on device, filtering out task-irrelevant information in multi-user chats and sending only task-relevant information to the system server for further processing, which enhances user privacy.
Interestingly, training a dialogue system directly on multi-user dialogues (Multi) underperforms Flat- ten by 5 points. This suggests that simple seq2seq model and medium-sized dataset are not enough to learn the dynamics of multi-user chats during training and that more research is needed to im-
prove modeling. We believe our dataset can pave the way toward this research direction.
Our work assumes that a dialogue system starts dialogue state tracking only after multi-user chat ends. This approach is common in practical dia- logue systems, where systems start processing user inputs only when the user signals their utterance is directed at the device (c.g.. by using a wake word). For multi-user task-oriented dialogues, an alternative approach is to track dialogue states after each utterance in multi-user chats; that is, dia- logue state is updated as soon a user finishes speaking to the other this approach is theoretically plausible and allows dialogue systems to proactively intervene in the middle of a multi-user chat, it requires substantial effort to annotate a dialogue state for each user turn in multi-user chats. By contrast, query rewrites in our dataset are a byproduct of our data collection and thus do not require annotation effort. Never- theless, it is an interesting question which method is more effective between turn-level dialogue state tracking and contextual query rewriting (as in our work). We leave this to future work.
Domain Transfer: Here we verify that our dataset is also useful for handling multi-user di- alogues in unseen domains. For this evaluation, use the 62 multi-user chats collected in our pilot study ($3.1) as two unseen domains: finding gifts and finding news. As before, BART trained for dialogue state tracking on our proprietary single- user dialogues for these two domains. During test- ing, the Flatten setting concatenates all utterances in the input multi-user chat as a "single" utterance. The Rewrite setting takes rewrite predicted by the query rewriting model. It is important that this rewriting model is trained only on our datasel with- out exposure to any data from the unseen domains.
or the system. While
Table 7b shows accuracy on intent prediction and informed slot value prediction of the two set- tings. Accuracy on requested slots is not reported, since dialogues in these domains do not have re- quested slots. According to the overall scores, user dialogue state tracking in the unseen domains is generally more challenging than the eight domains in MultiWOZ. Nevertheless, Rewrite still outper- forms Flatten for Intent by points and for Inform by 7 points. This suggests that our dataset can assist dialogue systems handle multi-user dialogues in even unseen domains via a query rewriting step in the language understanding pipeline.
Limitations
We used subset of dialogues in MultiWOZ 2.2 and the first four user utterances in each dialogue. Although this does not lose much information in terms of the diversity of services and slots, includ- ing more utterances for each dialogue would help train dialogue systems to be more robust to longer

---

conversations.
for Computational Linguistics: ACL-IJCNLP 2021. pages 5062-5074.
While the dialogues in our dataset reflect interest- ing social dynamics, some of the more challenging interactions, such as deliberation, are less frequent than other interactions. We can adjust the data col- lection manual to add more of such interactions. We leave this to future work.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. QuAC: Question Answering in Con- text. Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2174 2184.
Ethies Statement
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2009. TREC CAsT 2019: The Conversational Assistance Track Overview. Proceedings of the 28th Text RE- trieval Conference.
We tackle the problem of voice assistants process- ing a chat between users, and this could raise pri- vacy concerns among the users. To alleviate this concern, we also proposed the task of multi-user contextual query rewriting, which is supported by our data. This allows user chats to be rewritten on device and only rewrites that contain task-relevant information to be sent to the server.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Srewledge-Poweree Conversational agents. In International Conference on Learning Representations.
Alexander Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang. Haoran Li, Yashar Mehdad, and Dragomir Radev. 2021. ConvoSumm: Conversation Summa- rization Benchmark and Improved Abstractive Sum- marization with Argument Mining. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1Ith International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 6866-6880.
References
Jacob Andreas, John Bufe, David Burkett Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthyy Theo Lanman, Percy Liang, Christo- pher H Lin, Nya Lintshakh, Andy McGovern, Alek- sandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobew, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task- Oriented Dialogue as Dataflow Synthesis. Transac- tions of the Association for Computational Linguis- tics, 8:556-571.
Guy Feigenblat, Chulaka Gunasekara, Benjamin Szna- jder, Sachindra Joshi, David Konopnicki, and Ranit Aharonov. 2021. TWEETSUMM marization Dataset for Customer Service Findings of the Association for Computational Linguisties: EMNLP 2021, pages 245-260.
Dialog Sum-
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum Corpus: Human- annotated Dialogue Dataset for Abstractive Summa- rization. Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79.
Bill Byme, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. 2019. Taskmaster-1: To- ward a Realistic and Diverse Dialog Dataset. Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference Natural Language Processing (EMNIP-IJCNIP), pages 4515-4524.
Karthik Gopalakrishnan. Behnam Hedayatnia, Qin- lang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tüir. 2019. Topical-Chat. Towards Knowledge- Grruudde Open-Domain Conversations. Interspeech 2019, pages 1891-1895.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu. Semih Yavuz, and Richard Socher. 2020. A Simple Language Model for Task-Oriented Dialogue. In Advances in Neural Information Processing Systems, volume 33, pages 20179-20191. Curran Associates, Inc.
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur 2023. PLACES: Prompting Language Models for Social Conversation Synthesis. In Findings of the Association for Com- putational Linguisties: EACL 2023, pages 868, Dubrovnik, Croatia. Association for Computational Linguistics.
Kundan Krishna, Sopan Khosla, Jeffrey Bigham, and Zachary C Lipton. 2021. Generating SOAP Notes from Doctor-Patien Conversations Using Modular Summarization Techniques. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 4972-
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. Sum: A Real-Life Scenario Dialogue Summarization Dataset. Findings of the Association

---

Hung Le. Doyen Sahoo, Chenghao Liu, Nancy Chen. and Steven C H Hoi. 2020a. UniConv: A Unified Conversational Neural Architecture for Multi-domain Task-oriented Dialogues. Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNL.P). pages 1860-1877
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Tech- nical report.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the Lim- its of Transfer Learning with Unified Text-to- -ext Transformer. arXiv.
Hung Le, Richard Socher, and Steven C.H. Hoi. 2020b. Non-autoregressive dialog state tracking. In Interna- tional Conference on Learning Representations.
Abhinav Rastogi. Xiaoxue Zang. Srinivas Sunkara, Raghay Gupta, and Pranav Khaitan. 2020. Towards
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-tor Soaiicettt-eqquuee Se-uueueeee Pre- Pre-training for Natural Language Generation, Translation, and Com- prehension. arXiv.
Scalable Multi-Domain Conversational Agents: The Schema- Guided Dialogue Dataset. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8689-869
Yan Song, Yuanhe Tian, Nan Wang, and Fei Xia. 2020. Summarizing Medical Conversations via Identifying Important Utterances. Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 717-729.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqjang Cao, and Shuzi Niu. 2017. DailyDialgg A Manually Labelled Multi-turn Dialogue Dataset. In Proceed- ings of the Fighth International Joint Conference on Natural Language Processing (Volume 1: Long Pa- pers), aiviv, pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021. Question Rewriting for Conversational Question Answering. Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 355-363
Chin- Yew Lin. 2004. ROUGE: A Package for Auto- matic Evaluation of Summarics, In Text Summariza- tion Branches Out, pages 74-8) 81- Barcelona, Spain. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-ar Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, ACI., pages 45. Association for Computational Linguistics.
Haitao Lin, Liqun Ma, Junnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang. and Chengqing Zong. 2021. CSDS: A Fine-Grained Chinese Dataset for Customer Ser- vice Dialogue Summarization. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1436-4451.
Haitao Lin, Junnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2022. Other Roles Matter! Enhancing Role-Oriented Dialogue Summa- rization via Role Interactions. Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2545-2558.
Tom Young, Frank Xing, Vlad Pandelea, Jinjie Ni, and Erik Cambria. 2021. Fusing taik-oriented and open- domain dialogues in conversational agents. arXiv.
Shi Yu, Jiahua Liu, Jinggin Yang, Chenyan Xiong2, Paul Bennett2, Jianfeng Gao2, and Zhiyuan Liu. 2020. Few-Shot Generative Conversational Query Rewrit- ing. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1933-1936.
Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019. Automatic Dialogue Summary Generation for Customer Service. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1957-1965
Kai Nakamura, Sharon Levy, Yi-Lin Tuan, Wenhu Chen, and William Yang Wang. 2022. HybriDialogue: An CormatiionSeckingg Dialogue Dataset Grounded on Tabular and Textual Data. Findings of the Associa- tion for Computational Linguistics: ACI. 2022, pages 481 4992
Yifei Yuan, Chen Shi, Runze Wang, Liyi Chen, Feijun Jiang, Yuan You, and Wai Lam. 2022. McQueen: Benchmark for Multimodal Conversational Query Rewrite. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 4834-4844, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Martin Porcheron, Joel E Fischer, Stuart Reeves, and Sarah Sharples. 2018 Voice Interfaces in Everyday Life. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1-12.
Hamed Zamani, Johanne R Trippas, Jeff Dalton, and Filip Radlinski. 2022. Conversational Information Seeking. arXiv.

---

Xiaoxue Zang, Abhinav Rastogi, and Jindong Chen. 2020. MultiWOZ 2.2 A Dialogue Dataset with Additional Annotation Corrections and State Track- ing Baselines. Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117.
Shiyue Zhang, Asli Celikyilmaz, Jianfeng Gao, and Mohit Bansal. 2021. EmailSum: Abstractive Email Thread Summarization. Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 6895-6909.
Lulu Zhao, Fujia Zheng, Keqing He, Weihao Zeng, Yue- jie Lei, Huixing Jiang. Wei Wu, Weiran Xu, Jun Guo, and Fanyu Meng. 2021. TODSum: Task-Oriented Dialogue Summarization with State Tracking, arXiv.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma. Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5905-5921.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization. Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Compulational Linguistics: Human Language Technologies, pages 5927-5934.
Qi Zhu, Kaili Huang. Zheng Zhang, Xiaoyan Zhu, and Minlie Huang. 2020. CrossWOZ: A Scale Chi- nese Cross-Domain Task Oriented Dialogue Dataset. Transactions of the Association for Computational Linguistics, 8:281-295

---

A
Data Details
A.1 Example Dialogues
Example
User1:
We need to book train from cambridge. Yes, we'd like a train that leaves after 14:30.
User2:
Rewrite:
need to take a train from cambridge. would like to leave after 14:30. would be happy to help you with your request, first will need to know your destination and what day you would like to travel. Should we go on Tuesday or Wednesday? Let's go on Wednesday.
System:
User2:
User1:
User2:
Okay. Our destination is Peterborough.
Rewrite:
I've like to travel to Peterborough on Wednesday, please.
System:
have on departing at 14:34. Would you like me to book it?
User2:
Yes, that sounds good.
User1:
Go ahead and book that train.
User2:
There are eight people in our party.
Rewrite:
Yes, for eight please.
System:
Your booking was successful. That is 132 GBP at the station. Is there anything else can do for you today?
User1:
We should also find an expensive restaurant.
User2:
Please see if there any options that eastern european food.
Rewrite:
Can you find me a place eat? would like an expensive eastern european restaurant. Thanks.
System:
I'm not showing any eastern curopean restaurants. Would you like to try more specific type of food?
Example 2
User2:
Hello.
User1:
Let me talk.
User2:
No, I've got this.
User1:
No. Can you direct us to the nearest police station?
Rewrite:
Can you direct me to the closest police station?
System:
Yes I can. The Parkside Police Station is located at Parkside, Cambridge.
User1:
Can you give us the phone number?
User2:
And the postcode.
Rewrite:
Can you please give me the postcode and phone number?
System:
The phone number is 01223358966 and the postcode is CBIIJG.
User1:
Okay.
User2:
We got it, thanks.
Rewrite:
Ok, I've got it. Thanks.
System:
You are welcome, Have a nice day.
Example 3

---

User1:
Hi Xavier, how are you?
User2:
Great, you ready to get this trip settled? You have the info?
User1:
I am and do. We need an expensive hotel with free parking.
Rewrite:
am looking for hotel that is expensive and has free parking.
System:
have about 5 great options for you. Do you prefer certain area in the city?
User2:
I'll be on the east side that day. Can you do that?
User1:
Well, I'll be on the west but sure I can make it over there.
User2:
Great, and we'd like a guesthouse.
Rewrite:
I'm looking for guesthouse in the east side of town. am afraid I have nothing available with those specifications. Would you like a different are or a hotel?
System:
User1:
Well damn. What should do?
User2:
If not expensive then what about moderately priced you think?
User1:
That could work. Can you find that?
Rewrite:
Are there any moderately priced guesthouses that part of town? Yes, both the Carolina B&B and the Warkworth House are moderately priced guesthouses on the east side. Would you like room at one of these?
System:
User1:
Great! I like the Warkworth House.
User2:
Well I am partial to the Carolina B&B. Let's see which have availability.
Userl:
That's the best way to decide.
User2:
Do either have rooms for peeps 5 nights beginning Tuesday?
Rewrite:
Yes, could you see if either of them have availability starting on Tuesday for nights for people?
System:
You have reservation at the carolina bed and breakfast for Tuesday. Your reference number is BOHPIIFE
A.2 Distributions of Services, Intents, and Slots
We compare MultiWOZ 2.2 and MultiUserWOOZ (our data) in terms of the distributions of services (Figure 2), intents (Figure 3), informed slots (Figure 4), and requested slots (Figure 5). They are counted at the turn level; active intents and informed/requessedd slots are counted for each turn, and a service is counted if it has any active intents for each turn.
The least common services in both datasets are bus, hospital, and police. These services are represented substantially more in MultiUserWWOZ than MultiWOZ for both intents and informed/requested slots. We think this is desirable because it increases the exposure of these services to model training. Besides them, the attraction and restaurant services also have higher representation in MultiUserWOOZ than MultiWOZ. The taxi, train, and hotel services have lower representation in MultiUserOOZ than MultiWOZ, but this does not seem problematic because their proportions are high.

---

attraction
dataset
MuttiOz
MultiUserWOZZ
hospital
hotel
service
police
restaurant
taxi
10
20
25
%
Figure 2: Distributions of services.
book_ hotel
dataset
Multiwz
MultiUserWOZ
book_ train
find_attresd ind_sttraction
find bus
intent
find nnd_hospital
hotel
find_ ind_police
find restauraat
find_taxi taxi
find train
15 Figure Distributions intents.

---

atraction-area
dataset
MultiWOZ
attractionname
MultiUserOOZ
straction-type
bus-day
bus-deparlure
bus-destination
leavlea
hospital-department
hotel-area
hotel-bookday
hotel- hotel-bookpen bookpeople
booel-bookstay
hotel-internet
hotel-name
hotel-parking
hotel- hotel-pricerange
informed slot
hotel-stars
hotel-type
restaurant-area
restaurant-bookday
restaurant-bookk bookpple
restaurant-bookimme
restaurant
restaurant-pricerange
taxi arriveby
taxi-departure
taxi Soi-destination
taxi-laveeat
trin--ariveby
train bookpeople
train-day
train-departure
tain-destination
trin-llavveat
2
3
6
%
Figure 4: Distributions of informed slots.

---

altrcciio-aadress
attraction-area
atractionenttancceeee
attraction-name
attactton-parking
attractionphonee
attaction-poslcode
attraction-type
hospital-address
hospital-name
hospital-phone
hosprtal-postcode
hotel-address
hotel-area
hotel-internet
hotel-parking
holel-phone
hotel- postcods
hotel- pricerange
hotel-ref
read requested slot
hotel-stars
hotel-type
police-address
police-department
police-name
police-phone
police postcode
gestaurant-address
restaurant-food
restaurant-name
restaurant-phone
restaurant-postoode
restaurant- pricerange
restaurant-ref
taxi name
taxi-phone
taxi-type
rain-arriveby
tain-duration
train-leaveat
train-name
train-price
dalaset
train-ref
MultiWOZ
MultiUserWOZZ
train-trainid
%
Figure 5: Distributions of requested slots.

---

A.3 Dynamics Analysis
We analyzed three types of social dynamics in multi-user chats, namely, slot elicitation, social chatter, and deliberation.
Slot elicitation refers to users asking each other's preferences related to intents and slots. For example,
LLtt me think, there are how many in our group?"
"Do you need any amenities?"
"Do you think we should call first?"
Social chatter refers user utterances that are not included in original source utterances but used for
social conversation, such as jokes, feelings, etc. For example,
"I'm not sure, but I've heard good things."
"You don't 't have to tell the whole world how much money you have."
"That's way too early for us."
Deliberation refers to making a decision over multiple options. For example.
Userl: "Either 4-5 not sure." -> User2: "Better make it five to be safe."
Userl: "That might not be that bad." User2: "Depends the price" Userl: "It can't be that much."
"Do you think we should leave in the evening?" User2: "I would prefer night time honestly."
Userl:
To analyze the frequency these dynamics our data, we randomly sampled 100 multi-user chats for each split (a total of 300 chats). Next, a co-author of this paper marked each chat as to whether contains slot elicitation, social chatter, and deliberation. Slot elicitation appears in 24%, social chatter 23 %, and deliberation 2%.

---

B Data Collection Details
B.1 Turker Qualifications and Compensation
We recruited turkers who satisfied the following criteria.
Master Workers
Number of HITs Approved 2 95
HIT Approval Rate (%) for all Requesters' HITs > 95
We paid $2.00 for each HIT. Assuming each HIT takes minutes, we believe this pay reasonable.
B.2 Tasks
Figure 6-8 show MTurk task pages for data collection.
Our instructions do not specify how the collected data would be used. But since we asked turkers to create stories, there is no risk of unintentional privacy breaches.

---

You will be given a conversation between a customer and an agent. The goal this task to convert this conversation to conversation between TWO customers and the agent where the customers are making decisions together. Specifically, expand each utterance of the customer to a chat between two customers without changing the original intent of the utterance.
Example
Original Utterance
need train reservations from norwich to cambridge I'd like to know the price as well.
Note
Make sure to include preferences for ["train-ariveby]] if the original utterance contains them.
111 You should write customer utterances 111
Customer1 Customer2
We want to reserve a train. Are we leaving from norwich? Yes and get off at cambridge.
Customer1
Please let us know the price as well.
111111
Agent
have 133 trains matching your request. Is there specific day and time you would like to travel? They are all $50.
Blue texts indicate preferences of the customers and red texts indicate information requested from the agent. These texts must be included in the expanded chat in their literal form. If the original utterance has note, make sure to include the information requested in the note long as the original utterance contains it. If the original utterance does not contain the requested information, you can ignore it.
If the original utterance contains an answer to the agent's question (e.g. "Yes", "No"), make sure that the expanded chat too contains it.
The last utterance of the expanded chat is meant to be spoken to the agent.
Assume that the agent is listening to the chat between the customers. So, you don't have to
repeat summarize the chat when speaking the agent.
The expanded chat should naturally follow the past conversation and naturally lead to the agent's response.
The agent sometimes does not provide some information requested in the original utterance they can't. Do not omit a request in the original utterance just because the agent's response does not contain answer.
You may add some social chatter and deliberation make the conversation natural (see the good example below). But the resulting chat should reflect the same preferences and requests as the original utterance more, no less.
Figure 6: Data collection task instructions (1/2).

---

Good Example
Original Utterance Yes, one more thing. We need a taxi that will take us from Clare Hall to our guesthouse.
Customer2
Sounds good. Ah more thing. Should we take bus to get back to our guesthouse from Clare Hall? It's too much of a hassle. Let's take taxi. Okay. Taxi please.
Customer1
Customer1
Agent
What time would you like that taxi?
The original utterance contains an answer to the agent (i.e. "Yes") and the expanded chat also conveys it (l.e. "Sounds good").
There is deliberation (bus taxi) and social chatter ("too much of hassle"), ") but they do not change the original preferences and requests.
Bad Example
Original Utterance No thanks. Can you book train that goes to St. Louis Station. Can you tell me what time it arrives
Customer1
What was the station we'll be going to? Oh Louis. think we need two tickets. When will the train arrive?
Customer2
Agent
Booking was successful, the total fee is 13.28 GBP payable at the station. Reference number LMZE8AMH.
The original utterance begins with an answer the agent (i.e. "No thanks") but this missing the expanded chat.
The highlighted texts rephrased ("St. Louis Station" to "St. Louis" and "arrives" to "arrive"), which should not.
New information "two tickets" introduced, which can be inconsistent with the agent's original behavior and response.
Figure 7: Data collection task instructions (2/2).

---

Original Utterance Hello, I would like information on the Sesame Restaurant and Bar
Customer1
Write an utterance
Customer2
Write an utterance
Agent
Sesame Restaurant and bar number 01223358899 and it's address is 17 Hills Road City Centre.
Original Utterance would like appointment for
people
Monday
Note
Customer1
Write utterance
Customer2
Write utterance
Customer1
Write utterance
Customer2
Write utterance
Agent
have booked your request for people at the Sesame Restaurant and Bar, located at 17 Hills Road City
on Monday.
Original Utterance Can have the reference number please?
Note
Make to include preferences for [restaurantbbooktime] if the original utterance contains it/them.
Customer1
Write an utterance
Customer2
Write an utterance
Agent
Yes certainly. However, there seems to be an error was unable to book that time. Is there another time could book for you?
Original Utterance
Is
5PM available instead?
Customer1
Write an utterance
Customer2
Write an utterance
Customer1
Write utterance
Agent
Yes it have booked table for you 17:00 Monday for 7 people. The reference number is CZ3JGGJP.
Submit
Figure 8: Data collection task.

---

B.3 Number of Generated Utterances and Speaker Labels
The number of generated utterances and speaker labels for each source utterance were predefined as follows. We stochastically sampled the number of generated utterances kt E {2, 3, 4] where the weight for k is +T and n is the total number of informed or requested slots (so that is centered around this number). Further, we randomly chose Customerl or Customer2 as the first speaker of the generated chat.
B.4 Slot Constraints
When displayed source user utterances to turkers, we highlighted informed slot values and requested slot names. We constrained that the highlighted texts be mentioned in generated multi-user chats. However, the slot values names annotated in MultiWOZ do not always exactly match their expressions in user utterances. For example, informing hotel-parking.yes may be expressed as want free parking" rather than "yes", and train-bookpeople:1 may be expressed as "one ticket" rather than "1 ticket". Similarly, requesting train-ref may be expressed as "Give me a reference number", and restaurant- may be expressed as "What kind of food do they serve?". Therefore, to capture informed slot values and requested slot names as much as possible, we added regex patterns as in Table 8.
When informed slot values or requested slot names cannot be captured, it is usually because they are not mentioned and in rare cases they expressed in peculiar forms. We still provided turkers with the slot names and asked them to include information about those slots if found in the source utterance.

---

(Slot Name, Value)
Added Patterns
Slot Name
Added Patterns
(".*". "centre")
"center". "same area", "area"
"phone"
(phoneelcottacllleeppoonn
(".*" "east")
"same area", "area"
ber', "phone" "number"
(".*". "outh")
"same area", "area"
"postcode"
"postcodes?. (postlpostal)
(".*", "north")
"same area", "arca"
codes?"
(","", "west")
"same area", "arca" "one". "same group of people" "two". "same group people" "three". ""same group people" "four". "same group people" "five". "same group people"
address"
"address(es)?", "posteode" "price range", "price"
(".*", -1")
"-price"
(",",, "2")
"-ref"
freftreerreneelcmiimaatio
("""
(numberlit)", "reference" "entrance fees?" "travel times?"
("*","""
"entrancefee"
("*"."5)
"train-duration"
(6")
"six". "sume group people" "seven". "same group people" "eight" "same group of people" "nine", "same group of people" "for (melmyself)", "me" "guest house"
"taxi-type"
"car type" "arrival time". "arrives?", "when" "departure time". "depart", "leave", "when"
("."" *7")
"trann-arrieey"
(".""""8 "8")
"rain-leavcat"
(".*", "9")
("bnokpeople", "I")
traiintraiindd
"train id", "trainid", "id" "what area", "exact area". "area"
(".*", "guesthouse")
"arraa Trestauantt food"
(".*". swimmingpool"
"swimming pool". "pools", "pool"
"(what "food type" (((indisortttpp) food" "(what Ylypes? of attraction". "attraction types?". "what types?".
(",*", "mnnday))
"same day"
(".*", "uesday")
"same day"
"tttraction-ypp"
(".*". wednesday")
"same day"
("."", "thursday")
"same day"
"types?"
(".*". "friday")
"same day"
"hotel-intemet"
"(free (?(wi fiwiifilwifllitteeee" "(free Yparking" "hotel type". "what type of ho- tels?"
("."", "saturday")
"same day"
"hotel-parking"
("."". "sunday")
"same day"
"hotel-type"
(". "expensive")
"same price range", "price range", "high priced", "high-end", "high end". "luxurious", "pricey", "up- scale". "a lot of money" "same price range". price range". "reasonable", "inexpensive", "af- fordable"
"hotel-stars"
"how many stars". "stars of the hotel", "stars"
"cheap""""p")
(b) Added patterns for requested slot names.
(".*", "moderatee"
"same price range", "middle price range", "moderately priced", "price range", "isn't too high priced", "mid range". "not too expensive", "not cheap"
(""hotel-internetS" "yes")
"free internet", "free wifi". "inter- net", "wifi", "wi-fi" "free parking", "parking" "no (other iparticular Y?prefer- ences?", "no particular", "(don'th idd not) (really lactually Y?have mntferencelprffernncess] "(don' ttdonntdo not) mind", "(doesn' 'idoes notidon 'ido not) (re- ally lactually }(Ccarelmatter",, "what ever". "whatever", "whenver". "not picky", "(don tido notidoesn' 'iddees not) need"
(""hotcl-parkingS, "yes") (".". dontcaree")
(a) Added patterns for informed slot values.
Table 8: Added patterns for informed slot values and requested slot names.

---

C Data Validation Details
C.1 Turker Qualifications and Compensation
For the validation task ($3.3), we first ran a qualification task, and Figure 9-12 show MTurk task pages for validator qualification. We recruited turkers using the same criteria as above. 13/55 qualified turkers participated in the main validation tasks.
We paid $0.50 for each HIT. Assuming each HIT takes minute, believe this pay is reasonable.
C.2 Tasks
Figure 13-15 show MTurk task pages for validation. Our instructions do not specify how the collected data would be used. But since we asked turkers create stories, there is no risk of unintentional privacy breaches.

---

Customer Chat
Customer1 Let's visit college town.
Customer2
Sure, we will around the west side of town.
Summary
want to find college in the east.
Q1. the relationship between Customer1 and Customer2 really customers making decision together?
Yes: They are two customers making decision together
No: Their relationship is something else, e.g, one customer and the other is an agent
Q2. the customer chat realistic chat between humans?
Very much: Very realistic chat between humans Acceptable: Acceptable as chat between humans
Definitely not: Definitely not acceptable as a chat between humans
Q3. Is the last utterance of the customer chat realistic as an utterance toward the agent?
Very much: Very realistic
utterance toward the agent
Acceptable: Acceptable as an utterance toward the agent
Definitely not: Definitely not acceptable an utterance toward the agent
Q4-1, 1. is any important information in the customer chat missing in the summary to the extent that the agent perform action or provide information that is not intended by the customers? (See the instructions regarding what is acceptable and what is not)
O Yes: Some important information in the chat is missing in the summary
No: No missing information
Q4-2. If you chose "Yes", specify what information is missing.
Q5-1. Does the summary contain additional information that is not present in the customer chat to the extent that the agent perform action provide information that is not intended by the customers? (See the instructions regarding what is acceptable and what is not)
Yes: The summary contains additional information that is present in chat
No: The summary contains additional information
Q5-2. If you chose "Yes", specify what additional information is present.
Figure 9: Validation qualification task (1/4). Turkers should choose "No" for Q4-1 Q5-1 to pass.

---

Agent
There are 5 colleges in the west. churchill college, clare college, clare hall, magdalene college, and queens' college.
Customer Chat
Customer2
entrance
Hall?
Customer1
There entrance
Customer2
What time does open to the public?
Customer3
It's open from to the public. you need any other information?
Summary
Could you
entrance fee for clare hall please?
Q1. is the relationship between Customer1 and Customer2 really two customers making decision together?
Yes: They are two customers making decision together
Their relationship something else, e.g., one customer and other agent
the chat realistic chat between humans?
Very much: Very realistic as a chat between humans Acceptable: Acceptable as chat between humans
Definitely not: Definitely not acceptable as chat between humans
the utterance the customer chat realistic
toward the agent?
Very much: Very realistic as an utterance toward the agent
Acceptable: Acceptable as an utterance toward the agent
Definitely not: Definitely not acceptable as an utterance toward the agent
Q4-1. Is any important information in the customer chat missing in the summary to the extent that the agent may perform an action or provide information that not intended by the customers? (See the instructions regarding what is acceptable what not)
Yes: Some important information in the chat is missing in the summary
No: No missing information
Q4-2. If you chose "Yes", specify what information missing.
Q5-1. Does the summary contain additional information not present the customer chat to the extent that agent may perform an action or provide information that not intended by the customers? (See instructions regarding what acceptable what not)
Yes: The summary contains additional information that is not present in the chat
No: The summary contains no additional information
Q5-2. If you chose "Yes", specify what additional information is present.
Figure 10: Validation qualification task (2/4). Turkers should choose "No" for Q1 pass.

---

Agent
The entrance to Clare Hall is free. Anything else you need?
Customer Chat
Customer1
What the address and phone number for Clare Hall?
Customer2
Please give address and phone number Clare Hall.
Customer1
Please provide address phone number for clare hall.
Customer2
Phone number and addresss for Clare Hall, please.
Summary
Could you please provide me with the address and phone number?
Q1. the relationship between Customer1 and Customer2 really two customers making decision together?
Yes: They are two customers making decision together
No: Their relationship is something else, e.g, one is customer and the other is an agent
Q2. Is the customer chat realistic chat between humans?
Very much: Very realistic chat between humans Acceptable: Acceptable chat between humans
Definitely not: Definitely not acceptable chat between humans
the last
of the
realistic
toward the agent?
Very much: Very realistic as an utterance toward the agent
Acceptable: Acceptable as an utterance toward the agent
Definitely not: Definitely not acceptable as an utterance toward the agent
Q4-1. Is any important information in the customer chat missing in the summary to the extent that the agent may perform an action or provide information that is not intended by the customers? (See the instructions regarding what is acceptable and what is not)
O Yes: Some important information in the chat is missing in the summary
O No: No missing information
Q4-2. If you chose "Yes", specify what information is missing.
Q5-1. the summary additional information that is present the customer chat the that agent may perform action provide information that is intended by the customers? (See the instructions regarding what acceptable and what is not)
Yes: The summary contains additional information that is not present in the chat
No: The summary contains no additional information
Q5-2. If you chose "Yes", specify what additional information is present.
Figure 11: Validation qualification task (3/4). Turkers should choose "Definitely not" for Q2 to pass.

---

Agent
the phone number 01225332360 and address isherschel road
Customer Chat
Customer2
What moderately priced restaurant in the area of Clare Hall? What do you want to eat?
Customer1
Summary
What kind of moderately priced restaurants that area? want eat after visit the college.
Q1. Is the relationship between Customerl and Customer2 really two customers making decision together?
Yes: They are two customers making decision together
Their relationship something else, e.g. customer and other agent
Q2. the customer chat realistic chat between humans?
Very much: Very realistic as a chat between humans Acceptable: Acceptable as chat between humans
Definitely not: Definitely not acceptable as chat between humans
Q3. the last utterance of the customer chat realistic as an utterance toward the agent?
Very much: Very realistic utterance toward the agent Acceptable: Acceptable as utterance toward the agent
Definitely not: Definitely not acceptable utterance toward the agent
Q4-1. is any important Information in the customer chat missing the summary to the extent that the agent may perform action provide information that not intended by the customers? (See the instructions regarding what is acceptable and what not)
Some important information the chat missing the summary
O No: No missing information
Q4-2. you chose "Yes", specify what information is missing.
Q5-1. Does summary contain additional information that not present in customer chat to the extent that the agent may perform an action or provide information that not intended by customers? (See instructions regarding is acceptable what not)
O Yes: The summary contains additional information that not present in the chat
No: The summary contains no additional information
Q5-2. If you chose "Yes", specify what additional information is present.
Submit
Figure 12: Validation qualification task (4/4). Turkers should choose "Definitely not" for Q3 to pass.

---

will given conversation among two customers and one agent, where customers are (supposedly) making decisions together and getting help from the agent.
Each chat between the customers is labeled with summary. Your task is to evaluate the quality of the chat and the summary.
Example
Agent
Benny's 's is a good restaurant in the south area. Would you like me to make reservation for you?
Customer Chat
Customer2
Ill love that. Does Alice join us?
Customert
No, she needs to attend business meeting.
Customer2
Ok. We'd like to make reservation for people at 12:45 on Monday.
Customer1
No, it's dinner, not lunch. 17:45 please. Give their phone number too.
Summary
That would great. Please make reservation people 17:45 on Monday. Give me their phone number.
Q1. Is the relationship between Customert and Customer2 really two customers making a decision together?
Yes: They are two customers making decision together
No: Their relationship is something else, e.g., is customer and the other is agent
Q2. Is the customer chat a realistic chat between humans?
Very much: Very realistic as chat between humans Acceptable: Acceptable as chat between humans
Definitely not: Definitely not acceptable chat between humans
Q3. Is the last utterance of the customer chat realistic as an utterance toward the agent?
Very much: Very realistic as an utterance toward the agent Acceptable: Acceptable as an utterance toward the agent Definitely not: Definitely not acceptable as an utterance toward the agent
Q4-1. Is any important information the customer chat missing in the summary to the extent that the agent may perform an action or provide information that is not intended by the customers? (See below regarding what is acceptable and what is not)
Yes: Some important information in the chat is missing in the summary
No: No missing information
Q4-2. you chose "Yes", specify what information missing.
Q5-1. Does the summary contain additional information that is not present in the customer chat to the extent that the agent may perform an action or provide information that is not intended by the customers? (See below regarding what acceptable and what is not)
Yes: The summary contains additional information that is not present in the chat
No: The summary contains no additional information
Q5-2. you chose "Yes", specify what additional information present.
Figure 13: Validation task instructions (1/2).

---

What is good summary?
Summaries have to
Contain all essential information for the agent to perform an action or provide information intended by the customers, such as preferences for a search (e.g., "17:45", "Monday"). requested information (e.g., "phone number"), ) answers to the preceding question of the agent (e.g., "I'll love that").
Summaries do not have to
Contain unimportant details the chat, such as social chatter (e.g.. "attend business meeting") and declined preferences (e.g., "12:45"). )
first-person plural instead first-person singular. For example, it me" place of "give
summary contains "give
chat.
Use the exact wording of direemmen/diisageemment For example, it fine if the summary contains "That would be great" or "Yes" place of "P'll love that" in the chat.
Figure 14: Validation task instructions (2/2).

---

Agent
The University expensive, hotel with free wifi.. Comparatively, the Alexander Breakfast cheaply priced guesthouse, also
Customer Chat
Customer2
The University Arms will do.
Customer1 Can you book that for us? Yes, there'll be of us, staying 3 nights, starting wednesday. And also please give us the reference number.
Customer2
Customer1
Summary
Please book me some rooms for The University Arms accommodate people for nights starting wednesday. Can you also provide me the reference number after book?
Q1. Is the relationship between Customer1 and Customer2 really two customers making decision together?
Yes: They customers making decision together
No: Their relationship something else, e.g., customer and the other is agent
Is the customer chat realistic chat between humans?
Very much: Very realistic as chat between humans Acceptable: Acceptable chat between
Definitely Definitely not acceptable chat between humans
Q3. Is the utterance customer chat realistic as an utterance toward the agent?
Very much: Very realistic as an utterance toward the agent Acceptable: Acceptable as an utterance toward the agent
Definitely not: Definitely not acceptable as an utterance toward the agent
Q4-1. Is important information in the customer chat missing in the summary to the extent that the agent may perform an action or provide information that not intended by the customers? (See the instructions regarding what is acceptable and what is not)
Yes: Some important information in the chat is missing in the summary
No: No missing information
Q4-2. If you chose "Yes", specify what information is missing.
Q5-1. Does the summary contain additional information that is not present in the customer chat to the extent that the agent may perform an action provide information that is not intended by the customers? (See the instructions regarding what is acceptable and what is not)
The summary contains additional information that not present in chat The summary contains additional information
Q5-2, you chose "Yes", specify what additional information present.
Figure 15: Validation task.

---

D Experiment Details
D.1 Model Details
BART-base, BART-large, T5-base, and GPT-2-base have 139M, 406M, 223M, and 124M parameters, respectively. For all experiments, we set the hyperparameters as follows: batch size=8, epoch=10, learning rate=2e-5, weight decay=0.01, and beam size=5. The input length is set to be 64 for BART-base, BART- large, and T5-base, and 128 for GPT2. Each task takes about 40 60 minutes for training on Tesla V100.
All experiments and models are developed with Huggingface Library licensed under Apache 2.0. Our models are fine-tuned based on publicly released GPT-2 (MIT License), BART-base and BART-large (Apache 2.0 License), and TS-base (Apache 2.0 License)
D.2 Query Rewriting Results
Table 9 illustrates example outputs of the summarization models.
User2:
I think I want to arrive in the evening. User1: Yeah, but not after 9:00.
User2: Exactly.
Userl: So the urrival time of 20:54 should work.
Ground-True Truth Rewrite: Yes, I think the 20:54 arrival time should work.
T5: want to arrive in the evening at 20:54. (Good)
BART: i want to arrive in the evening after 9:00, (Picked up the wrong slot value)
GPT2: i would like to arrive at 20:54 on sunday. (Hallucination)
User1: We don't it travel much, this is special.
User2: Yes, we have enough saved.
Userl: We also need good experience.
User2: Yes, so will like it to be expensive.
Ground- Truth Rewrite: I would prefer that be expensive.
T5: i don't travel much, this is special. also need a good experience. (Picked up social chatter and missed a slot value) BART: i don't travel much, this special. (Picked up social chatter and missed slot value) GPT2: yes, and i'd like eat in moderate price range. (Hallucination and missing slot value)
User1: Oh, we are so going to have fun there. Make sure they know we have 5 total guests we will need to make sure get the reference number and book room for 4 nights starting Monday.
User2: Okay that is right Monday will be great for me and others as well.
User1: Yes so I' m in for Monday just make sure we get reference number to confirm.
User2: Okay will need the reference number for the stay starting Monday for people for 4 nights.THanks. THanks.
Ground- Truth Rewrite: Yes. Please book me room for nights starting Monday. We have 5 total guests. also need reference number please.
T5: would like book a room for 5 people for 4 nights starting monday. (Missed requested slot)
BART: yes please book for people and 4 nights starting monday. (Missed requested slot) GPT2: thursday from 3 pm and 3 people for nights starting 19:45 (Hallucination, wrong slot values, and missing requested slot)
User1: Thank you very much for all that information.
User2: We also need to make booking for the restaurant in the West.
User1: Please make sure that serves Italian food.
User2: And also make sure that the restaurant should be in the moderate price range.
Ground-Truth Rewrite: I need restaurant. The restaurant should be in the west and should serve italian food. The restaurant should be in the moderate price range
T5: thank you very much for all that information. also need make booking for the restaurant the west. please make sure that serves italian (Missed slot value)
thank you very much. also need make booking for the restaurant in the west. it serves italian food. (Missed slot value)
BART:
GPT2: thank you, also need to make a booking the restaurant in west and should serve italian food. (Missed slot value)
Table 9: Example outputs of different models.


Reviews:
[{"review_id": "04EeUGcvAf", "reviewer": "Reviewer_bSQy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. In comparison to regular task-oriented dialogues, multi-user task-oriented dialogues have their own characteristics. Should a specific evaluation metric be designed to address these characteristics?**\n\nThank you for bringing up this point. Yes, we agree that multi-user task-oriented dialogues have unique characteristics and need related evaluation metrics. We indeed introduced several criteria and metrics with which we evaluated and analyzed the generated dialogues in the paper and verified that our dataset has high quality. First, in Section 3.3, we used five criteria to measure (a) whether the users’ relationship is two customers making decision together, (b) whether the chat is a realistic chat between humans, (c) whether the last utterance in the chat is a realistic utterance toward the agent, (d) whether the rewrite has any missing information that is present in the chat, (e) whether the rewrite contains any information that is absent in the chat. All these criteria except (b) are specific to multi-user task-oriented dialogues, and our evaluation result shows the high quality of our dataset: (a) 0.99/1, (b-c) 1.8/2, (d-e) 0.98/1 (lines 282-288).\n\nIn addition, in Section 3.4, we measured five important quantities related to multi-user task-oriented dialogues. (a) instances of slot elicitation, (b) instances of social chatter, (c) instances of deliberation, (d) number of entities, (e) number of negations. In our dataset, (a) and (b) appear in 24% and 23% of dialogues, which are reasonably high, while (c) is quite low at 2% (lines 297-298). (d) and (e) are 1.2-1.9x as high as the original MultiWOZ (Table 2), which reveals that our dataset is semantically richer and dialectic than single-user dialogues.\n\nWe believe some of these metrics can be automated quite reliably. For instance, we can build a classifier that computes the probability that two users are customers (vs. having another relationship) using existing dialogue datasets, and classifiers that compute the probability that a dialogue contains slot elicitation, social chatter, or deliberation respectively. We can also use a public library, e.g., spaCy, to count named entities and negations, as we already did in Section 3.4. We will add this discussion in the final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper releases the Multi-User MultiWOZ dataset, task-oriented dialogues between two users and one agent.\n2. This paper propose the multi-user contextual query rewriting task that rewrites multi-user chats as concise task-oriented requests.", "reasons_to_reject": "1. In comparison to regular task-oriented dialogues, multi-user task-oriented dialogues have their own characteristics. Should a specific evaluation metric be designed to address these characteristics?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper releases the Multi-User MultiWOZ dataset: task-oriented dialogues among two users and one agent and proposes the novel\ntask of multi-user contextual query rewriting.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jq9JWq5m7H", "reviewer": "Reviewer_6huk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments.\n\n**1. Firstly, it only involves changing one user to two users instead of incorporating more users.**\n\nWe agree that incorporating more than two users would be nice and would be fruitful work. We want to highlight, though, that multi-user task-oriented dialogue is a completely new problem and there is no benchmark dataset at all. While our dataset is limited to two users, we believe our work, the first of its kind, has made meaningful contributions: defining the problem, releasing a good-size dataset, introducing a related task of contextual query rewriting, and verifying its effectiveness in dialogue state tracking. Our dataset would serve as a valuable first step and benefit the research and industry communities for improved datasets.\n\n**2. Secondly, multiple users do not interact separately with the dialogue system; instead, after the two users finish their conversation, the system interacts with them as a whole. This design may not pose sufficient challenges for the dialogue management module.**\n\nThe source utterance of each multi-user chat is exactly the case of each user interacting with the system separately. Therefore, we do not need to generate such utterances again. One can simply train a dialogue system on a mix of multi-user chats and source utterances to cover both user-user chats and user-agent 1:1 chats.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed Multi-User MultiWOZ dataset in this paper can aid researchers in conducting research in this field. The introduction of multiple users amplifies the technical challenges faced by the NLU and DST modules of dialogue systems. Furthermore, this was validated through experiments conducted in the study. ", "reasons_to_reject": "The constructed Multi-User MultiWOZ dataset has certain limitations. Firstly, it only involves changing one user to two users instead of incorporating more users. Secondly, multiple users do not interact separately with the dialogue system; instead, after the two users finish their conversation, the system interacts with them as a whole. This design may not pose sufficient challenges for the dialogue management module. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper releases the Multi-User MultiWOZ dataset, containing task-oriented dialogues between two users and one agent and proposes the multi-user contextual query rewriting task that rewrites multi-user chats as concise task-oriented requests.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fz7pZuCQqm", "reviewer": "Reviewer_CX5P", "summary": "", "questions": "- In Figure 1 (bottom, 2nd dialogue), the system seems to respond even without being prompted by users. Is this expected? In the real world, how does a system know when to respond without being prompted? For instance, in this example in Figure 1, the system can also respond right after user 2 utterance “Let’s do 19:45, Steve can’t get there till after 19:00”\n- From a human user perspective, it is probably more sensible to replace “you” in current dialogues with some special token “<system>” to prompt the system to respond. The current way will limit the way users can interact with each other (e.g. how can a human user address the other human party?)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments.\n\n**1. In the data generation process, each annotator/turker is tasked to expand all user utterances in each task. It is would be more natural to have two different humans interact with each other (that can reach the same dialogue goal as before). The current approach might create data bias (e.g. confirmation bias). However, this bias seems to be minimal as shown (though not explicit) by the data validation results in Section 3.3.**\n\nWe tried pairwise dialogue generation in our pilot study (Section 3.1), where we randomly paried up participants and let them naturally interact with each other via a texting interface. Unfortunately, the resulting dialogues were pretty poor and less realistic than dialogues generated by single annotators (lines 360-368). We speculate that this is because laypeople are usually not good at creating a collaborative scenario and holding a relevant dialogue in interactive settings. The same was observed even among family members who shared common social grounds. Tasking one annotator to generate the entire dialogue has been conducted in prior research as well to ensure quality. For instance, (1) Young et al. (2022), “Fusing Task-oriented and Open-domain Dialogues in Conversational Agents”, (2) Byrne et al. (2019), “Taskmaster-1: Toward a realistic and diverse dialog dataset”, (3) Nakamura et al. (2022), “HybriDialogue: An Information-Seeking Dialogue Dataset Grounded on Tabular and Textual Data”.\n\n**2. The current annotation of slot-value pairs, and intents are available after each multi-user chat, rather than after each user utterance. How would dialogue state tracking be applied here? How do we know when a chat is finished, then a dialogue state is required, and a system is expected to respond?**\n\nThe dialogue state is needed when a user utterance is device-directed and should be replied to by the system. This requires a device-directedness detector (or more generally an addressee detector), which is far from a solved problem but beyond the scope of this paper. In essence, all other turns between users are considered as a side-discussion and are not processed separately by the dialogue system. Once a device-directed (DD) utterance is detected, the relevant context from the side-discussion (i.e., multi-user chat) between the users is extracted and added to the dialogue state. For reliable DD utterance detection, dialogue systems can use a predefined protocol that indicates the user prompting the system, e.g., by saying “go ahead Alexa/Siri” or by looking at the device which is detected by the device camera. We did not include this in our dataset because this is a simple matter of protocol that varies by different dialogue systems.\n\nDetecting the DD utterances based solely on semantics is more difficult and ambiguous. But this ambiguity is neither a deficit nor a completely new challenge to our dataset. It is the nature of any turn-taking dialogues with two or more speakers involved. While we usually assume utterance boundaries are given in single-user dialogue state tracking, in reality, detecting utterance boundaries is nontrivial due to the user mumbling, making corrections in the middle, etc., and practical systems usually have to rely on non-semantic cues anyway, such as a long pause, as a signal for the end of an utterance.\n\nTracking dialogue states after each multi-user chat has some limitations as well, e.g., does not allow for many types of proactivity in the middle of the user-user chat. But it is a good compromise to stimulate multi-party interaction research. Alternatively one could parse a multi-user chat and track dialogue states at the utterance level within the chat as the reviewer seems to suggest; e.g., the intents in a multi-user chat might have to be augmented, and part of the corpus should be manually reannotated. This approach would be an interesting direction and we discuss this more in detail in 4 below.\n\n**3. The approach of contextual query rewriting is interesting and might be strong but it is not scalable to other domains and requires additional annotation of original single-user utterances to train such a model. Practically, those annotations are not available in many domains.**\n\nWhile it is true that the accuracy of contextual query rewriting on unseen domains may not be as good as in-domain accuracy (as in most NLP tasks indeed), we verified its scalability in Section 5.3 (lines 528-554) by showing that unseen domains (GiftFinder and NewsFinder) that are substantially different from training domains still benefit from contextual query rewiring in dialogue state tracking. Furthermore, as the capacity of language models increases, it is very likely that their ability to rewrite queries would generalize better across domains. \n\nThe query rewriting approach has additional advantages over other approaches. For instance, compared to training a dialogue state tracking model directly on multi-user chats, it performs better and helps preserve user privacy as emphasized in the paper. Dialogue state tracking within a multi-user chat, which is another alternative approach, requires even more annotation effort than query rewriting, partly because rewrite annotations in our dataset are a byproduct of our dialogue generation. That said, we are eager to see more research come up with better solutions, and we believe our dataset would serve as a valuable testbed. As an example research direction, co-authors have been able to generate multi-user dialogues for new domains automatically from single-user data, using a combination of rule-based expansion followed by paraphrasing.\n\n**4. I would advocate an approach to learn a multi-user dialogue state tracking model which can be considered a form of contextual rewriting (into compact slots and values). Here you would not assume access to the annotation of the original single-user utterances (which is harder to construct than dialogue state annotations).**\n\nWe assume that the reviewer is proposing utterance-level dialogue state tracking within a multi-user chat. This approach makes sense since it can hold rich information, such as different points of view of the users (different values for the same slot per user), meta-intents (e.g., talking about the performance of the system, talking about turn-taking), etc. While we believe this approach would be fruitful and its effectiveness is worth testing, this annotation is a significant investment. First and foremost, the proposed approach needs multi-user task-oriented dialogues to start with. But these are already a very scarce resource in the first place. It is indeed one of our contributions to obtain such dialogues by expanding single-user dialogues (which are comparably abundant) and publicly release them. Moreover, query rewriting annotation is a byproduct of this process, so no additional effort is needed on top of this dialogue generation. \n\nEven if we assume there are other cheap sources of multi-user task-oriented dialogues, we would humbly disagree that the utterance-level annotation of intents and slots is easier than query rewriting. First, annotators have to annotate dialogue states after every turn, which induces substantial cognitive load. Second, annotation of dialogue states is prone to errors, as shown by the fact that the original MultiWOZ had so many annotation errors and the SGD dataset avoided dialogue state annotation completely by using dialogue simulation. Lastly, it increases the dialogue state tracking model’s dependency on the domain schema since it has to predict domain-specific intents and slots for each utterance. By contrast, the query rewrite annotation requires reading the entire chat and rewriting it as a summary only once per multi-user chat. That said, we think that it would be an interesting methodological question which method is more scalable, reliable, and effective in dialogue state tracking.\n\n**5. In Figure 1 (bottom, 2nd dialogue), the system seems to respond even without being prompted by users. Is this expected? In the real world, how does a system know when to respond without being prompted? For instance, in this example in Figure 1, the system can also respond right after user 2 utterance “Let’s do 19:45, Steve can’t get there till after 19:00”**\n\nPlease refer to our answer to 1.\n\n**6. From a human user perspective, it is probably more sensible to replace “you” in current dialogues with some special token “<system>” to prompt the system to respond. The current way will limit the way users can interact with each other (e.g. how can a human user address the other human party?)**\n\nWhen users talk to voice assistants in general and in most task-oriented dialogue datasets, they call the systems “you” instead of a system token or the system name (e.g., “can YOU recommend books for kids?”). In our pilot study as well, participants naturally used \"you\" when speaking to the voice assistant. Hence, it is more reflective of user behavior to use “you” to refer to both other users and the system. This of course increases the complexity of the problem, but we think this is something we need to solve rather than avoid. For instance, we could introduce an explicit prompting protocol (e.g., a wake word), which is easy to add to our dataset, detect who the user looks at when they speak, and even improve semantic understanding.\n\n**7. Missing References:\n\"Non-autoregressive dialog state tracking.\" ICLR (2020). \"UniConv: A unified conversational neural architecture for multi-domain task-oriented dialogues.\" EMNLP (2020).**\n\nWe will discuss these papers in the Related Work section in the final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A unique and interesting extension of the current task-oriented dialogues to enable interactions with multiple users in a single dialogue. Additional components are interesting, including social chatter and deliberation, and make the dialogues more natural. \n- The data collection process is comprehensive. The data collection is grounded by a pilot study to reveal some key features in multi-user dialogues. There is also a validation round and quality assessment study to validate the generated dialogues based on a set of questionnaires and qualities.  \n- Good result analysis, including an analysis of query writing error types based on qualitative samples.", "reasons_to_reject": "- In the data generation process, each annotator/turker is tasked to expand all user utterances in each task. It is would be more natural to have two different humans interact with each other (that can reach the same dialogue goal as before). The current approach might create data bias (e.g. confirmation bias). However, this bias seems to be minimal as shown (though not explicit) by the data validation results in Section 3.3. \n- The current annotation of slot-value pairs, and intents are available after each multi-user chat, rather than after each user utterance. How would dialogue state tracking be applied here? How do we know when a chat is finished, then a dialogue state is required, and a system is expected to respond? \n- The approach of contextual query rewriting is interesting and might be strong but it is not scalable to other domains and requires additional annotation of original single-user utterances to train such a model. Practically, those annotations are not available in many domains. \n- I would advocate an approach to learn a multi-user dialogue state tracking model which can be considered a form of contextual rewriting (into compact slots and values). Here you would not assume access to the annotation of the original single-user utterances (which is harder to construct than dialogue state annotations).", "questions_to_authors": "- In Figure 1 (bottom, 2nd dialogue), the system seems to respond even without being prompted by users. Is this expected? In the real world, how does a system know when to respond without being prompted? For instance, in this example in Figure 1, the system can also respond right after user 2 utterance “Let’s do 19:45, Steve can’t get there till after 19:00”\n- From a human user perspective, it is probably more sensible to replace “you” in current dialogues with some special token “<system>” to prompt the system to respond. The current way will limit the way users can interact with each other (e.g. how can a human user address the other human party?)", "paper_topic_and_main_contributions": "The authors propose a new type of task-oriented dialogue for multiple users. In this task, the authors extend the MultiWOZ benchmark from single-user dialogue to multiple-user dialogues. To do that, the authors transform each of the original user utterances into a short conversation between 2 users. \n\nFor this task, the authors propose a new model for contextual query rewriting to rewrite multiple-user queries to a concise task-oriented query so that the newly transformed queries are compatible with existing task-oriented dialogue systems.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\"Non-autoregressive dialog state tracking.\" ICLR (2020).\n\"UniConv: A unified conversational neural architecture for multi-domain task-oriented dialogues.\" EMNLP (2020)."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The dataset introduced in the paper is an interesting contribution to the field of dialogue systems. It addresses a need for realistic, multi-user task-oriented dialogues and offers a comprehensive data collection process and insightful result analysis. This dataset is likely to aid researchers in advancing the development of dialogue systems for complex, real-world scenarios.
Decision: Accept-Findings