SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents
Authors: Shuzheng Si
Wentao Ma
Haoyu Gao
Yuchuan Wu
Ting-En Lin
Yinpei Dai
Hangyu Li
Rui Yan
Fei Huang
Yongbin Li
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Task-oriented Dialogue
Spoken Conversation
Dialogue State Tracking
Response Generation
Natural Language Processing

Abstract: Task-oriented dialogue (TOD) models have made significant progress in recent years. However, previous studies primarily focus on datasets written by annotators, which has resulted in a gap between academic research and real-world spoken con- versation scenarios. While several small-scale spoken TOD datasets are proposed to address robustness issues such as ASR errors, they ignore the unique challenges in spoken conversation. To tackle the limitations, we introduce SpokenWOZ, a large-scale speech-text dataset for spoken TOD, containing 8 domains, 203k turns, 5.7k dialogues and 249 hours of audios from human-to-human spoken conversations. SpokenWOZ further incorporates common spoken characteristics such as word-by-word processing and reasoning in spoken language. Based on these characteristics, we present cross-turn slot and reasoning slot detection as new challenges. We conduct experiments on various baselines, including text-modal models, newly proposed dual-modal models, and LLMs, e.g., ChatGPT. The results show that the current models still have substantial room for improvement in spoken conversation, where the most advanced dialogue state tracker only achieves 25.65% in joint goal accuracy and the SOTA end-to-end model only correctly completes the user request in 52.1% of dialogues. Our dataset, code, and leaderboard are available at https://spokenwoz.github.io/SpokenWOZ-github.io/.

Spoken WOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents
Shuzheng Si Wentao Ma Haoyu Gao Yuchuan Wu', Ting-En Lin', Yinpei D Hangyu Li', Rui Yan Fei Huang' and Yongbin Li
DAMO Academy, Alibaba Group
"Computer Science and Engineering Division, University of Michigan Gaoling School of Artificial Intelligence, Renmin University of China sishuzheng@foxmail com, (mawentao mwt, ghy385779)@alibabz com
Abstract
Task-oriented dialogue (TOD) models have made significant progress recent years. However, previous studies primarily focus datasets written by annotators, which has resulted gap between academic research and real-world spoken con- versation scenarios. While several small-scale spoken TOD datasets are proposed address robustness issues such as ASR errors, they ignore the unique challenges in spoken conversation. To tackle the limitations, we introduce SpokenWOL WOZ, a large-scale speech-text dataset for spoken TOD, containing 8 domains, 203k turns, 5.7k dialogues and 249 hours of audios from numan-to-human spoken conversations. SpokenWOZ further incorporates common spoken characteristics such word-by-word processing and reasoning in spoken language. Based on these characteristics, we present cross-turn slot and reasoning slot detection as new challenges. We conduct experiments various baselines, including text-modal models, newly proposed dual-modal models, and LLMs, e.g.. ChatGPT. The results show that the current models still have substantial room for improvement in spoken conversation, where the most advanced dialogue state tracker only achieves 25. 65% in joint goal accuracy and the SOTA end-to-end model only correctly completes the user request in 52. % of dialogues. Our dataset, code, and leaderboard are available
Introduction
Dialogue system modeling, classic research topic in the field of human-machine interaction, serves as an important application area [22, 21, 35, 44, 36, 34]. Recently, task-oriented dialogue (TOD) systems [14, 41, 3] have attracted significant attention. These systems are designed assist users in accomplishing specific goals, e.g., flight booking and restaurant reservation. Benefited from the previous well-designed data-collection pipeline, such Wizard-of-Oa [19], the TOD datasets [39, 2] can be constructed with low costs by writing from annotators, rather than being collected from realistic spoken conversations.
However, these TOD datasets constructed solely based on written texts may not accurately reflect the nuances of spoken conversations, leading to gap between academic research and real-world spoken TOD scenarios. To advance the research on more realistic spoken TOD, various datasets have been
Equul contribution.
i Work done while the author was working at Alibaba. Corresponning author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table 1: Dataset statistics of SpokenWOZ and existing TOD datasets dialogues in the training set.
Spoken contaain 4200
Metric DSTC2
KVRET
M2M
MultiOO
ABCD
DSTCI0
SpokenWOZ"
Dialogues
1,612
2,425
1,500
8,438
8,034
107
5,700
Turns
23,354
12,732
14,796
115,424
177,407
2,292
203,074
Domains
Single
Multi
Multi
Multi
Multi
Multi
Multi
Collection
H2M
H2H
M2M
H2H
H2H
H2H
H2H
Type
Spoken
Written
Written
Written
Written
Spoken
Spoken
Audio
Cross-turn Slot
Reasoning Slot
introduced to simulate and model spoken conversations [16, 17, 20]. Despite these efforts, existing datasets still exhibit three notable weaknesses:
Data Scale: Previous spoken TOD datasets have been limited in terms of data scale, posing challenges for developing strong and realistic systems. The pioneering ATIS dataset [16] consists of only 41 dialogues, which severely restricts the diversity and breadth of dialogues. While recent efforts, such as DSTC2 [17] and DSTCI0 [20], have renewed interest in modeling spoken TOD, they offer modest improvement with 1,612 and 107 dialogues, respectively. In contrast, written TOD datasets offer a significantly larger number of dialogues, exemplified by MultiWOZ [2]. Therefore, addressing this data scale limitation is crucial to advancing the spoken TOD systems.
Human-to-Human Audio: Another limitation is the lack of audio from human-to to-ma-t spoken conversations. DSTC2 collects the spoken audio via the Human-to-Maching schema, limiting the construction of human-like systems. DSTC10 only provides the automatic speech recognition (ASR) hypotheses and dialogue text content, limiting the investigation of the dual-modal TOD model.
Previous spoken TOD datasets focus robustness issues, especially ASR noise, ignoring the unique characteristics of spoken conversation, includ- ing word processing and reasoning in spoken language. These characteristics bring new challenges, such as handling incomplete utterances and reasoning in spoken languages.
Unique Challenges in Spoken Conversations:
Therefore, we aim at building spoken TOD dataset from human- ho-hummnn spoken conversations, named SOO.enWOZ, by extending the Wizard-of-Oz pipeline to realistic spoken conversations with real-time voice calls. To better model the spoken characteristics in SpokenWOZ, we introduce cross-turn slot and reasoning slot detection as new challenges to respectively address the dialogue state tracking in incomplete utterances and the indirect expression.
We spend more than 8 months building SpokenWOZ and implementing various quality improvements, achieving the final turn-level annotation accuracy of over 97%. The total cost of constructing SpokenWOZ. is approximate $55k, including $30k for audio collection, $20k for dialogue annotation, and $5k for the cost of using ASR tools.
In summary, contributions are threefold:
We introduce a large-scale speech-text TOD dataset named SpokenWOZZ which contains more than 203K annotated utterances, 5,700 dialogues, and the associated 249 hours of audio. To the best of our knowledge, this is the largest speech-text TOD dataset with annotation of the dialogue state and dialogue act.
We identify the new challenges in spoken conversation based on comprehensive analysis of Spoken WOZ, including cross-tumt slot detection and reasoning slot detection, which are not taken into account by the previous TOD datasets.
We also conduct comprehensive experiments on various baselines, including text-modal TOD baselines and newly proposed sual-modal models. We also explore the LLM's zero- shot performance on Spoken WOZ, e.g., ChatGPT. The results demonstrate the models still have much room for improvement in spoken conversation scenarios.

---

Preparation
Data Collection
SpokenhOZ
Audio Collaction
Pall Return
Pazza Continnue
HRY
Dialogue Audio
Pre-prepaaation
I I++
User
Agunt
Phone
Quality
Organizing
help
Annotating Process
Ewaat ta
Dallouuc State:
Hoeel-Arez: west
Training
Dialogue Bote--lllum-Ars
Quality Control
Dialogue Amnotation
Qualification Test
Diallggee
Figure 1: Construction schema of SpokenWOZ. Data collection includes (1) collection of dialogue audio and (2) annotation of dialogue. Strict quality control is performed at each collection stage.
2 Related Work
Written TOD datasets To push forward research in TOD modeling, various written TOD datasets have been proposed, ranging from
[31] [ natural human-to-human dialogues [2]. as well as from single-domain [39] to more realistic multi-domain scenarios [29]. Notable written TOD datasets include M2M [33], KVRET [11], MultiWOZ [2]. SGD [30], and ABCD [4]. Initially,
these datasets focus on the single-domain dialogue. Subsequently, MultiWOZ and SGD are proposed address more realistic molti-domain dialogue. Recently, numerous written TOD datasets attempt to simulate more realistic spoken conversations. For instance, RADDLE [27] manually introduces various numan-designed spoken noises to the written MultiWOZ. CGoDial [6] adds spoken features to existing Chinese TOD datasets via crowd-sourcing, but the language and lack of audio data both limit the research on spoken TOD. Similarly, SSTOD [43] introduces agmeen-by-segmennt interactions in simulated spoken language. Nonetheless, these studies concentrate solely on specific aspects of spoken TOD, therehy restricting more holistic examination.
Spoken TOD datasets Several datasets have been proposed for modeling the spoken TOD. ATIS [16] focuses on single-domain TOD and only contains 41 dialogues. DSTC2 [17] provides spoken corpora, but remains limited in scale. DSTCIO [20] revisits spoken TOD, providing 107 dialogues that include ASR hypotheses. EVI [37] provides transcript and ASR hypotheses. but only evaluates limited tasks, including enrolment (E), verification (V) and identification (I), ignoring more common tasks in TOD, such as querying and booking, To the best of our knowledge, SpokenWOZ WOZ represents the first large-scale speech-text dataset for TOD with annotation of the dialogue state and act.
Spoken WOokenWOZ WOZ Construction
To ensure the reliability of SpokenWOZ WOZ, we will discuss the stages involved in the construction schema in Figure 1. As shown in Table 2, SpokenWOZ consists 8 different domains, of which are inherited from widely used MultiWOZ, reducing the usage burden. Meanwhile, we design new domain "profile" to introduce realistic scenario where the agent needs to collect the user's personal information as the booking confirmation.
3.1 Collection of Dialogue Audio
To obtain dialogue audio in realistic spoken conversation, we organize 250 participants to generate 5,700 dialogues via phone calls. During the collection, one participant plays the role of user and asks questions based on a cemplate-generater task goal as shown Appendix A.7. The other participant plays the role of an agent and answers the user by searching the same database MultiWOZ, We build an online database for the agent as shown in Appendix A.8 enhance the realism of the dialogues. More details about dialogue audio can be found in Appendix A.1.4.
3

---

Table 2: Full ontology in SpokenWOZ. The upper script indicates which domains it belongs to. universal, I: restaurant, 2: hotel, 3: attraction, 4: taxi, 5: train, 6: hospital, 7: police, 8: profile.
inform" request" / ack" /select recommend nooffer / nobook 125 book edit / confirm² greet / bye / reqmore wait backchannel thanks
acts
12348 / a'araaa 123 price range 123 uddress 12367 postcode 12367 / phone 123678 / 12345 type 123 / 'internet parking" / stars" open hours / /eparturee / destination leavear arriveby ID' / email license plate number* / bookpeople 125
slots
trainID / ticket price" / booktime' department" / bbookayy | / day") bookstay!2
Qualification Test. Before the collection, conduct qualification test for each participant to ensure the quality of the data. The criteria include whether the task goal is completed and the naturalness of the audio. Each group of participants should submit completed dialogue audio, and we judge whether they pass the test. We initially receive 1520 applications, of which only 250 (16.4%) pass the qualification test.
Quality Control. We employ crowd-sourcing to evaluate the quality of each audio and remove the audio that exhibit poor communication quality did not fulfill the task goal.
Speaker Origins. To ensure dialogue diversity, we organize participants from various countries, including Canada, Singapore, China, and South Africa. More details are shown in Appendix A.1.3.
3.2 Annotation of Dialogue
We train annotators to annotate the clean dialogue state and dialogue act using both text transcrip- tions generated by the ASR tool and audio files. We inherit and expand the annotation schema of MultiWOZ, c.g., adding the new act "backchannel" to better model spoken conversations. Meanwhile, annotators need to transcribe the agent utterances according to the audio to obtain clean text. In order to keep the noise from ASR tool, we do not manually transcribe the user utterances and keep the ASR noise in the user utterances. Due to the finely adequate training (approximately 3 weeks), this annotating process was able to achieve better results.
Qualification Test. We conduct a qualification test for each annotator: The test is only passed if the test dialogues are annotated completely correct by annotators. This approach ensures that only qualified annotators could participate in the final annotation to improve the quality of Spoken WOZ.
Quality Control. Before annotation, the annotators will return the audio in low quality. To ensure the quality of the annotations, we implement a three-step quality control process, including script checking based on rules, full inspection by annotators, and random inspection by us. We use scripts to identify simple errors such as missing annotations and return the dialogues for revision. Next, annotators conduct a full inspection and correct any wrong labels. Finally, we carry out a random inspection on 10% of the dialogues in Spoken WOZ. If the correct rate of the random inspection was less than 97% at turn-level, we would return this batch of dialogues for re-inspection by the annotators. Due to the strict quality control, Spoken WOZ. can achieve much higher quality than MultiWOZ, which contains over 40% of dialogue turns that need to be modified [10].
4 SpokenWOR WOZ Dialogue Corpus
We build a novel, large-scale, human-to-human, multi-domain, dual-modal dataset that can be used to develop TOD systems on spoken conversations. SpokenWOZZ includes total of 5,700 dialogues and more than 203k utterances along with 249 hours of audio.
4.1 Characteristics of SpokenWOZZ
In this section, we will highlight the unique spoken characteristics in SpokenWOZ, including word-by- word processing, ASR noise, and reasoning in spoken language. We will use the dialogue segments
file- recognition

---

1135
Frequency (%)
Frequency
0.03
0.045
19 28 3 40 Lomaan
21 51 26 41
61 71
4.
10 15
Dialogue Length
Utterance Length
Figure 2:
The distribution of SpokenWOZ written MultiWOZ and spoken DSTCIO. The first two datasets have similar domains, and we can observe that spoken TOD tends have more turns but shorter utterances. Compared to DSTCI0, Spoken WOZ has more turns due to the cross-turn slots, and has shorter utterances because does not model the sowledge-grounder TOD as DSTC10 does.
from SpokenWOZ to illustrate these characteristics in this section. Understanding these complexities is essential for developing more realistic spoken TOD systems. Additionally, SpokenWOS has distinct distribution compared with the written MultiWOZ, as shown in Figure 2, demonstrating the difference between written and spoken TOD. More statistics are illustrated in Appendix A.5.
4.1.1 Word-by-Word Processing
Most TOD datasets only include written conversations collected via crowd-sourcing. As a result, the interactions turn-by-turn, with each turn containing complete semantics, such "I would like cheap restaurant in the north area.". ". However, language processing in spoken conversation is inherently incremental and word. The current word spoken is not necessarily well thought out and relies more on the preceding few words, such as "I would like restaurant, hnumm, cheap one please, meanwhile in the south area, oh, sorry, in the north.". ". This leads to less organized interactions, with utterances often characterized by various linguistic phenomena such as back- channel. disfluencies, and incomplete utterances.
Back-channel. Back-channel is a common phenomenon in human-to-human interaction. To effec- tively communicate with others, interlocutors often minor messages such "hmm" show their cooperation and engagement during a conversation. This can be quite challenging when one participant responds without enough semantics to follow the previous statement.
Disfluencies. Another spoken language phenomenon is disfluencies, such as repeating and pausing, which introduce noise in utterance text. The disfluencies in spoken language introduce noise, leading to increased demands on model robustness.
Incomplete Utterances. In spoken conversations, users tend to inform long information in multiple utterances rather than conveying it one utterance. The information is provided segment-by- segment over multi-turn interactions. For example, the user can tell the license plate in the following utterances, "Okay. My license plate ne", "86" and "g z w". written TOD datasets, one complete utterance contain all information, like "my license plate number is ne86gzw". How to integrate the information distributed in multiple turns is new challenge.
4.1.2 ASR Noise
In spoken dialogue systems, the ASR module is integrated to convert speech into text, but also introduces noise. Previous approaches, like RADDLE, attempt simulate ASR errors by manually writing noisy texts to the written TOD datasets. contrast, SpokenWOZ takes different approach by naturally introducing ASR noise from real-world scenarios. This is achieved by first obtaining
5

---

dialogue audio and then using an ASR module to transcribe the audio into text. The presence of ASR noise can pose challenges for the system to understand the user's intent and generate a response.
4.1.3 Reasoning in Spoken Language
Given the casual and informal nature spo- ken language, accurately determining the user's intent may require reasoning. For instance, Ta- ble 3 in SpokenWOR WOZ demonstrates scenario where the model needs to conduct reasoning to determine the total number of people involved.
Table 3:
The dialogue segment from SpokenWOZ shows the reasoning in spoken language.
Yeah. Yeah. Uh, can you book for me, parents and my grandparents?
Okay, so it's five people in total.
4.2 New Challenges of SpokenWOk WOZ
The inherent traits of spoken language pose unique challenges to TOD systems. To address this, we introduce two new types of slots in Spo- kenWOZ, including cross-turn slots and reason- ing slots, for incomplete utterances and indirect expression in spoken language, respectively.
Table 4:
The dialogue segment from Spoken WOZ shows the Cross-turn Slot Detection.
Oh, my id number is 5.5 8 ( "88" is missed by the ASR tool, but appears in the audio).
(Dialog State: od_number- = 5258)
So it's 5
4.2.1 Cross-turn Slot Detection
Yes. and then 5763.
Dialog State: number = 52585763)
In spoken conversations, users may provide the value of slot across multiple turns rather than in single turn. Each turn only provides a piece of the value, and this called the cross-turn slot. We introduce 5 cross-turn slots, including phone number, id number, user name, email, and license plate number in the domain "pro- file". More details are listed in Appendix A.1. Table 4 demonstrates an example where the user provides id number segments different turns sequentially. Differing from updating whole slot value in one turn, the agent needs to precisely locate the part of values that needs to be updated. Meanwhile, the agent should not only accumulate the slot value across the multiple turns but also make more fine-graing grained changes to the current slot value, such as correcting "525857637525" to "525857637524" by saying "I'm m sorry, 7,5 to 4".
5763.
And then 7525.
D Dialog State: id number = 525857637525)
7 5..
I'm sorry, 75 to 4.
Dialog State: id number 525857637524)
Yes. okay, it's 7524.
And then double 9 9
(Dialgg State: id_number = 52585763755299903)
4.2.2 Reasoning Slot Detection
To address the challenges presented by the sual nature of spoken language, TOD systems need to engage in reasoning effectively com- plete slot updates. SpokenWO2 introduces three kinds of reasoning, including Temporal Reason- ing [45], Mathematical Reasoning [26], and Se- mantic Reasoning [32]. More reasoning slot details can be found in Appendix A. I, includ-
Table 5: The dialogue segment from SpokenWOZ shows Temporal Reasoning.
Uh, yes, and on which day please?
Oh. yeah. And I think today is Friday, right.
Uh we will be there tomorrow. (Dialog State: Bookday Saturday)
ing the name of the specific slot and how to
construct the reasoning slots in dialogue.
Temporal Reasoning. Understanding temporal information is crucial for understanding user intent. Such as properly understanding the date of arrival at the hotel as shown in Table
Mathematical Reasoning. Once the user does not directly inform the number in the utterance, mathematical reasoning is required for the user's expression. As shown Section 4.1.3 and Table 3, the agent needs to know from the user utterance that the value of the Bookpeople slot is 5.

---

Semantic Reasoning, To capture the intent from indirect expressions, the agent needs to perform semantic reasoning based on the available information. As shown Table 6, since the user expresses a desire for sushi, it can be inferred that the purpose is to find Japanese restaurant.
Tasks & Settings
The complexity and diverse spoken characteris- tics in SpokenWOZ make it useful dataset for different TOD tasks, including dialogue state tracking and response generation.
Table 6: The dialogue segment from SpokenWOZ shows Semantic Reasoning.
Yes, ma am, the restaurant should serve sushi and should be in the center.
Dialogue State Tracking (DST). As high- lighted in Section 4, the challenges in DST in- volve understanding noisy utterances, managing cross-turn slots, and handling reasoning slots. We use the joint goal accuracy (JGA) [2] as the metric, which measures the ratio of turns for which the value of each slot is correct.
(Dialog State: Restaurant- Type = Japanese; Restaurant-Area = centre)
Just confirm that the restaurants are serving Japanese food in the centre.
That's 's correct, man.
For response generation, the challenges are twofold:
Policy Optimization. System optimizes dialogue policy and generates the appropriate response based on user utterance and dialogue state.
End-to-end Modeling. System generates the correct response solely based on the user utterance, without explicit dialogue state information.
Following the previous TOD setting [2], we report the metrics of whether the system provides correct entity (INFORM) and answers all the requested information (SUCCESS). A combined score (Comb) computed by (NNFORM+SUCCESS)).. 5+BLU [25] as an overall quality measure.
6 Experiments
6.1 Baselines
We compare text-modal baselines, which include the following models:
BERT+TripPy: TripPy [15] uses BERT [7] encoder and uses copy mechanisms fill slots for DST, including span prediction, inform operations, and slot copy.
UBAR [40] is acquired by fine-tuning the GPT-2 [28] on the sequence of the entire dialogue which is composed of user utterance, dialogue state, database result, system act, and system response.
UBAR:
GALAXY: GALAXY [13] is a pre-trained conversation model based on UniLM [9] that learns dialogue policy via semi-supervised learning. GALAXY keeps the same input and output format as UBAR to generate the dialogue state or system response.
SPACE [12] is pre-trained conversation model based on UniLM that takes into account both dialogue understanding and dialogue generation. SPACE keeps the same input and output format UBAR to generate the dialogue state or system response.
SPACE:
SPACE+TripPy: We use SPACE replace the original encoder BERT in TripPy as a new baseline.
We utilize WavLM [5] as the speech encoder and implement the following dual-modal models:
We use bi-encoder framework and then concatenate the representation outputs from SPACE and WavLM, then use Transformer [38] encoder as fusion layer to allow the interaction between the different modalities. We use the multi-modal outputs from the fusion layer as the representations for TripPy.
SPACE-Wav
M: To utilize the generation ability of SPACE and the speech-modal information, we concatenate the user utterance embeddings from SPACE and user audio embeddings from WavLM as new user-side inputs and then generate the state or response by the SPACE decoder.
7

---

SPACE
Embedding
Text-Speech Alignment
Speech Embedding
Final Embodding
Text Speechh Alignment
warna
find
hotel
SPACE
WavLM
Text
Speech
Figure 3: The architecture of MPACE+WavLM,,n It remains the same text-side inputs and formats SPACE, i.e., sequence of the entire dialogue context, but additionally introduces aligned speech- modal information of user utterances.
SPACE+WavLMr As shown in Figure 3, align the user-side word context and the corre- sponding audio pieces based on the annotations in BookenWOZD Then we add the user utterance embeddings from SPACE and the speech embeddings from WavLM During the inference, SPACE+ WayLMalign uses dual-modal input to generate the state and response.
user-side embeddings. Table 7: DST experimental results.
As LLM makes amazing progress on NLP
tasks [l], we also evaluate LLM's zero-shot performance. We use the prompts form Hude- cek et al. [18] and Bang et [1] as reference.
Model
JGA
-w/o cross-ture slot
BERT+TripPy
14.78
15.58
ChatGPT: ChatGPT (GPT-3.5-turbo) is con- versational LLM [24], which has brought re- markable success on various zero-shot tasks.
SPACE+TripPy
16.24
17.31
18.71
20.90
UBAR
20.54
23.51
InstructGPT oous: InstructGPT ous (text- davinci-003) [24] a 175B LLM trained by instruction tuning and reinforcement learning.
SPACE
22.73
26.99
SPACE&WavLM
24.09
27.34
SPACE+
25.65
28.15
Please refer Appendix for additional details and descriptions of the baseline models.
ChatGPT
13.75
16,30
InstructGPT
14.15
16.49
6.2 Results and Discussion
The results are reported in Tables 7 and 8. These tables offer several key insights follows:
SpokenWOZZ WOZ is challenging. Compared with the results in written MultiWOZ, such as SPACE achieving JGA of 57.5 in DST task and Combined Score of 110.95 in End-to-end Modeling task, the metrics are significantly lower in SpokenWOZ. This indicates that the current models learn the written TOD well, but ignore the characteristies of spoken TOD. As shown in Table all baselines get improved JGA performance without evaluating cross-turn slots, meaning modeling cross-turn slots is challenging for current models. Furthermore, we propose Macro Average Mentioned Slot Accuracy (MAMS Acc) to measure the difficulty of different types of slots in DST. MAMS Acc is calculated by (i) separately determining the accuracy of each slot, excluding instances where the none" value is present in the final turn; (ii) in each slot category. we calculate the macro average for the accuracy of each slot belonging to this category get the MAMS Acc. We divide all the slots into four categories: reasoning slot, cross-turn slot, select "name" in restaurant, hotel and attraction domains as ASR-sensitin slots, because the special entity names are difficult to correctly recognized via ASR tool. Normal slot includes slots that do not belong to the above three types. As shown Figure 4, the spoken characteristic such as ASR noise, reasoning, cross-turn slots are quite challenging and there is still much room for progress in research of spoken TOD model. In addition face their unique challenge of more diverse act flow modeling. SpokenWOZ extends the ontology of
slot, and normal slot. Then we
challenges in DST, response generation tasks

---

Table 8: Policy Optimization and End-to-end Modeling experimental results.
Policy Optimization
End-to-end Modeling
Model
INFORM
SUCCESS
BLEU
Comb
INFORM
SUCCESS
BLEU
Comb
UBAR
62.50
48.10
9.69
64.99
60.20
47.40
63.70
GALAXY
70.60
42.20
16.52
72.92
65.80
38.50
20.10
72.25
SPACE
76,00
57.60
18.72
85,52
66.40
50,60
21.34
79,84
SPACE+ +
76.80
58.40
18.54
86.14
67.20
51.30
21.46
80.71
SPACE+ WavLMaagg
77.20
59,20
19.81
88.01
68.30
52.10
22.12
82.32
ChatGPT
73.40
39.50
4.58
61.03
23.40
13.80
3.59
22.19
InstructGPPoom
78.20
56.90
7.72
75.27
25.30
18.50
6.13
28.03
MultiWOZ. which introduces the difficulties to predict the right system acts. Meanwhile, the act flow in SpokenWOZ more diverse than in written MultiWOZ as shown Appendix A.4.
Dual-modal TOD models is what you need. The significant size of the dialogues and audios in Spoken WOZ allows researchers to build large data-driven neural models given textual inputs and modll imputs. To use dual- data in SpokenWOZ, we introduce several dual-modal baselines. Dual-modal methods achieve improved performance, showing the necessity of dual-modal methods in realistic scenarios. We observe that SPACE+ SRACE+WavI.My Wavi.Matign outperforms SPACE+ WavLM due to the speech-text alignment. We show Case Study in Appendix A.3 to confirm our claim.
Supervised generative methods are helpful. We can empirically observe that generative meth- ods, e.g. UBAR, achieve better performance than extractive methods, e.g., TripPy. We fur- ther conduct Case Study in Appendix A.3. It shows that the existing extractive methods can not handle cross-turn slots and reasoning slots,
SPACE SPACE WarLM
SPaCT-WWel Marg
0.6
MAMS Acc
as the target values do not directly appear in the
utterance. Meanwhile, due to the ASR noise,
0.2
the extractive methods easily extract the wrong
value, even in the right location. However, the
generative methods can be robust to ASR noise and modify the wrong word in the original utter- ance to the correct value.
Normal
Reasoning
ASR Sersitive
Cross Tur
Figure 4: MAMS Acc of four categories of slot in advanced generative-methods.
Dialogue state is the bottleneck of LLMs. As shown 7 and 8, LLMM do not outperform
supervised methods in DST and End-to-end Modeling. But, is worth noting that L.MMs achieve comparable performances in Policy Optimization task. shows LLMS can complete the user request if provided the golden dialogue state and database results. However, when LLMS use their predicted dialogue state and database results in End-to-end Modeling, the performance is much lower. This suggests that LLM's s ability accomplish dialogue goals depends on the predictions of the dialogue state. To explore the reasons for the poor performance of LLMs, we provide an analysis in Appendix A.9. We find the main reason in DST is that the hallucination phenomenon [23] is very serious. i.e., LLMs generate erroneous results at slots that are not involved in the conversation. Meanwhile, LLMs sensitive noisy utterances, e.g.. LLMs tend directly copy the noisy word in the utterance as result, which may be due to the inability of LLMS to perceive the speech information.
Conclusion
In this paper, we introduce SpokenWOZ, a large-scale spoken Task-oriented Dialogue (TOD) dataset that incorporates both text and speech inputs. SpokenWOZ encompasses with unique characteristics of spoken conversations, including word-by-word processing, ASR noise, and reasoning. In addition, we introduce two new slot types, cross-turn and reasoning slots, as novel challenges. We further present range of baseline results to demonstrate the usability of the dataset. We hope that SpokenWOZ, with its rich, dual- dual-modal data and considerable volume, will drive the progress of spoken TOD modeling.

---

Acknowledgement
This work was supported by Alibaba Group. Meanwhile, we would like thank Prof. Milica Gasic for her appreciation and advice on our idea at the beginning of the project. We thank Dr. Bowen Yu for his constructive comments on our writing. Finally, we would like to thank all the annotators for their efforts. We look forward to our dataset advancing the research on spoken TOD.
References
[1] Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do. Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CoRR, abs/2302.04023, 2023.
[2] Pawel Budzianowski, Tsung-Hssen Wen, Bo-IIsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašić. MultiWOZ- a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages Linguistics.
Brussels, Belgium, October- Noobbr-Novemmbrr222 2018. Association for Computational
[3] Yucheng Cai, Wentao Ma, Yuchuan Wu, Shuzheng Si, Yuan Shao, Zhijian Ou, and Yonghin Li. Unipem: Universal pre-trained conversation model with fask-aware automatic prompt, 2023.
[4] Derek Chen, Howard Chen, Yi Yang, Alexander Lin, and Zhou Yu. action-based conversations dataset: corpus for building more in-depth task-oriented dialogue systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3002-3017, Online, June 2021. Association for Computational Linguistics.
[5] Sanyuan Chen. Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. Wavim: Large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Signal Process., 16(6): (505):500-1155 1518, 2022.
[6] Yinpei Dai, Wanwei He, Bowen Li, Yuchuan Wu, Zheng Cao, Zhongqi An, Jian Sun, and Yongbin Li. Cgodial: large-scale benchmark for chinese goal-oriented dialog evaluation. arXiv preprint arXiv:2211.11617, 2022.
[7] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Pupers). pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties.
[8] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, M. Zhou, and Hsiao-Wuse Hon. Unified language model pre-training for natural language understanding and generation. ArXiv, 1ts/1995.03199 2019.
[9] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. Advances in Neural Information Processing Systems, 32, 2019.
[10] Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani- Tur.. MultWWZZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 422-428 Marseille, France, May 2020. European Language Resources Association.
[11] Mihail Eric and Christopher D Manning, Key-value retrieval networks for task-oriented dialogue. arXiv preprint /705::1705.05411 2017.
[12] Wanwei He, Yinpei Dai, Min Yang, Jian Sun, Fei Huang, Luo Si, and Yongbin Li. Unified dialog model pre-training for task-triented dialog understanding and generation. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai, editors, SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 15, 2022, pages 187-200. ACM, 2022.
10

---

[13] Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang. Min Yang, Fei Huang, Luo Si. Jian Sun, and Yongbin Li. GALAXY: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Four Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposiun on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022. pages 10749-10757 AAAI Press, 2022.
[14] Michael Heck, Carel van Nickerk, Nural I Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasie. TripPy: A triple copy strategy for value independent neural dialog state tracking. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 35-44, 1st virtual meeting, July 2020. Association for Computational Linguistics.
[15] Michael Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. Trippy: A triple copy strategy for value independent neural dialog state tracking. In Olivier Pietquin, Smaranda Muresan, Vivian Chen, Casey Kennington. David Vandyke, Nina Dethlefs, Koji Inoue, Erik Ekstedt, and Stefan Ultes, editors, Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2020, 1st virtual meeting, July 1-3, 2020, pages 35-44, Association for Computational Linguistics, 2020.
[16] Charles T. Hemphill, John J. Godfrey, and George R. Doddington. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990, 1990.
[17] Matthew Henderson, Blaise Thomson, and Jason D Williams. The second dialog state tracking challenge. In Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL). pages 263-272, 2014.
[18] Vojtech Hudecek and Ondrej Dusek, Are Ilms all you need for task-oriented dialogue? abs/2304.06556, 2023.
CoRR,
[19] J. F. Kelley. An itcrative design methodology for user-friendly natural language office information applications. ACM Trans. Inf. Syst., 2(1):26-41, 1984.
[20] Seokhwan Kim, Yang Liu, Di Jin, Alexandros Papangelis, Karthik Gopalakrishnan, Behnam Hedayat- nia, and Dilek Hakkani-Tur. "how robust r u?": Evaluating task-oriented dialogue systems on spoken conversations, 2021.
[21] Yunshui Li. Binyuan Hui, Zhaochao Yin, Wanwei He, Run Luo, Yuxing Long, Min Yang. Fei Huang, and Yongbia Li. Vdialogue: A unified evaluation benchmark for visually-grunnded dialogue, 2023.
[22] Yunshui Li, Binyuan Hui, ZhiChao Yin, Min Yang. Fei Huang, and Yongbin Li. Pace: Unified multi modal dialogue pre-training with progressive and compositional experts, 2023.
[23] Benjamin Marie and Atsushi Fujita. Synthesizing parallel data of user-generated texts with zero-shot neural machine translation. Transactions of the Association for Computational Linguistics, 8:710-725, 2020.
[24] Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
[25] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
[26] Arkil Patel, Satwik Bhattamishra, and Navin Goyal, Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Longuage Technologies, pages 2080-2094, Online, June 2021. Association for Computational Linguistics.
[27] Baolin Peng. Chunyuan Li, Zhu Zhang, Chenguang Zhu, Jinchao Li, and Jianfeng Gao. RADDLE: An evaluation benchmark and analysis platform for robust taik-triented dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4418 4429, Online, August 2021. Association for Computational Linguistics.
11

---

[28] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
[29] Abhinav Rastogi, Xiaoxue Zang. Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In The Thirty-Fourrte AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secni Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8689-8696. AAAI Press, 2020.
Innovative Applications of Artificial
[30] Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The tahema--gudde dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8689 8696, 2020.
[31] Antoine Raux, Brian Langner, Dan Bohus, Alan W Black, and Maxine Eskenazi. Let's go public! taking a spoken dialog system to the real world. In in Proc. of Interspeech 2005. Citeseer, 2005.
[32] Jaydeep Sen, Tanaya Babtiwale. Kanishk Saxena, Yash Butala, Sumit Bhatia, and Karthik Sankara- narayanan. Schema aware semantic reasoning for interpreting natural language queries in enterprise settings. In Proceedings of the 28th International Conference on Computationnl Linguistics, pages 1334- 1345, Barcelona, Spain (Online), December 2020. International Committce on Computational Linguistics.
[33] Pararth Shah, Dilek Hakkami Tür, Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. Building a conversational agent overnight with dialogue self- arXiv preprint 1801044871 2018.
[34] Shuzheng Si, Zefan Cai, Shuang Zeng, Guoqiang Feng, Jiaxing Lin, and Baobao Chang. Santa: Separate strategies for inaccurate and incomplete annotation noise in distantly-supervisst named entity recognition, 2023.
[35] Shuzheng Si. Shuang Zeng, and Baobao Chang. Mining clues from incomplete utterance: A query- enhanced network for incomplete utterance rewriting, 2023.
[36] Shuzheng Si, Shuang Zeng, Jiaxing Lin, and Baobao Chang. Scl-rai: Span based contrastive learning with retrieval augmented inference for unlabeled entity problem in ner, 2022.
[37] Georgios Spithourakis. Ivan Vulié, Michal Lis, Inigo Casanueva, and Pawel Budzianowski. EVI: Multilin- gual spoken dialogue tasks and dataset for knowledge-basse enrolment, verification, and identification. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1647-1659, Scattle, United States, July 2022. Association for Computational Linguistics.
[38]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[39] Tsung-Hsien Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona Pei-Hao Su, Stefan Uhes, and Steve Young. A network-hased end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguis- tics: Volume 1, Long Papers, pages 438 449, Valencia, Spain, April 2017. Association for Computational Linguistics.
[40] Yunyi Yang, Yunhao Li, and Xiaojun Quan. Ubar: Towards fully end-to-end task-oriented dialog system with gpt-2. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14230- 14238, 2021.
[41] Yunyi Yang, Yunhao Li, and Xiaojun Quan. Ubar: Towards fully end-to-end lask-oriented dialog systems with gpt-2. AAAI, 2021.
[42] Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. MultWOZ 2.2 dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online, July 2020. Association for Computational Linguistics.
[43] Sai Zhang. Yuwei Hu, Yuchuan Wu. Jiaman Wu, Yongbin Li, Jian Sun, Caixia Yuan, and Xiaojie Wang. slot is not built in one utterance: Spoken language dialogs with sub-slots. In Findings of the Association for Computational Linguistics: ACL 2022, pages 309-321, Dublin, Ireland, May 2022. Association for Computational Linguistics.
12

---

[44] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An. Liang Chen. Zixuan Liu, Sheng Wang. Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in- context learning. 2023.
[45] Ben Zhou, Daniel Khashabi, Qiang Ning. and Dan Roth. "going vacation" takes longer than "going walk": A study of temporal commonsense understanding. Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing TEMNLP-IJCNLP), pages 3363-3369, Hong Kong, China, November 2019. Association for Computational Linguistics.
13

---

A
Appendix
A.1
Construction & Schema Details
A.1.1
Conversation Details
To make spoken conversations that close to the real scenarios, we change the following interaction pattern in MultiWOZ. In SpokenWOZ, once the user's booking is successful, the agent will provide the entity booked and ask for the user's profile information, rather than providing a reference code in MultiWOZ. Profile information including name, ID, email, license plate number, and phone. We will explain in detail when agents will actively collect profile information from users.
Name. When a user makes a successful hotel and restaurant reservation, the agent will request the user's s name the reserved information. The user's name is randomly generated by the script'.
Table 9: The 36 slots are tracked in the dialogue state.
attraction
area name / type
hospital
department
area bookday / bookpeople/
ID number. When user books train, the agent will ask for the user's ID number as registration information, which a randomly generated 16-digit string.
hotel
bookstay internet name
parking pricerange stars type
booktime bookday / bookpeople area food name pricerange
restaurant
Email. When the completes the ho- tel restaurant reservation, the agent will ask the user if she/he wants to receive the or- der via email. the user agrees to receive the order, the agent will request the user's email. The mailhox number consists of the first letter
arrivehy departure leaveat destination
taxi
arriveby departure / destination
train
leaveat / bookpeople / day
license plate number / name / ID email / phone
user's first name plus the user's last name, plus four randomly gener- ated characters, and randomly choose one of "@ gmai".com", "@ yahoo.comm', "@ "@outlook.com" "@ hotmaill.cmm as the suffix.
profile
License plate number. When reserves parking space hotel, attraction, restaurant, the agent request the user's license plate number. The license plate number string 7 random characters, first two are letters, the middle two are numbers, and the last three are letters.
Phone number. When a user successfully books taxi, the agent will request the user's phone number to contact the taxi driver, which is randomly generated 10-digit string. In another case, when users inquire about police station information, the agent will also ask for the user's phone number contact number.
A.1.2 Slot Details
The following slots tracked in the di- alogue state shown in Table 9, We also list the reasoning slot in Table 10. To control the number of cases where the value needs to be reasoned about in the reasoning slots, require participants to implicitly express the values specified in the task goal. 20% of the reasoning slot values will be automatically marked requiring implicit expression in the conversation.
Table 10: Reasoning slot in SpooknWWOZ The upper script
indicates which domains it belongs to. *; universal, restaurant, 2: hotel, 3: attraction, 4: taxi, 5: train, 6:
hospital, 7: police, 8: profile.
leaveat 4.5
arriveby
Temporal Reasoning
booktime" day
bookday 1.2
bookpcople 1,2,5 bookstay²
A.1.3 Speaker Origins Details
Mathematical Reasoning
Considering the different of access of the different countries, we chose Canada, Singapore, China, South Africa to collect the audio data, which will enable us open source the audio data. We also found that the
2,,3
area 1,2,3 internet?
Semantic Reasoning
department parking"
names: httpss//gituubbcoo///r//mmmeeee
14

---

cost of collecting audio data from Canada and Singapore is about three times that of collecting from South Africa. Therefore, within the same budget, chose to collect audios from South Africa, and believe that a larger data set would further prompt the research the community. The distribution of speaker origins are shown in Table II.
A.1.4 Audio Details
Our audio files are two-track. One track represents the voice of the user and the other represents the voice of the agent. Meanwhile, the sample rate of our audio files is 8000Hz. Each dialogue corresponds to an audio file, and each word is recorded in the text annotation corresponding to the word context, start time and end time. To avoid the problem of overlapping utterances, we follow the rules below during the collection: (i) prohibit the agent from using the backchannel to interrupt the user; (ii) when a user uses backchannel expression, the agent should respond to the backchannel correctly, rather than ignoring it and continuing the previous utterance. Finally, the word error rate of ASR is 6.1% calculated from the manually modified agent utterances and the agent utterances recognized by ASR tool.
Table 11: The origins diversity of SpokenWOZZ Participants come from four different countries to improve the diversity of spoken conversations.
Country
Dialogues
Percenlage
Peuple Percenlage
Canada
500
8.77%
24%
Singapore
500
8,77%
40
16%
China
2100
36.84%
30
12%
South Africa
2600
45.61%
120
48%
A.1.5 Data Division
Table 12: Statistics of SpokenWOZ
SpokenWOZin is divided into 4200/500//000 dialogues in order by train/dev/test. More details can be found in Table 12. The results of experiments are evaluated by the test set.
Dataset
Train
Dev
Test
Audio Hours
183
22
44
Dialogues
4,200
1000
Turns
149,126
18,384
35,564
Tokens
1.672.984
204,644
396,933
A.2 Experiment Details
Avg. Tums
35.50
36.77
35,56
Avg. Tokens
11.21
11.13
11.16
A.2.1 DST Baselines
TripPy makes use of copy mechanisms to fill slots. A slot is filled by one of threc copy mcchanisms, including: (1) span prediction: values are dircetly extracted from the user's utterances; (2) inform operations: a value may be copied from the system's inform operations; (3) slot copy; a value may copied over from a different slot.
BERT+TripPy
SPACE+TripP?
We use SPACE to replace the original encoder BERT in TripPy. SPACE a semi-supervised pre-trained conversation model learning from argeescale dialogue corpora with limited annotations, which can be effectively fine-tuned on different downstream dialogue tasks.
SPACE+ WaiLy+ To use both speech and text data, we concatenate the embeddings from SPACE and WavLM. Then we use Transformer encoder as the fusion layer to allow the interaction between the different modalitics. Then use the fused outputs as the representations in the TripPy.
UBAR UBAR is acquired by fine-tuning the GPT-2 on the sequence of the entire dialogue session which is composed of user utterance, dialogue state, database result, system act, system response every dialogue turn. During the inference time, it formulates DST as a sequence-to-see to-sequence task. It takes the current user utterance, dialogue history, and the previously predicted dialogue state as input, and gets the dialogue state of the current user utterance.
SPACE is a semi- semi-supervised pre-trained conversation model learning from large-scale dialogue corpora, which is based UniLM [8]. Such UBAR, use SPACE as pre-trained language model to fine-tuning on the sequence of the entire dialogue session. During the inference, give the history and user utterance, SPACE generates dialogue states by autoregression. SPACE uses the same special token as UBAR to split user utterances, dialogue state, act and system response.
SPACE
To utilize the generation ability of the SPACE model and the speech- modal information, we concatenate the user utterance embeddings from SPACE and user audio embeddings from WavLM as new user-side inputs. During the inference, the model uses dual-modal inputs generate the state by autoregression.
SPACE+ WavLM
15

---

WPACE+WayLMMaiie Using the annotations in SpokenWOZ, the text of a word can be aligned with its audio segment. To further explore how make full use of the speech information. align the token and audio segment of every word user utterances. embeddings from WavLM as new user-side embeddings. During the inference, the model uses sual-modal input to generate the dialogue state by autoregression.
we the text embeddings from SPACE and corresponding
ChatGPT: ChatGPT (gpt-3.5-turbo) is conversational LLM that has been trained by reinforcement learning
instruction tuning [24]. demonstrating a surprising ability completing conversations.
InstructGPT Toos
InstructGPPons (text-davinci-003)) [24] a 175B LLM trained by reinforcement learning with human feedback and instruction tuning.
A.2.2 Response Generation Baselines
UBAR UBAR fine-tum tunes the GPT-2 on the sequence of the entire dialogue, including user utterance, dialogue state, database result, system act, and system response. During the inference, UBAR uses the fine-tuned GPT-2 generate responses given different inputs based different task settings.
GALAXY GALAXY pre-trained dialogue model that explicitly learns dialogue policy from limited labeled dialogues and large-scale unlabeled dialogue corpora via supiisupereised learning, which based on UniLM, GALAXY keeps the same input format as UBAR.
SPACE SPACE is pre-trained dialogue model that benefiting from large-scale dialogue corpora via multi- task learning, including dialog understanding module, dialog policy module dialog generation module. SPACE is based on UniLM. SAPCE keeps the same imput format as UBAR.
We concatenate the user utterance embeddings from SPACE and user audio embeddings from WavLM new user-side inputs. During the inference, SPACE4WavL.M uses dual-modal input to generate dialogue state, act, and final system response by autoregression.
PPyCE+WayLMs
SPACE+ WayLMaiigned SPACE+ MAACE+WavLAAe WavLMallgned adds the text embeddings from SPACE and the correspond- ing embeddings from WavLM new user-side embeddings. During the inference, the model uses dual-modal input generate dialogue state, act, and final system response.
A.2.3 Hyperparameters
For text-modal methods, we use the code and hyperparameters provided by their respective papers. For dual- modal methods, ues the same hyperparameters as text-modal methods. To the fair comparison, we train all the basclines 10 epoch for DST, 25 epoch for response generation and use the final epoch checkpoint get the results on SpokenWOK WOZ. The results we report are the average of the results using five different seeds. We trained the baselines in NVIDIA A100 and V100.
A.3
Case Study
We will show the predicted cases to confirm our insights proposed in section Experiments.
Supervised generative methods helpful. We give the comparison between the cenerative-methor UBAR seractive-methor BERT+TripPy. Extractive methods can not get the value it does not directly exists in utterance, such as the reasoning slot. Meanwhile, the generative-men methood can be robust to ASR noise and modify the wrong word in the utterance to the right one.
Table 13: The Case shows that supervised generative methods are helpful.
There train, the id is called tr8925, do you make a booking. Yes, please make booking for me.
Okay. How many people?
Um let me think ah it's me and have six friends with BERT+TripPy: Train-Bookpeend = none)
(UBAR: Train- Brain-Bookpeople =
16

---

The value 7 of slot Bookpeople can be correctly predicted by UBAR. However, value 7 is not existing which is predicted wrongly by B
Table 14: The Case shows that supervised generative methods are helpful.
Uh, I'm looking for particular hotel in cambridge.
Okay. So may I know its name, please?
Um let me check. I think the name is called lavelle lodge.
(BERT+Tripyy: Hotel-Name lavelle lodge)
(UBAR: Hotel-Name lovell lodge)
In this case, the correct name of the hotel " lovel lodge" is not predicted correctly by BERT TripPy, even it extracts the correct span in utterance. We also find that UBAR get the name correctly, which shows that generative-method can learn the ability correct errors from ASR.
Table 15: The Case shows that supervised generative methods are helpful.
My id is 8
(BERT+TriPPy: Profile-ID None) (UBAR: Profile-ID= = 88716)
You can proceed.
46859 9.
BERTTT Profile-ID None) (UBRR: Profile-ID = 8871 1646859)
am still working with.
6 8 1 1.
(BERT+TripPy: Profile-ID None) (UBAR: Profile-ID = (88166663963)111
senerative-metheed can learn ability to concatenate the value segment from different turns, which can be hardly learned by the extractive-methed BERT + TripPy.
Dual-modal TOD models is what you need. We give the case study and comparison between text-modal methods SPACE TripPy, SPACE and dual-modal methods SPACE TripPy WavLM, SPACE WavLM. Although the main experimental results reflect that speech information improves overall performance, more concerned with the performance the ASR-sensitive slots.
Table 16: The Case shows that dual-mode TOD models is what you need.
Good afternoon. Yes. Uh, could you please assist me looking for particular restaurant, please.
No problem. Do you have name for a restaurant? Yes. um, it's called the bangkok cutting.
Bangkok city, okay. Just give me second. I'll look for it on the system.
Okay, not problem at all.
SPACE+TriPPy: Restaurant- Name bangkok cutting)
Restaurant-Name bangkok city )
(SPACE: Restaurant-Name = bangkok city)
SPACE+WWvyLM Restaurant-Name bangkok city) )
Name is not correctly predicted by SPACE+TripPy. We find span prediction copy mechanism is performed in SPACE+TripPy However, PPACE-WayLM-TTippP performed the copy mechanism and copy the value from Inform Operations to get right value. This indicates that the speech information can be used to further improve performance.
this case, value
17

---

Table 17: The Case shows that dual- TOD models is what you need.
want find particular hotel to rise. I remember his name, but can't find the location.
So may have the name of the hotel, please.
Uh. the name of the hotel is work worth house.
Okay, I found hotel called warkworth house for you.
Uh, yes, ycs. That's ' the hotel. Thank you.
SPiPP+TripPy: Hotel-Name None)
Hotel-Name work worth house)
(SPACE: Hotel-Name worth house)
(SPACE+WavLM: Hotel-Name warkworth house
In this case. SPACE can not predict the value in the right pattern, but dual-modal SPACE+ TriMMr successfully predict it. This shows that speech modality also help generative methods learn the correct pattern, even in the presence ASR noise from user utterances.
A.4 Heatmap of acts
We show the act flow in SpokenWOZ, in Figure 6. Given the user dialogue present the frequency of agent heat map. As shown in Figure and 6, SpokenWOZ not only contains more types of acts, but also contains more diverse act flow. more difficult for the model to predict the right action and give the right response.
grawi
55
82
roquest
inform
2736
2444
8238
2325
4786
114
2013
1136
123
3151
28463
13681
919
thank
16
1500
76
4054
55
7143
404
242
686
906
74
nootfer
offerbooc
offerbooied nobook
recommend
Figure 5: Heat map of agent acts in MultWWZ. The heat map shows the frequency of the agent act (horizontal axis) after the given act (vertical axis).
019et
1387
12
87
40
45201
8b
25
123
146
7540
confrm
12
16
41
404
285
27
backchannd
24
28
119
26
15
40
201
211
Zye
2012
thanks
314
53
3184
90
1834
46
2892
255
183
1005
864
70
noofftr
select
roomore
book
welcome
nobock
bye
ack:
recommend confim
Inform
roquest
greet
Figure 6: Heat map of agent in SpokenWOZ. The heat map shows the frequency of the agent act (horizontal axis) after the given user act (vertical axis).
18

---

A.5 Statistics
We give the distribution of domains in Table 18. Meanwhile, the dataset distributions of dialog length and turn length shown in the following figures. We give the statisties of SpokenWOZ Figure and Shown in Figure 7, the length dialogue history is concentrated above 30 turns. The excessive number of dialogue turns also makes it difficult for the model learn. We compared the multi-domain and single- domain dialogue Figure 9, intuitively, the number of turns for multi-domain dialogue larger than the number of turns for single-domain dialogue. In Fig- 10, there is significant difference between the utterance lengths of user and agent, because Spo- KOn is constructed using Human-tov tt--nannn schema. We also show the distribution of the dialogue acts and slots in Figure 11.
Table 18: The distribution dataset domains.
Domains
Number
profilerrettuurat-train
720
hotel- finel-profile--o profile-train
702
attraction hotel profile- taxi
295
hotel profile restaurant-taxi
dutraction-prffllee
291
profile-taxi
285
attraction-train
278
duraction-profiin
profi----eeetttaattt--e-ee-- tastsaurann--ax
275
hotel-profile-restaaeee restaurant
profile-restaurant
238
attraction
238
hotel-profile
237
Inraction-hotee profile
altractioon-roofle-reess
profile-train train
193
149
hotel-train
148
restaurant
132
hotel
104
restaurant
102
iitractionrestaurrat
85
attraction-hotel
62
hospital
police-profile
56
attraction-proole proffle
47
hotel-restaurant
39
Total
5700
0.08
Frequency (%)
0.06
Dialogue Length
Figure 7: The distribution of the length of turn in Spoken WOZ.
19

---

0.1
0.00
Frequency (%)
0.00
0.04
0.02
33
25
27
Utterance Length
Figure 8: The distribution of the length of turn in Spoken WOZ.
0.12
Frequency (%)
0.06
0.02
Dialogue Length
Figure 9: The distribution of the number of turns in two kinds of dialog in Spoken WOZ: Multi- domain, Single-domain.
20

---

0.1
Frequency (%
0.02
19 15 17
23
25
27
29
Utterance Length
Figure 10: The distribution of the length of turn in two kinds of dialog in Spoken WOZ: User, Agent.
destnattion
Cmall
bookday
departure
Dhonerummer
arriveyy
leaveat
idnumber
name
dernttation
Bookiaay
phonenumbor
departure
request
ernvetiy
email
Inform
idnumber
ack
inform
request
brain
profile
name
bookday
attraction
name
booknnlople
Inform
hotel
restaurant
Inform
pricerande
request
interne
ack
Inform
inform
ack
food
pncerrngge
booktime
bookday
area
g00)
Figure 11: The distribution of domains, dialogue acts and slots in Spoken WOZ.
A.6 Dialogue Example
A dialogue example be found in Figure 12.
21

---

Helo
1||||
Hi, hello. how may help you
l10.
Yes, I'm looking for 18S and I'm leaving binmingham now street to Cambridge. Can you help?
Yes, can help you find do, uh, du you
1|||-
9.9
leave.
Ib].
I'd like to leave an the day befone thursday and I warii to
Jeave after 18:50 the evening.
|||-
P.
Olay, so 2's en sodaasdaz and then leave after 06:15 in altemoon. Yes, there train. Do you want to book a licket?
Yes, faschook_is
parents and four other siblings..
-
909
Okay, so there will be total people of you.
l
Yos, plense.
Oksy. have your number the, uh,
1111-
ticket booking
is 2
1111--
6.6.666
4. 9.
90
4.9.1.
=||1-
909
l/1--
d.dcubbe
$.5.2. suys.
Oksy, already booked the tickets for
1111-
11|-
Yas
Uh, still need anything else?
-||1--
P.9
|
Yos, also I'm looking for place to dic and want have it the center the city. Do you have any recommendation:.
Yes you
speciñio food kind you and shout price runge?
-|0--
P.O
Well, really try Asian oriental food maybe not that expensive, modemtte it's fine.
Oiny let there noscle called noodle they serve pretty good
111--
P.9
food.
l
Okay. May have your address, please.
Oh yes and the address will 4042 king
city peniler.
|.--
Okay, thanks. think that's all.
Okay. you want have your email
P.9
confimmatinn for the train booking..
|
Wow, Ulanks.
right
1|11-
P.9.
Okay. Yeal. you any thing elae?
I|..
No, that's
you
Okay, thanks for culling bye bye.
Ctoss-tum Slot
Reasoning Slot
Yerbafized Expressions
ASR Frront
Figure 12: A dialogue example from SpokenWOZ.
22

---

A.7 Online Database
We give the interface of our built datahase for participants in Figure 13.
Data
Infor
1r10
sDart
350
c3r1ds
charbrrion
Uibar now bed Lnvat
123005755
59120
Nimbal
23136
D17s cal thant
103011625
edllaroot
993064619
puarticcuss narts
1M
c3d3pd
seriica
Figure 13: The built online database in Spoken WOZ.
A.8 Task Goal Example
We give example of task goal for participants in Table 19.
Table 19: An example of task goal.
(Tips!) The important information marked Message veiled expeession) veiled expression needed here.
(Attention!) customer information try your problem
turns
(Background) Your name <Misty Imberts, your telephone is <93107291300 330s, your number <84851140014699133 email <MImbent06ja @ outlook. outlook.com.. comp. your number <OLDSODU>
(Background) You are planning
in Cambridge
You are looking <place stay>, The hotel <doesn't need have free parkings and should <include free will>
The should
have
parkings you find chotels you want
people>> and nightss
starting from <friday>
You also looking for <restaurant>. The restaurunt should be emndeealees> price range and should <indiant> food
The should
hotel>
Once you crestaurants you want
same group people> Souus veiled expression)> <the same day>
(Background) Once you have made booking, you do not want to give out your email address for receiving orders.
23

---

A.9 Analysis of LLMs
A.9.1 Prompts for LLMs
We imitate the format of prompt form Hudecek et al. [18] and Bang et al. [1]. We list prompt example for DST task in Table 20.
Table 20: An example of zero-shot version of the prompt used for DST.
Definition: Give the dialogue of the last utterance the following dialogue JSON (for example: STATE:
"hotel-parking:: "yes". "hotel-type:: "guest house") by using the following pre-delined slots and possible values:
Slot Name: priee--ricerange: Sloe Deserption: price hudget of the botel; Possible values: ('expensive', "cheap', 'moderate']
Slot Name: hotel-type; Slot Descrption: type
botel: Possible values: I' l'guest houss', 'hotel'] Slot Name: hotel-parking: Slot Descrption: whether the hotel has parking: Prssible values: ["no', 'yes']
Slot Name: hotel-day; Slot Descrption: day of the hotel booking: Possible values: I'mondny', 'tuesday'. 'wodnesday', 'thursday'. - 'triday.. 'saturday,, "sunday') I
Slot Name: hotel- hotel-people; Slot Descrption: number of people booking the botel: Possible values: ['1', '2', "3", 14, "S", '6', '.. '8']
Slot Name: hotel-stay; Slot Descrption: length of stay at the hotel; Possible values: [",,, '2', '3", '4', '5", '6', '7', '8']
Slot Name: hotel-intemet. Slot Descrption: whether the botel has the free internet; Possible values: l'no', 'yes'l
Slot Name: hotel-name; Slot Descrption: name of the hotel, Possible values: []
Slot Name: hotel-area: Slot Deserption; area of the botel: Possible values: I'centre', , 'cast", 'north'. "south", 'west']
13", '4", Skot Name: train arriveby; Slot Descrption: the arrival time of the train, 24 bour standard time, e.g. 06:00, 18:30; Possible values: []
Name: hotel-star; Slot Descrption: star
Possl.Possible values: I'0',
Slot Name: train-day: Slot Deserption: day of the train departure: Possible values: [' I'monday', 'tuesday', "wodnesday", "thursday.. "triday'. 'saturday'. : 'sunday"] ]
Slot Name: train-poople, Slot Deserption: number people travelling by train; Possible values: [',, 2", 22,33"," '5,'5",666 '?'. '8']
Name: neavete Slot Deverption: leaving of the train. 24-hour standard time, e.g. 06:00, 18:30:
Possible values: []
Name: miindestiination; Descrption: destination the train; Possible values: ('hirmingham new street',
"bishops stortford', 'broxbouree' 'cambridge', 'ely', "kings lyan'. 'leicester', 'london kings cross', 'london liverpool street'.
'norwich', "peterborough', ; 'stansted airport'. 'stevenuge"l I
Slot Name: train-departure; Slot Deserption: departure
train: Possible values: l'birmingham new street'.
'bishops stortford',
bruxbwurne' 'cambridge, 'ely'. 'kings lynn', 'leicester,, : 'london kings cross", 'london liverpoul street', 'nuewich". ppeerborouuhh; 'stansted nirport', 'stevenage']
Slot Name: attraction-area; Slot Desorption: area the attraction; Possible values: I'centre', , 'cast', 'north', 'south", 'west']
Slot Name: sttraction-name: Slot Descrption: name of the attraction: Possible values: II
Slot Name: altraction-lype: Slot Descrption: type of the altraction; Pussible values: I' l'anchiieccture, "boal". 'cinema', 'college', "concerthall', 'entertainment. 'museum'; 'multiple sports', 'nightclub', 'park', "swinmingposl" 'theatre'] Skot Name: restaurant restaurat-priccennge: Slot Descrption: price budget for the restaurant: Possible values: I' I'expensive', "cheap". 'moderate'l
Slot Name: restaurnnt-aren Slot Deserption: area of the restaurant; Possible values: ['centre', "east', 'north', "south", 'west"]
Slot Name: restaurant- Slot Descrption: the cuisine of the restaurant; Possible values: []
Slot Name: restaurant name: Slot Descrption: name of the restaurant; Possible values: II
Slot Name: restaurunt day; Slot Deserption: day of the restaurant booking: Possible values: ('monday'. , "tuesday". 'wednesday', 'thursday, , 'riday', 'saturday', "sunday'] ]
Skot Name: restaurant people; Slat Descrption: number of people for the restaurant booking; Possible values: ["T', '2", "3',
4", '5", '6','', '7', "7'
Slot Name: restaurant-time Slot Desorption: time of the restaurant booking, 24-hour standard time, e.g. 06:00, 18:30;
Possible valuea: []
Slot Name: depurtmeetprttmett:: Slot Descrption: department of the hospital; Possible values: II
Slot Name: taxi lexvelt Slot Deserption: leaving time of taxi, 24 hour standard time, e.g. 06:00, 18:30; Possible values: []
Slot Name: txxi-destination; Slot Descrption: destination taxi; Possible values:
Slot Name: taxi departure; Slot Deserption: departure location of tasi: Pussible values: []
Name: tuxi-aniveby; Slot Descrption: anival time taxi, 24-hour standard time, e.g. 06:00, 18:30; Possible values: []
Slot Name: profile-name; Slot Desciption: the name of the user. Possible values: []
Slot Name: protile-email; Slot Descrption: the email of the user. Possible values: II
Slot Name: profile kdnumberr; Slot Descrption: the idnumber of the user, Possible values:
Slot Name: phonlle-phonnmmmeer Slot Deserption: the phonemumher of the user; Prosible values: []
Slot Name: poffle-platnuumber Deserption: platenumber of the user, Possible values: []
USER: Hello, need some help.
SYSTEM: Okay, how can help you.
USER: Yes, I'm looking for train.
STATE:
For response generation tasks, we follow the prompt as Hudecek et al. [18]. We use the dialogue state to query the database provided to get the entity. For the Policy Optimization task, we use the ground truth dialogue state to query the provided database. For Sund-toend Modeling, we use the predicted dialogue state generated from ChatGPT or InstructGPTog to query the provided database. We the contents of the database and the
24

---

generated system response to match the strings and complete the de-lexicalization process. The de-lexicalized response will be used to calculate INFORM and SUCCESS. We list a prompt example for response generation task in Table 21.
Table 21: An example of a zero-shot version of the prompt used for Response Generation.
Please continue the dialogue as a lask-oriented dialogue system called SYSTEM. The answer of SYSTEM should follow the DATABASE provided in ISON format and answer the USER's last utterance. SYSTEM can recommend and inform the contents in the DATABASE according
Definition:
utterance of the USER and retur name of the entity when and the trainid when il comes trains, But only when the USER requests information about an entily the DATABASE, such as phone number, should SYSTEM inform the comesponding content.
restaurants, hotels and attractions,
DATABASE:
[ "restaurani": :
"midress": "Cambridge Retail Park Newmarket Road Fen Ditton,, "area": "emst", "Toud": "italian,, "name"" : "pizza hut fen ditton". " "phone": "12235237570", "postcods": "cb58wr", "priocrango": "moderate") ]
DIALOGUE CONTENT:
USER: Hello,
SYSTEM: Hello! How can I assist you today? Are you looking for any specific information or rocomeendation? I'm in east part city and
some food. hope to serve food moderate price range.
SYSTEM:
A.9.2 Analysis on DST
As shown in the section Experiments. the performances of LLMs do not surpass that of supervised small models and show noticeable gap compared the supervised generative dual-modal baselines the DST task However, it is worth noting that LLMs slightly outperform the BERT+ when cross-turn slots are not taken into account, showing the potential capabilitics of LLMs. Mcanwhile, we find that the main reason for the poor results of LLMs is that the hallucination phenomenon [23] is very scrious, c.g., LLMs often generate additional results that do not the dialogue utterances. When we measure only the slots where the ground truth value is not "none", we find that ChatGPT achieves JGA of 30.81 and InstructGPTone achieves JGA of 34.42. Compared to the standard JGA, ChatGPT and InstructGPPoor improve their scores by 17.06 and 20.27 respectively. This indicates that LLMs generate a large number of erroneous results at slots that are not involved in the conversation, i.e., the hallucination phenomenon very serious. We show in Table 24.
Table 22: The Case shows the hallucination phenomenon.
Hello.
Hello, how can help?
Yes, I'm looking for restaurant.
CChatGPT: Restaurant-Food = international)
(Ground Truth: Restaurant- Food none)
Meanwhile, due to the inability of LLMs to perceive information speech, LLMs tend to generate the value directly from user utterance. We show a case in Table 23.
Table 23: The Case shows that LLMs are sensitive to the noisy utterance.
Uh, I'm looking for particular hotel in cambridge.
Okay. So may I know its name, please?
Um let me check. I think the name is called lavelle lodge.
(ChatGPT: Hotel-Name = lavelle lodge)
(InstructGPTanc Hotel-Name lavelle lodge)
(Ground Truth: Hotel-Name = lovell lodge)
A.9.3 Analysis on Response Generation
As introduced in the section Experiments, LLMs achieve comparable performances in Policy Optimization task but poor performances in the End-to-dn -one-toonddddd Modeling task. We find that the main reason for the poor results in End-to-end Modeling is that the entity returned by the database does not meet the user's needs. Mcanwhile.
25

---

the poor perfornances of BLEU show that there is big difference between LLM's 's response style and human response style. Meanwhile, LLMs may feel confused about noisy utterances and generate statement requesting clarification as shown Table 24.
Table 24: The Case shows that LLMs is sensitive to the noisy utterance.
Mm. I'm looking for place to die (In the audio it is actually "dinner"). ChatGPT: I'm sorry, I'm not sure I understand. Could you please rephrase your question?
B
Limitations
Even though we tried hard to build realistic spoken TOD benchmark for further studies, we could not audio data from real conversations due to privacy concerns. This brings the following limitations: (1) For "profile" domain, we use a designed script to generate random personal information, which may not be realistic, and the number of slots also limits the further in scenario that agents need to collect personal information from users; (2) As the limited ontology, our benchmark should mainly be used for research instead of deployed in realistic applications.
C Ethics Statement
We construct SpokenWOZ, a task-oriented dialogue benchmark containing both audio data and text data, and we will detail our ethical considerations for each part of our collection process:
Ontology Consideration. We inherited and expanded MultWWO''s s ontology, which is open-source and under the MIT License. We have used it in compliance with its terms of use.
Privacy Concern. In our dataset, we have designed the scenarios where an agent needs proactively collects user information, however, user information is all generated by scripts in random manner, there will not be any privacy leakage issues.
Audio Collection. We informed each participant that the collected audio data will be used as public dataset for research. Participants who agree to participate in data collection will sign a contract with us, and the ownership and use rights of their data belong to us. We will not disclose which specific participant the audio came from. At the same time, we have reviewed the legal regulations of four countries and regions, and will release the data in legal manner, this will cause any legal problems. The distribution of our data sources has been discussed the Appendix A. 1.3, the diversity of Spoken WOZ makes our data unbiased. We pay $30k 249 hours audio. The average cost per hour of audio is $120.
Dialogue Annotation. During the annotation process, the annotators' personal information is not collected, which will not cause privacy leakage. At the same time, during the annotation process, we signed non disclosuuuee agreement with each annotator, therefore, the audio data will not be leaked during the annotation process. We pay $20k for 5,700 dialogues. The average cost per dialogue annotation is $3.5. After our statistics, average of one hour can annotate dialogues.
D
Data Format
D.1 Audio Format
As detailed in Appendix A.1.4, audio files are two-track with sample rate of 8000Hz. One track represents the voice the user and the other represents the voice the agent. Each dialogue corresponds to an audio file, and each word is recorded in the text annotation corresponding to the word context, start time, and end time. We use the wav format to save our audio files. The file name of the audio is consistent with the id of the dialogue, for example, the corresponding audio file for MUT.0032 is MUL.003.wav.
D.2
Text Format
Our text data is given in json format, and we take the same fields as popular MultiWZZ 2.2 [42] to store the corresponding information, so researchers ontology and database json files. There are 5,700 dialogues ranging form single-domaie in SpokenWOZZ The test sets contain 1k examples. Dialogues with MUL in the name refers to multi- domti- dialogues. Dialogues with SNG refers to mingle-domin dialogues. Each dialogue consists of goal, multiple user and system utterances, dialogue state, dialogue act, corresponding audio and ASR transcription.
easily use our data. addition, we additionally provide the data
multi-doman
26

---

The dialogue goal for each dialogue is recorded in the "goal" field. The dialogue goal holds the fields involved in the dialogue well the slots involved and the corresponding values.
The dialogue state for each dialogue recorded in the "metadata" field in every the MuluWOZ 2.2. The state have two sections: semi, book. Semi refers slots from a particular domain. Book refers to booking slots for a particular domain. The joint accuracy metrics includes ALL slots.
The dialogue act for each dialogue is recorded in the "dialogue_ac" act" and "span_info" field in every tum:
" "Sdialogue id
"log :1
Sturr_ _dur
dialogue_act"
Sact name": [
Sslot_name",
"acction_vall value"
}
span sran_info":
Sact name
$slot name".
Saction value"
The ASR transcription for cach dialogue is recorded in the "words" field in every tum.
"ddialogue id "log 1
"Sturn_ii id
wordss" ";
"evord_context"" "Sword", Sbeein_tiim time "Sbegintime",
end uad_timee: "Sendtime",
channel_id id": "Schannei", word_index"; "Sindex",
E Datasheets for Spoken WOZ
E.1 Dataset documentation and intended uses
For what purpose was the dataset created? Was there specific task in mind? Was there specific gap that needed to be filled? Please provide description. Task- dialogue (TOD) models have made significant progress in recent years. These systems are designed to assist users accomplishing specific goals, e.g., flight booking and restaurant reservation. However, these TOD datasets constructed solely based on written texts may not accurately reflect the nuances of spoken conversations, leading gap between academic research and real-worle spoken TOD scenarios. We introduce the common tasks of TOD, including dialogue state tracking, policy Optimization, and End-to-end Modeling.
Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., 15 company, institution, organization)? This dataset is created by researchers at Alibaba Group, Renmin University of China, and University of Michigan.
27

---

Who funded the creation of the dataset? Alibaba Group.
The creation of dataset was funded by DAMO Academy.
Any other comments?
N/A
E.2
Composition
What do the instances that comprise the dataset represent (e.g,, documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description. The dataset contains text data and aduio data of spoken task-oriented dialogue. For each dialogue texi, we also annotate both the dialogue state and dialogue act. For the dialogue audio, we also give the ASR transcription and audio file.
How many instances there in total (of each type, if appropriate)? SpokenWOZ contains 8 domains, 203k turns. 5.7k dialogues and 249 hours audios from spoken conversations.
What data does each instance consist of? "Raw" data (e.g,, unprocessed text or images) or features? either case, please provide a deseription. For a conversation data, includes text part and audio part. For cach dialogue text, SpokenWOZ contains dialogue context, annotated dialogue state and annotated dialogue act. For the dialogue audio, SpokenWOZ contains corresponding audio data and ASR transcription.
Is there label or target associated with each instance? so, please provide a description Yes, we inherit and extend the MultiWOZ, annotation schema, which is widely used for task-oriented dialogue.
Is any information missing from individual instances? If so, please provide a description, explaining why this information missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g,, redacted text. For individual instances, there is no missing information.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit. Each dialogue in SpokenWOZ is relatively independent, and the domains involved are different.
If so, please provide a description of these splits, explaining the rationale behind them. We give the data splits in Appendix A.1.5. The data is split into training, development, and unrelcased test sets. Once researchers have the dev set, they can submit it
Are there recommended data splits (e.g., training, ewvelopmen//vaiddattioneetttt
built model that works to your expectations hidden test set. To mitigate the misestimation of the generalization error of the model, we increase the number of test set to 1000 dialogues. At the same time we keep the training and test data domain distributions roughly, but not exactly, the same.
get official scores the
Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description. Since dataset requires manual annotation of dialogue state and dialogue act, annotation noise inevitably introduced, the same time the dialogue audio collection there are cases substandard audio quality, such as low communication quality. As shown section Spoken WOZ Construction, strict quality control is performed at each collection stage.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? SpokenWOZZ is self-contained.
Does the dataset contain data that might be considered confidential (e.g,, data that is pro- tected by legal privilege by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? so, please provide a description. No.
Does the dataset relate people? If not, you may skip the remaining questions in this section. No. Detailed in Ethics Statement, we have designed the scenarios where an agent needs to proactively collects user information, however. our user information is all generated by scripts in a andom manner, so there will not be any privacy leakage issues.
28

---

E.3
Collection process
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), indirectly inferred/derived from other data (e-g., part-of-speech tags, model-based guesses for age or language)? data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If' so, please describe how. We report the construction schema of SpokenWOZ the section SpokenWOZ Construction.
What mechanisms procedures were used to collect the data (e.g,, hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated? We organized 250 particinants to generate 5,700 dialogues via phone calls. The details of the audio file can be found in Appendix A.1.4, The dialogue state and dialogue act are annotated using the Appen platform".
If the dataset is a sample from a larger set, what was the sampling strategy (e.g,, deterministic, probabilistic with specific sampling probabilities? SpokenWOZ not sampled from larger set.
Over what timeframe the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e-g., recent crawl old news articles)? If not, please describe the time-frame in which the data associated with the instances was created. Our data collection starts in July 2022. The contents of our data instances are independent of the time of collection.
Were any ethical review processes conducted (e-g., by an institutional review board)? If so, please provide description of these review processes, including the outcomes, well link or other access point to any supporting documentation. Not applicable. We consider these contents Ethics Statement.
Does the dataset relate to people? If not, you may skip the remainder of the questions in this section. No.
E,4 Prepooesinn/deaainn/kbliig
Was any preprocesinn//eennn///aeee tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? ? If so, please provide a description. If not, you may skip the remainder of the questions in this section. We report the construction schema of SpokenWOOZ in the section SpokenWOZ Construction.
of the data done (e.g., discretization or bucketing,
Was the "raw" data saved in addition to the preppocesed/cleane//abel data (e.g., to support unanticipated future uses)? If so, please provide link or other access point to the "raw" data. Raw data was not saved to prevent misuse. We will only open source the cleaned data.
Is the software used to preprocess/cean//lbel the instances available? If so, please provide link or other access point. We have used Python language implement data cleaning. We will share the scripts details in our codebase.
E.5
Uses
Has the dataset been used for any tasks already? If so, please provide description. The complexity and diverse spoken characteristics SPO make it a useful dataset different TOD tasks, including dialogue state tracking and response generation. For response generation, the challenges are twofold: Policy Optimization and End-to-end Modeling. More details can be found in section Tasks & Settings.
Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point. We provide links to the papers of all the baseline models the leaderboard' built.
What (other) tasks could the dataset be used for? The dataset be used for the full range of tasks related to lask-oriented dialogue and can used dual-modal task-oriented dialogue studies.
hnttss//apppen.com/
29

---

Is there anything about the composition of the dataset or the way it was collected and prepro- cessed/cdeanee//bbelel that might impact future uses? For example, is there anything that future user might need to know avoid uses that could result in unfair treatment of individuals or groups (e.g,, stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms? NA.
Are there tasks for which the dataset should not be used? If so, please provide a description. NA.
E.6 Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was Ir so, please provide description. SpokenWOZ dataset and codebases for reproducing the experiments are available at: https:/spokenwoz.gthhe githh.. io/SpokenWOZZ WOrk github.io/.
How will the dataset will be distributed (e.g,, tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? The dataset is now available at: https//spokenwwzzgiluuu githuu...
When will the dataset be distributed?
The dataset is available now.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide link other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions. SpokenWOL WOZ is distributed under the CC BY-NC 4.0 license. CC BY Y-NC 4.0 allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only long attribution is given to the creator. If you remix, adapt, build upon the material, you must license the modified material under identical terms.
Have any third parties imposed IP-based other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions. No.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation. No.
E.7 Maintenance
Who is case of violation of rights. Shuzheng Si (sishuzheng @ foxmail.com) and Wentao Ma (mawentao.ng www @alibbba inc.com) will be responsible for maintaining this dataset.
the dataset? Authors of this work bear all responsibility in
How can the owner/curraorr/maaaer of the dataset be contacted (e.g., email address)? you wish to extend or contribute to our dataset SpokenWOZ, please contact us via email Shuzheng Si (sishuzheng @ foxmail.com) and Wentao Ma www.eentao.mwtt-allaaaaaaaa
Is there an erratum? If so, please provide a link or other access point. Any updates to the dataset will be shared via GitHHb
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g.,mailing GitHuG? If we find inconsistencies the dataset or extend the dataset, we will release the new version on the website and Github.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals question told that their data would be retained for a fixed period of time and then deleted)? N/A
30

---

Will older versions of the dataset continue to be supported/ootee/nniiaaaaiii If so, please describe how. If not, please describe how its obsolescence will be communicated to users. All versions of SpokenWOZ will be continue to supported maintained on website. We will post the updates on the website and Github.
others want to extend/augmment/huild on/contribute to the dataset, there a mechanism for them to do so? so, please provide a description. Will these contributions be validated/veriffed?? If so, please describe how. If not, why not? Is there a process for commmicatingddistributtii these contributions to other users? If so, please provide description. Yes. Please contact the authors of this paper building upon this dataset.
E.8 Responsibility
The authors bear all responsibility in case of violation of rights, etc. We confirm that the dataset is licensed under CC BY-NC 4.0 license.
31


Reviews:
[{"review_id": "LfG7NrfzKP", "reviewer": "Reviewer_cbUA", "summary": "", "questions": "", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The dataset contains 8 domains, 203k turns, 5.7k dialogues and 249 hours of audios.\n2. The datasets addresses some unique characteristics of spoken conversation.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Fcihbv0JeW", "reviewer": "Reviewer_F6kt", "summary": "", "questions": "", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. It contributes a large-scale speed-text TOD dataset across 8 domains and 249 hours, which is a valuable contribution to the dialogue research community.\n2. It includes detailed analysis of the dataset and conducts quality control to improve the annotation quality of the dataset.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mb8puGRsMQ", "reviewer": "Reviewer_fz7a", "summary": "", "questions": "", "limitations": "There are no negative societal impacts or limitations that should be addressed.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The main strength of the submission is sharing with the research community ready to use, free, large dataset that costs to make around 55k according to estimates (excluding researchers' time). \nWhat is another contribution is providing insights into creation processes making it extensible and reproducible. Moreover, it is thoroughly documented.\nNext, introducing new slot (cross-turn and reasoning) types and proposing Macro Average Mentioned Slot Accuracy and using dual-model is noteworthy. Nonetheless demonstrated results and conclusions from experiments were to be expected. The generative method's helpfulness and dual-models superiority is not a surprising conclusion. \nAchievement in itself is by means of reporting experiment results on SpokenWOZ (for both dialogue state tracking and response generation), successfully demonstrating that the SpokenWOZ is much more challenging than for example MultiWOZ - its properties stemming from its spoken nature are as planned posing issues for the vast majority of models performing well on written datasets. \nIt is great to see exposing shortcomings of LLMs, identifying what they are and tackling them. \nSubmission with primarily supplementary material proves beyond the shadow of a doubt the high quality of the research.\nThe ethical and social implications of a paper are positive. The research community will gain great materials to experiment with and build upon.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "6GsXWycRxH", "reviewer": "Reviewer_QaYG", "summary": "", "questions": "", "limitations": "I don't see any limitations or flaws in the paper. It is sound, interesting, the effort to prepare the dataset was significant, there is really nothing I can point out as a drawback. ", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "It is a very good paper. I like everything about it. The dataset is very large (when compared to other annotated datasets used to train ASRs). The idea to put challenging annotations that test models' abilities to discover cross-turn entities, or to perform various types of reasoning on spontaneous speech is very good. I like the strict protocol used to generate conversations and to test the quality of recordings, transcriptions, and annotations. The fact that such significant work has been undertaken in the field of spontaneous speech is very important.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "4cvMzktecq", "reviewer": "Reviewer_bThz", "summary": "", "questions": "", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The dataset can be used for advancing spoken TOD research and its related area.\n2. The evaluation results revealed current LLMs' ability of TOD understanding.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents a new spoken, multi-domain, task-oriented dialogue dataset, that closely parallels the widely used MultiWOZ multi-domain task-oriented dataset. Reviewers included good reviews, mainly suggestions for including clarifications in the paper, and the authors positively responded to these suggestions. I agree with the reviewers that this dataset will be useful.
Some additional concerns include: whether the annotation of the ASR outputs caused any noise in the annotations, especially for dialogue state categories. Please clarify this in the paper.
Furthermore, it would have been better if the authors had extended the databases and locales for this collection, as the limited database has been a concern for MultiWOZ. I hope the authors can consider this in the future updates.
Decision: Accept (Poster)