SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shuzheng Si $^{1*}$ , Wentao Ma $^{1*}$ , Haoyu Gao $^{1*}$ , Yuchuan Wu $^{1}$ , Ting-En Lin $^{1}$ , Yinpei Dai $^{2\dagger}$ , Hangyu Li $^{1}$ , Rui Yan $^{3\dagger}$ , Fei Huang $^{1}$  and Yongbin Li $^{1\dagger}$ $^{1}$ DAMO Academy, Alibaba Group
$^{2}$ Computer Science and Engineering Division, University of Michigan  $^{3}$ Gaoling School of Artificial Intelligence, Renmin University of China sishuzheng@foxmail.com, {mawentao.mwt, ghy385779}@alibaba-inc.com https://spokenwoz.github.io/SpokenWOZ-github.io/
Task-oriented dialogue (TOD) models have made significant progress in recent years. However, previous studies primarily focus on datasets written by annotators, which has resulted in a gap between academic research and real-world spoken conversation scenarios. While several small-scale spoken TOD datasets are proposed to address robustness issues such as ASR errors, they ignore the unique challenges in spoken conversation. To tackle the limitations, we introduce SpokenWOZ, a large-scale speech-text dataset for spoken TOD, containing 8 domains, 203k turns, 5.7k dialogues and 249 hours of audios from human-to-human spoken conversations. SpokenWOZ further incorporates common spoken characteristics such as word-by-word processing and reasoning in spoken language. Based on these characteristics, we present cross-turn slot and reasoning slot detection as new challenges. We conduct experiments on various baselines, including text-modal models, newly proposed dual-modal models, and LLMs, e.g., ChatGPT. The results show that the current models still have substantial room for improvement in spoken conversation, where the most advanced dialogue state tracker only achieves  $25.65\%$  in joint goal accuracy and the SOTA end-to-end model only correctly completes the user request in  $52.1\%$  of dialogues. Our dataset, code, and leaderboard are available at https://spokenwoz.github.io/SpokenWOZ-github.io/.
Dialogue system modeling, a classic research topic in the field of human-machine interaction, serves as an important application area [22, 21, 35, 44, 36, 34]. Recently, task-oriented dialogue (TOD) systems [14, 41, 3] have attracted significant attention. These systems are designed to assist users in accomplishing specific goals, e.g., flight booking and restaurant reservation. Benefited from the previous well-designed data-collection pipeline, such as Wizard-of-Oz [19], the TOD datasets [39, 2] can be constructed with low costs by writing from annotators, rather than being collected from realistic spoken conversations.
However, these TOD datasets constructed solely based on written texts may not accurately reflect the nuances of spoken conversations, leading to a gap between academic research and real-world spoken TOD scenarios. To advance the research on more realistic spoken TOD, various datasets have been
†Work done while the author was working at Alibaba.
Table 1: Dataset statistics of SpokenWOZ and existing TOD datasets*: SpokenWOZ contains 4200 dialogues in the training set.
Metric	DSTC2	KVRET	M2M	MultiWOZ	ABCD	DSTC10	SpokenWOZ*
Dialogues	1,612	2,425	1,500	8,438	8,034	107	5,700
Turns	23,354	12,732	14,796	115,424	177,407	2,292	203,074
Domains	Single	Multi	Multi	Multi	Multi	Multi	Multi
Collection	H2M	H2H	M2M	H2H	H2H	H2H	H2H
Type	Spoken	Written	Written	Written	Written	Spoken	Spoken
Audio	✓	✗	✗	✗	✗	✗	✓
Cross-turn Slot	✗	✗	✗	✗	✗	✗	✓
Reasoning Slot	✗	✗	✗	✗	✗	✗	✓
introduced to simulate and model spoken conversations [16, 17, 20]. Despite these efforts, existing datasets still exhibit three notable weaknesses:
- Data Scale: Previous spoken TOD datasets have been limited in terms of data scale, posing challenges for developing strong and more realistic systems. The pioneering ATIS dataset [16] consists of only 41 dialogues, which severely restricts the diversity and breadth of dialogues. While recent efforts, such as DSTC2 [17] and DSTC10 [20], have renewed interest in modeling spoken TOD, they offer a modest improvement with 1,612 and 107 dialogues, respectively. In contrast, written TOD datasets offer a significantly larger number of dialogues, exemplified by MultiWOZ [2]. Therefore, addressing this data scale limitation is crucial to advancing the spoken TOD systems.
- Human-to-Human Audio: Another limitation is the lack of audio from human-to-human spoken conversations. DSTC2 collects the spoken audio via the Human-to-Machine schema, limiting the construction of human-like systems. DSTC10 only provides the automatic speech recognition (ASR) hypotheses and dialogue text content, limiting the investigation of the dual-modal TOD model.
- Unique Challenges in Spoken Conversations: Previous spoken TOD datasets focus on robustness issues, especially ASR noise, ignoring the unique characteristics of spoken conversation, including word-by-word processing and reasoning in spoken language. These characteristics bring new challenges, such as handling incomplete utterances and reasoning in spoken languages.
Therefore, we aim at building a spoken TOD dataset from human-to-human spoken conversations, named SpokenWOZ, by extending the Wizard-of-Oz pipeline to realistic spoken conversations with real-time voice calls. To better model the spoken characteristics in SpokenWOZ, we introduce cross-turn slot and reasoning slot detection as new challenges to respectively address the dialogue state tracking in incomplete utterances and the indirect expression.
We spend more than 8 months building SpokenWOZ and implementing various quality improvements, achieving the final turn-level annotation accuracy of over  $97\%$ . The total cost of constructing SpokenWOZ is approximate  $\$55\mathrm{k}$ , including  $\$30\mathrm{k}$  for audio collection,  $\$20\mathrm{k}$  for dialogue annotation, and  $\$5\mathrm{k}$  for the cost of using ASR tools.
In summary, our contributions are threefold:
- We introduce a large-scale speech-text TOD dataset named SpokenWOZ, which contains more than 203K annotated utterances, 5,700 dialogues, and the associated 249 hours of audio. To the best of our knowledge, this is the largest speech-text TOD dataset with annotation of the dialogue state and dialogue act.
- We identify the new challenges in spoken conversation based on a comprehensive analysis of SpokenWOZ, including cross-turn slot detection and reasoning slot detection, which are not taken into account by the previous TOD datasets.
- We also conduct comprehensive experiments on various baselines, including text-modal TOD baselines and newly proposed dual-modal models. We also explore the LLM's zero-shot performance on SpokenWOZ, e.g., ChatGPT. The results demonstrate the models still have much room for improvement in spoken conversation scenarios.
Figure 1: Construction schema of SpokenWOZ. Data collection includes (1) collection of dialogue audio and (2) annotation of dialogue. Strict quality control is performed at each collection stage.
**Written TOD datasets** To push forward research in TOD modeling, various written TOD datasets have been proposed, ranging from human-to-machine [31] to more natural human-to-human dialogues [2], as well as from single-domain [39] to more realistic multi-domain scenarios [29]. Notable written TOD datasets include M2M [33], KVRET [11], MultiWOZ [2], SGD [30], and ABCD [4]. Initially, these datasets focus on the single-domain dialogue. Subsequently, MultiWOZ and SGD are proposed to address more realistic multi-domain dialogue. Recently, numerous written TOD datasets attempt to simulate more realistic spoken conversations. For instance, RADDLE [27] manually introduces various human-designed spoken noises to the written MultiWOZ. CGoDial [6] adds spoken features to existing Chinese TOD datasets via crowd-sourcing, but the language and lack of audio data both limit the research on spoken TOD. Similarly, SSTOD [43] introduces segment-by-segment interactions in simulated spoken language. Nonetheless, these studies concentrate solely on specific aspects of spoken TOD, thereby restricting a more holistic examination.
Spoken TOD datasets Several datasets have been proposed for modeling the spoken TOD. ATIS [16] focuses on single-domain TOD and only contains 41 dialogues. DSTC2 [17] provides spoken corpora, but remains limited in scale. DSTC10 [20] revisits spoken TOD, providing 107 dialogues that include ASR hypotheses. EVI [37] provides a transcript and ASR hypotheses, but only evaluates limited tasks, including enrolment (E), verification (V) and identification (I), ignoring more common tasks in TOD, such as querying and booking. To the best of our knowledge, SpokenWOZ represents the first large-scale speech-text dataset for TOD with annotation of the dialogue state and act.
3 SpokenWOZ Construction
To ensure the reliability of SpokenWOZ, we will discuss the stages involved in the construction schema in Figure 1. As shown in Table 2, SpokenWOZ consists of 8 different domains, 7 of which are inherited from widely used MultiWOZ, reducing the usage burden. Meanwhile, we design a new domain "profile" to introduce a realistic scenario where the agent needs to collect the user's personal information as the booking confirmation.
3.1 Collection of Dialogue Audio
To obtain dialogue audio in realistic spoken conversation, we organize 250 participants to generate 5,700 dialogues via phone calls. During the collection, one participant plays the role of a user and asks questions based on a template-generated task goal as shown in Appendix A.7. The other participant plays the role of an agent and answers the user by searching the same database as MultiWOZ. We build an online database for the agent as shown in Appendix A.8 to enhance the realism of the dialogues. More details about dialogue audio can be found in Appendix A.1.4.
Table 2: Full ontology in SpokenWOZ. The upper script indicates which domains it belongs to.  
*: universal, 1: restaurant, 2: hotel, 3: attraction, 4: taxi, 5: train, 6: hospital, 7: police, 8: profile.
acts	inform* / request* / ack* /select123 / recommend123 / nooffer123 / nobook125 / book125 edit8 / confirm8 / greet / bye / reqmore / wait / backchannel / thanks
slotslots	address12367 / postcode12367 / phone123678 / name12348 / area123 / price range123 type123 / internet2 parking2 / stars2 / open hours3 / departure45 / destination45 leaveat45 / arriveby45 / ID8 / email8 / license plate number8 / bookpeople125 trainID5 / ticket price5 / booktime1 / department7 / bookday12 / day5 / bookstay12
Qualification Test. Before the collection, we conduct a qualification test for each participant to ensure the quality of the data. The criteria include whether the task goal is completed and the naturalness of the audio. Each group of participants should submit completed dialogue audio, and we judge whether they pass the test. We initially receive 1520 applications, of which only 250 (16.4%) pass the qualification test.
Quality Control. We employ crowd-sourcing to evaluate the quality of each audio and remove the audio that exhibit poor communication quality or did not fulfill the task goal.
Speaker Origins. To ensure dialogue diversity, we organize participants from various countries, including Canada, Singapore, China, and South Africa. More details are shown in Appendix A.1.3.
3.2 Annotation of Dialogue
We train 15 annotators to annotate the clean dialogue state and dialogue act using both text transcriptions generated by the ASR tool  ${}^{4}$  and audio files. We inherit and expand the annotation schema of MultiWOZ, e.g., adding the new act "backchannel" to better model spoken conversations. Meanwhile, annotators need to transcribe the agent utterances according to the audio to obtain a clean text. In order to keep the noise from ASR tool, we do not manually transcribe the user utterances and keep the ASR noise in the user utterances. Due to the finely adequate training (approximately 3 weeks), this annotating process was able to achieve better results.
Qualification Test. We conduct a qualification test for each annotator. The test is only passed if the test dialogues are annotated completely correct by annotators. This approach ensures that only qualified annotators could participate in the final annotation to improve the quality of SpokenWOZ.
Quality Control. Before annotation, the annotators will return the audio in low quality. To ensure the quality of the annotations, we implement a three-step quality control process, including script checking based on rules, full inspection by annotators, and random inspection by us. We use scripts to identify simple errors such as missing annotations and return the dialogues for revision. Next, annotators conduct a full inspection and correct any wrong labels. Finally, we carry out a random inspection on  $10\%$  of the dialogues in SpokenWOZ. If the correct rate of the random inspection was less than  $97\%$  at turn-level, we would return this batch of dialogues for re-inspection by the annotators. Due to the strict quality control, SpokenWOZ can achieve a much higher quality than MultiWOZ, which contains over  $40\%$  of dialogue turns that need to be modified [10].
4 SpokenWOZ Dialogue Corpus
We build a novel, large-scale, human-to-human, multi-domain, dual-modal dataset that can be used to develop TOD systems on spoken conversations. SpokenWOZ includes a total of 5,700 dialogues and more than 203k utterances along with 249 hours of audio.
4.1 Characteristics of SpokenWOZ
In this section, we will highlight the unique spoken characteristics in SpokenWOZ, including word-by-word processing, ASR noise, and reasoning in spoken language. We will use the dialogue segments
4https://www.alibabacloud.com/help/en/intelligent-speech-interaction/latest/recording-file-recognition
Figure 2: The distribution of SpokenWOZ, written MultiWOZ and spoken DSTC10. The first two datasets have similar domains, and we can observe that spoken TOD tends to have more turns but shorter utterances. Compared to DSTC10, SpokenWOZ has more turns due to the cross-turn slots, and has shorter utterances because it does not model the knowledge-grounded TOD as DSTC10 does.
from SpokenWOZ to illustrate these characteristics in this section. Understanding these complexities is essential for developing more realistic spoken TOD systems. Additionally, SpokenWOZ has a distinct distribution compared with the written MultiWOZ, as shown in Figure 2, demonstrating the difference between written and spoken TOD. More statistics are illustrated in Appendix A.5.
4.1.1 Word-by-Word Processing
Most TOD datasets only include written conversations collected via crowd-sourcing. As a result, the interactions are turn-by-turn, with each turn containing complete semantics, such as "I would like a cheap restaurant in the north area." However, language processing in spoken conversation is inherently incremental and word-by-word. The current word spoken is not necessarily well thought out and relies more on the preceding few words, such as "I would like a restaurant, hmm, cheap one please, meanwhile in the south area, oh, sorry, in the north." This leads to less organized interactions, with utterances often characterized by various linguistic phenomena such as back-channel, disfluencies, and incomplete utterances.
Back-channel. Back-channel is a common phenomenon in human-to-human interaction. To effectively communicate with others, interlocutors often use minor messages such as "hmm" to show their cooperation and engagement during a conversation. This can be quite challenging when one participant responds without enough semantics to follow the previous statement.
Disfluencies. Another spoken language phenomenon is disfluencies, such as repeating and pausing, which introduce noise in utterance text. The disfluencies in spoken language introduce noise, leading to increased demands on model robustness.
Incomplete Utterances. In spoken conversations, users tend to inform long information in multiple utterances rather than conveying it all in one utterance. The information is provided segment-by-segment over multi-turn interactions. For example, the user can tell the license plate in the following utterances, "Okay. My license plate is n e", "8 6" and "g z w". In written TOD datasets, one complete utterance can contain all information, like "my license plate number is ne86gzw". How to integrate the information distributed in multiple turns is a new challenge.
4.1.2 ASR Noise
In spoken dialogue systems, the ASR module is integrated to convert speech into text, but it also introduces noise. Previous approaches, like RADDLE, attempt to simulate ASR errors by manually writing noisy texts to the written TOD datasets. In contrast, SpokenWOZ takes a different approach by naturally introducing ASR noise from real-world scenarios. This is achieved by first obtaining
dialogue audio and then using an ASR module to transcribe the audio into text. The presence of ASR noise can pose challenges for the system to understand the user's intent and generate a response.
4.1.3 Reasoning in Spoken Language
Given the casual and informal nature of spoken language, accurately determining the user's intent may require reasoning. For instance, Table 3 in SpokenWOZ demonstrates a scenario where the model needs to conduct reasoning to determine the total number of people involved.
Table 3: The dialogue segment from SpokenWOZ shows the reasoning in spoken language.
: Yeah. Yeah. Uh, can you book it for me, my parents and my grandparents?
: Okay, so it's five people in total.
4.2 New Challenges of SpokenWOZ
The inherent traits of spoken language pose unique challenges to TOD systems. To address this, we introduce two new types of slots in SpokenWOZ, including cross-turn slots and reasoning slots, for incomplete utterances and indirect expression in spoken language, respectively.
Table 4: The dialogue segment from SpokenWOZ shows the Cross-turn Slot Detection.
Oh, my id number is 5258 (“8” is missed by the ASR tool, but appears in the audio).
(Dialog State: id_number = 5258)
So it's 5 2 5 8.
Yes. and then 5763.
(Dialog State: id_number = 52585763)
5763.
And then 7525.
(Dialog State: id_number = 525857637525)
7525.
I'm sorry, 75 to 4.
(Dialog State: id_number = 525857637524)
Yes. okay, so it's 7524.
And then double 903.
(Dialog State: id_number = 5258576375249903)
4.2.1 Cross-turn Slot Detection
In spoken conversations, users may provide the value of a slot across multiple turns rather than in a single turn. Each turn only provides a piece of the value, and this is called the cross-turn slot. We introduce 5 cross-turn slots, including phone number, id number, user name, email, and license plate number in the domain "profile". More details are listed in Appendix A.1. Table 4 demonstrates an example where the user provides id number segments in different turns sequentially. Differing from updating a whole
slot value in one turn, the agent needs to precisely locate the part of values that needs to be updated. Meanwhile, the agent should not only accumulate the slot value across the multiple turns but also make more fine-grained changes to the current slot value, such as correcting "525857637525" to "525857637524" by saying "I'm sorry, 7, 5 to 4".
4.2.2 Reasoning Slot Detection
To address the challenges presented by the casual nature of spoken language, TOD systems need to engage in reasoning to effectively complete slot updates. SpokenWOZ introduces three kinds of reasoning, including Temporal Reasoning [45], Mathematical Reasoning [26], and Semantic Reasoning [32]. More reasoning slot details can be found in Appendix A.1, including the name of the specific slot and how to construct the reasoning slots in dialogue.
Table 5: The dialogue segment from SpokenWOZ shows Temporal Reasoning.
:Uh,yes,and on which day please?
:Oh.yeaH.And I think today is Friday,right.
Uh we will be there tomorrow.
(Dialog State:Bookday=Saturday)
Temporal Reasoning. Understanding temporal information is crucial for understanding user intent. Such as properly understanding the date of arrival at the hotel as shown in Table 5.
Mathematical Reasoning. Once the user does not directly inform the number in the utterance, mathematical reasoning is required for the user's expression. As shown in Section 4.1.3 and Table 3, the agent needs to know from the user utterance that the value of the Bookpeople slot is 5.
Semantic Reasoning. To capture the intent from indirect expressions, the agent needs to perform semantic reasoning based on the available information. As shown in Table 6, since the user expresses a desire for sushi, it can be inferred that the purpose is to find a Japanese restaurant.
5 Tasks & Settings
The complexity and diverse spoken characteristics in SpokenWOZ make it a useful dataset for different TOD tasks, including dialogue state tracking and response generation.
Dialogue State Tracking (DST). As highlighted in Section 4, the challenges in DST involve understanding noisy utterances, managing cross-turn slots, and handling reasoning slots. We use the joint goal accuracy (JGA) [2] as the metric, which measures the ratio of turns for which the value of each slot is correct.
Table 6: The dialogue segment from SpokenWOZ shows Semantic Reasoning.
: Yes, ma'am, the restaurant should serve sushi and should be in the center.
(Dialog State: Restaurant-Type = Japanese; Restaurant-Area = centre)
: Just to confirm that the restaurants are serving Japanese food in the centre.
: That's correct, man.
For response generation, the challenges are twofold:
Policy Optimization. System optimizes dialogue policy and generates the appropriate response based on user utterance and dialogue state.
End-to-end Modeling. System generates the correct response solely based on the user utterance, without explicit dialogue state information.
Following the previous TOD setting [2], we report the metrics of whether the system provides a correct entity (INFORM) and answers all the requested information (SUCCEED). A combined score (Comb) is computed by  $(\mathrm{INFORM} + \mathrm{SUCCEED}) \times 0.5 + \mathrm{BLEU}$  [25] as an overall quality measure.
6.1Baselines
We compare text-modal baselines, which include the following models:
BERT+TriPy: TripPy [15] uses BERT [7] as encoder and uses copy mechanisms to fill slots for DST, including span prediction, inform operations, and slot copy.
UBAR: UBAR [40] is acquired by fine-tuning the GPT-2 [28] on the sequence of the entire dialogue which is composed of user utterance, dialogue state, database result, system act, and system response.
GALAXY: GALAXY [13] is a pre-trained conversation model based on UniLM [9] that learns dialogue policy via semi-supervised learning. GALAXY keeps the same input and output format as UBAR to generate the dialogue state or system response.
SPACE: SPACE [12] is a pre-trained conversation model based on UniLM that takes into account both dialogue understanding and dialogue generation. SPACE keeps the same input and output format as UBAR to generate the dialogue state or system response.
SPACE+TripPy: We use SPACE to replace the original encoder BERT in TripPy as a new baseline.
We utilize WavLM [5] as the speech encoder and implement the following dual-modal models:
SPACE+WavLM+TripPy: We use a bi-encoder framework and then concatenate the representation outputs from SPACE and WavLM, then use Transformer [38] encoder as fusion layer to allow the interaction between the different modalities. We use the multi-modal outputs from the fusion layer as the representations for TripPy.
SPACE+WavLM: To utilize the generation ability of SPACE and the speech-modal information, we concatenate the user utterance embeddings from SPACE and user audio embeddings from WavLM as new user-side inputs and then generate the state or response by the SPACE decoder.
Figure 3: The architecture of  $\mathrm{SPACE + WavLM_{align}}$ . It remains the same text-side inputs and formats as SPACE, i.e., sequence of the entire dialogue context, but additionally introduces aligned speech-modal information of user utterances.