SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 18: The distribution of dataset domains.
Domains	Number
profile-restaurant-train	720
hotel-profile-train	702
attraction-hotel-profile-taxi	295
hotel-profile-restaurant-taxi	294
attraction-profile-train	291
profile-taxi	285
attraction-train	278
attraction-profile-restaurant-taxi	275
hotel-profile-restaurant	252
profile-restaurant	238
attraction	238
hotel-profile	237
attraction-hotel-profile	212
attraction-profile-restaurant	209
profile-train	193
train	149
hotel-train	148
restaurant-train	132
hotel	104
restaurant	102
attraction-restaurant	85
attraction-hotel	62
hospital	57
police-profile	56
attraction-profile	47
hotel-restaurant	39
Total	5700
Figure 7: The distribution of the length of turn in SpokenWOZ.
Figure 8: The distribution of the length of turn in SpokenWOZ.
Figure 9: The distribution of the number of turns in two kinds of dialog in SpokenWOZ: Multi-domain, Single-domain.
Figure 10: The distribution of the length of turn in two kinds of dialog in SpokenWOZ: User, Agent.
Figure 11: The distribution of domains, dialogue acts and slots in SpokenWOZ.
A.6 Dialogue Example
A dialogue example can be found in Figure 12.
Figure 12: A dialogue example from SpokenWOZ.
A.7 Online Database
We give the interface of our built database for participants in Figure 13.
Figure 13: The built online database in SpokenWOZ.
A.8 Task Goal Example
We give an example of our task goal for participants in Table 19.
Table 19: An example of task goal.
(Tips!) The important information is marked with < >. Message with (use veiled expression) means that an veiled expression is needed here.
(Attention!) Please ask customer service for information and try to solve your problem in the Shortest number of turns
(Background) Your name is <Misty Imbert>, your telephone is <9310729130>, your ID number is <8485147021469113>,
your email is <MImbert06jn@outlook.com>, your car_number is <OL09ODU>
(Background) You are planning your trip in Cambridge
You are looking for a <place to stay>. The hotel <doesn't need to have free parking> and should <include free wifi>
The hotel should be in the <west> and should have <a star of 4>
If there is no such hotel, how about one that has <free parking> Once you find the <hotel> you want to book it for <4 people> and <2 nights> starting from <friday>
You are also looking for a <restaurant>. The restaurant should be in the <moderate> price range and should serve <indian> food
The restaurant should be <in the same area as the hotel>
Once you find the <restaurant> you want to book a table for <the same group of people> at <12:15>(use veiled expression)> on <the same day>
(Background) Once you have made a booking, you do not want to give out your email address for receiving orders.
A.9 Analysis of LLMs
A.9.1 Prompts for LLMs
We imitate the format of prompt form Hudecek et al. [18] and Bang et al. [1]. We list a prompt example for DST task in Table 20.
STATE:
For response generation tasks, we follow the prompt as Hudecek et al. [18]. We use the dialogue state to query the database provided to get the entity. For the Policy Optimization task, we use the ground truth dialogue state to query the provided database. For End-to-end Modeling, we use the predicted dialogue state generated from ChatGPT or InstructGPT $_{003}$  to query the provided database. We use the contents of the database and the
generated system response to match the strings and complete the de-lexicalization process. The de-lexicalized response will be used to calculate INFORM and SUCCESS. We list a prompt example for a response generation task in Table 21.
Table 21: An example of a zero-shot version of the prompt used for Response Generation.
Definition: Please continue the dialogue as a task-oriented dialogue system called SYSTEM. The answer of SYSTEM should follow the DATABASE provided in JSON format and answer the USER's last utterance. SYSTEM can recommend and inform the contents in the DATABASE according to the utterance of the USER and return the name of the entity when it comes to restaurants, hotels and attractions, and the trainid when it comes to trains. But only when the USER requests information about an entity in the DATABASE, such as a phone number, should SYSTEM inform the corresponding content.
DATABASE:
{“restaurant”:{“address”：“Cambridge Retail Park Newmarket Road Fen Ditton”，“area”：“east”，“food”：“italian”，“name”：“pizza hut fen ditton”，“phone”：“12233237370”，“postcode”：“cb58wr”，“pricerange”：“moderate”}
DIALOGUE CONTENT:
USER:Hello.
SYSTEM: Hello! How can I assist you today? Are you looking for any specific information or recommendations?
USER: I'm in east part of the city and I want to have some food. Uh, hope for the. to serve Italian food and with moderate price range.
SYSTEM:
A.9.2 Analysis on DST
As shown in the section Experiments, the performances of LLMs do not surpass that of supervised small models and show a noticeable gap compared to the supervised generative dual-modal baselines in the DST task. However, it is worth noting that LLMs slightly outperform the BERT+TripPy when cross-turn slots are not taken into account, showing the potential capabilities of LLMs. Meanwhile, we find that the main reason for the poor results of LLMs is that the hallucination phenomenon [23] is very serious, e.g., LLMs often generate additional results that do not fit the dialogue utterances. When we measure only the slots where the ground truth value is not "none", we find that ChatGPT achieves JGA of 30.81 and InstructGPT<sub>003</sub> achieves JGA of 34.42. Compared to the standard JGA, ChatGPT and InstructGPT<sub>003</sub> improve their scores by 17.06 and 20.27 respectively. This indicates that LLMs generate a large number of erroneous results at slots that are not involved in the conversation, i.e., the hallucination phenomenon is very serious. We show a case in Table 24.
Table 22: The Case shows the hallucination phenomenon.
:Hello.
Hello, how can I help?
: Yes, I'm looking for restaurant.
(ChatGPT: Restaurant-Food = international)
(Ground Truth: Restaurant-Food = none)
Meanwhile, due to the inability of LLMs to perceive the information of speech, LLMs tend to generate the value directly from user utterance. We show a case in Table 23.
Table 23: The Case shows that LLMs are sensitive to the noisy utterance.
Uh, I'm looking for a particular hotel in cambridge.
Okay. So may I know its name, please?
Um let me check. I think the name is called lavelle lodge.
(ChatGPT: Hotel-Name = lavelle lodge)
(InstructGPT003: Hotel-Name = lavelle lodge)
(Ground Truth: Hotel-Name = lovell lodge)
A.9.3 Analysis on Response Generation
As introduced in the section Experiments, LLMs achieve comparable performances in Policy Optimization task but poor performances in the End-to-end Modeling task. We find that the main reason for the poor results in End-to-end Modeling is that the entity returned by the database does not meet the user's needs. Meanwhile,
the poor performances of BLEU show that there is a big difference between LLM's response style and human response style. Meanwhile, LLMs may feel confused about noisy utterances and generate a statement requesting clarification as shown in Table 24.
Table 24: The Case shows that LLMs is sensitive to the noisy utterance.
: Mm. I'm looking for a place to die (In the audio it is actually “dinner”).
ChatGPT: I'm sorry, I'm not sure I understand. Could you please rephrase your question?
Even though we tried hard to build a realistic spoken TOD benchmark for further studies, we could not use the audio data from real conversations due to privacy concerns. This brings the following limitations: (1) For "profile" domain, we use a designed script to generate random personal information, which may not be realistic, and the number of slots also limits the further in scenario that agents need to collect personal information from users; (2) As the limited ontology, our benchmark should mainly be used for research instead of deployed in realistic applications.
C Ethics Statement
We construct SpokenWOZ, a task-oriented dialogue benchmark containing both audio data and text data, and we will detail our ethical considerations for each part of our collection process:
Ontology Consideration. We inherited and expanded MultiWOZ's ontology, which is open-source and under the MIT License. We have used it in compliance with its terms of use.
Privacy Concern. In our dataset, we have designed the scenarios where an agent needs to proactively collects user information, however, our user information is all generated by scripts in a random manner, so there will not be any privacy leakage issues.
Audio Collection. We informed each participant that the collected audio data will be used as public dataset for research. Participants who agree to participate in data collection will sign a contract with us, and the ownership and use rights of their data belong to us. We will not disclose which specific participant the audio came from. At the same time, we have reviewed the legal regulations of four countries and regions, and will release the data in a legal manner, so this will not cause any legal problems. The distribution of our data sources has been discussed in the Appendix A.1.3, the diversity of SpokenWOZ makes our data unbiased. We pay  $30k for 249 hours audio. The average cost per hour of audio is$ 120.
Dialogue Annotation. During the annotation process, the annotators' personal information is not collected, which will not cause privacy leakage. At the same time, during the annotation process, we signed a non-disclosure agreement with each annotator, therefore, the audio data will not be leaked during the annotation process. We pay $20k for 5,700 dialogues. The average cost per dialogue annotation is $3.5. After our statistics, an average of one hour can annotate 5 dialogues.
D Data Format
D.1 Audio Format
As detailed in Appendix A.1.4, audio files are two-track with a sample rate of  $8000\mathrm{Hz}$ . One track represents the voice of the user and the other represents the voice of the agent. Each dialogue corresponds to an audio file, and each word is recorded in the text annotation corresponding to the word context, start time, and end time. We use the wav format to save our audio files. The file name of the audio is consistent with the id of the dialogue, for example, the corresponding audio file for MUL0032 is MUL0032.wav.
D.2 Text Format
Our text data is given in json format, and we take the same fields as popular MultiWOZ 2.2 [42] to store the corresponding information, so researchers can easily use our data. In addition, we additionally provide the data ontology and the database json files. There are 5,700 dialogues ranging from single-domain to multi-domain in SpokenWOZ. The test sets contain 1k examples. Dialogues with MUL in the name refers to multi-domain dialogues. Dialogues with SNG refers to single-domain dialogues. Each dialogue consists of a goal, multiple user and system utterances, dialogue state, dialogue act, corresponding audio and ASR transcription.
The dialogue goal for each dialogue is recorded in the "goal" field. The dialogue goal holds the fields involved in the dialogue as well as the slots involved and the corresponding values.
The dialogue state for each dialogue is recorded in the "metadata" field in every turn the same as MultiWOZ 2.2. The state have two sections: semi, book. Semi refers to slots from a particular domain. Book refers to booking slots for a particular domain. The joint accuracy metrics includes ALL slots.
The dialogue act for each dialogue is recorded in the "dialogue actu" and "span_info" field in every turn:
{
"\$"dialogue_id": {
"log":{
"\$"turn_id": {
"dialogue_ACT": {
"\$"act_name": [
"\$"slot_name",
"\$"action_value"
]
],
},
"span_info": [
"\$"act_name"
"\$"slot_name",
"\$"action_value"
"\$"start_charater_index",
"\$"exclusive_end_charater_index"
]
}
}
The ASR transcription for each dialogue is recorded in the "words" field in every turn.
\{
	"\$"dialogue_id": \{
	"log": \{
	"\$"turn_id": \{
	"words": [ \{
	"\$"word_context": "\$"word",
	"\$"begin_time": "\$"begintime",
	"end_time": "\$"endtime",
	"channel_id": "\$"channel",
	"word_index": "\$"index",
	\}
\}
\}
E Datasheets for SpokenWOZ
E.1 Dataset documentation and intended uses
For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description. Task-oriented dialogue (TOD) models have made significant progress in recent years. These systems are designed to assist users in accomplishing specific goals, e.g., flight booking and restaurant reservation. However, these TOD datasets constructed solely based on written texts may not accurately reflect the nuances of spoken conversations, leading to a gap between academic research and real-world spoken TOD scenarios. We introduce the common tasks of TOD, including dialogue state tracking, policy Optimization, and End-to-end Modeling.
Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., 15 company, institution, organization)? This dataset is created by researchers at Alibaba Group, Renmin University of China, and University of Michigan.
Who funded the creation of the dataset? The creation of dataset was funded by DAMO Academy, Alibaba Group.
Any other comments? N/A
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description. The dataset contains text data and adio data of spoken task-oriented dialogue. For each dialogue text, we also annotate both the dialogue state and dialogue act. For the dialogue audio, we also give the ASR transcription and audio file.
How many instances are there in total (of each type, if appropriate)? SpokenWOZ contains 8 domains, 203k turns, 5.7k dialogues and 249 hours of audios from spoken conversations.
What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features? In either case, please provide a description. For a conversation data, it includes a text part and a audio part. For each dialogue text, SpokenWOZ contains dialogue context, annotated dialogue state and annotated dialogue act. For the dialogue audio, SpokenWOZ contains corresponding audio data and ASR transcription.
Is there a label or target associated with each instance? If so, please provide a description Yes, we inherit and extend the MultiWOZ annotation schema, which is widely used for task-oriented dialogue.
Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. For individual instances, there is no missing information.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit. Each dialogue in SpokenWOZ is relatively independent, and the domains involved are different.
Are there recommended data splits (e.g., training, development/validation,testing)? If so, please provide a description of these splits, explaining the rationale behind them. We give the data splits in Appendix A.1.5. The data is split into training, development, and unreleased test sets. Once researchers have built a model that works to your expectations on the dev set, they can submit it to us to get official scores on the hidden test set. To mitigate the misestimation of the generalization error of the model, we increase the number of test set to 1000 dialogues. At the same time we keep the training and test data domain distributions roughly, but not exactly, the same.
Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description. Since our dataset requires manual annotation of dialogue state and dialogue act, annotation noise is inevitably introduced. At the same time the dialogue audio collection there are cases of substandard audio quality, such as low communication quality. As shown in section SpokenWOZ Construction, strict quality control is performed at each collection stage.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? SpokenWOZ is self-contained.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description. No.
Does the dataset relate to people? If not, you may skip the remaining questions in this section. No. Detailed in Ethics Statement, we have designed the scenarios where an agent needs to proactively collects user information, however, our user information is all generated by scripts in a tandem manner, so there will not be any privacy leakage issues.
E.3 Collection process
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how. We report the construction schema of SpokenWOZ in the section SpokenWOZ Construction.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated? We organized 250 participants to generate 5,700 dialogues via phone calls. The details of the audio file can be found in Appendix A.1.4. The dialogue state and dialogue act are annotated using the Appen platform<sup>6</sup>.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? SpokenWOZ is not sampled from a larger set.
Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the time-frame in which the data associated with the instances was created. Our data collection starts in July 2022. The contents of our data instances are independent of the time of collection.
Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation. Not applicable. We consider these contents in Ethics Statement.
Does the dataset relate to people? If not, you may skip the remainder of the questions in this section. No.
E.4 Preprocessing/cleaning/labeling
Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section. We report the construction schema of SpokenWOZ in the section SpokenWOZ Construction.
Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data. Raw data was not saved to prevent misuse. We will only open source the cleaned data.
Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point. We have used Python language to implement data cleaning. We will share the scripts details in our codebase.
Has the dataset been used for any tasks already? If so, please provide a description. The complexity and diverse spoken characteristics in SpokenWOZ make it a useful dataset for different TOD tasks, including dialogue state tracking and response generation. For response generation, the challenges are twofold: Policy Optimization and End-to-end Modeling. More details can be found in section Tasks & Settings.
Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point. We provide links to the papers of all the baseline models on the leaderboard<sup>7</sup> we built.
What (other) tasks could the dataset be used for? The dataset can be used for the full range of tasks related to task-oriented dialogue and can be used for dual-modal task-oriented dialogue studies.
$^{6}$ https://appen.com/  
 $^{7}$ https://spokenwoz.github.io/SpokenWOZ-github.io/
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms? NA.
Are there tasks for which the dataset should not be used? If so, please provide a description. NA.
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. SpokenWOZ dataset and codebases for reproducing the experiments are available at: https://spokenwoz.github.io/SpokenWOZ-github.io/.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? The dataset is now available at: https://spokenwoz.github.io/ SpokenWOZ-github.io/.
When will the dataset be distributed? The dataset is available now.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions. SpokenWOZ is distributed under the CC BY-NC 4.0 license. CC BY-NC 4.0 allows reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions. No.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation. No.
Who is supporting/hosting/maintaining the dataset? Authors of this work bear all responsibility in case of violation of rights. Shuzheng Si (sishuzheng@foxmail.com) and Wentao Ma (mawentao.mwt@alibaba-inc.com) will be responsible for maintaining this dataset.
How can the owner/curator/manager of the dataset be contacted (e.g., email address)? If you wish to extend or contribute to our dataset SpokenWOZ, please contact us via email - Shuzheng Si (sishuzheng@foxmail.com) and Wentao Ma (mawentao.mwt@alibaba-inc.com).
Is there an erratum? If so, please provide a link or other access point. Any updates to the dataset will be shared via GitHub