SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathbf{SPACE} + \mathbf{WavLM}_{\mathrm{align}}$ : As shown in Figure 3, we align the user-side word context and the corresponding audio pieces based on the annotations in SpokenWOZ. Then we add the user utterance embeddings from SPACE and the speech embeddings from WavLM as new user-side embeddings. During the inference,  $\mathbf{SPACE} + \mathbf{WavLM}_{\mathrm{align}}$  uses dual-modal input to generate the state and response.
As LLM makes amazing progress on NLP tasks [1], we also evaluate LLM's zero-shot performance. We use the prompts form Hudecek et al. [18] and Bang et al. [1] as reference.
ChatGPT: ChatGPT (GPT-3.5-turbo) is a conversational LLM [24], which has brought remarkable success on various zero-shot tasks.
InstructGPT003: InstructGPT003 (text-davinci-003) [24] is a 175B LLM trained by instruction tuning and reinforcement learning.
Please refer to Appendix A.2 for additional details and descriptions of the baseline models.
Table 7: DST experimental results.
Model	JGA	-w/o cross-turn slot
BERT+TripPy	14.78	15.58
SPACE+TripPy	16.24	17.31
SPACE+WavLM+TripPy	18.71	20.90
UBAR	20.54	23.51
SPACE	22.73	26.99
SPACE+WavLM	24.09	27.34
SPACE+WavLM align	25.65	28.15
ChatGPT	13.75	16.30
InstructGPT003	14.15	16.49
6.2 Results and Discussion
The results are reported in Tables 7 and 8. These tables offer several key insights as follows:
SpokenWOZ is challenging. Compared with the results in written MultiWOZ, such as SPACE achieving a JGA of 57.5 in DST task and a Combined Score of 110.95 in End-to-end Modeling task, the metrics are significantly lower in SpokenWOZ. This indicates that the current models learn the written TOD well, but ignore the characteristics of spoken TOD. As shown in Table 7, all baselines get improved JGA performance without evaluating cross-turn slots, meaning modeling cross-turn slots is challenging for current models. Furthermore, we propose Macro Average Mentioned Slot Accuracy (MAMS Acc) to measure the difficulty of different types of slots in DST. MAMS Acc is calculated by (i) separately determining the accuracy of each slot, excluding instances where the "none" value is present in the final turn; (ii) in each slot category, we calculate the macro average for the accuracy of each slot belonging to this category to get the MAMS Acc. We divide all the slots into four categories: reasoning slot, cross-turn slot, ASR-sensitive slot, and normal slot. Then we select "name" in restaurant, hotel and attraction domains as ASR-sensitive slots, because the special entity names are difficult to correctly recognized via ASR tool. Normal slot includes slots that do not belong to the above three types. As shown in Figure 4, the spoken characteristic such as ASR noise, reasoning, cross-turn slots are quite challenging and there is still much room for progress in the research of spoken TOD model. In addition to the challenges in DST, response generation tasks face their unique challenge of more diverse act flow modeling. SpokenWOZ extends the ontology of
Table 8: Policy Optimization and End-to-end Modeling experimental results.
Model	Policy Optimization	End-to-end Modeling
INFORM	SUCCEED	BLEU	Comb	INFORM	SUCCEED	BLEU	Comb
UBAR	62.50	48.10	9.69	64.99	60.20	47.40	9.90	63.70
GALAXY	70.60	42.20	16.52	72.92	65.80	38.50	20.10	72.25
SPACE	76.00	57.60	18.72	85.52	66.40	50.60	21.34	79.84
SPACE+WavLM	76.80	58.40	18.54	86.14	67.20	51.30	21.46	80.71
SPACE+WavLMalign	77.20	59.20	19.81	88.01	68.30	52.10	22.12	82.32
ChatGPT	73.40	39.50	4.58	61.03	23.40	13.80	3.59	22.19
InstructGPT003	78.20	56.90	7.72	75.27	25.30	18.50	6.13	28.03
MultiWOZ, which introduces the difficulties to predict the right system acts. Meanwhile, the act flow in SpokenWOZ is more diverse than in written MultiWOZ as shown in Appendix A.4.
Dual-modal TOD models is what you need. The significant size of the dialogues and audios in SpokenWOZ allows researchers to build large data-driven neural models given textual inputs and dual-modal inputs. To use dual-modal data in SpokenWOZ, we introduce several dual-modal baselines. Dual-modal methods achieve improved performance, showing the necessity of dual-modal methods in realistic scenarios. We observe that  $\mathrm{SPACE} + \mathrm{WavLM}_{\mathrm{align}}$  outperforms  $\mathrm{SPACE} + \mathrm{WavLM}$  due to the speech-text alignment. We show Case Study in Appendix A.3 to confirm our claim.
Supervised generative methods are helpful. We can empirically observe that generative methods, e.g., UBAR, achieve better performance than extractive methods, e.g., TripPy. We further conduct Case Study in Appendix A.3. It shows that the existing extractive methods can not handle cross-turn slots and reasoning slots, as the target values do not directly appear in the utterance. Meanwhile, due to the ASR noise, the extractive methods easily extract the wrong value, even in the right location. However, the generative methods can be robust to ASR noise and modify the wrong word in the original utterance to the correct value.
Dialogue state is the bottleneck of LLMs. As shown in 7 and 8, LLMs do not outperform
Figure 4: MAMS Acc of four categories of slot in advanced generative-methods.
supervised methods in DST and End-to-end Modeling. But, it is worth noting that LLMs achieve comparable performances in Policy Optimization task. It shows LLMs can complete the user request if provided the golden dialogue state and database results. However, when LLMs use their predicted dialogue state and database results in End-to-end Modeling, the performance is much lower. This suggests that LLM's ability to accomplish dialogue goals depends on the predictions of the dialogue state. To explore the reasons for the poor performance of LLMs, we provide an analysis in Appendix A.9. We find the main reason in DST is that the hallucination phenomenon [23] is very serious, i.e., LLMs generate erroneous results at slots that are not involved in the conversation. Meanwhile, LLMs are sensitive to noisy utterances, e.g., LLMs tend to directly copy the noisy word in the utterance as a result, which may be due to the inability of LLMs to perceive the speech information.
In this paper, we introduce SpokenWOZ, a large-scale spoken Task-oriented Dialogue (TOD) dataset that incorporates both text and speech inputs. SpokenWOZ encompasses with unique characteristics of spoken conversations, including word-by-word processing, ASR noise, and reasoning. In addition, we introduce two new slot types, cross-turn and reasoning slots, as novel challenges. We further present a range of baseline results to demonstrate the usability of the dataset. We hope that SpokenWOZ, with its rich, dual-modal data and considerable volume, will drive the progress of spoken TOD modeling.
This work was supported by Alibaba Group. Meanwhile, we would like to thank Prof. Milica Gasic for her appreciation and advice on our idea at the beginning of the project. We thank Dr. Bowen Yu for his constructive comments on our writing. Finally, we would like to thank all the annotators for their efforts. We look forward to our dataset advancing the research on spoken TOD.
[44] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning, 2023.
[45] Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. "going on a vacation" takes longer than "going for a walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363-3369, Hong Kong, China, November 2019. Association for Computational Linguistics.
A.1 Construction & Schema Details
A.1.1 Conversation Details
To make spoken conversations that close to the real scenarios, we change the following interaction pattern in MultiWOZ. In SpokenWOZ, once the user's booking is successful, the agent will provide the entity booked and ask for the user's profile information, rather than providing a reference code in MultiWOZ. Profile information including name, ID, email, license plate number, and phone. We will explain in detail when agents will actively collect profile information from users.
Name. When a user makes a successful hotel and restaurant reservation, the agent will request the user's name as the reserved information. The user's name is randomly generated by the script<sup>5</sup>.
ID number. When a user books a train, the agent will ask for the user's ID number as registration information, which is a randomly generated 16-digit string.
Email. When the user completes the hotel or restaurant reservation, the agent will ask the user if she/he wants to receive the order via email. If the user agrees to receive the order, the agent will request the user's email. The mailbox number consists of the first letter of the user's first name plus the user's last name, plus four randomly generated characters, and randomly choose one of
"@gmail.com", "@yahoo.com", "@outlook.com", "@@hotmail.com" as the suffix.
Table 9: The 36 slots are tracked in the dialogue state.
attraction	area / name / type
hospital	department
hotel	area / bookday / bookpeople / bookstay / internet / name / parking / pricerange / stars / type
restaurant	booktime / bookday / bookpeople / area / food / name / pricerange
taxi	arriveby / departure / leaveat / destination
train	arriveby / departure / destination / leaveat / bookpeople / day
profile	license plate number / name / ID / email / phone
License plate number. When a user reserves a parking space at a hotel, attraction, or restaurant, the agent will request the user's license plate number. The license plate number is a string of 7 random characters, the first two are letters, the middle two are numbers, and the last three are letters.
Phone number. When a user successfully books a taxi, the agent will request the user's phone number to contact the taxi driver, which is a randomly generated 10-digit string. In another case, when users inquire about police station information, the agent will also ask for the user's phone number as a contact number.
A.1.2 Slot Details
The following 36 slots are tracked in the dialogue state shown in Table 9. We also list the reasoning slot in Table 10. To control the number of cases where the value needs to be reasoned about in the reasoning slots, we require participants to implicitly express the values specified in the task goal.  $20\%$  of the reasoning slot values will be automatically marked as requiring implicit expression in the conversation.
A.1.3 Speaker Origins Details
Considering the different laws of data access of the different countries, we chose Canada, Singapore, China, South Africa to collect the audio data, which will enable us to open source the audio data. We also found that the
Table 10: Reasoning slot in SpokenWOZ. The upper script indicates which domains it belongs to. *: universal, 1: restaurant, 2: hotel, 3: attraction, 4: taxi, 5: train, 6: hospital, 7: police, 8: profile.
Temporal Reasoning	leaveat4,5
arriveby4,5
booktime1 / day5
bookday1,2
Mathematical Reasoning	bookpeople1,2,5
bookstay2
Semantic Reasoning	type1,2,3
area1,2,3
internet2
department6
parking2
5names: https://github.com/treyhunner/names
cost of collecting audio data from Canada and Singapore is about three times that of collecting from South Africa. Therefore, within the same budget, we chose to collect more audios from South Africa, and we believe that a larger data set would further prompt the research in the community. The distribution of speaker origins are shown in Table 11.
A.1.4 Audio Details
Our audio files are two-track. One track represents the voice of the user and the other represents the voice of the agent. Meanwhile, the sample rate of our audio files is  $8000\mathrm{Hz}$ . Each dialogue corresponds to an audio file, and each word is recorded in the text annotation corresponding to the word context, start time and end time. To avoid the problem of overlapping utterances, we follow the rules below during the collection: (i) prohibit the agent from using the backchannel to interrupt the user; (ii) when a user
Table 11: The origins diversity of SpokenWOZ. Participants come from four different countries to improve the diversity of spoken conversations.
Country	Dialogues	Percentage	People	Percentage
Canada	500	8.77%	60	24%
Singapore	500	8.77%	40	16%
China	2100	36.84%	30	12%
South Africa	2600	45.61%	120	48%
uses a backchannel expression, the agent should respond to the backchannel correctly, rather than ignoring it and continuing the previous utterance. Finally, the word error rate of ASR is  $6.1\%$ , calculated from the manually modified agent utterances and the agent utterances recognized by ASR tool.
A.1.5 Data Division
SpokenWOZ is divided into 4200/500/1000 dialogues in order by train/dev/test. More details can be found in Table 12. The results of experiments are evaluated by the test set.
A.2 Experiment Details
A.2.1 DSTBaselines
Table 12: Statistics of SpokenWOZ.
Dataset	Train	Dev	Test
Audio Hours	183	22	44
Dialogues	4,200	500	1000
Turns	149,126	18,384	35,564
Tokens	1,672,984	204,644	396,933
Avg. Turns	35.50	36.77	35.56
Avg. Tokens	11.21	11.13	11.16
BERT+TripPy TripPy makes use of copy mechanisms to fill slots. A slot is filled by one of three copy mechanisms, including: (1) span prediction: values are directly extracted from the user's utterances; (2) inform operations: a value may be copied from the system's inform operations; (3) slot copy: a value may be copied over from a different slot.
SPACE+TriPy We use SPACE to replace the original encoder BERT in TripPy. SPACE is a semi-supervised pre-trained conversation model learning from large-scale dialogue corpora with limited annotations, which can be effectively fine-tuned on different downstream dialogue tasks.
SPACE+WavLM+TripPy To use both speech and text data, we concatenate the embeddings from SPACE and WavLM. Then we use a Transformer encoder as the fusion layer to allow the interaction between the different modalities. Then we use the fused outputs as the representations in the TripPy.
UBAR UBAR is acquired by fine-tuning the GPT-2 on the sequence of the entire dialogue session which is composed of user utterance, dialogue state, database result, system act, and system response of every dialogue turn. During the inference time, it formulates DST as a sequence-to-sequence task. It takes the current user utterance, dialogue history, and the previously predicted dialogue state as input, and gets the dialogue state of the current user utterance.
SPACE SPACE is a semi-supervised pre-trained conversation model learning from large-scale dialogue corpora, which is based on UniLM [8]. Such as UBAR, we use SPACE as pre-trained language model to fine-tuning on the sequence of the entire dialogue session. During the inference, give the history and user utterance, SPACE generates dialogue states by autoregression. SPACE uses the same special token as UBAR to split user utterances, dialogue state, act and system response.
SPACE+WavLM To utilize the generation ability of the SPACE model and the speech-modal information, we concatenate the user utterance embeddings from SPACE and user audio embeddings from WavLM as new user-side inputs. During the inference, the model uses dual-modal inputs to generate the state by autoregression.
SPACE+WavLMaligned Using the annotations in SpokenWOZ, the text of a word can be aligned with its audio segment. To further explore how to make full use of the speech information, we align the token and audio segment of every word in user utterances. Then we add the text embeddings from SPACE and the corresponding embeddings from WavLM as new user-side embeddings. During the inference, the model uses dual-modal input to generate the dialogue state by autoregression.
ChatGPT: ChatGPT (gpt-3.5-turbo) is a conversational LLM that has been trained by reinforcement learning and instruction tuning [24], demonstrating a surprising ability in completing conversations.
InstructGPT<sub>003</sub>: InstructGPT<sub>003</sub> (text-davinci-003) [24] is a 175B LLM trained by reinforcement learning with human feedback and instruction tuning.
A.2.2 Response Generation Baselines
UBAR UBAR fine-tunes the GPT-2 on the sequence of the entire dialogue, including user utterance, dialogue state, database result, system act, and system response. During the inference, UBAR uses the fine-tuned GPT-2 to generate responses given different inputs based on different task settings.
GALAXY GALAXY is a pre-trained dialogue model that explicitly learns dialogue policy from limited labeled dialogues and large-scale unlabeled dialogue corpora via semi-supervised learning, which is based on UniLM. GALAXY keeps the same input format as UBAR.
SPACE SPACE is a pre-trained dialogue model that benefiting from large-scale dialogue corpora via multitask learning, including dialog understanding module, dialog policy module and dialog generation module. SPACE is based on UniLM. SAPCE keeps the same input format as UBAR.
SPACE+WavLM We concatenate the user utterance embeddings from SPACE and user audio embeddings from WavLM as new user-side inputs. During the inference, SPACE+WavLM uses dual-modal input to generate dialogue state, act, and final system response by autoregression.
SPACE+WavLM aligned SPACE+WavLM aligned adds the text embeddings from SPACE and the corresponding embeddings from WavLM as new user-side embeddings. During the inference, the model uses dual-modal input to generate dialogue state, act, and final system response.
A.2.3 Hyperparameters
For text-modal methods, we use the code and hyperparameters provided by their respective papers. For dual-modal methods, we uses the same hyperparameters as text-modal methods. To the fair comparison, we train all the baselines 10 epoch for DST, 25 epoch for response generation and use the final epoch checkpoint to get the results on SpokenWOZ. The results we report are the average of the results using five different seeds. We trained the baselines in NVIDIA A100 and V100.
We will show the predicted cases to confirm our insights proposed in section Experiments.
Supervised generative methods are helpful. We give the comparison between the generative-method UBAR and extractive-method BERT+TripPy. Extractive methods can not get the value if it does not directly exist in utterance, such as the reasoning slot. Meanwhile, the generative-method can be robust to ASR noise and modify the wrong word in the utterance to the right one.
Table 13: The Case shows that supervised generative methods are helpful.
There is a train, the id is called tr8925, do you want to make a booking.
Yes, please make a booking for me.
Okay. How many people?
Um let me think ah it's me and I have six friends with me.
(BERT+TriPy: Train-Bookpeople = none)
UBAR: Train-Bookpeople = 7)
The value 7 of slot Bookpeople can be correctly predicted by UBAR. However, value 7 is not existing which is predicted wrongly by BERT+TripPy.
Table 14: The Case shows that supervised generative methods are helpful.
:Uh,I'm looking for a particular hotel in cambridge.
:Okay.So may I know its name,please?
:Um let me check.I think the name is called lavelle lodge.
(BERT+TripPy:Hotel-Name=lavelle lodge)
(UBAR:Hotel-Name=lovell lodge)
In this case, the correct name of the hotel "lovel lodge" is not predicted correctly by BERT + TripPy, even it extracts the correct span in utterance. We also find that UBAR can get the name correctly, which shows that generative-method can learn the ability to correct errors from ASR.
Table 15: The Case shows that supervised generative methods are helpful.
: My id is 88716.
(BERT+TriPy: Profile-ID = None)
(UBAR: Profile-ID = 88716)
: You can proceed.
46859.
(BERT+TriPy: Profile-ID = None)
(UBAR: Profile-ID = 8871646859)
: I am still working with.
638141.
(BERT+TriPy: Profile-ID = None)
(UBAR: Profile-ID = 8871646859638141)
The generative-method can learn the ability to concatenate the value segment from different turns, which can be hardly learned by the extractive-method BERT + TripPy.
Dual-modal TOD models is what you need. We give the case study and comparison between text-modal methods  $\text{SPACE} + \text{TripPy}$ ,  $\text{SPACE} + \text{dual-modal methods}$ $\text{SPACE} + \text{TripPy} + \text{WavLM}$ ,  $\text{SPACE} + \text{WavLM}$ . Although the main experimental results reflect that speech information improves overall performance, we are more concerned with the performance of the ASR-sensitive slots.
Table 16: The Case shows that dual-modal TOD models is what you need.
Good afternoon. Yes. Uh, could you please assist me looking for a particular restaurant, please.
No problem. Do you have a name for a restaurant?
Yes. um, it's called the bangkok cutting.
Bangkok city, okay. Just give me a second. I'll look for it on the system.
Okay, not a problem at all.
(SPACE+TripPy: Restaurant-Name = bangkok cutting)
(SPACE+WavLM+TripPy: Restaurant-Name = bangkok city)
(SPACE: Restaurant-Name = bangkok city)
(SPACE+WavLM: Restaurant-Name = bangkok city)
In this case, the value of the slot Name is not correctly predicted by SPACE+TripPy. We find that the span prediction copy mechanism is performed in SPACE+TripPy. However, SPACE+WavLM+TripPy performed the copy mechanism and copy the value from Inform Operations to get right value. This indicates that the speech information can be used to further improve performance.
Table 17: The Case shows that dual-modal TOD models is what you need.
: I want to find a particular hotel to rise. I remember his name, but I can't find the location.
: So may I have the name of the hotel, please.
: Uh. the name of the hotel is work worth house.
: Okay, so I found a hotel called warkworth house for you.
: Uh, yes, yes. That's the hotel. Thank you.
(SPACE+TripPy: Hotel-Name = None)
(SPACE+WavLM+TripPy: Hotel-Name = work worth house)
(SPACE: Hotel-Name = worth house)
(SPACE+WavLM: Hotel-Name = warkworth house)
In this case, SPACE can not predict the value in the right pattern, but dual-modal SPACE+WavLM+TripPy successfully predict it. This shows that speech modality can also help generative methods learn the correct pattern, even in the presence of ASR noise from user utterances.
A.4 Heatmap of acts
We show the act flow in SpokenWOZ in Figure 6. Given the user dialogue act, we present the frequency of agent act in heat map. As shown in Figure 5 and 6, SpokenWOZ not only contains more types of acts, but also contains more diverse act flow. It is more difficult for the model to predict the right action and give the right response.
Figure 5: Heat map of agent acts in MultiWOZ. The heat map shows the frequency of the agent act (horizontal axis) after the given user act (vertical axis).
Figure 6: Heat map of agent acts in SpokenWOZ. The heat map shows the frequency of the agent act (horizontal axis) after the given user act (vertical axis).
A.5 Statistics
We give the distribution of domains in Table 18. Meanwhile, the dataset distributions of dialog length and turn length are shown in the following figures. We give the statistics of SpokenWOZ in Figure 7 and 8. Shown in Figure 7, the length of dialogue history is concentrated above 30 turns. The excessive number of dialogue turns also makes it difficult for the model to learn. We compared the multi-domain and single-domain dialogue in Figure 9, intuitively, the number of turns for multi-domain dialogue is larger than the number of turns for single-domain dialogue. In Figure 10, there is no significant difference between the utterance lengths of the user and agent, because SpokenWOZ is constructed using the Human-to-Human schema. We also show the distribution of the dialogue acts and slots in Figure 11.