# STOD: Towards Scalable Task-Oriented Dialogue System on MultiWOZ-API
## 5 Experiment and Analysis  
### Multi-Domain Dialogue Modeling  
#### 5.1.1 Experiment Setting  
Dataset  
We evaluate our proposed multi-domain scalable task-oriented dialogue model MSTOD on MultiWOZ-API described in Section 3. Note that we do not modify the raw dialogue text but the annotations, so we can compare our MSTOD with previous TOD models on MultiWOZ 2.1.  
Evaluation Metrics  
We use the official evaluation script (we noticed that related works use different scripts, which may lead to unfair comparisons, so we adopted the official benchmark results from [[https://github.com/budzianowski/multiwoz](https://github.com/budzianowski/multiwoz)]([https://github.com/budzianowski/multiwoz](https://github.com/budzianowski/multiwoz)) (accessed on 1 May 2024); these results are generally slightly lower than in the original paper) proposed by [PERSON] and [PERSON] (2020) to evaluate various dialogue models. The evaluation includes the following metrics: **Inform** measures whether the system provides suitable entities according to the user requirements; **Success** checks if the system offers the appropriate entity and the information requested; and **BLEU** assesses the fluency of the generated responses. The overall quality of the dialogue system is determined using the **Combined** score, calculated as (Inform + Success) \(\times\) 0.5 + BLEU.  
Baseline  
To set the stage for evaluating our approach against established methods, we include several strong baseline models that have demonstrated effectiveness in multi-domain task-oriented dialogues. Here is a brief overview of each:  
**DAMD**[26] A multi-action data augmentation model that utilizes one-to-many relationships to generate diverse and appropriate dialogue responses.  
**AuGPT**[27] A pre-trained model that enhances GPT-2 with a new dialogue consistency classification task, aiming to maintain dialogue coherence across turns by learning to classify consistent responses.  
**MinTL**[28] An end-to-end model that uses pre-trained language models (PLMs) in a Seq2 Seq manner. It introduces two different decoders to separately track belief states and generate responses, optimizing the task-oriented dialogue system.  
**SOLOIST**[12] A model that introduces an auxiliary task, where the target belief state is replaced by the belief state from unrelated samples to predict consistency in dialogues, thereby improving the generalization of the model to various dialogue scenarios.  
**UBAR**[29] An end-to-end model incorporating all belief states across all dialogue turns, which enriches the context available for generating responses.  
**PPTOD**[13] An innovative model that recasts task-oriented dialogue subtasks into prompts, leveraging the multi-task transfer learning capabilities of T5 to handle diverse dialogue tasks effectively.  
**BORT**[14] A robust model that adds a denoising reconstruction task to the encoder-decoder framework, which helps in reconstructing the original context from altered dialogue states, thus enhancing the model's understanding of dialogue flow.  
**MTTOD**[30] A pre-trained model that introduces a span prediction task during pre-training, helping the model to better identify and utilize relevant information within the dialogue for response generation.  
**GALAXY**[31] A pre-trained model that focuses on pre-training tasks such as dialogue act prediction to optimize dialogue policies, aiming to improve the strategic decision-making of dialogue systems.  
To draw a fair comparison, we only use the raw dialogue context as the model input and output the final natural language response for evaluation. Specifically, these baselines use the generated dialogue states, while our model uses generated API queries.