Multi-Domain Dialogue State Tracking with Disentangled Domain-Slot Attention
Year: 2023

Abstract:
As the core of task-oriented dialogue systems, dialogue state tracking (DST) is designed to track the dialogue state through the conversation between users and systems. Multi-domain DST has been an important challenge in which the dialogue states across multiple domains need to consider. In recent mainstream approaches, each domain and slot are aggregated and regarded as a single query feeding into attention with the dialogue history to obtain domain-slot specific representations. In this work, we propose disentangled domain-slot attention for multi-domain dialogue state tracking. The proposed approach disentangles the domain-slot specific information extraction in a flexible and context-dependent manner by separating the query about domains and slots in the attention component. Through a series of experiments on MultiWOZ 2.0 and MultiWOZ 2.4 datasets, we demonstrate that our proposed approach outperforms the standard multi-head attention with aggregated domain-slot query.

heading: 1 Introduction
text: Task-oriented dialogue system is designed to assist users to accomplish sorts of certain tasks. For example, by using dialogue-based automated customer service, users can online query information and make reservations. Multi-domain dialogue state tracking has been an important challenge introduced by Budzianowski et al. (2018), in which numerous mixed-domain conversations are involved. In this case, DST has to track the dialogue states at each turn through the conversation, which contains a huge space involving the combinations of the ontology of different domains, slots, and values. It is a challenging task since spoken language is not formal, in which ellipsis and cross-reference are barrier to handling the correlations among different domains and slots.
Several studies have explored sorts of approaches to handle the correlations among domains
and slots. In recent mainstream approaches, each domain and slot are aggregated into a single vector regarded as a query. The query and the dialogue history are fed into attention to generate domain-slot specific representations (Wu et al., 2019). Then the information interchange across different domains and slots are performed with them to model the correlation among different domain and slots (Hu et al., 2020; Wang and Lemon, 2013; Ye et al., 2021). However, these approaches introduce too much human prior knowledge and they only consider the correlations among domains and slots names or overestimate these correlations (Yang et al., 2022).
To tackle this problem, we propose a disentangled domain-slot attention (DDSA), which disentangles information extraction about domains and slots in a flexible and context-dependent manner. In detail, we disentangle the query about domains and slots in the domain-slot attention component. Firstly, domain specific representations are obtained using the domain query and the dialogue history. Then the model utilizes these representations and slot query to retrieve slot specific information (in this context, slot means the slot only) and generate domain-slot specific representations. Finally, state prediction is performed with these domain-slot specific representations.
We conduct experiments to verify our approach on MultiWOZ 2.0 and MultiWOZ 2.4 datasets. The experimental results show that the proposed approach can effectively improve the performance of multi-domain dialogue state tracking. The contributions of this work can be addressed as follows. (1) We propose a disentangled domain-slot attention mechanism to handle the correlations among domains and slots, in which the process of domainslot specific information extraction is disentangled in a flexible and context-dependent manner. (2) We demonstrate that the performance of DST benefits from our proposed approach and make a detailed empirical study that shows that our model performs
4928
better than the baseline models based on standard attention with aggregated domain-slot query1.

heading: 2 Related Works
text: Dialogue state tracking (DST) is the core of taskoriented dialogue systems. In the early years, DST highly relies on hand-crafted semantic features to predict the dialogue states (Williams and Young, 2007; Thomson and Young, 2010; Wang and Lemon, 2013), which is hard to handle lexical and morphological variations in spoken language (Lee et al., 2019). Benefiting from the rapid development of deep learning methods, neural networkbased DST models have been explored. Mrkšić et al. (2017) proposes a novel neural belief tracking (NBT) framework with learning n-gram representations of utterances. Inspired by it, a lot of neural network models are investigated (Nouri and Hosseini-Asl, 2018; Ren et al., 2018; Zhong et al., 2018; Hu et al., 2020; Ouyang et al., 2020; Wu et al., 2019) and achieve further improvement.
Pre-trained models have brought natural language processing to a new era in recent years. Many substantial works have shown that the pretrained models can learn universal language representations, which are beneficial for downstream tasks (Mikolov et al., 2013; Pennington et al., 2014; McCann et al., 2017; Sarzynska-Wawer et al., 2021; Devlin et al., 2019; Mittal et al., 2021). More recently, the very deep pre-trained language models, such as Bidirectional Encoder Representation from Transformer (BERT) (Devlin et al., 2019) and Generative Pre-Training (GPT) (Radford et al., 2018), trained with an increasing number of selfsupervised tasks have been proposed to make the models capturing more knowledge from a large scale of corpora, which have shown their abilities to produce promising results. In view of it, many pieces of studies about DST have explored to establish the models on the basis of these pre-trained language models (Hosseini-Asl et al., 2020; Kim et al., 2020; Lee et al., 2019; Zhang et al., 2020; Chen et al., 2020; Chao and Lane, 2019; Ye et al., 2021; Heck et al., 2020; Lin et al., 2020).
Related to handling the correlations among domains and slots in multi-domain DST, several approaches have been investigated. In recent mainstream approaches, domain-slot specific representations are first achieved using attention mechanism
1The code is available at https://github.com/ couragelfyang/DDSA
with aggregated domain-slot query, and then the correlations are modeled with them. (Balaraman and Magnini, 2021) utilizes domain and slot information to extract both domain and slot specific representations and then combines such representations to predict the values. Chen et al. (2020) manually constructs a schema graph modeling the dependencies of different slots and introduces a graph attention matching network to mix the information from utterances and graphs to control the state updating. Hu et al. (2020) introduces a matrix representing the similarity among different slots and then perform slot information sharing among similar slots. The above two approaches are name-based since they only consider the semantics dependencies of slot names to measure the correlation among different slots, which may result in overlooking the dependencies of some slots. More recently, Ye et al. (2021) proposes a data-driven approach to handle these correlations, in which slot self-attention is introduced. However, this approach may inevitably result in overestimating some correlations (Yang et al., 2022).

heading: 3 Dialogue State Tracking with Disentangled Domain-Slot Attention
text: Figure 1(a) presents the overview of the proposed model. It consists of a dialogue encoder, a domain, slot and value encoder, disentangled domain-slot attention (DDSA), and slot value matching. The context representations of dialogue history, domains, slots and values are firstly obtained by feeding dialogue history, domains, slots and values into encoders respectively. And then these representations are passed to our proposed disentangled domainslot attention, as shown detailedly in Figure 1b, to achieve domain-slot specific representations. Finally, the corresponding values are chosen to predict the state values with these representations and slot value matching.

heading: 3.1 Encoding
text: We employ BERT as the encoder to generate semantic representations. The BERTcontext whose parameters are fine-tuned during training is used for encoding the dialogue context. Let’s define the dialogue context history CT = {R1, U1, ..., RT , UT } as a set of system responses R and user utterances U in T turns of dialogue, where R = {Rt}Tt=1 and U = {Ut}Tt=1, 1 ≤ t ≤ T . We define ET = {B1, ..., BT } as the dialogue states of T
turns, and each Et is a set of slot value pairs {(S1, V1), ..., (SJ , VJ)} of J slots. Although the dialogue history Ct = {Rt, Ut} contains integrated information for the conversation until the t-th turn, the previous study (Ye et al., 2021) has indicated that it is helpful to combine it along with a compact representation E′t−1, which only includes the slots whose values are not none, as part of the input. In view of this, the context encoder accepts the dialogue history till turn t, which can be denoted as Xt = {Ct, E′t−1}, as the input and generates context vector representations Ht = BERTcontext(Xt).
Another pre-trained model BERTdsv is employed to encode the domains, slots, and candidate values, in which the parameters of BERTdsv remain frozen. For those slots and values containing multiple tokens, the vector corresponding to the special token [CLS] is employed to represent them. For each domain Di slot Sj and value Vk, hdi = BERTdsv(Di), hsj = BERTdsv(Sj), hvk = BERTdsv(Vk).

heading: 3.2 Disentangled Domain-Slot Attention
text: Figure 1(b) demonstrates the structure of our proposed disentangled domain-slot attention. The extraction with query about domains and slots is disentangled into two stages. The domain specific representations are first obtained using the domain query and the dialogue context. The slot query is employed to retrieve slot specific information based on the output of the previous stage. Finally, domainslot specific context representations are achieved for the subsequent state prediction.

heading: 3.2.1 Domain Query
text: Domain specific representations are achieved using the hidden representations of domains hd and
that of dialogue context Ht2. The process can be described as follows:
Qd = W nd dqhd + bQd (1) Kd = W nd Kd Ht + bKd (2) Vd = W nd Vd Ht + bVd (3)
αndd = softmax( QdK
⊺ d√
kdim , axis = domain) (4)
hndd = α nd d Vd (5)
Where Wdq,bQd ,WKd ,bKd ,WVd ,bVd are the parameters of the linear layers for projecting query, key, and value respectively at the domain query stage. kdim = kmodel/nd, in which kmodel is the hidden size of the model and nd ∈ Nd is the heads of the multi-head dot-product attention at this stage.

heading: 3.2.2 Slot Query
text: After the domain query stage, slot specific representations can be obtained using the output of the domain query stage and the hidden representations of slots hs. Note that here "slot" means the slot only rather than the concatenation or the average on the representations of domains and slots pairs. The process is shown as follows:
Qs = W ns sq hs + bQs (6) Ks = W ′ns Ksh nd d + bKs (7) Vs = h nd d (8)
αnss = softmax( QsK ⊺ s√
kddsa , axis = slot) (9)
hnsds = α ns s Vs (10) hds = WosConcat(h 1 ds, ...,h Ns ds ) (11)
2Here we omit the indices of domains and slots for simplification.
Where Wsq,bQs ,W ′ Ks ,bKs ,WVs ,bVs are the parameters of the linear layers for projecting query, key and value respectively at the slot query stage, and Wos is the parameters of the linear layer for aggregating the heads of slot query. kddsa is a hyperparameter indicating the hidden dimension in this component, and ns ∈ Ns is the number of heads at this stage.
Since the number of combinations of domains and slots is generally larger than that of the actual domain-slot pairs, a linear layer is employed to project domain-slot specific representation hds to the representation of the actual size.
hds = WodConcat(h 1 ds, ...,h Nd ds ) (12)
h′ds = Linear(hds, axis = domain× slot) (13)
Where Wod is the parameters of the linear layer for aggregating the heads of domain query.

heading: 3.3 Slot Value Matching
text: A Euclidean distance-based value prediction is performed for each slot. Firstly, the domain-slot specific vector is fed into a normalization layer. Then the distances between domain-slot specific vector and value are measured. Finally, the nearest value is chosen to predict the state value.
rDSmt = LayerNorm(Linear(h ′ ds)), (14) p(V kt |Xt, DSm) = exp(−d(hVk , rDSmt ))∑
V ′k∈νk exp(−d(h V ′k , rDSmt ))
(15)
where d(·) is Euclidean distance function, and νk denotes the value space of the actual domain-slot DSm. The model is trained to maximize the joint probability of all slots. The loss function at each turn t is denoted as the sum of the negative loglikelihood.
Lt = M∑
m=1
− log(p(V kt |Xt, DSm)) (16)

heading: 4 Experimental Settings
text: We conduct the experiments using MultiWOZ 2.0 and MultiWOZ 2.4 datasets in this work. MultiWOZ 2.0 (Budzianowski et al., 2018) is one of the largest open-source human-human conversational datasets of multiple domains. It contains
over 10,000 dialogues in which each dialogue averages 13.68 turns. MultiWOZ 2.4 is the latest refined version (Ye et al., 2022). It mainly fixes the annotation errors in the validation and test set. To make a fair comparison with the models evaluated on these two datasets, we follow the pre-processing and evaluation procedure in several previous works (Wu et al., 2019; Lee et al., 2019; Wang et al., 2020; Ye et al., 2021) to keep consistent. We present the settings of the model in Appendix A.

heading: 5 Results and Discussions

heading: 5.1 Main Results
text: Joint goal accuracy (JGA) and slot accuracy (SA) are employed to evaluate the overall performance. The joint goal accuracy is a strict measurement comparing the predicted values of each slot with ground truth for each dialogue turn, and the prediction is considered correct if and only if all the predicted values match the ground truth values without any error at each turn. The slot accuracy compares each value to the corresponding ground truth individually without seeing other turns. For the results of baselines, we use the results reported in the corresponding references.
Table 1 presents the results of the different models on the test set of MultiWOZ 2.0 and 2.4 datasets. As shown in it, overall, our proposed model achieves the best performance on these two datasets. We utilize the Wilcoxon signed-rank test, the proposed method is statistically significantly better (p < 0.05) than baselines. Comparing to the previous SOTA models SAVN on the original MultiWOZ 2.0 dataset, which utilizes slot attention with the concatenated domain-slot query extracting slot specific information and value normalization on the ontologies to varying degrees, and STAR, which uses slot self-attention with the aggregated domain-slot query to model the correlations among different slots, our model obtains a JGA of 54.70% and a SA of 97.49% outperforming SAVN with a JGA of 54.52% and a SA of 97.42% , and STAR with a JGA of 54.53% and a SA of 97.38%. For the latest refined MultiWOZ 2.4 dataset, our proposed model improves the performance by a relatively larger margin comparing to the previous SOTA STAR model from a JGA of 73.62% to 75.58% and a SA of 98.87% to 98.94%. To have a better understanding, an error analysis, a discussion about the effects of different hyperparameter settings, and a case study are made and presented in
Appendix B. These additional results also indicate the effectiveness of our approach.

heading: 5.2 Ablation Study
text: A simple ablation study is performed to verify the effectiveness of our proposed disentangled domainslot attention. As we can see in Table 1. The performance on the two datasets drops seriously when removing the proposed DDSA , which verifies the effectiveness of our proposed approach. In this case of model w/o DDSA, the domain specific and the slot specific information are extracted by feeding into the dialogue context and the domains and slots to the traditional domain and slot attention respectively, then they are concatenated and sent to the slot value matching component to perform state prediction.

heading: 6 Conclusion
text: In this work, we propose a model based on disentangled domain-slot attention for multi-domain dialogue state tracking to handle the correlation among different domains and slots. Unlike the conventional approach in recent mainstream models, we disentangle the query about domains and slots in a flexible and context-dependent manner. The experimental results on MultiWOZ 2.0 and MultiWOZ 2.4 datasets show that, comparing to the models based on conventional approaches of slot attention using the aggregated domain-slot pairs, our approach effectively improves the performance of multi-domain dialogue state tracking. In future
works, we will investigate to utilize the proposed approach to generative models and generalize them to more complicated scenarios.

heading: Acknowledgement
text: This work was supported by JSPS KAKENHI Grand Number JP22K12069 and partially supported by JSPS KAKENHI Grant Number 23K11227 and 23H03402.
Limitations
This paper shows the effectiveness of our proposed disentangled domain-slot attention mechanism in multi-domain dialogue state tracking. The limitation of this paper is that this work mainly focuses on ontology-based DST, which need a list of predefined candidate values in advance. The condition may be different in the case of generative DST since entire successive information involved in language modeling may be important for language generation. Therefore, how to tackle the problems in generated manners need to further investigate, which we intend to take up in future works.

heading: A Experimental Settings
text: The dialogue context encoder BERTcontext in this work is a pre-trained BERT-base-uncased model, which has 12 layers with 768 hidden units and 12 self-attention heads. We also employ another BERT-base-uncased model as the domain, slot and value encoder BERTdsv. For the proposed disentangled domain-slot attention, the number of heads of domains Nd and that of slots Ns in disentangled domain-slot attention are hyperparameters and investigated in the experiments. The dimension kddsa in it is set to 768. Adam optimizer is adopted with a batch size of 8, which trains the model with a learning rate of 4e-5 for the encoder and 1e-4 for other parts. The hyperparameters are selected from the best-performing model over the validation set. We use a dropout with a probability of 0.1 on the dialogue history during training. The ground-truth states at previous turns are involved in the input during training. The previously predicted states are used as part of the input when inferring.

heading: B Supplementary Results
text: B.1 Effects of Different Hyperparameter Settings
To investigate the effects of different hyperparameter settings, Table 2 presents the results of using different numbers of heads Nd for domain query and that Ns for slot query in the DDSA component in our model. It can be found that the model achieves the best performance when the number of heads for domain Nd = 16 and that for slot Ns = 32 in the experiment. These hyperparameters are selected by tuning on the validation set.
B.2 Error Analysis An error analysis of each slot for the previous SOTA model STAR and our model on MultiWOZ 2.4 is shown in Figure 2, in which the lower the better. It can be observed that the error rates of several name and area-related slots are improved significantly. Specifically, the performance of restaurant−name, hotel−type, hotel−area, attraction− area and hotel − bookstay are improved to a relatively large margin.
B.3 Case Study A case study below demonstrates some cases in MultiWOZ 2.4 dataset. Table 3 presents three dialogue episodes and the predicted dialogue states by the previous SOTA STAR and our proposed

heading: 4 8 71.58

heading: 4 16 71.14
text: model. It can be found that, for the first example, the system recommends "downing college" to the user’s request for an attraction. Although STAR captures the adjective phrase, the value for the slot attraction − name is not the referencing object "all saints church". Since there is a full slot selfattention is applied to the concatenated domain-slot query specific information, the mistake may be introduced from other domain-slot specific represen-
tations. In the second case, the user would like to confirm the asked hotel in the east area while STAR fails to get the point. In the third case, the user is open to any area. But STAR still overestimated the correlation between hotel and the previously mentioned restaurant. Our model successfully predicts the dialogue states for it.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitation
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3; cited in References

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3; cited in References

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 3; cited in References

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3; cited in References

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3; cited in References

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3; cited in References
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 states that we take the step as same in other works for consistency.
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Therefore, how to tackle the problems in generated manners need to further investigate, which we intend to take up in future works