Dialogue State Tracking with Sparse Local Slot Attention
Year: 2023

Abstract:
Dialogue state tracking (DST) is designed to track the dialogue state during the conversations between users and systems, which is the core of task-oriented dialogue systems. Mainstream models predict the values for each slot with fully token-wise slot attention from dialogue history. However, such operations may result in overlooking the neighboring relationship. Moreover, it may lead the model to assign probability mass to irrelevant parts, while these parts contribute little. It becomes severe with the increase in dialogue length. Therefore, we investigate sparse local slot attention for DST in this work. Slot-specific local semantic information is obtained at a sub-sampled temporal resolution capturing local dependencies for each slot. Then these local representations are attended with sparse attention weights to guide the model to pay attention to relevant parts of local information for subsequent state value prediction. The experimental results on MultiWOZ 2.0 and 2.4 datasets show that the proposed approach effectively improves the performance of ontology-based dialogue state tracking, and performs better than token-wise attention for long dialogues.

heading: 1 Introduction
text: Task-oriented dialogue systems aim to assist users to complete certain tasks and have drawn great attention in both academia and industry (Young et al., 2010, 2013; Chen et al., 2017). As the core of taskoriented dialogue systems, dialogue state tracking (DST) is designed to track the dialogue states during the conversation between users and systems, which is generally expressed as a list of {(domain, slot, value)} representing user’s goal (Rastogi et al., 2017, 2018). The estimated dialogue states are used for subsequent actions.
To achieve the dialogue state, value prediction is made for each slot given the dialogue history. At each turn, the model inquires of the dialogue history and predicts the state values accordingly (Xu
and Hu, 2018; Ren et al., 2018; Wu et al., 2019; Zhang et al., 2019; Heck et al., 2020). With it, how to extract appropriate context information in the noisy dialogue history is crucial and challenging (Hu et al., 2020). Yang et al. (2021) make an empirical study about the effect of different contexts on the performance of DST with several manually designed rules. It indicates that the performance of DST models benefits from selecting appropriate context granularity.
In recent mainstream models, a fully token-wise slot attention mechanism is widely used to capture slot-specific information with dialogue history. The attention assigns an attention weight to each token, measuring the relationship of each token in dialogue history for the specified slot, and then attends them with these weights. Although encouraging results have been achieved, it also brings some risks. First, such operations disperse the distribution of attention, which results in overlooking the neighboring relation (Yang et al., 2018). Some entities (e.g., restaurant and attraction names) in spoken dialogue are generally informal, diverse, and local-compact, where the non-semantic tokens may be included. Moreover, a limitation of the used softmax computation is that the probability distribution in the outputs always has full support (Martins and Astudillo, 2016), i.e., softmax(z) > 0 for every vector z. It may lead a model to assign probability mass to implausible parts of dialogue history. Involving noise may make the model difficult to focus on the essential parts, and it may be more severe with the increase in dialogue length (Peters et al., 2019).
To tackle this problem, we propose a sparse local slot attention mechanism for this task. In our approach, local semantic information is firstly achieved at a sub-sampled temporal resolution capturing local dependencies for each slot. Then, these local information is attended with sparse attention weights generated by sparsemax function (Martins
39
and Astudillo, 2016), which outputs sparse posterior distributions by assigning zero probability to irrelevant contents in the dialogue history.
We conduct experiments to verify our approach on MultiWOZ 2.0 and 2.4 datasets. The contributions can be addressed as follows: 1) We propose a sparse local slot attention mechanism to lead the model to focus on relevant local parts to the specific slot for the DST task; 2) We demonstrate that the performance of DST benefits from introducing local information with our proposed approach, and make an empirical study that shows that our model performs better in state prediction for name-related slots and long dialogues than the models based on fully token-wise attention.

heading: 2 Related Works
text: Dialogue state tracking (DST) is the core of taskoriented dialogue systems. In the early years, DST highly relies on hand-crafted semantic features to predict the dialogue states (Williams and Young, 2007; Thomson and Young, 2010; Wang and Lemon, 2013), which is hard to handle lexical and morphological variations in spoken language (Lee et al., 2019). Benefiting from the rapid development of deep learning methods and their successful application in natural language processing, neural method-based DST models have been proposed. (Mrkšić et al., 2017) proposes a novel neural belief tracking (NBT) framework with learning ngram representation of the utterance. Inspired by it, sorts of neural network-based models have been investigated for DST task (Nouri and Hosseini-Asl, 2018; Ren et al., 2018; Zhong et al., 2018; Hu et al.,
2020; Ouyang et al., 2020; Wu et al., 2019) and achieves encouraging results.
Pre-trained models have brought natural language processing to a new era in recent years. Many substantial works have shown that the pretrained models can learn universal language representations, which are beneficial for downstream tasks (Mikolov et al., 2013; Pennington et al., 2014; McCann et al., 2017; Sarzynska-Wawer et al., 2021; Devlin et al., 2019). More recently, very deep pre-trained language models, such as bidirectional encoder representation from the transformer (BERT) (Devlin et al., 2019) and generative pretraining (GPT) (Radford et al., 2018), trained with an increasing number of self-supervised tasks have been proposed to make the models capturing more knowledge from a large scale of corpora, which have shown their abilities to produce promising results in downstream tasks. In view of it, many pieces of research of DST have explored to establish the models on the basis of pre-trained language models (Hosseini-Asl et al., 2020; Kim et al., 2020; Lee et al., 2019; Zhang et al., 2019; Chen et al., 2020; Chao and Lane, 2019; Ye et al., 2021b; Heck et al., 2020; Lin et al., 2020).
Related to extracting slot-specific information, most of the previous studies rely on dense tokenwise attention (Lee et al., 2019; Wang et al., 2020; Ye et al., 2021b). However, several pieces of research have indicated that local information may be missing with it (Yang et al., 2018; Shaw et al., 2018; Sperber et al., 2018; Luong et al., 2015; Yang et al., 2022). Motivated by it, we investigate introducing local modeling in this task. The most rele-
vant research is (Yang et al., 2021), which makes a comprehensive study of how different granularities affect DST. However, this research employs simple hand-crafted rules to neglect several utterances in a dialogue history. Our proposed approach in this work is data-driven.

heading: 3 Dialogue State Tracking with Sparse Local Slot Attention

heading: 3.1 Encoding
text: As shown in Figure 1(a), BERTcontext is used for encoding the dialogue context, whose parameters are fine-tuned during training. Let’s define the dialogue history DT = {R1, U1, ..., RT , UT } as a set of system responses R and user utterances U in T turns of dialogue, where R = {Rt}Tt=1 and U = {Ut}Tt=1. We define ET = {B1, ..., BT } as the dialogue states of T turns, and each Et is a set of slot value pairs {(S1, V1), ..., (SJ , VJ)} of J slots. The context encoder accepts the dialogue history till turn t, which can be denoted as Xt = {Dt, E′t−1}, as the input and generates context vector representations Ht = BERTcontext(Xt).
Another pre-trained BERTsv is employed to encode the slots and candidate values. Its parameters remain frozen during training. For those slots and values containing multiple tokens, the vector corresponding to the [CLS] token is employed to represent them. For each slot Sj and value Vj , hSj = BERTsv(Sj), hVj = BERTsv(Vj).

heading: 3.2 Sparse Local Slot Attention
text: To extract slot-specific information, we propose sparse local slot attention (SLSA). As shown in Figure 1(b), sparse local slot attention accepts the dialogue history Ht and the representation hSj of the specific slot Sj . To obtain local information, we employ a convolutional layer whose kernel has size l and stride m over the context vector representation of dialogue history. The convolutional kernel accepts the local area in the dialogue history representation and multiplies it with the learnable parameters to obtain the local semantic representations.
H′t = ReLU(Conv(Ht) +Ht) (1)
After that, multi-head attention with the sparsemax function is employed to retrieve relevant information for each slot. It generates sparse distribution to each local area. The sparsemax function
returns the Euclidean projection of the input vector z onto the probability simplex ∆K−1 := {p ∈ RK |1Tp = 1,p ≥ 0}. The projection is likely to hit the boundary of the simplex, in which case sparsemax(z) becomes sparse (Martins and Astudillo, 2016).
Sparsemax(z) := arg min p∈∆K−1
||p− z||2 (2)
Then the output is concatenated with each slot to generate slot-specific representations through a feed-forward layer.
Q Sj t = hSjWQ + bQ (3) K Sj t = H ′ tWK + bK (4) V Sj t = H ′ tWV + bV (5)
α Sj t = Sparsemax(
Q Sj t K Sj t
T
√ dk )V Sj t (6)
C Sj t = W2ReLU(W1[hSj ,α Sj t ] + b1) + b2
(7)
Where WQ,bQ,WK ,bK , WV , and bV are the parameters of the linear layers for projecting query, key, and value respectively. dk = dh/N in which dh is the hidden size of the model, and N is the number of heads.

heading: 3.3 Slot Self-Attention
text: Slot self-attention is introduced to communicate information across different slots. Each sub-layer in the self-attention layer consists of the self-attention block and two fully connected layers of ReLU activation with layer normalization and residual connection. Let Ct = [CS1t , ...,C SJ t ] and F 1 t = Ct at the first sub layer, then for the l-th sub-layer,
F̃lt = LayerNorm(F l t), (8)
Glt = MultiHead(F̃ l t, F̃ l t, F̃ l t) + F̃ l t. (9)
For the l-th feed forward sub-layer,
G̃lt = LayerNorm(G l t), (10)
Fl+1t = FFN(G̃ l t) + G̃ l t. (11)
The output of the final layer is regarded as the final slot specific vector FL+1t = [f S1 t , ..., f SJ t ].

heading: 3.4 Slot Value Matching
text: A Euclidean distance-based value prediction is performed for each slot, the nearest value is chosen to predict the state value.
p(V jt |Xt, Sj) = exp(−d(hVj , fSjt ))∑
V ′j∈νj exp(−d(h V ′j , f Sj t ))
(12)
where d(·) is Euclidean distance function, and νj denotes the value space of the slot Sj . The model is trained to maximize the joint probability of all slots. The loss function at each turn t is denoted as the sum of the negative log-likelihood, Lt =∑J
j=1− log(p(V j t |Xt, Sj)).

heading: 4 Experiments

heading: 4.1 Datasets
text: We conduct experiments using a couple of mainstream datasets of task-oriented dialogue: MultiWOZ 2.0 and 2.4 datasets. MultiWOZ2.0 (Budzianowski et al., 2018) is currently the largest open-source human-human conversational dataset of multiple domains. MultiWOZ 2.4 is the latest version and fixes the incorrect and inconsistent annotations (Ye et al., 2021a).

heading: 4.2 Implementation Details
text: The BERTcontext is a pre-trained BERT-baseuncased model, which has 12 layers with 768 hidden units and 12 self-attention heads. Another BERT-base-uncased model is used as the BERTsv. For the sparse local slot attention, window size and stride are investigated in the experiment. Padding is added on both sides of the input if needed. The number of attention heads is 4. Adam optimizer is adopted with a batch size of 16, which trains the model with a learning rate of 4e-5 for the encoder and 1e-4 for other parts. The hyper-parameters are selected from the best-performing model over the validation set. We use a dropout with a probability of 0.1 on the dialogue history during training.

heading: 4.3 Main Results
text: The main results are shown in Table 1. As we can see, our model achieves the best performance on all the datasets. We utilize the Wilcoxon signed-rank test, the proposed method is statistically significantly better (p < 0.05) than baselines. For the MultiWOZ 2.0 dataset, our proposed SLSA model (window size is 3 and stride is 1) achieves a JGA of 54.83% performing better than STAR with a JGA of 54.53%, which is the previous SOTA. Moreover, on the latest refined version MultiWOZ 2.4 fixing
many annotations in the test set, our model obtains a JGA of 77.92%. To sum up, our proposed model achieves a slight improvement on the original MultiWOZ 2.0 dataset, and a significant improvement on the latest refined MultiWOZ 2.4 dataset with a clean test set. We also make an investigation about the effects of local granularities, as shown in Appendix A.1.

heading: 4.4 Ablation Study
text: To further verify the proposed approach, we present some results that show the effectiveness of the components in the proposed approaches. Table 2 presents the joint goal accuracy and slot accuracy obtained when we progressively remove the components in our proposed model on MultiWOZ 2.4 dataset. On one hand, comparing SLSA and "w/o Local" (or "w/o Sparse" and "w/o both"), when the local pattern component is removed, the performance of corresponding model decreases. On the other hand, comparing SLSA and "w/o Sparse" (or "w/o Local" and "w/o both" when the sparse component is removed, the performance of the corresponding model decreases. It shows that the sparse and the local components are effective and important to the proposed model.

heading: 4.5 Error Analysis
text: An error analysis of each slot for the previous SOTA model STAR and our models on MultiWOZ 2.4 is shown in Figure 2, in which the lower the better. The four slots with the highest error rates
are hotel-type with 3.62%, attraction-name with 2.52%, restaurant-name with 2.32% and hotelname with 2.17%. It can be noticed that the later three are name-related whose values are diverse, local-compact, and may includes several nonsemantic tokens. Our proposed models perform better than STAR on these three slots, evidenced by that the error rates are lower. In addition, our model performs better in several categorical slots such as hotel-internet, hotel-parking, hotel-stars and book stay. We make a case study shown in Appendix A.2 to have a straightforward understanding of our proposed approach.

heading: 4.6 Performance for Long Dialogues
text: Figure 3 depicts the joint goal accuracy per turn of our models and STAR on MultiWOZ 2.4 dataset. Joint goal accuracy per turn is to measure the performance for long dialogues. It is considered correct if and only all of the values are correctly predicted for each slot until the n-th turn. In the beginning, the performance of these two models for short turns is comparable. Then it decreases as the dialogue length becomes longer since the previous states are employed as part of the input where some mistakes may be included. The trend of our model is a little milder. For very long dialogues whose length is larger than 7, our model performs better than
STAR. It shows our model performs better for the long dialogues DST.

heading: 5 Conclusion
text: In his work, we propose a sparse local slot attention for dialogue state tracking to alleviate allocating attention weights to content unrelated to the specific slot of interest. In our approach, local semantic information is firstly achieved at a sub-sampled temporal resolution capturing local dependencies for each slot. Then, these local information is attended with sparse attention weights generated by sparsemax function. The experimental results show that, comparing to several existing models based on dense token-wise attention, our approach effectively improves the performance of ontology-based dialogue state tracking in the state prediction for name-related slots and long dialogues.

heading: Acknowledgement
text: This work was supported by JSPS KAKENHI Grand Number JP22K12069 and partially supported by JSPS KAKENHI Grant Number 23K11227 and 23H03402.
Limitations
In this work, we propose a sparse local slot attention (SLSA) mechanism to make the model pay attention to slot-specified local areas in dialogue history, and then attend them with sparse distribution generated by sparsemax to neglect some redundant parts. This paper shows the effectiveness of our proposed approaches in state prediction for some specified slots and long dialogues. While we show that the model with SLSA is competitive in dialogue state tracking, there are limitation of that provide avenues for future works. First, it is not as easy to apply SLSA to generation-based dialogue state tracking. Different from ontology-based manners, the condition may be different in the case of generative DST since entire successive information involved in language modeling may be important for language generation. Therefore, how to handle the local and sparse properties for the generative model need to further consider. Second, convolution operation considers a fixed bounded local context. It is a challenge to handle local properties of various lengths.

heading: A Appendix
text: A.1 Effects of Different Locality Granularities We compare our model with different sizes and strides of the window of the local pattern to see how different granularities affect the performance on MultiWOZ 2.4 dataset, as shown in Table 3.
It shows that the best result is achieved when the size of 3 and the stride of 1, while the performance is not improved by enlarging the size of the local window or decreasing it. Note that, as mentioned in the experimental settings, in the main results, the hyperparameters of window size and stride are selected by tuning on the validation set.
A.2 Case Study Figure 4 and 5 demonstrate the predicted states of STAR and our model on two pieces of dialogues from the MultiWOZ 2.4 dataset. We color the input with the weights generated by sparse local slot attention in our model and the dense token-wise attention used in STAR. Note that in our model, one position with a dark background means the local area around this position is focused. It is different from STAR, in which one position denotes a token.
As shown in Figure 4, although STAR captures the relevant information for attraction-name but not the best. Our models are able to focus on the local area covering the entity. As shown in Figure 5, the user says "nothing in particular" indicating he/him does not prefer "a certain area". STAR fails to capture this information, and its attention is scattered. Our model realizes this and successfully gets the user’s point. Although the values "none" and "do not care" indicate the attraction-area does not need concrete values, they denote the user’s different intentions.

heading: Future Work
text: While we show that the model with SLSA is competitive in dialogue state tracking, there are limitation of that provide avenues for future works. First, it is not as easy to apply SLSA to generation-based dialogue state tracking. Different from ontology-based manners, the condition may be different in the case of generative DST since entire successive information involved in language modeling may be important for language generation. Therefore, how to handle the local and sparse properties for the generative model need to further consider. Second, convolution operation considers a fixed bounded local context. It is a challenge to handle local properties of various lengths