Value type: the bridge to a better DST model
Year: 2023

Abstract:
Value type of the slots can provide lots of useful information for DST tasks. However, it has been ignored in most previous works. In this paper, we propose a new framework for DST task based on these value types. Firstly, we extract the type of token from each turn. Specifically, we divide the slots in the dataset into 9 categories according to the type of slot value, and then train a Ner model to extract the corresponding type-entity from each turn of conversation according to the token. Secondly, we improve the attention mode which is integrated into value type information between the slot and the conversation history to help each slot pay more attention to the turns that contain the same value type. Meanwhile, we introduce a sampling strategy to integrate these types into the attention formula, which decrease the error of Ner model. Finally, we conduct a comprehensive experiment on two multi-domain taskoriented conversation datasets, MultiWOZ 2.1 and MultiWOZ 2.4. The ablation experimental results show that our method is effective on both datasets, which verify the necessity of considering the type of slot value.

heading: 1 Introduction
text: Task-oriented dialogue systems have become more and more important as people’s demand for life increases(booking flights or restaurants), which have become increasingly important in the field of NLP(Nature Language Process). (Henderson et al., 2019; Hung et al., 2021; Zheng et al., 2022) Traditionally, the task-oriented dialogue system consists of four modules (Zhang et al., 2020): Natural language understanding(NLU), Dialogue state tracking(DST), Dialogue manager(DM) and Natural language generation(NLG). This module directly affects the decision-making behavior of the dialogue system, and plays an extremely important
⇤The first two authors contribute equally. Weiran Xu is the corresponding author.
role in the task-based dialogue system. (Lee et al., 2019)
The recent methods in DST work are mainly divided into two categories. The first category is based on ontology which means the candidate slot value is assumed to be known eg (Zhou et al., 2022; Ye et al., 2021b; Guo et al., 2021). The second is the way without ontology. These studies have completely abandoned ontology, and they assume that the slot value is unknown. eg(Wu et al., 2019; Kim et al., 2019; Kumar et al., 2020; Lin et al., 2021). However, most of their work is based on dialog state, dialog and slot modeling, ignoring that the value type of each slot may be different. If these slots are modeled uniformly, then there is a lack of a specific feature of each slot.
In this work, we propose a new DST framework named SVT-DST, which uses the Slot-Value Type as the bridge to increrase the model performance. With this method, each slot has specificity for the attention of the conversation history to better identify the slot value. Specifically, we first classify all the slots in the dataset according to their slot value types. As shown in Figure 1, adjectives, time and numbers correspond to pricerange, arrive-time and book-people respectively. We train a sequence annotation model with dialogue training which is used to extarct entities and corresponding entitytypes in each on the turn. We hope that the attention
1211
between the dialogue and slots can be higher when the turn is near to current turn with the same slotvalue type. In order to achieve the goal, we use monotonically decreasing functions to integrate the attention weights, which will be described in detail in the method. we use monotonically decreasing functions to integrate these types into the attention operation.
Our main contributions are as follows: 1) We classify the slot according to the slot-value type, then train the Ner model to extract these types to improve the attention formula. 2)We design a sampling strategy to integrate these types into the attention formula, which decrease the error of Ner model. 3)We have achieved competitive results on MultiWOZ 2.1 and 2.4. We analyze the results and point out the future work.

heading: 2 Method
text: Figure 2 shows the structure of our DST model, including encoder, attention module and slot value processing module. In this section, we will introduce each module of this method in detail.
A T-turn conversation can be expressed as Ct = {(U1, R1), ..., (Rt 1, Ut)}, where Rt represents system discourse and Ut represents user discourse. We define the dialogue state of the t-th turn as Bt = {(Sj , V tj )| 1 <= j <= J}, where V tj represents the value of the j-th slot Sj in the t-th turn. J represents the number of predefined slots. Follow (Ren et al., 2018), we express the slot as a "domain slot" pair, such as ’restaurant-price range’.

heading: 2.1 Encoder
text: Follow(Ye et al., 2021b) , we use two bert (Devlin et al., 2018) models to encode context and slot respectively.

heading: 2.1.1 Context encoder
text: We express the dialogue at turn t as Dt = Rt Ut, where represents sentence connection. Then the history of the dialogue including t-th turn as Mt = D1 D2 ... Dt.The input of the context encoder is Xt = [CLS] Mt [SEP ]. The output of the encoder is:
Ct = bertfinetuned(Xt) (1)
Where Ct 2 R|Xt|⇥d, |Xt| is the length of Mt and d is the hidden size of bert. bertfinetuned indicates that the bert model updates a part of parameters during training.

heading: 2.1.2 Slot-value related encoder
text: We employ the first token to represent the aggregate representation of the entire input sequence. Therefore, for any slot Sj 2 S(1  j  J) and any value vtj 2 Vj we have:
hSj = bertfixed(Sj) 2 R1⇥d (2) hv t j = bertfixed(v t j) 2 R1⇥d (3)
For the last turn of dialogue state Bt 1, we have
hBt 1 = bertfixed(Bt 1) (4)
Where hBt 1 2 R|Bt 1|⇥d, B 1 = Null. bertfixed indicates that the bert model has fixed parameters during training.

heading: 2.2 Cross-Attention
text: We use the multi-head-attention module(Vaswani et al., 2017) as the basis of our attention module.

heading: 2.2.1 Slot-Context Attention
text: We first calculate the bias term of the attention formula. For each dialogue history Mt, we first use the monotonically decreasing distribution function ⌘(n) to initialize the weight of each turn of dialogue Dt in the dialogue history:
(n) =
Z n+1
n ⌘(n)dn (5)
Where n = T t, n represents the distance between the last turn and the current turn. The closer the distance is, the greater the weight will be obtained. Note that (T ) represents the weight of distance T for this turn (turn 0) and the latest turn t. We record the turns of the value type typej with slot Sj in the history:
! = [m, ..., n] (6)
Where n>m, which represents the turn indexs. Then we calculate the weight of these turns:
⌦ij,t =
( (T i), i 2 !
0, else (7)
Finally, we add these two weights according to the turn indexs to get bias:
biasj,t = ⌦j,t = [⌦ 0 j,t, ...,⌦ t j,t] (8)
The attention between Sj and Ct can be calculated as:
ACj,t = Softmax( QjK T tp
dk + '(biasj,t)Wbias)Vt
(9)
AC,FFNj,t = W r 2 ReLU(W r 1 [(h Sj , ACj,t] + b r 1) + b r 2
(10)
Where '() indicates a learnable mapping built by embedding. Wbias, W r1 and W r 2 indicates a linear layer, respectively.

heading: 2.2.2 Slot-State Attention
text: For Sj and Bt 1, their attention can be expressed as:
ABj,t 1 =MultiHead(h Sj , hBt 1 , hBt 1)
(11)
AB,FFNj,t 1 =W r 4 ReLU(W r 3 [(h Sj , ABj,t 1] + b r 1)
+ br2 (12)

heading: 2.2.3 Gate Fusion
text: inspired by (Zhou et al., 2022), we employ a gate module to combine the attention between Slotcontext and Slot-state:
gtj = (W s j ⌦ [AC,FFNj,t ; A B,FFN j,t 1 ]) (13) mtj = g t j · ACj,t + (1 gtj) · ABj,t 1 (14)
Where ⌦ indicates vector product, indicates the sigmoid function and · indicates element-wise product operation.

heading: 2.3 Self-Attention And Value Matching
text: In this part, we have followed the relevant part of (Ye et al., 2021b).

heading: 2.4 Ner Model And Sampling Strategy
text: We employ the W2NER model(Li et al., 2022) as our tagging model. The strategy of our labelmaking is that: for each value in the ontology, if the value is in current turn, we will tagging this value. For sampling strategy, only when the target entities are different from entities extracted from previous turns, this turn will be marked with the entities’ type. This strategy helps to reduce the interference of duplicate entities. For the specific classification of each slot, please refer to the appendix. In particular, for bool type, we train the annotation model to extract keywords, such as internet, parking, etc.

heading: 2.5 Optimization
text: We use the sum of the negative log-likelihood as the loss function at each turn t:
Lt = JX
j=1
log(P (V tj | Xt, St)) (15)
Where
P (V tj | Xt, St) = exp( || t Stj hV tj ||2)
P V 0j 2Vj exp( || t Stj hV 0j ||2)
(16) t Stj indicates the output of self-attention module
corresponding to Sj at the t-th turn.

heading: 3 Experiments

heading: 3.1 Dataset, metric and Evaluation
text: We evaluate our method on these datasets: MultiWOZ 2.1 (Eric et al., 2019) and MultiWOZ 2.4 (Ye et al., 2021a) which provide turn-level annotations of dialogue states in 7 different domains. We evaluate our method on this dataset and follow the pre-processing and evaluation setup from (Wu et al., 2019), where restaurant, train, attraction, hotel, and taxi domains are used for training and testing. We use Joint Goal Accuracy that is the average accuracy of predicting all slot assignments for a given service in a turn correctly to evaluate the main results of models.

heading: 3.2 Baselines
text: (1) Trade: Transferable dialogue state generator (Wu et al., 2019) which utilizes copy mechanism to facilitate domain knowledge transfer. (2) Tripy: It applies three copying mechanisms to extract all values (Heck et al., 2020) (3) MinTL: An effective transfer learning framework for task-oriented dialogue systems(Lin et al., 2020),which uses T5 (Raffel et al., 2020) and Bart (Lewis et al., 2019). (4) Star: Framework with self-attention modules to learn the relationship between slots better (Ye et al., 2021b) from the dialogue context. (5) SST: a multi-domain dialogue state tracker which employs graph methods to fuse utterance and schema graph.(Chen et al., 2020) (6) TripyR: The model with a new training strategy based on Tripy (Heck et al., 2022). (7) MSP-B: An extraction model with mentioned slot pool(MSP) (Sun et al., 2022) (8)
LUNA: It applies a slot-turn alignment strategy to accurately locate slot values and their associated context. (Wang et al., 2022)

heading: 3.3 Main Results And Analysis Experiments
text: Table 1 shows the results of our main test and ablation study. Our base model achieved 53.28% for the joint-acc, while our Ner-based model achieved 55.37% , a significant improvement of 2.09% compared with the base model. In 2.4 dataset, our model achieved 68.28%, a significant improvement of 2.93% compared with the base model. And When we use the correct type labels for training, the model performance reaches 59.27%, which has exceeded all baseline models. Ground truth is extracted according to the slot-type in the turn label, similar to our sampling strategy. In order to model the attention of state and dialog history separately, we changed the attention in Star(Ye et al., 2021b) to the fusion of slot attention and dialog history attention. Such changes reduced the performance of the model. However, the ablation experiment shows that the method we proposed can really benefit the model indicators.
Table 2 shows the results of our analysis experiments, which use different distribution functions to model attention. For both 2.1 and 2.4 datasets, the experimental results show that under different distribution function modeling, the distribution with constant term bias may produce higher results such as 0.5 ⇤ (1 + x) + 1 and 1 x/30. And it often has a positive impact on the experiment when the power of the independent variable is 1.

heading: 3.4 Case Study
text: We conducted a series of analytical experiments on attention weights. As shown in the Table 3, we randomly selected a slot, "attraction-name," and then chose an example PMUL4648 from the test set to observe the attention distribution of this slot
for each turn in the test samples. In the example, the attraction-name slot is activated in the turn 2. It can be seen that function 3 noticed this turn with a large weight, followed by function 1. As a comparison, function 2 assigned larger weights to the first turn, which is sufficient to indicate that the fitting effect of function 2 is weaker compared to the other two functions. Our analysis is as follows: If there is no constant term in the distribution function, the difference between score+bias and score is not significant, resulting in limited performance improvement of the model. On the other hand, the power of the independent variable is greater than 1 such as function 2, the magnitude changes too obviously after Softmax. This leads to not smooth transitions between turns, resulting in limited performance improvement.
The result of using the ground truth labels training model shows that there is still huge space for improvement in Ner model annotation. One of the biggest challenges is that the annotation model often assigns certain entities to labels based on some fragmented tokens, without considering the impact of context, which leads to the proliferation of labels. We will solve this problem in future work.

heading: 4 Conclusion
text: In this paper, we propose an effective method to integrate slot-types into the DST model. Specifically, we propose the SVT-DST. This framework incorporates the slot-types information into the attention operation to help model pay more attention to these turns that include the type of one slot. Further, We design a sampling strategy to integrate these types into the attention formula to decrease the error of Ner model. Results on MultiWOZ dataset show
that our method has significant improvement on this task.
Limitation This work has two main limitations: (1) The performance of the model largely depends on the performance of the annotation model. If the annotation model is too simple, it may cause the performance of the DST model to decline. On the contrary, it will increase the complexity of the overall model and prolong the reasoning time. (2) Even for the labeling model with good performance, the tagging values may also interfere with the DST model. For details, please refer to the analysis experiment.

heading: A Appendix
text: A.1 Slot-Value Type A.2 Implementation we implement SVT-DST model based on the bertbase-uncased (110M parameters) model which has
12 layers and the hidden size is 768. The quantity of trainable parameters of the whole model is 24.85M. Our model is trained with a base learning rate of 0.0001 for 12 epochs about 4 hours. We use 1 NVIDIA 3090 GPU for all of our experiments.Joint goal accuracy is used to evaluate the performance of the models. Predicted dialogue states are correct only when all of the predicted values exactly match the correct values.The result of the model comes from the result of two averages. The annotation model is based on w2ner, which uses bert-largecased (330M parameters) as encoder.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
5.Limitation
7 A2. Did you discuss any potential risks of your work? Our work is devoted to improving the quality of the dialogue state tracking model, and there is basically no potential risk in theory.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3.3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We will solve this problem in future work