SlowGuess/ACL_2020_OCR huggingface dataset:
Jiaying Hu $^{1}$ , Yan Yang $^{1,3*}$ , Chengcai Chen $^{2}$ , Liang He $^{1,3}$ , Zhou Yu $^{4}$
<sup>1</sup>East China Normal University
$^{2}$ Xiao Research, Xiao Robot Technology Co., Ltd
$^{3}$ Shanghai Key Laboratory of Multidimensional Information Processing
<sup>4</sup>University of California, Davis
51184506058@stu.ecnu.edu.cn, {yanyang,lhe}@cs.ecnu.edu.cn
arlenecc@xiaoi.com, joyu@ucdavis.edu
Dialogue state tracker is responsible for inferring user intentions through dialogue history. Previous methods have difficulties in handling dialogues with long interaction context, due to the excessive information. We propose a Dialogue State Tracker with Slot Attention and Slot Information Sharing (SAS) to reduce redundant information's interference and improve long dialogue context tracking. Specially, we first apply a Slot Attention to learn a set of slot-specific features from the original dialogue and then integrate them using a Slot Information Sharing. The sharing improves the models ability to deduce value from related slots. Our model yields a significantly improved performance compared to previous state-of-the-art models on the MultiWOZ dataset.
The recent global adoption of personal assistants such as Alexa and Siri made dialogue system a more popular topic in research. The major difference between dialogue systems and question-answering is that dialogue systems need to track dialogue history effectively. So, we normally use a dialogue state tracking component to track user's intention throughout the conversation. A dialogue state is typically composed as a set of slot value pairs in a task-oriented dialogue, such as "hotelinternet-yes". It means the slot "hotel-internet" has a value of "yes".
Early dialogue state tracking model needs a predefined ontology which means the values of every slot are enumerated in advance (Henderson et al., 2014; Mrkšić et al., 2017; Zhong et al., 2018; Sharma et al., 2019). Such practice is inefficient and costly. The large number of possible slot-value pairs makes deploying these models in the real-life
applications difficult (Rastogi et al., 2017). This difficulty is further amplified in multi-domain dialogue state tracking where the dialogues have more than one tasks. Because the manual effort grows exponentially with the complexity of the dialogues. In (Wu et al., 2019), Wu et al. introduced a transferable dialogue state generator (TRADE), which can generate dialogue states from utterances using a copy mechanism. This generative model achieved relative good performance, but it still has trouble in extracting relevant information from the original dialogues. For example, a user may tell the agent that he/she needs a taxi in a turn, but the taxi's departure location is implicitly mentioned several turns ago. Inspired by the (Chen et al., 2017; Chen, 2018), (Chen et al., 2019) studied on utilizing attention mechanism to deal with the long distance slot carryover problem. In their work, they first fused the information of the slot, its corresponding value and the dialogue distance into a single vector. Then they computed the attention between this single vector and the concatenation of dialogue and intent information. We simplify the attention method and introduce it into the dialogue state tracking task.
Moreover, it is a common sense that there is some kind of relevance between two slots involving the same domain or the same attribute. For example, people tend to have a meal near the attraction they visit, so slot "attraction-area" and slot "restaurant-area" have the same value at most times. For these slots with a common or related value, if a slot never or seldom appears in the training set, sharing the learned feature of data-sufficient slot may benefit the model's tracking ability on these rare or unknown slots.
So we propose SAS, a new multi-domain dialogue state tracking model to resolve this issue to some extent. To be specific, we use an Slot Attention to localize the key features from the original information-excessive dialogue and a Slot Infor
6366
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6366-6375 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
mation Sharing to improve the models ability to deduce value from related slots. The processed information provided by the slot attention and the sharing module makes the generator more sensitive to the location of the values in the dialogue history and thus generates correct slot values. Experiments on the multi-domain MultiWOZ dataset (Budzianowski et al., 2018) shows SAS can achieve  $51.03\%$  joint goal accuracy and outperform previous state-of-the-art model by  $2.41\%$ . On the single domain dataset which only contains the restaurant domain, we achieve  $67.34\%$  joint goal accuracy, outperforming prior best by  $1.99\%$ . In addition, we conduct an analysis of the experimental results to evaluate the quality of values generated by our model.
The early research of DST focused on the pipelined approach which involves a special module named Spoken Language Understanding (SLU) before the DST module (Wang and Lemon, 2013; Williams, 2014; Perez and Liu, 2017). But obviously, it was not reasonable to train SLU and DST respectively since the accumulated error in SLU may be passed to the DST. In order to alleviate this problem, later study focuses on the joint training methods (Henderson et al., 2014; Zilka and Jurcicek, 2015; Wen et al., 2017). Although the higher performance shows the effectiveness of models without SLU, there still remains some shortcomings. For example, these models typically rely on semantic dictionaries which list the potential rephrasings for all slots and values in advance. Make such a list is costly. Fortunately, the recent development of deep learning and representation learning helps the DST to get rid of this problem. (Mrksic et al., 2017) proposed a novel Neural Belief Tracking (NBT) framework which was able to learn distributed representations of dialogue context over pre-trained word vectors, while (Dernoncourt et al., 2017) described a novel tracking method which used elaborate string matching and coreference resolution to detect values explicitly presented in the utterance. These models greatly improve the performance of DST, but they are not good at handling rare and unknown slot value pairs which seldom or never appear in the training set.
There were many efforts to exploit general features between rare slot value pairs and common ones. (Zhong et al., 2018) proposed GLAD, a
model which built global modules to share parameters between estimators for different slots and local modules to learn slot-specific features. (Nouri and Hosseini-Asl, 2018) improved GLAD by reducing the latency in training and inference time, while preserving its powerful performance of state tracking. But as the dialogues become increasingly complex, the performance of these models on multi-domain is not as satisfying as on single domain. Because of the dependency on the dialogue ontology, they have difficulty in scaling up with domains. Once the number of domains increases, the amount of slot value pairs will boom. With the copy mechanism, the sequence-to-sequence model TRADE (Wu et al., 2019) successfully got rid of any predefined slot value pairs and generated dialogue states from conversation utterances.
But we find there still remain several crucial limitations which have not been well solved on multi-domain dialogues. First, these models rely on the long dialogue history to identify the values which belong to various domains and slots. Sometimes the information contained in the dialogue history is too rich for these models to efficiently utilize and the redundant information tends to interfere with their value identification or value generation. Second, the related information among similar slots is wasted. To alleviate these problems, a slot attention and a slot information sharing module are suggested. The former can isolate the most valuable information for each slot, while the latter integrates information kept by its all similar slots and improve the models ability to deduce value from related slots.
The dialogue state tracking models take the interaction context as input and extract slot value pairs explicitly or implicitly presented in conversations. The combinations of these slot value pairs are the representations of the user's goal. In this paper, we denote  $X = \{(u_{1},r_{1}),\dots ,(u_{T},r_{T})\}$  as the dialogue history, where  $u_{1},\dots ,u_{T}$  and  $r_1,\dots ,r_T$  are respectively the set of user utterances and the set of system responses in  $T$  turns. The dialogue state of turn  $t$  is marked as  $ST_{t} =$  (slot:  $s_j$ , value:  $y_{j}^{value}$ ). Here,  $s_j$  indicates the  $j$ -th slot, while  $y_{j}^{value}$  means the ground turth value sequence for this slot. All the slots in ontology are obtained by preprocessing the original MultiWOZ dataset with the delexicalization. Moreover, we extend the def
Figure 1: SAS model's architecture. This model consists of four parts: an encoder, a slot attention, a slot information sharing and a decoder.
inition of the slot to include the domain name for convenience. For instance, a slot in this paper will be "hotel-star", rather than "star".
Our primary goal is to learn a generative dialogue state tracker model  $M: X \times O \to ST$  that can efficiently capture the user's intentions for dialogues including multiple domains. And unlike most of the previous models, the ontology  $O$  mentioned in this paper only contains the predefined slots and excludes their values.
4 Our Proposed Model
Figure 1 shows the architecture of SAS. SAS is a sequence-to-sequence model augmented with slot attention and slot information sharing. Slot attention enables better feature representation and slot information sharing helps understanding less-seen slots. We describe the details of every component in SAS as follows:
We use a 1-layer bidirectional gated recurrent unit (GRU) (Chung et al., 2014) to encode the dialogue history. As TRADE (Wu et al., 2019), our input to the model is the concatenation of all words in the recent  $l$ -turn dialogue history  $\mathbf{X}_{\mathbf{t}} = [u_{t - l + 1},r_{t - l + 1},\dots ,u_{t},r_{t}]\in \mathbb{R}^{|X_{t}| \times d_{emb}}$ , where  $d_{emb}$  means the embedding size. First, each word in the dialogue history  $X$  is mapped to a distributed embedding vector. Then, a GRU is utilized to obtain the hidden state corresponding to each word in the text and we denote these hidden state as the his
tory hidden states  $\mathbf{H}_{\mathbf{t}} = \{h_1^{enc}, h_2^{enc}, \dots, h_{|X_t|}^{enc}\} \in \mathbb{R}^{|X_t| \times d_{hdd}}$ .
4.2 Slot Attention
To isolate key features from the noisy dialogue history, we build the slot attention. In fact, the multidomain dialogues are usually complex and contain rich features. This challenges the model's ability to cope with the excessively rich information.
To be specific, in one dialogue, user can mention various information, such as wanting to book a restaurant for a meal and then planning to see an attraction after the meal by ordering a taxi. There are in total 10 slots mentioned spanning across restaurant, attraction and taxi domains. Information from one domain maybe not useful for other domain and can even cause confusion. For example, both restaurant and taxi mention time and people.
So we propose the slot attention to only extract useful history information to every slot. More concretely, for a particular slot  $s_j$ , we first encode its slot name into slot hidden states  $\mathbf{SH}_{\mathbf{j}} = [sh_{j_1}^{enc}, \dots, sh_{j|N|}^{enc}]$ , where  $|N|$  is the maximum size of the slot name phrase. Since the last hidden state  $sh_{j|N|}^{enc}$  provided by the GRU contains the context information of the entire phrase, we pick it as the representation of slot  $s_j$ .
After that, we calculate the attention between the slot information,  $sh_{j|N|}^{enc}$  and the hidden states of the dialogue history  $\mathbf{H}_{\mathbf{t}} = [h_1^{enc},\dots ,h_{|X_t|}^{enc}]$  to obtain the context vector  $c_{j}$ :
6368
$$
\mathbf {a} ^ {j} = \left(\mathbf {h} ^ {\mathbf {e n c}}\right) ^ {\top} s h _ {j _ {| N |}} ^ {e n c} \tag {1}
$$
$$
s c _ {i} ^ {j} = \frac {\exp \left(a _ {i} ^ {j}\right)}{\sum_ {i = 1} ^ {| X _ {t} |} \exp \left(a _ {i} ^ {j}\right)} \tag {2}
$$
$$
c _ {j} = \sum_ {i = 1} ^ {| X _ {t} |} s c _ {i} ^ {j} h _ {i} ^ {\text {e n c}} \tag {3}
$$
Here, the score  $sc_{t}^{j}$  indicates the relevance between info slots  $s_{j}$  and dialogue history. The context vector  $c_{j} \in \mathbb{R}^{d_{hdd}}$  denotes the slot-specific information grabbed from the entire dialogue history. Finally, we obtain the context vectors  $\mathbf{c} = [c_1, c_2, \dots, c_J] \in \mathbb{R}^{d_{hdd} \times J}$  for all  $J$  slots.
4.3 Slot Information Sharing
In the slot information sharing, there is a special matrix called the slot similarity matrix. This matrix controls the information flow among similar slots. We introduce two sharing methods according to their different calculation of the slot similarity matrix: fix combination sharing and the k-means sharing. We will compare the effectiveness of the two methods in Section 6.
4.3.1 Fix Combination Method
We calculate the similarity between every two slots to construct switch matrix. We first compute the cosine similarity over the two slot names and then calculate the similarity over the slot types. Specifically, the slot types can be divided into several categories such as "date", "location". For example, if there are two slots "restaurant-area" and "restaurant-book day", then the similarity in the first part may be high since the two slot names share a common word "restaurant", while the similarity in the second part is quite low: slot "restaurant-area" has a value whose type is "location", and "restaurant-book day" has a value which belongs to "date". Next, the two calculated similarities  $sname$  and  $vtype$  will be integrated with a hyperparameter  $\alpha \in [0,1]$  and we can get a special matrix  $sim \in \mathbb{R}^{J \times J}$  as a result.
$$
s i m = \alpha \cdot s n a m e + (1 - \alpha) \cdot v t y p e \tag {4}
$$
Here, the integration ratio  $\alpha$  actually controls the final similarity of the slots. In Table 2, we show that different choices of this ratio will impact the model's tracking performance.
After that, matrix sim is transformed into the slot similarity matrix  $M$  by the mask mechanism.
$$
M _ {i j} = \left\{ \begin{array}{l l l} 1 & i f & \operatorname {s i m} _ {i j} \geq \beta \\ 0 & i f & \operatorname {s i m} _ {i j} <   \beta \end{array} \right. \tag {5}
$$
Here, hyperparameter  $\beta$  acts as a threshold to decide whether the two slots are similar enough to trigger the sharing switch and open the information path between them.
4.3.2 K-means Sharing Method
Since the fix combination method needs manual efforts to search for the best hyperparameter, we propose another method, K-means Sharing Method, which requires no hyperparameter tuning and can achieve an averagely good performance. In this sharing method, we also compute the slot name similarity  $sname_{ij}$  and the value type similarity  $vtype_{ij}$  between slot  $s_i$  and  $s_j$  as the way in the fix combination one. Then we put vectors  $(sname_{ij}, vtype_{ij})$  onto flat space and divide these vectors into two groups by the k-means clustering algorithm. One group stands for the slot  $s_i$  and  $s_j$  are similar enough, while the other one not similar. The element in  $M_{ij}$  is 1 if they are in similar group, 0 if they are in unsimilar group.
After getting the slot similarity matrix whose value is either 1 or 0, we do the matrix multiplication between the context vectors  $\mathbf{c} = [c_1, c_2, \dots, c_J] \in \mathbb{R}^{d_{hdd} \times J}$  and the slot similarity matrix  $M \in \mathbb{R}^{J \times J}$ . Then we get the integrated vectors  $\mathbf{int} = [int_1, int_2, \dots, int_J] \in \mathbb{R}^{d_{hdd} \times J}$ . These new vectors keep more expressive information for every slot. Specifically,  $int_j$  is calculated as following:
$$
i n t _ {j} = \sum_ {i = 1} ^ {J} c _ {i} \cdot M _ {i j}, M _ {i j} \in \{0, 1 \} \tag {6}
$$
As shown in the above equation,  $int_{j}$  is essentially the integrated result of all related context vectors  $c_{i}$  in  $c$  and the integration is guided by the slot similarity matrix  $M$ . The matrix  $M$  actually plays the role of a switch which controls the information flow between slots and provides a selective integration. For example, this integration makes the data-insufficient slot "attraction-type" receive the information from its related and data-sufficient slot "attraction-name", and helps our model deduce the related value for data-insufficient slots.
6369
The value prediction process of our decoder can be divided into two steps: first, predicting whether the value of a certain slot is constrained by the user; and then extracting the value if the constraint is mentioned in the dialogue.
In the first step, a three-way classifier called slot gate is used and it can map a vector taken from the encoded hidden states  $H_{t}$  to a probability distribution over "ptr", "none", and "dontcare" labels. Once the slot gate predicts "ptr", the decoder will fill the slots with the values extracted from the dialogues. Otherwise, it just fills the slots with "not-mentioned" or "does not care".
In the second step, another GRU is utilized as the decoder. During the decoding step of the  $j$ -th slot, given a sequence of word embeddings  $[w_{j_1}, w_{j_2}, \dots, w_{j_{|N|}}]$ , the GRU transforms them into decoded hidden states  $[h_{j_1}^{dec}, h_{j_2}^{dec}, \dots, h_{j_{|N|}}^{dec}]$  with the slot's integrated vector  $int_j$ :
$$
z _ {k} ^ {j} = \sigma \left(U _ {z 1} w _ {j _ {k}} + U _ {z 2} h _ {j _ {k - 1}} ^ {\text {d e c}}\right) \tag {7}
$$
$$
r _ {k} ^ {j} = \sigma \left(U _ {r 1} w _ {j _ {k}} + U _ {r 2} h _ {j _ {k - 1}} ^ {\text {d e c}}\right) \tag {8}
$$
$$
\tilde {h} _ {k} ^ {j} = \tanh  \left(U _ {1} w _ {j _ {k}} + U _ {2} \left(r _ {k} ^ {j} \circ h _ {j _ {k - 1}} ^ {\text {d e c}}\right)\right) \tag {9}
$$
$$
h _ {j _ {k}} ^ {d e c} = \left(1 - z _ {k} ^ {j}\right) \circ h _ {j _ {k - 1}} ^ {d e c} + z _ {k} ^ {j} \circ \tilde {h _ {k} ^ {j}} \tag {10}
$$
Here,  $|N|$  is the length of the slot sequence and  $int_{j}$  is the initial hidden state input  $h_{j0}^{dec}$ . The integrated vector  $int_{j}$  makes the decoded hidden states contain more information about the dialogue history. So they are more sensitive about whether the value of slot  $j$  is mentioned in the dialogue and where it locates. With the decoded hidden state  $h_{jk}^{dec}$ , the generator computes  $P_{jk}^{gen}$ , the probability of the value generated from the vocabulary list  $E \in \mathbb{R}^{|V| \times d_{hdd}}$  and  $P_{jk}^{copy}$ , the one copied from the interaction history.  $|V|$  is the vocabulary size and  $d_{hdd}$  is the dimension of the hidden state. In the end, we sum the probability  $P_{jk}^{gen}$  and  $P_{jk}^{copy}$  to yield the final prediction  $P_{jk}$ :
$$
P _ {j k} ^ {\text {g e n}} = \operatorname {S o f t m a x} \left(E \cdot \left(h _ {j k} ^ {\text {d e c}}\right) ^ {\top}\right) \tag {11}
$$
$$
P _ {j k} ^ {\text {c o p y}} = \operatorname {S o f t m a x} \left(H _ {t} \cdot \left(h _ {j k} ^ {\text {d e c}}\right) ^ {\top}\right) \tag {12}
$$
$$
P _ {j k} = g _ {j k} \times P _ {j k} ^ {\text {g e n}} + (1 - g _ {j k}) \times P _ {j k} ^ {\text {c o p y}} \tag {13}
$$
$$
g _ {j k} = \operatorname {S i g m o i d} \left(W _ {g} \cdot \left[ h _ {j k} ^ {\text {d e c}}; w _ {j k}; P _ {j k} ^ {\text {c o p y}} \cdot H _ {t} \right]\right) \tag {14}
$$
Here,  $g_{jk}$  is a scalar which controls the model behaviour. It determines whether to generate values from the vocabulary list or copy words from the historical context.
In this section, we first introduce the dataset and the evaluation metrics. We then describe our model's implementation details. Finally, we show our baseline models.
MultiWOZ (Budzianowski et al., 2018) is a fully-labelled collection of human-human written conversations spanning over multiple domains and topics. There are 7,032 multi-domain dialogues consisting of 2-5 domains in MultiWOZ. Because these dialogues have multiple tasks, so the long dialogue history makes state tracking more difficult. Since there are no dialogues from hospital and police domains in validation and testing sets of MultiWOZ, we follow TRADE (Wu et al., 2019) and use five out of the seven domains to train, valid and test, including restaurant, hotel, attraction, taxi and train. These domains involve 30 slots.
We also test our model on a subset of MultiWOZ which only contains the dialogues from the restaurant domain to verify whether our model still works for single-task dialogues.
We evaluate all the models using two metrics, slot accuracy and joint goal accuracy, similar to (Nouri and Hosseini-Asl, 2018):
- Slot accuracy. We use slot accuracy to check whether each single slot in the ground truth dialogue states is correct. The metric only focuses on if the slot requested is correct or not.
- Joint goal accuracy. The joint goal accuracy is used to evaluate whether the user's goal in each turn is captured. Only when every slot in the ground-truth dialogue state is considered and has correct value, can we consider the joint goal is achieved. It is the most important metric in the dialogue state tracking task.
We use the concatenation embedding of GloVe embedding (Pennington et al., 2014) and the character-wise embedding (Hashimoto et al., 2017) in the
Model	MultiWOZ	MultiWOZ(res)
Joint	Slot	Joint	Slot
MDBT	15.57	89.53	17.98	54.99
GLAD	35.57	95.44	53.23	96.54
GCE	36.27	98.42	60.93	95.85
SpanPtr	30.28	93.85	49.12	87.89
TRADE	48.62	96.92	65.35	93.28
SAS	51.03	97.20	67.34	93.83
Table 1: Performances of various models on Multi-WOZ dataset and MultiWOZ (restaurant) dataset.
Model	Joint	Slot
SAS-att-shr	55.52	92.66
SAS-shr	60.68	89.53
SAS(RT shr-0.7, 0.8)	60.59	96.92
SAS(RT shr-0.8, 0.7)	60.78	96.94
SAS(RT shr-0.8, 0.8)	61.04	97.02
SAS(RT shr-0.8, 0.9)	60.54	96.91
SAS(RT shr-0.9, 0.8)	61.47	97.00
SAS(KM shr)	60.92	96.96
SAS(HM shr)	60.28	96.89
Table 2: Results evaluated on the MultiWOZ(except hotel) dataset. "RT shr" means the fix combination sharing method, "KM shr" is the k-means sharing method, and "HM shr" is the human evaluated sharing method. The two numbers after "- represents the integration ratio  $\alpha$  and the threshold  $\beta$  respectively.
experiment. The model is trained with ADAM optimizer (Kingma and Ba, 2014) and a batch size of 32. Both the encoder and the decoder use 400 hidden dimensions. The learning rate is initially set to 0.001, but once the joint goal accuracy does not rise with the training, the network will automatically decrease its learning rate to improve the performance. We apply dropout with 0.2 dropout rate for regularization (Srivastava et al., 2014). Besides that, a word dropout technique is also utilized in the way proposed by (Bowman et al., 2015) which simulates the out-of-vocabulary setting. Our k-means clustering algorithm is implemented with the sklearnn module, and we set all the hyperparameter in k-means algorithm as default.
We compare SAS with several previous methods: MDBT, GLAD, GCE, SpanPtr and TRADE. Based on the classical NBT model, MDBT (Ramadan et al., 2018) extended the task into multiple domains. MDBT makes full use of the semantic simi