SlowGuess/ACL_2020_OCR huggingface dataset:
Yong Shan $^{12}$ , Zekang Li $^{12}$ , Jinchao Zhang $^{3}$ , Fandong Meng $^{3}$ , Yang Feng $^{12*}$ , Cheng Niu $^{3}$ , Jie Zhou $^{3}$
<sup>1</sup> Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
2 University of Chinese Academy of Sciences
$^{3}$  Pattern Recognition Center, WeChat AI, Tencent Inc, China {shanyong18s, lizekang19g, fengyang}@ict.ac.cn {dayerzhang, fandongmeng, chengniu, withtomzhou}@tencent.com
Recent studies in dialogue state tracking (DST) leverage historical information to determine states which are generally represented as slot-value pairs. However, most of them have limitations to efficiently exploit relevant context due to the lack of a powerful mechanism for modeling interactions between the slot and the dialogue history. Besides, existing methods usually ignore the slot imbalance problem and treat all slots indiscriminately, which limits the learning of hard slots and eventually hurts overall performance. In this paper, we propose to enhance the DST through employing a contextual hierarchical attention network to not only discern relevant information at both word level and turn level but also learn contextual representations. We further propose an adaptive objective to alleviate the slot imbalance problem by dynamically adjust weights of different slot durations training. Experimental results show that our approach reaches  $52.68\%$  and  $8.55\%$  joint accuracy on MultiWOZ 2.0 and MultiWOZ 2.1 datasets respectively and achieve new state-of-the-art performance with considerable improvements  $(+1.2$  and  $-98\%)$ .
Recently, task-oriented dialogue systems have attracted increasing attention in both industry and academia due to their broad application for helping users accomplish tasks through spoken interactions (Young, 2002; Young et al., 2013; Gao et al., 2019a). Dialogue state tracking (DST) is an essential part of dialogue management in task-oriented dialogue systems. Given current utterances and dialogue history, DST aims to determine the set of
User: Hello, I'm looking for a restaurant with fair prices.
State: price range=moderate
Sys: OK. There are Golden Wok Chinese restaurant and Nirala which serves Indian food, which is to you like?
User: Are they both have a reasonable price?
Sys: Of course.
User: Please tell me the address of Go on Wok.
State: price range  $= \{m\}$  - value; foo-chinese
Table 1: An simple dialog at the last turn, it is necessary to capture relevant information in dialogue history to correctly predict the value of slot "food", which is underlined. "User" and "Sys" represent user utterance and system response respectively, and the italic text means dialogue states.
that a user tries to inform at each turn which are represented as slot-value pairs (Williams et al., 2013; Henderson et al., 2014a).
As Table 1 shows, the dialogue state is usually dependent on relevant context in the dialogue history, which is proven in previous studies (Sharma et al., 2019; Wu et al., 2019). However, traditional DST models usually determine dialogue states by considering only utterances at current turn (Henderson et al., 2014b; Mrkšić et al., 2017; Zhong et al., 2018; Chao and Lane, 2019) which neglects the use of dialogue history. Recent researches attempt to address this problem through introducing historical dialogue information into the prediction of slot-value pairs. Most of them leverage a naive attention between slots and concatenated historical utterances (Wu et al., 2019; Zhou and Small, 2019; Gao et al., 2019b; Zhang et al., 2019; Le et al., 2020a,b) or only utilize partial history (Ren et al., 2019; Kim et al., 2019; Sharma et al., 2019) or lack direct interactions between slots and history (Ren et al., 2018; Lee et al., 2019; Goel et al., 2019). Briefly, these methods are deficient in exploiting relevant context from dialogue history.
This paper was retracted. For more information, see https://aclanthology.org/2020.acl-main.563.
† Joint work with Pattern Recognition Center, WeChat AI, Tencent Inc.
$^{1}$ Code is available at https://github.com/ictnlp/CHAN-DST
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6322-6333
Furthermore, there are differences in the frequency of different slots and different slot-value pairs. For example, in MultiWOZ 2.0 train set, there are 15384 samples related to the slot "train-day" while 5843 for the slot "attraction-name"; the slot-value pair (attraction-area, center) occurs 5432 times and (taxi-departure, royal spice) occurs only 9 times; etc. We refer to this problem as "slot imbalance", which makes the learning difficulties of different slots varies (Refer to Appendix for details). However, existing approaches usually ignore the slot imbalance problem and treat all slots indiscriminately, which limits the learning of those hard slots and eventually damages the overall performance.
To address the two aforementioned problems, we propose an effective model equipped with a contextual hierarchical attention network (CHAN) to fully exploit relevant context from dialogue history, and an adaptive objective to alleviate the slot imbalance problem. In CHAN, the slot firstly retrieves word-level relevant information from utterances at each turn. Then, these word-level relevant information will be encoded into contextual representations by rich interactions. Finally, the slot aggregates all contextual representations into turnlevel relevant information and then we combine it with word-level relevant information to obtain the outputs. To further enhance the ability to exploit relevant context, we employ a state transition prediction task to assist DST learning. For this slot imbalance problem, our adaptive piecewise can dynamically evaluate the difficulties in an accuracy-sensitive manner and then adaptively adjust the learning weights for different slots. Thus, it can balance the learning of all slots as far as possible.
We evaluate the effectiveness of our model on MultiWOZ 2.0 and Multi-WOZ 2.1 datasets. Experimental results show that our model reaches  $52.68\%$  and  $-55.97\%$  accuracy, outperforming previous state-of-the-art by  $+1.24\%$  and  $+5.98\%$ , respectively. The evaluation study also demonstrates each module's effectiveness in our model. Our contributions are as follows:
- We propose an effective contextual hierarchical attention network to fully exploit relevant context from dialogue history and employ a state transition prediction task to further enhance it.
- We design an adaptive objective to address the slot imbalance problem by dynamically
adjusting the weight of each slot. To the best of our knowledge, our method is the first to address the slot imbalance problem in DST.
- Experimental results show that our model achieves state-of-the-art performance with significant improvements over all previous models.
As shown in Figure 1, the proposed model consists of three components: 1) the contextual hierarchical attention network (CHAN); 2) the state transition prediction module; 3) the adaptive objective. We share all the model parameters for each slot to keep our model universal for all sles.
Given a dialogue  $X = \{(U_1, T), (U_T, R_T)\}$  of  $T$  turns where  $c$  represents utterance and  $R_t$  represents system response of turn  $t$ , we define the dialogue state as each turn  $t$  as  $\mathcal{B}_t = \{(s, v_t), s \in S\}$  where  $\mathcal{S}$  is a set of slots and  $v_t$  is the corresponding value of the slot  $s$ . Following Lee et al. (2019), we use the term "slot" to refer to the concatenation of a domain name and a slot name in order to represent both domain and slot information. For example, "restaurant-food". Similar to (Ren et al., 2018; Lee et al., 2019), we decompose the dialogue state tracking to a multi-label classification problem where we score each value with slot-related features in a non-parametric way and then choose the best candidate. We also add a literally "none" into the value set of each slot to represent that no corresponding value is tracked.
2.2 Contextual Hierarchical Attention Network
Recently the pre-trained BERT language model (Devlin et al., 2019) shows powerful ability in universal contextual semantics representation, thus we employ BERT to encode utterances, slots and values. To better retrieve relevant context from dialogue history, we devise Slot-Word Attention and Slot-Turn Attention to query both relevant keywords and turns. Specifically, we exploit a Context Encoder between word-level and turn-level attention to capture contextual representations of relevant information from dialogue history. Furthermore, we devise a Global-Local Fusion Gate to balance the information from global context and local utterances.
6323
Figure 1: The architecture of our model. At turn  $t$ , the slot retrieves relevant information along  $\{1, \dots, t\}$  turns at both word level and turn level. Specifically, we utilize a context encoder between word levels and turn level to capture the relationships between historical relevant information. Finally, we combine the global relevant context  $c_{s,t}^{turn}$  and local dialogue information  $c_{s,t}^{word}$  as outputs. During training, we first run the D'F task and the state transition prediction task jointly, then fine-tune our model with the adaptive objective.
Sentence Encoder. BERT leverages a special token [CLS] to aggregate the whole representation of a sentence and a special token [SEP] to indicate the end of a sentence. For user utterance  $U_{t} = \{w_{1}^{u},\dots,w_{l}^{u}\}$  and system response  $R_{t} = \{w_{1}^{r},\dots,w_{l^{\prime}}^{r}\}$  at dialogue turn  $t$ , we concatenate them with special tokens and encode them into contextual word representations  $\mathbf{h}_t$  as follows:
$$
\mathbf {h} _ {t} = \operatorname {B E R T} _ {\text {f i n e t u n e}} ([ F, U _ {t} ], \quad (1)
$$
where  $\mathrm{BERT}_{\mathrm{finetune}}$  means that it will be tuned during training. The fore,  $\mathrm{BERT}_{\mathrm{finetune}}$  will learn a corresponding generalization of sentence representation and adapt to dialogue state tracking task.
For slot  $s$  we define  $v_{t}$ , we adopt another pretrained BFI  $\Gamma_{fix}$  to encode them into contextual semantics vectors  $\mathbf{n}^{s}$  and  $\mathbf{A}_{t}^{v}$  respectively. Different from utterances, we use the output vector of the special token [CLS] to obtain the whole sentence representation:
$$
\mathbf {h} ^ {s} = \operatorname {B E R T} _ {\text {f i x e d}} (s) \tag {2}
$$
$$
\mathbf {h} _ {t} ^ {v} = \mathrm {B E R T} _ {\text {f i x e d}} (v _ {t})
$$
where the weights of  $\mathrm{BERT}_{\text{fixed}}$  are fixed during training thus our model can be scalable to any unseen slots and values with sharing the original BERT representation.
Slot-Word Attention. The slot-word attention is a
multiHead (MultiHead(Q, K, V)), which takes a query matrix Q, a key matrix K and a value matrix V as inputs. Refer to (Vaswani et al., 2017) for more details. For each slot s, the slot-word attention summarizes word-level slot-related information from each turn t into a d-dimensional vector  $\mathbf{c}_{s,t}^{word}$ , which can be determined as follows:
$$
\mathbf {c} _ {s, t} ^ {\text {w o r d}} = \operatorname {M u l t i H e a d} \left(\mathbf {h} ^ {s}, \mathbf {h} _ {t}, \mathbf {h} _ {t}\right) \tag {3}
$$
Context Encoder. The context encoder is a unidirectional transformer encoder, which is devised to model the contextual relevance of the extracted word-level slot-related information among  $\{1,\dots,t\}$  turns. The context encoder contains a stack of  $N$  identical layers. Each layer has two sub-layers. The first sub-layer is a masked multi-head self-attention (MultiHead), in which  $\mathbf{Q} = \mathbf{K} = \mathbf{V}$ . The second sub-layer is a position-wise fully connected feed-forward network (FFN), which consists of two linear transformations with a ReLU activation (Vaswani et al., 2017).
$$
\operatorname {F F N} (x) = \max  \left(0, x W _ {1} + b _ {1}\right) W _ {2} + b _ {2} \tag {4}
$$
Formally, the output of the context encoder  $\mathbf{c}_{s,\leq t}^{ctx}$  can be denoted as follows:
$$
\mathbf {m} ^ {n} = \operatorname {F F N} (\operatorname {M u l t i H e a d} (\mathbf {m} ^ {n - 1}, \mathbf {m} ^ {n - 1}, \mathbf {m} ^ {n - 1}))
$$
$$
\mathbf {m} ^ {0} = \left[ \mathbf {c} _ {s, 1} ^ {\text {w o r d}} + \operatorname {P E} (1), \dots , \mathbf {c} _ {s, t} ^ {\text {w o r d}} + \operatorname {P E} (t) \right]
$$
$$
\mathbf {c} _ {s, \leq t} ^ {c t x} = \mathbf {m} ^ {N} \tag {5}
$$
6324
where  $\mathbf{m}^n$  is the output of the  $n$ -th layer of context encoder and  $\mathrm{PE}(\cdot)$  denotes positional encoding function. Note that residual connection and layer normalization are omitted in the formula.
Slot-Turn Attention. To retrieve turn-level relevant information from contextual representation, we devise a slot-turn attention which is the multi-head attention as follows:
$$
\mathbf {c} _ {s, t} ^ {\text {t u r n}} = \operatorname {M u l t i H e a d} \left(\mathbf {h} ^ {s}, \mathbf {c} _ {s, \leq t} ^ {\text {c t x}}, \mathbf {c} _ {s, \leq t} ^ {\text {c t x}}\right) \tag {6}
$$
Therefore, the model can access word-level and turn-level relevant information from the historical dialogues.
Global-Local Fusion Gate. To balance the information of global context and local utterances, we propose to dynamically control each proportion of contextual information and current turn information so that the model can not only benefit from relevant context but also keep a balance between global and local representations. Similar to Hochreiter and Schmidhuber (1997), we leverage a fusion gate mechanism, which computes a weight to decide how much global and local information should be combined according to  $c_{s,t}^{word}$  and  $c_{s,t}^{turn}$ . It can be defined as follows:
$$
g _ {s, t} = \sigma \left(\mathbf {W} _ {g} \odot \left[ \mathbf {c} _ {s, t} ^ {\text {w o r d}}; \mathbf {c} _ {s, t} ^ {\text {t u r n}} \right]\right) \tag {7}
$$
$$
{\mathbf {c} _ {s, t} ^ {g a t e}} = {g _ {s, t} \otimes \mathbf {c} _ {s, t} ^ {w o r d} + (1 - g _ {s, t}) \otimes \mathbf {c} _ {s, t} ^ {t u r n}}
$$
where  $\mathbf{W}_g\in \mathbb{R}^{2d\times d}$  are parameters,  $\sigma$  means sigmoid activation function,  $\odot$  and  $\otimes$  mean the joint-wise and element-wise multiplication operators.
Finally, we use a linear projection to obtain query results with layer normalization and dropout
$$
\mathbf {o} _ {s, t} = \text {L a y e r N o r m} (\operatorname {L i n} \quad \mathrm {r} (\operatorname {D r o p o} ^ {\left(\mathbf {c} _ {t} ^ {\text {g a t e}}\right)})) (8)
$$
We follow Ren et al. (2018) to adopt L2 norm to compute the distance. Therefore, the probability distribution of the  $v_{t}$  as the training objective can be defined as:
$$
p (v _ {t} | U _ {\leq t}, h _ {\underline {{-}}} \quad s) = \frac {\exp (- \| o _ {s , t} - h _ {t} ^ {v} \| _ {2})}{\sum_ {v ^ {\prime} \in \mathcal {V} _ {s}} \exp (- \| o _ {s , t} - h _ {t} ^ {v ^ {\prime}} \| _ {2})}
$$
$$
\mathcal {L} _ {d s t} = \sum_ {s \in \mathcal {S}} \sum_ {t = 1} ^ {T} - \log \left(p \left(\hat {v} _ {t} \mid U _ {\leq t}, R _ {\leq t}, s\right)\right) \tag {9}
$$
where  $\mathcal{V}_s$  is the candidate value set of slot  $s$  and  $\hat{v}_t\in \mathcal{V}_s$  is the ground-truth value of slot  $s$ .
2.3 State Transition Prediction
To better capture relevant context, we further introduce an auxiliary binary classification task to jointly train with DST: State Transition Prediction
(STP), which is to predict if the value for a slot is updated compared to previous turn. This module reads  $\mathbf{c}_{s,t - 1}^{gate}$  and  $\mathbf{c}_{s,t}^{gate}$  as inputs and the transition probability  $p_{s,t}^{stp}$  can be calculated as follows:
$$
\mathbf {c} _ {s, t} ^ {s t p} = \tanh  \left(\mathbf {W} _ {c} \odot \mathbf {c} _ {s, t} ^ {\text {g a t e}}\right) \tag {10}
$$
$$
{p _ {s, t} ^ {s t p}} = {\sigma (\mathbf {W} _ {p} \odot [ \mathbf {c} _ {s, t} ^ {s t p}; \mathbf {c} _ {s, t - 1} ^ {s t p} ])}
$$
where  $\mathbf{W}_c\in \mathbb{R}^{d\times d}$ $\mathbf{W}_p\in \mathbb{R}^{2d}$  are parameters. Note that when  $t = 1$  , we simply concatenate  $\mathbf{c}_{s,t}^{stp}$  with zero vectors.
For this task, we calculate the binary cross entropy loss between ground-truth transition labels  $y_{s,t}^{stp}$  and the transition probability  $p_{s,t}^{stp}$ , which is defined as follows:
$$
\mathcal {L} _ {s t p} = \sum_ {s \in \mathcal {S}} \sum_ {t = 1} ^ {T} - \frac {x p}{p} \cdot \log \left(p _ {x}\right) \tag {11}
$$
2.4 Adaptive Objective
Essentially, the not in- dence problem can be considered as a kind of class - balance because there is an imbalance among both different slots and different samples. Instead of treating all slots indiscriminately, it is important to balance the learning of different slots. Recently, Lin et al. (2017) propose a so- nampling method, Focal Loss, to re-weight the losses of different classes.
Based by their work, we design a novel adaptive objective for DST which evaluates the difficulty from each slot's accuracy on the validation set and adaptively adjusts the weight of each slot during optimization. We define the accuracy of slot  $s$  on validation set as  $acc_{s}^{val}$ . Our adaptive objective is based on the following intuitions:
(1) If  $acc_{s}^{val} \leq acc_{s'}^{val}$ ; then slot  $s$  is more difficult than slot  $s'$ . Suppose this slot-level difficulty is defined as  $\alpha$ ; then
$$
\alpha_ {s} = \frac {1 - a c c _ {s} ^ {\text {v a l}}}{\sum_ {s ^ {\prime} \in \mathcal {S}} 1 - a c c _ {s ^ {\prime}} ^ {\text {v a l}}} \cdot | \mathcal {S} | \tag {12}
$$
(2) Suppose there are two samples  $\{(U_t, R_t), (s, v_t)\}$  and  $\{(U_{t'}, R_{t'})\}$ . If the former confidence is lower than the latter, then sample  $\{(U_t, R_t), (s, v_t)\}$  is more difficult than  $\{(U_{t'}, R_{t'})\}$ . Suppose this sample-level difficulty is defined as  $\beta$ ; then
$$
\beta (s, v _ {t}) = (1 - p (s, v _ {t})) ^ {\gamma} \tag {13}
$$
where  $p(s,v_t)$  is the confidence of sample  $\{(U_t,R_t),(s,v_t)\}$  and  $\gamma$  is a hyper-parameter.
6325
Thus, the adaptive objective is defined as follows:
$$
\mathcal {L} _ {\text {a d a p t}} (s, v _ {t}) = - \alpha_ {s} \beta (s, v _ {t}) \log (p (s, v _ {t})) \tag {14}
$$
Focal Loss assigns static learning weights on slots and doesn't change them anymore during the whole training. Compared to Focal Loss, our adaptive objective can fit data better by dynamically evaluate the difficulties in an accuracy-sensitive manner and then adaptively control the learning weights for different slots, which is proved in our experiments. If the difficulty of slot  $s$  is greater than the average difficulty of all slots,  $\alpha_{s}$  would increase and enlarge the loss of  $s$ . Similarly, the optimization of sample  $\{(U_t,R_t),(s,v_t)\}$  with a low confidence  $p(s,v_t)$  would be encouraged by a larger loss. When an epoch ends, the adaptive objective re-evaluates the difficulty of each slot and updates  $\alpha_{s}$ . Therefore, it can not only encourage the optimization of those hard slots and samples but also balance the learning of all slots.
2.5 Optimization
In our model, we firstly jointly train the DST and STP tasks to convergence and then fine-tune DST task with the adaptive objective.
During joint training, we optimize the sum of these two loss functions as following:
$$
\mathcal {L} _ {\text {j o i n t}} = \mathcal {L} _ {\text {d s t}} + \mathcal {L} _ {\text {s t p}} \tag {13}
$$
At the fine-tuning phase, we adopt the adaptive objective to fine-tune DST task as follows:
$$
\mathcal {L} _ {\text {f i n e t u n e}} = \sum_ {s \in \mathcal {S}} \sum_ {t = 1} ^ {T} \mathcal {L} _ {\text {a d a r}} (\hat {v} _ {t}) \tag {10}
$$
3.1 Datasets & Metrics
Hotel	Atrition	Restaurant	Taxi
Slots	price, type, parking, stay, day, people, area, stars, internet, name	location, day, arrive at, leave at, people	area, name, type	food, price, area, name, time, day, people	destination, departure, arrive by, leave by
Train	3381	3103	2717	3813	1654
Valid	416	484	401	438	207
Test	394	494	395	437	195
Table 2: The dataset statistics of MultiWOZ 2.0 & 2.1.
We evaluate our model on MultiWOZ 2.0 (Budzianowski et al., 2018) and MultiWOZ 2.1 (Eric et al., 2019), which are two of the largest
public task-oriented dialogue datasets, including about 10,000 dialogues with 7 domains and 35 domain-slot pairs. MultiWOZ 2.1 shares the same dialogues with MultiWOZ 2.0 but it fixed previous annotation errors. The statistics are shown in Table 2. Following (Wu et al., 2019), we use only 5 domains {restaurant, hotel, train, attraction, taxi} excluding hospital and police since these two domains never occur in the test set. We preprocess the datasets following (Lee et al., 2019) $^2$ .
We use joint accuracy and slot accuracy as our evaluation metrics. Joint accuracy is the accuracy of the dialogue state of each turn and a dialogue state is evaluated correctly only if all the values of slots are correctly predicted. Slot accuracy only considers individual slot-level accuracy.
We compare our results with the following competitive baselines:
DSTreader proposes a node DST as a machine reading comprehension task and extract spans from dialogue history (Zao et al., 2019b).
GI D-KCFS uses heuristic rule to extract relevant returns and lets slot-value pairs to query relevant content from them (Sharma et al., 2019).
HySys uses a hierarchical encoder and takes a rigid way combining both predefined-ontology and open-vocabulary settings (Goel et al., 2019).
TRADE encodes the whole dialogue context and decodes the value for every slot using a copy-augmented decoder (Wu et al., 2019).
DST-QA proposes to model DST as a question answering problem and uses a dynamically-evolving knowledge graph to learn relationships between slot pairs (Zhou and Small, 2019).
SOM-DST considers the dialogue state as an explicit fixed-size memory and proposes a selectively overwriting mechanism (Kim et al., 2019).
SUMBT exploits BERT as the encoder of the utterances, slots and values. It scores every candidate slot-value pair in a non-parametric manner using a distance measurement (Lee et al., 2019).
DST-picklist performs matchings between candidate values and slot-context encoding considering all slots as picklist-based slots (Zhang et al., 2019).
GLAD-RCFS, HyST, SUMBT, DST-picklist are predefined-ontology models as well as our model and DSTreader, TRADE, DST-QA, SOM-DST are open-vocabulary models.
$^{2}$ https://github.com/SKTBrain/SUMBT
6326
Model	Ontology	MultiWOZ 2.0	MultiWOZ 2.1
Joint (%)	Slot (%)	Joint (%)	Slot (%)
DSTreader (Gao et al., 2019b)	×	39.41	-	36.40*	-
GLAD-RCFS (Sharma et al., 2019)	✓	46.31	-	-	-
HyST (Goel et al., 2019)	✓	42.33	-	38.10*	-
TRADE (Wu et al., 2019)	×	48.60	96.92	45.60*	-
DST-QA (Zhou and Small, 2019)	×	51.44	97.24	51.17	97.21
SOM-DST (Kim et al., 2019)	×	51.38	-	52.57	-
SUMBT (Lee et al., 2019)	✓	48.81†	97.33†	52.75‡	97.56‡
DST-picklist (Zhang et al., 2019)	✓	-	-	53.30	-
Our Model	✓	52.68	97.69	58.55	98.14
We employ the pre-trained BERT model that has 12 layers of 784 hidden units and 12 self-attention heads  ${}^{3}$  . For the multi-head attention, we set heads count and hidden size to 4 and 784,respectively. For the context encoder, we set the transformer layers to 6. We set the max sequence length of all inputs to 64 and the batch size to 32. In all training, we use Adam optimizer (Kingma and Ba, 2015) and set the warmup proportion to 0.1. Specifically, in the joint training phase, we set the peak learning rate to 1e-4. At the fine-tuning phase, we set  $\gamma$  to 2,peak learning rate to 1e-5. The training stopped early when the validation loss was not improved for 15 consecutive epochs. For all experiments, we report the mean joint accuracy over all up different random seeds to reduce statistical errors.