SlowGuess/ACL_2020_OCR huggingface dataset:
Yi Huang, Junlan Feng, Min Hu, Xiaoting Wu, Xiaoyu Du, Shuo Ma  
JIUTIAN Team, China Mobile Research
{huangyi,fengjunlan,humin}@chinamobile.com {wuxiaoting,duxiaoyu,mashuo}@chinamobile.com
A Dialogue State Tracker (DST) is a core component of a modular task-oriented dialogue system. Tremendous progress has been made in recent years. However, the major challenges remain. The state-of-the-art accuracy for DST is below  $50\%$  for a multi-domain dialogue task. A learnable DST for any new domain requires a large amount of labeled indomain data and training from scratch. In this paper, we propose a Meta-Reinforced Multi-Domain State Generator (MERET). Our first contribution is to improve the DST accuracy. We enhance a neural model based DST generator with a reward manager, which is built on policy gradient reinforcement learning (RL) to fine-tune the generator. With this change, we are able to improve the joint accuracy of DST from  $48.79\%$  to  $50.91\%$  on the MultiWOZ corpus. Second, we explore to train a DST meta-learning model with a few domains as source domains and a new domain as target domain. We apply the model-agnostic meta-learning (MAML) algorithm to DST and the obtained meta-learning model is used for new domain adaptation. Our experimental results show this solution is able to outperform the traditional training approach with extremely less training data in target domain.
A Dialogue State Tracker (DST) is a core component of a modular task-oriented dialogue system (Young et al., 2013). For each dialogue turn, a DST module takes the user utterance and the dialogue history as input, and outputs a belief estimate of the dialogue state. The dialogue state as of today is simplified as a set of requests and goals, both of which are represented as (slot, value) pairs such as (area, centre), (food, Chinese) for a user request I'm looking for a Chinese restaurant in the centre of the city. A highly accurate DST is crucial to ensure
Figure 1: An example of dialogue state tracking process for booking a hotel, looking for an attraction and booking a taxi between them. Each turn contains a user utterance (grey) and a system utterance (blue). The dialogue state tracker (yellow) tracks all the (domain, slot, value) until the current turn. Blue color texts indicate mentions of slot values appeared at that turn. Best viewed in color.
the quality and smoothness of a human-machine dialogue.
Budzianowski et al. (2018) recently introduced a multi-domain dialogue dataset Multi-domain Wizard-of-Oz (MultiWOZ), which is more than one order of magnitude larger than all previous annotated task-oriented corpora with around 10k dialogues and involves more than 7 domains. A domain of a task-oriented system is often defined by an ontology, which defines all entity attributes called slots and all possible values for each slot. MultiWOZ presents conversation scenarios much similar to those in real industrial applications. Figure 1 shows an example of a multi-domain dialogue, where a user starts a conversation about hotel reservation and moves on to look for attractions nearby
7109
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7109-7118 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
of his interest. It adds a layer of complexity to the DST and brings new challenges.
The first new challenge is how to appropriately model DST for a multi-domain dialogue task. Multi-domain DST is in its infancy before MultiWOZ (Rastogi et al., 2017). Most previous work on DST focus on one given domain (Henderson et al., 2013, 2014; Mrkšić et al., 2017; Zhong et al., 2018; Korpusik and Glass, 2018; Liu et al., 2019). As Wu et al. (2019) pointed out, to process the MultiWOZ data, the DST model has to determine a triplet (domain, slot, value) instead of a pair (slot, value) at each turn of dialogue. MultiWOZ contains 30 (domain, slot) pairs over 4,500 possible slot values in total. The prediction space is significantly larger. This change seems quantitative. However, it challenges the foundation of most successful DST models, where DST is casted as a neural model based classification problem, each (slot, value) pair is an independent class and the number of classes is relatively limited. When the number of classes is large enough as the case in MultiWOZ, classification-based approaches are not applicable. In real industry scenarios, the prediction space is even larger and it is often not possible to have full ontology available in advance (Xu and Hu, 2018). It's hard to enumerate all possible values for each slot. The second challenge is how to model the commonality and differences among domains. The number of domains is unlimited in real-life. It won't be able to scale up if each new domain requires a large amount of annotated data.
To overcome these challenges, Wu et al. (2019) proposed a TRAnsferable Dialogue statE generator (TRADE) that generates dialogue states from utterances using a copy mechanism, facilitating knowledge transfer between domains. The prominent difference from previous one-domain DST models is that TRADE is based on a generation approach instead of a close-set classification approach. The generation model parameters are shared among various domains and slots. TRADE is able to help boost the DST accuracy up to  $48.62\%$  with the MultiWOZ corpus. It is obvious this accuracy is far from being acceptable.
In this paper, we are motivated to enhance this generation-based approach for two objectives, higher accuracy and better domain adaptability. To improve DST accuracy, we propose a new framework which contains the state generator and reward manager. The state generator follows the same setup
of TRADE. The Reward Manager calculates the reward to fine-tune the generator through policy gradient reinforcement learning (PGRL). We use the reward manager to help the generator alleviate the objective mismatch challenge. Objective mismatch is a limitation of encoder-decoder generation approaches, where the training process is set to maximize the log likelihood, but it doesn't assure producing the best results on discrete evaluation metrics such as the DST accuracy. Since MultiWOZ provides data for multiple domains, it enables us to study the long-standing domain adaptability problem. It is a hope we can train a general DST model from multi-domain data and this model can be adapted to a new domain with minimal examples from a new domain. We apply the meta-learning algorithm, MAML, for this study. Our key contributions in this paper are as follows:
- We propose a new framework as the DST model, which contains a neural model based DST generator and a reward manager.
- With our proposal, we are able to improve the joint accuracy of DST from  $48.79\%$  to  $50.91\%$ , which is  $2.12\%$  absolute improvement over the latest state-of-the-art on the MultiWOZ corpus.
- We apply MAML to train a meta-learning DST model with a few domains as the training domains and a new domain as the testing domain. Our experimental results show this solution is able to outperform the traditional training approach with only  $30\%$  of the indomain training data.
- To our knowledge, we are the first to apply RL and MAML into DST.
2 Model MERET
The overview of our model is illustrated in Figure 2. It consists of a generator model and a reward manager.
2.1 The Generator
In this paper, we take TRADE as our baseline. The TRADE model comprises three components: (1) an utterance encoder, (2) a context-enhanced slot classifier, (3) a state generator. We briefly describe the TRADE model in this Section.
The utterance encoder encodes dialogue utterances into a sequence of fixed-length vectors.
Figure 2: The architecture of the proposed MERET model, which contains a Generator and a Reward Manager in general. The Generator includes (a) an utterance encoder, (b) a context-enhanced slot classifier, and (c) a state generator. The Reward Manager calculates the reward values based on the reward functions to fine-tune the generator through PGRL.
TRADE uses Bi-GRU (Chung et al., 2014), to encode. Instead of initializing by concatenating GloVe embeddings (Pennington et al., 2014), our model explore to use BERT (Devlin et al., 2019) as embedding model. We denote a sequence of dialogue turns as a matrix  $X_{t} = [U_{t - l},R_{t - l},\dots,U_{t},R_{t}] \in \Re^{|X_{t}| \times d_{emb}}$ , where  $l$  is the length of the dialogue history selected,  $U$  is the user turn,  $R$  represents the system response and  $d_{emb}$  indicates the turn-level embedding size. The encoder encodes  $X_{t}$  into a hidden matrix  $H_{t} = [h_{1}^{enc},\dots,h_{|X_{t}|}^{enc}] \in \Re^{|X_{t}| \times d_{hdd}}$ ,  $hdd$  is the hidden size.
The state generator uses GRUs as the decoder, which takes the embedding of the  $j$ th (domain,slot) pair as well as the  $k$ th word as input and outputs a hidden vector  $h_{jk}^{dec}$  at the  $k$ th decoding step. This hidden vector is then mapped to distribution over the vocabulary  $V$  and over the dialogue history as shown in Eq (1).
$$
P _ {j k} ^ {\text {v o c a b}} = \operatorname {S o f t m a x} \left(E \cdot \left(h _ {j k} ^ {\text {d e c}}\right) ^ {\top}\right) \in \Re^ {| V |} \tag {1}
$$
$$
P _ {j k} ^ {\text {h i s t o r y}} = \operatorname {S o f t m a x} \left(H _ {t} \cdot \left(h _ {j k} ^ {\text {d e c}}\right) ^ {\top}\right) \in \Re^ {| X _ {t} |}
$$
These two distributions are combined as Eq (2) as the final results,
$$
P _ {j k} ^ {\text {f i n a l}} = p _ {j k} ^ {\text {g e n}} \times P _ {j k} ^ {\text {v o c a b}} + \left(1 - p _ {j k} ^ {\text {g e n}}\right) \times P _ {j k} ^ {\text {h i s t o r y}} \tag {2}
$$
The context-enhanced slot classifier takes as input  $H_{t}$  and classifies it into one of the three class-
es: ptr, none, dontcare. With a linear layer parameterized by  $W_{g} \in \Re^{3 \times d_{hdd}}$ , the slot classifier for the jth (domain, slot) pair is defined as
$$
G _ {j} = \operatorname {S o f t m a x} \left(W _ {g} \cdot \left(P _ {j 0} ^ {\text {h i s t o r y}} \cdot H _ {t}\right) ^ {\top}\right) \in \Re^ {3} \tag {3}
$$
If this slot classifier determines none or dontcare, the system ignores any output from the state generator.
Optimization is performed jointly for both the state generator and the slot classifier. The cross-entropy loss is used for both, with  $L_{s}$  representing the loss for the slot classifier and  $L_{g}$  for the generator. They are combined with hyper-parameters  $\eta$  and  $\sigma$ .
$$
L _ {m i x} = \eta L _ {s} + \sigma L _ {g} \tag {4}
$$
2.2 A Reward Manager
Generally, the cross-entropy loss is used to train a generator. In our task, the true words  $Y_{j}^{label}$  is used and the cross-entropy loss can be defined as:
$$
\operatorname {l o s s} _ {g} = - \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {| Y _ {j} |} \log \left(P _ {j k} ^ {\text {f i n a l}} \cdot \left(y _ {j k} ^ {\text {l a b e l}}\right) ^ {\top}\right) \tag {5}
$$
where  $y_{jk}^{label}$  is the ground truth of the value word for the  $jth$  (domain, slot) pair.
In this paper, we propose a RL-based Reward Manager to work the generator. The Reward Manager is used for calculating the reward to fine-tune the Generator through PGRL.
7111
The specific modeling process of reinforcement learning adaptation for DST task is summarized in Algorithm 1: We treat the Generator as the target agent to be trained. The agent interacts with an external environment (utterances, domains, slots and reward manager) by taking actions and receiving environment state and reward. The actions are the choices of tokens for slot value that generates for any given (domain, slot) pair. The action space is the vocabulary. Following each action, the reward manager calculates a reward by comparing the generated token to the corresponding ground-truth token. When reaching the last decoding step, the agent updates its parameters towards maximizing the expected reward. RL loss is defined as follows:
$$
L _ {r l} = - \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {| Y _ {j} |} r \left(y _ {j k} ^ {s}\right) \log \left(P ^ {f i n a l} \left(y _ {j k} ^ {s}\right)\right) \tag {6}
$$
where  $y_{jk}^{s}$  is a token sampled from the vocabulary probability distribution and  $r(y_{jk}^{s})$  means the reward for the sampled token  $y_{jk}^{s}$ , computed by a reward function. Intuitively, the loss function  $L_{rl}$  enlarges the probability of the sampled  $y_{jk}^{s}$  if it obtains a higher reward for the  $kth$  token in  $jth$  (domain, slot) pair.
We also define a combined loss function:
$$
L = \mu L _ {r l} + \lambda L _ {m i x} \tag {7}
$$
where  $L_{rl}$  is defined as the reinforcement learning loss,  $L_{mix}$  is the cross-entropy loss from TRADE,  $\mu$  and  $\lambda$  are the combined hyper-parameters. Algorithm 1 shows how this method works.
3 MAML-adaptive DST
The traditional paradigm of supervised learning is to train a model for a specific task with plenty of annotated data. Meta-learning aims at learning new tasks with few steps and little data based on existing tasks. MAML (Finn et al., 2017) is the most popular meta-learning algorithm. It has been successfully employed in various tasks. We propose to apply MAML to perform dialogue state tracking for new domains. The MAML algorithm tries to build an internal representation of multiple tasks and maximize the sensitivity of the loss function when applied to new tasks, so that small update of parameters could lead to large improvement of new task loss value. In this paper, we explore how it works with DST, a key component in task-oriented dialogue systems.
Algorithm 1 REINFORCE algorithm
Input: Dialogue history sequence  $X$ , ground-truth output slot value sequences  $Y$ , a pre-trained model  $\pi_{\theta}$ .
Output: Trained model  $\pi_{\theta'}$  with REINFORCE algorithm.
1: Training Steps:
2: Initialize  $\pi_{\theta}$  with random weights  $\theta$
3: Pre-train  $\pi_{\theta}$  using cross-entropy loss of generator and classifier on dataset  $(X,Y)$ ;
4: Initialize  $\pi_{\theta^{\prime}} = \pi_{\theta}$
5: while not done do
6: Select a batch of size  $N$  from  $X$  and  $Y$ ;
7: for each slot do
8: Sample  $\{Y^{s} = (y_{1}^{s},\dots ,y_{|Y_{j}|}^{s})\}_{1}^{N}$  from the final probability distribution of vocabulary;
9: Compute reward  $\{r(y_1^s),\dots ,r(y_{|Y_j|}^s)\}_{1}^{N}$  defined in the Reward Manager;
10: end for
11: Compute  $L_{rl}$  and  $L$  using Eq (6) and Eq (7);
12: Update the parameters of network with learning rate  $\rho, \theta' \gets \theta' + \rho \nabla_{\theta'} L_{\theta'}$ ;
13: end while
14: Testing Steps:
15: for batch of  $X$  and  $Y$  do
16: Generate the output  $\widehat{Y}$
17: end for
18: return The evaluated model  $\pi_{\theta'}$ ;
MAML is compatible for any model training based on gradient descent. We can denote the baseline model as  $M$ . Training a typical gradient descent model  $M$  involves (1) providing training data and initializing parameters of  $M$ ; (2) computing a given objective loss; (3) applying gradient descent to the loss to update  $M$  parameters. With MAML, the training steps become: (1) Initialize  $M$  and making  $nd$  copies of  $M$  to be  $M_d'$ ; (2) Select training data from each domain and updating  $M_d'$  parameters based on gradient descent and a loss function; (3) Calculate a loss for each domain with their updated temporary model  $M_d'$ ; (4) Sum up the new loss from each training domain to be a total loss; (5) Update parameters of the original  $M$  based on the total loss; (6) Repeat above steps until  $M$  converges.
Algorithm 2 shows step-by-step how MAML combines with our model MERET. Suppose we consider  $nd$  dialogue domains, we take  $ntr$  do
7112
mains as source domains for meta-training and nts domains as target domains for meta-testing. For each source domain, we divide the source domain data into  $D_{d}^{train}$  as the support dataset and  $D_{d}^{valid}$  as the query dataset,  $d$  is the domain index.  $\alpha, \beta$  are two hyper-parameters for MAML,  $\alpha$  as the learning rate for each domain and  $\beta$  as the learning rate for meta-learning update.
There are two cycles. The outer cycle is for meta-learning, updating model parameters of  $M$ . The inner cycle is for task learning, updating the temporary model  $M_d'$  of each domain  $d$ . For task learning, we select  $K$  examples from  $D_d^{train}$  for each domain  $d$ , evaluate the gradient of the loss function as Eq (7), update the parameters  $\theta_d'$  with respect to the  $K$  examples (Step 4). After each domain model is updated once, the  $M$  model parameters are updated using the sum of the loss with respect to  $K'$  examples sampled from each  $D_d^{valid}$ . Specifically, we sum the loss of  $M_d'$  in each domain to obtain the meta loss  $L_M$ ,
$$
L _ {M} = \sum_ {d} L _ {d} \left(M _ {d} ^ {\prime}, D _ {d} ^ {v}\right) \tag {8}
$$
Finally, we minimize the meta loss for updating the current model  $M$  until an ideal meta-learned model  $M$  is achieved,
$$
M \leftarrow M - \beta \nabla_ {M} \sum_ {d} L _ {d} \left(M _ {d} ^ {\prime}, D _ {d} ^ {v}\right) \tag {9}
$$
To adapt to a new domain, we start with the metalearned model  $M$  instead of initializing randomly, new-domain training data is used to update model parameters as multiple batches and the learnt task model is fit for the new domain.
4.1 Dataset and Evaluation Matrix
In this paper, we use MultiWOZ as our training and testing corpus. MultiWOZ is a fully-labeled collection of human-human written conversations spanning over multiple domains and topics. It contains 8438 multi-turn dialogues with on average 13.7 turns per dialogue. It has 30 (domain, slot) pairs and over 4,500 slot values. We use the most frequent five domains (restaurant, hotel, attraction, taxi, train) in our experiments.
Two common metrics to evaluate DST models are joint goal accuracy and slot accuracy. Joint accuracy measures the accuracy of dialogue states, where a dialogue state is correctly predicted only if
Algorithm 2 MAML algorithm
Input:  $D_{d}^{train}$ ;  $D_{d}^{valid}$ ;  $\alpha$ ;  $\beta$ .
Output: Trained model  $M$  with MAML algorithm.
1: while not done do
2: for each domain  $d$  do
3: Select a batch of size from  $D_{d}^{train}$  and  $D_{d}^{valid}$  to get  $D_{d}^{t}$  and  $D_{d}^{v}$ ;
4: Pre-update model with gradient descent:  $M_d' \gets M - \alpha \nabla_M L_d(M, D_d^t)$
5: Compute  $L_{d}(M_{d}^{\prime},D_{d}^{v})$  using  $D_{d}^{v}$
6: end for
7: Update the current model  $M$ :
$$
M \gets M - \beta \nabla_ {M} \sum_ {d} L _ {d} (M _ {d} ^ {\prime}, D _ {d} ^ {v})
$$
8: end while
9: return meta-learned model  $M$ ;
all the values of for all the (domain, slot) pairs are correctly predicted. Slot accuracy is the accuracy of the (domain, slot, value) tuples. Joint accuracy is a more challenging metric.
For all experiments, we choose Bi-GRU networks with a hidden size of 768 to be the encoder and the decoder. The model is optimized using Adam (Kingma and Ba, 2015) with a learning rate of 0.001. We reduce the learning rate to half if the validation loss increases. We set the batch (Ioffe and Szegedy, 2015) size to 32 and the dropout (Zaremba et al., 2014) rate to 0.2. Different reward functions have been tried through the experiment progress. We choose a binary reward that a positive value is given when the output token equals the target and a punishment otherwise, 1 and -0.1 respectively. We evaluate the model every epoch and adopt early stopping on the validation dataset. In meta-training phase, we set different numbers of updating  $M'$  due to the differences in slot complexity for each domain. The model was implemented in the pyTorch.
4.3 Multi-domain Results
Table 1 shows our experimental results with MERET. MERET achieves the joint goal accuracy of  $50.91\%$ , which is  $2.12\%$  above the latest state-of-the-art DST model COMER and is  $2.29\%$  higher than TRADE. Table 1 also shows accuracies of a few latest systems on the same corpus. MERET is also able to obtain the best slot accura
7113
DST Models	Joint Acc	Slot Acc
MultiWOZ Benchmark (Budzianowski et al., 2018)	25.83	-
GLAD (Zhong et al., 2018)	35.57	95.44
HyST (ensemble)(Goel et al., 2019)	44.22	-
TRADE (Wu et al., 2019)	48.62	96.92
COMER (Ren et al., 2019)	48.79	-
MERET	50.91	97.07
-BERT	50.35	96.98
-RL	50.09	97.01
Table 1: The evaluation of existing multi-domain DSTs on MultiWOZ. MERET has the highest joint accuracy, which surpasses current state-of-the-art model. The baseline for the MultiWOZ dataset is taken from Budzianowski et al. (2018)
New Domain (Proportion)	Training Model	Joint Acc	Slot Acc
Taxi (1%)	Training from scratch	60.57	73.25
Fine-tuning TRADE	59.03	78.65
MERET	64.37	83.20
Attraction (1%)	Train from scratch	27.88	63.43
Fine-tuning TRADE	29.05	62.24
MERET	43.10	74.32
Table 2: Evaluation on taxi and attraction new domains. MERET outperforms learning from scratch and TRADE fine-tune with the same data on both new domains.
Figure 3: K-shot results of different experimental settings. Performance of our model surpasses training from scratch on attraction domain with  $K = 5$ .
cy  $97.07\%$  which is slightly higher than TRADE, but not substantial. To prove the effectiveness of our structure, we conduct ablation experiments in different setups. MERET-BERT(remove BERT, acc  $50.35\%$ ,  $+1.73\%$ ) has the same embedding Glove with TRADE, the improvement here mainly comes from RL, benefitting from the reward manager, which provides an ability for the entire model to explore rather than to be greedy at every single step and overcomes the existing limitation of encoder-decoder generation approach as men
tioned in the intro. MERET-RL(remove RL, acc  $50.09\%$ ,  $+1.47\%$  ) shows the increment due to embedding changes, which uses BERT instead of Glove, integrating powerful pre-trained language representation of BERT. We can see that MERET's advantage mainly comes from the RL. The way we employ RL with the generator in this paper is a good baseline. We are encouraged by these experimental results for future exploration in this line of research.
4.4 New Domain Results
To test the effectiveness of MERET, we choose hotel, train and restaurant as the source domains, taxi and attraction as the target domains. For each source domain, we utilize 3000 dialogues on average and 200 dialogues for training and testing. We utilize 30 dialogues (1% of source domain) for training on new domains with the pre-trained model. In our experiments, we conducted comparison studies with three setups, (1) Training a MERET model from scratch using  $1\%$  sampled data from each target domain, (2) Meta-training a MERET model using the source domain data and then fine-tuning with  $1\%$  sampled data from each target domain, (3) Training a TRADE model using the source domain data and then fine-tuning
7114
(a) Error type of TRADE.
(b) Error type of MERET.
Figure 4: Distributions of different error type for two models' comparison.
Figure 5: Overview of correct-error rate for multidomain slots. The book stay slot in hotel domain and name slot in restaurant domain has the highest and lowest correct rate respectively,  $98.97\%$  and  $91.06\%$  correspondingly.
with  $1\%$  sampled data from each target domain. Experimental results are listed in Table 2. MERET achieves substantial higher accuracy,  $64.7\%$  joint goal accuracy for the Taxi domain and  $43.10\%$  for the Attraction domain, comparing to the other two setups. Similar advantages are obtained for slot accuracies for both target domains.
To explore the K-shot performance of the MERET model, we conduct experiments to measure the impact of the number of training examples from the target domain. We meta-train MERET with source domains and meta-test on the taxi and attraction domain. The number of training samples  $K$  from the target domains varies from 1 to 10. We use  $K = (1,3,5,10)$  as the testing point. Figure 3 illustrates our experiments. It's natural that the accuracy increases as the training data increases. We can observe that the accuracy with  $K = 5$  of
Figure 6: The changes of joint accuracy over dialogue turns. The performance of our model MERET gradually emerges as the number of dialogue turns increases with the help of RL maximizing reward expectations.