SlowGuess/ACL_2020_OCR huggingface dataset:
Yinpei Dai†, Hangyu Li†, Chengguang Tang†, Yongbin Li†*, Jian Sun†, Xiaodan Zhu‡ †Alibaba Group, Beijing
$^{\ddagger}$ Ingenuity Labs Research Institute & ECE, Queen's University
{yinpei.dyp, hangyu.lhy, chengguang.tcg}@alibaba-inc.com {shuide.lyb, jian.sun}@alibaba-inc.com, zhu2048@gmail.com
Existing end-to-end dialog systems perform less effectively when data is scarce. To obtain an acceptable success in real-life online services with only a handful of training examples, both fast adaptability and reliable performance are highly desirable for dialog systems. In this paper, we propose the Meta-Dialog System (MDS), which combines the advantages of both meta-learning approaches and human-machine collaboration. We evaluate our methods on a new extended-bAbI dataset and a transformed MultiWOZ dataset for low-resource goal-oriented dialog learning. Experimental results show that MDS significantly outperforms non-meta-learning baselines and can achieve more than  $90\%$  per-turn accuracies with only 10 dialogs on the extended-bAbI dataset.
End-to-end neural models have shown a great potential in building flexible goal-oriented dialog systems. They can be directly trained on past dialogs without any domain-specific handcrafting, which makes it easy to automatically scale up to new domains (Bordes et al., 2017). However, these models are normally data-hungry and have only been successfully applied to domains with rich datasets (Perez et al., 2017; Luo et al., 2019; Kim et al., 2019).
In real-world scenarios, common issues with end-to-end dialog models include: (1) the shortage of proper training dialogs because of the high cost of data collection and cleaning, i.e., the data scarcity problem (Zhao and Eskenazi, 2018), and (2) a large gap between limited data and unknown online test examples, i.e., the covariate shift effect (Liu et al.). Such problems can lead to a significant performance degradation in dialog systems, which
may harm the users' experience and result in loss of customers in commercial applications. Therefore, both fast adaptability and reliable performance are strongly desirable for practical system deployment. Fast adaptability reflects the efficiency of adapting dialog systems to domains with low-resource data. Reliable performance reflects the robustness of handling unpredictable user behaviors in online services.
To boost the online performance of dialog systems, there have been some recent work (Rajendran et al., 2019; Wang et al., 2019; Lu et al., 2019) on designing end-to-end models in a human-machine joint-teaming manner. For instance, the dialog system in (Rajendran et al., 2019) can identify an ongoing dialog during testing when the system might fail and transfer it to a human agent. But all these methods are trained with sufficient data, which hinders the possibility of rapidly prototyping the models in new domains with restricted resources.
In this paper, we formulate the low-resource goal-oriented dialog learning as a few-shot learning problem, where a limited numbers of dialogs are used for training and the remaining for the test. We propose the Meta-Dialog System (MDS), an end-to-end human-machine teaming framework optimized by the model-agnostic meta-learning (MAML) algorithm (Finn et al., 2017). In general, MDS learns to make prediction and requests human by finding good initial parameters, which can be adapted to new tasks fast and reliably by using fewer dialogs. We evaluate our methods on a new multi-domain dialog dataset called extended-bAbI. Results show that MDS achieves obvious performance improvement over baselines and attains more than  $90\%$  per-turn accuracy on new domains with only 10 dialogs. We also perform experiments on MultiWOZ dataset (Eric et al., 2019) which has been transformed into simplified bAbI format and observe similar superior results with MDS.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 609-618 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In summary, the main contributions of this paper are three-fold: (1) To the best of our knowledge, this is the first study on applying meta-learning to retrieval-based end-to-end goal-oriented dialog systems; (2) we leverage the MAML algorithm to optimize a human-machine collaborative dialog system and show very promising results on the low-resource dialog tasks; and (3) we propose a new dataset and hope that can help bring forward the research in this area.
In this section, we first introduce the problem definition and our new dataset; we then elaborate the framework of MDS and meta-learning procedures.
Problem Definition. We focus on the retrieval-based goal-oriented dialog tasks (Perez et al., 2017), where a training data  $d_{i}$  usually contains a triple  $(H_{i},y_{i},\mathcal{R})$ .  $H_{i}$  denotes the dialog history consisting of all user utterances and system responses up to the current turn,  $\mathcal{R}$  is a set of given candidate responses and  $y_{i}$  is the index of the correct response in  $\mathcal{R}$ . The main task is to train an end-to-end dialog model to predict  $y_{i}$  from  $\mathcal{R}$  based on  $H_{i}$ .
Extended-bAbI Dataset. The original bAbI dataset (Bordes et al., 2017) is not suitable for low-resource settings due to the lack of domains and tasks. We extend it into a multi-domain dataset through complicated simulation rules and construct templates with a more diversity to raise the difficulty. There are 7 domains in total: restaurant, flights, hotels, movies, music, tourism and weather, each of which has its own ontology and the candidate response set. Similar to (Bordes et al., 2017), a complete dialog in extended-bAbI contains four phases of interactions: (1) the system asks for required attributes to constrain the search and issues the first API call; (2) the user updates their requests for revised API calls; (3) the system confirms for multiple times to determine the entity the user wants; (4) the user requests more attributes for extra information based on the final entity. The total number of dialogs is 21,000 and the detailed examples and statistics are given in Appendix A.1.
2.1 Model Architecture
In MDS, there is an encoding module to extract neural features of dialogs and a policy module to make system actions of either predicting responses or requesting human. All modules are jointly optimized
with the MAML algorithm. The main framework of training MDS is shown in Figure 1.
Encoding Module. It contains a history encoder to compute the dialog state vector  $s_i$  for  $H_i$  and a response encoder to compute the response embedding  $r_j$  for the  $j$ -th response in  $\mathcal{R}$ . The dimensions of  $s_i$  and  $r_j$  are set as the same. In this paper, we use the MemN2N (Sukhbaatar et al., 2015) as the history encoder and a simple additive model for the response encoder, but many other models optimized by gradient descent may be applied here.
Policy Module. This module consists of a switch  $S$  that makes a binary decision whether to request human to select the response, and a response predictor  $P$  that predicts the right response itself if human is not requested. We assume that the response chosen by human is always correct.
For the optimization of  $P$ , the widely used largemargin cosine loss (Wang et al., 2018; Lin and Xu, 2019) is employed since it maximizes the decision margin in the angular space and is able to force the model to learn more discriminative deep features. Suppose a batch of training data is  $\mathcal{D} = \{d_1,\dots d_i,\dots,d_{|\mathcal{D}|}\}$ , then the formulation is:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {L M C}} = \\ \sum_ {i = 1} ^ {| \mathcal {D} |} - \log \frac {e ^ {a \cdot (\cos \left(s _ {i} , r _ {y _ {i}}\right) - b)}}{e ^ {a \cdot (\cos \left(s _ {i} , r _ {y _ {i}}\right) - b)} + \sum_ {j \neq y _ {i}} e ^ {a \cdot \cos \left(s _ {i} , r _ {j}\right)}} \tag {1} \\ \end{array}
$$
where  $\cos (\cdot ,\cdot)$  is a function that calculates the cosine similarity of two input vectors.  $a$  is the scaling factor and  $b$  is the cosine margin  $(a = 30,m = 0.1$  in our experiments). In the test phase, the model predicts an answer according to the maximal cosine angle  $y_{i}^{*} = \mathrm{argmax}_{j}\cos (s_{i},r_{j})$
The switch  $S$  is a neural binary classifier that also takes  $s_i$  and each  $r_j$  as input and calculate the decision probability of requesting human as follows:
$$
w _ {i j} = e ^ {s _ {i} ^ {\top} W r _ {j}} / \sum_ {k = 1} ^ {| \mathcal {R} |} e ^ {s _ {i} ^ {\top} W r _ {k}} \tag {2}
$$
$$
c _ {i} = \sum_ {j = 1} ^ {| \mathcal {R} |} w _ {i j} r _ {j} \tag {3}
$$
$$
f _ {i} = s _ {i} \oplus c _ {i} \tag {4}
$$
$$
p _ {i} = \sigma \left(\operatorname {F C} \left(f _ {i}\right)\right) \tag {5}
$$
where  $\sigma$  is the sigmoid function and  $\oplus$  the concatenation function for vectors.  $\mathrm{FC}(\cdot)$  is a fully-connected neural network with one hidden layer that has half size of the input layer and is activated
Figure 1: An overview of training the Meta-Dialog System.
by tanh function.  $|\mathcal{R}|$  is the size of  $\mathcal{R}$  and  $W$  is a trainable square matrix.
Learning to switch. Since there are no actual labels for  $S$  to indicate whether it is correct to ask human or not, some previous work (Woodward and Finn, 2016; Rajendran et al., 2019) proposes to use the REINFORCE algorithm (Williams, 1992) for weakly-supervised training, but their reward settings fail to penalize the case when the model asks human while it can give right prediction, which may lead to redundant requests. To consider this effect, we propose a new reward definition here. For the batch data  $\mathcal{D}$ , we calculate the F1 scores<sup>1</sup> for positive data and negative data, respectively, and take the average of them to get a scalar value score  $(\mathcal{D})$ . Then each data  $d_{i} \in \mathcal{D}$  is assigned with a reward by computing an incremental value as below:
$$
r _ {t} = \operatorname {s c o r e} (\mathcal {D}) - \operatorname {s c o r e} (\mathcal {D} - d _ {i}) \tag {6}
$$
Through maximizing such rewards, the switch  $S$  learns to be more effective and asks human when it is necessary. The reinforcement learning loss for  $S$  is  $\mathcal{L}_{\mathrm{RL}} = \sum_{i=1}^{|\mathcal{D}|} -r_i \log p_i$ , and the final loss of our model is  $\mathcal{L} = \mathcal{L}_{\mathrm{LMC}} + \mathcal{L}_{\mathrm{RL}}$ .
2.2 Training Procedure
We rewrite the final loss  $\mathcal{L}$  as  $\mathcal{L}(\mathcal{M}_{\theta},\mathcal{D})$  for clarity, where  $\mathcal{M}_{\theta}$  denotes the dialog model with trainable parameters  $\theta$  and  $\mathcal{D}$  is the batch data for training.
During meta-learning, we first choose one domain as the target domain and the rest as source domains. Then we uniformly sample  $K$  different domains  $\mathcal{T} = \{\tau_1,\dots ,\tau_K\}$  from source domains as meta-tasks. For each meta-task  $\tau_{k}$ , we sample  $N$  data as the support set  $\mathcal{D}_k^{\mathrm{sup}}$  and other  $N$  data with the same answers as the query set  $\mathcal{D}_k^{\mathrm{que}}$ .
$\mathcal{M}_{\theta}$  is first updated on support sets for each  $\tau_{k}$ :
$$
\theta_ {k} ^ {\prime} = \theta - \alpha \nabla_ {\theta} \mathcal {L} \left(\mathcal {M} _ {\theta}, \mathcal {D} _ {k} ^ {\sup }\right) \tag {7}
$$
Then  $\mathcal{M}_{\theta}$  is evaluated on each  $\mathcal{D}_k^{\mathrm{que}}$  with  $\theta_k^\prime$  respectively and is optimized as follows:
$$
\theta \leftarrow \theta - \beta \nabla_ {\theta} \sum_ {k = 1} ^ {K} \mathcal {L} \left(\mathcal {M} _ {\theta_ {k} ^ {\prime}}, \mathcal {D} _ {k} ^ {\text {q u e}}\right) \tag {8}
$$
where  $\alpha, \beta$  are learning rates. By training on multiple tasks via MAML,  $\mathcal{M}_{\theta}$  can learn good initial parameters that is applicable on new tasks or domains (Finn et al., 2017; Mi et al., 2019). The algorithm is summarised in Algorithm 1.
After this meta-learning as pre-training, we finetune  $\mathcal{M}_{\theta}$  on the target domain with the first  $L$  dialogs of its training set, where  $L$  is a small number. To mimic the situation of online testing, we evaluate  $\mathcal{M}_{\theta}$  on the whole test sets and regard those unseen user utterances as new user behaviours.
In our experiments, we first verify the capability of MDS on our newly simulated dialog dataset
Detailed explanations can be found in Appendix A.2.
extended-bAbI, and then conduct extra evaluation on the more realistic dataset MultiWOZ 2.1 (Eric et al., 2019).
We select each domain as the target domain in turn and take the average of the results in all domains.
Metric. Following (Wang et al., 2019), we report the user-perceived per-turn accuracy ('per-turn accuracy' is used in the remainder of the paper), where the prediction of one turn is considered correct if the model either selects the right response by itself or asks human. To be fair, we also report the human request rate. The less the request rate and higher per-turn accuracy are, the more reliable the model performs online.
Implementation details. For the meta-learning, we use SGD for the inner loop and Adam for the outer loop with learning rate  $\alpha = 0.01$  and  $\beta = 0.001$ . The meta-task size  $K$  is 4 and the support or query set size  $N$  is 16. For the standard MLE training, we use Adam with a learning rate of 0.001 and set the batch size as 32. Both schemes are trained for a maximum of 5000 iterations with early stopping on the validation set. During fine-tuning on new domains, we use SGD with the learning rate 0.01 for all models and report the final results after fine-tuning 10 iterations on  $L$  training dialogs of the target domain, where  $L = 0, 1, 5, 10$ . The word vector size is 25 and all MemN2Ns take 3 hops.
We compare MDS with the following baselines:
- Mem: A MemN2N (Sukhbaatar et al., 2015) model trained with standard MLE.
MetaMem: A MemN2N trained with MAML. Both Mem and MetaMem can not request human.
- Mem+C: A MemN2N model combined with a binary classifier in (Rajendran et al., 2019), which has different objective functions and optimization.
- IDS: The incremental dialog system used in (Wang et al., 2019), which requests human by estimating the uncertainty through a variational autoencoder.
- MDS-switch: A MDS without the switch  $S$ .
-  $\mathbf{MDS}_{\mathrm{rand}}$ : A MDS whose switch is replaced with a random classifier that has the same request rate.
-  $\mathbf{MDS}_{\mathbf{mle}}$ : A MDS whose meta-learning optimization is replaced with standard MLE.
Figure 2: The per-turn accuracy of different methods on the test set during fine-tuning with 1 dialog adaptation where the target domain is restaurant.
3.3 Results on Extended-bAbI
Table 1 shows few-shot adaptation results for different methods. MDS significantly outperforms other models under all adaptation sizes of new dialogs and can achieve a  $91.31\%$  per-turn accuracy on average with only 10 new dialogs.
There is a gap between methods without the switch (such as Mem, MetaMem and MDS-switch) and methods with the switch in Table 1, indicating that the switch  $S$  is crucial for improving the overall per-turn accuracy because of the human agent. However, without proper objective functions and meta-learning optimization, Mem+C and IDS² have poorer performances in both metrics than MDS even if they contain the switch module.
In the ablation study, we see a steady increase of about  $10\%$  per-turn accuracy from the comparison between MDS and  $\mathrm{MDS}_{\mathrm{rand}}$ , suggesting that the switch does identify intractable dialogs.  $\mathrm{MDS}_{\mathrm{mle}}$  is the closest baseline to MDS, but we still observe an obvious improvement, which means joint optimization of  $S$  and  $P$  via meta-learning allows faster and better adaptation while maintaining similar request rates. Appendix A.3 illustrates detailed case studies for different methods.
To further investigate the adaptation process, we present the fine-tuning curves for different methods with 1 dialog adaptation in Figure 2. As it can be seen, MDS achieves the best accuracy at the beginning and converges fastest as well, showing that it can transfer on new tasks quickly by finding better parameter initialization.
2We only report the result of IDS with 10 dialog adaptation since its request rates are too high to be fair in other settings.
Method	No adaptation	Adapt with 1 dialog	Adapt with 5 dialogs	Adapt with 10 dialogs
accuracy	request	accuracy	request	accuracy	request	accuracy	request
Mem	32.28±1.86	n.a.	45.02±1.39	n.a.	64.07±0.76	n.a.	71.56±0.48	n.a.
MetaMem	39.45±1.13	n.a.	48.95±1.18	n.a.	65.57±0.69	n.a.	72.19±0.81	n.a.
Mem+C	58.74±2.89	37.34±5.23	68.27±2.19	34.83±4.35	81.41±2.26	36.96±5.05	87.46±2.07	38.09±5.29
IDS	-	-	-	-	-	-	90.91±4.29	83.98±6.43
MDS-switch	41.03±0.98	n.a.	50.31±1.16	n.a.	65.72±1.13	n.a.	72.35±0.90	n.a.
MDSrand	61.05±1.20	34.75	66.02±0.91	32.31	77.27±0.76	34.31	79.70±0.98	35.26
MDSmle	59.89±3.11	34.36±6.09	69.40±2.25	32.46±4.06	83.04±2.07	33.90±5.22	88.13±1.63	35.28±5.08
MDS	64.93±2.39	34.75±5.87	74.71±2.15	32.31±4.34	86.49±2.01	34.31±4.36	91.31±1.16	35.26±4.23
3.4 Results on MultiWOZ
MultiWOZ (Budzianowski et al., 2018) is a widely-used multi-domain Wizard-of-Oz dialog dataset spanning 7 distinct domains and containing 10k dialogs. This realistic dataset has been a standard benchmark for various dialog tasks such as belief tracking and policy optimization.
In our experiment, we use the corrected version MultiWOZ 2.1 (Eric et al., 2019) for evaluation. To translate the MultiWOZ dialogs into bAbI-format data, we first delexicalize the slot-values in user utterances using dialog labels, and then produce a set of canonical system acts as the candidate responses by simplifying the original dialog acts. Only dialogs containing single domain are used in our experiments and a MultiWOZ dialog sample is given in Appendix A.4.
Table 2 shows the adaptation results for different models on MultiWOZ 2.1. It can be seen that MDS still largely outperforms other models with the adaptation of 10 dialogs. The degradation of per-turn accuracy from extended-bAbI to MultiWOZ is reasonable since the user utterance is more diverse and the dialog policy is more flexible.
End-to-end neural approaches of building dialog systems have attracted increasing research interest. The work of (Bordes et al., 2017) is the first attempt to solve goal-oriented dialog tasks with end-to-end models. Further improvements has been made in (Williams et al., 2017) to combine explicit domain-specific knowledge and implicit RNN features. Luo et al. (2019) take user personalities into consideration for better user satisfaction. Rajendran et al. (2018) learn dialogs with multiple possible answers. Our work is inspired by the work of (Rajendran et al., 2019; Wang et al., 2019), which
Table 1: Few-shot results on the extended-bAbI dataset. The numbers represent the average of means and standard deviations of Task 5 in all target domains. Each experiment run 10 times with different seeds; 'n.a.' means no switch in the model; 'accuracy' is the user-perceived per-turn accuracy and 'request' is the request rate.
Method	Adapt with 10 dialogs
accuracy	request
Mem	56.87±1.63	n.a.
MetaMem	62.78±2.05	n.a.
Mem+C	80.59±3.13	38.18±5.01
MDSswitch	64.50±3.75	n.a.
MDSrand	74.78±4.35	38.34
MDSmle	80.92±3.02	37.91±4.20
MDS	83.52±3.30	38.34±6.96
Table 2: Few-shot test results on MultiWOZ 2.1.
propose to solve unseen user behaviors through human-machine teamwork. The research of (Liu et al.; Chen et al., 2017; Lu et al., 2019) also show the advantages of incorporating the role of human to teach online. However, dialog learning in low-resource scenarios has not been investigated.
Meta-learning aims to learn new tasks rapidly with a few training examples (Sung et al., 2018; Finn et al., 2017), which fits well to our task. There have been some work applying meta-learning to other tasks in dialog research, such as that in (Dou et al., 2019; Geng et al., 2019) for natural language understanding and (Qian and Yu, 2019; Mi et al., 2019) for natural language generation.
In this paper, we leverage the MAML algorithm to optimize a human-machine collaborative dialog system, which shows good results for both fast adaptability and reliable performance. In the future, we plan to use more powerful encoders and evaluate our methods on real dialog data.
The research of the last author is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).
Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. ICLR.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Lu Chen, Xiang Zhou, Cheng Chang, Runzhe Yang, and Kai Yu. 2017. Agent-aware dropout DQN for safe and efficient on-line dialogue policy learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2454-2464, Copenhagen, Denmark. Association for Computational Linguistics.
Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. 2019. Investigating meta-learning algorithms for low-resource natural language understanding tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1192-1197, Hong Kong, China. Association for Computational Linguistics.
Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, San-chit Agarwal, Shuyang Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. arXiv preprint, 1907.01669.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126-1135. JMLR.org.
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3902-3911, Hong Kong, China. Association for Computational Linguistics.
Byoungjae Kim, KyungTae Chung, Jeongpil Lee, Jungyun Seo, and Myoung-Wan Koo. 2019. A bilstm memory network for end-to-end goal-oriented dialog learning. Computer Speech & Language, 53:217-230.
Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5491-5496, Florence, Italy. Association for Computational Linguistics.