SlowGuess/ACL_2020_OCR huggingface dataset:
Follow the prior work (Eric et al., 2017; Madotto et al., 2018; Wen et al., 2018; Wu et al., 2019a; Qin et al., 2019b), we adopt the BLEU and Micro Entity F1 metrics to evaluate model performance. The results on the two datasets are shown in Table 2, we can observe that: 1) The basic shared-private framework outperforms the best prior model GLMP in all the datasets. This indicates that the combination of domain-shared and domain-specific features can better enhance each domain performance compared with only utilizing the implicit domain-shared features. 2) Our framework achieves the state-of-the-art performance on two multi-domain task-oriented dialog datasets, namely SMD and Multi-WOZ 2.1. On SMD dataset, our model has the highest BLEU compared with baselines, which shows that our framework can generate more fluent response. More importantly, our model outperforms GLMP by  $2.0\%$  overall,  $3.3\%$  in the Navigate domain,  $1.1\%$  in the Weather domain and  $0.6\%$  in Schedule domain on entity F1 metric, which indicates that considering relevance between target domain input and all domains is effective for enhancing performance of each domain. On Multi-Woz 2.1 dataset, the same trend of improvement has been witnessed, which further shows the effectiveness of our framework.
We study the strengths of our model from several perspectives on SMD dataset. We first conduct several ablation experiments to analyze the effect of different components in our framework. Next, we conduct domain adaption experiments to verify the transferability of our framework given a new domain with little or no labeled data. In addition, we provide a visualization of the dynamic fusion layer and case study to better understand how the module affects and contributes to the performance.
For Multi-WOZ 2.1 dataset, most dialogs are supported by more than single row, which can not processed by KB-retriever, so we compare our framework with it in SMD and Camrest datasets.
Note that, we find that Wu et al. (2019a) report macro entity F1 as the micro F1, so we rerun their models (https://github.com/jasonwu0731/GLMP) and obtain results.
6349
(a) Navigate Domain
(b) Weather Domain
(c) Schedule Domain
Figure 5: Performance of domain adaption on different subsets of original training data.
Figure 6: Zero-shot performance (F1 score) on each domain on SMD dataset. The x-axis domain name represents that the domain is unseen and other two domains is the same as original dataset.
Model	Correct	Fluent	Humanlike
GLMP	3.4	3.9	4.0
Our framework	3.6	4.2	4.2
Agreement	53%	61%	74%
Table 4: Human evaluation of responses on the randomly selected dialogue history.
3.5.1 Ablation
Several ablation experiments and results are shown in Table 3. In detail, 1) w/o Domain-shared Knowledge Transfer denotes that we remove domain-shared feature and just keep fused domain-specific feature for generation. 2) w/o Domain Fusion mechanism denotes that we simply sum all domain-specific features rather than use the MOE mechanism to dynamically fuse domain-specific features. 3) w/o Multi-Encoder represents that we remove multi-encoder module and adopt one shared encoder in our framework. 4) w/o Multi-Decoder represents that we remove the multi-decoder module and adopt one shared decoder. 5) w/o Adversarial Training denotes that we remove the adversarial training in experimental setting. Generally, all the proposed components are effective to contribute the final performance. Specifically, we can clearly observe the effectiveness of our dynamic fusion mechanism where w/o domain-specific knowledge fusion causes  $1.8\%$  drops and the same trend in removing domain-shared knowledge fusion. This
Figure 7: Distribution of Mix-of-the-expert mechanism across source domains for randomly selected 100 examples in each domain on SMD dataset.
further verifies that domain-shared and domain-specific feature are benefit for each domain performance.
3.5.2 Domain Adaption
Low-Resource Setting To simulate low-resource setting, we keep two domains unchanged, and the ratio of the except domain from original data varies from  $[1\%, 5\%, 10\%, 20\%, 30\%, 50\%]$ . The results are shown in Figure 5. We can find that: (1) Our framework outperforms the GLMP baseline on all ratios of the original dataset. When the data is only  $5\%$  of original dataset, our framework outperforms GLMP by  $13.9\%$  on all domains averagely. (2) Our framework trained with  $5\%$  training dataset can achieve comparable and even better performance compared to GLMP with  $50\%$  training dataset on some domains. This implies that our framework effectively transfers knowledge from other domains to achieve better performance for the low-resources new domain.
Zero-Shot Setting Specially, we further evaluate the performance of domain adaption ability on the zero-shot setting given an unseen domain. We randomly remove one domain from the training set, and other domain data remained unchanged to train the model. During test, the unseen domain input use the MoE to automatically calculate the correlation between other domains and the current input and get the results. Results are shown in
Dialogue History: Find location and address to home that is nearest me. Your home is at 56 cadwell street. Thanks, set the gps for there. 0.5 0.1996 0.0242 Response: No problem. Navigation Weather Schedule
Figure 8: Case of of expert gate distribution in SMD dataset. Text segments with red color represents appearing in both schedule and navigation domain.
Figure 6, we can see our model significantly outperforms GLMP on three domains, which further demonstrate the transferability of our framework.
3.5.3 Visualization of Dynamic Fusion Layer
To better understand what our dynamic fusion layer has learned, we visualize the gate distribution for each domain in low-resource  $(5\%)$  setting, which fuses domain-specific knowledge among various cases. As shown in the Figure 7, for a specific target domain, different examples may have different gate distributions, which indicates that our framework successfully learns how to transfer knowledge between different domains. For example, the navigation column contains 100 examples from its test set and each row show the corresponding expert value. More specifically, in the navigation column, we observe that the expert value in schedule domain is bigger than weather domain, which indicates schedule domain transfers more knowledge to navigation than weather domain.
3.5.4 Case Study
Furthermore, we provide one case for navigation domain and their corresponding expert gate distribution. The cases are generated with  $5\%$  training data in the navigation domain and other two domain datasets keep the same, which can better show how the other two domains transfer knowledge to the low-resource domain. As shown in Figure 8, the expert value of schedule domain is bigger than the weather domain, which indicates the schedule contributes more than weather domain. In further exploration, we find word "location" and "set" appear both in navigation and schedule domain, which shows schedule has closer relation with navigation than weather, which indicates our model successfully transfers knowledge from the closest domain.
3.5.5 Human Evaluation
We provide human evaluation on our framework and other baseline models. We randomly generated 100 responses. These responses are based on
distinct dialogue history on the SMD test data. Following Wen et al. (2018) and Qin et al. (2019b), We hired human experts and asked them to judge the quality of the responses according to correctness, fluency, and humanlikeness on a scale from 1 to 5.
Results are illustrated in Table 4. We can see that our framework outperforms GLMP on all metrics, which is consistent with the automatic evaluation.
Existing end-to-end task-oriented systems can be classified into two main classes. A series of work trains a single model on the mixed multi-domain dataset. Eric et al. (2017) augments the vocabulary distribution by concatenating KB attention to generate entities. Lei et al. (2018) first integrates track dialogue believes in end-to-end task-oriented dialog. Madotto et al. (2018) combines end-to-end memory network (Sukhbaatar et al., 2015) into sequence generation. Gangi Reddy et al. (2019) proposes a multi-level memory architecture which first addresses queries, followed by results and finally each key-value pair within a result. Wu et al. (2019a) proposes a global-to-locally pointer mechanism to query the knowledge base. Compared with their models, our framework can not only explicitly utilize domain-specific knowledge but also consider different relevance between each domain. Another series of work trains a model on each domain separately. Wen et al. (2018) leverages dialogue state representation to retrieve the KB implicitly. Qin et al. (2019b) first adopts the KB-retriever to explicitly query the knowledge base. Their works consider only domain-specific features. In contrast, our framework explicitly leverages domain-shared features across domains.
The shared-private framework has been explored in many other task-oriented dialog components. Liu and Lane (2017) applies a shared-private LSTM to generate shared and domain-specific features. Zhong et al. (2018) proposes a global-local architecture to learn shared feature across all slots and specific feature for each slot. More recently, Zhang et al. (2018) utilizes the shared-private model for text style adaption. In our work, we explore shared-private framework in end-to-end task-oriented dialog to better transfer domain knowledge for querying knowledge base. In addition, we take inspiration from Guo et al. (2018), who successfully apply the mix-of-the-experts (MoE) mechanism in multi-sources domain and cross-lingual
6351
adaption tasks. Our model not only combines the strengths of MoE to incorporate domain-specific feature, but also applies adversarial training to encourage generating shared feature. To our best of knowledge, we are the first to effectively explore shared-private framework in multi-domain end-to-end task oriented dialog.
In this paper, we propose to use a shared-private model to investigate explicit modeling domain knowledge for multi-domain dialog. In addition, a dynamic fusion layer is proposed to dynamically capture the correlation between a target domain and all source domains. Experiments on two datasets show the effectiveness of the proposed models. Besides, our model can quickly adapt to a new domain with little annotated data.
We thank Min Xu, Jiapeng Li, Jieru Lin and Zhouyang Li for their insightful discussions. We also thank all anonymous reviewers for their constructive comments. This work was supported by the National Natural Science Foundation of China (NSFC) via grant 61976072, 61632011 and 61772153. Besides, this work also faxed the support via Westlake-BrightDreams Robotics research grant.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37-49, Saarbrücken, Germany. Association for Computational Linguistics.
Mihail Eric and Christopher Manning. 2017. A copy-augmented sequence-to-sequence architecture gives good performance on task-oriented dialogue. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 468-473,
Valencia, Spain. Association for Computational Linguistics.
Revanth Gangi Reddy, Danish Contractor, Dinesh Raghu, and Sachindra Joshi. 2019. Multi-level memory for task oriented dialogs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3744-3754, Minneapolis, Minnesota. Association for Computational Linguistics.
Yaroslav Ganin and Victor Lempitsky. 2014. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495.
Jiang Guo, Darsh Shah, and Regina Barzilay. 2018. Multi-source domain adaptation with mixture of experts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4694-4703, Brussels, Belgium. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1437-1447, Melbourne, Australia. Association for Computational Linguistics.
Bing Liu and Ian Lane. 2017. Multi-domain adversarial learning for slot filling in spoken language understanding. arXiv preprint arXiv:1711.11310.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1-10, Vancouver, Canada. Association for Computational Linguistics.
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1468-1478, Melbourne, Australia. Association for Computational Linguistics.
Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019a. A stack-propagation framework with token-level intent detection for spoken
6352
language understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2078-2087, Hong Kong, China. Association for Computational Linguistics.
Libo Qin, Yijia Liu, Wanxiang Che, Haoyang Wen, Yangming Li, and Ting Liu. 2019b. Entity-consistent end-to-end task-oriented dialogue system with KB retriever. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 133-142, Hong Kong, China. Association for Computational Linguistics.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2440-2448.
Haoyang Wen, Yijia Liu, Wanxiang Che, Libo Qin, and Ting Liu. 2018. Sequence-to-sequence learning for task-oriented dialogue with dialogue state representation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3781-3792, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Chien-Sheng Wu, Richard Socher, and Caiming Xiong. 2019a. Global-to-local memory pointer networks for task-oriented dialogue. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Haiming Wu, Yue Zhang, Xi Jin, Yun Xue, and Ziwen Wang. 2019b. Shared-private LSTM for multi-domain text classification. In *Natural Language Processing and Chinese Computing - 8th CCF International Conference*, NLPCC 2019, Dunhuang, China, October 9-14, 2019, Proceedings, Part II, pages 116-128.
Steve J. Young, Milica Gasic, Blaise Thomson, and Jason D. Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
Ye Zhang, Nan Ding, and Radu Soricut. 2018. SHAPED: Shared-private encoder-decoder for text style adaptation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1528-1538, New Orleans, Louisiana. Association for Computational Linguistics.
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1458-1467, Melbourne, Australia. Association for Computational Linguistics.
6353
Hyperparameter Name	SMD	Multi-WOZ 2.1
Batch Size	16	32
Hidden Size	128	128
Embedding Size	128	128
Learning Rate	0.001	0.001
Dropout Ratio	0.2	0.1
Teacher Forcing Ratio	0.9	0.9
Number of Memory Network's Hop	3	3
Table 5: Hyperparameters we use for SMD and Multi-WOZ 2.1 dataset.
A.1 Hyperparameters Setting
The hyperparameters used for SMD and MultiWOZ 2.1 dataset are shown in Table 5.
A.2 Basic Loss Function
The loss  $\mathcal{L}_{\text{basic}}$  used in our Shared-Private Encoder-Decoder Model is the same as GLMP. Different with the standard sequence-to-sequence with attention mechanism model, we use  $[h_{\mathrm{dec},t}^{f}, h_{\mathrm{dec},t}^{f'}]$  to replace  $[h_{\mathrm{dec},t}, h_{\mathrm{dec},t}']$  and then get the sketch word probability distribution  $P_t^{\text{vocab}}$ . Based on the gold sketch response  $\pmb{Y}^s = (y_1^s, \dots, y_n^s)$ , we calculate the standard cross-entropy loss  $\mathcal{L}_v$  as follows:
$$
P _ {t} ^ {\text {v o c a b}} = \operatorname {S o f t m a x} \left(\boldsymbol {U} \left[ \boldsymbol {h} _ {\mathrm {d e c}, t} ^ {f}, \boldsymbol {h} _ {\mathrm {d e c}, t} ^ {f ^ {\prime}} \right]\right), \tag {19}
$$
$$
\mathcal {L} _ {v} = \sum_ {t = 1} ^ {n} - \log \left(P _ {t} ^ {\text {v o c a b}} \left(y _ {t} ^ {s}\right)\right). \tag {20}
$$
Given the system response  $Y$ , we get the global memory pointer label sequence  $G^{label} = (\hat{g}_1, \dots, \hat{g}_{b + T})$  and local memory pointer label sequence  $L^{label} = (\hat{l}_1, \dots, \hat{l}_n)$  as follows:
$$
\hat {g} _ {i} = \left\{ \begin{array}{l l} 1 & \text {i f O b j e c t} (m _ {i}) \in Y \\ 0 & \text {o t h e r w i s e} \end{array} , \right. \tag {21}
$$
$$
\hat {l} _ {t} = \left\{ \begin{array}{l l} \max  (z) & \text {i f} \exists z \text {s . t .} y _ {t} = \operatorname {O b j e c t} \left(m _ {z}\right) \\ b + T + 1 & \text {o t h e r w i s e} \end{array} , \right. \tag {22}
$$
where  $m_{i}$  represents one triplet in the external knowledge  $M = [B;X] = (m_{1},\ldots ,m_{b + T})$  and  $\mathrm{Object}(\cdot)$  function is denoted as getting the object word from a triplet.
Then, the  $\mathcal{L}_g$  can be written as follows:
$$
\mathcal {L} _ {g} = - \sum_ {i = 1} ^ {b + T} \left(\hat {g} _ {i} \cdot \log g _ {i} + \left(1 - \hat {g} _ {i}\right) \cdot \log \left(1 - g _ {i}\right)\right). \tag {23}
$$
Based on the  $L^{label}$  and  $P_{t} = (p_{1}^{k},\dots ,p_{b + T}^{k})$  we can calculate the standard cross-entropy loss  $\mathcal{L}_l$  as follows:
$$
\mathcal {L} _ {l} = \sum_ {t = 1} ^ {n} - \log \left(P _ {t} \left(\hat {l} _ {t}\right)\right). \tag {24}
$$
Finally,  $\mathcal{L}_{\text{basic}}$  is the weighted-sum of three losses:
$$
\mathcal {L} _ {\text {b a s i c}} = \gamma_ {g} \mathcal {L} _ {g} + \gamma_ {v} \mathcal {L} _ {v} + \gamma_ {l} \mathcal {L} _ {l}, \tag {25}
$$
where  $\gamma_{g},\gamma_{v}$  and  $\gamma_{l}$  are hyperparameters.
A.3 Adversarial Training
We apply a Convolutional Neural Network (CNN) as domain classifier both in the shared encoder and shared decoder to identify the domain of shared representation of dialogue history  $H_{\mathrm{enc}}^s$  and response  $H_{\mathrm{dec}}^s$ . Take the encoder for example, based on the  $H_{\mathrm{enc}}^s$ , we can extract the context representation  $c_{\mathrm{enc}}^s$  by CNN and then  $\beta_{\mathrm{enc}} \in \mathbb{R}^{|\mathbb{D}|}$  can be calculated as follows:
$$
\boldsymbol {\beta} _ {\mathrm {e n c}} = \text {S i g m o i d} (\text {L e a k y R e L U} \left(\boldsymbol {W} _ {\mathrm {e n c}} \left(\boldsymbol {c} _ {\mathrm {e n c}} ^ {\mathrm {s}}\right)\right), \tag {26}
$$
Then we train the domain classifier by optimizing its parameters  $\theta_{d}$  to minimize the sequence-level binary cross-entropy loss  $L_{\mathrm{enc}}^{adv}$  as follows:
$$
\begin{array}{l} \max  _ {\boldsymbol {\theta} _ {s}} \min  _ {\boldsymbol {\theta} _ {d}} L _ {\mathrm {e n c}} ^ {a d v} = - \sum_ {i = 1} ^ {| \mathbb {D} |} \left(e _ {i} \cdot \log \left(\beta_ {\mathrm {e n c}, i} \mid \boldsymbol {\theta} _ {s}, \boldsymbol {\theta} _ {d}\right) \right. \\ + \left(1 - e _ {i}\right) \cdot \log \left(1 - \beta_ {\mathrm {e n c}, i} \mid \boldsymbol {\theta} _ {\boldsymbol {s}}, \boldsymbol {\theta} _ {\boldsymbol {d}}\right), \tag {27} \\ \end{array}
$$
where  $\beta_{\mathrm{enc},i}$  represents the probability of the input dialogue history belongs to the domain  $d_{i}$ . Similarly, we can get the  $L_{\mathrm{dec}}^{adv}$  and sum up them as:  $\mathcal{L}_{adv} = L_{\mathrm{enc}}^{adv} + L_{\mathrm{dec}}^{adv}$ .
In order to update the encoder-decoder model parameters  $\theta_{s}$  underlying the domain classifier, we introduce the gradient reversal layer to reverse the gradient direction which trains our model to extract domain-shared features to confuse the classifier. On the one hand, we train the domain classifier to minimize the domain classification loss. On the other hand, we update the parameters of the network underlying the domain classifier to maximize the domain classification loss, which works adversarially towards the domain classifier. This encourages that our shared encoder and decoder are trained to extract domain-shared features.
6354