SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 6: Perplexity v.s. Number of epochs for English-to-German.
stage of training. The possible reason is that their training enforces one unified model to fit data from two different domains simultaneously, which is computationally more difficult.
Table 3 shows the BLEU scores of the baselines and our domain mixing methods for English-to-French translation. Note that though the data from the Medical and TED domains are slightly imbalanced (about 1:2.5), our methods can still outperform the baselines on both domains.
Table 2: English-to-German.
Method	TED	Medical
Direct Training
TED	28.22	7.32
Medical	7.03	53.73
Medical + TED	39.21	53.40
Embedding based Methods
MTL	39.14	53.37
AdvL	39.54	53.46
PAdvL	39.56	53.23
WDC + WL	39.79	53.85
Our Domain Mixing Methods
Encoder	40.30	54.05
Encoder + WL	40.43	54.14
E/DC	40.52	54.28
E/DC + WL	40.60	54.39
Table 4 shows the BLEU scores of the baselines and our domain mixing methods for Chinese-to
Table 3: English-to-French.
Method	Laws	News	Speech	Thesis
Direct Training
Laws	51.98	3.80	2.38	2.64
News	6.88	31.99	8.12	4.17
Speech	3.33	4.90	18.63	3.08
Thesis	5.90	5.55	4.77	11.06
Mixed	48.87	26.92	16.38	12.09
Embedding based Methods
MTL	49.14	27.15	16.34	11.80
AdvL	48.93	26.51	16.18	12.08
PAdvL	48.72	27.07	15.93	12.23
WDC + WL	42.16	25.81	15.29	10.14
Our Domain Mixing Methods
Encoder	50.21	27.94	16.85	12.03
Encoder + WL	50.11	27.48	16.79	11.93
E/DC	50.64	28.48	17.41	11.71
E/DC + WL	50.04	28.17	17.60	11.59
Table 4: Chinese-to-English.
English translation. As can be seen, our methods outperform the baselines on all domains except Thesis. We remark that the translation for the Thesis domain is actually very difficult, and all methods obtain poor performance.
Moreover, we find that for Chinese-to-English task, all our baselines are sensitive to the architecture of the Transformer. Their training will fail, if we place the layer normalization at the end of each encoder and decoder layer (as Vaswani et al. (2017) suggest). Therefore, we move the layer normalization to their beginnings. Surprisingly, our domain mixing methods are very stable regardless of the position of the layer normalization. More details can be found in Table 8 of Appendix A.
We further shows that the performance gains are from the domain mixing methods, instead of from the new model architecture design. Table 5 shows the BLEU scores with and without using domain labels under the same network structure and the same number of parameters as in the domain mixing methods. The only difference is that we remove domain label to guide the training of domain proportion, i.e., only  $L_{\mathrm{gen}}$  is used in the training loss, and  $L_{\mathrm{mix}}$  is removed. Training without domain labels shows a slight improvement over baseline, but is still significantly worse than our proposed method for most of the tasks. Therefore, we can conclude that our proposed domain mixing approach indeed improves performance.
4.5 Visualizing Domain Proportions
To further investigate our domain mixing methods, we plot the domain proportions of the word em
Method	Direct Training	w/o DL	with DL (Ours)
English-to-Germany
News	26.06	26.25	27.78
TED	28.11	28.27	30.30
English-to-French
TED	39.21	39.39	40.30
Medical	53.40	53.33	54.05
Chinese-to-English
Laws	48.87	48.96	50.21
News	26.92	27.02	27.94
Speech	16.38	16.15	16.85
Thesis	12.09	12.03	12.03
Table 5: BLEU Scores with and without domain labels (DL) under equal model capacity.
bedding at different layers. A uniform proportion, e.g., (0.5, 0.5), is encouraging knowledge sharing across domains, while a skewed proportion, e.g., (0.1, 0.9), means there is little knowledge to share across domains. Figure 7 illustrates how the knowl
Figure 8: Domain proportions of a sentence pair for English-to-German task. White represents the News domain and black represents the TED domain. The domain proportions of both the encoder (bottom) and the decoder (top) are presented.
edge sharing is controlled via the domain proportion. The selected sentence is from the English-to-French task, containing TED and Medical domains. Specifically, we observe :
- The domain proportions of different words at different layers have various patterns.
- At the bottom layers, the domain proportion of a word is closely related to its frequency of occurrence.
- Some words with simple semantic meanings do not need to borrow much knowledge from other domains, e.g., and; Some other words need to borrow knowledge from other domains to better understand their own semantic meaning. For example, the
word phenomenon keeps borrowing/sharing knowledge from/to the medical domain at every layer.
- The ending of the sentence only conveys a stopping signal, and thus is shared across all domains.
- The domain proportions at the bottom layers tend to be more diverse, while those at the top layers tend to be more skewed, as shown in Figure 8 for English-to-German task.
- The domain proportions of the decoder tend to be more skewed than those of the encoder, which demonstrates little knowledge sharing. Figure 9 shows the histograms of word-level domain proportions at different layers in both the encoder and decoder. This might explain why the mixing decoder only contributes limited performance gain for the English-to-German task.
Figure 7: Domain proportion of a sentence from the TED domain for English-to-French task. The domain proportion is extracted from all layers of the encoder.
Figure 9: Histograms of the domain proportions of each layer in our domain mixing model for English-to-German Task. Within each histogram, 0 means pure News domain, and 1 means pure TED domain.
4.6 Combining Domain Mixing with Domain Aware Embedding
The embedding based methods can be naturally combined with our domain mixing methods. As we mentioned in 4.2, the domain proportion is trained solely, meaning gradient does not propagate between the domain proportion layers  $\mathcal{D}$  and
Figure 10: Back-propagation for different embedding based methods.
the Transformer. The computation of the gradient, on the other hand, is the key to combining two methods. Specifically, we encourage the embedding to be domain aware via MTL, AdvL and PAdvL, where we use the domain proportion layers to guide the training of the embedding. Figure 10 illustrates the back-propagation under different methods. Table 6 shows the performance for Chinese-to-English task under this setting. Here we consider applying domain mixing only to the encoder as the baseline. As can be seen, by applying appropriate domain aware embedding, the performance can be further improved.
Method	Laws	News	Speech	Thesis
Encoder	50.21	27.94	16.85	12.03
+MTL	49.15	26.82	15.72	11.93
+Adv	50.18	27.72	16.99	12.16
+PAdvL	49.01	26.63	16.06	12.15
+Multitask + WL	48.75	26.78	16.53	12.11
+Adv + WL	50.24	28.21	16.98	12.00
+PAdv + WL	48.87	26.86	16.14	11.89
One major challenge in multi-domain machine translation is the word ambiguity in different domains. For example, the word "article" has different meanings in the domains of laws and media. When translating "article" into Chinese, the translated words are "条款" and "文章", meaning a separate clause of a legal document and a piece of writing. Our proposed word-level layer-wise domain mixing approach tends to reduce the word ambiguity. As mentioned in Section 3.3, our model extracts different representations of each word from contexts at different layers. Accordingly, the domain proportion of each word evolves from bottom to top layers, and can eventually help identify the corresponding domains.
Table 6: BLEU Scores of Domain Mixing + Domain Aware Embedding for Chinese-to-English Task
Laws	“Article 37 The freedom of marriage ...”
“第三十七条:婚姻的自由...”
Media	“... working on an article about the poems ...”
“...正在写一篇诗的文章...”
Table 7: The ambiguity of "articles".
Moreover, as mentioned in Section 3.2, the positional embedding also contributes to the word disambiguation in multi-domain translation. For example, in the law domain, we find that "article" often appears at the beginning of a sentence, while in the media domain, the word "article" may appear in other positions. Therefore, varying domain proportions for different positions can help with word disambiguation.
We remark that word disambiguation across domains actually requires  $\mathcal{D}(x)$  to be powerful for predicting the domain of the word. However, a powerful  $\mathcal{D}(x)$  tends to yield skewed domain proportions and is not flexible enough for domain knowledge sharing. To trade off between strength and flexibility of  $\mathcal{D}(x)$ , the smoothing parameter  $\epsilon$  of  $\mathcal{D}(x)$  (see Section 3.1) needs to be properly set.
We present a novel multi-domain NMT with word-level layer-wise domain mixing, which can adaptively exploit the domain knowledge. Unlike the existing work, we construct multi-head dot-product modules for each domain and then combine them by the layer-wise domain proportion of every word. The proposed method outperforms the existing embedding based methods. We also show mixing method can be combined with embedding based methods to make further improvement.
Moreover, we remark that our approach can be extended to other multi-domain or multi-task NLP problems.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Denny Britz, Quoc Le, and Reid Pryzant. 2017. Effective domain mixing for neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 118-126.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Katsuitho, Yoshino Koichiro, and Federmann Christian. 2017.
Overview of the iwslt 2017 evaluation campaign. In International Workshop on Spoken Language Translation, pages 2-14.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th iwslt evaluation campaign, iwslt 2014. In Proceedings of the International Workshop on Spoken Language Translation, Hanoi, Vietnam, page 57.
Boxing Chen, Colin Cherry, George Foster, and Samuel Larkin. 2017. Cost weighting for neural machine translation domain adaptation. In Proceedings of the First Workshop on Neural Machine Translation, pages 40-46.
Chenhui Chu and Raj Dabre. 2019. Multilingual multi-domain adaptation approaches for neural machine translation. arXiv preprint arXiv:1906.07978.
Chenhui Chu and Rui Wang. 2018. A survey of domain adaptation for neural machine translation. arXiv preprint arXiv:1806.00258.
The Linguistic Data Consortium. 1992. The linguistic data consortium description.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Zi-Yi Dou, Junjie Hu, Antonios Anastasopoulos, and Graham Neubig. 2019. Unsupervised domain adaptation for neural machine translation with domain-aware feature embeddings. arXiv preprint arXiv:1908.10430.
M Amin Farajian, Marco Turchi, Matteo Negri, Nicola Bertoldi, and Marcello Federico. 2017. Neural vs. phrase-based machine translation in a multi-domain scenario. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 280-284.
Shuhao Gu, Yang Feng, and Qun Liu. 2019. Improving domain adaptation translation with domain invariant and specific information. arXiv preprint arXiv:1904.03879.
Barry Haddow and Philipp Koehn. 2012. Analysing the effect of out-of-domain data on smt systems. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 422-432. Association for Computational Linguistics.
Kazuma Hashimoto, Akiko Eriguchi, and Yoshimasa Tsuruoka. 2016. Domain adaptation and attention-based unknown word replacement in chinese-to-japanese neural machine translation. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 75-83.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700-1709.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In  $MT$  summit, volume 5, pages 79-86.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, pages 177-180.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. arXiv preprint arXiv:1804.07755.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Minh Quang Pham, Josep-Maria Crego, Jean Senellart, and François Yvon. 2019. Generic and specialized word embeddings for multi-domain machine translation.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
Christophe Servan, Josep Crego, and Jean Senellart. 2016. Domain specialization: a post-training domain adaptation for neural machine translation. arXiv preprint arXiv:1612.06141.
Yan Shen, Dahlmann Leonard, Petrushkov Pavel, Hewavitharana Sanjika, and Khadivi Shahram. 2017. Word-based domain adaptation for neural machine translation. In Proceedings of the 15th International Workshop on Spoken Language Translation, pages 31-38.
Jinsong Su, Jiali Zeng, Jun Xie, Huating Wen, Yongjing Yin, and Yang Liu. 2019. Exploring discriminative word-level domain contexts for multidomain neural machine translation. IEEE transactions on pattern analysis and machine intelligence.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826.
Sander Tars and Mark Fishel. 2018. Multi-domain neural machine translation. arXiv preprint arXiv:1805.02282.
Liang Tian, Derek F Wong, Lidia S Chao, Paulo Quaresma, and Francisco Oliveira. Um-corpus: A large english-chinese parallel corpus for statistical machine translation.
Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A conditional random field word segmenter for sigan bakeoff 2005. In Proceedings of the fourth SIGHAN workshop on Chinese language Processing.
Brian Tubay and Marta R Costa-jussà. 2018. Neural machine translation with the transformer and multisource romance languages for the biomedical wmt 2018 task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 667-670.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Rui Wang, Masao Utiyama, Andrew Finch, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2018. Sentence selection and weighting for neural machine
translation domain adaptation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(10):1727-1741.
Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017. Instance weighting for neural machine translation domain adaptation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1482-1488.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Antonio Jimeno Yepes, Aurélie Néveol, Mariana Neves, Karin Verspoor, Ondrej Bojar, Arthur Boyer, Cristian Grozea, Barry Haddow, Madeleine Kittner, Yvonne Lichtblau, et al. 2017. Findings of the wmt 2017 biomedical translation shared task. In Proceedings of the Second Conference on Machine Translation, pages 234-247.
Jiali Zeng, Jinsong Su, Huating Wen, Yang Liu, Jun Xie, Yongjing Yin, and Jianqiang Zhao. 2018. Multidomain neural machine translation with word-level domain context discrimination. In Conference on Empirical Methods in Natural Language Processing, pages 447-457.
A Complementary Experiments - Chinese to English
Experiment results of the original Transformer, where layer normalization is at the end each layer.
Method	Laws	News	Spoken	Thesis
Laws	10.37	0.45	0.27	0.27
News	0.39	5.12	0.91	0.57
Spoken	0.70	1.11	6.19	0.83
Thesis	0.63	0.25	0.16	1.24
Mixed	5.45	4.09	2.67	1.85
Multitask	6.16	3.83	1.91	1.53
Adversarial	5.93	3.38	1.85	1.37
PAdv	6.58	3.90	2.32	1.80
WDC. w/ WL	7.13	3.87	2.45	1.88
Our Proposed Mixing Method
Encoder	50.16	27.61	16.92	11.85
+ Decoder	50.45	28.15	17.45	11.62
Table 8: Chinese to English
Layer normalization after attention
Layer normalization before attention
Figure 11: Two variants of layer normalization