SlowGuess/ACL_2020_OCR huggingface dataset:
Haoming Jiang
jianghm@gatech.edu
cjiang73@gatech.edu
Chong Wang
chong.wang@bytedance.com
Many multi-domain neural machine translation (NMT) models achieve knowledge transfer by enforcing one encoder to learn shared embedding across domains. However, this design lacks adaptation to individual domains. To overcome this limitation, we propose a novel multi-domain NMT model using individual modules for each domain, on which we apply word-level, adaptive and layer-wise domain mixing. We first observe that words in a sentence are often related to multiple domains. Hence, we assume each word has a domain proportion, which indicates its domain preference. Then word representations are obtained by mixing their embedding in individual domains based on their domain proportions. We show this can be achieved by carefully designing multi-head dot-product attention modules for different domains, and eventually taking weighted averages of their parameters by word-level layer-wise domain proportions. Through this, we can achieve effective domain knowledge sharing, and capture fine-grained domain-specific knowledge as well. Our experiments show that our proposed model outperforms existing ones in several NMT tasks.
Neural Machine Translation (NMT) has made significant progress in various machine translation tasks (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2014; Luong et al., 2015; Wu et al., 2016). The success of NMT heavily relies on a huge amount of annotated parallel sentences as training data, which is often limited in certain domains, e.g., medical domain. One approach to address this is to explore unparalleled corpora, such as unsupervised machine translation (Lample et al., 2017, 2018). Another approach is to train a multi-domain NMT model and this is
the focus of this paper. The simplest way is to build a unified model by directly pooling all training data from multiple domains together, as the languages from different domains often share some similar semantic traits, e.g., sentence structure, textual style and word usages. For domains with less training data, the unified model usually shows significant improvement.
Researchers have proposed many methods for improving multi-domain NMT. Though certain semantic traits are shared across domains, there still exists significant heterogeneity among languages from different domains. For example, Haddow and Koehn (2012) show that for a domain with sufficient training data, a unified model may lead to weaker performance than the one trained solely over the domain; Farajian et al. (2017); Luong et al. (2015); Sennrich et al. (2015a); Servan et al. (2016) also show that to improve the translation performance over certain domains, fine-tuning the unified model is often needed, but at the expense of sacrificing the performance over other domains. This indicates that a unified model might not well exploit the domain-specific knowledge for each individual domain.
To overcome this drawback, two lines of recent research focus on developing new methods by exploiting domain-shared and domain-specific knowledge to improve multi-domain NMT (Britz et al., 2017; Zeng et al., 2018; Tars and Fishel, 2018; Hashimoto et al., 2016; Wang et al., 2017; Chen et al., 2017; Wang et al., 2018; Gu et al., 2019; Chu and Wang, 2018; Dou et al., 2019; Pham et al., 2019; Chu and Dabre, 2019).
One line of research focuses on instance weighting, which assigns domain related weights to different samples during training. For example, Wang et al. (2017) consider sentence weighting and domain weighting for NMT. The sentence weight is determined by the bilingual cross-entropy of each
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1823-1834
sentence pair based on the language model of each domain. The domain weight can be modified by changing the number of sentences from that domain in a mini-batch. Chen et al. (2017) propose a cost weighting method, where the weight of each pair of sentences is evaluated by the output probability of a domain classifier on the encoder embedding. Wang et al. (2018) propose a dynamic training method to adjust the sentence selection and weighting during training. We remark that many of these methods are complementary to our proposed model, and can be applied to improve the training of our model.
Another line of research attempts to design specific encoder-decoder architectures for NMT models. For example, Britz et al. (2017) consider domain-aware embedding given by the encoder, and then jointly train a domain classifier, taking the embedding as input to incorporate the domain information. Zeng et al. (2018); Su et al. (2019) further extend their approach by separating the domain-shared and domain-specific knowledge within the embedding. In addition, Zeng et al. (2018) and Shen et al. (2017) propose a maximum weighted likelihood estimation method, where the weight is obtained by word-level domain aware masking to encourage the model to pay more attention to the domain-specific words. The aforementioned methods, however, have a notable limitation: They enforce one single encoder to learn shared embedding across all domains, which often lacks adaptivity to each individual domain.
To better capture domain-shared knowledge beyond shared embedding from a single encoder, we propose a novel multi-domain NMT model using individual modules for each domain, on which we apply word-level, adaptive and layer-wise domain mixing. Our proposed model is motivated by the observation that although every sentence of the training data has a domain label, the words in the sentence are not necessarily only related to that domain. For instance, the word "article" appears in the domains of laws and business. Therefore, we expect the knowledge for translating the word "article" to be shared between these two domains. Our proposed model assigns a context-dependent domain proportion<sup>1</sup> to every word in the sentence. The domain proportions of the words can be naturally integrated into the Transformer model for capturing domain-shared/specific knowledge, as
the multi-head dot-product attention mechanism is applied at the word-level. Specifically, we carefully design multi-head dot-product attention modules for different domains, and eventually mix these modules by taking weighted averages of their parameters by their layer-wise domain proportions.
Compared with existing models, ours has the following two advantages:
- Our proposed model is more powerful in capturing the domain-specific knowledge, as we design multiple dot-product attention modules for different domains. In contrast, existing models rely on one single shared encoder, and then one single unified translation model is applied, which often cannot adapt to each individual domain very well.
- Our proposed model is more adaptive in the process of domain knowledge sharing. For common words across domains, their domain proportions tend to be uniform, and therefore can significantly encourage knowledge sharing. For some words specific to certain domains, their domain proportions tend to be skewed, and accordingly, the knowledge sharing is encouraged only within the relevant domains. For example, the word "article" appears less in the medical domain than the domains of laws and business. Therefore, the corresponding domain proportion tends to favor the domains of laws and business more than the medical domain.
We evaluate our proposed model in several multi-domain machine translation tasks, and the empirical results show that our proposed model outperforms existing ones and improves the translation performance for all domains.
The rest of the paper is organized as follows: Section 2 introduces the background; Section 3 describes our proposed model in detail; Section 4 presents numerical experiments on EN-DE, ENFR and ZH-EN datasets; Section 5 discusses the connection to word disambiguation.
Neural Machine Translation (NMT) directly models the conditional distribution of the translated sentence  $\mathbf{y} = (y_{1},\dots,y_{\ell})$  given a source sentence  $\mathbf{x} = (x_{1},\dots,x_{\ell})^{2}$ . The conditional probability density function  $p(\mathbf{y}|\mathbf{x})$  is parameterized by an encoder-decoder neural network: The encoder
<sup>1</sup>A word actually has multiple domain proportions at different layers of our model. See more details in Section 3
2Here we assume that we have applied padding to all sentences, and therefore, they are all of the same length.
encodes the source sentence into a sequence of hidden representations  $\mathcal{H}(\mathbf{x}) = (h_1,\dots,h_n)$ , and the decoder generates target sentence one token at a time using these intermediate representations. More specifically, the decoder usually contains a recursive structure for computing  $p(y_t|y_{<t},\mathbf{x})$  by
$$
p (y _ {t} | y _ {<   t}, \mathbf {x}) = \mathcal {F} (\mathcal {G} _ {t}, \mathcal {H} (\mathbf {x}), y _ {t - 1}),
$$
where  $\mathcal{G}_t$  denotes the hidden representation of the decoder for the  $t$ -th position of the sequence, and  $\mathcal{F}$  denotes a multi-layered network that outputs the probability of  $y_{t}$ . Notice that  $\mathcal{G}_t$  is generated by the  $\mathcal{G}_{t-1}, \mathcal{H}(\mathbf{x})$ , and the previous word  $y_{t-1}$ . Given  $N$  pairs of source/target sequences denoted by  $\{\mathbf{x}_i, \mathbf{y}_i\}_{i=1}^n$ , we train the NMT model by minimizing the cross-entropy loss as follows,
$$
\min _ {\mathcal {H}, \mathcal {G}, \mathcal {F}} L _ {\mathrm {g e n}} = \frac {1}{n} \sum_ {i = 1} ^ {n} - \log p (\mathbf {y} _ {i} | \mathbf {x} _ {i})
$$
where  $p(\mathbf{y}_i|\mathbf{x}_i) = \prod_{t = 1}^{m}p(y_{i,t}|y_{i, < t},\mathbf{x}_i)$ .
Transformer is one of the most popular NMT models (Vaswani et al., 2017; Tubay and Costa-jussa, 2018; Devlin et al., 2018). The encoder and decoder in Transformer contain stacked self-attention and point-wise, fully connected layers without any explicit recurrent structure, which is different from existing RNN-based NMT models.
Specifically, Vaswani et al. (2017) propose a new attention function using the scaled dot-product as the alignment score, which takes the form,
$$
\operatorname {A t t e n t i o n} (Q, K, V) = \operatorname {s o f t m a x} \left(\frac {Q K ^ {\top}}{\sqrt {d}}\right) V, (1)
$$
where  $Q, K, V \in \mathbb{R}^{\ell \times d}$  are the vector representations of all the words in the sequences of queries, keys and values accordingly. For the self-attention modules in the encoder and decoder,  $Q = K = V$ ; For the attention module that takes into account the encoder and the decoder sequences,  $Q$  is different from the sequence represented by  $V$  and  $K$ .
Based on the above attention function in (1), Vaswani et al. (2017) further develop a multi-head attention module, which allows the NMT model to jointly attend to information from different representations at different positions. In particular, we consider a multi-head attention module with  $m$  heads. For the  $i$ -th head  $H_{i}$ , three point-wise linear transformations  $W_{i,Q}, W_{i,K}, W_{i,V} \in \mathbb{R}^{d \times d / m}$  are first applied to the input  $Q$ ,  $K$  and  $V$ , respectively, and then the scaled dot-product attention
Figure 1: Multi-head Scaled Dot-Product Attention.
is applied: Let  $\widetilde{Q}_i = QW_{i,Q}$ ,  $\widetilde{K}_i = KW_{i,K}$  and  $\widetilde{V} = VW_{i,V}$ ,
$$
H _ {i} = \operatorname {A t t e n t i o n} \left(\widetilde {Q} _ {i}, \widetilde {K} _ {i}, \widetilde {V} _ {i}\right). \tag {2}
$$
Eventually, the final output applies a point-wise linear transformation  $W_O \in \mathbb{R}^{d \times d}$  to the concatenation of the output from all heads:
$\operatorname{MultiHead}(Q, K, V) = \operatorname{Concat}(H_1, \dots, H_m)W_O.$
An illustrative example of the multihead attention architecture is provided in Figure 1.
In addition to the above multi-head attention modules, each layer in the encoder and decoder in Transformer contains a point-wise two-layer fully connected feed-forward network.
We present our Transformer-based multi-domain neural machine translation model with word-level layer-wise domain mixing.
3.1 Domain Proportion
Our proposed model is motivated by the observation that although every sentence in the training data has a domain label, a word in the sentence does not necessarily only belong to that single domain. Therefore, we assume that every word in the vocabulary has a domain proportion, which indicates its domain preference. Specifically, given the embedding  $x \in \mathbb{R}^d$  of a word,  $k$  domains and  $R \in \mathbb{R}^{k \times d}$ , our model represents the domain proportion by a smoothed softmax layer as follows,
$$
\mathcal {D} (x) = (1 - \epsilon) \cdot \operatorname {s o f t m a x} (R x) + \epsilon / k,
$$
where  $\epsilon \in (0,1)$  is a smoothing parameter to prevent the output of  $\mathcal{D}(x)$  from collapsing towards 0 or 1. Specifically, setting  $\epsilon$  as a large value encourages the word to be shared across domains.
3.2 Word-Level Adaptive Domain Mixing
In our proposed model, each domain has its own multi-head attention modules. Recall that the pointwise linear transformations in the multi-head attention module  $W_{i,Q}$ 's,  $W_{i,K}$ 's,  $W_{i,V}$ 's and  $W_O$  are applied to each word separately and identically, as shown in Figure 2. Therefore, we can naturally
Figure 2: The Point-wise Linear Transformations are applied at the word-level.
integrate the domain proportions of the words with these multi-head attention modules. Specifically, we take the weighted averaging of the linear transformation based on the domain proportion  $\mathcal{D}(x)$ . For example, we consider the point-wise linear transformations  $\{W_{i,Q,j}\}_{j=1}^{k}$  on the  $t$ -th word of the input,  $Q_t$ , of all domains. The mixed linear transformation can be written as
$$
\overline {{Q}} _ {i, t} = \sum_ {j = 1} ^ {k} Q _ {t} ^ {\top} W _ {i, Q, j} \mathcal {D} _ {Q, j} (Q _ {t}),
$$
where  $\mathcal{D}_{Q,j}(Q_t)$  denotes the  $j$ -th entry of  $\mathcal{D}_Q(Q_t)$ , and  $\mathcal{D}_Q$  is the domain proportion layer related to  $Q$ . Then we only need to replace  $\widetilde{Q}_i$  in (2) with
$$
[ \overline {{Q}} _ {i, 1}, \dots , \overline {{Q}} _ {i, n} ].
$$
An illustrative example is presented in Figure 3. For other linear transformations, we applied the domain mixing scheme in the same way. We re
Figure 3: Word-level mixing with 3 domains. For simplicity, we omit the subscripts  $Q$ ,  $i$ .
mark that the Transformer model, though does not have any explicit recurrent structure, handles the sequence through adding additional positional embedding for each word (in conjunction with sequential masking). Therefore, if a word appears
in different positions of a sentence, its corresponding embedding is different. This indicates that the domain proportions of the same word can also be different across positions. This feature makes our model more flexible, as the same word in different positions can carry different domain information.
3.3 Layer-wise Domain Mixing
Recall that the Transformer model contains multiple multi-head attention modules/layers. Therefore, our proposed model inherits the same architecture and applies the word-level domain mixing to all these attention layers. Since the words have different representations at each layer, the corresponding domain proportions at each layer are also different, as shown in Figure 4. In addition to the multi-head attention layers, we also apply similar word-level domain mixing to the point-wise two-layer fully connected feed-forward network.
The layer-wise domain mixing allows the domain proportions to be context dependent. This is because the domain proportions are determined by the word embedding, and the word embedding at top layers is essentially learnt from the representations of all words at bottom layers. As a result, when the embedding of a word at some attention layer is already learned well through previous layers (in the sense that it contains sufficient contextual information and domain knowledge), we no longer need to borrow knowledge from other domains to learn the embedding of the word at the current layer. Accordingly, the associated domain proportion is expected to be skewed and discourages knowledge sharing across domains. This makes the process of knowledge sharing of our model more adaptive.
Recall that  $\mathcal{H}$  denotes the encoder,  $\mathcal{F}$  denotes the decoder, and  $\mathcal{D}$  denotes the domain proportion. Define  $\Theta = \{\mathcal{F},\mathcal{H},\mathcal{D}\}$ . The proposed model can be efficiently trained by minimizing a composite loss function defined as follows,
$$
L ^ {*} = L _ {\mathrm {g e n}} (\Theta) + L _ {\mathrm {m i x}} (\Theta),
$$
where  $L_{\mathrm{gen}}(\Theta)$  denotes the cross-entropy loss over the training data  $\{\mathbf{x}_i,\mathbf{y}_i\}_{i = 1}^n$  , and  $L_{\mathrm{mix}}(\Theta)$  denotes the cross entropy loss over the words/domain (hard) labels.
For  $L_{\mathrm{mix}}(\Theta)$ , the domain labels are obtained from the training data. Specifically, for all words
Figure 4: Illustration of Our Multi-domain NMT Model: Normalization and residual connection are omitted for simplicity. For all other detail, please refer to Vaswani et al. (2017).
in a sentence belonging to the  $J$ -th domain, we specify their domain hard labels as  $J$ . Then given the embedding  $x$  of a word, we compute the cross entropy loss of its domain proportion  $\mathcal{D}(x)$  as  $-\log (\mathcal{D}_J(x))$ . Accordingly,  $L_{\mathrm{mix}}(\Theta)$  is the sum of the cross entropy loss over all such pairs of word/domain label of the training data.
We conduct experiments on three different machine translation tasks:
- English-to-German. We use a dataset from two domains: News and TED. We collect the News domain data from Europarl (Koehn, 2005) and
the TED domain data from IWLST (Cettolo et al., 2014).
- English-to-French We use a dataset containing two domains: TED and Medical domain. We collect TED domain data from IWLST (Cettolo et al., 2017) and medical domain data from Medline (Yepes et al., 2017).
- Chinese-to-English We use a dataset containing four domains: News, Speech, Thesis and Laws. We collect the Laws, Speech, and Thesis data from UM-Corpus (Tian et al.), and the News data from LDC (Consortium, 1992). The translation from Chinese-to-English is inherently difficult. The four-domains setting makes it even more challenging. This dataset is also used in Zeng et al. (2018).
The sizes of training, validation, and testing sets for different language pairs are summarized in Table 1. We tokenize English, German and French sentences using MOSES script (Koehn et al., 2007) and perform word segmentation on Chinese sentences using Stanford Segmenter (Tseng et al., 2005). All sentences are then encoded using byte-pair encoding (Sennrich et al., 2015b). We evaluate the performance using two metrics: BLEU (Papineni et al., 2002) and perplexity following the default setting in fairseq with beam search steps of 5.
Language	Domain	Train	Valid	Test
EN-DE	News	184K	18K	19K
TED	160K	7K	7K
EN-FR	TED	226K	10K	10K
MEDICAL	516K	25K	25K
ZH-EN	Laws	219K	600	456
News	300K	800	650
Speech	219K	600	455
Thesis	299K	800	625
Table 1: The numbers of sentences in the datasets.
Our baselines include the Transformer models trained using data from single and all domains. We also include several domain aware embedding based methods, which train the embedding of the encoder along with domain information.
- Multitask Learning (MTL) proposed in Britz et al. (2017) uses one sentence-level domain classifier to train the embedding. Note that their classifier is only used to predict the domain, while our model
uses multiple word-level domain classifiers to obtain the domain proportions for different layers (further used for domain mixing).
- Adversarial Learning (AdvL) proposed in Britz et al. (2017) is a variant of MTL, which flips the gradient before it is back-propagated into the embedding. This encourages the embedding from different domains to be similar.
- Partial Adversarial Learning (PAdvL) To combine the advantages of the above two methods, we split the embedding into half of multitask part and half of adversarial part.
- Word-Level Domain Context Discrimination (WDC) Zeng et al. (2018) integrates MTL and AdvL with word-level domain contexts. This method requires the dimension of the embedding to be doubled and, thus, is not directly applicable in Transformer. We use a point-wise linear transformation to reduce the dimension.
Moreover, Zeng et al. (2018) consider the word-level domain aware weighted loss (WL). Specifically, they assign a domain-aware attention weight  $\beta_{j}$  to the  $j$ -th position in the output sentence, and the corresponding weighted loss is:
$$
L _ {\mathrm {g e n}} = - \frac {1}{n} \sum_ {j = 1} ^ {n} (1 + \beta_ {j}) \log p (y _ {j} | \mathbf {x}, y _ {<   j}).
$$
Here  $\beta_{j}$  is obtained by an attention based domain classifier built upon the last hidden layer.
4.2 Details of Our Implementation
All of our experiments are conducted under fairseq (Ott et al., 2019) environment. We follow the fairseq re-implementation of 12-layer Transformer designed for IWLST data. Specifically, the embedding dimension is 512 for both the encoder and decoder, the number of heads is 4, and the embedding dimension in the feed-forward layer is 1024. Such a model is actually larger than the base model in Vaswani et al. (2017) (76M vs. 65M parameters). Notice that, the number of parameters of the mixing model is  $k$  times larger ( $k$  is the number of domains). For a fair comparison, all baselines are tested using both the above model and an enlarged model, which has  $\sqrt{k}$  times larger embedding dimension (so the weight matrices are  $k$  times larger). The enlarged model and the mixing model has the same number of parameters. The presented baseline results are the best of the two. In terms of the optimization, we follow the training recipe
provided by faireq. Specifically, we use Adam (Kingma and Ba, 2014) with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$  with a weight decay parameter of  $10^{-4}$ . The learning rate follows the inverse square root schedule (Vaswani et al., 2017) with warm-up steps of 4000, initial warm-up learning rate of  $10^{-7}$ , and the highest learning rate of  $5 \times 10^{-4}$ . For effective training,  $L_{\mathrm{gen}}$  is replaced by a label-smoothing cross-entropy loss with a smoothing parameter of 0.1 (Szegedy et al., 2016).
For our domain mixing methods, we set the smoothing parameter  $\epsilon$  of the domain proportion as 0.05. Besides applying domain mixing to both the encoder and decoder (E/DC), we consider applying domain mixing to only the Encoder. The domain proportion layers  $\mathcal{D}$  are only used for estimating the domain proportion and should not intervene in the training of the translation model. So the gradient propagation is cut off between the Transformer and the domain proportion as Figure 5 shows. More discussion about the training procedure can be found in Section 4.6.
Figure 5: Computational graph for training the domain proportion layers.
Table 2 shows the BLEU scores of the baselines and our domain mixing methods for English-to-German translation. As can be seen, our methods outperform the baselines on both domains. Notice that, our baseline method achieves 29.09 BLEU when training and testing on TED domain only, where Liu et al. (2019) only achieves 28.56 with the same training/testing data, the codebase (i.e., fairseq), and the network structure. This indicates that our reimplemented baseline is rather strong.
We also compare the perplexity on the validation set in Figure 6. As can be seen, our domain mixing methods converge faster than the baselines and all methods converge after 50 epochs. We also observe that the baselines get stuck at plateaus at the early
Method	News	TED
Direct Training
News	26.09	6.15
TED	4.90	29.09
News + TED	26.06	28.11
Embedding based Methods
MTL	26.90	29.27
AdvL	25.68	27.46
PAdvL	27.06	29.49
WDC + WL	27.25	29.43
Our Domain Mixing Methods
Encoder	27.78	30.30
Encoder + WL	27.67	30.11
E/DC	27.58	30.33
E/DC + WL	27.55	30.22