SlowGuess/ACL_2020_OCR huggingface dataset:
wangwe@google.com
Ye Tian
ytian@google.com
Jiquan Ngiam
jngiam@google.com
Yinfei Yang
yinfeiy@google.com
Isaac Caswell
icaswell@google.com
Zarana Parekh
zarana@google.com
Most data selection research in machine translation focuses on improving a single domain. We perform data selection for multiple domains at once. This is achieved by carefully introducing instance-level domain-relevance features and automatically constructing a training curriculum to gradually concentrate on multi-domain relevant and noise-reduced data batches. Both the choice of features and the use of curriculum are crucial for balancing and improving all domains, including out-of-domain. In large-scale experiments, the multidomain curriculum simultaneously reaches or outperforms the individual performance and brings solid gains over no-curriculum training.
In machine translation (MT), data selection, e.g., (Moore and Lewis, 2010; Axelrod et al., 2011), has remained as a fundamental and important research topic. It has played a crucial role in domain adaptation by selecting domain-matching training examples, or data cleaning (aka denoising) by selecting high-quality examples. So far, the most extensively studied scenario assumes a single domain to improve.
It becomes both technically challenging and practically appealing to build a large-scale multidomain neural machine translation (NMT) model that performs simultaneously well on multiple domains at once. This requires addressing research challenges such as catastrophic forgetting (Goodfellow et al., 2014) at scale and data balancing. Such a model can easily find potential use cases, i.e., as a solid general service, for downstream transfer learning, for better deployment efficiency, or for transfer learning across datasets.
Unfortunately, existing single-domain data-selection methods do not work well for multiple domains. For example, improving the translation
Static	Dynamic
Single	domain noise	Y	Y
Y	Y
Multi	domain noise	Y	N (Our Work)
N
Table 1: Data selection and data mixing research in NMT. 'Y': There is previous research that studies this case. 'N': No previous research has studied this case.
accuracy of one domain will often hurt that of another (van der Wees et al., 2017; Britz et al., 2017), and improving model generalization across all domains by clean-data selection (Koehn et al., 2018) may not promise optimization of a particular domain. Multiple aspects need to be considered for training a multi-domain model.
This paper presents a dynamic data selection method to multi-domain NMT. Things we do differently from previous work in mixing data are the choice of instance-level features and the employment of a multi-domain curriculum that is additionally able to denoise. These are crucial for mixing and improving all domains, including out-of-domain. We experiment with large datasets at different noise levels and show that the resulting models meet our requirements.
In MT, research that is most relevant to our work is data selection and data mixing, both being concerned with how to sample examples to train an MT model, usually for domain adaptation. Table 1 categorizes previous research by two aspects and shows where our work stands. These two aspects are:
1. Is the method concerned with a single domain or multiple domains?
2. Does the method use data statically or dynamically?
7711
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7711-7723
Static data selection for a single domain. Moore and Lewis (2010) select in-domain data for n-gram language model (LM) training. It is later generalized by Axelrod et al. (2011) to select parallel data for training MT models. Chen and Huang (2016); Chen et al. (2016) use classifiers to select domain data. Clean-data selection (Koehn et al., 2019, 2018; Junczys-Dowmunt, 2018) reduces harmful data noise to improve translation quality across domains. All these works select a data subset for a single "domain" and treat the selected data as a static/flat distribution.
Dynamic data selection for a single domain. Static selection has two shortcomings: it discards data and it treats all examples equally after selection. When data is scarce, any data could be helpful, even if it is out of domain or noisy<sup>2</sup>. Dynamic data selection is introduced to "sort" data from least in-domain to most in-domain. Training NMT models on data sorted this way effectively takes advantage of transfer learning. Curriculum learning (CL) (Bengio et al., 2009) has been used as a formulation for dynamic data selection. Domain curricula (van der Wees et al., 2017; Zhang et al., 2019) are used for domain adaptation. Model stacking (Sajjad et al., 2017; Freitag and Al-Onaizan, 2016) is a practical idea to build domain models. CL is also used for denoising (Kumar et al., 2019; Wang et al., 2018a,b), and for faster convergence and improved general quality (Zhang et al., 2018; Platanios et al., 2019). Wang et al. (2018a) introduce a curriculum for training efficiency. In addition to data sorting/curriculum, instance/loss weighting (Wang et al., 2017; Chen et al., 2017; Wang et al., 2019b) has been used as an alternative. CL for NMT represents the SOTA data-selection method, but most existing works target at a single "domain", be it a specific domain or the "denoising domain".
Static data mixing for multiple domains. When mixing data from multiple domains, a fundamental challenge is to address catastrophic forgetting (Goodfellow et al., 2014)-training an NMT model to focus on one domain can likely hurt another (van der Wees et al., 2017; Britz et al.,
2017). Britz et al. (2017) learn domain-discerning (or -invariant) network representation with a domain discriminator network for NMT. The methods, however, require that domain labels are available in data. Tars and Fishel (2018) cluster data and tag each cluster as multi-domain NMT training data, but the method treats data in each cluster as a flat distribution. Farajian et al. (2017) implement multi-domain NMT by on-the-fly data retrieval and adaptation per sentence, at increased inference cost. Most existing methods (or experiment setups) have the following problems: (i) They mix data statically. (ii) They don't consider the impact of data noise, which is a source of catastrophic forgetting. (iii) Experiments are carried out with small datasets, without separate examination on the data regularization effect. (iv) They do not examine out-of-domain performance.
Automatic data balancing for multi-domains. (Wang et al., 2020) automatically learn to weight (flat) data streams of multi-languages (or "domains"). We perform dynamic data selection and regularization through a mulit-domain curriculum.
Automatic curriculum learning. Our work falls under automatic curriculum construction (Graves et al., 2017) and is directly inspired by Tsvetkov et al. (2016), who learn to weight and combine instance-level features to form a curriculum for an embedding learning task, through Bayesian Optimization. A similar idea (Ruder and Plank, 2017) is used to improve other NLP tasks. Here, we use the idea for NMT to construct a multi-domain data selection scheme with various selection scores at our disposal. The problem we study is connected to the more general multi-objective optimization problem. Duh (2018) uses Bandit learning to tune hyper-parameters such as the number of network layers for NMT.
More related work. Previously, catastrophic forgetting has mostly been studied in the continued-training setup (Saunders et al., 2019; Khayrallah et al., 2018), to refer to the degrading performance on the out-of-domain task when a model is fine-tuned on in-domain data. This setup is a popular topic in general machine learning research (Aljundi et al., 2019). Thompson et al. (2018) study domain adaptation by freezing subnetworks. Our work instead addresses forgetting in the data-balancing scenario for multi-domains. We use curriculum to generalize fine-tuning.
<sup>1</sup>We treat denoising as a domain in the paper, inspired by previous works that treat data noise using domain adaptation methods, e.g., (Junczys-Dowmunt, 2018). <sup>2</sup>We refer to data regularization (using more data) and to transfer learning (fine-tuning) to exploit both data quantity and quality, the idea behind dynamic data selection. See Appendix C.
7712
Figure 1: Data order in single-domain curricula and a potential multi-domain curriculum. (1) A toy training dataset of 3 examples. Each example has three scores, representing relevance to three domains, grey/dark/white domains, respectively. The higher the bar the more relevant. (2) Grey-domain order. (3) Dark-domain order. (4) White-domain order. (5) A potential multi-domain data order.
3 Curriculum Learning for NMT
We first introduce curriculum learning (CL) (Bengio et al., 2009), which serves as a formulation for SOTA single-domain dynamic data selection and which our method is built upon and generalizes. In CL, a curriculum,  $\mathcal{C}$ , is a sequence of training criteria over training steps. A training criterion,  $Q_{t}(y|x)$ , at step  $t$  is associated with a set of weights,  $W_{t}(x,y)$ , over training sentence pairs  $(x,y)$  in a parallel dataset  $D$ , where  $y$  is the translation for  $x$ .  $Q_{t}(y|x)$  is a re-weighting of the original training distribution  $P(y|x)$ :
$$
Q _ {t} (y | x) \propto W _ {t} (x, y) P (y | x), \quad \forall (x, y) \in D \tag {1}
$$
Hence, for  $T$  maximum training steps,  $\mathcal{C}$  is a sequence:
$$
\mathcal {C} = \left\langle Q _ {1}, \dots , Q _ {t}, \dots , Q _ {T} \right\rangle \tag {2}
$$
At step  $t$ , an online learner randomly samples a data batch from  $Q_{t}$  to fine-tune model  $m_{t-1}$  into  $m_{t}$ . Therefore,  $\mathcal{C}$  corresponds to a sequence of models,
$$
\langle m _ {1}, \dots , m _ {t}, \dots , M \rangle . \tag {3}
$$
$M$  is the final model that the entire curriculum has been optimizing towards. Intermediate models,  $m_{t}$ , serve as "stepping stones" to  $M$ , to transfer knowledge through them and regularize the training for generalization. A performance metric  $\mathcal{P}(\mathcal{C})$  evaluates  $M$  on a development or test set, after training on  $\mathcal{C}$ .
Table 2: Curriculum examples characterized by reweighting,  $W_{t}(x,y)$ , over three steps, to stochastically order data to benefit a final domain. Strikethrough discards examples. (1) corresponds to data order Figure 1 (2). (2) corresponds to data order Figure 1 (5).
In NMT, CL is used to implement dynamic data selection. First, a scoring function (Section 4.3) is employed to measure the usefulness of an example to a domain and sort data. Then mini-batch sampling, e.g., (Kocmi and Bojar, 2017), is designed to realize the weighting  $W_{t}$ , to dynamically evolve the training criteria  $Q_{t}$  towards in-domain. Figure 1 (1)-(4) illustrates the basic idea of the curriculum we use. (1) shows three sentence pairs,  $S_{1}, S_{2}, S_{3}$ , each having three scores, respectively representing usefulness to three domains. A grey-domain training curriculum, for example, relies on the data order in (2), gradually discards least useful examples according to  $W_{t}(x,y)$  (Eq. 1) in Table 2 (1): At step 1, the learner uniformly samples from all examples  $(W_{1})$ , producing model  $m_{1}$ . In step 2, the least-in-domain  $S_{3}$  is discarded (strikethrough) by  $W_{2}$  so we sample from subset  $\{S_{1}, S_{2}\}$  uniformly to reach  $m_{2}$ . We repeat this until reaching the final model  $M$ . In this process, sampling is uniform in each step, but in-domain examples (e.g.,  $S_{1}$ ) are reused more over steps. Similarly, we can construct the dark-domain curriculum in Figure 1 (3) and the white-domain (4).
4 Our Approach: Learning a Multi-Domain Curriculum
4.1 General Idea
The challenges in multi-domain/-task data selection lie in addressing catastrophic forgetting and data balancing. In Figure 1, while curriculum (2) moves a model to the grey-domain direction, this direction may not necessarily be positively consistent with the dark domain (Figure 1 (3)), causing dropped dark-domain performance. Ideally, a training example that introduces the least forgetting across all domains would have gradients that move the model in a common direction towards all domains. While this may not be easily feasible by selecting a single example, we would like the intuition to work in a data batch on average. Therefore, our idea is to carefully introduce
<sup>3</sup>As a preview, in our paper,  $W_{t}(x,y)$  uses uniform weights over selected examples and assigns zero weights for filtered examples, similar to a mask.
7713
Figure 2: Learning a multi-domain curriculum.
per-example data-selection scores (called features) to measure "domain sharing", intelligently weight them to balance the domains of interest, and dynamically schedule examples to trade-off between regularization and domain adaptation.
A method to realize the above idea has the following properties:
1. Features of an example reflect its relevance to domains.
2. Feature weights are jointly learned/optimized based on end model performance.
3. Training is dynamic, by gradually focusing on multi-domain relevant and noise-reduced data batches.
Furthermore, a viable multi-domain curriculum meets the following performance requirements:
(i) It improves the baseline model across all domains.
(ii) It simultaneously reaches (or outperforms) the peak performance of individual single-domain curricula.
Above requires improvement over out-of-domain, too.
4.2 The Framework
Formally, for a sentence pair  $(x,y)$ , let  $f_{n}(x,y) \in \mathbb{R}$  be its  $n$ -th feature that specifies how  $(x,y)$  is useful to a domain. Suppose we are interested in  $K$  domains and each example has  $N$  features. For instance, each sentence pair of  $S1, S2, S3$  in Figure 1 (1) has three features  $(N = 3)$ , each for one domain  $(K = 3)$ . We represent  $(x,y)$ 's features using a feature vector  $F(x,y) =$
$[f_0(x,y),\dots,f_{N - 1}(x,y)]$  . Given a weight vector  $V = [v_{0},\ldots ,v_{N - 1}]$  for all sentence pairs, we compute an aggregated score
$$
f (x, y) = V \cdot F (x, y) \tag {4}
$$
for each sentence pair and sort the entire data in increasing order. We then construct a curriculum  $\widehat{\mathcal{C}}(V)$  to fine-tune a warmed-up model, evaluate its performance and propose a next weight vector. After several iterations/trials, the optimal weight vector  $V^{*}$  is the one with the best end performance:
$$
V ^ {*} = \arg \max  _ {V} \mathcal {P} (\widehat {\mathcal {C}} (V)) \tag {5}
$$
Figure 2 shows the framework. For the process to be practical and scalable,  $\widehat{C}$  fine-tunes a warmed-up model for a small number of steps. The learned  $V^{*}$  can then eventually be used for retraining a final model from scratch.
4.3 Instance-Level Features
We design the following types of features for each training example and instantiate them in Experiments (Section 5).
NMT domain features  $(q_{Z})$  compute, for a pair  $(x,y)$ , the cross-entropy difference between two NMT models:
$$
q _ {Z} (x, y) = \frac {\log P (y | x ; \theta_ {Z}) - \log P (y | x ; \theta_ {\text {b a s e}})}{| y |} \tag {6}
$$
$P(y|x;\theta_{base})$  is a baseline model with parameters  $\theta_{base}$  trained on the background parallel corpus,  $P(y|x;\theta_Z)$  is a  $Z$ -domain model with  $\theta_Z$  by finetuning  $\theta_{base}$  on a small,  $Z$ -domain parallel corpus  $\widehat{D}_Z$  with trusted quality and  $|y|$  is the length of  $y$ .  $q_Z$  discerns both noise and domain  $Z$  (Wang et al., 2019a). Each domain  $Z$  has its own  $\widehat{D}_Z$ .
Importantly, Grangier (2019) shows that, under the Taylor approximation (Abramowitz and Stegun, 1964),  $q_{Z}$  approximates the dot product between gradient,  $g(x,y;\theta_{base})$ , of training example  $(x,y)$  and gradient,  $g(\widehat{D}_Z,\theta_{base})$ , of seed data  $\widehat{D}_{Z}$ . Thus an example with positive  $q_{Z}$  likely
$$
\begin{array}{l} q z (x, y) \times | y | = \\ \log P (y | x; \theta_ {Z}) - \log P (y | x; \theta_ {\text {b a s e}}) \approx \\ \lambda g (x, y; \theta_ {\text {b a s e}}) ^ {\top} g (\widehat {D} _ {Z}, \theta_ {\text {b a s e}}) \tag {7} \\ \end{array}
$$
when  $\theta_{base}$  and  $\theta_Z$  are close, which is the case for finetuning:  $\theta_Z = \theta_{base} + \lambda g(\widehat{D}_Z,\theta_{base})$ .
That is, according to Grangier (2019):
<sup>4</sup>But  $N$  does not necessarily equal  $K$  because we can introduce multiple features for one domain or a single feature for multiple domains.
moves a model towards domain  $Z$ . For multiple domains,  $Z_{1},\ldots,Z_{K}$ , selecting a batch of examples with  $q_{Z_k}$ 's all being positive would move a model towards a common direction shared across multiple domains, which alleviates forgetting.
The  $Z$ -domain feature  $q_{Z}(x,y)$  can be easily generalized into a single multi-domain feature,  $q_{\mathcal{Z}}$  for a set of domains  $\mathcal{Z}$ :
$$
q _ {\mathcal {Z}} (x, y) = \frac {\log P (y | x ; \theta_ {\mathcal {Z}}) - \log P (y | x ; \theta_ {\text {b a s e}})}{| y |} \tag {8}
$$
by simply concatenating all the seed parallel corpus  $\widehat{D}_{\mathcal{Z}}$  from the constituent domains into  $\widehat{D}_{\mathcal{Z}}$  and use it to fine-tune the baseline  $\theta_{base}$  into  $\theta_{\mathcal{Z}}$ . A benefit of  $q_{\mathcal{Z}}$  is scalability: using a single feature value to approximate  $(x,y)$ 's gradient consistency with the multiple domains at once. Simple concatenation means, however, domain balancing is not optimized as in Eq. 5.
NLM domain features  $(d_Z)$  (Moore and Lewis, 2010; van der Wees et al., 2017) compute  $Z$ -domain relevance of sentence  $x$  with neural language models (NLM), like  $q_Z$ :
$$
d _ {Z} (x) = \frac {\log P (x ; \vartheta_ {Z}) - \log P (x ; \vartheta_ {\text {b a s e}})}{| x |} \tag {9}
$$
where  $P(x; \vartheta_{base})$  is an NLM with parameters  $\vartheta_{base}$  trained on the  $x$  half of the background parallel data, and  $P(x; \vartheta_Z)$  is obtained by fine-tuning  $P(x; \vartheta_{base})$  on  $Z$ -domain monolingual data. Although  $d_Z$  may not necessarily reflect the translation gradient of an example under an NMT model, it effectively assesses the  $Z$ -domain relevance and, furthermore, allows us to include additional larger amounts of in-domain monolingual data. We do not use its bilingual version (Axelrod et al., 2011), but choose to consider only the source side, for simplicity.
Cross-lingual embedding similarity feature (emb) computes the cosine similarity of a sentence pair in a cross-lingual embedding space. The embedding model is trained to produce similar representations exclusively for true bilingual sentence pairs, following Yang et al. (2019).
BERT quality feature (BERT) represents quality scores from a fine-tuned multilingual BERT model (Devlin et al., 2018). We fine-tune a pre-trained BERT model<sup>6</sup> on a supervised dataset with positive and negative translation pairs.
These features compensate each other by capturing the information in a sentence pair from different aspects: NLM features capture domain. NMT features additionally discern noise. BERT and emb are introduced for denoising, by transferring the strength of the data they are trained on. All these features are from previous research and here we integrate them to solve a generalized problem.
4.4 Performance Metric  $\mathcal{P}$
Eq. 5 evaluates the end performance  $\mathcal{P}(\widehat{\mathcal{C}}(V))$  of a multi-domain curriculum candidate. We simply combine the validation sets from multi-domains into a single validation set to report the perplexity of the last model checkpoint, after training the model on  $\widehat{\mathcal{C}}(V)$ . The best multi-domain curriculum minimizes model's perplexity (or maximizes its negative per Eq. 5) on the mixed validation set. We experiment with different mixing ratios.
4.5 Curriculum Optimization
We solve Eq. 5 with Bayesian Optimization (BayesOpt) (Shahriari et al., 2016) as the optimizer in Figure 2. BayesOpt is derivative-free and can optimize expensive black-box functions, with no assumption of the form of  $\mathcal{P}$ . It has recently become popular for training expensive machine-learning models in the "AutoML" paradigm. It consists of a surrogate model for approximating  $\mathcal{P}(\widehat{\mathcal{C}}(V))$  and an acquisition function for deciding the next sample to evaluate. The surrogate model evaluates  $\widehat{\mathcal{C}}(V)$  without running the actual NMT training, by the Gaussian process (GP) priors over functions that express assumptions about  $\mathcal{P}$ . The acquisition function depends on previous trials, as well as the GP hyper-parameters. The Expected Improvement (EI) criterion (Srinivas et al., 2010) is usually used as acquisition function. Algo
<sup>6</sup>We use the public cased 12 layers multilingual model: multi_cased_L-12_H-768_A-12
rithm 1 depicts how BayesOpt works in our setup. We use Vizier (Golovin et al., 2017) for Batched Gaussian Process Bandit, but open-source implementations of BayesOpt are easily available. $^{7}$ .
4.6 Curriculum Construction
We pre-compute all features for each sentence pair  $(x,y)$  in training data and turn its features into a single score  $f(x,y)$  by Eq. 4, given a weight vector. We then construct a curriculum by instantiating its re-weighting  $W_{t}(x,y)$  (Eq. 1). To that end, we define a Boolean, dynamic data selection function  $\chi_{\rho}^{f}(x,y;t)$  to check, at step  $t$ , if  $(x,y)\in D$  belongs to the top  $\rho (t)$ -ratio examples in training data  $D$  sorted in increasing order of  $f(x,y)$ ,  $(0 < \rho \leq 1)$ . So  $\chi_{\rho}^{f}$  is a mask. Suppose  $n(t)$  examples are selected by  $\chi_{\rho}^{f}(x,y;t)$ , the re-weighting will then be
$$
W _ {t} (x, y) = 1 / n (t) \times \chi_ {\rho} ^ {f} (x, y; t). \tag {10}
$$
Filtered examples have zero weights and selected ones are uniformly weighted. We set  $\rho(t) = (1/2)^{t/H}$  to decay/tighten over time<sup>8</sup>, controlled by the hyper-parameter  $H$ . During training,  $\chi_{\rho}^{f}(x,y;t)$  progressively selects higher  $f(x,y)$ -scoring examples. In implementation, we integrate  $\chi_{\rho}^{f}(x,y;t)$  in the data feeder to pass only selected examples to the downstream model trainer; we also normalize  $f(x,y)$  offline to directly compare to  $\rho(t)$  online to decide filtering. As an example, the  $W_{t}(x,y)$  for the multi-domain curriculum order in Figure 1 (5) can look like Table 2 (2).
Data and domains. We experiment with two English  $\rightarrow$  French training datasets: the noisy ParaCrawl data $^9$  (290 million sentence pairs) and the WMT14 training data (38 million pairs). We use SentencePiece model (Kudo, 2018) for subword segmentation with a source-target shared vocabulary of 32,000 subword units. We evaluate our method with three "domains": two specific domains, news and TED subtitles, and out-of-domain. News domain uses the WMT14 news
testset (N14) for testing, and WMT12-13 for validation in early stopping (Prechelt, 1997). The TED domain uses the IWSLT15 testset (T15) for testing, and the IWSLT14 testset for validation. Out-of-domain performance is measured by two additional testsets, patent testset (PA) (2000 sentences) $^{10}$  and WMT15 news discussion testset (D15). We report SacreBLEU $^{11}$  (Post, 2018).