SlowGuess/ACL_2020_OCR huggingface dataset:
Xuebo Liu $^{1*}$  Houtim Lai $^{2*}$  Derek F. Wong $^{1\dagger}$  Lidia S. Chao $^{1}$
$^{1}$ NLP $^{2}$ CT Lab, Department of Computer and Information Science, University of Macau
$^{2}$ NewTranx Information Technology, Shenzhen, China
nlp2ct.xuebo@gmail.com, haotian.li@newtranx.com,
{derekfw,lidiasi}@um.edu.mo
A neural machine translation (NMT) system is expensive to train, especially with high-resource settings. As the NMT architectures become deeper and wider, this issue gets worse and worse. In this paper, we aim to improve the efficiency of training an NMT by introducing a novel norm-based curriculum learning method. We use the norm (aka length or module) of a word embedding as a measure of 1) the difficulty of the sentence, 2) the competence of the model, and 3) the weight of the sentence. The norm-based sentence difficulty takes the advantages of both linguistically motivated and model-based sentence difficulties. It is easy to determine and contains learning-dependent features. The norm-based model competence makes NMT learn the curriculum in a fully automated way, while the norm-based sentence weight further enhances the learning of the vector representation of the NMT. Experimental results for the WMT'14 English-German and WMT'17 Chinese-English translation tasks demonstrate that the proposed method outperforms strong baselines in terms of BLEU score  $(+1.17 / + 1.56)$  and training speedup  $(2.22\mathrm{x} / 3.33\mathrm{x})$
The past several years have witnessed the rapid development of neural machine translation (NMT) based on an encoder-decoder framework to translate natural languages (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015). Since NMT benefits from a massive amount of training data and works in a cross-lingual setting, it becomes much hungrier for training time than other natural language processing (NLP) tasks.
Based on self-attention networks (Parikh et al., 2016; Lin et al., 2017), Transformer (Vaswani et al., 2017) has become the most widely used architecture for NMT. Recent studies on improving Transformer, e.g. deep models equipped with up to 30-layer encoders (Bapna et al., 2018; Wu et al., 2019; Wang et al., 2019; Zhang et al., 2019a), and scaling NMTs which use a huge batch size to train with 128 GPUs (Ott et al., 2018; Edunov et al., 2018), face a challenge to the efficiency of their training. Curriculum learning (CL), which aims to train machine learning models better and faster (Bengio et al., 2009), is gaining an intuitive appeal to both academic and industrial NMT systems.
The basic idea of CL is to train a model using examples ranging from "easy" to "difficult" in different learning stages, and thus the criterion of difficulty is vital to the selection of examples. Zhang et al. (2018) summarize two kinds of difficulty criteria in CL for NMT: 1) linguistically motivated sentence difficulty, e.g. sentence length, word frequency, and the number of coordinating conjunctions, which is easier to obtain (Kocmi and Bojar, 2017; Platanios et al., 2019); 2) model-based sentence difficulty, e.g. sentence uncertainties derived from independent language models or the models trained in previous time steps or epochs, which tends to be intuitively effective but costly (Zhang et al., 2017; Kumar et al., 2019; Zhang et al., 2019b; Zhou et al., 2020).
In this paper, we propose a novel norm-based criterion for the difficulty of a sentence, which takes advantage of both model-based and linguistically motivated difficulty features. We observe that the norms of the word vectors trained on simple neural networks are expressive enough to model the two features, which are easy to obtain while possessing learning-dependent features. For example, most of the frequent words and context-insensitive rare words will have vectors with small norms.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 427-436 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Batch	Len.	Source sentence
Vanilla
B1	16	In catalogues, magazines ...
27	Nevertheless, it is an ...
B2	38	The company ROBERT ...
37	Ottmar Hitzfeld played ...
The Proposed Method
B1*	3	Second Part.
4	It was not.
B2*	5	Thank you very much.
4	We know that.
Table 1: Training batches on the WMT'14 English-German translation task. "Len." denotes the length of the sentence. The proposed method provides a much easier curriculum at the beginning of the training of the model.
Unlike existing CL methods for NMT, relying on a hand-crafted curriculum arrangement (Zhang et al., 2018) or a task-dependent hyperparameter (Platanios et al., 2019), the proposed norm-based model competence enables the model to arrange the curriculum itself according to its ability, which is beneficial to practical NMT systems. We also introduce a novel paradigm to assign levels of difficulty to sentences, as sentence weights, into the objective function for better arrangements of the curricula, enhancing both existing CL systems and the proposed method.
Empirical results for the two widely-used benchmarks show that the proposed method provides a significant performance boost over strong baselines, while also significantly speeding up the training. The proposed method requires slightly changing the data sampling pipeline and the objective function without modifying the overall architecture of NMT, thus no extra parameters are employed.
NMT uses a single large neural network to construct a translation model that translates a source sentence  $\mathbf{x}$  into a target sentence  $\mathbf{y}$ . During training, given a parallel corpus  $\mathcal{D} = \{\langle \mathbf{x}^n,\mathbf{y}^n\rangle \}_{n = 1}^N$ , NMT aims to maximize its log-likelihood:
$$
\begin{array}{l} \hat {\boldsymbol {\theta}} = L (\mathcal {D}; \boldsymbol {\theta} _ {0}) \\ = \arg \max  _ {\boldsymbol {\theta} _ {0}} \sum_ {n = 1} ^ {N} \log P \left(\mathbf {y} ^ {n} \mid \mathbf {x} ^ {n}; \boldsymbol {\theta} _ {0}\right) \tag {1} \\ \end{array}
$$
where  $\theta_0$  are the parameters to be optimized during the training of the NMT models. Due to the intractability of  $N$ , the training of NMT employs mini-batch gradient descent rather than batch gradient descent or stochastic gradient descent, as follows:
$$
\mathcal {B} _ {1}, \dots , \mathcal {B} _ {t}, \dots , \mathcal {B} _ {T} = \operatorname {s a m p l e} (\mathcal {D}) \tag {2}
$$
$$
\hat {\boldsymbol {\theta}} = L \left(\mathcal {B} _ {T}; L \left(\mathcal {B} _ {T - 1}; \dots L \left(\mathcal {B} _ {1}, \boldsymbol {\theta} _ {0}\right)\right)\right) \tag {3}
$$
where  $T$  denotes the number of training steps and  $\mathcal{B}_t$  denotes the  $t$ th training batch. In the training of the  $t$ th mini-batch, NMT optimizes the parameters  $\theta_{t-1}$  updated by the previous mini-batch.
CL supposes that if mini-batches are bucketed in a particular way (e.g. with examples from easy to difficult), this would boost the performance of NMT and speed up the training process as well. That is, upgrading the sample  $(\cdot)$  to
$$
\mathcal {B} _ {1} ^ {*}, \dots , \mathcal {B} _ {t} ^ {*}, \dots , \mathcal {B} _ {T} ^ {*} = \operatorname {s a m p l e} ^ {*} (\mathcal {D}) \tag {4}
$$
where the order from easy to difficult (i.e.  $\mathcal{B}_1^* \to \mathcal{B}_T^*$ ) can be: 1) sentences with lengths from short to long; 2) sentences with words whose frequency goes from high to low (i.e. word rarity); and 3) uncertainty of sentences (from low to high uncertainties) measured by models trained in previous epochs or pre-trained language models. Table 1 shows the sentences of the training curricula provided by vanilla Transformer and the proposed method.
3 Norm-based Curriculum Learning
3.1 Norm-based Sentence Difficulty
Most NLP systems have been taking advantage of distributed word embeddings to capture the syntactic and semantic features of a word (Turian et al., 2010; Mikolov et al., 2013). A word embedding (vector) can be divided into two parts: the norm and the direction:
$$
\boldsymbol {w} = \underbrace {\left| \left| \boldsymbol {w} \right| \right|} _ {\text {n o r m}} \cdot \underbrace {\frac {\boldsymbol {w}}{\left| \left| \boldsymbol {w} \right| \right|}} _ {\text {d i r e c t i o n}} \tag {5}
$$
In practice, the word embedding, represented by  $\mathbf{w}$ , is the key component of a neural model (Liu et al., 2019a,b), and the direction  $\frac{\mathbf{w}}{||\mathbf{w}||}$  can also be used to carry out simple word/sentence similarity and relation tasks. However, the norm  $||\mathbf{w}||$  is rarely considered and explored in the computation.
Figure 1: Word vector norm of the word embedding model trained on the WMT'14 English-German (source side) training data. The  $x$ -axis is the word frequency, ranked in descending order. Rare words and significant words have higher norms.
Surprisingly, the norm which is simply derived from a single model parameter, can also capture delicate features during the optimization of a model. Schakel and Wilson (2015) observe that in the word embedding model (Mikolov et al., 2013), the word vector norm increases with a decrease of the word frequency, while polysemous words, such as "May", tend to have an average norm weighted over its various contexts. Wilson and Schakel (2015) further conduct controlled experiments on word vector norm and find that besides the word frequency, the diversities of the context of the word are also a core factor to determine its norm. The vector of a context-insensitive word is assigned a higher norm. In other words, if a word is usually found in specific contexts, it should be regarded as a significant word (Luhn, 1958). The word embedding model can exactly assign these significant words higher norms, even if some of them are frequent. The sentences consisting of significant words share fewer commonalities with other sentences, and thus they can also be regarded as difficult-to-learn examples.
Figure 1 shows the relationship between the word vector norm and the word frequency in the English data of the WMT'14 English-German translation task. The results stay consistent with prior works (Wilson and Schakel, 2015), showing that the rare words and significant words obtain a high norm from the word embedding model. Motivated by these works and our preliminary experimental results, we propose to use the word vector norm as a criterion to determine the difficulty
of a sentence. Specifically, we first train a simple word embedding model on the training corpus, and then obtain an embedding matrix  $\mathbf{E}^{\mathrm{w2v}}$ . Given a source sentence  $\mathbf{x} = x_{1},\dots ,x_{i},\dots ,x_{I}$ , it can be mapped into distributed representations  $x_{1},\dots ,x_{i},\dots ,x_{I}$  through  $\mathbf{E}^{\mathrm{w2v}}$ . The norm-based sentence difficulty is calculated as
$$
d (\mathbf {x}) = \sum_ {i = 1} ^ {I} | | \boldsymbol {x} _ {i} | | \tag {6}
$$
Long sentences and sentences consisting of rare words or significant words tend to have a high sentence difficulty for CL.
The proposed norm-based difficulty criterion has the following advantages: 1) It is easy to compute since the training of a simple word embedding model just need a little time and CPU resources; 2) Linguistically motivated features, such as word frequency and sentence length, can be effectively modeled; 3) Model-based features, such as learning-dependent word significance, can also be efficiently captured.
3.2 Norm-based Model Competence
Besides finding an optimal sentence difficulty criterion, arranging the curriculum in a reasonable order is equally important. As summarized by Zhang et al. (2019b), there are two kinds of CL strategies: deterministic and probabilistic. From their observations, probabilistic strategies are superior to deterministic ones in the field of NMT, benefiting from the randomization during mini-batch training.
Without loss of generality, we evaluate our proposed norm-based sentence difficulty with a typical probabilistic CL framework, that is, competence-based CL (Platanios et al., 2019). In this framework, a notion of model competence is defined which is a function that takes the training step  $t$  as input and outputs a competence value from 0 to  $1$ :
$$
c (t) \in (0, 1 ] = \min (1, \sqrt {t \frac {1 - c _ {0} ^ {2}}{\lambda_ {t}} + c _ {0} ^ {2}}) \quad (7)
$$
where  $c_{0} = 0.01$  is the initial competence at the beginning of training and  $\lambda_{t}$  is a hyperparameter determining the length of the curriculum. For the sentence difficulty, they use cumulative density function (CDF) to transfer the distribution of sentence difficulties into (0, 1]:
$$
\hat {d} (\mathbf {x} ^ {n}) \in (0, 1 ] = \operatorname {C D F} \left(\left\{d \left(\mathbf {x} ^ {n}\right) \right\} _ {n = 1} ^ {N}\right) ^ {n} \tag {8}
$$
<sup>1</sup>We introduce the square root competence model since it has the best performance in Platanios et al. (2019).
Figure 2: Norm of NMT source embedding and BLEU score of a vanilla Transformer on the WMT'14 English-German translation task. The BLEU scores are calculated on the development set. Both the norm and BLEU score grow rapidly until the 30K training step.
The score of difficult sentences tends to be 1, while that of easy sentences tends to be 0. The model uniformly samples curricula whose difficulty is lower than the model competence at each training step, thus making the model learn the curriculum in a probabilistic way.
One limitation of competence-based CL is that the hyperparameter  $\lambda_{t}$  is task-dependent. In detail, for each system, it needs to first train a vanilla baseline model and then use the step reaching  $90\%$  of its final performance (BLEU score) as the value of the length hyperparameter. As we know, training an NMT baseline is costly, and arbitrarily initializing the value might lead to an unstable training process.
To alleviate this limitation and enable NMT to learn curricula automatically without human interference in setting the hyperparameter, it is necessary to find a way for the model to determine the length of a curriculum by itself, according to its competence, which should be independent of the specific task.
To this aim, we further introduce a norm-based model competence criterion. Different from the norm-based difficulty using the word vector norm, the norm-based model competence uses the norm of the source embedding of the NMT model  $\mathbf{E}^{\mathrm{nmt}}$ :
$$
m _ {t} = | | \mathbf {E} _ {t} ^ {\mathrm {n m t}} | | \tag {9}
$$
where  $m_{t}$  denotes the norm of  $\mathbf{E}^{\mathrm{nmt}}$  at the tth training step, and we write  $m_0$  for the initial value of the norm of  $\mathbf{E}^{\mathrm{nmt}}$ . This proposal is moti
vated by the empirical results shown in Figure 2, where we show the BLEU scores and the norms of the source embedding matrix at each checkpoint of a vanilla Transformer model on the WMT'14 English-German translation task. We found the trend of the growth of the norm  $m_{t}$  to be very similar to that of the BLEU scores. When  $m_{t}$  stays between 15K to 20K, which is about from twice to three times larger than the initial norm  $m_{0}$ , both the growth of the norm and that of the BLEU score have slowed down. It shows strong clues that  $m_{t}$  is a functional metric to evaluate the competence of the model, and thus we can avoid the intractability of  $\lambda_{t}$  in Equation 7:
$$
\hat {c} (t) = \min  \left(1, \sqrt {\left(m _ {t} - m _ {0}\right) \frac {1 - c _ {0} ^ {2}}{\lambda_ {m} m _ {0}} + c _ {0} ^ {2}}\right) \tag {10}
$$
where  $\lambda_{m}$  is a task-independent hyperparameter to control the length of the curriculum. With this criterion, the models can, by themselves, fully automatically design a curriculum based on the feature (norm). At the beginning of the training, there is a lower  $m_{t}$ , so the models tend to learn with an easy curriculum. But with an increase of the norm  $m_{t}$ , more difficult curricula will be continually added into the learning.
3.3 Norm-based Sentence Weight
In competence-based CL, the model uniformly samples sentences whose difficulty level is under the model competence, and then learns with the samples equally. As a result, those simple sentences with low difficulty (e.g.  $\hat{d}(\mathbf{x}) < 0.1$ ) are likely to be repeatedly used in the model learning. This is somewhat counterintuitive and a waste of computational resources. For example, when students are able to learn linear algebra, they no longer need to review simple addition and subtraction, but can keep the competence during the learning of hard courses. On the other hand, a difficult (long) sentence is usually made up of several easy (short) sentences. Thus, the representations of easy sentences can also benefit from the learning of difficult sentences.
To alleviate this limitation of competence-based CL and further enhance the learning from the curriculum of different levels of difficulty, we propose a simple yet effective norm-based sentence weight:
$$
w (\mathbf {x}, t) = \left(\frac {\hat {d} (\mathbf {x})}{\hat {c} (t)}\right) ^ {\lambda_ {w}} \tag {11}
$$
where  $\lambda_w$  is the scaling hyperparameter smoothing the weight,  $\hat{d}(\mathbf{x})$  is the norm-based sentence difficulty, and  $\hat{c}(t)$  is the model competence. For each training step  $t$ , or each model competence  $\hat{c}(t)$ , the weight of a training example  $w(\mathbf{x}, t)$  is included in its objective function:
$$
l (\langle \mathbf {x}, \mathbf {y} \rangle , t) = - \log P (\mathbf {y} | \mathbf {x}) w (\mathbf {x}, t) \tag {12}
$$
where  $l(\langle \mathbf{x},\mathbf{y}\rangle ,t)$  is the training loss of an example  $\langle \mathbf{x},\mathbf{y}\rangle$  at the  $t$ th training step. With the use of sentence weights, the models, at each training step, tend to learn more from those curricula whose difficulty is close to the current model competence. Moreover, the models still benefit from the randomization of the mini-batches since the length weight does not change the curriculum sampling pipeline.
3.4 Overall Learning Strategy
Algorithm 1 illustrates the overall training flow of the proposed method. Besides the component and training flow of vanilla NMT models, only some low-cost operations, such as matrix multiplication, have been included in the data sampling and objective function, allowing an easy implementation as a practical NMT system. We have also found, empirically, that the training speed of each step is not influenced by the introduction of the proposed method.
4.1 Data and Setup
We conducted experiments on the widely used benchmarks, i.e. the medium-scale WMT'14 English-German (En-De) and the large-scale WMT'17 Chinese-English (Zh-En) translation tasks. For En-De, the training set consists of 4.5M sentence pairs with 107M English words and 113M German words. The development is newest13
and the test set is newstest14. For the Zh-En, the training set contains roughly 20M sentence pairs. The development is newsdev2017 and the test set is newstest2017. The Chinese data were segmented by jieba, $^{2}$  while the others were tokenized by the tokenizer.perl script from Moses. $^{3}$  We filtered the sentence pairs with a source or target length over 200 tokens. Rare words in each data set were split into sub-word units (Sennrich et al., 2016). The BPE models were trained on each language separately with 32K merge operations.
All of the compared and implemented systems are the base Transformer (Vaswani et al., 2017) using the open-source toolkit Marian (Junczys-Dowmunt et al., 2018). We tie the target input embedding and target output embedding (Press and Wolf, 2017). The Adam (Kingma and Ba, 2015) optimizer has been used to update the model parameters with hyperparameters  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$ ,  $\varepsilon = 10^{-9}$ . We use the variable learning rate proposed by Vaswani et al. (2017) with 16K warm up steps and a peak learning rate 0.0003.
We employed FastText (Bojanowski et al., 2017) with its default settings to train the word embedding model for calculating the norm-based sentence difficulty; an example is given in Figure 1. The hyperparameters  $\lambda_{m}$  and  $\lambda_{w}$  controlling the norm-based model competence and norm-based sentence weight were tuned on the development set of En-De, with the value of 2.5 and 0.5, respectively. To test the adaptability of these two hyperparameters, we use them directly for the Zh-En translation task without any tuning. We compare the proposed methods with the re-implemented
$^{2}$ https://github.com/fxsjy/jieba
<sup>3</sup>http://www.statmt.org/moses/
4https://marian-nmt.github.io/
$^{5}$ https://github.com/facebookresearch/fastText
ID	Model	Dev.	Test	Updates	Speedup
Existing Baselines
1	GNMT (Wu et al., 2016)	-	24.61	-	-
2	ConvS2S (Gehring et al., 2017)	-	25.16	-	-
3	Base Transformer (Vaswani et al., 2017)	25.80	27.30	-	-
4	Big Transformer (Vaswani et al., 2017)	26.40	28.40	-	-
Our Implemented Baselines
5	Base Transformer (Vaswani et al., 2017)	25.90	27.64	100.0K	1.00x
6	5 + Competence-based CL (Platanios et al., 2019)	26.39	28.19	60.0K	1.67x
Our Proposed Method (Individual)
7	6 + Norm-based Model Competence	26.59	28.51	50.0K	2.00x
8	6 + Norm-based Sentence Complexity	26.61	28.61	50.0K	2.00x
9	6 + Norm-based Sentence Weight	26.63	28.32	52.5K	1.90x
Our Proposed Method (All)
10	5 + Norm-based CL	26.89	28.81	45.0K	2.22x
Table 2: Results on the WMT'14 English-German translation task. Dev. is the newtest2013 while Test is newtest2014. 'Updates' means the step of each model reaching the best performance of model (5)  $(\mathrm{K} = \text{thousand})$  while 'Speedup' means its corresponding speedup.
λm	Dev.	λw	Dev.
1.0	26.63	0	26.71
2.0	26.72	1/3	26.80
2.5	26.89	1/2	26.89
3.0	26.65	1	26.78
4.0	26.62	2	26.77
Table 3: Effects of different  $\lambda_{m}$  of the norm-based model competence function and  $\lambda_{w}$  of the norm-based sentence weight function.
competence-based CL (Platanios et al., 2019).<sup>6</sup>
During training, the mini-batch contains nearly 32K source tokens and 32K target tokens. We evaluated the models every 2.5K steps, and chose the best performing model for decoding. The maximum training step was set to 100K for En-De and 150K for Zh-En. During testing, we tuned the beam size and length penalty (Wu et al., 2016) on the development data, using a beam size of 6 and a length penalty of 0.6 for En-De, and a beam size of 12 and a length penalty of 1.0 for Zh-En. We report the 4-gram BLEU (Papineni et al., 2002) score given by the multi-bleu.perl script. The codes and scripts of the proposed norm-based CL and our re-implemented competence-based CL are freely available at https://github.com/NLP2CT/norm-nmt.
Table 2 shows the results of the En-De translation task in terms of BLEU scores and training speedup. Models (1) to (4) are the existing baselines of this translation benchmark. Model (5) is our implemented base Transformer with 100K training steps, obtaining 27.64 BLEU scores on the test set. By applying the competence-based CL with its proposed sentence rarity and square root competence function, i.e. model (6), it reaches the performance of model (5) using 60K training steps and also gets a better BLEU score.
For the proposed method, we first show the performance of each sub-module, that is: model (7), which uses the norm-based model competence instead of the square root competence of model (6); model (8), which uses the proposed norm-based sentence complexity instead of the sentence rarity of model (6); and model (9), which adds the norm-based sentence weight to model (6). The results show that after applying each sub-module individually, both the BLEU scores and the learning efficiency are further enhanced.
Model (10) shows the results combining the three proposed norm-based methods for CL, i.e. the norm-based sentence difficulty, model competence, and sentence weight. We call the combination of the proposed method norm-based CL. It shows its superiority in the BLEU score, which has an increase of 1.17 BLEU scores compared to the Trans
<sup>6</sup>We use its best settings, i.e. the rarity-based sentence difficulty and the square root competence function.