SlowGuess/ACL_2020_OCR huggingface dataset:
Yikai Zhou† Baosong Yang‡ Derek F. Wong†* Yu Wan† Lidia S. Chao†  
†NLP²CT Lab, Department of Computer and Information Science, University of Macau nlp2ct.{yzhou,ywan}@gmail.com, {derekfw,lidiasiç}@um.edu.mo ‡Alibaba DAMO Academy yangbaosong.ybs@alibaba-inc.com
Neural machine translation (NMT) has proven to be facilitated by curriculum learning which presents examples in an easy-to-hard order at different training stages. The keys lie in the assessment of data difficulty and model competence. We propose uncertainty-aware curriculum learning, which is motivated by the intuition that: 1) the higher the uncertainty in a translation pair, the more complex and rarer the information it contains; and 2) the end of the decline in model uncertainty indicates the completeness of current training stage. Specifically, we serve cross-entropy of an example as its data difficulty and exploit the variance of distributions over the weights of the network to present the model uncertainty. Extensive experiments on various translation tasks reveal that our approach outperforms the strong baseline and related methods on both translation quality and convergence speed. Quantitative analyses reveal that the proposed strategy offers NMT the ability to automatically govern its learning schedule.
Neural machine translation (NMT) has advanced the state-of-the-art on various translation tasks (Hassan et al., 2018; Chen et al., 2018). A well-performed NMT is trained using an end-to-end framework (Sutskever et al., 2014) that profits from large-scale training corpus and various optimization tricks (Ott et al., 2018; Xu et al., 2019; Li et al., 2020). These techniques boost the translation quality, in the meanwhile, leading to massive hyper-parameters to be tuned and expensive development costs (Popel and Bojar, 2018). Recent studies (Zhang et al., 2018, 2019; Platanios et al., 2019; Liu et al., 2020) have proven that feeding training examples in a meaningful order rather than considering them randomly can accelerate the model
Figure 1: The change of confidence in an area during the learning. Humans (red) experience the process of overconfidence  $\Rightarrow$  despair  $\Rightarrow$ enlightenment (Dunning-Kruger Curve), while prior work that exploits CL in NMT assumes a monotonically increased curve (green, Platanios et al., 2019). Interestingly, our model automatically draws a similar tendency as humans (blue).
convergence thus reducing the computational cost. Such methods refer to curriculum learning (CL, Bengio et al., 2009), in which a model is taught as a human from simple concepts to complex ones.
There exists two open problems in the integration of CL with NMT, i.e. the assessment of data difficulty and the programme of learning schedule. Considering the former, prior studies (Kocmi and Bojar, 2017; Platanios et al., 2019) intuitively treat human linguistic knowledge, e.g. either sentence length or word rarity, as the measure of difficulty. Nevertheless, each linguistic feature merely considers an aspect of sentences which fails to fully cope with the data difficulty for a model (Jiang et al., 2015). For the latter, existing methods pre-define the duration of curriculum based on an assumption that the model confidence monotonically increases with the training (Zhang et al., 2018, 2019). We argue that this assumption does not conform to human behavior, i.e. Dunning-Kruger Curve (Figure 1, Kruger and Dunning, 1999), and limits the adaptability and flexibility of curriculum learning.
In response to these problems, we propose to strengthen CL for NMT through determining the
6934
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6934-6944 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
data difficulty and scheduling the curriculum according to model ability rather than human intuitions. We introduce a novel uncertainty-aware curriculum learning framework, which serves uncertainty as its principle to order the input examples and control the duration of each training stage. Specifically, we measure the data uncertainty of a sentence pair according to its joint distribution that is estimated by a language model pre-trained on the training corpus. The intuition behind is that the higher the cross-entropy and uncertainty have in an example, the harder it is to learn and translate (Brown et al., 1990). Besides, we calculate the model uncertainty using the variance of the distribution over the network presented by Bayesian neural networks (Buntine and Weigend, 1991). Accordingly, the model uncertainty reflects whether our model can best describe the data distribution (Xiao and Wang, 2019), and the stop of its decline indicates the completeness of the current training stage.
One principle in our work is to maintain the simplicity and efficiency in CL. Several researchers may doubt that the use of Bayesian inference over the training corpus may significantly raise the computational cost. To this end, we apply Monte Carlo Dropout (Gal and Ghahramani, 2016) to approximate Bayesian inference. Besides, we categorize examples into subsets according to their difficulty, which is then be progressively added into the training set at different training stages, namely baby step (Cirik et al., 2016). The model uncertainty can be calculated after each epoch using the samples randomly selected from the current training set, thus avoiding affect training efficiency.
We evaluate the effectiveness of our methods on WMT16 English-to-German, IWSLT15 English-to-Vietnamese, and WMT17 Chinese-to-English translation tasks. The experimental results demonstrate that the proposed model consistently improves translation performance over the strong TRANSFORMER (Vaswani et al., 2017) baseline and related methods that exploit CL into NMT. Extensive analyses confirm that: 1) our approach significantly speeds up the model convergence; 2) using data uncertainty to present the translation difficulty surpasses its sentence length and word rarity counterparts, and this superiority can be further expanded by exploiting a language model that is trained on large-scale external data, i.e. BERT (Devlin et al., 2019); 3) the model uncertainty performs a self-adaptive manner to assess the model competence
regardless the pre-defined patterns.
NMT uses a single, large neural network to build translation model, aiming to maximize the conditional distribution of sentence pairs using parallel corpus (Sutskever et al., 2014; Bahdanau et al., 2015; Yang et al., 2019; Wan et al., 2020). Formally, the learning objective is to minimize the following loss function over the training corpus  $\mathcal{D} = \{(x^n,y^n)\}_{n = 1}^N$ , with the size being  $N$ :
$$
\mathcal {L} = \mathbb {E} _ {\left(x ^ {n}, y ^ {n}\right) \sim \mathcal {D}} [ - \log \mathbf {P} \left(y ^ {n} \mid x ^ {n}; \theta\right) ] \tag {1}
$$
where  $x^n$  and  $y^n$  indicate the source and target sides of the  $n$ -th example in training data.  $\theta$  denotes the trainable parameters of NMT model. During the training, the examples randomly feed to vanilla model, regardless of their order, making the development of a well-performed NMT system time-consuming (Sennrich et al., 2016a; Popel and Bojar, 2018; Yang et al., 2020). An alternative way to speed up the training process and boost the performance of a neural network is to exploit CL (Elman, 1993; Krueger and Dayan, 2009; Bengio et al., 2009).
Related Work on Exploring CL Several studies have shown the effectiveness of CL in the field of computer vision (Sarafianos et al., 2017; Wang et al., 2019c; Guo et al., 2018), as well as a range of NLP tasks, including math word problem (Zaremba and Sutskever, 2014), sentiment analysis (Cirik et al., 2016), and natural answer generation (Liu et al., 2018). They point out that CL can solve the problem in some tasks that is hard to train through presenting training data in an easy-to-hard order.
Kocmi and Bojar (2017) first apply CL into NMT and suggest two sticking points, i.e. data difficulty and learning schedule. Partially inspired by their findings, Thompson et al. (2018), Zhang et al. (2019), Wang et al. (2019b) and Kumar et al. succeed on handling the problem in domain translation. Concerning the general translation tasks, Zhang et al. (2018) investigate a variety of difficulty criteria based on human intuition, e.g. sentence length and word rarity, which show distinct performance across language pairs and model settings. While Platanios et al. (2019) pay attention to the schedule that determines the duration of each curriculum. They introduce monotonically increased curves, e.g. either linear or square root, to represent the
6935
changes of the model ability across the training process. These early successes presuppose the limited heuristic knowledge on both the data difficulty and the tendency of model competence.
Motivation As mentioned above, one of the main challenges in CL is the identification of easy and hard samples which is onerous and conceptually difficult in translation community. For example, neither the sentence length or word rarity can fully express the complexity of a translation. Another problem in CL is the programme of learning schedule, in which the patterns pre-defined by humans lack in adaptability and lead to massive additional hyper-parameters that have to be tuned. Even if these artificial supervisions are feasible, what is intuitively "easy" and "competent" for a human may not match that for neural networks (Kumar et al., 2010; Jiang et al., 2015).
To this end, we approach these problems from the model perspective. In this section, we first introduce data uncertainty to quantify the translation difficulty for each training example (Section 3.1). Then, we propose to predict the model uncertainty at the training time which is a self-adaptive manner to govern curriculum by the model itself (Section 3.2). Finally, we describe how to exploit the proposed two factors in NMT training (Section 3.3). The proposed framework is illustrated in Figure 2.
3.1 Data Uncertainty
In order to estimate the data uncertainty, we propose to pre-train a language model (LM) over the monolingual sentences from the parallel training corpus  $\mathcal{D}$  to account the cross-entropy of each sentence. The intuition behind this is that the higher cross-entropy and perplexity represents an uncertain sentence, since it is hard to be generated and determined by the LM (Brown et al., 1990). This provides an explainable and comprehensive way to evaluate the difficulty of an example. Accordingly, we assign several types of data uncertainty, which can be used individually or combined together:
Source Difficulty The difficulty of a source sentence affects the language understanding of NMT model. Inspired by Zhang et al. (2018) and Platanios et al. (2019), an interpretable way is to use the source difficulty to approximate the complexity of a sentence pair. Given the source sentence  $x^n$ , we can calculate the source uncertainty  $u^{data}(x^n)$  by
Figure 2: Illustration of the proposed uncertainty-aware curriculum learning framework. We categorize training corpus into baby steps according to their data uncertainty. The sign of entering the next curriculum is the stop of decline in model uncertainty which is estimated over random samples in the current training stage.
estimating its perplexity, namely:
$$
u ^ {d a t a} (x ^ {n}) = - \frac {1}{I} \sum_ {n = 1} ^ {N} \log \mathbf {P} \left(x _ {i} ^ {n} \mid x _ {<   i} ^ {n}\right) \tag {2}
$$
where  $I$  indicates the length of source sentence.
Target Difficulty Since the complex and rare target sentence directly makes NMT have a harder time in generating the sentence (Kocmi and Bojar, 2017), another natural choice is to apply the target uncertainty to present the data difficulty. Analogous to the source side, the target uncertainty  $u^{data}(y^n)$  is:
$$
u ^ {d a t a} (y ^ {n}) = - \frac {1}{J} \sum_ {j = 1} ^ {J} \log \mathbf {P} \left(y _ {i} ^ {n} \mid y _ {<   i} ^ {n}\right) \tag {3}
$$
where  $J$  denotes the length of target sentence  $y^n$ .
Joint Difficulty Intuitively, the complexity of a translation pair should be contributed by two sides, thus reflecting the difficulty of both understanding and generating processes in NMT. We can combine the concepts of source and target uncertainty:
$$
u ^ {d a t a} (x ^ {n}, y ^ {n}) = u ^ {d a t a} (x ^ {n}) + u ^ {d a t a} (y ^ {n}) \tag {4}
$$
To our best knowledge, due to the lack of interpretability on scoring the joint difficulty in a sentence pair, all the existing methods that exploit CL into NMT merely measure data difficulty on either source or target. Our method provides an alternative way to tackle this problem with the concept of joint probability distribution. We expect the joint uncertainty to further improve the performance.
6936
In this paper, we examine three widely used LMs to appraise the data uncertainty: a statistical  $n$ -gram LM - KENLM (Heafield, 2011), a neural LM - RNNLM (Mikolov et al., 2010), and a multilingual neural LM that trained on billions of external sentences - BERT (Devlin et al., 2019). Note that, the modeling of data uncertainty is not limited to our approach. It can be also quantified by other manners, e.g. estimating the data likelihood with Monte Carlo approximation (Der Kiureghian and Ditlevsen, 2009) or validating the translation distribution using a well-trained NMT model (Zhang et al., 2018). In contrast to these time-consuming techniques, LM marginally increases the computational cost and easy to be applied, conforming to the original motivation of CL.
3.2 Model Uncertainty
Moreover, we propose to regulate the duration of each curriculum by quantifying the model uncertainty rather than presetting before the training. Model uncertainty, which is also known as epistemic uncertainty (Kendall and Gal, 2017), can be used to measure whether the model parameters are able to best describe the data distribution (Dong et al., 2018; Xiao and Wang, 2019). In our work, a small score of model uncertainty indicates the model is confident that the current training data has been well learned (Wang et al., 2019a), and the termination of the decline in scores represents the signal to shift to the next curriculum stage.
The model uncertainty can be quantified by Bayesian neural networks (Buntine and Weigend, 1991; Neal, 1996), which place a probabilistic distribution over the model parameters on constant input and output data, and serve its variance as the uncertainty. For reasons of computational efficiency, we adopt widely used Monte Carlo Dropout (Gal and Ghahramani, 2016) to approximate Bayesian inference. Given a dataset used to examine the model uncertainty  $\mathcal{D}^U = \{(x^m,y^m)\}_{m = 1}^M$  which consists of  $M$  sentence pairs, we perform  $K$  passes of forward propagation through the NMT model. In each pass, part of neurons in network  $\theta$  are randomly deactivated. Eventually, we yield  $K$  samples on model parameters  $\{\hat{\theta}^1,\dots ,\hat{\theta}^K\}$  and corresponding translation probabilities. The model
uncertainty on  $\mathcal{D}^U$  can be formally expressed as:
$$
u ^ {m o d} (\theta) = \frac {1}{M} \sum_ {m = 1} ^ {M} \mathbf {V a r} \left[ \mathbf {P} \left(y ^ {m} \mid x ^ {m}, \hat {\theta} ^ {k}\right) \right] _ {k = 1} ^ {K} \tag {5}
$$
Here,  $\mathbf{Var}[\cdot ]$  denotes the variance of a distribution which calculated following the common setting in Dong et al. (2018) and Xiao and Wang (2019). In this way, the model is offered the ability to determine its model competence by itself.
3.3 Self-Adaptive Training Strategy
In this work, we adopt a widely used CL strategy called baby step (Cirik et al., 2016; Zhang et al., 2018) to arrange training data and organize the training process. Specifically, the whole training set  $\mathcal{D}$  is divided into different buckets, i.e. steps  $\{\mathcal{D}_1,\dots ,\mathcal{D}_T\}$ , in which those examples with similar data uncertainty scores  $u^{data}$  are categorized into the same bucket. The training starts from the step that consists of examples with the lowest uncertainty. After that, data in the next step is aggregated to the current training dataset  $\mathcal{C}$  when the model uncertainty ceases its reduction. Following existing studies (Platanios et al., 2019; Kocmi and Bojar, 2017) that the model should be trained from easy samples to hard ones, we schedule the curriculum with the order of increasing uncertainty. To avoid overfitting and useless training, partially inspired by early stopping, we treat the third time when current model uncertainty is higher than the score
2Our preliminary experiments show that the model with a reverse order does not gain any performance improvement to the baseline model.
${}^{1}K$  is empirically set to 10 in our work.
6937
evaluated last time as the sign that the model is at the level of "expert" for the current curriculum. The hyperparameter of stopping criterion is important. A small value makes the training to easily enter the next baby step, and the current baby step fails to be fully trained, while a large value reduces training efficiency and cause over-fitting.
Considering that performing Monte Carlo Dropout over the NMT model on all the examples in  $\mathcal{C}$  is time-consuming, while the superiority of CL lies in its ability to accelerate the model convergence. In order to maintain this advantage, we propose to estimate the model uncertainty after each epoch rather than every model updating steps. Furthermore, we randomly extract  $M = 1\mathrm{k}$  samples from current training dataset  $\mathcal{C}$  as  $\mathcal{D}^U$ . Then, the evaluation of model uncertainty is conducted on  $\mathcal{D}^U$  to mirror the confidence over the current curriculum. Therefore, our approach reserves the efficiency in CL, in the meanwhile, guiding the duration of each curriculum in a self-adaptive fashion. The overall procedure is described in Algorithm 1.
We examine our method upon advanced TRANSFORMER (Vaswani et al., 2017) and conduct experiments on widely used translation tasks: IWSLT15 English-to-Vietnamese  $(\mathrm{En} \Rightarrow \mathrm{Vi})$ , WMT16 English-to-German  $(\mathrm{En} \Rightarrow \mathrm{De})$  and WMT17 Chinese-to-English  $(\mathrm{Zh} \Rightarrow \mathrm{En})$ .<sup>3</sup>
Dataset To compare with the results reported by previous work (Platanios et al., 2019), we evaluate our methods on IWSLT15  $\mathrm{En} \Rightarrow \mathrm{Vi}$  and WMT16  $\mathrm{En} \Rightarrow \mathrm{De}$  translation tasks. Our models are trained using all of the available parallel corpora from the IWSLT15 and WMT16 datasets, consisting of  $133\mathrm{k}$  and  $4.5\mathrm{M}$  sentence pairs. In order to verify the universality of the proposed method, we also conduct experiments on the large-scale training corpus, i.e. WMT17  $\mathrm{Zh} \Rightarrow \mathrm{En}$ , in which,  $20\mathrm{M}$  examples are extracted as the training set. We use the standard validation and test sets provided in each translation task. The Chinese sentences are segmented by the word segmentation toolkit Jieba, $^{4}$  while the sentences in other languages are tokenized using the scripts provided in Moses. $^{5}$  All the data are
processed by byte-pair encoding to alleviate the Out-of-Vocabulary problem (Sennrich et al., 2016b) with  $32\mathrm{K}$  merge operations for both language pairs. The case-sensitive 4-gram NIST BLEU score (Papineni et al., 2002) is used as the evaluation metric.
Model Our experiments are based on TRANSFORMER (Vaswani et al., 2017) and the compared methods are re-implemented on top of our in-house codes. Considering the small-scale translation task  $\mathrm{En} \Rightarrow \mathrm{Vi}$ , we use the setting same as Platanios et al. (2019) in which the dropout ratio is set to 0.3 and each iteration batch consists of 4,096 tokens. For translation models on  $\mathrm{En} \Rightarrow \mathrm{De}$  and  $\mathrm{Zh} \Rightarrow \mathrm{En}$ , we follow the common Base setting in Vaswani et al. (2017) except that we set dropout ratio to 0.1 and train models with a total batch of 32,768 tokens. As to LMs, we train 4-gram KENLM (Heafield, 2011) $^6$  and 2 layers RNNLM (Mikolov et al., 2010) with dimensionality being 200 on monolingual side of each training corpus. Besides, we also score sentences using multilingual BERT (Devlin et al., 2019) that pre-trained on external data with Base setting for comparison.
We investigate the following methods:
- LENGTH measures data difficulty with sentence length (Kocmi and Bojar, 2017).
- RARITY measures data difficulty with word rarity (Zhang et al., 2018).
- DATA-U represents the proposed method which measures difficulty with data uncertainty on source sentence (src), target sentence (trg), and both sides (joint).
- SQRT governs curriculum with the square root model competence (Platanios et al., 2019).
- MOD-U governs curriculum with the proposed model uncertainty. In our experiments, we set baby steps to 4 as default.
In this section, we evaluate the effectiveness of different components in CL on the  $\mathrm{En} \Rightarrow \mathrm{De}$  task. In the first two series of experiments, we investigate the effects of different measures of data difficulty and model competence. Then, we check how the baby steps applied in our training strategy affect the performance. The results are concluded in Table 1.
$^{3}$ Our code is available at https://github.com/NLP2CT/ua-cl-nmt
4https://github.com/fxshy/jieba
<sup>5</sup>https://github.com/mosesdecoderoder
$^{6}$ https://github.com/kpu/kenlm
6938
Model	SQRT	MOD-U
TRANSFORMER	32.76
LENGTH	32.80	33.23↑
RARITY	32.84	33.39↑
DATA-U	KENLM (src)	33.03	33.64↑
KENLM (trg)	33.09	33.69↑
KENLM (joint)	33.15	33.85↑
RNNLM (joint)	33.17	33.73↑
BERT (joint)	33.35*	33.93↑*
Table 1: Ablation study of various measures with respect to data difficulty and model competence for CL in NMT. The results are evaluated using BLEU on  $\mathrm{En}\Rightarrow \mathrm{De}$  translation task, where \* indicates that the result is produced with a LM trained on external data.  $" \uparrow / "\uparrow "$  indicates statistically significant difference from the SQRT counterpart  $(p <   0.05 / 0.01)$  , tested by bootstrap resampling (Koehn, 2004).
Effectiveness of Data Uncertainty We first compare different difficulty measures in CL. Considering the existing methods, both the LENGTH and RARITY yield improvements over the baseline model, which is consistent with prior findings in Kocmi and Bojar (2017), Zhang et al. (2018) and Platanios et al. (2019). The proposed data uncertainty strategies outperform the baseline and existing measures. This verifies our hypothesis that data uncertainty is of higher relevance in respect to the difficulty of an example for a NMT model than its sentence length and word rarity counterparts.
Specifically, the results show the utility of estimating the uncertainty on either the source or target side of a translation pair. Among the two strategies, the target one performs better. We attribute this to the fact that the target uncertainty brings a more direct reflex of the sentence generation difficulty, thus playing a crucial role in CL. Moreover, "joint", which provides a more comprehensive way to model data uncertainty, achieves the best results. This success indicates that the two strategies are complementary to each other and the complexity of a translation pair is contributed by both sides.
We attempt three kinds of LMs to quantify data uncertainty. As seen, all the models contribute to the model performance. Concerning LMs trained on the monolingual side of a parallel corpus, KENLM and RNNLM get comparable translation qualities. Besides, as a state-of-the-art LM, BERT has recently attracted a lot of interests since it learns from billions of external sentences. As expected,
it outperforms all the LMs trained on internal data. Although this comparison is unfair, the results suggest that the performance of LM significantly affects the evaluation of data uncertainty. Since the statistical approach can be faster developed and it does not rely on external data, we choose KenLM as the default in the subsequent experiments.