SlowGuess/ACL_2020_OCR huggingface dataset:
ID	Model	Dev.	Test	Updates	Speedup
Existing Baselines
11	Base Transformer (Ghazvininejad et al., 2019)	-	23.74	-	-
12	Big Transformer (Ghazvininejad et al., 2019)	-	24.65	-	-
Our Implemented Baselines
13	Base Transformer (Vaswani et al., 2017)	22.29	23.69	150.0K	1.00x
14	13+Competence-based CL (Platanios et al., 2019)	22.75	24.30	60.0K	2.50x
Our Proposed Method
15	13+Norm-based CL	23.41	25.25	45.0K	3.33x
Table 4: Results on the large-scale WMT'17 Chinese-English translation task. Dev. is the newsdev2017 while Test is newtest2017. 'Updates' means the step of each model reaching the best performance of model (13) ( $\mathrm{K} =$  thousand), while 'Speedup' means its corresponding speedup.
(a) Length
(b) Frequency
Figure 3: Translation performance of each NMT system in (a) length-based, (b) frequency-based, and (c) norm-based difficulty buckets. The reported BLEU scores are evaluated on the three subsets evenly divided by the En-De test set based on sentence difficulty. NBCL and CBCL denote norm-based and competence-based CL, respectively. CBCL+NBSW denotes the integration of norm-based sentence weight and competence-based CL.
(c) Norm
former baseline, as well as speeding up the training process by a factor of 2.22. One can note that all of our implemented systems have the same number of model parameters; besides, the training step of each model involves essentially the same execution time, resulting in a deployment-friendly system.
4.3 Effect of  $\lambda_{m}$  and  $\lambda_{w}$
Table 3 shows the effects of the two hyperparameters used in the proposed method. For each experiment, we kept the other parameters unchanged and only adjusted the hyperparameter. For  $\lambda_{m}$ , controlling curriculum length, the higher the value, the longer the curriculum length. When setting  $\lambda_{m}$  to 2.5 with the curriculum length of nearly 29K steps, it achieves the best performance. For  $\lambda_{w}$ , the scaling sentence weight of the objective function, one achieves satisfactory results with a value of 0.5, which maintains the right balance between the learning of simple and hard examples.
4.4 Results on the Large-scale NMT
Although the hyperparameters  $\lambda_{m}$  and  $\lambda_{w}$  have been sufficiently validated on the En-De translation, the generalizability of the model trained using these two hyperparameters is still doubtful. To clear up any doubts, we further conducted the experiments on the large-scale Zh-En translation without tuning these two hyperparameters, that is, directly using  $\lambda_{m} = 2.5$  and  $\lambda_{w} = 0.5$ . Specifically, the only difference is the use of a large number of training steps in Zh-En, namely,  $150\mathrm{K}$ , for the purpose of better model fitting.
We first confirm the effectiveness of competence-based CL in large-scale NMT, that is model (14), which shows both a performance boost and a training speedup. Model (15), which trains NMT with the proposed norm-based CL, significantly improves the BLEU score to 25.25 (+1.56) and speeds up the training by a factor of 3.33, showing the generalizability of the proposed method. The results
Source Reference	Last year a team from the University of Lincoln found that dogs turn their heads to the left when looking at an aggressive dog and to the right when looking at a happy dog. Letztes Jahr fand ein Team der Universität von Lincoln hereaus, dass Hunde den Kopf nach links drehen, wenn sie einen aggressiven Hund ansehen, und nach rechts, wenn es sich um einen zufriedenen Hund handelt.
Vanilla NBCL	Im vergangenen Jahr stellte ein Team der Universität Lincoln fest, dass Hunde beim Blick auf einen aggressiven Hund nach links abbiegen. Letztes Jahr fand ein Team von der Universität von Lincoln hereaus, dass Hunde ihren Kopf nach links drehen, wenn sie einen aggressiven Hund sehen und rechts, wenn sie einen,GLICKlichen Hund sehen.
Table 5: Example of a translation which is regarded as a difficult sentence in terms of the norm-based sentence difficulty, from the En-De test set. The vanilla Transformer omits translating the bold part of the source.
show that large-scale NMT obtains a greater advantage from an orderly curriculum with enhanced representation learning. The proposed norm-based CL enables better and faster training of large-scale NMT systems.
4.5 Effect of Sentence Weight
As discussed in Section 3.3, competence-based CL over-trains on the simple curriculum, which might lead to a bias in the final translation. To verify this, we quantitatively analysed the translations generated by different systems. Figure 3 presents the performance of the vanilla Transformer, and of the NMTs trained by competence-based CL and norm-based CL. By dividing the En-De test set (3,003 sentences) into three subsets (1001 sentences) according to the length-based sentence difficulty, the frequency-based sentence difficulty, and the norm-based sentence difficulty, we calculated the BLEU scores of each system on each subset.
The results confirm our above assumption, although competence-based CL performs much better in translating simple sentences due to its overtraining, the translation of sentences of medium difficulty worsens. However, the norm-based CL benefits from the norm-based sentence weight, successfully alleviating this issue by applying a scale factor to the loss of simple curricula in the objective function, leading to a consistently better translation performance over the vanilla Transformer.
To further prove the effectiveness of the proposed norm-based sentence weight, we explore the model integrating norm-based sentence weight with competence-based CL, and find that it can also strike the right balance between translating simple and medium-difficulty sentences.
4.6 A Case Study
Table 5 shows an example of a translation of a difficult sentence consisting of several similar clauses in the norm-based difficulty bucket. We observe that the translation by the vanilla model omits translating the last clause, but NMT with norm-based CL translates the entire sentence. The proposed method enhances the representation learning of NMT, leading to better understandings of difficult sentences, thus yielding better translations.
The norm of a word embedding has been sufficiently validated to be highly correlated with word frequency. Schakel and Wilson (2015) and Wilson and Schakel (2015) train a simple word embedding model (Mikolov et al., 2013) on a monolingual corpus, and find that the norm of a word vector is relevant to the frequency of the word and its context sensitivity: frequent words and words that are insensitive to context will have word vectors of low norm values.
For language generation tasks, especially NMT, there is still a correlation between word embedding and word frequency. Gong et al. (2018) observe that the word embedding of NMT contains too much frequency information, considering two frequent and rare words that have a similar lexical meaning to be far from each other in terms of vector distance. Gao et al. (2019) regard this issue as a representation degeneration issue that it is hard to learn expressive representations of rare words due to the bias in the objective function. Nguyen and Chiang (2019) observe a similar issue during NMT decoding: given two word candidates with similar lexical meanings, NMT chooses the more frequent one as the final translation. They attribute
this to the norm of word vector, and find that target words with different frequencies have different norms, which affects the NMT score function. In the present paper, for the sake of obtaining an easy and simple word vector norm requirement, we use the norm derived from a simple word embedding model. In the future, we would like to test norms of various sorts.
There are two main avenues for future research regarding CL for NMT: sentence difficulty criteria and curriculum training strategies. Regarding sentence difficulty, there are linguistically motivated features (Kocmi and Bojar, 2017; Platanios et al., 2019) and model-based features (Zhang et al., 2017; Kumar et al., 2019; Zhang et al., 2019b; Zhou et al., 2020). Both types of difficulty criteria have their pros and cons, while the proposed norm-based sentence difficulty takes the best of both worlds by considering simplicity and effectiveness at the same time.
Regarding the training strategy, both deterministic (Zhang et al., 2017; Kocmi and Bojar, 2017) and probabilistic strategies (Platanios et al., 2019; Zhang et al., 2019b; Kumar et al., 2019) can be better than the other, depending on the specific scenario. The former is easier to control and explain, while the latter enables NMT to benefit from the randomization of mini-batch training. However, both kinds of strategy need to carefully tune the CL-related hyperparameters, thus making the training process somewhat costly. In the present paper, we have designed a fully automated training strategy for NMT with the help of vector norms, removing the need for manual setting.
We have proposed a novel norm-based curriculum learning method for NMT by: 1) a novel sentence difficulty criterion, consisting of linguistically motivated features and learning-dependent features; 2) a novel model competence criterion enabling a fully automatic learning framework without the need for a task-dependent setting of a feature; and 3) a novel sentence weight, alleviating any bias in the objective function and further improving the representation learning. Empirical results on the medium- and large-scale benchmarks confirm the generalizability and usability of the proposed method, which provides a significant performance boost and training speedup for NMT.
This work was supported in part by the National Natural Science Foundation of China (Grant No. 61672555), the Joint Project of the Science and Technology Development Fund, Macau SAR and National Natural Science Foundation of China (Grant No. 045/2017/AFJ), the Science and Technology Development Fund, Macau SAR (Grant No. 0101/2019/A2), and the Multi-year Research Grant from the University of Macau (Grant No. MYRG2017-00087-FST). We thank the anonymous reviewers for their insightful comments.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015.
Ankur Bapna, Mia Xu Chen, Orhan First, Yuan Cao, and Yonghui Wu. 2018. Training Deeper Neural Machine Translation Models with Transparent Attention. In EMNLP 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In ICML 2009.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. In TACL 2017.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In EMNLP 2018.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration problem in training natural language generation models. In ICLR 2019.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In ICML 2017.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-Predict: Parallel Decoding of Conditional Masked Language Models. In EMNLP 2019.
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2018. FRAGE: Frequency-Agnostic Word Representation. In NIPS 2018.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andre F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in  $\mathrm{C + + }$  .In ACL 2018.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP 2013.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR 2015.
Tom Kocmi and Ondrej Bojar. 2017. Curriculum Learning and Minibatch Bucketing in Neural Machine Translation. In RANLP 2017.
Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. 2019. Reinforcement learning based curriculum optimization for neural machine translation. In NAACL 2019.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In ICLR 2017.
Xuebo Liu, Derek F. Wong, Lidia S. Chao, and Yang Liu. 2019a. Latent attribute based hierarchical decoder for neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):2103-2112.
Xuebo Liu, Derek F. Wong, Yang Liu, Lidia S. Chao, Tong Xiao, and Jingbo Zhu. 2019b. Shared-private bilingual word embeddings for neural machine translation. In ACL 2019.
Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2):159-165.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In *ICLR* 2013.
Toan Nguyen and David Chiang. 2019. Improving lexical choice in neural machine translation. In *NAACL-HLT* 2019.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In WMT@EMNLP 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In ACL 2002.
Ankur P Parikh, Oscar Tackström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In EMNLP 2016.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom M. Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In NAACL 2019.
Ofir Press and Lior Wolf. 2017. Using the Output Embedding to Improve Language Models. In EACL 2017.
Adriaan M J Schakel and Benjamin J Wilson. 2015. Measuring Word Significance using Distributed Representations of Words. arXiv.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL 2016.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS 2014.
Joseph P Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In ACL 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS 2017.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In ACL 2019.
Benjamin J Wilson and Adriaan M J Schakel. 2015. Controlled Experiments for Word Embeddings. arXiv.
Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2019. Depth growing for neural machine translation. In ACL 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv.
Biao Zhang, Ivan Titov, and Rico Sennrich. 2019a. Improving deep transformer with depth-scaled initialization and merged attention. In EMNLP 2019.
Dakun Zhang, Jungi Kim, Josep Crego, and Jean Senellart. 2017. Boosting Neural Machine Translation. In IJCNLP 2017.
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy G Winnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. 2018. An Empirical Exploration of Curriculum Learning for Neural Machine Translation. arXiv.
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019b. Curriculum Learning for Domain Adaptation in Neural Machine Translation. In NAACL 2019.
Yikai Zhou, Baosong Yang, Derek F. Wong, Yu Wan, and Lidia S. Chao. 2020. Uncertainty-aware curriculum learning for neural machine translation. In ACL 2020.