SlowGuess/ACL_2020_OCR huggingface dataset:
Effectiveness of Model Uncertainty In this experiment, we evaluate the impacts of different assessments on model competence. Obviously, our approach “MOD-U” consistently gains improvements over the vanilla method “SQRT” with the same setting. These results reveal that applying model uncertainty to determine the duration of each curriculum by the model itself is conducive to CL in NMT. Moreover, the combination of data uncertainty and model uncertainty can progressively boost the model performance, confirming that the two methods are complementary to each other.
Different Baby Steps We further explore the effects of the number of baby steps on  $\mathrm{En} \Rightarrow \mathrm{De}$  translation task. The experiments are conducted on the proposed uncertainty-aware CL model as plotted in Figure 3. The vanilla NMT system without using any curriculum strategy could be considered as the model that sets the total number of steps to 1. As seen, dividing training corpus into 4 baby steps is superior to other settings. Before that, the translation performance increases with progressively subdividing baby steps, since the model with fine-grained steps can benefit more from CL. When the total number of subsets is greater than 4, the tendency of translation qualities decreases. A plausible explanation is that to train the model on an over-small subset leads to the problem of overfitting.
In this section, we evaluate the proposed approach on both IWSLT15  $\mathrm{En} \Rightarrow \mathrm{Vi}$ , WMT16  $\mathrm{En} \Rightarrow \mathrm{De}$ , as well as WMT17  $\mathrm{Zh} \Rightarrow \mathrm{En}$  tasks, as listed in Table 2. Our baseline TRANSFORMER and re-implemented existing methods outperform the reported results in Platanios et al. (2019), which we believe makes the evaluation convincing. As seen, the proposed uncertainty-aware curriculum learning strategy consistently outperforms strong baselines and recent studies that exploit CL into NMT across language pairs. These results demonstrate the universality and effectiveness of the proposed approach.
Model	WMT16 En⇒De	IWSLT15 En⇒Vi	WMT17 Zh⇒En
Baseline & Related Methods
TRANSFORMER	32.76	-	30.01	-	24.19	-
+SQRT+LENGTH	32.80	+0.04	29.83	-0.18	24.17	-0.02
+SQRT+RARITY	32.84	+0.08	30.10	+0.09	24.31	+0.12
The Proposed Models
Uncertainty-Aware	33.85↑	+1.09	30.75↑	+0.74	25.04↑	+0.85
Uncertainty-Aware with BERT	33.93↑	+1.17	30.94↑	+0.93	25.02↑	+0.83
Table 2: Comparing with baseline and existing methods that exploit CL on IWSLT15  $\mathrm{En}\Rightarrow \mathrm{Vi}$ , WMT16  $\mathrm{En}\Rightarrow \mathrm{De}$  as well as WMT17  $\mathrm{Zh}\Rightarrow \mathrm{En}$  translation tasks. The evaluation metric is BLEU.
Figure 3: Evaluation of our models trained with different total number of baby steps, where the number of baby step being 1 represents the vanilla NMT system. The experiments are conducted on  $\mathrm{En} \Rightarrow \mathrm{De}$  task.
It is encouraging to see that the improvement does not diminish but enlarges with the increase of training data, indicating that the model is conducive to the large scale translation tasks. Interestingly, our model with BERT is superior to that with KENLM trained on small scale data, while the gap becomes minor when KENLM learns from a larger training corpus (e.g.  $20\mathrm{M}\mathrm{Zh}\Rightarrow \mathrm{En}$  task). We attribute this to the fact that, with the use of the large-scale training examples, KENLM can describe its data distribution well, and the superiority of BERT tends to marginal in these tasks.
We conduct extensive analyses on  $\mathrm{En} \Rightarrow \mathrm{De}$  task to better understand our model. We investigate three problems: 1) whether the proposed model indeed speeds up the model convergence; 2) how different are between difficulty measures; and 3) how the model uncertainty exactly changes during training.
Figure 4: Convergence curves of different models on  $\mathrm{En} \Rightarrow \mathrm{De}$  development set. Obviously, our model is able to achieve the same performance as baseline with the reduction in update steps of  $53.6\%$ .
5.1 Model Convergence
As aforementioned, one intuition of CL is to speed up the model convergence. Figure 4 shows the learning curves of different models on  $\mathrm{En} \Rightarrow \mathrm{De}$  validation set. As seen, the conventional NMT model reaches the highest BLEU at  $140\mathrm{k}$  steps, while related CL method SQRT+RARITY obtains the same performance at step  $98\mathrm{k}$ , which achieves  $30\%$  acceleration rate. The acceleration effect is slightly asthenic than that reported in Platanios et al. (2019). This could be explained by the fact that their examined models are trained with a batch of 5,120 tokens, which is much smaller than 32,768 used in our experiments. The large batch facilitates the training (Popel and Bojar, 2018), thus weakening the acceleration effect. In spite of that, our model converges  $53.6\%$  faster than the baseline to get the same BLEU score (step  $65\mathrm{k}$ ), showing the action of the proposed method on speeding up the training.
6940
Figure 5: Statistics on the percentage of difference set between the corresponding baby steps that produced by our model (KENLM) and others. As seen, there exist obvious diversities among these methods.
5.2 Difference among Difficulty Measures
It is interesting to investigate the discrepancy among data difficulty measures. Accordingly, we compare the composition of the corresponding baby steps sorted by different difficulty methods. Figure 5 shows the percentage of distinct sentence contained in each subset of our method (KENLM) to that in others (LENGTH, RARITY, and BERT). As seen, there exists considerable diversity among associated baby steps produced by our method and existing approaches. Moreover, the difference in the middle period of curriculums, i.e. step 2 and step 3, is greater than that in step 1 and step 4. This phenomenon reveals that the most "simple" and "complex" sentences quantified by different measures are relatively similar, and the main diversity lies in those sentences of which the difficulties hardly to be distinguished. Therefore, we argue that the improvements of the proposed method may mainly contribute by the differences in these two steps. Besides, the subsets divided by KENLM and BERT have big gaps, which suggest again that the performance of LM plays a crucial role in our approach.
5.3 Variety in Model Uncertainty
In this section, we discuss the training process from the model uncertainty perspective. For better illustration, we define the model confidence as the reciprocal of model uncertainty  $(1 / u^{mod})$ , since the two features are negative correlation (Dong et al., 2018; Wang et al., 2019a). Figure 6 visualizes the curves concerning the average of model confidence on  $\mathrm{En} \Rightarrow \mathrm{De}$  validation set during the curriculum learning. We analyze those models trained on two baby
steps divided by different data difficulty measures, i.e. KenLM, BERT, and RARITY, for comparison.
Obviously, different models draw similar changing trends of model confidence during training, that is, the model confidence first increases sharply, then drops and rises, eventually balances. Surprisingly, the tendency highly accords with the psychology of human students when they getting into a new area, i.e. Dunning Kruger Curve (Figure 1, Kruger and Dunning, 1999). That is, starting from scratch, peoples rapidly grow their knowledge, they therefore have a large amount of confidence. Then, peoples begin to have awareness about how little they really know and are discouraged by their inability. Over time, humans gradually improve, making them more and more confident, and experienced. To some extent, both the artificial neural networks and human beings can be regarded as connectionist models (Munakata and McClelland, 2003). Accordingly, this interpretation can be also used to explain the phenomenon in NMT training. Such kind of fluctuates model confidence confirms that the curriculum duration should not be fixed, and the predefined strategies may be insufficient to cope with the model training. In addition, the models trained in different curriculums with various difficulty measures perform distinct change amplitudes on model uncertainty, indicating the adaptability of our method. These findings support our assumption that the model uncertainty is an effective and self-adaptive indicator to guide the CL.
We propose a novel uncertainty-aware framework to improve the two key components in CL for NMT, i.e. data difficulty measurement and curriculum arrangement. Our contributions are mainly in:
- We propose to estimate the data uncertainty of each training example as its difficulty, which is more explainable and comprehensive.
- We introduce a self-adaptive CL strategy that evaluates the model uncertainty to govern the curriculum by the model itself.
- The extensive experiments on various translation tasks and model settings demonstrate the universal-effectiveness of the proposed framework. Our method is able to achieve over  $50\%$  accelerate rate on model convergence.
- Quantitative and qualitative analyses indicate
6941
Figure 6: Curves of model confidence  $(1 / u^{mod})$  on  $\mathrm{En} \Rightarrow \mathrm{De}$  validation set at different checkpoints. We evaluate the model uncertainties of CL models that exploit different data difficulty measures. It is clear to see that different methods have the same change trend of model confidence but distinct change amplitudes.
that the model confidence is fluctuant at the training time. It surprisingly draws a similar changing curve as human confidence.
As our model is not limited to machine translation, it is interesting to validate the proposed framework into other NLP tasks that need to exploit CL. Another promising direction is to design more powerful training strategies to replace the baby step.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum Learning. In ICML.
Peter F. Brown, John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Fredrick Jelinek, John D. Lafferty, Robert L. Mercer, and Paul S. Roossin. 1990. A Statistical Approach to Machine Translation. Computational Linguistics, 16(2):79-85.
Wray L Buntine and Andreas S Weigend. 1991. Bayesian Back-Propagation. Complex systems, 5(6):603-643.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation. In ACL.
Volkan Cirik, Eduard H. Hovy, and Louis-Philippe Morency. 2016. Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks. CoRR, abs/1611.06204.
Armen Der Kiureghian and Ove Ditlevsen. 2009. Aleatory or Epistemic? Does It Matter? Structural Safety, 31(2):105-112.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In NAACL.
Li Dong, Chris Quirk, and Mirella Lapata. 2018. Confidence Modeling for Neural Semantic Parsing. In ACL.
Jeffrey L Elman. 1993. Learning and Development in Neural Networks: The Importance of Starting Small. Cognition, 48(1):71-99.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In ICML.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R. Scott, and Dinglong Huang. 2018. CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images. In ECCV.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving Human Parity on Automatic Chinese to English News Translation. CoRR, abs/1803.05567.
Kenneth Heafield. 2011. KenLM: Faster and Smaller Language Model Queries. In EMNLP.
6942
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. 2015. Self-Paced Curriculum Learning. In AAAI.
Alex Kendall and Yarin Gal. 2017. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? In NIPS.
Tom Kocmi and Ondrej Bojar. 2017. Curriculum Learning and minibatch Bucketing in Neural Machine Translation. In RANLP.
Philipp Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. In EMNLP.
Kai A Krueger and Peter Dayan. 2009. Flexible Shaping: How Learning in Small Steps Helps. Cognition, 110(3):380-394.
Justin Kruger and David Dunning. 1999. Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments. Journal of personality and social psychology, 77 6:1121-34.
Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. Reinforcement Learning Based Curriculum Optimization for Neural Machine Translation. In NAACL.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-Paced Learning for Latent Variable Models. In NIPS.
Jian Li, Xing Wang, Baosong Yang, Shuming Shi, Michael R Lyu, and Zhaopeng Tu. 2020. Neuron Interaction Based Representation Composition for Neural Machine Translation. In AAAI.
Cao Liu, Shizhu He, Kang Liu, and Jun Zhao. 2018. Curriculum Learning for Natural Answer Generation. In *IJCAI*.
Xuebo Liu, Houtim Lai, Derek F. Wong, and Lidia S. Chao. 2020. Norm-based curriculum learning for neural machine translation. In ACL 2020.
Tomáš Mikolov, Martin Karafiat, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent Neural Network based Language Model. In ISCA.
Yuko Munakata and James L. McClelland. 2003. Connectionist Models of Development. Developmental Science, 6(4):413-429.
Radford M. Neal. 1996. Bayesian Learning for Neural Networks. Springer-Verlag.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling Neural Machine Translation. In ACL.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In ACL.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019. Competence-Based Curriculum Learning for Neural Machine Translation. In NAACL.
Martin Popel and Ondrej Bojar. 2018. Training Tips for The Transformer Model. The Prague Bulletin of Mathematical Linguistics, 110(1):43-70.
N. Sarafianos, T. Giannakopoulos, C. Nikou, and I. A. Kakadiaris. 2017. Curriculum Learning for Multi-task Classification of Visual Attributes. In ICCV(Workshops).
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh Neural Machine Translation Systems for WMT 16. In WMT.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In ACL.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In NIPS.
Brian Thompson, Huda Khayrallah, Antonios Anastasopoulos, Arya D. McCarthy, Kevin Duh, Rebecca Marvin, Paul McNamee, Jeremy Gwinnup, Tim Anderson, and Philipp Koehn. 2018. Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation. In ACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In NIPS.
Yu Wan, Baosong Yang, Derek F. Wong, Lidia S. Chao, Haihua Du, and Ben CH Ao. 2020. Unsupervised Neural Dialect Translation with Commonality and Diversity Modeling. In AAAI.
Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019a. Improving Back-Translation with Uncertainty-Based Confidence Estimation. In EMNLP.
Wei Wang, Isaac Caswell, and Ciprian Chelba. 2019b. Dynamically Composing Domain-Data Selection with Clean-Data Selection by "Co-Curricular Learning" for Neural Machine Translation. In ACL.
Yiru Wang, Weihao Gan, Wei Wu, and Junjie Yan. 2019c. Dynamic Curriculum Learning for Imbalanced Data Classification. CoRR, abs/1901.06783.
Yijun Xiao and William Yang Wang. 2019. Quantifying Uncertainties in Natural Language Processing Tasks. In AAAI.
Mingzhou Xu, Derek F. Wong, Baosong Yang, Yue Zhang, and Lidia S. Chao. 2019. Leveraging Local and Global Patterns for Self-Attention Networks. In ACL.
6943
Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019. Assessing the Ability of Self-Attention Networks to Learn Word Order. In ACL.
Baosong Yang, Derek F. Wong, Lidia S. Chao, and Min Zhang. 2020. Improving Tree-based Neural Machine Translation with Dynamic Lexicalized Dependency Encoding. Knowledge-Based System, 188.
Wojciech Zaremba and Ilya Sutskever. 2014. Learning to Execute. CoRR, abs/1410.4615.
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy G Winnup, Marianna J. Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. 2018. An Empirical Exploration of Curriculum Learning for Neural Machine Translation. CoRR, abs/1811.00739.
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019. Curriculum Learning for Domain Adaptation in Neural Machine Translation. In NAACL.