SlowGuess/ACL_2020_OCR huggingface dataset:
The results for MRC tasks are presented in Table 3. In all experiments, our CL approach outperforms its baseline with considerable margin. On SQuAD 2.0, we obtain  $+1.30 \mathrm{EM} / +1.15$  F1 improvements using base model and  $+0.31 \mathrm{EM} / +0.57$  F1 using large model compared to our competitive re-implemented baseline. Note that the performance gain is more significant with Base model. On NewsQA, we also get  $+0.02 \mathrm{EM} / +0.47$  F1 and  $+0.10 \mathrm{EM} / +0.30$  F1 improvements for base and large model respectively.
4.4 GLUE Results
We summarize our GLUE results in Table 4. Results on dev sets show that our CL method consistently outperforms their competitive baseline on all 8 tasks, which proves that our CL is not only robustly effective but also generalizable on a wide range of NLU tasks. Because the model architecture and hyper-parameters setting are identical, all
the performance gains can be attributed to our CL approach alone.
Specifically, we observe that our CL approach is doing better on more challenging tasks. For CoLA and RTE, the margin is up to  $+3.3$  and  $+1.8$  in respective metrics, which is relatively larger than less challenging tasks where the model performance already reached a plateau. Such results are understandable: when learning harder tasks, the model can be overwhelmed by very difficult examples at early stages, and a well-arranged curriculum thus can be more helpful. And for tasks where the baselines are already approaching the human performance like SST-2, our CL approach is still able to provide another  $+0.4$  improvements, which demonstrates the robustness of our approach. Overall, our CL approach obtains  $+0.9$  average score gain on GLUE benchmark compare to our re-implemented baseline.
Results on test sets further demonstrate the effectiveness of our approach. We obtain  $+0.4$  average score gain compare to our re-implementation and the baseline on the leaderboard.
In this section, we delve into our approach on a series of interesting topics including: (i) what is the best CL design strategy for NLU tasks, (ii) can Cross Review really distinguish easy examples from difficult ones, (iii) the best choice of  $N$ . We choose SQuAD 2.0 task in most experiments for generality, and all experiments are performed with BERT Base model.
Comparison with Heuristic CL Methods To demonstrate our advantage over manually designed CL methods, we compare our approach with sev-
6100
Method	SQuAD 2.0	Δ
EM	F1
No Curriculum	-	76.30	-
No Curriculum*	73.66	76.78	-
Rarity+Annealing	73.75	76.90	+0.12
Answer+Annealing	74.02	77.15	+0.37
Question+Annealing	74.35	77.37	+0.59
Paragraph+Annealing	74.45	77.54	+0.76
Cross-Review+Naive order	74.31	77.29	+0.51
Cross-Review+Annealing	74.96	77.93	+1.15
Table 5: Comparisons with heuristic CL design (written in italic). * indicates our re-implementation, Δ indicates absolute improvements on F1.
Figure 2: Statistical illustration on different levels of difficulty examples in SQuAD 2.0. Four line respectively indicate average answer length, question length, paragraph length and the proportion of unanswerable examples with respect to level of difficulty. Bar indicates the number of examples in each bucket. Best viewed in color mode.
eral heuristic curriculum design in Table 5. For Difficulty Review methods, we adopt word rarity, answer length, question length, and paragraph length as difficulty metrics similar to (Tay et al., 2019; Platanios et al., 2019). We calculate word rarity as the average word frequency of the question, where the frequency is count from all questions in trainset. We define difficult examples as those with lower words frequencies, longer answer, question, and paragraph length. We first sort all examples using these metrics, and divide them evenly to obtain 10 example buckets with a corresponding level of difficulty, and the Curriculum Arrangement strategy remains unchanged as Annealing. For Curriculum Arrangement method, we try Naive order for comparison. We directly implement the curriculum as  $\{C_i\}$  (instead of  $\{S_i\}$ ) without any sampling algorithm, only that  $S_{N+1}$  is still retained for fair comparison. In the meantime, the Difficulty Eval
uation method remains unchanged as Cross Review. The results show that these intuitive design indeed works well with various improvements ranging from  $+0.12$  to  $+0.76$  on F1 score. But they are all outperformed by our Cross Review + Annealing approach.
Case study: Easy VS Difficult In our Cross-Review method, the dataset was divided into  $N$  buckets  $\{C_i\}$  with different levels of difficulty. Here we further explore what do these easy/difficult examples in various tasks actually look like. Earlier in the introduction (see Table 1), we have provided a straightforward illustration of easy cases versus hard cases in SST-2 dataset. Among ten different levels of difficulty, these cases are sampled from the most easy bucket  $(C_1)$  and the most difficult bucket  $(C_{10})$ , respectively. The results are very clear and intuitive.
We further choose SQuAD 2.0 as a more complex task to perform in-depth analysis. Under the  $N = 10$  setting, we reveal the statistical distinctions of all buckets  $\{C_i\}$  in Fig 2. With three monotonically increasing curve, it is very clear that difficult examples tend to entail longer paragraph, longer questions, and longer answers. Such conclusions conform to our intuition that longer text usually involves more complex reasoning patterns and context-dependency. And these challenging examples are now successfully excluded in the early stages attributing to our CL approach. Another interesting result is that the percentage of unanswerable examples drops consistently from  $40\%$  to  $20\%$  along the difficulty axis. We assume that simply doing classification is easier than extracting the exact answer boundaries.
On Different Settings of N One argument that needs to be specified ahead in our approach is  $N$ , which decides the number of meta-datasets, learning stages, and also the granularity of our difficulty score. Assume the metric is between 0 and 1, which fits almost all the cases, then the difficulty score  $c_{ji}$  should range from 0 (when all teacher models fail) to  $N - 1$  (when all teacher models succeed), so all examples can be distinguished into  $N$  different levels. With  $N$  becoming larger, the granularity is also finer.
To examine the impact of different settings, we perform ablation study on SQuAD 2.0 task given a wide range of choices: from 2 to 20 (see Fig 3). It is obvious that under all settings our approach
6101
Figure 3: F1 score on SQuAD 2.0 with respect to  $N$ . Dotted line is the baseline, solid line is our re-implementation, best viewed in color mode.
outperforms the baseline by at least  $+0.5$  F1 score (even including  $N = 2$ , where the difficulty evaluation results may be affected by the fluctuation of single-teacher review). We also experiment with extremely large  $N$  value. For  $N = 100$ , the result is 74.10 on F1 score (2.68 below our baseline), which is as expected because the meta-dataset is too small to prepare a decent teacher that is capable of evaluating. In general, our approach is very robust with the settings of  $N$ .
The idea of training a neural network in an easy-to-difficult fashion can be traced back to (Elman, 1993). (Krueger and Dayan, 2009) revisited the idea from a cognitive perspective with the shaping procedure, in which a teacher decomposes a complete task into sub-components. Based on these works, Curriculum Learning is first proposed in (Bengio et al., 2009). They designed several toy experiments to demonstrate the benefits of curriculum strategy both in image classification and language modeling. They also propose that curriculum can be seen as a sequence of training criteria, and at the end of it, the reweighting of examples should be uniform with the target distribution, which inspired the design of our Curriculum Arrangement algorithm.
Although CL has been successfully applied to many areas in computer vision (Supancic and Ramanan, 2013; Chen and Gupta, 2015; Jiang et al., 2017), it was not introduced to solve NLU tasks until (Sachan and Xing, 2016). By experimenting with several heuristics, they migrated the success of CL (Kumar et al., 2010) to machine reading com
prehension tasks. (Sachan and Xing, 2018) further extended this work to question generation. More recently, (Tay et al., 2019) employed CL strategy to solve reading comprehension over long narratives. Apart from them, there aren't very many works that discuss CL in the context of NLU to the best of our knowledge.
On the methodology of designing CL algorithms, our approach is closely related to (Guo et al., 2018; Wang et al., 2019; Platanios et al., 2019; Tay et al., 2019), where a curriculum is formed via two steps: evaluating the difficulty first, then sampling the examples into batches accordingly. For different target tasks, the evaluation methods also vary greatly. (Guo et al., 2018) first examined the examples in their feature space, and define difficulty by the distribution density, which successfully distinguished noisy images. (Wang et al., 2019) incorporated category information into difficulty metric to address imbalanced data classification. In language tasks, (Platanios et al., 2019) and (Tay et al., 2019) propose to consider the length of context as extent of difficulty. Another line of works see curriculum construction as an optimization problem (Kumar et al., 2010; Graves et al., 2017; Fan et al., 2018), which usually involves sophisticated design and is quite different from our approach.
In this work we proposed a novel Curriculum Learning approach which does not rely on human heuristics and is simple to implement. With the help of such a curriculum, language models can significantly and universally perform better on a wide range of downstream NLU tasks. In the future, we look forward to extend CL strategy to the pretraining stage, and guide deep models like transformer from a language beginner to a language expert.
We thank all anonymous reviewers for their valuable comments. This work is supported by the National Natural Science Foundation of China, Grant No.U19A2057, No.61876223, the National Science Fund for Distinguished Young Scholars No.61525206, the Fundamental Research Funds for the Central Universities, Grant No.WK3480000008, and the grant of Tianjin New Generation Artificial Intelligence Major Program No.19ZXZNGX00110.
6102
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48. ACM.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In TAC.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14.
Xinlei Chen and Abhinav Gupta. 2015. Webly supervised learning of convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1431-1439.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Jeffrey L Elman. 1993. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99.
Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and TieYan Liu. 2018. Learning to teach. arXiv preprint arXiv:1805.03643.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP.
Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. 2017. Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1311-1320. JMLR.org.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. 2018. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 135-150.
Guy Hacohen and Daphna Weinshall. 2019. On the power of curriculum learning in training deep networks. arXiv preprint arXiv:1904.03626.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2017. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv:1712.05055.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Kai A Krueger and Peter Dayan. 2009. Flexible shaping: How learning in small steps helps. Cognition, 110(3):380-394.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, pages 1189-1197.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Gail B Peterson. 2004. A day of great illumination: Bf skinner's discovery of shaping. Journal of the experimental analysis of behavior, 82(3):317-328.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell. 2019. Competence-based curriculum learning for neural machine translation. arXiv preprint arXiv:1903.09848.
Martin Popel and Ondrej Bojar. 2018. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 110(1):43-70.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper.pdf.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
6103
Mrinmaya Sachan and Eric Xing. 2016. Easy questions first? a case study on curriculum learning for question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 453-463.
Mrinmaya Sachan and Eric Xing. 2018. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 629-640.
Kornl Csernai Shankar Iyer, Nikhil Dandekar. 2016. DIY corpora: the www and the translator.
Burrhus F Skinner. 1958. Reinforcement today. American Psychologist, 13(3):94.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
James S Supancic and Deva Ramanan. 2013. Self-paced learning for long-term tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2379-2386.
Yi Tay, Shuohang Wang, Luu Anh Tuan, Jie Fu, Minh C Phan, Xingdi Yuan, Jinfeng Rao, Siu Cheung Hui, and Aston Zhang. 2019. Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives. arXiv preprint arXiv:1905.10847.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. 2019. Dynamic curriculum learning for imbalanced data classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 5017-5026.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
6104