SlowGuess/NAACL_2022_OCR huggingface dataset:
Guy Hacohen and Daphna Weinshall. 2019. On the power of curriculum learning in training deep networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2535-2544. PMLR.
Sanggyu Han and Sung-Hyon Myaeng. 2017. Tree-structured curriculum learning based on semantic similarity of text. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 971-976. IEEE.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401.
Borna Jafarpour, Dawn Sepehr, and Nick Pogrebnyakov. 2021. Active curriculum learning. In Proceedings of the First Workshop on Interactive Learning for Natural Language Processing, pages 40-45.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: Crossing format boundaries with a single qa system. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1896-1907.
Tom Kocmi and Ondrej Bojar. 2017. Curriculum learning and minibatch bucketing in neural machine translation. In Proceedings of the International Conference, Recent Advances in Natural Language Processing, RANLP 2017, pages 379-386.
Yajing Kong, Liu Liu, Jun Wang, and Dacheng Tao. 2021. Adaptive curriculum learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5067-5076.
Antonio Laverghetta Jr, Jamshidbek Mirzakhalov, and John Licato. 2020. Towards a task-agnostic model of difficulty estimation for supervised learning tasks. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 16-23.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and
the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. 2019. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems.
Gustavo Penha and Claudia Hauff. 2020. Curriculum learning strategies for ir. In Advances in Information Retrieval, pages 699-713, Cham. Springer International Publishing.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In Proceedings of NAACL-HLT, pages 1162-1172.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Mrinmaya Sachan and Eric Xing. 2018. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 629-640.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on EMNLP and the 9th IJCNLP, pages 4453-4463.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951-2959.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP), pages 9275-9293.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Peiyi Wang, Liang Chen, Tianyu Liu, Baobao Chang, and Zhifang Sui. 2021a. Hierarchical curriculum learning for amr parsing. arXiv preprint arXiv:2110.07855.
Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, and Bryan Hooi. 2021b. Curgraph: Curriculum learning for graph classification. In Proceedings of the Web Conference 2021, pages 1238-1248.
Daphna Weinshall, Gad Cohen, and Dan Amir. 2018. Curriculum learning by transfer learning: Theory and experiments with deep networks. In International Conference on Machine Learning, pages 5238-5246.
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095-6104.
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. G-daug: Generative data augmentation for commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1008-1025.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754-5764.
Seonghyeon Ye, Jiseon Kim, and Alice Oh. 2021. Efficient contrastive learning via novel data augmentation and curriculum learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1832-1838.
Wojciech Zaremba and Ilya Sutskever. 2014. Learning to execute. arXiv preprint arXiv:1410.4615.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800.
Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods
in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2495-2509.
Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, et al. 2020. Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense. Engineering, 6(3):310-345.
Figure 4: Comparison of QAP scores on the (a) training and (b) validation sets of SocialIAQ, for the RoBERTa models trained with and without CL.
A Prefix-Tuning Model
For this scenario, we finetune a GPT2 $_{LARGE}$  model for the task of multiple-choice reasoning using prefix-tuning, wherein the weights of pretrained GPT2 are frozen, and only a matrix of prefix embeddings are trained for the target task. Following Li and Liang (2021), we prepend a prefix to the inputs of GPT2 model i.e.  $z = \text{[PREFIX}, x, y]$  where  $(x, y)$  is the question-answer input pair. We also a parameterization network to compute the prefix embeddings i.e.,  $P_{\theta} = MLP(P_{\theta'})$  where  $P_{\theta}$  is the prefix embedding used during inference,  $P_{\theta'}$  is a smaller prefix matrix used for reparameterization during training and MLP is the dense layer used as the reparameterization network. We set the prefix length to 16, and the dimensions of  $P_{\theta'}$  to 512. The dimensions of  $P_{\theta}$  is 1024, according to the hidden size of GPT2 $_{LARGE}$ . This leads to a total of  $0.2\%$  trainable parameters in our prefix-tuning based GPT2 model.
Datasets. We use two large multiple-choice commonsense-based question answering datasets for our experiments i.e. SocialQA (Sap et al., 2019) and CosmosQA (Huang et al., 2019). Additionally, we also use the CODAH dataset (Chen et al., 2019) folds released in Yang et al. (2019) and follow their method to create HellaSwag-2K (Zellers et al., 2019) for testing our methods on low-resource scenario. All of the above datasets are being for their intended purposes i.e. research only, in our work. All of these datasets are in the English language. Models are evaluated using the respective task-specific accuracies. See Table 9 for dataset statistics.
Hyperparameter	SocialIQA	CosmosQA	CODAH	HellaSwag2K	WinoGrande (XL)
Starting Percent (sp)	0.26	0.37	0.5	0.26	0.39
Increase Factor (inc)	1.89	1.74	1.32	1.05	1.8
Update Steps (S)	1789	1745	332	199	405
Adaptive Factor (α)	0.93	0.99	0.55	0.09	0.19
Table 7: Best adaptive curriculum learning pacing function parameters for finetuning RoBERTa on various datasets
Parameter	Bounds
Grid-search Based Optimization
Learning Rate	[1e-5, 5e-5, 1e-6, 5e-6]
Batch Size (inc)	[8, 16]
Total Train Epochs	[3, 4, 5]
Bayesian Optimization
Starting Percent (t)	[0.01, 0.5]
Increase Factor (λ)	[1.05, 2.0]
*Update Steps (S)	[0.01, 3]*250
Adaptive Factor (α)	[0, 1.0]
Table 8: Optimization bounds for grid-search based tuning of training hyperparameters and bayesian optimization of curriculum learning parameters. *We expand the search window for update steps to 2500 for larger datasets in adaptive CL
Split	SIQA	CosQA	CDH	H2K	WG
Train	33,410	25,778	1666	2000	40,498
Dev	1954	3000	566	1000	1266
Test	2224	7000	566	5000	1767
Table 9: Number of samples in each split of the datasets used in our experiments. (SIQA = SocialIQA, CosQA = CosmosQA, CDH = Codah, H2K = HellaSwag2K, WG=WinoGrande-XL)
Models & Hyper-parameters. We use RoBERTa $_{LARGE}$  Liu et al. (2019) as the main PTLM in our experiments. We perform Bayesian optimization for finding the optimum parameters for fixed as well as adaptive CL. For adaptive learning, we set  $L = S$  and optimize for  $\alpha$ ,  $t$ ,  $\lambda$  and  $S$  parameters. All models are also subjected to grid-search based tuning of training hyperparameters.
B.1 Hyperparameter Tuning Bounds
We perform grid-search based optimization for finding the best training hyperparameters for each training scenario and dataset. For curriculum learning based parameters, we first perform bayesian optimization (Snoek et al., 2012) of curriculum learning parameters using the best training hyperparameters for the baseline (non-CL) model. For bayesian optimization, each model is subjected to 15 iterations of optimization with 3 restarts and the tuning
Figure 5: Demonstration of the optimized learning pace of RoBERTa for various datasets. Usage percentage  $k$  (y-axis) at training step  $n$  (x-axis) refers to the top  $k\%$  easiest examples of the ranked dataset.
is based on accuracy on validation set.
For the prefix-tuning models, we perform grid-search based tuning of training hyperparameters according to the bounds reported in Table 8 and in addition, up to 10 training epochs.
Hyperparameter	SocialIQA	CosmosQA	CODAH	HellaSwag2K	WinoGrande (XL)
Learning Rate	5e-6	5e-6	1e-5	1e-5	1e-5
Epochs	3	4	5	5	5
Max Gradient Norm	1.0	1.0	1.0	1.0	1.0
Weight Decay	0.01	0.01	0.01	0.01	0.01
Batch Size	8	8	16	16	16
Max Length	128	128	90	128	70
Warmup Ratio	0.06	0.06	0.06	0.06	0.06
LR Decay	Linear	Linear	Linear	Linear	Linear
Optimizer	AdamW	AdamW	AdamW	AdamW	AdamW
Hardware	RTX 2080 Ti	RTX 2080 Ti	RTX 2080Ti	RTX 2080Ti	RTX 2080Ti
Single GPU training time	5 hours	5 hours	2 hours*	2 hours	5 hours
Table 10: Training hyperparameters for finetuning RoBERTa on various datasets.