SlowGuess/NAACL_2022_OCR huggingface dataset:
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 4171-4186.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu. 2021. Few-nerd: A few-shot named entity recognition dataset. arXiv preprint arXiv:2105.07464.
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1-10.
DSTL. 2017. Relationship and entity extraction evaluation dataset. https://github.com/dstl/re3d.
Dilek Hakkani-Tür, Gokhan Tur, Asli Celikyilmaz, YunNung Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. 2016. Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM. In InterSpeech.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with a single qa system.
Eliyahu Kiperwasser and Miguel Ballesteros. 2018. Scheduled multi-task learning: From syntax to translation. Transactions of the Association for Computational Linguistics, 6:225-240.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Quentin Lhoest, Albert Villanova del Moral, Patrick von Platen, Thomas Wolf, Mario Šaško, Yacine Jernite, Abhishek Thakur, Lewis Tunstall, Suraj Patil, Mariama Drame, Julien Chaumont, Julien Plu, Joe Davison, Simon Brandeis, Victor Sanh, Teven Le Scao, Kevin Canwen Xu, Nicolas Patry, Steven Liu, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Nathan Raw, Sylvain Lesage, Anton Lozhkov, Matthew Carrigan, Théo Matussière, Andre von Werra, Lysandre Debut, Stas Bekman, and Clément Delangue. 2021a. huggingface/datasets: 1.15.1.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumont, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierrick Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021b. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Swee Kiat Lim, Aldrian Obaja Muis, Wei Lu, and Chen Hui Ong. 2017. Malwaretextdb: A database for annotated malware articles. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1557-1567.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R Curran. 2013. Learning multilingual named entity recognition from wikipedia. *Artificial Intelligence*, 194:151-175.
Tomoko Ohta, Sampo Pyysalo, Jun'ichi Tsujii, and Sophia Ananiadou. 2012. Open-domain anatomical entity mention detection. In Proceedings of the workshop on detecting structure in scholarly discourse, pages 27-36.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. In International Conference on Learning Representations.
Jason Phang, Thibault Fevry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R Bowman. 2020. Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work? arXiv preprint arXiv:2005.00628.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Juan Diego Rodriguez. 2020. Datasets for entity recognition. https://github.com/juand-r/entity-recognition-datasets.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multitask prompted training enables zero-shot task generalization.
MIT CSAIL Spoken Language Systems Group. 2020. Mit movie corpus. https://groups.csail.mit.edu/sls/downloads/.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2021. Spot: Better frozen model adaptation through soft prompt transfer.
Tu Vu, Tong Wang, Tsendvarsen Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020. Exploring and predicting transferability across NLP tasks. CoRR, abs/2005.00770.
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019a. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465-4476, Florence, Italy. Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems (NeurIPS).
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019c. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR).
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Edward Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. Crossfit: A few-shot learning challenge for cross-task generalization in nlp.
Amir Zeldes. 2017. The gum corpus: Creating multilayer resources in the classroom. *Language Resources and Evaluation*, 51(3):581-612.
Sequence Labelling (SL) Tasks: We source the following sequence labeling datasets and use them in the CONLL data format. Rodriguez (2020) provides preprocessed data in the original splits. We report F1 score on all unseen target tasks. The total number of train examples across all 15 datasets is 225,433.
Pre-finetuning tasks:
- CONLL - English, Spanish, Dutch, German (Sang and De Meulder, 2003)
- Ontonotes - English, Chinese (Weischedel et al., 2013)
ANEM (Ohta et al., 2012)
GUM (Zeldes, 2017)
GMB (Bos et al., 2017)
SEC Filings (Alvarado et al., 2015)
Re3d (DSTL, 2017)
- Malware TextDB (Lim et al., 2017)
Few-NERD (Ding et al., 2021)
SNIPS (Coucke et al., 2018)
- ATIS (Hemphill et al., 1990) with preprocessing from the repository of Hakkani-Tür et al. (2016)
Unseen Target Tasks:
- WNUT17 (Derczynski et al., 2017)
- Wikigold (Balasuriya et al., 2009; Nothman et al., 2013)
- MIT Movie Corpus - EN (Spoken Language Systems Group, 2020)
- NCBI Disease Corpus (Doogan et al., 2014)
Classification (C) Tasks The pre-finetuning classifications tasks are from the Glue benchmark (Wang et al., 2019c) and the 4 unseen tasks are from the Superglue benchmark (Wang et al., 2019b). We use the versions made available via the Huggingface Datasets library (Lhoest et al., 2021b,a). In total, we have 943984 train examples across the
pre-finetuning classification tasks. The unseen target tasks Superglue tasks, again made available via Huggingface. We retain the original splits and report performance on the validation sets. For Commitment Bank (CB) we report F1 score and accuracy for the other 3 unseen tasks as dictated by the metrics module from Huggingface.
Question Answering (QA) Tasks Our 6 prefinetuning QA tasks Figure 1 are obtained from the MRQA dataset. We use the versions made available via the Huggingface Datasets library (Lhoest et al., 2021b,a). In total, we have 435,624 train examples across the pre-finetuning question answering tasks. We use the Books subset of the SubjQA dataset. For QED and SquadAdv, we split the data in a 4:1 ratio for train and validation. These datasets are collected as challenge sets explaining the big difference in performance. For XQuad, we use only the English language data and the original train and validation splits. We report F1 again as the metric of comparison for all QA tasks using the Huggingface metrics module.
The Huggingface Datasets library is released under the Apache License 2.0. The license information for all the sequence labelling datasets are available at Rodriguez (2020).
Heterogeneous Batches When we run prefinetuning on any combination of groups, we pool all the examples from the corresponding datasets and create heterogeneous batches from this pool. This is the chief reason for a gain in terms of computational cost on selecting a smaller subset and we provide statistics to measure this in Appendix B.3.
The model we use for our experiments is an XLM-Roberta model. We learn a shared encoder during pre-finetuning along with separate task specific heads for each pre-finetuning task. Our tasks are of three different formats that use the output of the encoder differently to make predictions—for classification we predict a single label for the entire sequence using the representation of the  $
$  token, for sequence labelling we predict a label for each token in the input sequence and for extractive question answering we demarcate a span of the input sequence that corresponds to the answer for the question. The implementation of each of these follows from standard task-specific heads released
by Huggingface where the corresponding output is fed to a linear classifier. When we initialize the model, we provide it with a list of pre-finetuning tasks to index the various task-specific heads. For each forward pass, the batch consists of the tokenized input as well as the task indices to be used for all the examples. When we use the model to report performance on a target task, the encoder checkpoint is loaded and a new task-specific head is initialized.
B.2 Pre-Finetuning and Target Task Finetuning Details
During pre-finetuning, the model sees examples from multiple tasks in the train and validation sets. We train each pre-finetuning model with an early stopping condition based on validation loss. We compute the validation loss for each task separately at the end of each epoch and the early stopping condition is when the average of the validation losses of all tasks does not improve for 3 epochs. This scheme is kept uniform for all pre-finetuning runs. We also keep the same batch size, 128, and search space of learning rates for pre-finetuning. We use validation data to search for the best learning rate, sweeping from  $1e^{-3}$  to  $1e^{-5}$ . The rest of the Adam optimizer parameters are retained as the default values from Huggingface Trainer (Wolf et al., 2020).
For target task finetuning, we load the saved prefinetuned model and train the model to convergence defined as when the average validation loss does not improve for 3 consecutive epochs. We again sweep for the best learning rate from  $1e^{-3}$  to  $1e^{-5}$  and report performance on the best selection across 5 random restarts.
B.3 Computational Cost
All experiments are run on an Amazon p3.16xlarge EC2 instance containing 8 Tesla V100 GPUs. The relative improvement in computational cost on selecting a smaller subset of tasks is mainly due to having fewer examples since we make use of heterogeneous batches. Following was the per-epoch runtimes during pre-finetuning on the various runs:
- Only-SL - 1131 seconds per epoch
Only-C - 1643 seconds per epoch
- Only-QA - 2237 seconds per epoch
-  $\mathrm{SL} + \mathrm{C}$  - 2661 seconds per epoch
- QA+C - 3413 seconds per epoch
- SL+QA - 3079 seconds per epoch
- SL+QA+C - 4884 seconds per epoch
Grouping of Tasks We note that grouping of NLP tasks is often fuzzy and imprecise. Our chosen grouping was based on task format, as is used in recent work in the field (Vu et al., 2020; Sanh et al., 2021). We note that task similarity measures (Achille et al., 2019) calculate a more principled relationship between tasks. We acknowledge that a more optimum grouping could be found for each target task set but our results (Table 2) show that our chosen heuristic is a reasonable choice to isolate the effect of transferability. Selecting an optimal grouping would be a combinatorial task which would take significantly more compute. The way we choose to group tasks makes it easy to select 'related' tasks for each target task set. This might become more challenging for non-standard tasks.
Controlling the Size of Each Task Group Our chosen task groups have an unequal number of examples per task and group. We chose to retain all the examples from the various datasets since controlling for the number of examples also doesn't account for the relative difficulty of tasks (in particular we see that some of our chosen QA tasks seem to be more difficult for the model) but this could be a future line of research.
Potential Risks The main risks of our project are the risks associated with training large language models. We do not collect the datasets ourselves and use publicly released data which might contain biases against certain protected groups that will be reflected on models trained in this manner. To the best of our knowledge, these are standard datasets and we use them for the released tasks but we do not manually check them for offensive content.