SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: Natural-Instructions results. The results for GPT3 are taken from Mishra et al. (2021). The best result except SameTask in each column for each model size is boldfaced. MS=model size.
Model	BQ	MC	CA	RE	CB	Avg.
LM	52.2	26.6	63.0	50.8	38.3	46.2
SelfSup.	55.7	33.6	67.6	53.0	44.9	51.0
NSG	52.1	25.9	64.0	51.0	41.2	46.9
CL	52.5	26.8	61.4	50.9	48.1	47.9
MWP	51.9	26.3	61.8	50.8	36.1	45.4
LPP	53.5	29.5	61.6	52.0	40.3	47.4
points using the new templates (NewTemplate) to study the effect of new templates.
We report the results for SuperGLUE and Natural-Instructions in Table 2 and Table 3. Our findings are as follows:
1. Our proposed self-supervised training achieves the best performance on average for both benchmarks.
2. ExtraLM and NewTemplate show similar perfor
Figure 4: Average results for the 1.3B model on SuperGLUE and Natural-Instructions when varying the number of examples used for self-supervised training.
Table 4: SuperGLUE results when training the 125M model with only one of the self-supervised tasks.
Model	QG	AG	MM	VF	Avg.
LM	33.7	12.9	53.0	14.7	28.6
SelfSup.	16.9	14.6	70.1	18.9	30.0
NSG	32.3	12.5	54.0	13.8	28.2
CL	8.3	0.3	1.0	2.7	3.1
MWP	15.2	19.4	50.5	17.8	25.7
LPP	11.3	16.6	49.5	19.9	24.3
Table 5: Natural-Instructions results when training the 125M model with only one of the self-supervised tasks.
mances as the pretrained language model checkpoints, suggesting that the improvements from our self-supervised training is unlikely to come from the additional training on the data and the task template changes.
3. Compared to the pretrained language model checkpoints, CrossTask shows worse performances on both benchmarks, which is likely due to the differences between training tasks and evaluation tasks.
5.1 Effect of Amount of Data
In Figure 4, we report model performances for the 1.3B model on SuperGLUE and Natural-Instructions with  $1\%$ ,  $10\%$ ,  $20\%$ ,  $50\%$ , and  $200\%$
ALL	ALL+MoreTask
SuperGLUE Results
125M	51.0	50.9
1.3B	55.6	55.6
Natural-Instructions Results
125M	30.0	31.7
1.3B	45.5	45.4
Table 6: Average results when adding denoising autoencoding and gap sentence prediction to the self-supervised training. ALL: use all of the self-supervision described in Sec. 3.
LM	Correct Label	Random Label
SuperGLUE Results
125M	46.2	51.0	38.2
1.3B	50.7	55.6	42.5
Natural-Instructions Results
125M	28.6	30.0	19.1
1.3B	43.8	45.5	31.5
of training examples. $^{8}$  We train the models for ten epochs. $^{9}$  As shown in the figure, when the amount of training data for self-supervised tasks is similar to that for the CrossTask setting (i.e.,  $1\%$  data), the self-supervised tasks also lead to worse performances. The improvements become clearer when we increase the number of training data, but it begins to plateau at around  $100\%$  data. This suggests that one of the advantages of the self-supervised tasks compared to the tasks in the CrossTask setting is the amount of training data. We hypothesize that further increasing the amount of data not being helpful is because the data used for constructing the self-supervised tasks has already been used for language model pretraining. So, our models manage to learn to solve these tasks with a relatively limited amount of data. We have similar observations for the 125M model. See appendix A.8 for more details. $^{10}$
5.2 Effect of Individual Self-Supervised Tasks
We investigate the effect of individual self-supervised tasks by training models with only one
Table 7: Average model performance comparing whether we assign random labels to the self-supervised tasks.
MS	GPT3 Template	Our Template
LM	125M	48.4	46.2
SelfSup	125M	47.2	51.0
LM	1.3B	51.8	50.7
SelfSup	1.3B	51.1	55.6
Table 8: Average results for SuperGLUE when using different task templates. MS=model size.
Model	MS	QG	AG	MM	VF	Avg.
LM	125M	33.7	12.9	53.0	14.7	28.6
CrossTask	125M	22.0	24.8	66.9	17.9	32.9
SelfSup.	125M	16.9	14.6	70.1	18.9	30.0
Combined	125M	23.5	25.2	70.3	18.5	34.4
LM	1.3B	40.9	32.5	74.0	27.8	43.8
CrossTask	1.3B	38.1	41.6	69.2	23.0	42.9
SelfSup.	1.3B	43.9	37.5	72.3	28.6	45.5
Combined	1.3B	42.1	42.5	74.1	28.7	46.9
Table 9: Natural-Instructions results when combining the self-supervised tasks and the tasks in the CrossTask setting. The best performance in each column for each model size is boldfaced. MS=model size.
task. We report the experiment results in Table 4 and Table 5. More results and discussions are in appendix A.9. Our findings are:
1. Combining all four self-supervised tasks results in the biggest improvements for most tasks, suggesting that the tasks are complementary.
2. Each self-supervised task improves a few downstream task performances (e.g., NSG helps COPA; CL helps MultiRC and CB). This is likely due to similarities between tasks.
3. It is worth noting that while CL hurts model performances on Natural-Instructions, it helps on the SuperGLUE. We hypothesis that this is because unlike Natural-Instructions, SuperGLUE is ranking based and, therefore, more favorable to classification-related training.
4. It is interesting to see that NSG and CL tasks are the two most beneficial to downstream performance among the four self-supervised tasks. This is likely due to (1) the generic task formulation of NSG, and (2) CL requires different inference abilities compared to the other self-supervised tasks. It is also interesting that training on only one of the self-supervised tasks can hurt the performance on Natural-Instruction.
5.3 Effect of More Self-Supervised Tasks
To investigate the effect of having more self-supervised tasks during training, we add two extra self-supervised tasks to the self-supervised training,
<sup>8</sup>We apply the same ratio to all the self-supervised tasks and use the same development sets for each task across these settings.
<sup>9</sup>Upon manual inspection, we found that the development set loss values in these experiments have converged.
10 Our goal for this analysis is to show the rough trends of model performance when varying the amount of training data, rather than to provide an exact estimate of the training data required for the self-supervised training.
Task Prompt	Task Input	Reference	LM	Self-Supervised
Construct a question from the given fact by a simple rearrangement of words.	Fact: Pollen seeds come from male gametes of plants.	what seeds come from male gametes of plants?	What might cause harm to plants?	What would you use to measure the number of male gametes of plants?
Ask a question on “event duration” based on the provided sentence.	Sentence: At the sight of the great man, Spear flushed crimson, and then his look of despair slowly disappeared; and into his eyes there came incredulously hope and gratitude.	How long did Spear see the great man?	How long did he stay in the Embassy?	How long did it take for Spear to look at the great man?
Answer the given question. Your answer must be a single span in the passage.	Passage: ... The following year he won a scholarship to the Royal Academy of Music, ... The principal of the Academy, Sir Alexander Mackenzie, had forbidden ... Question: What was the full name of the school Sir Alexander Mackenzie was principal of?	Royal Academy of Music.	Oliver.	the Royal Academy of Music.
Answer the given question. Your answer must be a single span in the passage.	Passage: ... Epitaph Records, founded by Brett Gurewitz of Bad Religion, was the base for many future pop punk bands ... The mainstream pop punk of latter-day bands such as Blink-182 is criticized by many punk rock devotees; in critic Christine Di Bella’s words ... Question: What is the full name of the person that is very critical of modern mainstream pop punk bands?	Christine Di Bella.	the “Bad Religion”.	many punk rock devotees.
Table 10: Generation examples by the 1.3B model. The examples are taken from Natural-Instructions. The first two examples are from QG, and the other two are from AG. We only show part of the passages relevant to the outputs for QA for brevity.
following the same procedure as the other tasks. The additional tasks are: denoising autoencoding (Lewis et al., 2020) and gap sentence generation (Zhang et al., 2020). Denoising autoencoding is the task of reconstructing the original sentences from sentences corrupted by random noises, which has been shown effective for training generic language representations; gap sentence generation is to recover the missing sentence and has been found useful for abstractive summarization.
We report the results in Table 6 where we do not find adding the two tasks improves downstream tasks. This is likely because the two tasks share similarities with our existing tasks (e.g., gap sentence generation shares a similar inference style as MWP). So, adding them does not promote diversity in the self-supervised tasks, leading to the fact that the models are not encouraged to learn different information.
5.4 Effect of Few-Shot Templates
The self-supervised training brings two benefits: making models familiar with the few-shot templates and task semantics. To differentiate the effect of the two, we train models on the self-supervised tasks with random labels. For example, for NSG,
we use random sentences as outputs rather than the true next sentences; for the binary classification tasks, we randomly select binary labels. As shown in the results in Table 7, random labels hurt model performances, suggesting that what the models have learned is more than the few-shot templates.
We also investigate the effect of task templates for SuperGLUE by evaluating models using different templates. We report results in Table 8 where we find that having the templates for downstream tasks similar to the ones used for self-supervised training gives the models significantly better performances.
5.5 Zero-Shot vs. One-Shot vs. Few-Shot
We show zero-shot, one-shot, and few-shot performances for the LM and the self-supervised model in Table 11. We find that among the three settings, the self-supervised training is the most helpful in the few-shot setting and does not help in the zero-shot setting, suggesting that the self-supervised training improves the models' in-context learning capabilities.
Zero-Shot	One-Shot	Few-Shot
LM	SS	LM	SS	LM	SS
125M Δ	46.7	44.3	42.6	46.1	46.2	51.0
(-2.4)	(+3.5)	(+4.8)
1.3B Δ	49.5	49.9	46.5	50.8	50.7	55.6
(+0.4)	(+4.3)	(+4.9)
Table 11: Average results for SuperGLUE showing the zero-shot, one-shot, and few-shot model performances for the LM and the self-supervised model (SS). The numbers in parenthesis are the performance differences between the LM and the SS with the positive numbers indicating improvements. We boldface the largest improvement for each model.
5.6 Combine Self-Supervision with Cross-Task Human-Supervision
We investigate the relations between the self-supervised tasks and the human-annotated tasks. We combine the tasks from the self-supervision and those from the CrossTask and report the results in Table 9. Interestingly, combining the two kinds of tasks results in better performances on average, showing that they are complementary.
5.7 Generation Examples
We show generation examples in Table 10. In general, we find that compared to the vanilla pretrained language models, the self-supervised models are better at using information from task input following task requirements. Specifically, for the first two examples in Table 10, the LM suffers from more severe semantic drift than the self-supervised model (e.g., "male gametes of plants" is more specific and relevant to the task input than "plants"). We have similar observations for the third example, where "Oliver" is a name from the task demonstration rather than the passage. Interestingly, for the last example, the answer generated by the LM is from the passage but is actually "the base for many future pop punk bands" instead of what the question looks for (i.e., "very critical of modern mainstream pop punk bands"). While the answer generated by the self-supervised model does not exactly match the reference, it is partially correct as the mainstream pop punk "is criticized by many punk rock devotees".
We evaluated four self-supervised objectives on two few-shot benchmarks by casting the self-supervised training as an intermediate training stage between language model pretraining and
downstream few-shot evaluation. Empirically, we have shown that the models trained by the self-supervised objectives show the best performances compared to strong baselines on average. Analysis showed that (1) the amount of self-supervised training data and the diversity of the self-supervised tasks can affect the downstream performances.; (2) the self-supervised tasks are complementary to the human-annotated datasets; and (3) the self-supervised-trained models are better at following task requirements.
Armen Aghajanyan, Anchit Gupta, Akshit Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021a. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5799-5811, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021b. HtIm: hyper-text pre-training and prompting of language models. arXiv preprint arXiv:2107.06955.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. 2021. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684.
Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. 2020. Self-supervised meta-learning for few-shot natural language classification tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 522-534, Online. Association for Computational Linguistics.
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge.
Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021. Flex: Unifying evaluation for few-shot nlp. In Advances in Neural Information Processing Systems, volume 34, pages 15787-15800. Curran Associates, Inc.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Ting-Yun Chang and Chi-Jen Lu. 2021. Rethinking why intermediate-task fine-tuning works. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 706-713, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mingda Chen, Zewei Chu, and Kevin Gimpel. 2019. Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 649-662, Hong Kong, China. Association for Computational Linguistics.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. KGPT: Knowledge-grounded pretraining for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8635-8648, Online. Association for Computational Linguistics.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. *BoolQ: Exploring the surprising difficulty of natural yes/no questions.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In *Machine learning challenges: evaluating predictive uncertainty*, visual object classification, and recognising textual entailment, pages 177-190. Springer.
Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5925-5932, Hong Kong, China. Association for Computational Linguistics.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The Commitment-Bank: Investigating projection in naturally occurring discourse. To appear in proceedings of Sinn und Bedeutung 23. Data can be found at https://github.com/mcdm/CommitmentBank/.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jingfei Du, Edouard Grave, Belize Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. 2021. Self-training improves pre-training for natural language understanding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5408-5418, Online. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions? arXiv preprint arXiv:2010.11982.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computational Linguistics.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2021. Pvt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022. Incontext learning for few-shot dialogue state tracking. arXiv preprint arXiv:2203.08568.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China. Association for Computational Linguistics.
Yacine Jernite, Samuel R Bowman, and David Sontag. 2017. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557.
Robin Jia, Mike Lewis, and Luke Zettlemoyer. 2021. Question answering infused pre-training of general-purpose contextualized representations. arXiv preprint arXiv:2106.08190.
Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Unifying question answering, text classification, and regression via span extraction. arXiv preprint arXiv:1904.09286.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262, New Orleans, Louisiana. Association for Computational Linguistics.
Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2017. Learning what is essential in questions. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 80-89, Vancouver, Canada. Association for Computational Linguistics.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Alexander Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In AAAI.
Tven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627-2636, Online. Association for Computational Linguistics.
Chen-Yu Lee, Chun-Liang Li, Chu Wang, Renshen Wang, Yasuhisa Fujii, Siyang Qin, Ashok Popat, and Tomas Pfister. 2021. ROPE: Reading order equivariant positional encoding for graph-based document