SlowGuess/NAACL_2022_OCR huggingface dataset:
information extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 314-321, Online. Association for Computational Linguistics.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, page 47.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. 2022. Unsupervised cross-task generalization via retrieval augmentation. arXiv preprint arXiv:2204.07937.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
Fei Mi, Wanhao Zhou, Lingjing Kong, Fengyu Cai, Minlie Huang, and Boi Faltings. 2021. Self-training improves pre-training for few-shot learning in task-oriented dialog systems. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1887-1898, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773.
Nikita Moghe, Mark Steedman, and Alexandra Birch. 2021. Cross-lingual intermediate fine-tuning improves dialogue state tracking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1137-1150, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Preprint.
Denis Paperno, Germán Kruszewski, Angeliki Lazari-dou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525-1534, Berlin, Germany. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop.
Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. English intermediate-task training improves zero-shot crosslingual transfer too. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language
Processing, pages 557-575, Suzhou, China. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computational Linguistics.
Clifton Poth, Jonas Pfeiffer, Andreas Rückle, and Iryna Gurevych. 2021. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10585-10605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247, Online. Association for Computational Linguistics.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203-5212, Online. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In *Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems*, pages 1-7.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series.
Raphael Rubino and Eiichiro Sumita. 2020. Intermediate self-supervised learning for machine translation quality estimation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4355-4360, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021b. Few-shot text generation with natural language instructions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 390-402, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021c. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
Taylor Sorensen, Joshua Robinson, Christopher Michael Ryting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An information-theoretic approach to prompt engineering without ground truth labels. arXiv preprint arXiv:2203.11364.
Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4980-4991, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tu Vu, Minh-Thang Luong, Quoc Le, Grady Simon, and Mohit Iyyer. 2021. STraTA: Self-training with task augmentation for better few-shot learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5715-5731, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021a. Entailment as few-shot learner. arXiv preprint arXiv:2104.14690.
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021b. List: Lite self-training makes efficient few-shot learners. arXiv preprint arXiv:2110.06274.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. 2020. Learning from task descriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361-1375, Online. Association for Computational Linguistics.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. 2022. Zero-prompt: Scaling prompt-based pretraining to 1,000 tasks improves zero-shot generalization. arXiv preprint arXiv:2201.06910.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7163-7189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3914-3923, Hong Kong, China. Association for Computational Linguistics.
Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020.
Universal natural language processing with limited annotations: Try few-shot textual entailment as a start. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8229-8239, Online. Association for Computational Linguistics.
Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021. Few-shot intent detection via contrastive pre-training and finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1906-1912, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328-11339. PMLR.
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. Differentiable prompt makes pre-trained language models better few-shot learners. In International Conference on Learning Representations.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885.
Shiwei Zhang and Xiuzhen Zhang. 2021. Does QA-based intermediate training help fine-tuning language models for text classification? In Proceedings of the The 19th Annual Workshop of the Australasian Language Technology Association, pages 158-162, Online. Australasian Language Technology Association.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2856-2878, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "going on a vacation" takes longer than "going for a walk": A study of temporal commonsense understanding. In Proceedings of the
2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363-3369, Hong Kong, China. Association for Computational Linguistics.
A.1 Additional Details for LPP and Classification Tasks
The label strings we used for LPP are as follows: Yes and No, Y and N, True and False, and T and F. We randomly choose from Yes, Y, True, and T as the label string for the positive label and use the other one in the selected pair as the negative label.
The label strings we used for the binary classification task are the same as the classification LPP task. For the three-way classification task, we use the following label strings: Positive and Negative and Neutral, True and False and Neither, T and F and N, Yes and No and Unknown, Y and N and U.
We report dataset statistics for SuperGLUE and Natural-Instructions in Table 12 and Table 13, respectively.
We train our models in PyTorch (Paszke et al., 2017) using FAIRSEQ (Ott et al., 2019).
A.4 More Details for Natural-Instructions
Dataset Sources. CosmosQA (Huang et al., 2019), DROP (Dua et al., 2019), EssentialTerms (Khashabi et al., 2017), MCTACO (Zhou et al., 2019), MultiRC (Khashabi et al., 2018), QASC (Khot et al., 2020), Quoref (Dasigi et al., 2019), ROPES (Lee et al., 2021) and Winogrande (Sakaguchi et al., 2020).
Training Datasets. We used the following 8 datasets when training models in the cross-task setting: subtask026_drop_question_generation, subtask060_ropes_question_generation, subtask028_drop_answer_generation, subtask047_misc Answering_science_questions, subtask061_ropes_answer_generation, subtask059_ropes_story_generation, subtask027_drop_answer_type_generation, subtask046_miscellaenous_question_typing.
Dataset	Task Category	Metrics	#Train	#Test	#Class
BoolQ	Question Answering	Accuracy	9427	3270	2
MultiRC	Question Answering	F1a/EM	5100	953	2
COPA	Question Answering	Accuracy	400	100	2
RTE	Natural Language Inference	Accuracy	2500	278	2
CB	Natural Language Inference	Accuracy/F1	250	57	3
Table 12: Dataset statistics for SuperGLUE. We use the official development sets as test sets.
Dataset	Task Category	#Train	#Test
subtask003_mctaco_question_generation_event_duration	Question Generation	330	100
subtask040_qasc_question_generation	Question Generation	6400	100
subtask002_quoref_answer_generation	Answer Generation	6400	100
subtask033_winogrande_answer_generation	Answer Generation	6400	100
subtask034_winogrande_question_modification_object	Minimal Modification	6400	100
subtask045_miscellaneoussentence_paraphrasing	Minimal Modification	93	100
subtask039_qasc_find_overlapping_words	Verification	6400	100
subtask044 essentials_termsIdentifying_essential_words	Verification	2138	100
Table 13: Dataset statistics for Natural-Instructions.
A.5 List of Function Words for the Last Phrase Prediction Task
We used the following function words for identifying the last phrase: the, a, an, for, including, and, in, is, are, were, was, neither, or, nor, be, at, in, on, by, to, would, will, before, after, of, about, from, excluding, except, during, under, above, then, into, onto, should, shall, must, may, might, than, with, using, can, could, about, as, from, within, without, have, had, been.
A.6 Templates for SuperGLUE
We show the SuperGLUE templates in Table 14.
We tune the hyperparameters based on development set performances. We tune the learning rate in  $\{1\mathrm{e} - 7,5\mathrm{e} - 7,1\mathrm{e} - 6,3\mathrm{e} - 6,5\mathrm{e} - 6,8\mathrm{e} - 6,1\mathrm{e} - 5,3\mathrm{e} - 5,$ $5\mathrm{e} - 5\}$ , and the attention dropout rate in  $\{0.0,0.1\}$ .
A.8 Effect of Amount of Data
In Figure 5, we report model performances for the 125M and 1.3B models on SuperGLUE and Natural-Instructions with  $1\%$ ,  $10\%$ ,  $20\%$ ,  $50\%$ , and  $200\%$  of training examples.
A.9 Effect of Individual Self-Supervised Task to Downstream Tasks
We investigate the effect of individual self-supervised task by considering two experiment settings: training models with only one task and training models with one task excluded. We report the experiment results in Table 15, Table 16, Table 17, and Table 18. Our findings are:
Figure 5: Average results on SuperGLUE and Natural-Instructions when varying number of examples used for training.
1. Combining all the four self-supervised tasks gives the largest improvements for most tasks, suggesting that these tasks are mostly complementary.
2. Each self-supervised task improves a few downstream task performances (e.g., NSG helps COPA; CL helps MultiRC and CB). This is likely due to the semantic similarities between tasks.
3. It is worth noting that (1) while CL hurts model performances on Natural-Instructions, it helps on the SuperGLUE; and (2) excluding NSG hurts the model performances most on Natural-Instructions whereas excluding CL hurts the most on SuperGLUE. This presumably is because SuperGLUE is ranking based and therefore is more favorable to classification-related training, whereas the tasks in Natural-Instructions are generation tasks and thus benefits more from generation-related tasks.
GPT3	${Context} <newline> question: ${Question} <newline>answer:{Answer}
Ours	Input: ${Context} question: ${Question} answer: True<newline>Output: ${Answer}
(a) BoolQ Template.
GPT3	${Context} <newline> question: ${Question} True or False? <newline>answer:${Answer}
Ours	Input: ${Context} question: ${Question} answer: True<newline>Output: ${Answer}
(b) RTE Template.
GPT3	${Context} <newline>${Answer}
Ours	Input: ${Context} <newline>Output:${Answer}
(c) COPA Template.
GPT3	${Context} <newline> question: ${Question} true, false, or neither? <newline>answer: ${Answer}
Ours	Input: ${Context} question: ${Question} true, false, or neither? <newline>Output: ${Answer}
Table 14: Evaluation templates for SuperGLUE.  $\mathbb{S}\{\cdot \}$  represents values drawn from a particular data field. We alter the GPT3 templates for these tasks to share similar formats with one of our self-supervised tasks. The red, boldfaced texts are used to compute the language modeling perplexities for ranking the labels. We note that the shown templates are for a single example, and there could be multiple examples within an instance.
(d) CB Template.
Model	BQ	MC	CA	RE	CB	Avg.
LM	52.2	26.6	63.0	50.8	38.3	46.2
SelfSup.	55.7	33.6	67.6	53.0	44.9	51.0
NSG	52.1	25.9	64.0	51.0	41.2	46.9
CL	52.5	26.8	61.4	50.9	48.1	47.9
MWP	51.9	26.3	61.8	50.8	36.1	45.4
LPP	53.5	29.5	61.6	52.0	40.3	47.4
Table 15: SuperGLUE results when training the 125M model with one of the self-supervised tasks.
Model	QG	AG	MM	VF	Avg.
LM	33.7	12.9	53.0	14.7	28.6
SelfSup.	16.9	14.6	70.1	18.9	30.0
NSG	32.3	12.5	54.0	13.8	28.2
CL	8.3	0.3	1.0	2.7	3.1
MWP	15.2	19.4	50.5	17.8	25.7
LPP	11.3	16.6	49.5	19.9	24.3
4. It is interesting to see that among the four self-supervised tasks, NSG and CL tasks are the two most important factors in terms of affecting the downstream performances. This is likely due to (1) the generic task formulation of NSG and it being the only sentence generation tasks; and (2) the drastic differences between CL and the other self-supervised tasks with respect to their inference styles. Unlike NSG/MWP/LPP, which models can rely on input within each example to solve the task, CL require models to make comparisons across examples in a training instance.
Table 16: Natural-Instructions results when training the 125M model with one of the self-supervised tasks.
Model	BQ	MC	CA	RE	CB	Avg.
LM	52.2	26.6	63.0	50.8	38.3	46.2
ALL	55.7	33.6	67.6	53.0	44.9	51.0
ALL-NSG	55.0	31.8	62.7	52.5	45.9	49.6
ALL-MWP	55.6	33.5	67.3	52.7	45.5	50.9
ALL-LPP	53.5	30.5	67.6	51.9	46.6	50.0
ALL-CL	54.0	32.9	67.4	52.8	39.0	49.2
Table 17: SuperGLUE results when excluding one of the self-supervised tasks. The results are based on the 125M model.
Model	QG	AG	MM	VF	Avg.
LM	33.7	12.9	53.0	14.7	28.6
ALL	16.9	14.6	70.1	18.9	30.0
ALL-NSG	10.5	18.7	46.5	14.9	22.7
ALL-MWP	17.2	14.9	67.1	17.9	29.3
ALL-LPP	17.3	14.8	67.6	18.1	29.5
ALL-CL	23.1	15.1	59.0	18.2	28.9
Table 18: Natural-Instructions results when excluding one of the self-supervised tasks. The results are based on the 125M model.