SlowGuess/NAACL_2021_OCR huggingface dataset:
We investigate the importance of several factors for few-shot performance: the choice of patterns and verbalizers, the usage of both unlabeled and labeled data, and properties of the underlying language model. We also look into our proposed modification for PET to work with multiple masks and compare it to various baselines. Finally, we measure how choosing different sets of training examples affects performance. Our analysis focuses on PET as GPT-3 is not publicly available. $^6$
5.1 Patterns
The way in which tasks are reformulated as cloze questions can have a huge impact on performance (Jiang et al., 2020; Schick and Schütze, 2021). These reformulations can be arbitrarily complex; for example, the pattern used by GPT-3 for WSC contains an introductory section of almost 30 words; it is unclear if and how this formulation has been optimized.7 To investigate the importance
of patterns and verbalizers, we compare three sets of PVPs: our initial set as defined in Section 4.1 (denoted  $\mathbf{p}_{\mathrm{ours}}$ ), the single PVP used by GPT-3 ( $\mathbf{p}_{\mathrm{GPT-3}}$ ), and the combination of both ( $\mathbf{p}_{\mathrm{comb}}$ ).
We train ALBERT using PET with all three sets of patterns; results for selected SuperGLUE tasks are shown in Table 2 (top). As can be seen, the PVP used by GPT-3 outperforms our PVPs on RTE whereas our initial set of patterns performs much better on MultiRC. These large differences in performance highlight the importance of finding good ways to express tasks as cloze questions. As it is difficult to ascertain which patterns perform well without trying them on a large set of examples, a key challenge for few-shot approaches is to compensate for PVPs that the LM fails to understand well. As seen in the performance of the model trained with  $\mathbf{p}_{\mathrm{comb}}$ , PET is able to do so: not only does combining all PVPs compensate for the worse performance of  $\mathbf{p}_{\mathrm{ours}}$  on RTE and of  $\mathbf{p}_{\mathrm{GPT-3}}$  on MultiRC, it even further improves average performance across the three tasks compared to the best-performing set of patterns. This clearly demonstrates the potential of carefully engineering a set of suitable patterns as opposed to just choosing a single formulation without means of evaluating its effectiveness.
5.2 Unlabeled Data Usage
Unlike GPT-3, PET requires unlabeled data to distill the knowledge of all models based on individual PVPs into a single classifier; for iPET, unlabeled data is additionally used to generate training sets for future generations. The underlying assumption
<sup>6</sup>We could not obtain access to OpenAI's GPT-3 API.  
<sup>7</sup>While the authors use a different terminology, GPT-3 also makes use of PVPs (Brown et al., 2020, pp. 50–61).
Model	CB Acc. / F1	RTE Acc.	MultiRC EM / F1a	Avg -
PET (pours)	85.1 / 59.4	69.8	37.9 / 77.3	66.6
PET (pGPT-3)	83.3 / 58.1	71.8	25.4 / 68.3	63.1
PET (pcomb)	84.5 / 59.0	74.7	39.1 / 77.7	68.3
PET (pours) ¬dist	83.9 / 76.2	66.4	38.9 / 76.2	68.0
PET (pcomb) ¬dist	83.9 / 76.2	72.9	39.6 / 76.6	70.4
Figure 4: Average performance (± standard deviation) of all MLMs trained on individual patterns for three generations and of the distilled classifier ("dist.") across three individual training runs
is that unlabeled data can easily be obtained, which may not always be the case in real-world settings. We thus investigate the importance of unlabeled data for regular PET. To this end, we compare the performance of the final classifier in PET to that of directly using the ensemble of models corresponding to individual PVPs. While using this ensemble entirely removes the need for unlabeled data, the ensemble for  $k$  PVPs is larger than the distilled model by a factor of  $3 \cdot k$  as we follow the default setting of PET and train three models per PVP. However, even for a large number of PVPs the ensemble is smaller than GPT-3 by two orders of magnitude.
Results without distillation can be seen in Table 2 (bottom). Averaged across the three tasks, the ensemble performs even better than the distilled classifier. This shows that if the goal is only to achieve good performance, then unlabeled data is not necessary; however, it is required to obtain a single, lightweight model as final classifier.
Figure 4 illustrates the benefit of training multiple generations with iPET. For all tasks except MultiRC, there are substantial improvements from
Table 2: Results on selected tasks for various sets of PVPs for regular PET and for an ensemble of PET models with no knowledge distillation ("-dist")
Model	CB Acc. / F1	RTE Acc.	MultiRC EM / F1a	Avg -
PET	85.1 / 59.4	69.8	37.9 / 77.3	66.6
unsupervised	33.5 / 23.1	55.0	3.9 / 60.3	38.5
supervised	60.7 / 42.5	50.2	4.3 / 49.8	43.0
PET (XLNet)	88.7 / 83.0	60.4	21.4 / 66.6	63.4
Priming (XLNet)	56.3 / 37.7	49.5	- / -	-
Table 3: Results on selected tasks for various ways of using the labeled examples available in FewGLUE
the first to the second generation, whereas the third generation achieves only slight additional improvements. On average, standard deviation is reduced in later generations, illustrating that the models learn from each other and their predictions converge. The final distillation step brings further improvements for all tasks except MultiRC and reduces standard deviation across three training runs to almost zero, illustrating that PET and iPET are effective means of reducing finetuning instability (Dodge et al., 2020).
Of course, there are further ways to leverage unlabeled data such as keeping an auxiliary language modeling objective during finetuning (Chronopoulou et al., 2019). While we leave investigating the impact of additionally using such methods to future work, we note that they can easily be applied to PET while there is no straightforward way to combine them with priming.
5.3 Labeled Data Usage
We next investigate the effect of how labeled data is used, which is one of the key differences between priming and PET. We first compare PET with regular supervised training (i.e., without using any patterns), and with a fully unsupervised model (i.e., an ensemble using all PVPs but no labeled training examples). Given 32 examples, PET clearly outperforms both baselines (Table 3).
We next compare PET directly to priming. However, we cannot do so using ALBERT as it is only able to process sequences of up to 512 tokens, which is not enough for a set of 32 examples; we instead use XLNet (Yang et al., 2019) for this comparison. As shown in Table 3, XLNet in general performs worse than ALBERT. More importantly, XLNet with PET performs much better than priming. We were not able to obtain results with priming on MultiRC because the 32 examples in FewGLUE would require more than 10,000 tokens, so processing them with a standard Transformer (Vaswani
Figure 5: Accuracy differences between priming with 32 examples and one-shot priming for all GPT-3 models as well as between ALBERT with PET (without distillation) and unsupervised ALBERT (bottom row)
et al., 2017) is infeasible due to the quadratic complexity of self-attention. This highlights another important issue with priming: It does not scale well to more than a few examples; even GPT-3 is only able to process sequences of up to 2,048 tokens. While there are some Transformer variants that can deal with much longer contexts (e.g., Kitaev et al., 2020; Beltagy et al., 2020), it has yet to be investigated to what extent such models make good use of priming examples over long context spans.
We further investigate the effectiveness of priming by looking at results obtained with GPT-3 more closely. To this end, Figure 5 shows the performance difference between priming GPT-3 with 32 examples and priming it with just a single example for each task and model size. As can be seen, priming with 32 examples only slightly improves performance for most tasks and model sizes. For some tasks, adding more examples even leads to worse performance, especially for smaller models. For ReCoRD, even the largest model's performance slightly drops when adding more examples.
The bottom row of Figure 5 shows the performance difference between ALBERT trained with PET (without distillation) and a fully unsupervised ALBERT model on all tasks. While results are not directly comparable due to different underlying models and PVPs, PET results in much stronger performance improvements compared to priming and does not worsen results for any task.
Model	Params	CB Acc. / F1	RTE Acc.	MultiRC EM / F1a	Avg -
ALBERT	223M	87.5 / 78.7	74.7	38.9 / 76.2	71.8
RoBERTa	355M	85.7 / 77.5	62.8	23.3 / 70.0	63.7
GPT-2	345M	73.2 / 73.7	47.7	12.4 / 57.4	52.0
Table 4: Results on selected tasks for PET without knowledge distillation combined with various LMs using  $\mathbf{p}_{\mathrm{GPT-3}}$  for CB/RTE and  $\mathbf{p}_{\mathrm{ours}}$  for MultiRC
5.4 Model Type
We next look into the impact of the underlying LM on PET by comparing ALBERT with RoBERTa large (Liu et al., 2019) and GPT-2 medium (Radford et al., 2019). As GPT-2 is a unidirectional model similar to GPT-3, it can only process patterns where the mask token is the very last token. We therefore use  $\mathbf{p}_{\mathrm{GPT - 3}}$  for CB and RTE; for MultiRC, we stick with our original set of patterns as they already fulfill this requirement. We also do not perform distillation and instead report the ensemble's performance as there is no established way of equipping GPT-2 with a sequence classification head.
Results for training all three LMs with PET in Table 4 show that using ALBERT as underlying LM is crucial for PET's strong performance; exchanging ALBERT with RoBERTa results in an average performance drop of 8 points. However, RoBERTa still clearly outperforms GPT-3 13B, which is larger by two orders of magnitude. Importantly, PET with GPT-2 performs much worse than with the two other models. As anticipated by Brown et al. (2020), a reason for this drop in performance may be that like GPT-3, GPT-2 is unidirectional, making tasks that require comparing two sequences a challenge. However, it is important to note that there are also other substantial differences between GPT-2 and the other two models, most notably the pretraining dataset. Regardless of whether unidirectionality is the reason for GPT-2's bad performance, bidirectionality of the underlying LM is important for PET as it removes the need for the mask token to be at the very end and thus allows for more flexibility in the creation of patterns.
5.5 PET with Multiple Masks
We modified PET to work for outputs that require more than a single token. To investigate the impact of this modification, we look at the three tasks for which this is required: COPA, WSC and ReCoRD. We compare our decoding strategy of predicting to
<sup>8</sup>We do not compare priming to zero-shot performance as for unknown reasons, zero-shot GPT-3 performs well below random guessing for some tasks (e.g., 0.0% accuracy for WiC). To not overestimate the benefit of priming, we therefore show gains from providing 32 examples compared to just one.
Model	COPA Acc.	WSC Acc.	ReCoRD Acc. / F1	Avg -
PET	95.0	80.1	86.0 / 86.5	87.1
PET ¬dist (max-first)	90.0	80.8	86.0 / 86.5	85.7
PET ¬dist (ltr)	89.0	79.8	84.7 / 85.3	84.6
PET ¬dist (parallel)	77.0	80.8	82.5 / 83.1	80.2
untrained	72.5	59.9	84.7 / 85.4	72.5
kens in order of the probability assigned to them, to which we refer as max-first, with two alternatives: decoding left-to-right (ltr) as is common for many autoregressive language models, and decoding all tokens simultaneously (parallel) as is done during training. Additionally, we compare PET with untrained ALBERT to measure the effectiveness of our proposed training loss.
Results are shown in Table 5. PET clearly outperforms untrained ALBERT for the three tasks. Not performing distillation hurts performance for COPA, but leads to slight improvements on WSC; for ReCoRD, we did not perform distillation in the first place as we only use a single PVP. Our decoding strategy is clearly superior to parallel decoding except for WSC, for which most predictions consist only of one or two tokens, and performs slightly better than left-to-right decoding.
5.6 Training Examples
Recall that we conduct our experiments with training examples from FewGLUE, a randomly selected subset of the original SuperGLUE training examples. We used a fixed random seed  $s_0$  to generate FewGLUE. Let  $\Sigma_i$  be the randomly selected subset of SuperGLUE for random seed  $s_i$ , so  $\Sigma_0 =$  FewGLUE. In this subsection, we create two additional subsets of SuperGLUE,  $\Sigma_1$  and  $\Sigma_2$ , based on different seeds. This allows us to investigate how different sets of training examples affect performance. To this end, we run PET for CB, RTE and MultiRC using the three  $\Sigma_i$ . To measure only the effect of varying the training set while ignoring unlabeled examples, we do not use distillation.
Table 6 shows that for all tasks, changing the set of training examples can result in large performance differences for PET. This highlights the importance of using the same set of examples when comparing different few-shot approaches, which is why we make the particular set of examples in FewGLUE publicly available. However, we note
Table 5: Results on selected tasks for our proposed variant of PET as well as other decoding strategies and for untrained ALBERT
Model	CB Acc. / F1	RTE Acc.	MultiRC EM / F1a	Avg -
GPT-3	82.1 / 57.2	72.9	32.5 / 74.8	65.4
PET ¬dist (Σ0)	83.9 / 76.2	66.4	38.9 / 76.2	68.0
PET ¬dist (Σ1)	82.1 / 57.4	61.4	39.2 / 77.9	63.2
PET ¬dist (Σ2)	87.5 / 84.0	61.4	34.7 / 76.3	67.6
Table 6: Results on selected tasks for GPT-3 and for PET using training sets  $\Sigma_0, \Sigma_1, \Sigma_2$
that the average performance of PET is similar to that of GPT-3 for all seeds.
While our results may seem contrary to the insight that for GPT-3, the exact choice of examples does not play a major role, we suspect this to be due to the fact that priming benefits much less from training examples than PET (cf. Section 5.3); accordingly, the influence of the exact set of training examples on the model's performance is smaller.
We have proposed a simple yet effective modification of PET, enabling us to use it for tasks that require predicting multiple tokens. In extensive experiments, we have identified several factors responsible for the strong performance of PET combined with ALBERT: the possibility to concurrently use multiple patterns for transforming examples into cloze questions, the ability to compensate for patterns that are difficult to understand, the usage of labeled data to perform parameter updates, and the underlying LM itself.
We have shown that using PET, it is possible to achieve few-shot text classification performance similar to GPT-3 on SuperGLUE with LMs that have three orders of magnitude fewer parameters. This not only lowers financial cost, but above all reduces environmental impact immensely and leads to a much smaller carbon footprint. We see this as an important contribution to achieving the goal of an environmentally more friendly NLP. To enable comparisons with our work, we make our code, models and datasets publicly available.
For future work, it would be interesting to see whether PET also works for generative tasks when combined with generative LMs and whether further improvements are possible in multi-task settings.
Acknowledgments This work was funded by the European Research Council (ERC #740516). We thank the anonymous reviewers for their helpful comments.
Mark Anderson and Carlos Gómez-Rodríguez. 2020. Distilling neural networks for greener and faster dependency parsing. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, pages 2-13, Online. Association for Computational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. Computing Research Repository, arXiv:2004.05150.
Sergey Brin. 1999. Extracting patterns and relations from the world wide web. In *The World Wide Web and Databases*, pages 172–183, Berlin, Heidelberg. Springer Berlin Heidelberg.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. 2019. An embarrassingly simple approach for transfer learning from pretrained language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2089-2095, Minneapolis, Minnesota. Association for Computational Linguistics.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. *BoolQ: Exploring the surprising difficulty of natural yes/no questions.* In *Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies*, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising textual entailment, pages 177-190. Springer.
Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173-1178, Hong Kong, China. Association for Computational Linguistics.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank: Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung 23.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. Computing Research Repository, arXiv:2002.06305.
Urun Dogan, Tobias Glasmachers, and Christian Igel. 2016. A unified view on multi-class support vector classification. J. Mach. Learn. Res., 17(45):1-32.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6121, Hong Kong, China. Association for Computational Linguistics.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. Computing Research Repository, arXiv:1412.6115.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394-398, Montréal, Canada. Association for Computational Linguistics.
Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural net
works with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR).
Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Computing Research Repository, arXiv:1503.02531.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online. Association for Computational Linguistics.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262, New Orleans, Louisiana. Association for Computational Linguistics.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In International Conference on Learning Representations.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations.
Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, page 47.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020. FastBERT: a self-distilling BERT with adaptive inference time. In