SlowGuess/NAACL_2021_OCR huggingface dataset:
Timo Schick<sup>1,2</sup> and Hinrich Schütze<sup>1</sup>
<sup>1</sup> Center for Information and Language Processing, LMU Munich, Germany  
<sup>2</sup> Sulzer GmbH, Munich, Germany
When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much "greener" in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.
Pretraining ever-larger language models (LMs) on massive corpora has led to large improvements in NLP (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020, i.a.). A standard approach is to replace the pretrained model's output layer with a task-specific head and finetune the entire model on a set of labeled training data. However, language modeling is not only a powerful pretraining objective, but many tasks can be reformulated as cloze questions (e.g., by appending phrases such as "the correct answer is _"), allowing pretrained LMs to solve them without any or with only very few labeled examples (Radford et al., 2019; Schick and Schütze, 2021).
Recently, Brown et al. (2020) introduced GPT-3, a pretrained LM with an enormous 175 billion parameters, and showed that it has amazing few-shot abilities: By reformulating tasks as LM problems,
Figure 1: Performance on SuperGLUE with 32 training examples. ALBERT with PET/iPET outperforms GPT-3 although it is much "greener" in that it has three orders of magnitude fewer parameters.
GPT-3 achieves near state-of-the-art results for some SuperGLUE (Wang et al., 2019) tasks given just 32 labeled examples. This is achieved through priming: GPT-3 is given a few demonstrations of inputs and corresponding outputs as context for its predictions, but no gradient updates are performed. While being straightforward to use, this method has two major drawbacks:
- It requires a gigantic LM to work well, making it unusable in many real-world scenarios and resulting in a large carbon footprint (Strubell et al., 2019).
- It does not scale to more than a few examples as the context window of most LMs is limited to a few hundred tokens.2
An alternative to priming is pattern-exploiting training (PET) (Schick and Schütze, 2021), which combines the idea of reformulating tasks as cloze questions with regular gradient-based finetuning. While PET additionally requires unlabeled data, unlabeled data is much easier to obtain than labeled
2While GPT-3 can process up to 2,048 tokens, this is still not enough to fit  $\geq 32$  examples for some SuperGLUE tasks.
1Our implementation is publicly available at https:// github.com/timoschick/pet.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352 June 6-11, 2021. ©2021 Association for Computational Linguistics
examples for many real-world applications. Crucially, PET only works when the answers to be predicted by the LM correspond to a single token in its vocabulary; this is a severe limitation as many tasks cannot easily be worded that way.
In this work, we adapt PET for tasks that require predicting multiple tokens. We then show that in combination with ALBERT (Lan et al., 2020), PET and its iterative variant (iPET) both outperform GPT-3 on SuperGLUE with 32 training examples, while requiring only  $0.1\%$  of its parameters (Figure 1). Moreover, training with PET can be performed in several hours on a single GPU without requiring expensive hyperparameter optimization. Finally, we show that similar performance can also be achieved without unlabeled data and provide a detailed analysis of the factors contributing to PET's strong performance: its ability to combine multiple task formulations, its resilience to wordings that are hard to understand, its usage of labeled data, and characteristics of the underlying LM. Given PET's "green" properties, we see our work as an important contribution to an environmentally sound NLP.
Enabling LMs to perform zero-shot learning by providing task descriptions was proposed by Radford et al. (2019) and has been applied to text classification (Puri and Catanzaro, 2019), commonsense knowledge mining (Davison et al., 2019) and argumentative relation classification (Opitz, 2019). It is also commonly used for probing the knowledge contained within LMs (Trinh and Le, 2018; Petroni et al., 2019; Talmor et al., 2020; Schick and Schütze, 2020; Ettinger, 2020, i.a.).
As finding ways to reformulate tasks as cloze questions that are understood well by LMs is difficult (Jiang et al., 2020), Schick and Schütze (2021) propose PET, a method that uses knowledge distillation (Hinton et al., 2015) and self-training (e.g., Scudder, 1965; Yarowsky, 1995; Brin, 1999; McClosky et al., 2006) to easily combine several reformulations. Our modified version of PET uses masked language models (Devlin et al., 2019) to assign probabilities to sequences of text; this is similar to using them in a generative fashion (Wang and Cho, 2019) and has previously been investigated by Salazar et al. (2020) and Ghazvininejad et al. (2019). In contrast to PET, which uses gradient-based optimization, Radford et al. (2019)
Figure 2: Application of a PVP  $\mathbf{p} = (P,v)$  for recognizing textual entailment: An input  $x = (x_{1},x_{2})$  is converted into a cloze question  $P(x);q_{\mathbf{p}}(y\mid x)$  for each  $y$  is derived from the probability of  $v(y)$  being a plausible choice for the masked position.
and Brown et al. (2020) investigate priming, where examples are given as context but no parameter updates are performed.
Finally, our focus on reducing the amount of compute required for few-shot learning is closely related to other efforts in Green AI (Schwartz et al., 2020a) that aim to improve model efficiency, including techniques for knowledge distillation (e.g., Hinton et al., 2015; Sanh et al., 2019; Jiao et al., 2020; Mao et al., 2020; Anderson and Gomez-Rodriguez, 2020), pruning (Han et al., 2015, 2016; Sanh et al., 2020) and quantization (Gong et al., 2014; Zafrir et al., 2019; Stock et al., 2021) as well as early exit strategies for inference (Liu et al., 2020; Schwartz et al., 2020b; Xin et al., 2020).
3 Pattern-Exploiting Training
Let  $M$  be a masked language model (MLM),  $T$  its vocabulary and  $\_ \in T$  the mask token; we denote the set of all token sequences as  $T^{*}$ . For some  $\mathbf{z} \in T^{*}$  containing at least  $k$  masks and  $t \in T$ , we denote with  $q_{M}^{k}(t \mid \mathbf{z})$  the probability that  $M$  assigns to  $t$  at the  $k$ th masked position in  $\mathbf{z}$ ; the model's logits before applying softmax are denoted with  $s_{M}^{k}(t \mid \mathbf{z})$ . We consider the task of mapping inputs  $x \in X$  to outputs  $y \in Y$ , for which PET requires a set of pattern-verbalizer pairs (PVPs). Each PVP  $\mathbf{p} = (P, v)$  consists of
- a pattern  $P: X \to T^*$  that maps inputs to cloze questions containing a single mask;
- a verbalizer  $v: Y \to T$  that maps each output to a single token representing its task-specific meaning in the pattern.
As illustrated in Figure 2, the core idea of PET is to derive the probability of  $y$  being the correct output for  $x$  from the probability of  $v(y)$  being
the "correct" token at the masked position in  $P(x)$ . Based on this intuition, a conditional probability distribution  $q_{\mathbf{p}}$  of  $y$  given  $x$  is defined as
$$
q _ {\mathbf {p}} (y \mid x) = \frac {\exp s _ {\mathbf {p}} (y \mid x)}{\sum_ {y ^ {\prime} \in Y} \exp s _ {\mathbf {p}} (y ^ {\prime} \mid x)} \tag {1}
$$
where  $s_{\mathbf{p}}(y \mid x) = s_M^1(v(y) \mid P(x))$  is the raw score of  $v(y)$  at the masked position in  $P(x)$ .
For a given task, identifying PVPs that perform well is challenging in the absence of a large development set. Therefore, PET enables a combination of multiple PVPs  $\mathbf{P} = \{\mathbf{p}_1,\dots ,\mathbf{p}_n\}$  as follows:
1. For each PVP  $\mathbf{p}$ , a MLM is finetuned on training examples  $(x, y)$  by minimizing the cross entropy between  $y$  and  $q_{\mathbf{p}}(y \mid x)$ . In practice, Schick and Schütze (2021) train three MLMs per pattern as performance can vary substantially between runs.
2. The ensemble of finetuned MLMs is used to annotate a set of unlabeled examples; each unlabeled example  $x \in X$  is annotated with soft labels based on the probability distribution
$$
q _ {\mathbf {P}} (y \mid x) \propto \exp \sum_ {\mathbf {p} \in \mathbf {P}} w _ {\mathbf {p}} \cdot s _ {\mathbf {p}} (y \mid x) \tag {2}
$$
similar to Eq. 1 where  $w_{\mathbf{p}}$  is a weighting term that is proportional to the accuracy achieved with  $\mathbf{p}$  on the training set before training.
3. The resulting soft-labeled dataset is used to train a regular sequence classifier by minimizing cross entropy between its output and  $q_{\mathbf{P}}$ .
As steps (2) and (3) above closely resemble knowledge distillation (Hinton et al., 2015), we also refer to them simply as distillation. Importantly, this process does not require holding the entire ensemble of MLMs in memory at the same time as each model's predictions can be computed sequentially; therefore, it is not more memory expensive than using a single model.
To give MLMs trained on different patterns further opportunity to learn from one another, Schick and Schütze (2021) also propose iPET, an iterative variant of PET in which several generations of models are trained on datasets of increasing size that are labeled by previous generations. This is achieved as follows: First, an ensemble of MLMs is trained as in regular PET. For each model  $M_{i}$ , a random subset of other models is used to generate
$$
\begin{array}{l} (a) \quad \mathbf {z} = \boxed {\underbrace {\text {A w f u l p i z z a !}} _ {x} \text {I t w a s} \quad .} \\ q _ {M} ^ {1} (\mathbf {t e r r i} | \mathbf {z}) <   q _ {M} ^ {2} (\bullet \mathbf {b l e} | \mathbf {z}) \\ \end{array}
$$
$$
\begin{array}{c} (\mathbf {b}) \quad \mathbf {z} ^ {\prime} = \boxed {\boxed {\text {A w f u l p i z z a !}} \text {I t w a s} \underset {x} {\longrightarrow} \bullet \text {b l e .}} \\ \vdots \\ q _ {M} ^ {1} (\text {t e r r i} | \mathbf {z} ^ {\prime}) \end{array}
$$
Figure 3: Inference for a verbalization consisting of the two tokens terri and •ble. (a) We first compute the probability of each token at its position in the cloze question  $P^2(x)$  and identify the token with the highest probability. (b) We insert this token into the cloze question and compute the probability of the remaining token.
a new training set  $T_{i}$  by assigning labels to those unlabeled examples for which the selected subset of models is most confident in its prediction. Each  $M_{i}$  is then retrained on  $T_{i}$ ; this process is repeated several times, each time increasing the number of examples in  $T_{i}$  by a constant factor. For further details, we refer to Schick and Schütze (2021).
3.1 PET with Multiple Masks
An important limitation of PET is that the verbalizer  $v$  must map each output to a single token, which is impossible for many tasks. We thus generalize verbalizers to functions  $v: Y \to T^{*}$ ; this requires some modifications to inference and training.<sup>3</sup> We further generalize PET in that we do not assume the output space to be identical for each input: for each  $x \in X$ , we denote with  $Y_{x} \subseteq Y$  the set of possible outputs given  $x$  as input. Given a PVP  $\mathbf{p} = (P, v)$ , we define  $l(x) = \max_{y \in Y_{x}} |v(y)|$  to be the maximum number of tokens required to express any output in  $Y_{x}$  and  $P^{k}(x)$  to be  $P(x)$  with the mask token replaced by  $k$  masks.
As a running example, we consider the task of binary sentiment classification for restaurant reviews with labels  $Y = \{+1, -1\}$ . We use the pattern  $P(x) = x$ . It was ____ and a verbalizer  $v$  that maps +1 to the single token great and -1 to the sequence terri •ble, i.e., we assume that the MLM's tokenizer splits the word "terrible" into the two tokens terri and •ble. For this example,  $l(x) = 2$  for all  $x$ ;  $P^2(x)$  is illustrated in Figure 3 (a).
3While PET can easily be adapted to generative MLMs (e.g., Lewis et al., 2020; Raffel et al., 2020), we stick with regular MLMs as they are more lightweight and performed better on simple cloze tasks in preliminary experiments.
Inference For  $x \in X$ ,  $y \in Y_x$  and  $|v(y)| = k$ , we redefine  $q_{\mathbf{p}}(y \mid x)$  in an autoregressive fashion: Starting from  $P^k(x)$ , we perform  $k$  consecutive predictions, where we always select the next token to predict based on the MLM's confidence. That is, we set  $q_{\mathbf{p}}(y \mid x) = q(v(y) \mid P^k(x))$  where
$$
q \left(t _ {1} \dots t _ {k} \mid \mathbf {z}\right) = \left\{ \begin{array}{l l} 1 & \text {i f} k = 0 \\ q _ {M} ^ {j} \left(t _ {j} \mid \mathbf {z}\right) \cdot q \left(t ^ {\prime} \mid \mathbf {z} ^ {\prime}\right) & \text {i f} k \geq 1 \end{array} \right. \tag {3}
$$
with  $j = \arg \max_{i=1}^{k} q_{M}^{i}(t_{i} \mid \mathbf{z})$ ,  $\mathbf{z}'$  is  $\mathbf{z}$  except  $\mathbf{z}_{j}' = t_{j}$  and  $t' = t_{1} \dots t_{j-1} t_{j+1} \dots t_{k}$ . Note that unlike in original PET (Eq. 1),  $q_{\mathbf{p}}$  is not a probability distribution as its values do not sum to one.
For our sentiment classification example, Figure 3 illustrates how  $q_{\mathbf{p}}(-1 \mid x)$  is computed: As  $|v(y)| = |\{\text{terri}, \bullet \text{ble}\}| = 2$ , we first use  $\mathbf{z} = P^2(x)$  to compute the probability of each token in  $v(y)$  (Figure 3a). We then choose the token with the highest probability, put it in place of the corresponding mask token, and use the resulting cloze question  $\mathbf{z}'$  to compute the probability of the remaining token (Figure 3b). The overall score for  $y = -1$  is then computed as
$$
q _ {\mathbf {p}} (- 1 \mid x) = q _ {M} ^ {2} (\bullet \mathrm {b l e} \mid \mathbf {z}) \cdot q _ {M} ^ {1} (\mathrm {t e r r i} \mid \mathbf {z} ^ {\prime})
$$
Training Computing  $q_{\mathbf{p}}(y \mid x)$  as in Eq. 3 for each training example  $(x, y)$  would be prohibitively expensive. To enable computation of all required probabilities in a single forward pass, we approximate  $q_{\mathbf{p}}(y \mid x)$  by (i) always inserting the maximum number of mask tokens required to express any output and (ii) for each  $y' \in Y_x$ , predicting all tokens in  $v(y') = t_1 \ldots t_k$  in parallel, where we simply ignore the model's predictions for all  $l(x) - k$  superfluous mask tokens:
$$
\tilde {q} _ {\mathbf {p}} \left(y ^ {\prime} \mid x\right) = \prod_ {i = 1} ^ {k} q _ {M} ^ {i} \left(t _ {i} \mid P ^ {l (x)} (x)\right) \tag {4}
$$
For our running example, this means we approximate the scores  $q_{\mathbf{p}}(y \mid x)$  by computing
$$
\tilde {q} _ {\mathbf {p}} (+ 1 \mid x) = q _ {M} ^ {1} (\operatorname {g r e a t} \mid \mathbf {z})
$$
$$
\tilde {q} _ {\mathbf {p}} (- 1 \mid x) = q _ {M} ^ {1} (\text {t e r r i} \mid \mathbf {z}) \cdot q _ {M} ^ {2} (\bullet \text {b l e} \mid \mathbf {z})
$$
which can be done in a single forward pass as it only requires processing the cloze question  $\mathbf{z} = P^2(x)$  shown in Figure 3 (a) once.
As  $\tilde{q}_{\mathbf{p}}$  is not a probability distribution over  $Y_{x}$ , cross entropy is not an ideal training objective as it
can also be minimized by reducing the probability assigned to sequences  $\mathbf{z} \notin v(Y_x)$  that are not part of the output space, despite this having no effect on the model's prediction. We instead opt for multiclass hinge loss (Weston and Watkins, 1999; Dogan et al., 2016) and minimize:
$$
\sum_ {y ^ {\prime} \in Y _ {x}} \max  \left(0; 1 - \log \tilde {q} _ {\mathbf {p}} (y | x) + \log \tilde {q} _ {\mathbf {p}} \left(y ^ {\prime} | x\right)\right) \tag {5}
$$
That is, we require the difference between the log probability of  $y$  and the log probability of any output  $y' \in Y_x \setminus \{y\}$  to be at least 1.
We compare PET and GPT-3 on SuperGLUE (Wang et al., 2019), a natural language understanding benchmark consisting of eight challenging tasks. We cannot evaluate PET using the exact same training data as GPT-3 because for most tasks, GPT-3 uses a different set of training examples for each test example and for the other tasks, training sets were not available upon request; however, the exact choice of examples has little impact on GPT-3's performance.4 We thus create new training sets by randomly selecting 32 examples for each task using a fixed random seed.
We additionally create sets of up to 20,000 unlabeled examples for each task; this is done by removing all labels from the original training sets. We refer to the resulting sets of training examples and unlabeled examples as FewGLUE.<sup>5</sup>
Below, we describe each of the SuperGLUE tasks and our corresponding PVPs. We use a vertical bar (|) to mark boundaries between text segments. Of the eight tasks considered, only COPA, WSC and ReCoRD require the use of PET with multiple masks as introduced in Section 3.1.
BoolQ (Clark et al., 2019) is a QA task where each example consists of a passage  $p$  and a yes/no question  $q$ . We use the following patterns:
p. Question: q? Answer:
-  $p$  . Based on the previous passage,  $q$  ? _____.
- Based on the following passage,  $q?$  _____.  $p$
Based on personal correspondence with the authors.
FewGLUE is publicly available at https://github. com/timoschick/fewglue.
We define two verbalizers mapping questions containing a true statement to yes/true and others to no/false, respectively, for a total of 6 PVPs.
CB (De Marneffé et al., 2019) and RTE (Dagan et al., 2006) are textual entailment tasks like MNLI, so we use PVPs similar to Schick and Schütze (2021). For a premise  $p$  and hypothesis  $h$ , we use
$$
h? | \_, p, \text {" h "} ? | \_, \text {" p "} , h? | \_, p , \text {" h "} ? | \_, \text {" p "}
$$
and a verbalizer that maps entailment to yes, disagreement to no and neutral to maybe.
Given a premise  $p$ , the task in COPA (Gordon et al., 2012) is to determine the cause or effect of the premise given two options  $c_{1}$  and  $c_{2}$ . For determining the effect, we use the following patterns:
$$
" c _ {1}" \text {o r} " c _ {2}"? p, \text {s o \_}. c _ {1} \text {o r} c _ {2}? p, \text {s o \_}.
$$
For determining the cause, we use the same patterns but replace so with because. The verbalizer for  $c_{1}$  and  $c_{2}$  is the identity function.
For WiC (Pilehvar and Camacho-Collados, 2019), given a word  $w$  and two sentences  $s_1$  and  $s_2$  in which it occurs, the task is to decide if  $w$  is used with the same sense in both sentences. We use:
- "s1"/"s2". Similar sense of "w"? _.
-  $s_1 s_2$  Does  $w$  have the same meaning in both sentences?
w. Sense (1) (a) “ $s_1$ ” (“ $s_2$ ”)
For the first two patterns, we use yes as verbalization for words used in the same sense and no for other words; for the third pattern, we use b and 2.
For WSC (Levesque et al., 2011), each example consists of a sentence  $s$  with a marked pronoun  $p$  and noun  $n$ , and the task is to determine whether  $p$  refers to  $n$ . We follow (Raffel et al., 2020; Brown et al., 2020) and treat WSC as a generative task. We highlight  $p$  in  $s$  by putting it in asterisks and use the following patterns:
-  $s$  The pronoun  $*p*$  refers to _____.
-  $s$  In the previous sentence, the pronoun ' $*p$ *' refers to _.
s In the passage above, what does the pronoun
$^{*}p^{*}$  refer to? Answer:
We use the identity function as verbalizer for  $n$ . Note that WSC is different from other tasks in that it requires free-form completion. This in
turn requires some modifications during training and inference that are discussed in Appendix A.
MultiRC (Khashabi et al., 2018) is a QA task. Given a passage  $p$ , a question  $q$  and an answer candidate  $a$ , the task is to decide whether  $a$  is a correct answer for  $q$ . We use the same verbalizer as for BoolQ and similar patterns:
-  $p$ . Question:  $q$ ? Is it  $a$ ?
-  $p$ . Question:  $q$ ? Is the correct answer "  $a$ ?" _____.
-  $p$ . Based on the previous passage,  $q$ ? Is "a" a correct answer? _____.
For ReCoRD (Zhang et al., 2018), given a passage  $p$  and a cloze question  $q$ , the task is to decide which of a given set of answer candidates is the correct replacement for the placeholder in the cloze question. As this task is already presented in the form of a cloze question, there is little room for designing PVPs, so we only use a trivial one: the concatenation of  $p$  and  $q$  as pattern and the identity function as verbalizer. With only one PVP, there is no need to perform knowledge distillation so we directly use the resulting model as our final classifier.
As underlying LM for PET we choose ALBERT-xxlarge-v2 (Lan et al., 2020), the best-performing MLM on SuperGLUE when training is performed on the regular, full size training sets. We use the same model, supplemented by a sequence classification head, as our final classifier. We run PET on the FewGLUE training sets for all SuperGLUE tasks. We do not use any development set to optimize hyperparameters; instead we use the exact same setup and hyperparameters as Schick and Schütze (2021). For COPA, WSC and ReCoRD, we use our proposed modification of PET to support verbalizers mapping labels to multiple tokens; for all other tasks, we use regular PET. We train iPET on all tasks except COPA and WSC, as their unlabeled sets contain well below 1,000 examples, as well as ReCoRD, for which iPET makes no sense as we only use a single PVP. For these three tasks, we simply reuse the results of regular PET.
Our main results are shown in Table 1. As can be seen, ALBERT with PET performs similar to the largest GPT-3 model, which is larger by a factor
Model	Params (M)	BoolQ Acc.	CB Acc. / F1	COPA Acc.	RTE Acc.	WiC Acc.	WSC Acc.	MultiRC EM / F1a	ReCoRD Acc. / F1	Avg -
dev	GPT-3 Small	125	43.1	42.9 / 26.1	67.0	52.3	49.8	58.7	6.1 / 45.0	69.8 / 70.7	50.1
GPT-3 Med	350	60.6	58.9 / 40.4	64.0	48.4	55.0	60.6	11.8 / 55.9	77.2 / 77.9	56.2
GPT-3 Large	760	62.0	53.6 / 32.6	72.0	46.9	53.0	54.8	16.8 / 64.2	81.3 / 82.1	56.8
GPT-3 XL	1,300	64.1	69.6 / 48.3	77.0	50.9	53.0	49.0	20.8 / 65.4	83.1 / 84.0	60.0
GPT-3 2.7B	2,700	70.3	67.9 / 45.7	83.0	56.3	51.6	62.5	24.7 / 69.5	86.6 / 87.5	64.3
GPT-3 6.7B	6,700	70.0	60.7 / 44.6	83.0	49.5	53.1	67.3	23.8 / 66.4	87.9 / 88.8	63.6
GPT-3 13B	13,000	70.2	66.1 / 46.0	86.0	60.6	51.1	75.0	25.0 / 69.3	88.9 / 89.8	66.9
GPT-3	175,000	77.5	82.1 / 57.2	92.0	72.9	55.3	75.0	32.5 / 74.8	89.0 / 90.1	73.2
PET	223	79.4	85.1 / 59.4	95.0	69.8	52.4	80.1	37.9 / 77.3	86.0 / 86.5	74.1
iPET	223	80.6	92.9 / 92.4	95.0	74.0	52.2	80.1	33.0 / 74.0	86.0 / 86.5	76.8
test	GPT-3	175,000	76.4	75.6 / 52.0	92.0	69.0	49.4	80.1	30.5 / 75.4	90.2 / 91.1	71.8
PET	223	79.1	87.2 / 60.2	90.8	67.2	50.7	88.4	36.4 / 76.6	85.4 / 85.9	74.0
iPET	223	81.2	88.8 / 79.9	90.8	70.8	49.3	88.4	31.7 / 74.1	85.4 / 85.9	75.4
SotA	11,000	91.2	93.9 / 96.8	94.8	92.5	76.9	93.8	88.1 / 63.3	94.1 / 93.4	89.3
Table 1: Results on SuperGLUE for GPT-3 primed with 32 randomly selected examples and for PET / iPET with ALBERT-xxlarge-v2 after training on FewGLUE. State-of-the-art results when using the regular, full size training sets for all tasks (Raffel et al., 2020) are shown in italics.
of 785. On average, PET performs 18 points better compared to GPT-3 Med, a model of similar size. iPET brings further improvements for 3 out of the 5 tasks that we use iPET for, most notably for CB, but results in a slight performance drop for MultiRC. Despite PET's strong performance, it still clearly performs worse than a state-of-the-art model trained on the regular, full size SuperGLUE training set.