SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
c _ {1} \text {o r} c _ {2}? e, \text {s o} <   \text {M A S K} >.
$$
and a cause  $c$  and possible effects  $e_1$  and  $e_2$ :
$$
\begin{array}{l} \text {" e _ {1}" o r " e _ {2}" ? <   M A S K > , b e c a u s e c .} \end{array}
$$
$$
e _ {1} \text {o r} e _ {2}? <   \text {M A S K} >, \text {b e c a u s e} c.
$$
and the verbalizer is the identity function.
- For MultiRC, given a passage  $p$ , question  $q$  and answer  $a$ , we estimate whether  $a$  is a proper answer with:
$$
p. Q u e s t i o n: q? I s i t a? <   \mathrm {M A S K} >.
$$
$p$ . Based on the previous passage,  $q$ ? Is the correct answer  $a$ ? <MASK>.
$p$  Question:  $q?$  Is the correct answer  $a?$  <MASK>.
- For WiC, given two sentences  $s_1$  and  $s_2$  and a word  $w$ , we classify whether  $w$  was used in the same sense.
$$
" s _ {1}" / " s _ {2}". \text {S i m i l a r s e n s e o f}" w"? <   \text {M A S K >}.
$$
$s_1$ $s_2$  Does  $w$  have the same meaning in both sentences?  $\langle \mathrm{MASK}\rangle$
With "yes" and "no" as verbalizers.
- For WSC, given a sentence  $s$  with a marked pronoun  $*p^*$  and noun  $n$ :
$s$  The pronoun  ${}^{*}p^{*}$  refers to  $<\mathrm{MASK}>$ .
$s$  In the previous sentence, the pronoun  ${}^{*}p^{*}$  refers to  $\langle \mathrm{MASK}\rangle$
s In the passage above, what does the pronoun  ${}^{\prime \ast}p^{\ast}$  refer to? Answer: <MASK>.
With the identity function as a verbalizer.
B Influence of the reporting method over runs
We chose to report the accumulated maximum performance across runs for every model. This means that if the maximum performance over random seeds is smaller than a maximum previously attained with less data points, we use the previous value. This appendix presents results with the maximum and mean at every point to condense several runs instead. Using either maximum is equivalent; using the mean, however, can make results vary significantly, as the distribution of outcomes is heavily left-skewed, or even bimodal, with poor-performance outliers.
Average Advantage (accumulated maximum reporting)
MNLI	BoolQ	CB	COPA	MultiRC*	RTE	WiC	WSC
P vs H	3506 ± 536	752 ± 46	90 ± 2	288 ± 242	384 ± 378	282 ± 34	-424 ± 74	281 ± 137
P vs N	150 ± 252	299 ± 81	78 ± 2	-	74 ± 56	404 ± 68	-354 ± 166	-
N vs H	3355 ± 612	453 ± 90	12 ± 1	-	309 ± 320	-122 ± 62	-70 ± 160	-
Table 2: Average prompting advantage in number of data points for MNLI & SuperGLUE tasks with accmax reporting.  $P$  denotes the prompt model,  $H$  the head model. On average across performance levels, an MNLI prompt model yields the results of an MNLI head model trained with 3500 additional data points. Confidence levels are based on a multiple random runs (see text).  $N$  indicates a null-verbalizer prompting task that replaces the verbalizer with a non-sensical mapping. *The comparison band of MultiRC is too small as the head baseline fails to learn beyond majority class; we use the full region for a lower-bound result.
Average Advantage (maximum reporting)
MNLI	BoolQ	CB	COPA	MultiRC*	RTE	WiC	WSC
P vs H	3506 ± 536	737 ± 53	86 ± 1	226 ± 189	448 ± 314	218 ± 24	-133 ± 65	297 ± 448
P vs N	150 ± 252	249 ± 80	75 ± 3	-	77 ± 64	331 ± 64	-341 ± 187	-
N vs H	3355 ± 612	488 ± 90	12 ± 2	-	371 ± 305	-113 ± 61	209 ± 176	-
Table 3: Average prompting advantage in number of data points for MNLI & SuperGLUE tasks with maximum reporting.
Average Advantage (mean reporting)
MNLI	BoolQ	CB	COPA	MultiRC*	RTE	WiC	WSC
P vs H	4526 ± 266	1139 ± 285	81 ± 2	292 ± 212	399 ± 77	72 ± 71	447 ± 190	-490 ± 402
P vs N	185 ± 307	514 ± 287	80 ± 2	-	58 ± 2	621 ± 215	10 ± 104	-
N vs H	4341 ± 347	625 ± 402	1 ± 2	-	342 ± 78	-549 ± 203	437 ± 189	-
Table 4: Average prompting advantage in number of data points for MNLI & SuperGLUE tasks with mean reporting.
Figure 4: Prompting vs head (classifier) performance across data scales, up to the full dataset, for seven SuperGLUE tasks & MNLI. Compares the best prompt and head performance at each level of training data across 4 runs. Highlighted region shows the accuracy difference of the models. Cross-hatch region highlights the lowest- and highest- accuracy matched region in the curves. The highlighted area in this region is used to estimate the data advantage from prompting.
C Curves on all tasks
Figure 5: Median performance across runs for each prompt on every task.