SlowGuess/NAACL_2022_OCR huggingface dataset:
Albert Webson<sup>1,2</sup> and Ellie Pavlick<sup>1</sup>
{albert_webson, ellie_pavlick}  $@$  brown.edu
$^{1}$ Department of Computer Science, Brown University
$^{2}$ Department of Philosophy, Brown University
Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. It is commonly argued that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. In this study, we experiment with over 30 prompt templates manually written for natural language inference (NLI). We find that models can learn just as fast with many prompts that are intentionally irrelevant or even pathologically misleading as they do with instructively "good" prompts. Further, such patterns hold even for models as large as 175 billion parameters (Brown et al., 2020) as well as the recently proposed instruction-tuned models which are trained on hundreds of prompts (Sanh et al., 2021). That is, instruction-tuned models often produce good predictions with irrelevant and misleading prompts even at zero shots. In sum, notwithstanding prompt-based models' impressive improvement, we find evidence of serious limitations that question the degree to which such improvement is derived from models understanding task instructions in ways analogous to humans' use of task instructions.
Suppose a human is given two sentences: "No weapons of mass destruction found in Iraq yet." and "Weapons of mass destruction found in Iraq." They are then asked to respond 0 or 1 and receive a reward if they are correct. In this setup, they would likely need a large number of trials and errors before figuring out what they are really being rewarded to do. This setup is akin to the pretrain-and-fine-tune setup which has dominated NLP in recent years, in which models are asked to classify a sentence representation (e.g., a CLS token) into some
arbitrary dimensions of a one-hot vector. In contrast, suppose a human is given a prompt such as: Given that "no weapons of mass destruction found in Iraq yet.", is it definitely correct that "weapons of mass destruction found in Iraq."?1 Then it would be no surprise that they are able to perform the task more accurately and without needing many examples to figure out what the task is.
Similarly, reformating NLP tasks with prompts such as the underlined text above has dramatically improved zero-shot and few-shot performance over traditional fine-tuned models (Schick and Schütze, 2021b; Le Scao and Rush, 2021; Sanh et al., 2021; Wei et al., 2021). Such results naturally give rise to the hypothesis that the extra prompt text included within each input example serves as semantically meaningful task instructions which help models to learn faster, in the way task instructions help humans to learn faster. This hypothesis is implicitly assumed by many and explicitly argued by Mishra et al. (2021), Schick and Schütze (2021a), and Brown et al. (2020).
While last years saw a gold rush of papers (summarized in §2) that proposed automatic methods for optimizing prompts, Logan IV et al. (2021) compare a representative sample of these newly proposed methods and report that Schick and Schütze (2021b)'s manually written prompts still on average outperform the automatically searched prompts across a range of SuperGLUE tasks (Wang et al., 2019). Such findings suggest that expert-crafted prompts are among the best, if not the best, which reinforces the above hypothesis that models benefit from meaningful instructions.
In this paper, we test this hypothesis by evaluating various models on NLI in zero-shot and few-shot settings using more than 30 manually written templates and 13 sets of LM target words for a
*Unabridged version available on arXiv. Code, interactive figures, and statistical test results available at https://github.com/awebson/prompt_semantics
This prompt is adapted from MultiNLI (Williams et al., 2018, p. 3)'s instructions to crowdsourced workers, while the example is the first one in RTE's training set.
Human Language Technologies, pages 2300 - 2344
total of over 390 prompts. We find that in most cases models learn identically as fast when given irrelevant or misleading templates as they do when given instructively good templates. Further, models ranging from 235 million to 175 billion parameters all exhibit this behavior, as do the instruction-tuned models, which are trained on hundreds of manually written prompts. While we confirm Sanh et al. (2021)'s finding that instruction tuning substantially improves the performance and robustness of prompts, we also find that instruction-tuned models can be, in some sense, too robust and less sensitive to the semantics of the prompts, as compared to their non-instruction-tuned equivalents. Finally, models are much more sensitive to the choice of the LM target words as opposed to the meaning of the instruction templates. In sum, despite prompt-based models' dramatic improvement in zero-shot and few-shot learning, we find limited evidence that models' improvement is derived from models understanding task instructions in ways analogous to humans' use of task instructions.
2.1 Prompt-Based Models
At the time of writing, the terms "prompt tuning" and "prompting" can refer to any one or combination of three approaches described below:
Discrete Prompts reformat each example with some template text. For example, in a sentiment analysis task, the template can be {sent} In summary, the restaurant is [prediction], where the predicted mask word is then converted to a class prediction by a predefined mapping, e.g., {"great"  $\rightarrow$  positive, "terrible"  $\rightarrow$  negative}. The prompts can be manually written (Schick and Schütze, 2021a; Bragg et al., 2021) or automatically generated (Gao et al., 2021b; Shin et al., 2020). This approach typically tunes all parameters of the model, but its few-shot performance can exceed that of very large models (e.g., GPT-3 175B) despite using a 3 orders of magnitude smaller LM (Schick and Schütze, 2021b; Tam et al., 2021).
Priming (a.k.a. in-context learning) prepends  $k$  priming examples to the evaluation example, where each example is optionally wrapped in a template such as Question:  $\{\mathrm{sent}_1\}$  True or false?  $\{\mathrm{label}_1\}$  ... Question:  $\{\mathrm{sent}_k\}$  True or false?  $\{\mathrm{label}_k\}$  Question:  $\{\mathrm{eval\_sent}\}$  True or
false? [prediction]. Notably, although models see labeled examples, their parameters do not receive gradient updates based on those examples. Although this approach is intriguing, Brown et al. (2020) report that it only performs well on the largest GPT-3 model, the API of which is costly and difficult to use for academic research (see Appendix B for details).
Continuous Prompts presuppose that the final embeddings of a model are not always perfect. This approach is based on the assumption that the final embeddings do not correspond to any real word in the vocabulary (e.g., Lester et al., 2021; Li and Liang, 2021; Qin and Eisner, 2021). This approach often efficiently tunes a much smaller set of model parameters, but these methods have not yet reported success in few-shot settings. Moreover, foregoing prompts as expressed in natural language makes it much harder to study their semantics, and it is not clear if continuous prompts serve as task-specific instructions or simply more efficient model parameters (see He et al., 2021 for a detailed analysis).
2.2 Analyses of Prompts
In this paper, we focus on discrete prompts because we can manually write and control their wording and semantics. We measure the effect of prompt semantics by the model's  $k$ -shot performance where  $k = \{0, 4, 8, 16, 32, 64, 128, 256\}$ . This setup resembles that of Le Scao and Rush (2021), but their study focuses on comparing Schick and Schütze (2021b)'s existing small set of prompts against traditional fine-tuning over the training trajectories of entire training sets, whereas our study focuses on the few-shot learning trajectories among a much more diverse set of prompts designed to test specific hypotheses about the effect of prompt semantics on few-shot learning speed.
At a high-level, our findings contradict Mishra et al. (2021)'s claim that models benefit from elaborate instructions adapted from crowdsourcing annotation guides. But note that they define "instructions" more broadly as including priming examples, and they find that "GPT-3 benefits the most from positive examples, mildly from definition, and deteriorates with negative examples." (p. 18). In other words, if we ablate priming and narrow "instructions" to just the description of a task, we in fact have the same finding that instructions are only modestly beneficial over no instructions (cf. our
irrelevant templates). In a similar vein, concurrent work by Lampinen et al. (2022) finds that other components of a prompt such as explanations of priming examples are helpful, but models are indifferent to whether the instructions in fact describe their tasks.
Finally, a growing body of concurrent work also questions the degree to which models need meaningful instructions (Khashabi et al., 2021; Prasad et al., 2022). One particularly noteworthy finding is that Min et al. (2022) show that models learn just as well with incorrect labels as opposed to correct labels in priming, concluding that prompts are helping models to learn the distribution of the input text and space of possible labels (as opposed to specifying instructions of the task).
3 Overall Setup
We implement a manual discrete prompt model which in essence is the same as that of Schick and Schütze (2021b), except their implementation includes several augmentations such as self-labeling and ensembling of multiple prompts for competitive results. In order to focus on measuring the effect of prompts themselves, our implementation does not include those augmentations. Following Sanh et al. (2021) and Wei et al. (2021), we evaluate by a rank classification of the target words.
Baseline Model In preliminary experiments, we fine-tuned and prompt-tuned BERT, DistilBERT, RoBERTa, ALBERT, and T5 (Devlin et al., 2019; Sanh et al., 2019; Liu et al., 2019; Lan et al., 2020; Raffel et al., 2020; all implemented via Wolf et al., 2020). Confirming prior work (Schick and Schütze, 2021b; Tam et al., 2021), we find that ALBERT consistently yields the best performance, so we use it as our baseline model.
To verify that our implementation is comparable with prior work, Figure 10 reports the RTE validation accuracy of our baseline model. At 32 shots, our implementation yields a median accuracy of  $70.22\%$  (mean  $= 69.29\%$ , std. dev.  $= 6.3\%$ ), which is comparable to the  $69.8\%$  reported by Schick and Schütze (2021b). Further, Figure 10 confirms Le Scao and Rush (2021)'s finding that, while both fine-tuning and prompt-tuning converge to similar results when fully trained on the entire set ( $n = 2490$  for RTE), prompt-tuning yields the largest improvement in the few-shot setting. Going forward, we focus on studying the few-shot learning trajectory between 4 and 256 examples.
Instruction-Tuned Model We additionally experiment with T0, a recently proposed instruction-tuned model which is trained on over 60 datasets formatted with hundreds of manually written prompts (Sanh et al., 2021). We experiment with both sizes of T0 (3B and 11B), as well as their non-instruction-tuned version, T5 LM-Adapted (Lester et al., 2021), as a baseline.
Very Large Model Lastly, we experiment with the largest GPT-3 (175B) via priming (a.k.a. in-context learning). Although fine-tuning is technically available, it is extremely limited by OpenAI's various quotas. See Appendix B for details on how we circumvent challenges in reproducing Brown et al. (2020)'s results.
Data NLI is a task where a model is asked to classify whether one piece of text (the "premise") entails another (the "hypothesis"). We focus on NLI because all T0 variants hold out all NLI prompts and all NLI datasets in its training, which makes it a fair comparison to other models in this paper.
We use Recognizing Textual Entailment (RTE, Dagan et al., 2006, inter alios), a series of expert-annotated NLI datasets. Specifically, we use the SuperGLUE collection of RTE (i.e., RTE1, 2, 3, and 5; all converted to binary classification) and report their validation accuracy for comparability with prior work on prompts.
We also experiment with Adversarial NLI (ANLI, Nie et al., 2020), Heuristic Analysis for NLI Systems (HANS, McCoy et al., 2019), and Winograd Schema Challenge (WSC, Levesque et al., 2012), reported in Appendices G.2, K, and L, respectively. We find no qualitative difference between their and the main RTE results except that ANLI requires much larger number of shots before obtaining any above-random accuracy, as it is designed to be a highly challenging set.
Random Seeds & Example Sampling All experiments are run over the same set of 4 random seeds. Within a given seed, all models see the same set of examples. For instance, under seed 1, the 4-shot models see examples 550-553, the 8-shot models see examples 550-557, and so on. Across different seeds, a different starting example index is drawn. The exact training example indices are also recorded in our GitHub repository for reproducibility.
Statistical Tests We use both ANOVA and its nonparametric equivalent, the Kruskal-Wallis test. After finding a significant difference among multiple categories of templates, we report pairwise significance with the independent two-sample  $t$ -test and the Wilcoxon rank-sum test. We set  $\alpha = 0.05$  and apply the Bonferroni correction to account for multiple comparisons. For all results reported in this paper, both  $t$ -test and Wilcoxon agree.
4 Effect of Templates
Our research question is whether models understand prompts as meaningful task instructions analogous to how humans would. For intuition, suppose an experimenter provides a human annotator with an informative instruction of a reasonably easy task. If the annotator understands the instruction, we expect them to perform better than when the experimenter provides intentionally misleading instructions, makes irrelevant chitchat, or says nothing at all. Accordingly, we write various prompt templates that correspond to these different scenarios and evaluate models' performance with these templates in zero-shot and few-shot settings.
We write 5 categories of templates (Table 1), with at least 5 templates for each category (10 for instructive):
- Instructive: how we would describe the NLI task to a human who has never seen this task before.
- Misleading-Moderate: instruct the models to perform a task related or tangential to NLI such that, if the model were to perform the task as explicitly instructed, it would perform poorly on NLI in general.2
- Misleading-Extreme: instruct the models to perform a task unrelated to NLI.
- Irrelevant: concatenate the premise, a sentence unrelated to any NLP task, and the hypothesis.
- Null: concatenate the premise and the hypothesis without any additional text.
Category	Examples
instructive	{prem} Are we justified in saying that “{hypo}”?
Suppose {prem} Can we infer that “{hypo}”?
misleading-
moderate	{prem} Can that be paraphrased as: “{hypo}”?
{prem} Are there lots of similar words in “{hypo}”?
misleading-
extreme	{prem} is the sentiment positive? {hypo}
{prem} is this a sports news? {hypo}
irrelevant	{prem} If bonito flakes boil more than a few seconds
the stock becomes too strong. “{hypo}”?
null	{premise} {hypothesis}
{hypothesis} {premise}
Table 1: Example templates for NLI.
See Table 1 for examples and Appendix F for the full list. We use "prompt" to mean a unique combination of a template and a predefined LM target word for each class label. For example, {"yes"  $\rightarrow$  entailment, "no"  $\rightarrow$  non-entailment} are the default targets for the template {premise} Should we assume that {hypothesis}? [prediction]. In this section, to control for the effect of target words, a template's performance is always reported with "yes"/"no" as its target words, which consistently perform best. In Section 5, we control for the templates and study the effect of different target words. We further control for punctuation, declarative vs. interrogative templates, and the order of concatenation (always {premise} some template text {hypothesis} [prediction]).
After preliminary experiments, to avoid cherry picking, all prompts reported in this paper were written prior to evaluation, i.e., we do not allow retroactively editing prompts for performance manipulations, except for an ablation study that explicitly studies the effect of punctuation (Appendix A).
Irrelevant Templates We find that models trained with irrelevant templates learn just as fast as those trained with instructive templates, with no practical difference<sup>3</sup> at any number of shots (Figure 1). This is true for all models and all datasets in our experiments, including the largest GPT-3 (Figure 2).
<sup>2</sup>An author manually labeled the 30 training examples seen by models under random seed 1 (example nos. 550-580), among which we find 17 pairs of entailment, 5 or 8 pairs (depending on how strictly one judges their acceptability) of summarizations, and only one pair of paraphrase.
<sup>3</sup>We acknowledge that a lack of a statistically significant difference does not entail "no difference". While it is true that we find no statistically significant difference with the independent two-sample  $t$ -test and the Wilcoxon rank-sum test whenever we say "no practical difference", note that our argument, here and through the paper, hinges on the very small effect sizes, not the significance tests, i.e., the two categories of prompts perform too similarly in absolute terms.
Figure 1: T0 (3B) on RTE. There is no practical difference between the performance of the models trained with instructive templates vs. those trained with irrelevant templates at any number of shots.
Figure 2: 16-shot accuracy of four large models on RTE. For GPT-3, there is no practical difference between any template categories except null (not plotted because they are below 0.5). For T5, there is no practical difference between instructive and irrelevant. For T0, there is no practical difference between instructive and irrelevant nor between instructive and misleading-moderate. For  $\mathrm{T}0 + +$ , there is no practical difference between instructive and irrelevant nor between instructive and misleading-extreme.
Misleading Templates There is no consistent relation between the performance of models trained with templates that are moderately misleading (e.g. {premise} Can that be paraphrased as {"\{hypothesis\}"?) vs. templates that are extremely misleading (e.g., {premise} Is this a sports news? {hypothesis}). T0 (both 3B and 11B) perform better given misleading-moderate (Figure 3), ALBERT and T5 3B perform better given misleading-extreme (Appendices E and G.4), whereas T5 11B and GPT-3 perform comparably on both sets (Figure 2; also see Table 2 for a summary of statistical significances.) Despite a lack of pattern between
Figure 3: T0 (3B) on RTE. There is no practical difference between models trained with instructive and misleading-moderate templates at any number of shots. But models trained with misleading-extreme templates are statistically significantly worse from 8 to 128 shots.
the two misleading categories, however, it is consistent that each model exhibits significantly better performance on instructive templates compared to at least one category of misleading templates.
Null Templates Models trained with null templates perform far worse than all other categories of templates (see Appendix G for all null results). Here, we focus on ALBERT (an encoder-only masked language model), which allows more permutation of concatenation orders by placing mask in the middle of sentences. We see that, although null templates are much worse in aggregate, some subset of them (e.g., {premise} [mask] {hypothesis}) are still able to learn nearly as fast as the average instructive template after 32 shots (Figure 13).
Zero-Shot So far, we have focused on few-shot results. At zero shots, all models (including GPT-3 175B) perform only marginally above random, except the instruction-tuned T0. Thus, for our analysis of zero shot performance, we focus on T0. Figure 4 shows that there is no practical difference between the performance of T0 3B given instructive templates and either category of misleading templates. T0 11B performs better, although it also shows no practical difference between misleading-moderate and instructive templates. Lastly,  $\mathrm{T0 + + }$  (trained on more datasets than other T0 variants), is the only model in this paper that shows statistically significantly different performance across all categories of prompts. However, there remains the caveat that it still performs arguably too well in absolute terms with pathological prompts, which we discuss in the next section.
Figure 4: Zero-shot accuracy of instruction-tuned models on RTE. Each prompt's performance is a single point (unlike the few-shot figures where each prompt is approximated by multiple points with multiple samplings of few-shot examples.) Arrows highlight some prompts with their excerpts. See Appendix I for the full results.
size	#shots	inst. > mis-moderate	inst. > mis-extreme	inst. > irrelevant	inst. > null
T0	3B	0	✓
T0	11B	0	✓	✓	✓
T0++	11B	0	✓	✓	✓	✓
ALBERT	235M	4 - 256	✓	✓
T5 LMA	770M	4 - 256
T5 LMA	3B	4 - 256	✓	✓
T0	3B	4 - 256	✓	✓
T5 LMA	11B	16	✓	✓	✓
T0	11B	16	✓	✓
T0++	11B	16	✓	✓
GPT-3	175B	16	✓
Table 2: Checkmarks indicate where two categories of templates lead to statistically significantly different performance, as measured by an independent two-sample  $t$ -test and a Wilcoxon rank-sum test; both tests always agree in this table. A lack of checkmark indicates where model performance fails to differentiate the two categories, i.e., models do not understand the differences between the prompt categories. We consider significant differences (checkmarks) between categories of prompts to be necessary—but not sufficient—for language understanding.
Recall that a common assumption in the literature is that prompts require experts to clearly and correctly describe the task at hand (§1). In contrast, Table 2 summarizes that, with the exception of  $\mathrm{T0}++$  at zero shots, all models perform essentially as well with some pathological prompts as they do with proper prompts. Notably, despite being much larger than its competitors, GPT-3 shows the same patterns of behaviors, suggesting that mere scaling does not address this issue. Meanwhile, the evidence from instruction tuning is mixed. Although Sanh et al. (2021) are right that instruction tuning yields substantial improvement in performance as
well as robustness as measured by variance, T0 is somewhat too robust and less sensitive to the semantics of the prompts in terms of distinguishing proper instructions from pathological ones, compared to T5 of the same size in the few-shot setting (Figure 2).
In the zero-shot setting, we do see that that the largest model instruction-tuned with the most datasets  $(\mathrm{T0}++)$  improves a model's sensitivity to prompt semantics. This is a positive result, but it comes with the caveat that there still exist numerous examples of pathological prompts that perform just as well as the proper ones do. To be charitable to randomness in neural models, we hold
Figure 5: The best-performing instructive template for ALBERT on RTE, {prem} Are we justified in saying that {"hypo"}? with select LM targets from each category.
Figure 6: T0 (3B) on RTE. Misleading templates + yes-no targets (red) learn substantially faster than instructive templates + arbitrary targets (green), which is the opposite of what we expect from humans.