The language of prompting: What linguistic properties make a prompt successful?
Authors: Alina Leidinger
Robert Van Rooij
Ekaterina Shutova
Venue: EMNLP 2023 Findings (2023)
Keywords: prompting
evaluation
LLMs
zero-shot
robustness
instability
instruction-tuning
TL;DR: zero-shot performance of LLMs varies considerably even for larger instruction-tuned models and doesn't correlate with perplexity in general

Abstract: The latest generation of LLMs can be prompted to achieve impressive zero-shot or few-shot performance in many NLP tasks. However, since performance is highly sensitive to the choice of prompts, considerable effort has been devoted to crowd-sourcing prompts or designing methods for prompt optimisation. Yet, we still lack a systematic understanding of how linguistic properties of prompts correlate with the task performance. In this work, we investigate how LLMs of different sizes, pre-trained and instruction-tuned, perform on prompts that are semantically equivalent, but vary in linguistic structure.  We investigate both grammatical properties such as mood, tense, aspect and modality, as well as lexico-semantic variation through the use of synonyms. Our findings contradict the common assumption that LLMs achieve optimal performance on prompts which reflect language use in pretraining or instruction-tuning data. Prompts transfer poorly between datasets or models, and performance cannot generally be explained by perplexity, word frequency, word sense ambiguity or prompt length. Based on our results, we put forward a proposal for a more robust and comprehensive evaluation standard for prompting research.

The language of prompting:
What linguistic properties make a prompt successful?
Alina Leidinger", Robert van Rooij and Ekaterina Shutova Institute for Logic, Language and Computation, University of Amsterdam
Abstract
2022: Sanh et al., 2022). Mounting evidence of per- formance variability (Köksal et al., 2022; Gonen et al., 2022, i.a.) has been met with an abundance of proposed methods for automatic prompt gen- eration (Shin et al., 2020; Gao et al.. 2021; Liu et al., 2023, i.a.) paired with an ongoing discus- sion on the superiority (or inferiority) of expert- written prompts
The latest generation of LLMs can be prompted achieve impressive zero-shot or few-shot performance in many NLP tasks. However, since performance highly sensitive to choice of prompts, considerable effort has been devoted to crowd-sourcing prompts or design- ing methods for prompt optimisation. Yet, we still lack a systematic understanding of how linguistic properties of prompts correlate with task performance. In this work, we investi- gate how LLMSs of different sizes, pre-trained and instruction-tune tunoon, perform on prompts that semantically equivalent, but vary in linguis- tic structure. We investigate both grammati- cal properties such as mood, tense, aspect and modality, well as dexico-semanté variation through the use of synonyms. Our findings contradict the common assumption that LLMs achieve optimal performance on lower perplex- ity prompts that reflect language use in pretrain- ing or instruction-tuning data. Prompts transfer poorly between datasets or models, and perfor- mance cannot generally explained by per- plexity, word frequency, ambiguity prompt length. Based on our results, we put forward proposal for more robust and comprehensive evaluation standard for prompting research',
generated ones (Logan IV et al., 2022: Webson and Pavlick, 2022). Funda- mentally, such variability in performance across prompting strategies raises the question of how LLMs process prompts based on language seen dur- ing training. Do they conform to linguistic intuition and respond better to lower perplexity prompts featuring straight-forward sentence structures, fre- quent and less anbiguous words, or language that has been seen during instruction-tuning??
To test this hypothesis, we examine prompting performance variability in LLMs through the lens of linguistics. To the best of our knowledge, we conduct the first controlled study of LLM per- formance variability across semantically equiv- alent prompts that differ in linguistic structure. Specifically, manually construct parallel sets of prompts that vary systematically grammatical mood, tense, aspect and modality (550 prompts in total?). We study the influence of word frequency and ambiguity by exchanging content words for alternative synonyms. We evaluate five LLMs of different sizes, both instruction-tuned and not instruction-tuned: LLaMA 30b (Touvron et al., 2023), OPT 1.3b, 30b (Zhang et al., 2022) and OPT-IML 1.3b and 30b (Iyer al., 2022). We focus on understanding of instructions, and there- fore, evaluate our models in a zero-shot fashion. We experiment with six datasets for three different tasks: sentiment classification, question answer- ing and natural language inference (NLI). For the instruction-tuned models, our choice of tasks cov-
1 Introduction
NLP has witnessed a rapid succession of large pre- trained language models (LLMs) being released accompanied by reports of impressive performance on multitude of tasks (Brown al., 2020; Tou- vron al., 2023; Zhang et al., 2022, i.a.). Many works show that increasing model scale decreases pre-training loss and improves downstream task performance on average (Brown et al., 2020; Rae et al., 2022). However, this does not hold in general across all samples and instructions (Ganguli et al.,
"Corrspooddig author: aj..eidinger@ur 'All resources are available at: ht https: httpgiithub.cco al Beaidinger/lanuuggeeeee -ttpeee____oppppporoe___ee
We use terms prompt instruction interchangeably. We release all prompts for all tasks in Appendix D.

---

ers the fully supervised, cross-dataset and cross- task setting.
tuning helps robustness against prompt variation, instruction-tur models respond favourably even to misleading instructions. Shaikh et al. (2023) show that GPT-3 scores drastically worse on bias and toxicity challenge sets with the addition of 'Let's think step by step.' to given prompt for Mhain-of-thougeee reasoning (Kojima et al.. 2022), Razeghi et al. (2022) find that performance on arith- metic tasks correlates with frequency of integers in the training data. Perhaps closest to our work, Gonen et al. (2022) find that lower perplexity of the prompt correlates with higher performance for OPT (Iyer et al., 2022) and BLOOM (Scao et al., 2022) on a variety of different tasks.
Overall, we observe large performance varia- tion due to changes in linguistic structure of the prompt ($5); and this holds even for instruction- tuned models on seen tasks. Furthermore, contrary previous findings (Gonen al., 2022), model performance does not appear correlate with per- plexity of the prompts ($6). Further, we find no cor- relation between performance and prompt length, word sense ambiguity or word frequency. In many cases more complex sentence structures and rare synonyms outperform simpler formulations. Our findings contradict the universal assumptions that LLMs perform best given low perplexity prompts featuring words which we assume (or know) to be frequent in pre-training and pnaructiio-tuning data, This stresses the need for further research into the link between statistical distribution of language at different training stages and model behaviour.
In priming in-context learning, LMs profit from being shown the required input-output format, the distribution of inputs and the label space, while ground truth labels don't seem to be required (Min et al., 2022). Performance is also sensitive to the or- dering of demonstration examples (Lu et al., 2022; Zhou et al., 2022; Zhao et al., 2021). Chinchilla per- forms better on abstract reasoning tasks when test samples cater to prior knowledge acquired through pretraining (Dasgupta al., 2022).
With regards evaluation practices in the field, our work highlights the limitations of benchmark- ing multiple models on single, fixed prompts and re- porting best-case performance. Prompts generally transfer poorly between datasets even for the same model, let alone across models ($5.2). Instruction- tuning ($5.3) or increasing model size ($5.4) do not preclude the possibility of considerable perfor- mance variation, even seen tasks. These find- ings, coupled with the fact that many works do not release their prompts, make the results of existing evaluations less reliable and difficult to reproduce. Section 7, we put forward a proposal for a more robust and comprehensive evaluation framework for prompting research.
Prompting evaluation practices. Cao et al. (2022) point out that evaluating models on the same prompt does not make for a direct compari- son, since models' exposure to different pretrain- ing data results in different responses to individual prompts. Ishibashi et al. (2023) find that machine- generated prompts do not achieve equal perfor- mance gains across datasets for the same task. Holtzman al. (2021) posit that subpar perfor- mance of LMs is due to different viable answers outside the answer choices competing for probabil- ity mass ("surface form competition" ")), reducing the score for the correct answer among answer choices, They mediate this using Domain Conditional PMI. Zhao et al. (2021) observe that models overpredict label words that occur more frequently in prompt, at its end, or are frequent in the pretraining data, They propose fitting an affine function to the LM scores, so that answer options are equally likely for 'content-free" dummy examples.
2 Related work
Instability in prompting Papers accompanying newly released models, rarely report prompts used for their evaluation and, typically, do not evaluate on multiple prompts (Brown al., 2020; Chowd- hery al., 2022; Rae et al., 2022). Sanh et al. (2022) stand alone in reporting performance varia- tion across prompts at the release of TO.
To date, few works have investigated robust- ness to different prompt formulations. Ishibashi et al. (2023) show that machine-generaaed prompts are not robust to token deletion or reordering. Webson and Pavlick (2022) evaluate pre-trained and instruction-tumed models on NLI in the few- shot setting. They find that while instruction-
Contrary to other works (Gonen et al., 2022; Sorensen et al., 2022; Liao et al., 2022) we do not aim proposing prompt selection methods or cal- ibrate predictions. While many nemii-automatte approaches to generating prompts have been pro- posed (Liu et al., 2023; Shin et al., 2020; Jiang et al., 2020; Gao et al., 2021, i.a.), we resort to crafting

---

prompts manually so as to maintain fine-grained control over sentence structures in our prompts, Logan IV et al. (2022) provide evidence that man- ually written prompts (Schick and Schütze, 2021) can yield better result than automatically sourced prompts. Further, Ishibashi et al. (2023) point out that automatically generated prompts contain atyp- ical language use, punctuation or spelling mistakes and generalise poorly across datasets. To isolate the effect of individual instructions we restrict our- selves to the zero-shot setting and do not include any demonstration examples in-context. Indeed, LMs have been shown to perform reasonably well in the few-shot setting given unrelated or mislead- ing instructions (Webson and Pavlick, 2022). Sim- ilarly, we abstain from prompt-tuning on demon- stration examples, so to not introduce additional sources of variance (Cao et al., 2022).
prop.
prompt
mood
inter.
Do you find this movie review positive?
indic.
You find this movie review positive.
imper.
Tell me if you find this movie review positive.
aspt.
active
Do you find this movie review positive?
pass.
Is this movie review found positive?
past
Did you find this movie review positive?
tense
pres.
Do you find this movie review positive?
future
Will you find this movie review positive?
can
Can you find this movie review positive?
could
Could you find this movie review positive?
modality
may
May you find this movie review positive?
might
Might you find this movie review positive?
must
Must you find this movie review positive?
should
Should you find this movie review positive?
would
Would you find this movie review positive?
Auu Aanonny  review
apprai.
Do you hind this movie appraisal positive?
comm.
Do you find this movie commentary positive?
criti.
Do you find this movie critique positive?
eval.
Do you find this movie evaluation positive?
Do you find this movie review positive?
Table I: Examples of variation of linguistic properties
3 Tasks and datasets
the latter category, we consider OPT-IML 1.3b and 30b (lyer et al., 2022)*. We make use of the Hug- gingFace transformers library (Wolf et al., 2020) for all our experiments.
To draw robust conclusions across tasks, we con- duct experiments on multiple datasets, namely Stan- ford Sentiment Treebank (SST-2; Socher et al., 2013) and IMDB (Maas et al., 2011) (Sentiment Analysis). SuperGLUE (Wang et al., 2019) Recog- nizing Textual Entailment (RTE; Wang et al., 2018) and CommitmentBank (CB; De Marneffe et al., 2019) (NLI), Boolean Questions (BoolQ: Clark et al., 2019) and A12 Reasoning Challenge Easy (ARC-E; Clark et al., 2018) (Question Answering). Our guiding principle in choosing datasets was to have diverse set of tasks on which pre-trained models such as LLaMA or OPT achieve above- chance performance in the zero-shot setting (lyer et ul., 2022). For OPT-IMI, our choice covers the supervised, cross-dataset, and cross-task setting. The OPT-IML models have been instruction-tuned on IMDB and SST using prompts from Prompt- Source (Sanh et al., 2022) and FLAN (Wei et al., 2021). They have been tuned on QA datasets such SeiQ, but not BoolQ ARC-E (Iyer et al., 2022) which we use. All textual entailment tasks (including RTE and CB) are fully-held out during training of OPT-IML (lyer et al., 2022).
4.2 Prompting setup
For our Sentiment Classification and NLI datasets, map the label space to target words yes/no yes/no/mann ns/maobe in the case of two three classes respectively. This was found to yield the best per- formance by Webson and Pavlick (2022) among alternative mappings. For question answering tasks, answer options are listed in a multiple-choice fash- ion using letters A, B or A, B, C, D which serve as target words. Additionally, we follow Gonen et al. (2022)'s advice regarding punctuation in prompts and add the postamble, e.g. "Choices: yes no? Answer:" to every prompt, as this was found to aid zero-shot performance and reduce surface- form competition (Holtzman et al., 2021). Each prompt is evaluated on 500 random samples for each dataset. For datasets belonging to the same task, we keep the set of prompts fixedS.
All our experiments are carried out in sero-shot setting. Following Sanh et al. (2022); Webson and Pavlick (2022); Wei et al. (2021), we predict by recording which target word assigned the largest log probability among all target words reggaddegss of whether that target word receives overall the
4 Method
4.1 Models
We examine both pretrained-only and instruction- tuned models in this study. The former category is represented by LLaMA 30b (Touvron et al., 2023) and OPT 1.3b and 30b (Zhang et al., 2022). In
"To showease similar performance variability for encoder- models, we include results on Flan-TS (Chung et al., 2022) in Appendix C.
Forr full test sample with answer options see App A.

---

largest log probability across the entire vocabulary. We choose accuracy as our evaluation metric".
Mood. As expected, LLMs generally respond more lavourably instructions phrased as ques- tions or orders rather than statements, with the ex- ception of LLaMA and OPT-IML 30b on BoolQ. However, we do not find that prompts in interroga- tive mood generally outperform imperative ones or vice versa. In most cases, instruction-tune broaden the gap between indicative and interroga- tive/imperative prompts dramatically. Notably, we find cases where an individual indicative prompt performs best across all prompts (e.g. Table de- tails that 'This movic review makes people want to watch this movie.' achieves the highest accuracy of 6. for OPT-IML 1.3b on IMDB.).
4.3 Linguistic variation in prompts
To contrast linguistic properties in prompts, we manually formulate parallel sets of prompts which differ in grammatical mood, tense, aspect or modality systematically and one at time. We present example in Table I and the full list of prompts" for all tasks in Appendix D.
did not
For example, in the case of mood, we design three sets of prompts each which are identi- cal, except that they are phrased either as a ques- tion, an order or a statement. The resulting sets of interrogative, indicative and imperative prompts are all in present tense and active voice, so to guarantee controlled setting. We pro- ceed similarly for aspect by formulating two sets of active and passive prompts (all in interrog- ative mood, present tense) and tense by crafting prompts simple past, present and future (all in active voice, interrogative mood). We also vary the degree of certainty in our instructions, while maintaining minimal edit distance, by in- troducing different epistemic modals (example in Table 1). Here, all prompts in active voice, present tense, interrogative mood. Lastly, we ask how model behaviouris is linked to word sense ambi- guity and word frequency. We thus replace content words in interrogative prompts with synonyms of varying word sense ambiguity and frequency.
Aspect. The hypothesis that active sentence constructions are simpler, shorter, more prevalent in the data and thus yield better performance was generally not confirmed. No model shows a clear preference for instructions phrased in active vs. pas- sive voice across all datasets, with the exception of OPT-IML 1.3b for which active voice works consistently better, albeit not always significantly. Interestingly, for all other models, passive prompts generally yield better results on BoolQ. For our 30b models, we find active prompts to be superior only for RTE and ARC-E (Tables 3, 4).
Tense. Similarly, the hypothesis that prompts present tense perform best, since they cater in par- ticular to prompts seen during instruction instruction-tuning, was only confirmed for CB. (In two cases, future prompts performed on par or slightly better.) On the other datasets our results proved to be more mixed. None of the tenses outperforms others for SST and IMDB, across models. On occasion, fu- ture and past prompts considerably outperformed present prompts, e.g. for OPT-IML 30b on IMDB (> 96% acc. (past/future) vs. 89% acc. (present)) or similarly for OPT 30b on SST. On BoolQ and ARC-E results varied less, with observed differ- ences under 2.5 percentage points.
Statistical tests & analysis We employ non- parametric tests, to quantify whether any observed performance variation between sets of prompts is indeed statistically significant. Specifically, we use the Friedman Two-Way Analysis of Variance by Ranks Test (Friedman, 1937) and the Wilcoxon Signed Rank Test (Wilcoxon, 1992) for paired sam- ples. We investigate the influence of prompt length, perplexity, word sense ambiguity and frequency on accuracy by computing the Spearman Rank-Order Correlation Coefficient (Spearman, 1961) and the Pearson Correlation Coefficient (Pearson, 1895).
Modality. Replacing different modal verbs in an instruction also results in considerable performance variation with differences up to 10 and 17 points on SST for LLaMA on 'must" vs. 'would" or OPT 30b on 'may" vs. 'might' (Table 2). For OPT-IML 1.3b and 30b, such variation is reduced on SST with average accuracies falling within the range of 92.23% 93.03% for both sizes. However, instruction tuning doesn't 't preclude the possibility of significant performance variation also for larger
5 Results
5.1
Performance variability
Our results per task are presented in Tables 2, 3,
"Run times and compute resources are detailed in App B. "Our prompts are loosely inspired by prompts in Prompt- Source (Sanh et al., 2022). However, we restrict ourselves to simple sentence structures consisting only of one main clause.

---

LLaMA 30b
OPT
1,3b
OPT-IMI.
OPT
30b
OPT-IML
SST
IMDB
IMDB
SST
IMDB
SST
IMDB
SST
IMDB
aspt, mood
indicative
82.53*
91.58
64.98'
71.88"
64.3"
91.92
76.2*
69.97
91.07'
71.33"
interrogative
83.35*
89.79"
87.98
58.38"
92.8
92.15'
79,9
69.95
92.25'
imperative
83.65
92.42
76.85
74.08
92.25"
92.92
79.08
66.98
92.68
87.5"
active
81.9
89.17
81.75'
64.35
93.3
93.2
69.5"
71.1
92.25'
91.25
passive
78.25
93.0
85.8
60.85*
92.8
93.15
71.0
70.3
92.55
92.75
past
80.8*
94.67
71.05*
80.53
86,95
94.85
85.8"
82.66"
91.68*
96, 12
tense
present
83.35
94.42
87.98
79.69"
92.8
94.7
79.9"
84.04
92.25
89.93"
Tuture
85.72
93.0"
76.6"
80.79
88.37'
94.88
87,35
83.79"
92.18'
96.15
can
85.22"
90.33'
$3.0"
64.78
92.5
90.78'
77.45"
73.5"
92.83
89.12
could
84.75*
91.38*
77.47"
67.12"
92.42
90.2"
70.97"
74,02
92.68*
87.58"
modality
may
84.62"
87.12"
82.63"
65.17"
92.35
90.83"
82.3"
71.1"
92.55*
87.38"
might
83.85"
91.25"
77.18
69.72
92.25
91.1"
65.98"
72.7"
92.92
85.58"
must
75.52"
90.62"
85.6"
59.77"
92.55
91.73
82.9
67.95"
92.6"
88.0"
should
82.92"
91.33"
85.47
63.08
92.78'
90.05'
81.32"
70.15*
92.73'
88.25'
would
85.97
92.54
86.05
62.12"
93.03
91.55
7-103"
71.25"
92.3"
85.25
appraisal
81.63"
87.49"
93.17"
90.46
76.26"
73.2
93.23
commentary
81.6*
92.95
86.37"
60.23
92.91"
88.03"
64.34"
71.86"
92.97'
93.09
critique
84.4
92.29"
85.66"
58.94"
92.46"
91.11"
68.63"
71.46"
92.43'
evaluation
83.63"
92.0"
89.89
52.34
93.29
91.74"
78.26"
70.23"
92.49"
91.26"
review
82.97'
92.95
87.94"
56.94"
92.97'
93.23
80.77
68.97'
92.17'
88.8"
null prompt
83.2
72.8
41.2
64.2
12.8
93.0
65.8
74.8
37.0
86.2
chance
50
50
50
50
50
50
50
50
50
50
Table Average accuracy per prompt in categories mood, aspect, tense, modality, synonymy SST and IMDB (Sentiment Classification), Highest accuracy per category for each model and dataset marked in bold. Significant lower results per category marked with an asterisk. The null prompt contains no instruction.
LLaMA 30b
OPT L,5h
OPT-IMI. 1,3b
OPT 30b
OPT-IML. 30b
BoolQ
ARC-E
BoolQ
ARC-E
BoolQ
ARC-E
BoolQ
ARC-E
BoolQ
ARC-E
aspt. mood
indicative
72.0
73.18
62.08
27.6
63.04
31.87
54.38
29.68
65.72
68.78
interrogative
67. 67.75*
75,43
61.92
28.77
64.25
31.53
60.4
31.72
63.44 44"
68.28
imperative
64.58*
74.65
62.72
27.52
63.7
34.1
60.52
30.47
64.81
69.16
active
67.6"
75.76
62.05
29.15
64.1
31.86
61.5
36.73
62.3
66,68
passive
73.9
72.98"
62.55
28.14
63.55*
30.05"
61.55
32.41
63.3
64.17"
tense
past
66.83'
75.0
59.28"
28,02
49.15
29.23
63.69
28.4
66.35
70,96
present
67.00 07.
72.68"
59.13"
28.19
49.5
30.2"
63.3"
27.92
67.13
71.11"
future
67.43
73.03"
59.61
27.93"
49.07'
30.3
63.03"
28.15"
66.91
71.19
can
64.5"
73.82"
61.75"
28.02"
63.62
31.41"
62.13
34.42
63.75*
67.22"
could
63.75
73.82"
61.88
28.27*
63.25
31.28"
61.38
33.92
63.5'
67.47"
modality
may
62.62"
74.21"
62.13
28.27
31.78
60.5*
32.41
63.5"
might
63.5
75.52"
61.88
28.39"
63.88"
31.41
60.38"
32.79"
62.5"
67.73
must
65.0"
75.79"
28.27*
63.5"
30.53"
57.38"
29.27'
65.38
67. 64"
should
67.75
75.65
61.87
28.77
64.13
30.4"
58.88"
29.77'
63.88"
68.9
would
67.12
77.09
61.5
28.39'
64.0
30.03"
60.62"
34.67
64.0"
68.23
ATTABONAS ÁurAuoBss
proper
66.02"
75.93
62.72
27.32
63.76
33.56"
61.68
32.13
63.9
68.67
right
62.76*
76.1
62.64
27.1"
63.94
34.08
61.12"
31.77
64.44"
68. 11
correct
62.9"
76.16
62.28
27.65"
63.78"
33.82
60.3"
31.73"
64.24
68 21
appropriate
67.24
75.85"
62.64
27.73
63,98
33.34"
61.32"
32.31
64.5
68,91
answer
61.82*
76.18
62.33"
27.71*
63.82"
33.69
60.07
31.94
64.15*
68.47
reply
62.93*
75.43"
62.22"
27.73"
63.91
33.42
60.73"
31.28"
64.64
67.81
response
65.98
76.39
62.35*
27.35*
63.85
32.91
61.45
31.61"
64.31*
68.3
solution
62.73*
73.11"
63.02
28.02
63.89*
33.14"
60.6
30.89*
64.33*
67.55*
null prompt
64.0
75.0
61.5
26.13
68.0
29.29
68.0
28.28
72.0
63.64
chance
50
50
25
25
25
Table 3: Average accuracy per prompt in categories mood, aspect, modality, synonymy BoolQ, ARC-E (Question Answering). Highest accuracy per category for each model and dataset marked in bold. Significant lower results per category marked with an asterisk. The null prompt contains no instruction.
models of eg. 4 percentage points of OPT-IML 30b on IMDB and CB. On QA (Table 3) and NLI (Table 4) datasets, we find numerous examples of drops in accuracy by up to 5 percentage points.
Synonymy. Perhaps most surprisingly. replacing content words with non-standard synonyms does not generally hurt performance, but rather improves it. In particular, for SST and IMDB the content

---

LLaMA 30b
OPT 1.3h
OPT-IMI. 1.3b
OPT
306
OPT-IMI. 306
RTE
CB
RTE
CB
RTE
CB
RTE
CB
RTE
CB
uspt, mood
indicative
53.28
49.7
47.9"
51.9"
58,55'
62.27
51.18"
46.77'
67.96
79.67
interrogative
57.6"
53.53
48.0
58.17
58.95'
62.33
51.01"
59.33
69.58
75.47*
imperative
57.68
52.4
47.35
52.13
60.4
61.4
52.4
59.27'
70.22
80.1
active
60.62
53,68
52.9'
56.28
55.65
59.88
52.38
58.56
69.3
70.84
passive
60.46*
53.08"
53.0
57.0
54.95
59.88"
51.8"
55.56*
69.16
68.
tense
past
53.11
51.0"
52.15
58.0
65.63"
62.17"
52.15
60.3
70.7*
12.13"
present
52.96"
51.63
51.16"
58.73
65,9
62.43
52.08
60.53
71.78
74,23
future
52.51
48.93"
52.1
55.87'
65.2"
62.63
51.43"
60.53
70.72
71.58"
can
60.55'
53.07
54.04
56.6
56.12
61.83"
53.17
59.07'
71.65
74.53"
could
61.37*
53.37
53.17
58.2
56.38
62.37
51.5"
58.97
70.7*
74.33*
modality
may
60.18*
52.27
54.75
58.07"
54.58"
64.03
52.28*
56.6"
72. 17
73.6"
might
60.37*
51.87"
53.5'
57.03'
54.29"
63.13
51.53'
56.43"
72.2
72.3*
must
57.25'
50.43
54.25
54.93"
52.92"
61.73
51.23'
56.33'
70.62
70.53"
should
58.38'
49.57"
54.04"
55.93'
55.67
60.7"
51.07'
60.73
71.55
71.3"
would
62.75
51.63"
53.75
56.03'
51.17"
60.43"
51.55'
60.17'
71.0"
74.87
syn. syn.
assertion
57.1
52.0
50.8
58.6
66.8
58.6
53.4
67.8
74.3
78.2
claim
55.5"
49.6"
49.6
58.6
68.0
60.6
51.9"
66.0"
74.4
78.8
entailment
55,4
42.4"
50.1*
55.4"
63.6
60.4*
49.0"
16.0
70.2
75.2
implication
54.8"
54.4
51.0
60.2
64.1
63.8
50.0
46.4
69.9
75.0
null prompt
45.0
55.2
40.0
57.6
16.5
61.6
46.6
66.8
11.2
79.2
chance
50
33.3
50
33.3
50
33.3
50
33.3
50
33.3
Table 4: Average accuracy per prompt in categories mood, aspect, modal verbs, synonymy on RTE, CB (Question Answering). Highest accuracy per category for each model and dataset marked in bold. Significant lower results per category marked with an asterisk.
word 'review' does not guarantee optimal perfor- mance (Table 2). This holds even for OPT-IML 30b which has been trained on instructions from FLAN and PromptSource most of which contain the word 'review". Instead rare synonyms such as 'appraisal' and 'commentary' yield better perfor- mance. Similarly, on BoolQ and ARC-E we did not find that prompts containing the words 'correct' and 'answer' worked best-even if many of the prompts in FLAN and PromptSource contain those words and none of the other synonyms we tested. Notably often, models respond more favourably to the rarer synonym 'appropriate" (See Table 3).
Prompts that were optimal for one model and dataset also transferred poorly to other models. We found many cases of drops by more than 20 per- centage points on SST and IMDB or 5 percentage points on BoolQ and CB.
5.3 The relation between robustness and instruction-tuning
Instruction-tuning holds promise improved per- formance and robustness, but how robust can we ex- pect our results to be? In line with previous works, we find our instruction-tunee models to perform more reliably seen tasks than their pre-trained counterparts of the same size. For OPT 30b, accu- racy can vary by 10 points or more for SST and IMDB. For OPT-IML 30b this gap narrows, but re- mains non non-negligible While performance on SST stabilises between 91% and 93% for SST, we still see performance in the range 85 92% on IMDB (Table 2). ARC-E is not part of the instruction- tuning tasks for OPT-IML 30b and performance here varies by 5 percentage points (Table 3). Sim- ilarly, performance on RTE and CB varies signifi- cantly with accuracies in the ranges 67.96 72.2% and 68.56 80.1% respectively (Table 4),
5.2 Prompt transfer
When evaluating ent models and datasets, one tacitly assumes that prompts are to some extent 'universal'. Our results largely contradict this assumption (Ta- bles 9, 11, 13). We found numerous cases which a prompt performed optimally for one model on one dataset, but gave staggeringly poor results on other datasets. For instance, the best prompt for OPT-IML 1.3b IMDB yields 97.66 97.6% accuracy, but barely above chance performance on SST. Sim- ilarly, we saw large drops in performance when transferring optimal prompts from IMDB SST and vice versa for LLaMA 30b, OPT 1.3b and OPT 30b.
fixed prompt for differ-
5.4
The relation between robustness and model size
We find numerous examples of increased model size not leading to increased stability. For instance,

---

changing "must' to "might" results in performance drop by 17 percentage points for OPT 30b on SST (Table 2). Overall, when comparing OPT-IML with OPT at 1.3b and 30b, the gup between best and worst prompts closes only for RTE, is stable for BoolQ, ARC-E and SST and widens for IMDB, and CB e.g. from 5 to 12pp (Tables 9, 11, 13).
difficulty for LLMs and lower accuracy. We largely do not find this to be the case (Table 5). While correlation between accuracy and degree of ambiguity is generally around zero for BoolQ and ARC-E, on CB is positive for all models. For SST and IMDB, OPT-IML 30b performs better on less ambiguous prompts, potentially due to its size, while the opposite is true of OPT-IML 1.3b.
6 Analysis
Prompt length Overall, none of our models per- form better on longer or shorter prompts (in number of tokens) across all tasks (Table 5). For LLaMA 30b and OPT 1.3b, longer prompts result in sig- nificantly higher accuracies only for IMDB and BoolQ. Performance of OPT-IML 1.3b correlates positively with prompt length (except on SST and CB). For OPT-IML 30b results are mixed.
Since accuracy varies considerably, we now anal- yse whether higher accuracy can be explained by lower prompt perplexity, prompt length, or the use of less ambiguous or more frequent words in the prompt. We present correlation results in Table 5.
Prompt perplexity Following Gonen et al. (2022), we average across perplexity for 500 ran- dom test samples each accompanied by the instruc- tion in question. We find that perplexity scores often reflect linguistic intuition, e.g. they are lower for prompts in imperative vs. indicative mood or prompts containing 'answer' vs. other synonyms (see Appendix E). Surprisingly however, we do not find lower perplexity to correlate significantly with higher accuracy across models or datasets (Table 5). For LLaMA 30b, higher perplexity correlates with higher accuracy (except on IMDB and BoolQ). OPT-IML 30b performs better given higher perplexity prompts. Overall, our findings contradict Gonen et al. (2022) and indicate that the success of high or low perplexity prompts is particular to any combination of dataset and model.
Lessons learnt and way forward
7.1 Implications of our research
Instability in prompting Our findings clearly demonstrate the instability of prompt-based eval- uation ($5) that is rarely featured in performance reports. For any model and task, differences in per- formance can be considerable at even the slightest change in wording sentence structure.
The connection between data distribution and model behaviour Our findings should be taken as an invitation to revisit the common assump- tion that LLMs respond best to lower perplexity prompts containing simple and frequent words and grammatical structures. We find numerous cases where LLMs do not learn from the data distribution in the way one would assume based perplex- ity scores and linguistic intuition (S6). This calls for further investigation into the interplay between model behaviour and the distribution of language use during pretraining and instruction-tuming
Frequency of synonyms We approximate word frequency as the number of occurrences the 14b Intelligent Web-based Corpus (Davies and Kim, 2019). In general, we do not find that frequent syn- onyms lead to better performance (Table 5). For BoolQ and ARC-E, correlation between word fre- quency and accuracy oscillates around zero with no clear preference for frequent synonyms emerging. For SST and IMDB, infrequent synonyms tend to work better for OPT IML 30b. For OPT, correlation is positive for SST and negative for IMDB's longer sentences. For I.LMAA correlation coefficients are mostly around zero. On RTE and CB, more fre- quent synonyms lead to better performance except for LLaMA and OPT 1.3b RTE.
The effect of instruction-tuning Contrary to pre- vailing opinion, the above holds also for compara- tively larger instruction-tuned LLMs. Our results indicate that cnrructtiontunnngg should not be taken as a panacea to performance instability without
further investigation ($5.3). While it does overall improve performance and robustness, performance can still vary by over 5pp seen tasks.
Ambiguity of synonyms We quantify word sense ambiguity as the number of word senses in WordNet (Miller, 1995). Intuitively, one would expect more ambiguous synonyms to pose greater
Limitations of current evaluation practices Im- portantly, our work highlights the limitations of benchmarking LLMs on the same prompt for a given task, or only a small set of prompts, which

---

LLaMA 30b
OPT 1,3b
OPT-IMI 1.3h
OPT 30h
OPT-IMI. 30b
Px
Pr
Pe
Pp
Pa
Pp
Px
Pe
Ps
Pp
0.23
0.3
0.27
0.16
0.49
0.22
0.11
ppl. VS acc.
IMDB
-0.17
-0.07
-0.06
0.04
0.06
0.02
-0.37'
-0.18"
0.18
0.13
BoolQ
-0.02
0.02
-0.21
-0.3*
0.04
-0.17
0.22
0.17
0.13
0.06
ARC-E
0.2"
0.16"
-0.07
-0.04
-0.14
-0.15
-0.12
-0.12
0.11
0.07
RTH
0.15
0.16"
0.08
0.08
-0.07
-0.0
-0.28*
-0.13
0.05
0.51
CB
0.46*
0.17
0.43*
0.29
0.15
0.58
0.59
-0.47
-0.12
0.42
SST
0.06
0.01
0.23
0.07
0.1
0.03
0.35
0.21
0.17
0.3
amb VS acc.
IMDB
0.08
0.09
-0.07
-0.09
0.53
0.53
0.08
-0.12
0.33
-0.77"
BoolQ
-0.11
-0.09
0.06
0.04
-0
0.02
0.02
0.03
0.04
00.11
ARC-E
0.06
0.05
-0.16
-0.21
-0.03
0.09
0.1
00.03
-0.14
-0.09
CB
0.12
0.4
0.1
0.25
0.22
0.39
0.41
0.21
0.05
0.08
RTE
0.24
0.36
-0.2
-0.14
0.59
0,43
0.39
0.15
0.27
0.14
SST
0.09
0.02
0.2
0.1
0.05
0.05
0.11
0.19
-0.33"
-0.35"
freq. VS acc.
IMDB
-0.03
0.07
-0.43"
-0.18
0.39
0.5"
0.18
-0.14
-0.77
-0.6"
BoolQ
-0.11
-0.08
0.06
0.05
0.05
0.06
0.08
0.03
-0.01
0.04
ARC-E
0.02
0.05
-0.05
0019
-0.03
0.12
0.02
0.02
-0.04
-0.04
CB
0.12
0.4
0.1
0.25
0.22
0.39
0.41
0.21
0.05
0.08
RTE
0.24
0.36
0.2
0.14
0.59
0.43
0.39
0.15
0.27
0.14
SST
-0.23*
-0.37*
-0.27
-0.19
-0.16
0.06
-0.14
-0.27
0.14
0.23*
len. vS acc.
IMDR
0.15
0.05
0.16
0.19
0.23
0.39*
0.54
0.47
-0.14
0.03
BoolQ
0.11
0.06
0.1
0.2
-0.09
0.18
0.16
-0.27*
-0.26"
ARC-E
-.144
-0.13
-0.0
-0.02
0,27
0.32
0.25
-0.15
-0.08
RTB -0.77
-0.45'
-0.46"
-0.55*
0.13
0.11
-0.08
-0.17
-0.31
-0.1
CB
0.38
0.33*
-0.4
0.27
0.19
0.03
0.22*
0.43*
0.35
Table 5: Spearman (ps) and Pearson correlation coefficients (0p) of (1) perplexity, (2)word sense ambiguity, (3)
frequency of synonyms, and (4) prompt length against accuracy. Significant results (p < 0.05) marked with an asterisk.
is the current practice. Large variability in perfor- mance ($5) and lack of transparency around used prompts make evaluations unreliable and hard to reproduce. Individual prompts are not transferable across datasets or models ($5.2). Instruction-tunm -utning does not appear to solve these problems ($5.3).
ambiguity, frequency, and model behaviour.
Generate sufficiently large of prompts. Given the high levels of performance variability be- tween different prompts (up to 17 pp, as observed in our experiments), is crucial to experiment with sufficiently large set of prompts, so as to obtain statistically reliable estimates of performance, in- dependent of individual prompt formulations.
7.2 Recommendations
Based on the lessons learnt in our study, we put forward proposal for more robust and compre- hensive evaluation framework for LLM prompting.
Include estimates of performance mean and vari- ance based on large set of prompts for more accurate picture of model capabilities.
Collect prompts that represent linguistic vari- ability. In order to obtain robust and accurate estimate of model performance, one needs ac- count for linguistic variability of prompts, ideally in a controlled and rigorous manner. This can be ac- complished by collecting sets of candidate prompts that are representative core linguistic structures.
Treat prompts as hyperparameters. When choos- ing a single prompt for evaluation, select it per model and dataset on a held-out development set.
Standardize and report metries characterising the prompt set and impact on performance.
Use semi-automatic approaches such as con- trolled paraphrasing (lyyer et al., 2018) to generate prompts that vary in grammatical constructions in systematic way.
For prompt collections, report metrics such perplexity, degree of ambiguity, and their distri- bution. Analyse how these correlate with perfor- mance metrics or log probabilities of true labels using correlation coefficients (Gonen et al., 2022).
Replace content words with synonyms to diver- sify vocabulary. Our results on synonyms point to simple method for expanding prompt sets to cover a more diverse vocabulary. This enables controlled investigation of the the link between word sense
Use mixed effects models (Gelman and Hill, 2006) to analyse how prompt characteristies in- fluence model performance per dataset and sam- ple (Lampinen et al., 2022).

---

8 Conclusion
of LLaMA. At the time of writing, we are only aware of models such as Vicuna and Alpaca? which are trained on data generated by OpenAl text-davinci-003 or interactions with human users, unlike OPT-IML which has been fine-tuned on a large collection of NLP tasks. We did thus not in- clude Vicuna and Alpaca in our experiments so to avoid skewed comparison.
We evaluated five LLMs, both pretrained and instruction-tuned, on parallel sets of prompts that systematically differ in grammatical mood, aspect, tense, modality and use of synonyms. We find that there is no favoured sentence structure that per- forms best across models and tasks. Prompts gener- ally transfer poorly across datasets and models. We find considerable performance variation, which can still persist even for larger instruction-tunet LLMs on seen tasks and is not explained by perplexity, word sense ambiguity or word frequency.
Ethies & broader impact
In this work, we analyse LLM behaviour given variety of linguistic properties provided to them as prompts. We uncover that models process lan- guage which varies in word sense ambiguity, fre- quency, perplexity, and length of expression in un- expected ways. Based on our findings, we provide recommendations for reliable and robust evaluation practices. Providing recommendations for prompt engineering from linguistic point of view is not the decided aim of this study and indeed any rec- ommendations that could be derived from our re- sults appear localised to the context of a particular models or dataset. We publicly release our set of 550 prompts in Appendix D, as a basis for further research on LLM behaviour through the lens of linguistics.
Limitations
In this work, we experiment with LLMs of up to 30b parameters that are able to perform a variety tasks in a zero-shot fashion. We did not include larger open-source LMs or OpenAl's GPT-3 due to (computational) cost. Further at the time of writing, the OpenAl API provides only the top 5 log probabilities given any input. This stands potentially at odds with the evaluation procedure of making prediction based on which answer option receives the highest log probability, independently of whether that answer option occurs in the top 5 log probabilities. We did not include results for smaller variants of OPT (IML) and LLaMA, since they did not perform significantly above chance across all our tasks in the zero-shot setting in initial experiments.
While our work does not include tasks close to real-world applications such as hate speech de- tection, our findings indicate that performance on such tasks might also vary considerably under dif- ferent instructions. We thus advise NLP practi- tioners working sensitive applications with very large LMs to carefully evaluate model performance across broader range of semantically equiva- lent instructions. In particular, evaluation reports should include measures of performance variability across prompts, seeds and demonstration examples. Further, prompts that have been engineered based on a particular model and dataset should not be transferred to other datasets or domains, since in general universality of optimal prompts cannot be assumed. Developers of LLMs for hate speech de- tection and related tasks should account for instabil- ities, particularly when using LLMs for automatic annotation.
So to not introduce an additional source of variation in our experiments and observe the effect linguistic variation in as much isolation as possi- ble, we did not include experiments on in-context learning or priming (Lu et al., 2022; Zhou et al., 2022; Zhao et al., 2021; Min et al., 2022). We fo- cus on the zero-shot setting only, so that models infer given task only based on instruction without any demonstrations.
Future experiments could include more exper- iments on other architectures such as encoder- decoder models, e.g. T5 (Raffel et al., 2020), TO (Sanh et al., 2022) or Flan-T5 (Chung et al., 2022), or multilingual models, e.g. bloom (Scao et al., 2022) or bloomz. (Muennighoff et al., 2022). In- vestigating model behaviour based on linguistic properties in languages that are morphologically richer than English would equally pose an interest- ing avenue for further research.
In future research, we would also like to draw a comparison with an instruction-tune tuned version
html

---

Acknowledgements
of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936.
We thank our anonymous reviewers for their in- sightful comments. The work for this publica- tion is financially supported by the project, 'From Learning to Meaning: A new approach to Generic Sentences and Implicit Biases' (project number 406. 18. TW.007) of the research programme SGW Open Competition, which (partly) financed by the Dutch Research Council (NWO).
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. (xXiv:8803.05577V
Ishita Dasgupta, Andrew K. Lamppnen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2022. Lan- guage models show human- like content effects on reasoning. ArXiv:2207.0705 [cs].
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Mark Davies and Jong-Bok Kim. 2019. The advan- tages and challenges of" big data": Insights from the 14 billion word iweb corpus. Linguistic Research, 36( 1-34-
Marie-Catherinn De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23. pages 107-124.
Boxi Cao, Hongyu Lin. Xianpei Han, Fangchao Liu, and Le Sun. 2022. Can prompt probe pretrained language models? understanding the invisible risks from a causal view. In Proceedings of the 60th Annual Meet- ing of the Association for Compulational Linguistics (Volume 1: Long Papers), pages 5796-5808, Dublin, Ireland. Association for Computational Linguistics.
Milton Friedman. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the american statistical associa- zion, 32(200):675-700
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin. Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsters Douglas Eek, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. ArXiv: 2204.02311.
Deep Ganguli, Danny Hernandez, Liane Lovitt. Amanda Askell, Yuntao Bai, Anna Chen, Tom Con- erly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds. Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kermian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, and Jack Clark. 2022. Predictability and Surprise in Large Generative Models. In 2022 ACM Confer- ence on Fairness, Accountaillity, and Transparency, pages 1747-1764, Seoul Republic of Korea. ACM.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 3816-3830, Online. Association for Computa- tional Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction- issruction-iinetuned language models. arXiv preprint arXiv:2210.11411
Andrew Gelman and Jennifer Hill. 2006. Data analysis using regression and multtleve//hierarchical models. Cambridge university press.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings
Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.040:

---

Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form com- petition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7038-7057 Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022. Cutting down prompts and parameters: Simple few-shot learning with language models. In Find- ings of the Association for Computational Linguis- ficx: ACI. 2022, pages 2824-2835, Dublin, Ireland. Association for Computational Linguisties.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel. and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Lonng Papers), pages 8086-801 8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Yoichi Ishibashi, Danushka Bollegala, Katsuhito Su- doh, and Satoshi Nakamura. 2023, Evaluating the robustness of discrete prompts. In Proceedings of the 17th Conference of the Earopean Chapter of the As- sociation for Computational Linguistics, pages 2373- 2384, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru. Todor Mihaylov. Dániel Simig, Ping Yu, Kurt Shus- ter, Tianlu Wang. Qing Liu, Punit Singh Koura, et al. 2022. Opt-iml: Scaling language model instruc- tion meta learning through the lens of generalization. arXiv preprint arXiv:2212. 12017.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 201 I. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Mohit lyyer. John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Pupers), pages 1875-1885, New Or- leans, Louisiana. Association for Computational Lin- guistics.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(1 1):3991111
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048- 11644, Abu Dhabi, United Arab Emirates. Association for Computational I Linguistics.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf, et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint arXiv:2211.0m 10017...
Abdullatif Köksal, Timo Schick, and Hinrich Schütze. 2022. Meal: Stable and active learning for few-shot prompting. arXiv preprint 2..00:2211.08358.
K Pearson. 1895 Notes on regression and inheritance in the case of two parents proceedings of the royal society of london, 58, 240-242. K Pearson.
Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. 2022. Can language models learn from explanations in con- text? In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 537-563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song. John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George Van Den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat Mcalcese. Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gri- bovskaya, Domenic Donato, Angeliki Lazaridou,
Chonghua Liao, Yanan Zheng, and Zhilin Yang. 2022. Zero-label prompt selection. arXiv preprint arXiv:2211.04665
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Computing Surveys, 15(5):1335.

---

Arthur Mensch, Jean- BannBBapistee Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien De Masson D' D'autume, Yujia Li. Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego De, Las Casas, Aurelia Guy. Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger. lason Gabriel, William Isaac. Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer. Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. 2022. Scal- ing Language Models: Methods, Analysis & Insights from Training Gopher. Technical report. ArXiv: 2112.1 144662..
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631- 1631-1642, Seattle, Washington, USA, Association for Computational Linguistics.
Taylor Sorensen, Joshua Robinson, Christopher Ryt- ting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An ieformationnthhoree engineering without ground truth labels. Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 819 1198662, Dublin, Ireland. Association for Computational Linguistics.
approach to prompt
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Charles Spearman. 1961. The proof and measurement of association between two things.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. 13X1: [cs].
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel Bowman. 2019. Superglue: A stick- ier benehmark for general-purpose language under- standing systems. Advances in neural information processing systems, 32.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Learning Representa- tions.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task -tsk benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP. Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 2rxXi:2211.051000
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistiex: Humun Lunguage Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2352, Online. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Zhao Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. 2021. Finetuned language mod- els are zero-shot leamnrr. In International Confer- ence on Learning Representations.
Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023. On second thought, let's not think step by step! bias and toxicity in zero- shot reasoning.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4235, Online. Association for Computational Linguistics.
Frank Wilcoxon, 1992. Individual comparisons by rank- ing methods. Springer.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen,

---

Clara Ma. Yacine Jernite. Julien Plu, Canwen Xu. Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. ArXiv: 2205.01068.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference Machine Learning, pages 12697-12706. PMLR.
Denny Zhou, Nathanael Schärli, Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le. and Ed Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.1062. 10625.

---

A Example prompt
SST
IMDB
RTE
HANS
ind.
86.6
94.6
86.1
76.1
In this section, we give an example prompt for sentiment classification featuring random input sample from SST-2.
mood
inter.
87.3
95.8
88.8
77.5
imp.
88.2
96.1
90.9
77.3
Table 6: Average accuracy per prompt category mood (indicative, interrogative, imperative) for Flan-T5 XI.
Prompt:
Do you want to watch this movie based on this movie review?
on SST-2, IMDB, RTE, HANS. Highest accuracy per dataset marked bold.
RTE
HANS
An entertaining British hybrid of comedy, caper thrills and quirky romance.
entailment
95
77
implication
96
78.4
Choices: yes or no? Answer:
assertion 91
81.5
Answer choices:
claim
92.5
81
yes, no
Table 7: Average accuracy per prompt in category syn- onymy for Flan-T5 XL on RTE, HANS. Highest accu- racy dataset marked in bold.
True answer: yes
SST
IMDB
B Average runtime
ywmuouus
appraisal
83.4
85.7
commentary
77.7
80.4
On average, evaluating LLaMA, OPT or OPT-IML 30b on all prompts for one dataset took approxi- mately two hours using four NVIDIA A100 GPUs. Evaluating OPT or OPT-IML 1.3b on all prompts for one dataset took around 15 minutes using one GPU.
critique
81.5
85.8
evaluation
83.7
88.4
Table 8: Average accuracy per prompt in category syn- onymy for Flan-T5 XL on SST-2, IMDB. Highest accu- racy per dataset marked in bold.
Performance variability in encoder-decoder models
per property, eg. for mood vs. aspect, might differ, since not all prompts listed under mood (Eg., 's this good movie?") passive voice. They thus omitted from active and passive. We detail accuracy per prompt for all models used in our experiments.
preliminary experiments we also considered encoder-decoder models, e.g. Flan-T5 (Chung et al., 2022). We later restricted our experimen- tal setup to decoder-only models, since they are more widely used to-date. To demonstrate perfor- mance variability similar to the main results in the paper also for an encoder-decoder model we in- clude illustrative results on mood and synonymy for Flan-T5 (XL) for sentiment classification and
be phrased in active and
E Perplexity per category
In Tables 15, 16, 17, 18. 19, 20 we list perplexity scores for each linguistic property and model in each task.
NLI in Tables 6, 7, 8.
D Performance per prompt
We list all prompts used in our experiments for sentiment analysis (Table 9), NLI (Table 13) and Question Answering (Table 11) grouped by the grammatical properties we investigate. Properties we investigate mood (indicative, imperative, in- terrogative), aspect (active, passive), tense (past, present, future), modality (can, could, may, might, must, should, would), and synonymy (different syn- onyms per task). Note that the number of prompts

---

property
prompt
LLMMA 306
1.36
UPFINL 1.36
UPT 30b
OPT HILL JUb
SST
SST
IMDB
IMDB
cnall prompee
R1.2
7258
41.3
64.2
128
65R
74.8
37.0
1N6.
You find this movie nestow positive
64.2
14.00
59.4
67.0
91.4
89.3
K7.4
65.2
91.4
59.h
inc.
This is good movie
52.4
88,33
47.0
77.2
35.8
79.4
53.2
70.6
59.0
66.8
ind.
The label for this movic review pasitive
64.2
92.67
58.3
75.6
92.0
92.3
81.8
65.6
90.6
60.8
You seviow makes people think
based this novee seview
55.0
91.67
85.6
64.6
89.2
92.6
good movic
63.6
91.33
43.8
77.2
46.4
97.3
neview makes people want watch movie
79.6
56.8
97.
ind.
This movic neviow is posilive
80.6
92.67
58.0
75.8
63.0
92.0
67.0
90.4
ind
You want to wasch this muvie based on this movie review
78.4
91.0
61.2
85.0
95.2
7000
75.1
91.4
76.8
inier.
Da you find this movie voveew prsdtive?
90.2
90.3
40.8
93.6
94.6
8O.2
56.6
91.2
K3.4
iner.
Is this a good movie
53.0
93.5
58.2
64.2
91.6
S0.5
73.8
75.2
92.8
91.6
inter.
Is the label for this asoiie ceview positive
56,0
90.0
90.4
59.5
928
95.4
R5.6
61.2
92.6
91.8
inerr.
Do like the
based this
63.8
89.3
39.1
59.5
95.0
90.8
85.2
this
resiew uuke people thisk good mavic
50.4
K5.0
79.2
94.4
this movic
make people
watch
67.0
884
60.6
934
insr.
movie seview positive
58.0
90.7
90.3
54.0
93.8
94.6
6122
22.2
inter.
Do you want watch music hased this movic mvviww
K7.0
85.7
88.4
56.2
92.0
93.
83.0
76.6
92.4
87.0
inp.
Tell me if you find this movic roview positive
86.8
94.67
85.5
67.2
93.2
94.0
86.6
60.2
91.8
K2.2
imp
Tell me if this is good movie
67.4
94.0
73.6
73.6
92.2
84.8
71.0
77.2
92.6
K5.0
imp.
Tell me if the label for this movie review positive
90.0
92,33
76.8
73.8
92.6
54.6
8LD
61.4
93.2
91D
Tell if like the movis hasrd this morie meriew
63.8
95.0
80.0
71.0
92.0
94.3
87.8
69.4
92.6
85.6
Tell
makes people think
good movis
70.2
N3.67
68.0
$1.4
91.8
93.8
Tell
makes people
watch
66.8
93.67
75.2
T8.6
93.0
70.6
imp.
Tell
asoiie review is positive
$7.4
93.0
80.5
71.6
93.2
94.6
45.6
92.4
inp
Tell me you want to watch this movie besel this movic reviow
76.8
93.0
74.6
75.4
90.2
94.4
83.2
75.4
81.2
active
Do people axmoder ihis movie miie positior
K7.4
V1.87
854
M1.
95.6
V..66
65.6
66X
V2.0
91.2
active
Do people label this movie restew as postive
57.8
89.33
80.0
64..8
93.6
95.0
80.2
65.4
91.5
90.8
sctive
Do people like this movie
67.4
93.07
81.0
68.0
93.0
90.2
75.2
73.4
93.0
93.0
this movie review make people think that is good cousidered positive by people
63.0
N2.0
77.6
64.2
93.0
94.0
paseive
this movie
$8.8
94.0
87.6
59.2
93.4
94.4
passive
this movie seview labelled as posstive by people
55.4
94,33
88.0
60.2
93.2
94.4
69.2
69.6
9/1.2
91.6
paseive
this movic Iied people
K5.4
91.0
89.3
59.8
92.8
90.6
72.8
92.6
93.6
pisdive
Are people mude think by this reviow that this is good muvie
51.4
92.67
77.8
64.2
91.8
93.3
79.2
71.6
93.4
94.2
cott
Can you fidd this movie review positive
K3.2
94.67
87.6
56.0
95.0
93.8
K1.6
67722
92.4
19.33
coll
Can this be good movie
56.8
87.33
85.2
66.8
91.8
79.8
K8.2
73.8
93.5
90067
com
Can the lahell for this movie ueview posiive
68.4
93.33
82.6
70.6
92.4
93.2
83.0
68.2
91.2
91.0
besed this movie review
56.6
91.33
88.6
65.2
92.88
92.6
KT.67
yeople flink
good
70.4
89:67
65.8
67.4
91.0
89:67
cit
Can movie seview make people want wutch this movie
54.2
83.33
82.6
68.0
93.0
21.3
75.8
79.2
93.0
88.0
com
Can movis miiis positive
65.4
89:67
83.3
68.0
92.8
81.0
75
70.8
92.8
00.33
con
Can you want to watch this movie based on this movee review
57.8
91.33
38.4
16.2
91.8
93.0
86.2
76.4
92.0
86.33
cuild
Could you find this movie miview positive
67.2
93.
82.6
54.2
9300
9344
W9.8
62-4
92.4
K3.67
could
Could this be good movic
89.5
87.0
75,5
68.2
91.8
79.8
05.0
71.2
93.4
89.33
could
Could the label for tlis movie review he positive
66.4
94.67
82.4
68.4
93.0
90.4
78.4
70.0
93.2
90167
could
Could like the movie based this
66.4
91.67
85.4
66.2
93.0
92.8
84.33
could
Could this movic review make people think this good movic
67.6
93.67
554
70.0
90.4
could
Could this movic review make people want watch asevie
55.0
88.0
74.0
7LS
92.8
91.4
90.6
92.8
87.67
could
Could this movic review he positive
67.0
89.0
76.0
68.0
92.8
91.
712
92.4
90.33
could
Could you want watch this movie based on this movie review
83.8
91.67
$8.2
65.2
90.2
95.
80.6
75.4
92.0
80:67
may
May you find this mosic reviow posilive
63.6
R5.
89.2
57.2
92.4
92.0
86.8
65.0
92.2
89:33
may
May this be a good movee
89.4
B9.0
82.2
70.0
92.6
834
73.8
75.0
93.4
90087
May the label for this movie review be positive
79,8
86.0
80.0
01.4
92.4
89.0
85.0
63.8
92.4
90.0
you like the movie based on this
review
61.8
86.0
84.6
62.5
92.6
95.0
84.67
people tlink this
mnoiee
66.8
88,33
70.6
69.0
91.6
may
May this movie review make people want
movie
76.8
$1.67
83.6
67.8
93.6
92.6
72.8
V2.4
may
May this movie review be gusitive
30.8
83.0
82.9
66.4
92.2
91A
70.0
90.33
may
May you want watch
movie fooad um music rexiow
63.0
92.67
88.0
59.0
90.0
93.6
85.8
74.4
92.0
70.0
might
Might you find this movie review positive
67.8
90.67
87.6
62.6
93.2
93.8
8A.8
66.4
92.0
90.33
might
Might this a goud mowis
64.2
89.0
78.4
71.
92.4
82.6
61.0
77.
91.4
63.33
might
Might the label for this movie review be positive
89.6
93.0
82.4
89.5
92.6
92.0
53.4
6910
92.8
9LD
might
Might like the movie booud un this musie review
67.0
91.0
80.8
67.2
91.2
91.8
76.8
73.8
75.33
might
Might this movic review make people
this
movie
62.0
95.0
58.5
76.2
92.4
92.2
might
Might this movie review make people
warch
66.2
87.67
76.6
73.2
9300
85.8
73.8
92.8
might
Might this movic review be positive
55.6
90.67
25.4
08.5
93.0
91.4
69.6
92.8
90.33
might
Might you want watch this movic based on movic reviow
$3.6
93.0
77.4
60.0
89.6
92.0
75.6
92.4
72.33
mist
Must you find this movic reviow positive
$1.8
89.33
89.8
$1.4
92.6
95.6
89.0
58.2
92.2
90.67
must
Must this he good movic
78.4
87.0
88.0
65.2
91.4
74.3
K1.4
72.6
93.0
84.67
mest
Must the label for this movie review be positive
62.6
95.0
$2.8
89.2
92.8
$0.0
W6.0
56.8
93.0
90.33
must
Musi like the movis bound on this mosie review
73.6
94.33
90.2
59.4
93.6
92.6
89.2
70.0
92.4
86.67
Must this movie rexlew make people think
good movie
63.2
21.8
61.5
92.8
950
89.0
this movie
people
watch
89.8
86.33
84.6
61.8
94.2
87.33
mist
this movie resiew be positive
66.5
90.0
88.6
54.8
94.3
61.8
92.6
89.07
must
Must you want wutch this movie based on this movic review
58.2
92.0
89.0
51.8
91.
94.0
71.8
92.9
85.67
should
Should you dind this movee review positive
89.5
92.0
38.5
58.0
93.6
95.3
81.2
68.4
92.0
83.67
should
Should this be a good movie
89.2
83.67
87.6
68.4
91.0
73.0
87.0
71.2
93.6
83.67
should
Should the lahee for this movic moiew he pusitive
53.5
94.0
86.6
65.8
93.2
93.1
65.8
56.4
91.4
91.0
should
Should you like the mavie based on this movic review
65.8
91.33
82.6
64.8
93.0
92.2
B5.2
73.8
92.2
88.67
should
Should this moviee movies make people think this is good movie
$7.8
94.33
73.2
66.0
92.0
91.4
43.4
79.
93.0
83.33
should
Should
zeview make people neview positive
watch this monie
$2.6
857.33
88.2
63.4
93.2
90.1
Should
K7.4
89.5
57.2
86.87
should
Should you wast to watch movie based this movie review
$7.4
91.67
87.0
61.0
928
93.1
76.0
92.9
87.0
would
Would you find this movie nestew posioive
89.5
95.33
90.6
$1.5
93.2
$4.3
89.
51.8
91.8
8033
would
Would this be a good morie
$8.0
D0L0
88.1
64.2
91.6
81.
70.0
76.8
98.2
88.67
woild
Wonld the lahel for this movie review be positive
58.8
94.33
88.0
65.8
93.2
93.0
89.4
57.6
92.2
69:67
would
Would you like the movie based en this movie revizw
55.4
93.0
87.2
61.8
93.6
93.4
82.2
75.0
92.2
82.0
woild
Would this movic reviow make people think this is good muvic
71.2
93.0
72.2
68.8
928
9200
27.4
80.8
92.8
85.67
would
Would this
make people positive
watch this movie
90.2
88.
86.0
66.5
934
926
82.8
86.33
wmild
World this
62.5
87.4
93.4
84.67
would
Would you want to watch this movee based this movie review
92.0
94.0
38.5
57.8
92.8
930
77.2
$2.0
84.67
Table 9: Detailed list of prompts for Sentiment Classification with accuracy per prompt on SST and IMDB across all models. Part 2.

---

projeity
prompe
SST
IMDB
SST
IMDB
SSE
IMDH
SST
IMDB
SST
IMDB
Does this move uppr, make people think this is good movid
72.
93.61
792
61.6
93.4
89.2
30.5
R00
93.6
$1.0
Does this movie appr. make people want watch this mnvie
89.0
91.67
85.0
65.6
93.4
91.4
09.5
90.8
93.5
92.9
this
58.2
91.67
88.3
61.0
93.4
90.4
67.4
appc.
apps. posttive
93.67
90.4
app-
the label this appe. movic apps.
79.2
90.0
91.8
appc
Do you movic
88.2
1pp.
you want worch this movie heool on movic appr
780
95.31
89.4
57.2
93.4
91.0
77.4
oumm.
Does this movie comm. make people thint this good moste
653
91.33
76.6
56.4
92.2
61.2
23.22
77.8
90.5
95.8
comaa
Does this movie comm. make people wast watch this movie
58.6
91.33
85.2
19.4
92.8
85.0
39.4
83.2
93.6
92.9
comms
Is this movie comm. posifive
88.4
90.67
86.2
61.2
93.4
90.4
70.2
67.2
94.0
93.8
87.6
95.67
89.2
the lakel
positive
65.4
92.67
89.3
87.6
Do you
movic
movic comum.
Do you want worch movie on
comm.
77.6
93.6
78.6
crtiquee
Does this movie cillique make peogle think this is good movic
68.6
93.0
72.8
$7.8
91.2
912
23.6
90.2
92.6
95.0
cetique
Does this movie critique make people want watch this movie Is this movie critique positive
87.4
90.0
83.4
0U22
92.0
91.0
47.0
838
92.2
91.2
criique
88.8
90.0
87.8
60.4
93.4
91.2
76.6
06.4
92.6
92.8
positive
906
91.8
the takel this criique positive
87.2
91.67
95.2
criiiuue
you
cnisgge
criique
Do you vorch movie
crilique
866
93.67
87.6
$7.0
91.2
90.2
76.4
evaluation
Does this movie evaluation make people think this is good movie
30.6
93.0
88.6
58.2
92.4
89.2
43.5
75.4
9344
93.6
evaluation
Does this movie evaluction make people want watch this zovt
36.6
$3.07
912
$1.8
93.8
91.0
72.6
MO.
92.5
$1.9
evalarion
Is this movis evuluation positive
39.0
90.0
91.2
$3.0
93.8
94.4
$2.4
66.8
92.2
93.0
etalutioon
Do find this music evalustion positior
$6.4
92.67
90.3
47.6
94.0
93.3
18.4
$3.0
91.5
69.2
the lakel
evalution pooitive
94.4
you
89.0
evaluation
you
movie evaluation
review
Do you find this movie resiew positie
90.2
93.0
89.8
49.0
93.6
94.2
99.2
55.8
91.1
$4.4
reviow
the hihel for this movie rivicw posiive
85.8
91.67
90.4
59.2
92.8
94.8
15.6
61.0
92.6
92.8
review
Do you like the movic based on diis movic revicw
30.4
95.33
89.2
59.2
93.0
90.4
58.2
72.6
92.2
$7.4
rosiew
Does this movic review make people think this good music
66.4
94.0
79.3
62.2
92.8
93.6
45.4
78.2
92.2
90.0
Does
people
this movie
91.0
90.33
88.1
92.8
positior
you
77.6
Table Detailed list of prompts for Sentiment Classification with accuracy per prompt on SST and IMDB across all models. Part 2 of 2.

---

peop-
pronge
LLMA 306
OPT 1,36
OPEINE 1.36
OFT 30h
OPT-IME 30t
BoolQ
ARC-E
BosQQ
ARC-F
BouiQ
ARC-
BealQ
ARC-E
BoxiQ
ARC-F
randum
50
50
cmull prompes
64.0
61.5
680
20.79
680
78.08
72.0
61.64
1nd.
You ans the
K1.0
19.17
62:0
50.6
62.5
32:16
66.00
30.3
75.0
53.00
utd.
You choese the best ens, the
60.0
66.67
62.0
29:15
62.5
32.66
00.0
31.31
72.0
64.65
ind.
You choose this oas.
64.0
60.70
62.0
27.14
62.0
33.67
43.0
22.22
69.0
66.67
This
77.0
65.0
31.16
67,68
This the
the
81.0
76.04
63.5
32.66
64.65
You give the contect
77.
79.17
22.61
64.0
29.55
69.0
You the comeett ans.
82.0
73.96
62.0
25
64.0
30.15
33.33
70.0
ndd
You pock the cornect ans.
66.0
73.0
02.0
25.63
633
30.55
42.0
30,3
70.0
69.7
idd.
You selecs the most vuitublle ons,
68.0
70.83
62.0
29
62.0
33.67
42.0
33.33
68.0
68 68.69
nd.
You solve the by chousing the correct ons.
72.0
70.04
61.5
26.65
65.0
35
47.0
39.19
72.0
72.73
ad.
You tell which ass, is contect
63.0
63.54
62.0
25.15
64.5
29.15
61D
34,34
70.0
65.
You think
the
73,0
620
31.16
71.9
you ane the
71.0
60.3
30.3
inter.
Cuuld you choose best
99.0
19.17
62.5
29
63.0
32.16
37.37
68.0
inter.
Which ans. do you choose
66.0
T8.
62.5
24.64
65.0
2965
620
38.08
65.0
63.61
utter.
Which acs. is conrect
61.0
T3396
61.5
25.13
65.5
27.4
60.0
32.32
70000
63.06
inter.
Which the correct ans. to the
76.0
73.94
610
25.04
65.5
32.16
67.0
31.31
09.0
54.65
intec.
Could you give me the correct ans.
70.0
TL,88
60.5
28.64
64.5
30.65
09.D
40.4
67.0
64.65
inter.
Cuuld you infor the corrct ane
84.0
77.08
61.5
28.14
65.0
31.66
76.0
37.37
69.0
64.65
Could you gick
09.0
76,04
64.3
31.16
65.66
Could you
the snituble
61.0
76.14
34.17
64.65
atter.
Could you selve the q choosing the conect ins.
70.0
76.04
62.5
28.14
64D
33.17
69D
37.37
70.0
64.63
inter
Cuuld you cell which one is coems
64.0
72.90
61.0
29
64.0
30.5
68.
33.33
69.0
utter
What do you think is the correct ans ass, the q.
00.0
T3.96
62.5
25.64
65.5
31.16
62.0
36.36
70.0
56.87
65.0
79.17
62.5
32,66
63.5
30.15
66.0
30.3
73.0
67.68
Choose the best ons, the
65.0
70.83
62.0
28.14
63.0
32.66
68.0
32.32
72.0
63.66
Tell which you choose
62.0
T2.96
65.0
30.15
69.0
65.66
imp-
Tall which
62.0
20.65
67.68
amp-
Tell which corect
71.0
27.64
60.5
31.16
71.0
imp-
Give the cormast sms.
78.0
73.96
61.0
26.63
64.5
33.17
68.0
34.34
69.0
69.7
ump-
Inder the correct ans.
78.0
76,01
62.5
23.64
06.5
31 66
7300
35.33
72.0
68
amp.
Pick the courect ans.
61.0
77,08
62.0
25.14
64.0
31.16
58.0
32.32
71.0
65.66
ump
Select the most suitable ans.
62.0
T5.0
62.0
26.14
65.0
35
70.0
26.26
89.0
67.68
Solve dee choosing the connet and.
50.0
T2.92
62.0
27.64
64.5
34.67
71.
35.35
7L.0
65.66
Tell what think
39.0
69.5
33.17
71.0
53.66
aci.
you ane
71.0
36.1Y
63.82
act.
Could you choose the best
59.0
76,96
29.15
63.0
32.16
37.19
61.5
66.83
act.
Which ars. do you choose
66.0
78.01
62.5
23.64
65.0
2965
58.5
37.09
63.0
66.83
act.
Could you give aee the correct ans.
70.0
74.35
60.5
25.64
64.5
3065
62.5
38.19
62.0
65.33
act.
Could you inferr the correct ans.
61.0
76.44
61.5
25.14
65.0
31.66
63.0
36.15
62.00
66.33
oct.
Could you pick the courect ans.
69.0
T7.49
62.5
25.14
64.5
31.16
6LD
38.19
62.0
67.34
acl.
Cuuld you select the most soitcble
63.0
76.44
62.5
31.66
63.0
3417
60.0
38.19
62.0
67.84
Could you the choosing
70,0
75.39
64.0
33.17
66.33
Curkd you sell which
64.0
71.73
30.65
68
oct.
do you think
correct
60.0
T5.92
25.64
69.5
31.16
37.5
34.67
63.0
67.34
pass.
Could the q. be ass.ed
74.0
70,08
63.0
$1.16
6500
2764
6300
32.36
63.5
50.3
pess.
Could the hest ans, to the q. he chosen
74.0
74.35
62.5
27.14
62.5
30.5
61.0
28.14
63.5
63.82
pass.
Which ass. could he chosea
71.
T4,35
62.5
26.63
64.5
30,55
6L5
33.67
65.0
66.83
Could the cornact ans. he given
74.0
72.25
62.5
26.63
63.0
31666
61.5
32.166
63.0
62.31
Could cornect be inferred
77.0
66,49
28
66.5
27.14
61.5
32.16
61.5
62.31
gass.
Cou'd
he selected
71.0
75:92
20.15
61.52
poss.
most suiable
75.0
70,68
27.64
62.5
32.16
33.07
60.5
65
pass.
Could be solved by choosing the contect ats.
75.0
71.73
63.0
30.55
60.0
34.67
63.0
62.81
pass.
Could it be sold which ans. is conrect
71.0
73.39
61.0
29:85
62.5
29
00.3
31,06
63.5
06.33
pass.
What thought to be the connect ats.
76.0
78,01
62.0
30.65
64.5
31.16
62.0
36,08
62.5
67,84
can
Which ans
choose
61.0
75.92
63.0
25:15
64.0
30.55
60.5
35.68
65.55
66.83
Which
be corect
59.5
72.25
61.5
21
66.0
29.15
62.5
33,67
63,5
65.3 65.33
which the conrcc
66.5
89.03
65.0
32.16
67.34
cun
cornect ans. Which ars could you choose
do you think can
71.0
77.49
62.5
33.67
60.5
65.33
coull
64.0
15.02
64.3
31.16
35.8
64.0
66.81
could
Which ass, could be conect
$7.0
72.25
61.5
27.64
64.0
28.44
60.5
34,67
63.5
66.33
ould
Which codl be the cumect anc to the
64.0
69.63
61.0
27.64
63.0
62
630
32.66
63.5
67.84
could
What do you think could be the correct ans.
70.0
77.49
62.0
21.64
61.5
32.66
60.0
33.17
64.0
64.82
Which
62.5
78,00
64.0
32.66
06.33
may
Which may
$7.5
65.0
30.15
64.0
66.81
aay
Which may
concect ans, to the
64.0
69.11
25.63
64.0
32.16
62.0
29.15
63.0
67.34
miy
do you think may he oorrect iIs..
66.5
76.44
62.5
15
63.0
32.16
34/67
63.5
63.83
night
Which acs. might you choose
63.0
75,39
63.0
25.64
65.0
32.16
61.5
33.17
62.0
66.83
might
Which ans, might be correct
55.5
T4J35
61.5
25.64
68.0
29.65
6LD
33.67
63.0
63.83
might
Which might be the coneect and to the
63.0
72.77
61.0
27.64
63.5
31.16
61.0
31,66
62.0
66,83
might
do think might be
70.5
79.58
650
32.66
64.82
mue
Wiich mux you choose
67.0
70.96
30.5
83.5
67.81
mus
Which ass. must be coneec
56.5
76.44
27.64
64.0
28.4
28.64
66.0
65.83
muet
Which must be the correct ans. to the
70.5
12.77
60.5
28.14
63.5
32.16
60.5
29.15
66.0
67.84
must
What do you think must be the courect ans.
70.0
76,96
62.5
29:65
62.5
3065
$7.5
35/68
64.0
66.33
should
Which ans. should you choose
65.0
75.92
62.0
$1.16
64.5
30.65
56.0
23.12
64.0
68.34
should
Which should be correct
65.5
75.93
61.5
28.64
64.5
29.15
57.0
30.15
64.5
67.84
should
Which should be
69.0
74.35
65.0
30.65
67.84
do you think should
67.5
65.5
31.16
64.5
would
Which ans would you choose
67.5
75.00
64.0
X6T5
30.5
35.8
63.0
66.1
would
Which ans. would be conrcct
64.00
77.49
61.0
29.15
64.0
28.14
60.5
34.17
64.0
66,83
weuld
Which would be the curmect anc un the
68.5
75.90
60.5
14
64.5
31.16
63.0
34.17
63.5
68. 54
would
What do think would be the correct ans.
73.5
76.96
62.5
29.15
63.5
30.65
59.5
35.18
64.5
66.33
Table 11: Detailed list of prompts for Question Answering with accuracy per prompt on BoolQ and ARC-E across all models. Part 2.

---

poop.
peorapt
LLaMA
OPT 1.36
UFF
30
BoolO
ARC-E
BoolQ
ARC-E
BoolO
ARC-E
BoolQ
BaslO
ARC-E
pm.
Could you chasss the peu ins to the
6200
17.15
614
27.16
64.0
14.51
62.6
11.30
652
67.61
pmo
Which ans, pro.
67.0
75.89
62.4
25.75
6-122
24,18
58.6
10.38
61.1
68.81
Which the pro, cas, the
70.2
73.55
61.8
26.36
63.2
33.8
60.0
30.99
65.0
70.02
Could give the
08.2
73.79
64.6
61.2
52.19
68.61
Could pio.
pro, you pick the pro,
768
76.94
64.0
31,79
68.01
pro.
63.2
79:01
63.4
62.2
31.59
08.81
pio.
you select the pro. ans.
61.2
79.87
29.14
63.6
34.81
62.6
34.61
63.8
68.61
pm.
Could you salve the ij by chousing the pro, ane.
63.6
76. 31
28.17
618
35:21
61.2
32.6
64
67.61
pro.
Could you tell me which ars. is poo.
64.6
68.76
62.2
25.06
63.4
31.39
6L88
33.2
63.2
70.22
pro
Whee du you think is the pro, ane.
63.4
77 99
632
27:57
63.4
34.81
62.6
32.6
64.4
68.41
Could choose the night
58.6
79:01
27.36
61.2
33.6
67.8
$1.39
67.4
Which
50.4
72:75
31.79
68.11
Which right
the you give me right uus.
71.0
74-21
69.01
right
69.2
74.84
27.36
64.6
35.01
31.79
64.4
68.41
right
Could you infer right ane.
790
77.57
27.36
64.3
34.0
62.2
29,38
63.8
66.8
right
Could you pick the right ans.
6288
78.62
634
26.36
63.
34.81
62.2
29.78
61.2
68.21
right
Could you whln the right ane
56.0
70:04
634
27.36
63.6
34.21
62.6
35.01
63.8
68.21
right
Could you solve the q by chooking the right ans.
61.0
77.15
63.4
27.77
64LA
35:21
61.0
$2.39
61.4
66.6
right
Coold tell which ins,
59.0
89 39
613
27.16
63.6
336
60.4
33.4
63.6
69.82
What think is
57.6
75.41
35,01
68.01
onr
Which ans, coer.
56.0
75.68
613
25.75
64.4
20.18
55.0
31.19
65.0
com:
Which the corr ans to the
69.0
73/79
60.4
27.77
63.6
33.4
59.5
29.78
61.4
69 22
oim
Goullyn give me the corn. ane.
678
74.84
614
28.17
64.4
34.21
62.4
32.6
64.2
67.2
com:
Could you infer the cor: ane.
71.6
77.78
6188
26.57
6412
33.6
62.5
10.55
63.0
67.2
con
Could you pick the core ans.
59.4
75.62
63.2
27.36
63.4
33.5
60.8
30.75
64.0
68.81
Could
60.0
75.01
63.2
35.01
60.0
53.6
64.4
68.01
Could
the by chooiiig
61.6
77.15
63.8
35.61
32.19
64.4
67.61
Could
which
61.4
634
90.2
69.22
con:
What you think is oon ars.
$5.0
77.57
63.0
27.16
63.5
35.41
57.6
31.99
64.8
68.81
app.
Could you choose the appe. ass, to the
63.0
77.99
634
27.16
64.4
33.6
63.4
30.38
65.0
63.41
appr.
Which ans, appr.
69.1
75.68
62.4
27.16
63.6
29.88
58.2
32.19
65.6
68.61
apps.
Which the
to the
70.6
73.17
61.
26.56
63.2
33.0
59.2
31.59
65.2
69.82
Could
me the apps, sas.
694
74.21
62.2
27.57
64.2
34,61
63.2
33.0
63.2
69.01
appc
Could infe: appr.
75.2
76.52
62.4
sppr
Could pick appe
64.4
77.99
$0.10
69.62
appr
select appt
75.41
64.0
35.01
68-81
appr
Could you salve y by chousing the uppr, ans.
62.8
75 89
28.57
64A
34.11
61.8
31.09
64.6
67.61
appr.
Could you tell me which ans, is appe
67.6
71 28
6200
2157
63.8
31,99
N0.
33.2
61.0
7002
epp
Whet du yus think is the uppr ans
67.6
27. 36
630
27.77
63.8
34.41
59.6
33.6
65.0
68.81
Could choose the hest mns. to q.
35.6
76.75
63A
28.57
63A6
33.6
63.0
32.6
63.4
68.21
Which ane do you chouse
63.0
73.83
63.0
26.76
63.8
33.6
$7.2
33.6
65.2
68.1
Which
56.0
75.68
20.18
68.41
ans.
to Could you give me the comn. ans.
Which
ans.
67.8
74 84
614
28.37
64.4
34.11
32.6
64.2
67.2
ans.
Could you infer the cor:. ans.
73.6
77.2K
61.8
28:57
64.2
33.6
62.6
30.58
63.0
67.2
aus.
Could you pick the con. ans.
59.4
78.62
632
27.36
63.
33.8
60.8
30.78
64.0
68.81
ans.
Could you select the most suiaalle ans.
53.6
77.36
63.4
29.18
63.6
35.01
61.4
33.2
63.6
69 82
aus
Could you solve the g by chooiing the cour, and
61.6
77.15
62.6
27.77
63.5
35,61
62.0
$2.19
64.4
67.61
Could tall which ans is 61.4
69.6
61.6
27.36
63.4
33.2
59.2
$2.8
63.6
60.32
What
think is
55.0
77.57
35.41
68.81
repoy
you choose the best reply
60.4
73.05
64.2
35.01
29.13
reply
Which reply do you chouse
59.6
77.57
623
27.77
64.6
33.6
$4.0
30.58
66.2
68.21
repty
Which reply conz
56.8
72.54
G0LD
28.17
63.2
28.57
58.2
32.8
66.4
68.41
reply
Which the ooer. relly to the
68.4
72.33
61.2
26.56
63.6
33.0
99.8
30.58
65.8
69.01
reppy
Could you give me the cour. reply
690
75.47
616
27.36
64.0
34,0
63.6
52.8
64.0
6666
eoply
Could you infer the sor reply
71.2
77.57
62.4
26.66
64.0
33.8
62.2
29.18
63.6
66.6
Could pick the reply
59.0
71.78
27
63.2
33.8
61.6
32.19
63.8
66.6
seply
Could
the suiaable rxply
5600
76.94
35.21
67.4
30.90
68.81
reppy
solve the q by choosing corr. reply
62.8
75.91
28577
64.0
34.81
eeply
Could you
which raply cum
58.4
69 39
60.0
26.76
61.8
30.78
60.0
$1.50
63.8
reppy
What do you think is the con reply
61.6
78.2
62.8
28:57
64.4
34.41
6L.8
$1.59
61.8
68.01
neg.
Could you choose the has reep to thin
623
77.7%
63A
28.17
640
3320
$7.4
$1.50
63.
K7.2
Which rogg, do you choose
64.0
78.62
62.8
28.57
64.0
33.0
58.6
$1.30
65.6
69.62
Which
59.2
75.15
60.6
27.36
63.4
27.97
58.6
31.79
65.6
68.21
sesp.
resp. Could you give me coen. reep
Which
68.4
74.21
32.6
70.22
resp.
71.0
64.2
32.19
67.61
resp.
Could you infes cons resp.
77.4
77.57
26.96
64.2
32.8
62.6
20.38
62.8
6668
resp.
Could you pick the com resp.
65.6
78.83
654
26.76
63.6
34.41
61.2
31.19
63.4
61.01
resp-
Could you select the most soitable resp.
63.4
76.94
6322
27:57
63.6
34.61
62.0
30.18
63.8
69:82
negp.
Codl you salve the i by chousing the oorr. map.
67.8
76.31
63.4
27.16
64.0
36.02
63.4
30.6
64.6
67.2
Could you tell me which
con:
60.4
71.25
59.2
25.96
62.8
31.39
61.0
33.2
63.0
69.01
What do you think is the oom
64.8
77.78
63.4
27.16
64.6
33.8
61.2
32.8
65.4
68.01
sul.
choose
sol.
Which sol you choose
65.4
75:89
27.77
64.0
32.39
59.8
29.95
65.0
6K.11
sol.
Which sol. com.
55.4
27.77
62.5
20.18
29.96
65.6
67.0
sol.
Whichs the con: sol, the 9,
71.4
71.7
622
25725
64.2
33.0
62.8
$1.99
66.0
69.62
sol.
Could you give me the coer. vol.
72.6
73.79
62.5
25.17
64.0
33.2
61.0
$2.19
62.8
66.8
sol.
Could you infer the som wol.
74.2
76.94
62.3
27.36
64.2
33.6
60.0
28.97
63.2
66.6
sol.
Coul pick the com sol.
58.4
77.57
63.6
28.77
64.2
33.2
55.8
31.70
64.4
67.2
sul.
Could
the suitabll sol.
52.2
24.63
63.4
27
64.0
33.6
62.4
28.73
63.6
68.21
Could
qg by chousing which sol, coer.
62.4
75.26
37.02
sol.
Could
59.0
27:17
63.6
32.39
80.8
31.59
sol.
What do you thinic is the con: sol.
62.0
74.84
63.4
28.7
63.6
34.21
61.0
31.99
65.4
67.4
Table 12: Detailed list of prompts for Question Answering with accuracy per prompt on BoolQ and ARC-E across all models. Part 1 of 2.

---

property
prongt
LLaMA 30b
OFF 136
OPFINL 1.36
SPT
OPF
RTE
CB
RTE
CB
RTE
CB
KTE
CB
RTE
CB
nill
45.0
40.5
576
46.5
616
46.6
Given [p] you assame
46
63.8
65.6
46.1
71.8
ind.
Given [p] the shim "Dh3" "s comeect
48.6
47.3
52.4
62.6
63.3
50.4
47.6
73.8
$1.6
Ind.
Given [p] you can deduce that "[n3"
52.4
50.
53.4
56.4
64.0
652
50.6
16.
70,0
80.0
ind.
Gaven [p] it follow tuat "UNJ"
$2.0
47.6
52.4
50.8
65.0
62.4
51.6
46.8
71.2
79.2
inc.
Gaven [p] this implies shat '[h]"
00.2
-8.3
53.2
52.8
03.2
62.8
53.8
18.1
71,2
70.0
ind.
Gaven [[] you can infor thet "Dig"
55.2
51.2
$3.0
53.2
64.0
62.0
50.5
46.0
73A
30.0
ind.
Given [p]
jastified in soying that "[h3"
50.0
48.0
$2.6
$2.4
65.2
62.4
31.2
46.4
76A
84.0
greuise" "[o"" hygonheis "DXd" label
55.2
64.4
Goven [p] you
'Ek]"
51.0
48
64.
ind.
eellltiiossh between [[]] "[h]" ent
40.4
47.3
52.6
54.8
62.0
60.0
48.6
46
67.0
74.8
ind
Given [p] it me (hse '[h]"
49.0
46
$3.6
64.6
60.3
49
47.6
75A
80-4
ind.
"Lo"" Using only the above deseription [hh)" is correct
31.2
46.4
52.4
47.6
63.5
62.8
30.8
46.8
71.1
$1.1
inter.
Gaven [9] you assume that '[h3" ?
52.6
56.3
$1.9
611.
65.0
67.2
51.8
56.8
73.0
74,8
inker.
Given [p] is the claim "Ih"" conect
53.2
48.4
50.8
39.6
67.4
39.6
52.2
66.0
73A
79.6
Goven [[]"
dedace "Dh3"
52.0
64.0
Given [p] doscctt fellow tust "[h]"
62.8
65
iner.
Geven [p] does imply that "Ohl"
52.8
60.
66.0
62.4
52.0
664
inter.
Given [p] tom you infer "D=3"
56.0
76.0
49.0
58.8
65.4
64.8
568
72.0
77.6
iner.
Geven [p] are you justifled in stying that "[h)"
36.2
33.2
50.6
50.8
65.5
39.2
53.4
396
73A4
79.6
iner.
Gaven grenise" [[]"" and hygothess "DId" is the lobel ont.
53.8
46.4
$1.0
49.6
62.5
68.0
49.0
464
75.4
81.2
imer.
Gaven [p] con reason that "[h]"
59.2
59.6
20.
58.0
64.5
60.5
56.0
67.2
75.1
78.0
inter.
& the relatiosship betwees "Co3" and "D'3" esc.
40.2
35.2
48.4
63.6
63.4
$7.6
49.0
46.0
67.5
67.2
Gaven [[D]" it that '[h]"
68.2
66.0
only the
deseription "CNT"
54.0
58.0
65.0
$7.6
78.8
imp.
Gaven [p] tell me you cua assume 'th)"
48.3
53.0
52.0
63.0
66.
49.2
55
77.1
30.0
imp.
Given [p] tell me if the cham "th3' correct
65.8
36.0
53.8
19.1
64.4
58.0
57.0
6912
76.3
25.2
imp.
Gaven [p] tell me if you csa dedace that "DO'"
54.4
51.2
53.6
$2.0
64.4
65.6
49.4
573
77.6
80.4
imp
Given [p] tell me in follow that 'EkJ"
54.6
49.2
$2.6
51.6
64.8
64.4
55.4
60
T1A
78.0
imp-
Given [[]] tell me it this amplics that '[h]'
57.2
65.2
$2.4
4916
63.5
62.4
52.3
56A
71A
75.2
Given [p] tell me if
infor that "D-3"
62.2
61.3
$2.8
56.0
65.0
64.4
49.5
55.6
76.6
81.6
Goven "Io]"
you justiflie in soying that "Dhg"
49.2
Given premisu "[o"" hyputhesss "Dk3"
lehel
52.3
Geven [p]" tell me you can reason that 'Ih]"
51.6
52.8
2044
63.5
612
36.0
7000
804
imp
Tell me if the reltionshpp betwoem "En3" and "Dr3" ent.
46.8
34.0
50.8
54.8
63.8
58.0
$1
49
62A
70.8
imp.
Gaven [p] tell me is tre that "[h]"
68:2
61.6
52.0
52.8
66.5
64.8
34.6
62.4
76,6
80.3
inp.
"Co"" Using only the above descriptio sell me of "[h]" is correct
58.2
48.0
$1.6
$7.6
67.6
52.4
$5.0
72.8
70.3
$4.4
autive
Given [p]
avemme that [63
32.6
36.3
51.0
603
65.
67.3
31A
363
73.0
74.3
active
Goven [p]
conciude that '[h]"
51.4
56.3
66.4
62.4
66
13.6
sufive
Given
you
"DD2"
52.0
$7.6
58.0
active
Goven" [h]"
follow that Given [o]] can you guess that '[h'"
38.8
53.8
59.6
59.6
51.2
30.8
25.6
active
60.2
65.6
52.8
56.8
65.5
64.8
51.0
72.8
74.8
74.4
active
Goven" [p] does this imply that "UhJ"
33.4
61.6
52.8
60.8
66.0
62.4
$2.0
376
70.1
68.
setive
Gaven gremis cas you indor tha "D'3"
53.4
47.2
50.0
50.4
52.8
39.6
40.2
45.6
59.2
47.6
active
Gaven [p] cas justiteely say that "Un."
57.2
51.2
51.4
58.4
05.5
62.0
S7.4
63.2
76,0
76.0
sstive
Given premise '[o"' and hygoiteis "Dh"" can you label this as est.
$1.2
47.2
49.4
47.2
63.0
64.4
40.0
46.0
76.0
82.8
adfive
Given [p]
(bst "[h3"
59.2
59.6
50.8
58.0
64.6
60.
56.0
67.3
75.2
78.0
passive
Goven [p]
assumed that "Uh""
52.4
62.3
61.3
67.3
72.8
passive
Given [o] be csucluded that "UI""
51.2
52.0
50.6
59.6
63.0
64.0
36.6
65.2
70.1
21.1
passive
Geenn [p] can it deduced that [hh""
31.6
58.8
51.2
61.1
63.0
62.8
51.4
37.6
31.0
79.1
passive
Goen [h]] is followwd that "[p]"
$2.6
38.8
54.6
57.6
40.0
50.0
49.4
49.6
G1A
54.8
puecive
Given [n] zsm it preveed thai "D"d"
55.6
54.4
51.6
58.8
64.6
64.4
49.5
59.2
70.2
70.4
passive
Given [[] is inpliedd Saxt "[h]"
55.0
60.4
52.4
61.6
67.5
60.5
50.5
60.0
71LS
71.6
puedive
Given
csm be inforel that 'D-J"
$1.0
45.6
49.4
47.6
52.0
18.8
49.2
46.8
40.0
48.0
Geven [p] justifielly sad that "[h]"
64.0
6200
68.6
predive
promiss [[]' hyputhoss "Dx3"
Lshelled
56.6
81.2
passive
Geven [p] con reasoned that "Uh'"
36.0
34.4
51.8
57.6
65.2
62.4
52.6
572
31.0
72.8
past
Gaven [p] did you assunee that "UF3"
$2.8
34.0
$3.0
36.4
63.5
65.2
49.6
36.8
08.2
06.8
pust
Given [n] was the clsim "Di3" cormett
51.4
46.6
50.6
60.0
68.2
58.4
50.5
66
70.8
74.8
past
Goen [p] did you deduce that 'Dh""
55.2
58.8
50.8
59.2
65.8
61.2
51.4
66.8
70.4
71.6
Given [[]' didii follow floe "[b]"
49.5
46.4
51.6
$6.8
65.4
62.4
59.0
66.0
67.6
65.6
past
Geven [p] this imply that "Oh3"
51.0
61.2
69.4
56.4
67.6
past
Goven [[]]' you infix
65.4
65.2
past
Gove [p]" were you jestitted saying that "Oho"
35.4
51.0
53.2
66.4
652
34.0
74.1
776
past
Gien prenise 'Co'' and hypotthess "DId"
label
52.8
48.0
53.6
54.4
63.5
65.2
49.0
46
74.8
84.8
post
Goen [p]" you reason that [hh]"
35.4
36.8
52.6
38.4
00.0
65.4
31.6
65.4
70,0
08.0
past
Was the relatiosship betweee "Co]" and "ENd" ene.
52.2
34.4
50.2
64.0
63.8
56.0
40.0
46.0
67.2
68.8
poe
Gven [p]" was it ins that 'Dx3"
57.4
50.8
53.6
55.2
70.0
63.2
58.2
64.8
75.0
73.2
"Co"" Using only the above description was "oneec connect
51.8
46.0
$3.0
58.8
65.2
59.2
$2.6
63.6
71L.
76.0
[p] do you
that "Oh3"
47.6
65.6
present
Given [p]
comeet
20.8
59.6
67.4
59.6
52.2
66.4
196
prssent
Giee [o]] do you deduce "Dx3"
52.4
55.6
50.6
59.6
66.4
64.4
50.0
66.8
70.8
72.0
proent
Given [p] doss fellow thut 'Ih""
49.0
51.6
50.2
60.0
66.3
62.8
36
648
68.6
65.6
preseut
Goen [[]' does this inpply thu "CPd"
53.4
61.6
52.8
60.8
66.0
62.4
$2.0
58D
70.3
68.4
pronnd
Given [p]" de you inftr that "Ch3"
34.0
64.4
$2.0
60.8
66.2
64.0
51
65.2
70A6
73.2
present
Given [p] are you justifted in saying that "[h]"
56.2
53.2
50,5
51.3
65.8
39.2
53.4
59.6
T5A
79.6
Given premior [[n"' and hyputhos.. "Dh3" te libel ent.
54.4
50.3
51.4
$4.0
61.4
66.3
49.0
46.4
75.1
84.8
Geven [p]" do you
the relationship betware 'Ga3" [hh''
60.0
65.1
preami
67.6
present
Geven" [o] is tue that [[h]"
39.8
34.0
52.4
56.0
89.2
66.0
57.0
66
75.1
77.6
prssent
"[o'" Using only the above description "CNJ" correct
34.0
42.0
52.2
5%.0
65.0
57.6
62.4
700.
78.8
Tolome
Gaven [p] will you assume that "F3""
40.6
46.
53.0
352
64.
66.4
51.0
35.
72.
74.0
fotrr
Given [p] will the chim "D=3" be correct
55.8
-8.4
19.8
19:2
61.0
59.6
$2.6
72.4
12.8
16.0
Tuture
Goven [[]] will you deduce duzt "D"'"
$1.0
50.4
$2.5
G0L4
64.5
63.2
51.2
64.4
71.0
73.6
future
Geven [p]"
follow
53.6
35.0
66.5
62.8
66.4
Gaven [[]" this auply "Ux'"
65.6
64.0
67.2
future
Goven [p]" you that "Ch]"
53.2
60.
66.4
65.6
51.0
30,0
136
fiture
Given [p]] will you justified saying "Dh2"
48.0
53.0
51.
66.5
58.8
52.6
591
74.0
75.6
falmm
Given premise "Gal" and hyputhoss "Odd will the ishel be ml.
32.4
46.0
53.0
46.
61.
67.
49.0
460
75.6
83.0
future
Gaven "ta'' will you reasin Bha "IN""
52.3
46.4
51.8
55.3
67.
63.6
51.8
65.
71.3
71.6
folm
Will the relatiundiip herwerrn "In]" and "INI" be ent.
48.4
40.4
52.0
62.0
60.4
57.6
49.0
47.6
67A
673
future
Gaven [p]" will it be mus that '[h)"
35.0
49.6
50.8
52.
66.4
62.0
30.3
592
74.4
76.3
foture
"Col" Using only the abose description will "[h]" he cumeet
34.6
47.6
52.4
336
65.0
603
55.4
76.4
70.0
72.4
Table 13: Detailed list of prompts for Natural Language Inference with accuracy per prompt on RTE and CB across all models. Part of 2.

---

priperty
peomp:
LLaMA
306
1PT
OPFIML 1.36
OPT
30b
OPFIML 306
RTE
CR
RTT
CR
RTF
CR
RTE
CR
RTTE
CR
Gives '[G3" you
"EKD"9
32.6
36.8
51.0
603
65.0
67.2
313
36.8
750
14.8
Gives "[p]" can daii "ONJ" be contect?
53.8
35.0
5033
52.4
67.4
57.6
51.8
65.8
74.8
77.6
Giver [[p]' can you dellase
"Ih1"
52.0
57.6
53.6
58.0
0
51.2
60.4
73.8
77.2
Gioe [[p]' can it follow that "[h2")
55.6
50.0
514
57.6
658
63.6
56.2
60-1
69.
70.0
1381
(iives '[n3" can this imply ths "[k3""
53.6
$6.4
$2.0
60.8
658
61.2
50.8
51.2
70.6
7000
Given "Lp3" can you infer that 'Ihi"?
55.0
76.0
1910
58.8
65.4
61.
524
55.8
72.0
73.6
690
Gives "[p]" can you be justifeed saying the "ON'"
58.4
51.2
512
57.6
664
59.6
57.8
62.0
74.2
75.2
Given 'Ip)" and typothesis "Ihi"
label be ext?
52.
47.6
520
06.8
65.6
6LD0
75.1
7966
Gives "[p3"
that 'Dh3'!
59.2
503
58.0
64.6
60.8
75.2
78.0
Can relatiosshp betwees "Lo]" "UhJ" ent?
51.
38.8
528
596
66.2
568
49.0
47.6
674
68D
Gives "p'" can tre "Dig"
54.6
47.2
518
55.6
68.4
64.8
56.2
54.8
74.4
74.0
[[]]" Using only the show descpption can "D-3"b commat
542
47.6
53.2
53.0
66.2
65.6
96.4
76.4
71.2
72.4
000d
Given [p]] could you assume that "Lh'F
534
55.4
513
601
65.2
0N.8
ST.
50.1
71.6
728
could
Gives "[p3" could fil claim 'Eh]" cocrect?
53.6
47.2
49.6
54.8
66.4
59.2
53.8
72.0
76.0
78.8
could
Given "[p3" could deduce that '[h]"?
52.0
55.6
50.1
01.6
66.8
66.0
50.2
02.4
75.6
76.0
could
Gives '[p]" could Sollow that "[h]")
45.4
5166
57.6
64.2
62.4
69.
70.4
cudld
Gives "[n]"' could this imply foxr 'Ih'"
53.4
914
60.4
65.
69.6
68.4
could
Gives "[o]" could you inter that "[h]"?
53.6
19.2
514
G4.4
65.8
64-4
52.0
57.2
72.0
74.8
anold
Give '[n]" could you be justifid in saying that "Ek3"9
59
54.0
31.3
608
66
$3.2
60.8
73.3
75.6
could
Given peemise '(p)" and hypothesis "[h)" could the label ent?
53.1
57.2
514
19.6
62.6
62.8
19.0
96.0
75.0
800
culd
Giver "[n3" could you noasun Huui "Eh3'1
57.8
8088
504
58.8
65.6
63.6
54.2
67.2
7344
77.2
could
Could the relatioship betwees "[o]" and "Uh"" be ent?
50.8
90.4
55.6
60.
64.0
35.6
4T1.2
67.2
08.4
ouild
Givm "[n3" could is be Inue that "D>3**
52.2
52.0
513
56.0
67.3
64.8
60.6
56.4
7388
74.0
could
[Ip]"Usng only the shore descrition could "UhJ" be cornect
522
500
52.8
64.6
58.0
71.0
75.6
may
Given '[p]" moy you asu.ase
35.6
520
62.4
720
203
may
Giver "[n=]" muy claim "[k]" comoet?
35.
90.0
52.3
53.2
66.2
60.4
54.6
69.2
74.8
79.6
may
Given '[p]" may you deduce that "[h3""
512
58.0
518
62.0
64.0
66.0
50.0
51.2
720
25.2
may
Giver "[n3" may it folllww
'Ch3"
54.0
54.4
514
61.
64.2
62.8
54.4
60.0
70.8
70.4
may
Given "[p]" may this imply that '[h)")
522
51.6
52.8
55.2
65.4
66.0
50.6
52.0
7000
67.6
Gives "[2]" may you lnfor that "U'3"
5244
64.4
52.4
62.4
64.8
67.2
514
25.2
71.0
73.2
Given "[p]"
be justifted
that "Ih??
55.6
52.0
520
59.6
66.0
6000
56.2
754
75.6
Gives
"[p3" hypothesis "Th3" may dee lakel om.?
54.6
50.4
54.0
52.8
62.6
64.4
75.2
81.6
may
Gives "[p3" may you
"Dh3"
$2.9
604
65.2
90.2
718
7200
may
May the celationdiip between "Iol" and "[h]" be an..?
50.6
42.4
54.3
59.2
62.6
62.4
49.0
45.8
67.2
69.6
may
Give "[n]" may
tme thet '[h]"?
57.
48.8
518
84.8
66.3
66.0
51.
$2.4
74.6
77.2
may
"[p]"Liong Uisng only the showe descriptins may "[h]" be conrect?
30.6
95.8
5288
5322
64.6
58.2
75.4
08.4
20.3
might
Gives '[p]" sight you asoume that "Oh3")
564
56.4
52.4
58.0
64.8
65.4
52.2
55.6
70.6
68.8
might
Gives "[p]" night the caiis "D-3" be cormerr?
12.0
46.8
49.8
560
66.6
61.2
52.8
68.4
75,0
77.2
might
Gives "[p3" night dedace Out "Ch]"?
60.2
55.0
6048
63.6
706
71.2
might
Givas "[n3" night follow tha 'Ihl'7
$2.6
67.6
70.2
708
might
Given "[p]" might imply that "Oh?"?
51.6
53.2
530
583
66.2
50.8
50.4
702
650
might
Give "[n3"' night you infor "[h]"
62.
86.0
$2.8
62.8
66.6
65.6
51
55.6
20
208
might
Given "[o]" might you be justified soving that "Ohr"y
5922
43.2
32.8
3600
0622
SK.
51.8
36.8
75.6
24.8
might
Give promier [[p]" and byp-shosis [bh] might the loeel be ?
55.4
56.8
63.6
$2.0
638
66.
46.4
74.3
70
might
Given might
might you resson that '[h)"?
61.1
25.4
52.4
58.4
65.8
61.8
51.6
57.2
70.6
71.6
migit
Might the relationship betwees "CoJ" and "DND" be cst?
46.0
39.2
55.0
56.0
65.0
36.8
49.2
45.4
67.2
67.6
might
Gives "[p]" might toee "th""
49.2
45.4
52.4
56.4
67.1
61.8
52.2
74.6
75.2
might
"[o]" USing the showe descdiptias might "Dh'" he concer?
$2.0
53.6
60.0
68.6
7200
must
Given '[n]' rns you soamme that [Eh" 7
51.6
518
56.4
64.8
61.2
50.0
80.0
69.6
660
mast
Given '[p3' must the claim "Ih3" be cormect?
52.0
57.2
1900
500
66.4
57.6
53.2
68.0
21.4
268
mist
Gives "[D3" zust you deduce that "Ih]"?
55.8
55.0
50.4
5922
66.0
64.8
51.4
59.2
70.6
7088
most
Given '[p]" must it follow that 'Ih'"
5388
52.0
51.4
51.4
65.6
60.1
614
53.2
08.
66.4
must
Gives '[p]" zust this ingly that "[h]"?
53.2
45.0
52.0
56.4
66.2
62.0
52.4
53.2
69.3
668
must
Given '[n]' muel infer foat '[h]"
54.6
57.6
51.2
58.4
6666
64.0
54.0
56.8
69.8
2088
Gives "[D3"
justited
tue "Di"?
$2.0
548
59.6
72.4
73.6
Giva promise "[p3' and hyp-shosis '[h3"
label he
$2.9
48.
638
74.8
77.6
mist
Gives "[=J" sust you rosson that "D'""
58.4
52.8
49.
5488
65.2
63.2
54.2
53.2
69.8
67.6
must
Musi relatiunshpp betwom "Sp]" and "Edd" bn
48.0
41.2
$3.4
560
62.4
62.8
402
48.4
67.6
6600
must
Gives "[p]" zust a tree that "Uhj"?
49.4
5000
51.4
55.2
66.8
62.0
36.8
51.6
75.4
09.6
meat
"[p]" Using only the showe descripiion most "Dx3" be commrt
52.4
48.0
53.8
$4.8
63.6
SK.0
55.0
76.4
68.8
74.4
sbould
Gives "[p]" Shoid you semme ihan "D3""
508
30.8
522
39.
63.6
628
3166
30.6
20.6
663
should
Given '[p3" should (he claim [hh'" comeet??
53.1
45.0
50.2
52.4
65.0
5K8
53.8
68.8
70.2
7600
should
Gioas "[p3" shuild youdislase ilut "Dh2"
$2.4
$7.2
64.4
69.8
68.0
should
Given "[p]' sbould follow
"Und"t
53.2
$2.0
563
66.2
5644
61.2
20.2
67.6
should
Giom "[n3' shoild (his imply Buat "[k2'9
54.0
$0.8
49.
560
65.
62.8
62.2
56.8
69
668
should
Given "Loj" should you inter that "!h3")
51.6
52.8
31.6
57.2
653
63.6
35.8
6288
69.6
65.6
should
Gives "[o3" should you be justfteed la saylag that "[h3"?
53.6
45.4
52.6
55.6
65.4
$7.2
54.8
61.6
73.8
72.8
should
Given premise "Ip)" and hypothesis "[hj"should the label be enc?
56.2
52.0
49.1
51.2
62.8
63.6
49.2
2044
75.6
should
Gives "GD3" should you cesson that "O'0"
52.2
45.0
52.2
57.2
65.0
63.6
35.0
60.8
71.0
7088
shald
Should the miliiiondii beturem "[p3" and "D=2" be mlt.
5000
40.8
$3.2
60L4
62.6
$7.6
49.2
48.8
66.8
60.2
Ghald
Gives "Ca3" should be that "Oid"?
55.8
52.4
513
55.6
$9.6
71.6
73.6
should
"[p]" Usin only the
description should
coemon?
46.0
$3.0
$2.4
74.0
69.4
74.0
would
Given "[p]" woold you assume thox "Uh]"?
313
52.
523
35.6
63.6
02.8
31.1
55.4
213
24.0
would
Gives "[D3' would the claim "ChJ" be connent
62.0
52.4
50.2
59.2
66.4
$600
51.
71.2
72.4
7922
wndld
Gives "[=3" would you dediae thet "Ik]"?
52.2
$0.8
52.4
$7.3
65.8
62.4
52.4
15..2
71.2
74.4
would
Gives '[o]" would is Edllow that "Oh""?
56.2
52.8
51.6
53.6
64.8
62.8
53.6
GOLD
70.0
70.4
woild
Give '[nJ' would this iirlly (hut "[h]""
552
5R.4
50.8
67.6
65.4
60.8
51.8
$4.8
70.8
200
Gives "[o3" would infe: "[hd"
55.8
$1.0
58.8
60.4
71.0
7400
woild
Gives "[n3' would
judiifidd saying "D-d"
$2.8
86.8
$7.0
73.8
78.4
would
Gives premise "Ip]" and hypothesis "[h)" would label be ext?
53.0
89.2
49.3
48.
65.2
61.8
05.0
75.4
804
wonld
Give '[n3' would you reason that "[h]")
56.6
51.6
$2.0
65.6
67.3
64.0
52.6
61.2
71.2
75.6
would
Would the colattondip between '[p] cad "[h]" be cm.
51.0
42.8
51.2
58.8
656
SLD
48.2
4T.6
67.4
68.4
would
Gives 'to3" would is tre that "Dig"t
60.8
53.2
52.1
56.4
66.8
60.4
55.2
59.6
75.2
78.4
would
'[p]" Li]] only the ahore descriptioa would "Uh]" be conrect
56.0
47.22
51.6
54.4
66.6
59.6
56.8
77.6
20.6
77.2
Given [[p]" is the
[h] correc??
35.6
52.
320
35.8
66.3
37.6
89.6
756
7K0
asseriion
Givas "[n3"
assetion 'Dk3' Ira??
$1.2
$3.4
$8.4
74.3
78.4
clats
Given "[p]" is clain "[x3" cornect?
50.8
59.6
674
59.6
52.2
66.0
2544
79.6
claiu
Gives "[o]" is the clain "UX3" tue?
55.0
50.8
53.2
57.6
66.2
61.6
52.6
66.0
73.6
7800
ent.
Given premise: "[p]" and hopottess: "[h]" is the label e
51.0
09.6
50.0
47.2
62.8
63.2
49.0
05.0
76.1
8522
cut.
the selationelip between [e]" and 'th3" ear?
49.2
35.2
48.4
63.6
63.4
$7.6
49.0
46.0
67.8
67.2
implication
Give prmise: "[n]" and hyputhoss: "[6]" is the label implication?
55.8
60.4
52.3
56.0
63.8
65.6
49.6
46.0
73.2
84.0
implicatios
Is the selationalip between "To"" and "[hi" implication?
53.8
45.4
49.4
64.4
63.6
62.0
45.4
46.8
66.3
66.0
Table 14: Detailed list of prompts for Natural Language Inference with accuracy per prompt on RTE and CB across all models. Part 2 of 2.

---

property
LLaMA 306
OPT 1.36
OPT-IMI. 1.3b
OPT 30h
OPT-IMI. 30b
imperative
12.94
8.64
5.19
5.39
7.75
indicative
13.01
8.68
5.33
5.42
7.76
interrogative
12.99
8.74
5.38
5,41
7.77
active
13.01
8.77
5,39
5.4
7.78
pussive
13.0
8.66
5.42
5.4
7.77
past
12.99
8.68
5.33
5.43
7.77
present
12.99
8.71
5.35
5.39
7.77
future
12.98
8.83
5.33
5,43
7.77
can
12.99
8.59
5.42
7.78
could
12.99
8.6
5.34
7.78
may
12,99
5.31
5.39
7.77
might
12.97
8.77
5.36
5.37
7.76
must
12.99
8.48
5.32
7.76
should
12.99
5.37
7.78
would
12.98
8.66
5.35
7.77
appraisal
12.95
8.74
5.38
5,42
7.76
commentary
12.97
8.73
5.36
5.4
7.76
critique
12.99
8.73
5.36
5,41
7.77
evaluation
12.99
8.74
5.38
5.41
7.77
review
12.99
8.72
5.35
5.4
7.77
Table 15:
Perplexity scores for prompts on IMDB
property
LLaMA 30b
OPT 1.3b
OPT-IML 1.3b
OPT 30b
OPT-IML 30b
imperative
13.21
8.7
7.8
7.32
8,34
indicative
13.66
8.99
7.91
7.49
8.52
interrogative
13.53
8.92
8.13
7.44
8.58
active
13.68
8.99
8.17
7.5
8.65
passive
13.57
8.95
8.13
7.5
8.54
past
13.57
8.87
8.03
7.43
8.56
present
13.55
8.88
8.05
7.38
8.57
future
13.5
8.84
7.99
7.4
8.52
can
13.53
8,91
7.42
could
13.54
8.03
7.39
8.59
may
13.58
8,91
7.41
might
13.46
8.87
7.98
7.34
must
13.54
8.88
8.07
7.4
13.54
8.87
8.12
7.36
would
13.46
8.86
8.05
7.39
appraisal
13.29
8.95
8.12
7.49
8,31
commentary
13.43
8.91
8.01
7.45
8.54
critique
13.53
8.91
8.09
7.48
8.57
evaluation
13.53
8.92
8.13
7.44
8.58
review
13.5
8.85
8.05
7.37
8.58
Table Perplexity scores prompts on SST

---

property
LLaMA 30b
OPT 1.3b
OPT-IMI 1.3b
OPT 30h
OPT-IMI. 30h
imperative
16.25
9.88
7.96
6.31
9.36
indicative
16.4
9.98
8.05
6.35
9.43
interrogative
16.41
9.96
8.08
6.31
9.43
active
15.92
9.74
7.99
6.42
9.21
passive
15.82
9.7
7.93
6.42
9.19
past
16.4
10.03
8.14
6.33
9.44
present
16.4
10.02
8.13
6.31
9.43
future
16.38
10.02
8.12
6,31
can
16.4
9,96
6,31
could
16.39
9.93
8.07
6.31
9.42
may
16.42
9,94
8.07
6.33
9.44
might
16.39
9.95
8.07
6.31
9.43
must
16.41
9,96
8.07
6.32
9,43
should
16.4
9,95
8.09
6.3
9.44
would
16.38
9.94
8.08
6.32
9.41
assertion
16.47
10.05
8,19
6.31
9.43
claim
16.45
10.06
8.2
6,3
9.43
implication
16.29
10.0
8.04
6.39
9,46
entailment
16.25
9.97
8.04
6,38
9.43
Table 17: Perplexity scores for prompts on CB
property
LLaMA 306
OPT 1.3b
OPT-IML 1.3h
OPT 30h
OPT-IMI. 30h
imperative
15.53
9.88
6.34
indicative
15.7
10.01
8.55
6.38
9.13
interrogative
15.68
9.95
8.57
6.3
9.13
active
15.33
9.74
8.49
6.43
8.99
passive
15.22
9.69
8.42
6.41
8.97
past
15.62
9.84
8.49
6,28
9.12
present
15.61
9,82
8.48
6.27
9.1
future
15.59
9,79
8.46
6,26
9.1
15.59
9.87
8.52
6.26
9.1
could
15.58
9.87
8.53
6.26
9.1
may
15.61
9.83
8.53
6.27
9.12
might
15.58
9.87
8.53
6.26
9.11
15.6
9.88
8.53
6.27
9.11
should
15.59
9.88
8.56
6.25
9.12
would
15.57
9.88
8.54
6.27
9.09
assertion
15.89
9.84
8.54
6.26
claim
15.88
8.54
entailment
15.6
9.73
8.35
6.33
9,09
implication
15.64
9.75
8.35
6.34
9.12
Table 18: Perplexity scores for prompts RTE

---

property
LLaMA 30b
OPT 1.3b
OPT-IML 1.3b
OPT
OPT-IML 30b
imperative
13.47
8.67
7.34
5.93
8.39
indicative
13.55
8.64
7.32
5.94
8.41
interrogative
13.5
8.63
7.3
5.92
8.4
active
13.48
8.61
7.28
5.87
8,37
passive
13.53
8.64
7.3
5.94
8.39
past
13.48
8,58
7.29
5.9
8.35
present
13.46
8.61
7.29
5.88
8.38
future
13.46
8.58
7.27
5.88
8.41
13.43
8.62
7.27
5,87
8.36
could
13.44
8.61
7.28
5.88
8.39
may
13.43
8.61
7.29
5,88
8.41
might
13.44
8.61
7.28
5.88
8.41
13.45
8.63
7.25
5.88
8.37
should
13.43
8.6
7.28
5,86
8.43
would
13.41
7.25
5.86
8.4
appropriate
13.49
8.62
5,88
correct
13.47
7.27
5,86
proper
13.5
8.63
7.29
5,88
right
13.47
8.61
7.25
5.87
answer
13.47
8.61
7.27
5.86
8,37
reply
13.5
8.66
7.33
5,91
8.43
response
13.49
8.63
7.31
5.9
8.42
solution
13.5
8.63
7.3
5.92
8.4
Table 19: Perplexity scores for prompts on ARC-E
property
LLaMA 30b
OPT 1,3b
OPT-IML 1.3b
OPT 30b
OPT-IML 30b
imperative
14.16
8.84
6.77
5.66
8.29
indicative
14.19
8.84
6.81
5.61
8.33
interrogative
14.14
8.82
6.74
5.6
8.26
active
14.14
8.82
6.75
5.6
8.27
passive
14.17
8.84
6.78
5.64
8.28
past
14.15
8.89
6.78
5.6
8.28
present
14.15
8.82
6.75
5.59
8.28
future
14.15
8.89
6.78
5.6
8.28
can
14.15
8.84
6.76
5,59
could
14.15
8.82
6.76
5.61
8.26
may
14.15
8,82
5.6
might
14.15
8,82
6.77
5.6
must
14.16
6.76
5.6
8.29
should
14.15
8.82
5,58
would
14.14
6.75
5.59
appropriate
14.15
8.83
6.75
5.61
8.28
correct
14.14
8.81
6.75
5.59
8.26
proper
14.16
8.83
6.75
5.61
8.28
right
14.14
8.82
6.74
5.6
8.26
answer
14.14
8.82
6.75
5.6
8.27
reply
14.16
8.83
6.79
5,62
8.29
response
14.16
8.83
6.77
5,61
8.28
solution
14.16
8.84
6.77
5,62
8.29
Table 20: Perplexity scores for prompts on BoolQ


Reviews:
[{"review_id": "ZqwCNyulIu", "reviewer": "Reviewer_4Bot", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, your helpful suggestions, and appreciation of our paper!\n\nThank you for pointing us to the missing reference, we will include it in an updated version of the paper. \n\nRegarding your question, we didn't paraphrase instructions to a larger extent, because we wanted to vary one linguistic property at a time and keep the variation between sentences minimal, in order to not introduce different sources of variation at once. Hence, we constructed all our prompts to only consist of a main clause with varying tense, mood, etc. Thank you for bringing this up, we will clarify this in an updated version of the paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A very comprehensive and systematic empirical study of the instability of prompt choices. Those experimental results could potentially be utilized in many future studies.\n* The authors present several interesting conclusions, such as the existence of the correlation between LM perplexity of prompts and performance.", "reasons_to_reject": "* Though it is not a serious flaw, it would make this paper stronger if the authors could give some explanations on why they choose certain types of linguistic variations, such as mood and tense. What if the prompts are paraphrased to a larger extent, e.g., using a constituency parser to generate sub-tree structures and making changes on the sub-tree level?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a large-scale systematic analysis of how the variations in prompts influence downstream performances. The authors conduct empirical studies on multiple tasks using different LMs under various settings. In these studies, the authors control (i) grammatical properties such as mood, tense and modality; and (ii)  lexico-semantic variation by replacing a word with its synonyms. Experimental results show that prompts transfer rather poorly across datasets and LMs. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "* Do Prompts Solve NLP Tasks Using Natural Language? Sen Yang, Yunchen Zhang, Leyang Cui and Yue Zhang. ArXiv 2022."}, "extra_text": {}}, {"review_id": "396OXbrxNc", "reviewer": "Reviewer_Nfib", "summary": "", "questions": "In your experiments, did you design all the prompts by yourself? Since you want to investigate the prompt robustness of instruction-tuned LMs, it should be better to choose the prompts used in tuning OPT-IML (seen by LMs during training), which can more effectively demonstrate your motivations.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review, for appreciating the novel perspective of our work and giving some actionable suggestions! We’d like to offer the following clarifications regarding your feedback and questions:\n\n*Re comment 1:* We agree that the fact that prompts transfer poorly is generally known. We deemed it valuable to investigate this thoroughly, systematically and comprehensively, since it is not known if there’s an upper bound to performance variation. In particular, instruction-tuning is often seen as a fix to this without rigorous justification. That is why we prompted, in particular, instruction tuned models with a large set of prompts to demonstrate how much performance can vary, even on seen datasets. \n\nWe agree that section 5 would benefit from an additional discussion and analysis. If the paper is accepted, we will use the additional page in the camera-ready version to expand the discussion of the relationships between different linguistic properties in the prompts and their effect on performance, e.g. by using hierarchical models which are commonly applied in the behavioural sciences and were recently also employed for the analysis of LMs (Lampinen et al. 2022). \n\n*Re comment 2:* Thank you for your input on the experimental set-up. We initially chose instruction-tuned and non-instruction-tuned decoder-only models as the scope of our paper, since decoder-only models are most widely used at the moment. We re-ran some of our experiments using an encoder-decoder model, Flan-T5 (XL), and found that average performance per linguistic property still varied between properties, similarly to what we reported on the encoder-only models in the paper. \n\n| Mood  | sst | imdb | rte | hans |\n|----------------|----|------|--------|-------------|\nInterrogative | 87.3 | 95.8 | 88.8 | 77.5 |\nIndicative      | 86.6 | 94.6 | 86.1 | 76.4 |\nImperative     | 88.2 | 96.1 | 90.9 | 77.3 |\n\n\n|Synonymy | rte | hans |\n|----|----------|---------------|\nEntailment | 95 | 77 |\nImplication | 96 | 78.4 |\nAssertion | 91 | 81.5 |\nClaim | 92.5 | 81 |\n\n|Synonymy | sst | imdb |\n|---|---|---|\nAppraisal | 83.4 | 85.7 |\nCommentary | 77.7 | 80.4 |\nCritique | 81.5 | 85.8 |\nEvaluation | 83.7 | 88.4 |\n\nIf the paper is accepted, we will include results on Flan-T5 in the camera ready version of the paper. We would also very much welcome your advice on which other types of models you would recommend including.\nMoving to the cross-task setting would indeed be a valuable avenue for future work. For this paper, we chose a cross-dataset setup, since the focus was comparing the same prompts on multiple dataset of the same task.\n\n*Re comment 3:*  The focus of our recommendations is on how to conduct evaluation in a more rigorous fashion, and our position is that a metric quantifying the dependence of model performance on a particular prompt formulation is a useful way to make progress in understanding of how to perform prompt engineering, be it manual engineering or automatic prompt optimization.\n\n*Re comment 4:* Thank you for the comment, we will rephrase the heading of section 5.3. for clarity. If this is possible, would you be able to offer more advice on which sections you found hard to follow? This would help us a lot in improving the paper. \n\n*Re question on prompt design:* Yes, we did construct the prompts ourselves in order to minimise the potential sources of variation and have a more homogeneous set of prompts in terms of sentence structure. We found prompts in public instruction tuning sets to vary quite widely. We do agree that experimenting with the exact prompts from OPT-IML would strengthen the paper and will include those experiments in an updated version of the paper. This is good advice, thank you!\n\n------------------------------------------------------\nReferences:\n\n* Lampinen, Andrew, et al. \"Can language models learn from explanations in context?.\" Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **A novel perspective**. The author utilizes a lexico-level perturbation and linguistic perspective to investigate the prompt robustness, which differs from the previous sentence-level paraphrasing or simple synonymy replacing.\n2. **Useful proposal**. The proposal concluded by this paper might also benefit future research.", "reasons_to_reject": "1. **Limited contribution**. Though this paper provides some useful suggestions, most conclusions have already been proposed by a bunch of previous works or are just commonsense for the community (e.g., prompt transfer). During the whole section 5, the authors simply introduce the results and compare scores, but there are no more insights, analysis, or explanations. After reading the whole paper, I didn't seem to learn much from it.\n\n2. **Concern about the experiment setting.** In the experiment, the authors adopt five models, but essentially, there are only two model categories --- vanilla LMs and instruction-tuned LMs. Similarly, all of these models (OPT, LLaMA) are decoder-only LMs. I am worried about the expandability of the experimental conclusion. Similarly, the datasets seem only cover the cross-dataset generalization setting; a better choice is to adopt a more challenging cross-task setting [1], which is a popular trend in the current community.\n\n3. **Some viewpoints are subjective.** For example, in line 537, the authors suggest \"include estimates of performance mean and variance based on a large set of prompts\". Personally, I think the prompt is just a fundamental \"feature\" for LLMs to deal with a specific task, \"prompt engineering\" is just similar to \"feature engineering\"; we have to tune an optimal (or sub-optimal) prompt for one specific task, but we cannot anticipate the LMs will be robust to the change of the prompt. So reporting statistic scores on large-set prompts for one task seems unrealistic. But I agree with \"treating prompts as hyperparameters\" --- tune the prompt on dev set, fix it, and then repot scores on the test set.\n\n4. **Writing.** I found it hard to follow some specific sections of this paper. For example, the title of section 5.3 --- \"Robustness and instruction-tuning\" --- sounds a little bit confusing.\n\n---\n\nReferences:\n\n[1]. Mishra S, Khashabi D, Baral C, et al. Cross-task generalization via natural language crowdsourcing instructions[J]. arXiv preprint arXiv:2104.08773, 2021.", "questions_to_authors": "In your experiments, did you design all the prompts by yourself? Since you want to investigate the prompt robustness of instruction-tuned LMs, it should be better to choose the prompts used in tuning OPT-IML (seen by LMs during training), which can more effectively demonstrate your motivations.", "paper_topic_and_main_contributions": "This paper conducts comprehensive experiments to show the prompt instability of LLMs. Unlike previous work, this paper investigates the prompt robustness of the model from a lexical perspective, where the authors slightly shift the linguistic properties (e.g., mood, tense) of prompt and find a significant performance variation of LLMs, including instruction-tuned LLMs. Some results are also different from the previous works, suggesting the vulnerability of numerous research findings in current prompt-related research. Thus, the author also concludes with some empirical advice for future research.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9eEVDw2FHU", "reviewer": "Reviewer_z3Ht", "summary": "", "questions": "- Can you please elaborate on the reason why you have decided to not apply synonymy to the main verb of the prompt?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review, and feedback on our paper! We’d like to offer the following clarifications on your inputs and questions:\n\n*Re presentation improvements:* Thank you for your feedback on the phrasing of the introduction in lines 0-56, we will adapt this in an updated version of the paper. \n\n*Re Q on synonymy for main verb of prompt:* We experimented with synonyms for the main verb of the instruction, but did not include the results, since usually there were more synonyms available for other content words in the sentence, so these results were more interesting. We will include the results in the camera-ready version of the paper. \n\n*Re comment 1:* The main focus of the paper was not to aid in choosing the best prompt for a task, but systematically examine current evaluation practices by analysing model robustness to linguistic variation. We thus take a novel perspective on the question of model robustness to prompt variability, and based on our results provide a set of recommendations for a more rigorous evaluation of model prompting. We believe that these findings and recommendations would help the field by critically examining commonly held assumptions and instigating a conversation on what transparent results reporting looks like for prompt-based evaluation.\n\n*Re comments 2, 3:* We acknowledge that LLMs can in principle solve tasks in an open-ended way, i.e., we can prompt them with an instruction and have them generate full sentences that will contain an answer. But such an open-ended generative set-up is very difficult to evaluate efficiently. That is why we chose to predict based on the maximum log probability among answer options. Is this what you meant by ‘explicit solvers’? With our evaluation method we are following what is the standard prompt-based evaluation procedure for LLMs (see Sanh et al. 2022, Wei et al. 2022, Webson and Pavlick 2022, i.a.). Due to surface form competition (Holtzman et al. 2021) this is the evaluation method of choice also for many QA tasks. \nIn our work, we aimed to have a diverse set of NLU tasks with a consistent evaluation procedure across tasks, and hence did not include open-ended generation tasks. We agree however that adding additional tasks, such as QA, would strengthen the paper. We are working on including those additional results in an updated version of the paper.\n\n------------------------------------------\nReferences:\n* Sanh, Victor, et al. \"Multitask Prompted Training Enables Zero-Shot Task Generalization.\" ICLR 2022-Tenth International Conference on Learning Representations. 2022.\n\n* Wei, Jason, et al. \"Finetuned Language Models are Zero-Shot Learners.\" (2022).\n\n* Webson, Albert, and Ellie Pavlick. \"Do Prompt-Based Models Really Understand the Meaning of Their Prompts?.\" Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.\n\n* Holtzman, Ari, et al. \"Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper aims to make a significant contribution to the understanding of how prompts should be phrased in order to obtain the best results over linguistic tasks. \n\n- The paper performs extensive experimental analysis", "reasons_to_reject": "- The large experimental analysis does not reveal any clear pattern and does not help in choosing what are the best ways to write a prompt for a classification task\n\n- The analysis is confined to classification tasks\n\n- It is even questionable if it is important to perform a linguistic classification task with LLMs. Indeed, solving many of the tasks is useless since we have LLMs that make decisions and generate responses without using explicit solvers for these tasks.", "questions_to_authors": "- Can you please elaborate on the reason why you have decided to not apply synonymy to the main verb of the prompt?\n", "paper_topic_and_main_contributions": "This is an experimental paper that aims to investigate the following research question: \nDo linguistic variations in prompts have an effect in performance over final NLP tasks for LLMs?\nTo pursue its goal, the paper examined the performance of LLMs on different classification tasks mainly contained in GLUE and SuperGLUE. Linguistic variations are characterized by different features: mood, aspect, tense, modality, and synonymy.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing references"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Writing the introduction:\n\n- please reorganize the introduction such that there is a clear paragraph describing the work of your paper, which is: studying the effect on performances of syntactic variations on semantically equivalent prompts.\n- \"To test ***this hypothesis***, we examine prompting ... \" (line 056). It is weird to start a paragraph with an anaphoric word referring to an idea expressed in the previous paragraph. This sentence needs rephrasing. \n- Make clear that the \"hypothesis\" is expressed in lines 051-055 (that is **this hypothesis***) and it is part of your intuition in building the paper "}}]

Author rebuttal:

Meta-review: This paper examines how linguistic variations in prompts affect the performance of large language models (LLMs) on various classification tasks. The paper tests different LLMs, including instruction-tuned LLMs, on datasets from GLUE and SuperGLUE. The paper manipulates the prompts by changing their grammatical properties (such as mood, tense, and modality) and lexical properties (such as synonyms). The paper finds that the prompts are unstable and sensitive to these variations, and that the performance varies significantly across datasets and LLMs. The paper provides some empirical suggestions for future prompt-related research. The paper elaborates upon the prompt variations aptly, is well written and definitely has merits for downstream use-cases, the scope limited to classification based tasks only for a long paper.
Decision: Accept-Findings