F.1 Main Experiment Templates
category	template	adapted from
instructive	{premise} Using only the above description and what you know about the world,
"{'hypothesis'} is definitely correct. Yes or no?	Williams et al. (2018, p. 3)
instructive	{premise} \nquestion: {hypothesis}Yes or no?\nanswer:	Brown et al. (2020, p. 59)
instructive	{premise} Are we justified in saying that "{hypothesis}?"
instructive	Given {premise} Should we assume that "{hypothesis}"} is true?
instructive	{premise} Based on the previous passage, is it true that "{hypothesis}?"
instructive	Given {premise} Is it guaranteed true that "{hypothesis}?"
instructive	Suppose {premise} Can we infer that "{hypothesis}"}?
instructive	Given that {premise} Does it follow that "{hypothesis}"}?
instructive	{premise} Question: Does this imply that "{hypothesis}"}?
instructive	Given that {premise} Therefore, it must be true that "{hypothesis}"}?
misleading-moderate	{premise} Do most of the above words appear in the following passage? {hypothesis}
misleading-moderate	{premise} Are there lots of similar words in "{hypothesis}"}?
misleading-moderate	{premise} Does that have the same meaning as "{hypothesis}"}?
misleading-moderate	{premise} Can that be paraphrased as: "{hypothesis}"}?
misleading-moderate	{premise} Can that be summarized as "{hypothesis}"}?
misleading-extreme	{premise} Does the paragraph start with "the"? {hypothesis}
misleading-extreme	{premise} Is this grammatically correct? {hypothesis}
misleading-extreme	{premise} Is the sentiment positive? {hypothesis}
misleading-extreme	{premise} Is this a sports news? {hypothesis}
misleading-extreme	{premise} Is this French? {hypothesis}
irrelevant	{premise} Single-family zoning is bad for American cities. "{hypothesis}"}?
irrelevant	{premise} Inflections are annoying and thank god that
Middle English got rid of most of them. "{hypothesis}"}?
irrelevant	{premise} When Bolyai sent Gauss his discovery of non-Euclidean geometry,
Gauss replied that he arrived at the same results 30 years ago. "{hypothesis}"}?	Greenberg (1974, p. 141)
irrelevant	{premise} If bonito flakes boil more than a few seconds,
the stock becomes too strong? "{hypothesis}"}?	Tsuji and Sutherland (1980, p. 148)
irrelevant	{premise} Is the pious loved by the gods because it is pious?
Or is it pious because it is loved by the gods? "{hypothesis}"}?	Plato (c. 399 BC, 10a)
null	{premise} {hypothesis}
null	{hypothesis} {premise}
null (MLM only)	{premise} {mask} {hypothesis}
null (MLM only)	{hypothesis} {mask} {premise}
null (MLM only)	{mask} {premise} {hypothesis}
null (MLM only)	{mask} {hypothesis} {premise}
Table 3: All prompts used in the main text of the paper. All templates use “yes”/“no” as target words for the entailment and non-entailment classes, respectively. For ternary NLI datasets, we use “unclear” for the neutral class, which performs best after preliminary experiments with other ternary words: “maybe”, “sometimes”, “perhaps”, “possibly”, and “neither”. Keen readers may notice that some of the instructive templates (e.g., should we assume) do not instruct a strict entailment task. We intentionally wrote a mixture of instructions that asks for strictly logical entailment and pragmatic inference, intending to measure if models can distinguish between the two on datasets such as HANS (McCoy et al., 2019) that magnify different predictions caused by pragmatic effects. Of course, this research question became moot as we found that models cannot even distinguish among much more pathological prompts.