SlowGuess/NAACL_2022_OCR huggingface dataset:
num. shots	template category	median	q3 - q1	mean	std. dev.
4	instructive	0.5830	0.0885	0.5907	0.0517
4	irrelevant	0.6300	0.1291	0.6170	0.0645
4	misleading-extreme	0.5884	0.0469	0.5787	0.0342
4	misleading-moderate	0.5650	0.0722	0.5753	0.0418
4	null	0.5560	0.0433	0.5599	0.0324
8	instructive	0.6155	0.0920	0.6186	0.0524
8	irrelevant	0.6570	0.0307	0.6471	0.0374
8	misleading-extreme	0.6101	0.0677	0.5899	0.0595
8	misleading-moderate	0.6047	0.0767	0.5969	0.0490
8	null	0.5632	0.0397	0.5586	0.0326
16	instructive	0.6697	0.0605	0.6594	0.0558
16	irrelevant	0.6787	0.0488	0.6787	0.0294
16	misleading-extreme	0.6390	0.0506	0.6413	0.0384
16	misleading-moderate	0.6083	0.0443	0.6072	0.0427
16	null	0.5722	0.0379	0.5767	0.0327
32	instructive	0.7022	0.0813	0.6929	0.0638
32	irrelevant	0.7292	0.0235	0.7206	0.0236
32	misleading-extreme	0.7076	0.0334	0.7056	0.0340
32	misleading-moderate	0.6516	0.0992	0.6350	0.0666
32	null	0.6318	0.0731	0.6414	0.0392
64	instructive	0.7545	0.0542	0.7353	0.0548
64	irrelevant	0.7491	0.0198	0.7455	0.0218
64	misleading-extreme	0.7509	0.0416	0.7451	0.0299
64	misleading-moderate	0.7310	0.0993	0.6953	0.0688
64	null	0.7004	0.0848	0.6998	0.0516
128	instructive	0.7834	0.0451	0.7661	0.0551
128	irrelevant	0.7671	0.0343	0.7704	0.0200
128	misleading-extreme	0.7798	0.0334	0.7729	0.0255
128	misleading-moderate	0.7744	0.0550	0.7354	0.0842
128	null	0.7329	0.0695	0.7369	0.0389
G.2 ALBERT on ANLI R1
num. shots	template category	median	q3 - q1	mean	std. dev.
32	instructive	0.3640	0.0232	0.3625	0.0166
32	irrelevant	0.3660	0.0140	0.3681	0.0134
32	misleading-extreme	0.3380	0.0100	0.3404	0.0081
32	misleading-moderate	0.3455	0.0130	0.3470	0.0098
32	null	0.3540	0.0177	0.3567	0.0122
64	instructive	0.3735	0.0408	0.3738	0.0251
64	irrelevant	0.3760	0.0210	0.3788	0.0178
64	misleading-extreme	0.3485	0.0135	0.3510	0.0129
64	misleading-moderate	0.3525	0.0197	0.3574	0.0171
64	null	0.3660	0.0208	0.3675	0.0184
128	instructive	0.4050	0.0562	0.3992	0.0356
128	irrelevant	0.4105	0.0240	0.4120	0.0176
128	misleading-extreme	0.3840	0.0262	0.3843	0.0204
128	misleading-moderate	0.3720	0.0295	0.3725	0.0199
128	null	0.3800	0.0235	0.3857	0.0247
256	instructive	0.4625	0.0490	0.4504	0.0450
256	irrelevant	0.4695	0.0175	0.4694	0.0147
256	misleading-extreme	0.4350	0.0297	0.4263	0.0231
256	misleading-moderate	0.4375	0.0492	0.4265	0.0353
256	null	0.4155	0.0475	0.4167	0.0365
512	instructive	0.5085	0.0235	0.4992	0.0434
512	irrelevant	0.5185	0.0230	0.5154	0.0186
512	misleading-extreme	0.5050	0.0172	0.5008	0.0177
512	misleading-moderate	0.4930	0.0285	0.4839	0.0413
512	null	0.4480	0.0550	0.4564	0.0399
1024	instructive	0.5555	0.0270	0.5557	0.0449
1024	irrelevant	0.5560	0.0345	0.5729	0.0351
1024	misleading-extreme	0.5330	0.0265	0.5477	0.0316
1024	misleading-moderate	0.5405	0.0247	0.5447	0.0388
1024	null	0.4990	0.0588	0.5062	0.0392
G.3 T5 770M on RTE
num. shots	template category	median	q3 - q1	mean	std. dev.
4	instructive	0.5433	0.0406	0.5493	0.0219
4	irrelevant	0.5469	0.0424	0.5532	0.0252
4	misleading-extreme	0.5560	0.0361	0.5561	0.0263
4	misleading-moderate	0.5542	0.0325	0.5531	0.0220
4	null	0.5451	0.0487	0.5451	0.0578
8	instructive	0.5487	0.0235	0.5516	0.0232
8	irrelevant	0.5415	0.0280	0.5480	0.0244
8	misleading-extreme	0.5632	0.0379	0.5545	0.0322
8	misleading-moderate	0.5487	0.0280	0.5543	0.0192
8	null	0.5217	0.0560	0.5122	0.0317
16	instructive	0.5668	0.0406	0.5662	0.0277
16	irrelevant	0.5578	0.0298	0.5558	0.0199
16	misleading-extreme	0.5632	0.0190	0.5634	0.0160
16	misleading-moderate	0.5632	0.0343	0.5666	0.0239
16	null	0.5542	0.0271	0.5469	0.0381
32	instructive	0.6047	0.0433	0.6078	0.0317
32	irrelevant	0.6029	0.0361	0.6025	0.0366
32	misleading-extreme	0.5939	0.0352	0.5996	0.0292
32	misleading-moderate	0.5884	0.0424	0.5986	0.0311
32	null	0.5722	0.0460	0.5772	0.0443
64	instructive	0.6264	0.0433	0.6318	0.0324
64	irrelevant	0.6697	0.0542	0.6585	0.0421
64	misleading-extreme	0.6318	0.0478	0.6336	0.0355
64	misleading-moderate	0.6227	0.0578	0.6195	0.0400
64	null	0.6173	0.0496	0.6115	0.0442
128	instructive	0.6859	0.0514	0.6820	0.0421
128	irrelevant	0.6805	0.0307	0.6749	0.0362
128	misleading-extreme	0.7022	0.0361	0.6987	0.0260
128	misleading-moderate	0.6516	0.0379	0.6597	0.0295
128	null	0.6191	0.1291	0.6277	0.0717
G.4 T5 3B on RTE
num. shots	template category	median	q3 - q1	mean	std. dev.
4	instructive	0.5433	0.0442	0.5524	0.0297
4	irrelevant	0.5560	0.0469	0.5611	0.0308
4	misleading-extreme	0.5668	0.0442	0.5671	0.0251
4	misleading-moderate	0.5379	0.0415	0.5497	0.0247
4	null	0.5523	0.0514	0.5575	0.0334
8	instructive	0.5650	0.0514	0.5680	0.0427
8	irrelevant	0.5704	0.0343	0.5676	0.0332
8	misleading-extreme	0.5848	0.0397	0.5773	0.0431
8	misleading-moderate	0.5523	0.0442	0.5485	0.0309
8	null	0.5542	0.0523	0.5553	0.0459
16	instructive	0.5866	0.0505	0.6005	0.0467
16	irrelevant	0.5921	0.0406	0.5907	0.0279
16	misleading-extreme	0.5921	0.0262	0.5953	0.0271
16	misleading-moderate	0.5704	0.0298	0.5693	0.0212
16	null	0.5848	0.0614	0.5833	0.0481
32	instructive	0.6227	0.1056	0.6463	0.0757
32	irrelevant	0.6336	0.0623	0.6349	0.0416
32	misleading-extreme	0.6191	0.0542	0.6315	0.0393
32	misleading-moderate	0.6011	0.0298	0.6134	0.0440
32	null	0.5939	0.0848	0.6031	0.0548
64	instructive	0.7220	0.1227	0.7113	0.0784
64	irrelevant	0.7040	0.0578	0.7032	0.0408
64	misleading-extreme	0.7076	0.0478	0.7039	0.0352
64	misleading-moderate	0.6697	0.0957	0.6792	0.0569
64	null	0.6390	0.0984	0.6397	0.0618
128	instructive	0.7996	0.0496	0.7769	0.0627
128	irrelevant	0.7473	0.0415	0.7468	0.0271
128	misleading-extreme	0.7653	0.0262	0.7604	0.0295
128	misleading-moderate	0.7690	0.0632	0.7685	0.0373
128	null	0.6661	0.1318	0.6640	0.0716
G.5 T0 3B on RTE
num. shots	template category	median	q3 - q1	mean	std. dev.
4	instructive	0.6805	0.0704	0.6677	0.0580
4	irrelevant	0.6534	0.0596	0.6695	0.0450
4	misleading-extreme	0.6336	0.0379	0.6368	0.0469
4	misleading-moderate	0.6805	0.0966	0.6644	0.0525
4	null	0.6282	0.0442	0.6223	0.0292
8	instructive	0.6859	0.0361	0.6850	0.0438
8	irrelevant	0.6769	0.0487	0.6579	0.0674
8	misleading-extreme	0.6444	0.0749	0.6401	0.0543
8	misleading-moderate	0.6968	0.0478	0.6747	0.0530
8	null	0.6047	0.0514	0.6137	0.0357
16	instructive	0.7238	0.0325	0.7290	0.0284
16	irrelevant	0.7166	0.0433	0.7171	0.0315
16	misleading-extreme	0.6895	0.0415	0.6879	0.0410
16	misleading-moderate	0.7166	0.0523	0.7191	0.0337
16	null	0.6227	0.0596	0.6322	0.0423
32	instructive	0.7545	0.0542	0.7627	0.0369
32	irrelevant	0.7599	0.0695	0.7621	0.0397
32	misleading-extreme	0.7256	0.0451	0.7278	0.0361
32	misleading-moderate	0.7491	0.0406	0.7551	0.0279
32	null	0.6968	0.0632	0.6859	0.0578
64	instructive	0.8014	0.0289	0.8027	0.0190
64	irrelevant	0.7978	0.0298	0.8040	0.0204
64	misleading-extreme	0.7834	0.0271	0.7827	0.0201
64	misleading-moderate	0.7978	0.0361	0.8000	0.0225
64	null	0.7112	0.0912	0.7053	0.0600
128	instructive	0.8303	0.0253	0.8292	0.0161
128	irrelevant	0.8231	0.0153	0.8244	0.0118
128	misleading-extreme	0.8087	0.0190	0.8088	0.0174
128	misleading-moderate	0.8195	0.0135	0.8215	0.0152
128	null	0.7238	0.0966	0.7401	0.0505
G.6 T0 3B on ANLI R1
num. shots	template category	median	q3 - q1	mean	std. dev.
32	instructive	0.3640	0.0185	0.3664	0.0129
32	irrelevant	0.3660	0.0190	0.3637	0.0119
32	misleading-extreme	0.3610	0.0200	0.3638	0.0117
32	misleading-moderate	0.3650	0.0175	0.3631	0.0125
32	null	0.3580	0.0115	0.3580	0.0096
64	instructive	0.3835	0.0395	0.3797	0.0255
64	irrelevant	0.3810	0.0160	0.3878	0.0141
64	misleading-extreme	0.3830	0.0340	0.3753	0.0223
64	misleading-moderate	0.3775	0.0400	0.3749	0.0259
64	null	0.3785	0.0368	0.3817	0.0275
128	instructive	0.4260	0.0233	0.4226	0.0214
128	irrelevant	0.4150	0.0170	0.4190	0.0219
128	misleading-extreme	0.3930	0.0340	0.3975	0.0227
128	misleading-moderate	0.4140	0.0318	0.4092	0.0274
128	null	0.3850	0.0247	0.3852	0.0179
256	instructive	0.4790	0.0197	0.4804	0.0181
256	irrelevant	0.4650	0.0185	0.4640	0.0161
256	misleading-extreme	0.4700	0.0355	0.4654	0.0259
256	misleading-moderate	0.4690	0.0242	0.4670	0.0167
256	null	0.4355	0.0460	0.4260	0.0388
512	instructive	0.5135	0.0185	0.5123	0.0147
512	irrelevant	0.5080	0.0205	0.5088	0.0147
512	misleading-extreme	0.5010	0.0265	0.5007	0.0233
512	misleading-moderate	0.5065	0.0105	0.5066	0.0127
512	null	0.4590	0.0565	0.4615	0.0389
1024	instructive	0.5375	0.0477	0.5539	0.0406
1024	irrelevant	0.5490	0.0740	0.5690	0.0406
1024	misleading-extreme	0.5350	0.0255	0.5447	0.0304
1024	misleading-moderate	0.5350	0.0467	0.5403	0.0279
1024	null	0.5225	0.0543	0.5353	0.0651
G.7 T5 11B, T0 11B, and GPT-3 175B (Figure 2)
model	template category	median	q3 - q1	mean	std. dev.
GPT-3 (175B)	instructive	0.6534	0.0722	0.6472	0.0429
GPT-3 (175B)	irrelevant	0.6101	0.0361	0.6260	0.0326
GPT-3 (175B)	misleading-extreme	0.6173	0.0072	0.6217	0.0143
GPT-3 (175B)	misleading-moderate	0.6498	0.0578	0.6318	0.0480
T5 LMA (11B)	instructive	0.6679	0.1462	0.6797	0.0823
T5 LMA (11B)	irrelevant	0.6426	0.0776	0.6368	0.0488
T5 LMA (11B)	misleading-extreme	0.5993	0.0794	0.6070	0.0619
T5 LMA (11B)	misleading-moderate	0.5957	0.1137	0.6072	0.0653
T5 LMA (11B)	null	0.5560	0.0442	0.5578	0.0332
T0 (11B)	instructive	0.7942	0.0623	0.7959	0.0392
T0 (11B)	irrelevant	0.7906	0.0632	0.7942	0.0384
T0 (11B)	misleading-extreme	0.7401	0.0650	0.7338	0.0496
T0 (11B)	misleading-moderate	0.7942	0.0397	0.7858	0.0356
T0 (11B)	null	0.6986	0.0695	0.6847	0.0484
T0++ (11B)	instructive	0.8321	0.0316	0.8319	0.0282
T0++ (11B)	irrelevant	0.8267	0.0433	0.8207	0.0323
T0++ (11B)	misleading-extreme	0.8051	0.0614	0.8029	0.0593
T0++ (11B)	misleading-moderate	0.8159	0.0487	0.8039	0.0333
T0++ (11B)	null	0.7509	0.0505	0.7379	0.0362
H Results of Individual Templates
H.1 ALBERT
Figure 17: ALBERT with all irrelevant templates and the aggregated instructive for reference.
aggregated instructive templates
premise} If bonito flakes boil more than a few seconds, the stock becomes too strong? " {hypothesi
premise} Inflections are annoying and thank god that Middle English got rid of most of them. "hy
premise] Is the pious loved by the gods because it is pious? Or is it pious because it is loved by the
—{premise} Single-family zoning is bad for American cities. " {hypothesis}"? {mask}
premise} When Bolyai sent Gauss his discovery of non-Euclidean geometry, Gauss replied that he
Figure 18: ALBERT with all misleading-moderate templates and the aggregated instructive for reference.
aggregated instructive templates
- {premise} Are there lots of similar words in {"hypothesis"}? {mask}
premise] Can that be paraphrased as: " {hypothesis}?"? {mask}
—{premise} Can that be summarized as "  $\{\mathrm{hypothesis}\}^{\prime \prime}$  ?{mask}
premise} Do most of the above words appear in the following passage? {hypothesis} {mask}
—{premise} Does that have the same meaning as {"hypothesis"}? {mask}
Figure 19: ALBERT with all misleading-extreme templates and the aggregated instructive for reference.
aggregated instructive templates
—{premise} Does the paragraph start with "the"? {hypothesis} {mask}
premise}Is the sentiment positive?{hypothesis} {mask}
premise}Is this French? {hypothesis} {mask}
premise?  $\{\mathrm{hypothesis}\}$ $\{\mathrm{mask}\}$
premise}Is this grammatically correct?  $\{\mathrm{hypothesis}\}$  {mask}
Figure 20: ALBERT with all instructive templates.
Given that {premise} Does it follow that {"hypothesis"}? {mask}
Given that {premise} Therefore, it must be true that {"hypothesis"}? {mask}
Given{premise}Isitguaranteedtrue that""{hypothesis}""?{mask}
Given{premise}Should we assume that"\{\hypothesis\}"is true?  $\{\mathrm{mask}\}$
Suppose{premise} Can we infer that""{hypothesis}"? {mask}
premise question: hypothesis Yes or no? answer: mask
premise}Are we justified in saying that""?{mask}
premise}Based on the previous passage, is it true that {"hypothesis"}? {mask}
premise}Question:Does this imply that""{hypothesis}]?"{mask}
premise}Using only the above description and what you know about the world, \*\*hypothesis" is a
H.2 T0 (3B)
Figure 21: T0 (3B) with all irrelevant templates and the aggregated instructive for reference.
aggregated instructive templates
premise} If bonito flakes boil more than a few seconds, the stock becomes too strong? " {hypothesi
premise} Inflections are annoying and thank god that Middle English got rid of most of them. "hy
premise] Is the pious loved by the gods because it is pious? Or is it pious because it is loved by the
— {premise} Single-family zoning is bad for American cities. " \{hypothesis\}"?
premise} When Bolyai sent Gauss his discovery of non-Euclidean geometry, Gauss replied that he
Figure 22: T0 (3B) with all misleading-moderate templates and the aggregated instructive for reference.
aggregated instructive templates
premise} Are there lots of similar words in "  $\{\mathrm{hypothesis}\}^{\prime \prime}$
premise] Can that be paraphrased as: "  $\{\mathrm{hypothesis}\} ?$
—{premise} Can that be summarized as " {hypothesis}"?
— {premise} Do most of the above words appear in the following passage? {hypothesis}
—{premise} Does that have the same meaning as""{hypothesis}""?
Figure 23: T0 (3B) with all misleading-extreme templates and the aggregated instructive for reference.
aggregated instructive templates
—{premise} Does the paragraph start with "the"? {hypothesis}
premise}Is the sentiment positive?{hypothesis}
premise?
premise? {hypothesis}
premise Is this grammatically correct? {hypothesis}
Figure 24: T0 (3B) with all instructive templates.
Given that {premise} Does it follow that ""hypothesis\}?
Given that {premise} Therefore, it must be true that "  $\{\mathrm{hypothesis}\}^{\prime \prime}$
Given {premise} Is it guaranteed true that {"hypothesis"}?
Given{premise}Should we assume that""{hypothesis}" is true?
Suppose{premise} Can we infer that  $\forall$  {hypothesis}\}?
premise question: {hypothesis} Yes or no? answer:
premise} Are we justified in saying that "  $\{\mathrm{hypothesis}\}^{\prime \prime ?}$
premise}Based on the previous passage, is it true that""hypothesis  $\text{出}$  ?
—{premise}Question:Does this imply that""{hypothesis}\?"?
premise}Using only the above description and what you know about the world, \*\*hypothesis" is a
H.3 T5 LM-Adapted (3B)
Figure 25: T5 LM-Adapted (3B) with all irrelevant templates and the aggregated instructive for reference.
aggregated instructive templates
premise} If bonito flakes boil more than a few seconds, the stock becomes too strong? " {hypothesi
premise} Inflections are annoying and thank god that Middle English got rid of most of them. "hy
premise] Is the pious loved by the gods because it is pious? Or is it pious because it is loved by the
— {premise} Single-family zoning is bad for American cities. " \{hypothesis\}"?
premise} When Bolyai sent Gauss his discovery of non-Euclidean geometry, Gauss replied that he
Figure 26: T5 LM-Adapted (3B) with all misleading-moderate templates and the aggregated instructive for reference.
aggregated instructive templates
premise} Are there lots of similar words in "  $\{\mathrm{hypothesis}\}^{\prime \prime}$
premise] Can that be paraphrased as: "  $\{\mathrm{hypothesis}\}^{\prime \prime ?}$
—{premise} Can that be summarized as " {hypothesis}"?
— {premise} Do most of the above words appear in the following passage? {hypothesis}
—{premise} Does that have the same meaning as " {hypothesis}"?
Figure 27: T5 LM-Adapted (3B) with all misleading-extreme templates and the aggregated instructive for reference.
aggregated instructive templates
—{premise}Does the paragraph start with"the"? {hypothesis}
premise Is the sentiment positive? {hypothesis}
premise?
premise? {hypothesis}
premise] Is this grammatically correct? {hypothesis}
Figure 28: T5 LM-Adapted (3B) with all instructive templates.
Given that {premise} Does it follow that ""hypothesis\}?
Given that {premise} Therefore, it must be true that "  $\{\mathrm{hypothesis}\}^{\prime \prime}$
Given {premise} Is it guaranteed true that {"hypothesis"}?
Given{premise}Should we assume that""{hypothesis}" is true?
Suppose{premise} Can we infer that  $\forall$  {hypothesis}\}?
premise question: {hypothesis} Yes or no? answer:
premise} Are we justified in saying that "  $\{\mathrm{hypothesis}\}^{\prime \prime ?}$
premise}Based on the previous passage, is it true that""hypothesis  $\text{出}$  ?
—{premise} Question: Does this imply that " {hypothesis}"?
premise}Using only the above description and what you know about the world, \*\*hypothesis" is a
I Zero-Shot Results (Figure 4)
model	category	template name	accuracy
T0 (3B)	instructive	MNLI_YN	0.7148
T0 (3B)	instructive	GPT_YN	0.6823
T0 (3B)	instructive	justified_in_saying	0.6426
T0 (3B)	instructive	should assum	0.6498
T0 (3B)	instructive	is_it_true	0.6462
T0 (3B)	instructive	guaranteed_true	0.6209
T0 (3B)	instructive	can_we_infer	0.6354
T0 (3B)	instructive	does_it_FOLLOW	0.6715
T0 (3B)	instructive	does.this_imply	0.6679
T0 (3B)	instructive	modal_be_true	0.6354
T0 (3B)	misleading-moderate	words_appear	0.6462
T0 (3B)	misleading-moderate	similar_words	0.6354
T0 (3B)	misleading-moderate	same Meaning	0.6968
T0 (3B)	misleading-moderate	paraphrase	0.6390
T0 (3B)	misleading-moderate	summarize	0.6462
T0 (3B)	misleading-extreme	start_with_the	0.6968
T0 (3B)	misleading-extreme	grammatical	0.6859
T0 (3B)	misleading-extreme	sentiment	0.6462
T0 (3B)	misleading-extreme	sportsball	0.6426
T0 (3B)	misleading-extreme	french	0.5668
T0 (3B)	irrelevant	zoning	0.5704
T0 (3B)	irrelevant	gauss	0.5523
T0 (3B)	irrelevant	katsuobushi	0.5668
T0 (3B)	irrelevant	inflection	0.6751
T0 (3B)	irrelevant	euthyphro	0.6606
T0 (3B)	null	concat_PHM	0.6426
T0 (3B)	null	concat_HPM	0.6029
model	category	template name	accuracy
T0 (11B)	instructive	MNLI_YN	0.8051
T0 (11B)	instructive	GPT_YN	0.8014
T0 (11B)	instructive	justified_in_saying	0.7112
T0 (11B)	instructive	should assum	0.7437
T0 (11B)	instructive	is_it_true	0.8051
T0 (11B)	instructive	guaranteed_true	0.6968
T0 (11B)	instructive	can_we_infer	0.7690
T0 (11B)	instructive	does_it_FOLLOW	0.7509
T0 (11B)	instructive	does.this_imply	0.8014
T0 (11B)	instructive	modal_be_true	0.6895
T0 (11B)	misleading-moderate	words_appear	0.7184
T0 (11B)	misleading-moderate	similar_words	0.7148
T0 (11B)	misleading-moderate	same Meaning	0.7256
T0 (11B)	misleading-moderate	paraphrase	0.7256
T0 (11B)	misleading-moderate	summarize	0.6679
T0 (11B)	misleading-extreme	start_with_the	0.6823
T0 (11B)	misleading-extreme	grammatical	0.6390
T0 (11B)	misleading-extreme	sentiment	0.6318
T0 (11B)	misleading-extreme	sportsball	0.5921
T0 (11B)	misleading-extreme	french	0.5271
T0 (11B)	irrelevant	zoning	0.6318
T0 (11B)	irrelevant	gauss	0.5560
T0 (11B)	irrelevant	katsuobushi	0.5740
T0 (11B)	irrelevant	inflection	0.7004
T0 (11B)	irrelevant	euthyphro	0.6931
T0 (11B)	null	concat_PHM	0.6570
T0 (11B)	null	concat_HPM	0.6209
T0++ (11B)	instructive	MNLI_YN	0.8592
T0++ (11B)	instructive	GPT_YN	0.8231
T0++ (11B)	instructive	justified_in_saying	0.7726
T0++ (11B)	instructive	should assum	0.8231
T0++ (11B)	instructive	is_it_true	0.8556
T0++ (11B)	instructive	guaranteed_true	0.8231
T0++ (11B)	instructive	can_we_infer	0.8303
T0++ (11B)	instructive	does_it_FOLLOW	0.7798
T0++ (11B)	instructive	does_this_imply	0.8664
T0++ (11B)	instructive	modal_be_true	0.8087
T0++ (11B)	misleading-moderate	words_appear	0.7076
T0++ (11B)	misleading-moderate	similar_words	0.7329
T0++ (11B)	misleading-moderate	same_meaning	0.7545
T0++ (11B)	misleading-moderate	paraphrase	0.7617
T0++ (11B)	misleading-moderate	summarize	0.6968
T0++ (11B)	misleading-extreme	start_with_the	0.6498
T0++ (11B)	misleading-extreme	grammatical	0.7762
T0++ (11B)	misleading-extreme	sentiment	0.7365
T0++ (11B)	misleading-extreme	sportsball	0.5307
T0++ (11B)	misleading-extreme	french	0.4838
T0++ (11B)	irrelevant	zoning	0.5018
T0++ (11B)	irrelevant	gauss	0.5090
T0++ (11B)	irrelevant	katsuobushi	0.4801
T0++ (11B)	irrelevant	inflection	0.7220
T0++ (11B)	irrelevant	euthyphro	0.6715
T0++ (11B)	null	concat_PHM	0.6426
T0++ (11B)	null	concat_HPM	0.6029
J Comparison of LM targets, Controlling for the Template
Figure 29: The best performing irrelevant prompt for ALBERT, {premise} Single-family zoning is bad for American cities. {"hypothesis"}? [mask] with all LM targets.
Figure 30: The best-performing misleading prompt for ALBERT, {premise} Does the paragraph start with "the"? [mask] {"hypothesis"} with all LM targets.
LM Targets
yes;no
write;sleep
good; bad
true;false
cake;piano
no;yes
Mary;John
the;a
cat:dog
she;he
Figure 31: The best-performing null prompt for ALBERT, {premise} [mask] {"hypothesis"} with all LM targets.
K Preliminary Results on HANS
Template irrelevant intuitive misleading
Figure 32: Few-shot RTE-trained ALBERT's zero-shot performance on HANS (McCoy et al., 2019).  $\mathrm{L} =$  lexical,  $\mathrm{S} =$  subsequence,  $\mathrm{C} =$  constituency.  $\mathrm{E} =$  true label is entailment.  $\mathrm{N} =$  true label is non-entailment. Apologies but note the template category colors are different from those in the main text. "Intuitive" = instructive templates. In general, models perform similarly with instructive and irrelevant templates, but models with misleading templates fare worse, especially for lexical non-entailment cases (LN, fourth row). A full analysis will be furnished in a future version of this paper.
L Preliminary Results on Winograd
category	template	accuracy
instructive	Is "\( pronoun" the same as {referent} ? Yes or No?	0.6538
instructive	Does "\( pronoun" refer to {referent} ? Yes or No?	0.6731
instructive	Is "\( pronoun" {referent} ? Yes or No?	0.5385
instructive	Should "\( pronoun" be {referent} ? Yes or No?	0.5962
instructive	Does "\( pronoun" mean {referent} ? Yes or No?	0.6442
instructive	Is "\( pronoun" equivalent to {referent} ? Yes or No?	0.6058
instructive	Does "\( pronoun" stand for {referent} ? Yes or No?	0.6346
instructive	Can the pronoun "\( pronoun" be replaced with {referent} ? Yes or No?	0.6250
misleading-extreme	Did "\( pronoun" eat cakes with {referent} ? Yes or No?	0.6346
misleading-extreme	Is "\( pronoun" mother of {referent} ? Yes or No?	0.6346
misleading-extreme	Was "\( pronoun" friend to {referent} ? Yes or No?	0.6058
misleading-extreme	Did "\( pronoun" marry {referent} ? Yes or No?	0.6346
misleading-extreme	Can "\( pronoun" rent a car with {referent} ? Yes or No?	0.6346
misleading-extreme	Should "\( pronoun" be brother of {referent} ? Yes or No?	0.6346
misleading-extreme	Did "\( pronoun" speak to {referent} ? Yes or No?	0.5673
misleading-extreme	Is "\( pronoun" cousins with {referent} ? Yes or No?	0.6154
Figure 33: Zero-shot accuracy of T0 on Winograd Schema Challenge (Levesque et al., 2012; SuperGLUE version). We find no statistically significant difference between instructive and misleading-extreme templates.