SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 29: Accuracy per label for 15-shot evaluation.
Model	Mean	World knowledge	Idiom	Rhetorical question
OPT-125m	51.86	44.93 +/- 4.10	53.41 +/- 8.48	43.94 +/- 19.93
OPT-350m	55.42	48.55 +/- 2.99	48.48 +/- 2.51	42.42 +/- 6.78
OPT-1.3b	61.61	64.49 +/- 5.28	68.94 +/- 4.67	42.42 +/- 6.78
OPT-2.7b	59.53	55.80 +/- 5.84	62.50 +/- 2.18	60.61 +/- 4.29
OPT-6.7b	64.72	55.80 +/- 7.28	68.18 +/- 3.47	60.61 +/- 16.32
OPT-13b	65.17	64.49 +/- 6.36	66.67 +/- 6.11	54.55 +/- 5.25
OPT-30b	64.06	68.84 +/- 4.64	60.23 +/- 5.98	43.94 +/- 8.16
OPT-66b	61.83	65.94 +/- 11.34	55.30 +/- 8.26	39.39 +/- 4.29
OPT-175b	64.78	76.09 +/- 11.16	67.05 +/- 9.44	50.00 +/- 6.94
BLOOM-560m	55.00	47.83 +/- 2.51	59.09 +/- 2.27	62.12 +/- 3.39
BLOOM-1b1	57.58	50.00 +/- 4.86	53.03 +/- 2.51	57.58 +/- 4.29
BLOOM-1b7	55.14	60.14 +/- 12.40	50.38 +/- 4.98	53.03 +/- 16.94
BLOOM-3b	58.69	44.93 +/- 3.24	61.36 +/- 6.94	57.58 +/- 6.78
BLOOM-7b1	55.67	55.07 +/- 7.80	61.36 +/- 2.93	56.06 +/- 8.16
BLOOM-176b	61.89	77.54 +/- 9.86	70.08 +/- 7.59	37.88 +/- 3.39
EleutherAI-125m	56.03	60.14 +/- 7.70	42.80 +/- 4.62	59.09 +/- 13.64
EleutherAI-1.3b	57.44	49.28 +/- 2.05	51.52 +/- 6.11	39.39 +/- 15.45
EleutherAI-2.7b	58.08	53.62 +/- 4.81	57.20 +/- 4.81	56.06 +/- 11.03
EleutherAI-6b	58.81	58.70 +/- 10.87	56.06 +/- 10.47	56.06 +/- 6.25
EleutherAI-20b	59.86	55.80 +/- 2.99	63.64 +/- 9.19	42.42 +/- 4.29
Cohere-409m	55.19	50.00 +/- 4.16	50.76 +/- 4.85	42.42 +/- 6.78
Cohere-6b	60.44	65.94 +/- 9.19	67.05 +/- 9.88	50.00 +/- 17.99
Cohere-13b	62.83	67.39 +/- 13.92	64.77 +/- 5.53	43.94 +/- 9.70
Cohere-52b	64.72	63.04 +/- 6.52	69.32 +/- 7.28	63.64 +/- 13.89
GPT-3-350m	58.83	50.00 +/- 7.43	53.41 +/- 1.74	42.42 +/- 13.55
GPT-3-1.3b	62.86	53.62 +/- 7.39	65.91 +/- 3.71	50.00 +/- 14.61
GPT-3-6.7b	65.17	62.32 +/- 6.95	63.64 +/- 2.27	51.52 +/- 10.05
GPT-3-175b	68.31	78.26 +/- 5.02	66.67 +/- 4.48	56.06 +/- 11.03
T0-3b	46.67	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
T0-11b	46.81	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
BlenderBot-90m	46.56	52.17 +/- 0.00	38.64 +/- 0.00	34.85 +/- 3.39
BlenderBot-3b	53.14	47.83 +/- 0.00	61.36 +/- 0.00	63.64 +/- 0.00
BlenderBot-9b	53.19	45.65 +/- 3.32	60.61 +/- 5.03	65.15 +/- 3.39
Flan-T5-780m	61.50	65.94 +/- 5.84	67.42 +/- 10.71	42.42 +/- 4.29
Flan-T5-3b	55.08	66.67 +/- 10.55	60.23 +/- 11.64	36.36 +/- 0.00
Flan-T5-11b	60.83	65.94 +/- 7.28	68.56 +/- 8.65	45.45 +/- 7.42
Cohere-command-6b	70.03	80.43 +/- 3.32	78.41 +/- 2.18	45.45 +/- 10.50
Cohere-command-52b	75.39	89.13 +/- 2.17	83.33 +/- 1.69	72.73 +/- 5.25
text-ada-001-unknown	58.28	55.07 +/- 5.98	56.06 +/- 5.67	63.64 +/- 13.89
text-babbage-001-unknown	65.19	63.04 +/- 4.86	77.27 +/- 3.71	68.18 +/- 6.94
text-curie-001-unknown	69.92	79.71 +/- 2.05	73.11 +/- 1.56	45.45 +/- 10.50
text-davinci-001-unknown	75.31	88.41 +/- 2.05	82.95 +/- 2.86	57.58 +/- 8.57
text-davinci-002-unknown	79.06	94.93 +/- 1.62	85.23 +/- 2.86	72.73 +/- 15.75
text-davinci-003-unknown	79.03	91.30 +/- 0.00	85.61 +/- 1.69	69.70 +/- 15.45
ChatGPT-unknown	75.56	86.23 +/- 4.64	86.74 +/- 4.03	60.61 +/- 10.05
GPT-4-unknown	82.08	95.65 +/- 2.51	81.44 +/- 2.43	90.91 +/- 0.00
Humans	86.23	93.04	92.73	92.73
Table 30: Accuracy per label for 15-shot evaluation.
Model	Mean	Particularised	Generalised	Other
OPT-125m	51.86	50.95 +/- 4.32	55.80 +/- 20.34	51.94 +/- 0.84
OPT-350m	55.42	55.30 +/- 3.39	60.51 +/- 4.24	56.29 +/- 0.75
OPT-1.3b	61.61	57.39 +/- 5.90	63.77 +/- 6.60	61.76 +/- 3.50
OPT-2.7b	59.53	49.24 +/- 2.98	75.00 +/- 1.66	59.78 +/- 5.03
OPT-6.7b	64.72	58.71 +/- 7.78	74.28 +/- 5.24	65.12 +/- 2.69
OPT-13b	65.17	64.02 +/- 2.34	65.22 +/- 7.32	65.50 +/- 1.25
OPT-30b	64.06	62.50 +/- 3.99	61.59 +/- 6.84	65.42 +/- 3.99
OPT-66b	61.83	58.71 +/- 7.21	57.61 +/- 5.14	64.25 +/- 3.20
OPT-175b	64.78	64.20 +/- 4.03	59.78 +/- 3.92	64.90 +/- 3.66
BLOOM-560m	55.00	44.13 +/- 2.58	74.64 +/- 4.64	54.78 +/- 1.75
BLOOM-1b1	57.58	45.45 +/- 1.74	64.13 +/- 5.58	60.42 +/- 2.34
BLOOM-1b7	55.14	52.27 +/- 5.21	57.61 +/- 8.40	55.73 +/- 0.84
BLOOM-3b	58.69	49.81 +/- 1.21	74.28 +/- 2.92	59.30 +/- 0.95
BLOOM-7b1	55.67	48.67 +/- 3.74	70.29 +/- 3.90	54.78 +/- 3.23
BLOOM-176b	61.89	64.02 +/- 4.43	46.38 +/- 11.20	62.02 +/- 4.53
EleutherAI-125m	56.03	56.82 +/- 3.47	54.35 +/- 5.89	57.11 +/- 0.65
EleutherAI-1.3b	57.44	51.14 +/- 3.21	61.59 +/- 9.11	59.95 +/- 2.44
EleutherAI-2.7b	58.08	57.58 +/- 2.24	61.96 +/- 6.49	58.27 +/- 1.50
EleutherAI-6b	58.81	54.17 +/- 6.07	66.67 +/- 8.48	59.35 +/- 4.11
EleutherAI-20b	59.86	55.11 +/- 3.98	62.68 +/- 11.87	60.85 +/- 4.53
Cohere-409m	55.19	52.65 +/- 1.69	61.23 +/- 8.08	56.12 +/- 2.19
Cohere-6b	60.44	49.62 +/- 2.60	71.01 +/- 6.48	60.85 +/- 4.38
Cohere-13b	62.83	57.77 +/- 3.17	71.01 +/- 5.98	63.09 +/- 2.62
Cohere-52b	64.72	57.39 +/- 2.69	71.01 +/- 4.81	65.16 +/- 1.07
GPT-3-350m	58.83	55.68 +/- 2.54	65.22 +/- 8.96	60.29 +/- 1.99
GPT-3-1.3b	62.86	56.06 +/- 5.67	63.04 +/- 7.63	64.86 +/- 1.78
GPT-3-6.7b	65.17	58.33 +/- 4.62	73.55 +/- 7.88	66.37 +/- 2.47
GPT-3-175b	68.31	64.77 +/- 4.10	71.38 +/- 6.33	68.60 +/- 3.97
T0-3b	46.67	55.68 +/- 0.00	23.91 +/- 0.00	48.32 +/- 0.00
T0-11b	46.81	55.68 +/- 0.00	25.00 +/- 1.09	48.41 +/- 0.12
BlenderBot-90m	46.56	55.68 +/- 0.00	23.91 +/- 0.00	48.19 +/- 0.13
BlenderBot-3b	53.14	44.32 +/- 0.00	75.36 +/- 1.02	51.46 +/- 0.23
BlenderBot-9b	53.19	44.13 +/- 1.79	75.72 +/- 1.49	51.72 +/- 0.74
Flan-T5-780m	61.50	56.63 +/- 2.22	71.74 +/- 11.30	60.90 +/- 4.55
Flan-T5-3b	55.08	56.82 +/- 1.31	45.65 +/- 12.92	55.04 +/- 3.39
Flan-T5-11b	60.83	57.01 +/- 3.37	60.14 +/- 15.81	61.02 +/- 5.34
Cohere-command-6b	70.03	60.80 +/- 4.72	72.83 +/- 8.30	70.84 +/- 1.68
Cohere-command-52b	75.39	69.89 +/- 2.43	76.81 +/- 3.90	74.76 +/- 1.01
text ADA-001-unknown	58.28	52.08 +/- 3.74	69.20 +/- 3.42	58.57 +/- 2.04
text-babbage-001-unknown	65.19	58.33 +/- 2.43	67.03 +/- 3.42	65.12 +/- 2.40
text-curie-001-unknown	69.92	62.50 +/- 1.47	68.84 +/- 6.72	71.40 +/- 0.93
text-davinci-001-unknown	75.31	64.58 +/- 2.12	83.70 +/- 1.66	75.54 +/- 0.95
text-davinci-002-unknown	79.06	72.92 +/- 1.02	86.96 +/- 3.97	77.99 +/- 2.77
text-davinci-003-unknown	79.03	69.32 +/- 2.37	87.68 +/- 1.62	78.94 +/- 1.12
ChatGPT-unknown	75.56	72.16 +/- 4.81	77.54 +/- 7.90	74.63 +/- 4.41
GPT-4-unknown	82.08	72.92 +/- 1.02	86.23 +/- 2.99	82.69 +/- 3.87
Humans	86.23	83.18	92.17	84.86
Table 31: Accuracy per label for 30-shot evaluation.
Model	Mean	World knowledge	Idiom	Rhetorical question
OPT-125m	51.50	55.80 +/- 5.28	54.55 +/- 9.28	54.55 +/- 9.09
OPT-350m	54.61	49.28 +/- 7.39	56.82 +/- 1.86	37.88 +/- 9.70
OPT-1.3b	61.67	71.01 +/- 3.24	67.80 +/- 5.48	28.79 +/- 11.03
OPT-2.7b	59.86	58.70 +/- 10.87	71.21 +/- 8.37	46.97 +/- 11.03
OPT-6.7b	63.61	62.32 +/- 11.13	67.05 +/- 2.86	46.97 +/- 19.93
OPT-13b	63.39	60.14 +/- 5.28	60.98 +/- 5.93	46.97 +/- 6.25
OPT-30b	65.47	71.74 +/- 8.23	62.88 +/- 7.38	37.88 +/- 3.39
OPT-66b	60.83	60.14 +/- 3.90	51.52 +/- 11.79	43.94 +/- 6.25
OPT-175b	62.44	65.94 +/- 10.77	62.50 +/- 13.43	60.61 +/- 4.29
BLOOM-560m	55.00	47.10 +/- 1.62	60.98 +/- 2.43	62.12 +/- 3.39
BLOOM-1b1	56.89	49.28 +/- 3.24	54.92 +/- 8.65	46.97 +/- 8.16
BLOOM-1b7	52.28	52.90 +/- 7.28	47.35 +/- 9.32	36.36 +/- 16.60
BLOOM-3b	58.64	50.72 +/- 2.05	62.50 +/- 5.68	59.09 +/- 6.94
BLOOM-7b1	57.61	50.72 +/- 5.98	61.74 +/- 3.81	54.55 +/- 9.09
BLOOM-176b	61.06	73.19 +/- 8.85	66.29 +/- 9.41	48.48 +/- 4.29
EleutherAI-125m	53.44	47.10 +/- 4.64	47.73 +/- 6.43	39.39 +/- 15.45
EleutherAI-1.3b	55.97	44.93 +/- 4.81	51.89 +/- 6.74	37.88 +/- 6.25
EleutherAI-2.7b	57.36	62.32 +/- 5.98	53.41 +/- 2.86	37.88 +/- 11.03
EleutherAI-6b	58.75	59.42 +/- 12.21	52.27 +/- 14.43	36.36 +/- 5.25
EleutherAI-20b	57.36	57.97 +/- 5.42	60.61 +/- 10.30	31.82 +/- 8.70
Cohere-409m	57.17	47.83 +/- 3.55	53.41 +/- 3.15	53.03 +/- 3.39
Cohere-6b	60.36	58.70 +/- 13.92	62.50 +/- 10.23	54.55 +/- 10.50
Cohere-13b	64.81	70.29 +/- 21.65	65.91 +/- 7.07	45.45 +/- 5.25
Cohere-52b	65.72	67.39 +/- 8.60	66.29 +/- 1.56	53.03 +/- 11.03
GPT-3-350m	60.25	55.07 +/- 2.05	57.95 +/- 5.83	51.52 +/- 10.05
GPT-3-1.3b	60.19	61.59 +/- 3.90	54.92 +/- 6.99	43.94 +/- 9.70
GPT-3-6.7b	62.86	56.52 +/- 4.35	65.53 +/- 3.32	50.00 +/- 6.94
GPT-3-175b	68.31	67.39 +/- 5.47	72.73 +/- 2.62	75.76 +/- 4.29
T0-3b	46.67	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
T0-11b	46.75	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
BlenderBot-90m	46.67	51.45 +/- 1.62	38.64 +/- 0.00	36.36 +/- 0.00
BlenderBot-3b	53.25	47.83 +/- 0.00	61.36 +/- 0.00	63.64 +/- 0.00
BlenderBot-9b	53.72	46.38 +/- 4.10	63.26 +/- 3.32	63.64 +/- 0.00
Flan-T5-780m	61.50	67.39 +/- 8.60	70.83 +/- 6.35	42.42 +/- 4.29
Flan-T5-3b	56.11	65.22 +/- 7.10	62.50 +/- 13.04	36.36 +/- 0.00
Flan-T5-11b	62.11	67.39 +/- 10.27	72.73 +/- 10.66	51.52 +/- 15.45
Cohere-command-6b	70.44	81.16 +/- 3.24	78.03 +/- 2.83	46.97 +/- 8.16
Cohere-command-52b	75.00	85.51 +/- 2.05	78.41 +/- 1.14	78.79 +/- 6.78
text-ada-001-unknown	55.58	50.72 +/- 7.39	57.58 +/- 4.29	57.58 +/- 8.57
text-babbage-001-unknown	66.00	67.39 +/- 5.47	71.59 +/- 3.15	63.64 +/- 5.25
text-curie-001-unknown	70.33	75.36 +/- 3.24	76.52 +/- 5.03	60.61 +/- 8.57
text-davinci-001-unknown	75.83	85.51 +/- 2.05	84.09 +/- 1.86	65.15 +/- 8.16
text-davinci-002-unknown	80.64	97.83 +/- 2.17	87.50 +/- 2.18	83.33 +/- 3.39
text-davinci-003-unknown	79.53	94.93 +/- 1.62	84.85 +/- 3.39	81.82 +/- 9.09
ChatGPT-unknown	75.64	87.68 +/- 4.64	89.02 +/- 4.43	83.33 +/- 9.70
GPT-4-unknown	82.17	95.65 +/- 3.55	87.12 +/- 3.12	90.91 +/- 0.00
Humans	86.23	93.04	92.73	92.73
Table 32: Accuracy per label for 30-shot evaluation.
Model	Mean	Particularised	Generalised	Other
OPT-125m	51.50	50.38 +/- 4.29	52.17 +/- 23.95	50.99 +/- 2.42
OPT-350m	54.61	55.30 +/- 1.26	49.28 +/- 3.48	55.51 +/- 1.92
OPT-1.3b	61.67	56.25 +/- 2.76	56.16 +/- 4.05	63.14 +/- 5.41
OPT-2.7b	59.86	50.19 +/- 2.89	69.57 +/- 4.16	59.95 +/- 5.23
OPT-6.7b	63.61	58.90 +/- 6.48	72.10 +/- 8.08	63.74 +/- 4.30
OPT-13b	63.39	59.47 +/- 2.14	64.86 +/- 6.07	65.07 +/- 2.24
OPT-30b	65.47	63.64 +/- 3.01	63.04 +/- 7.94	66.93 +/- 5.52
OPT-66b	60.83	63.07 +/- 3.64	55.43 +/- 9.29	62.62 +/- 4.98
OPT-175b	62.44	64.39 +/- 2.34	51.09 +/- 5.72	63.18 +/- 2.85
BLOOM-560m	55.00	47.92 +/- 2.31	71.01 +/- 6.23	54.18 +/- 1.82
BLOOM-1b1	56.89	49.81 +/- 4.93	56.88 +/- 8.46	59.35 +/- 3.16
BLOOM-1b7	52.28	55.49 +/- 1.79	40.94 +/- 8.82	53.83 +/- 1.25
BLOOM-3b	58.64	52.46 +/- 1.53	68.84 +/- 3.69	58.74 +/- 1.44
BLOOM-7b1	57.61	54.36 +/- 8.98	70.29 +/- 2.05	56.76 +/- 4.88
BLOOM-176b	61.06	67.42 +/- 1.93	50.00 +/- 9.05	60.08 +/- 4.01
EleutherAI-125m	53.44	59.47 +/- 2.43	46.74 +/- 4.65	54.18 +/- 1.21
EleutherAI-1.3b	55.97	51.70 +/- 5.20	56.52 +/- 10.80	58.40 +/- 1.35
EleutherAI-2.7b	57.36	57.39 +/- 2.25	51.81 +/- 3.18	58.79 +/- 0.89
EleutherAI-6b	58.75	57.77 +/- 4.22	58.33 +/- 9.92	60.38 +/- 4.52
EleutherAI-20b	57.36	51.52 +/- 4.08	59.42 +/- 14.51	58.79 +/- 2.39
Cohere-409m	57.17	51.14 +/- 3.99	64.49 +/- 8.57	58.66 +/- 2.15
Cohere-6b	60.36	51.52 +/- 2.51	70.29 +/- 6.11	61.20 +/- 5.15
Cohere-13b	64.81	58.14 +/- 2.96	68.12 +/- 8.20	65.98 +/- 3.57
Cohere-52b	65.72	64.96 +/- 4.17	69.93 +/- 2.32	65.50 +/- 2.01
GPT-3-350m	60.25	57.58 +/- 2.51	65.22 +/- 6.28	60.98 +/- 1.63
GPT-3-1.3b	60.19	57.77 +/- 5.66	57.61 +/- 9.12	62.06 +/- 4.02
GPT-3-6.7b	62.86	61.17 +/- 6.41	63.41 +/- 7.98	63.65 +/- 2.52
GPT-3-175b	68.31	64.58 +/- 5.06	70.29 +/- 6.23	68.17 +/- 2.01
T0-3b	46.67	55.68 +/- 0.00	23.91 +/- 0.00	48.32 +/- 0.00
T0-11b	46.75	55.87 +/- 0.42	23.91 +/- 0.00	48.41 +/- 0.12
BlenderBot-90m	46.67	55.68 +/- 0.00	23.55 +/- 0.81	48.41 +/- 0.24
BlenderBot-3b	53.25	44.32 +/- 0.00	76.09 +/- 0.00	51.55 +/- 0.20
BlenderBot-9b	53.72	45.08 +/- 1.82	75.36 +/- 1.02	52.11 +/- 0.93
Flan-T5-780m	61.50	57.01 +/- 3.85	73.19 +/- 8.67	60.16 +/- 3.75
Flan-T5-3b	56.11	56.44 +/- 1.56	47.83 +/- 11.71	56.29 +/- 4.18
Flan-T5-11b	62.11	61.36 +/- 4.50	57.25 +/- 16.68	61.58 +/- 6.25
Cohere-command-6b	70.44	64.96 +/- 1.21	78.62 +/- 5.53	69.81 +/- 1.60
Cohere-command-52b	75.00	71.78 +/- 1.66	75.00 +/- 3.49	74.55 +/- 0.49
text ADA-001-unknown	55.58	53.41 +/- 1.97	56.52 +/- 4.35	55.86 +/- 3.03
text-babbage-001-unknown	66.00	60.80 +/- 2.69	62.32 +/- 6.11	66.88 +/- 2.36
text-curie-001-unknown	70.33	60.42 +/- 5.01	74.28 +/- 6.20	71.32 +/- 1.42
text-davinci-001-unknown	75.83	67.05 +/- 1.97	83.33 +/- 3.48	75.67 +/- 1.02
text-davinci-002-unknown	80.64	74.43 +/- 1.83	83.70 +/- 2.74	79.76 +/- 1.44
text-davinci-003-unknown	79.53	72.92 +/- 2.49	86.59 +/- 1.95	78.55 +/- 1.20
ChatGPT-unknown	75.64	67.99 +/- 2.74	78.26 +/- 6.15	74.55 +/- 3.90
GPT-4-unknown	82.17	71.97 +/- 2.83	86.23 +/- 3.48	82.34 +/- 2.67
Humans	86.23	83.18	92.17	84.86
Table 33: Accuracy per label for model group Example IT for 5-shot chain-of-thought evaluation.
Model	Mean	World knowledge	Idiom	Rhetorical question
Cohere-command-6b	69.14	72.46 +/- 5.98	78.03 +/- 3.12	62.12 +/- 8.16
Cohere-command-52b	75.28	78.99 +/- 1.62	84.47 +/- 3.57	51.52 +/- 8.57
text-ada-001-unknown	15.33	11.59 +/- 8.20	17.42 +/- 9.43	10.61 +/- 9.70
text-babbage-001-unknown	47.67	47.83 +/- 11.50	55.30 +/- 16.21	42.42 +/- 8.57
text-curie-001-unknown	68.22	69.57 +/- 6.64	79.17 +/- 0.85	69.70 +/- 10.05
text-davinci-001-unknown	67.25	69.57 +/- 7.10	71.59 +/- 3.65	60.61 +/- 11.34
text-davinci-002-unknown	80.06	92.03 +/- 1.62	88.26 +/- 3.57	46.97 +/- 24.29
text-davinci-003-unknown	83.61	93.48 +/- 2.17	93.18 +/- 0.00	69.70 +/- 10.05
ChatGPT-unknown	77.19	89.86 +/- 4.10	87.88 +/- 3.63	65.15 +/- 9.70
GPT-4-unknown	86.47	93.48 +/- 3.32	93.18 +/- 2.93	87.88 +/- 4.29
Humans	86.23	93.04	92.73	92.73
Table 34: Accuracy per label for model group Example IT for 5-shot chain-of-thought evaluation.
Model	Mean	Particularised	Generalised	Other
Cohere-command-6b	69.14	58.33 +/- 1.93	81.52 +/- 2.08	69.04 +/- 2.04
Cohere-command-52b	75.28	68.94 +/- 3.19	77.17 +/- 2.08	75.88 +/- 0.53
text-ada-001-unknown	15.33	15.53 +/- 7.73	14.86 +/- 9.26	15.50 +/- 8.33
text-babbage-001-unknown	47.67	45.27 +/- 11.94	40.94 +/- 19.34	48.19 +/- 14.11
text-curie-001-unknown	68.22	59.47 +/- 5.15	74.28 +/- 7.88	68.04 +/- 1.75
text-davinci-001-unknown	67.25	64.58 +/- 3.85	64.13 +/- 5.14	67.92 +/- 3.30
text-davinci-002-unknown	80.06	75.95 +/- 3.68	80.07 +/- 6.69	80.23 +/- 1.07
text-davinci-003-unknown	83.61	77.46 +/- 1.02	87.32 +/- 3.18	83.25 +/- 0.96
ChatGPT-unknown	77.19	72.35 +/- 1.56	80.43 +/- 5.47	76.23 +/- 1.11
GPT-4-unknown	86.47	81.63 +/- 2.58	88.77 +/- 4.05	86.05 +/- 1.17
Humans	86.23	83.18	92.17	84.86
Table 35: Accuracy per prompt template for BERT-cased.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	47.3	48.8	50.5	49.8	46.7	46.7
2	46.8	50.3	45.5	50.2	46.7	46.5
3	57.3	51.5	50.0	50.0	47.0	46.7
4	48.8	51.0	49.5	48.5	46.8	46.7
5	46.7	50.3	44.5	47.7	46.7	46.7
6	46.7	50.3	45.8	47.8	46.8	46.7
Mean	48.9	50.4	47.6	49.0	46.8	46.7
- std	3.81	0.832	2.42	1.04	0.107	0.0745
Structured	51.1	50.4	50.0	49.4	46.8	46.7
- std	4.4	1.17	0.408	0.665	0.125	7.11e-15
Natural	46.7	50.3	45.3	48.6	46.7	46.6
- std	0.0471	7.11e-15	0.556	1.16	0.0471	0.0943
Table 36: Accuracy per prompt template for BERT-uncased.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	57.0	53.2	51.8	55.2	51.7	49.3
2	53.7	50.3	54.0	48.7	49.0	49.3
3	54.7	54.7	57.3	55.5	53.3	52.8
4	56.7	51.5	52.3	54.0	50.3	49.5
5	53.2	50.2	50.2	48.3	48.2	47.2
6	53.3	50.3	54.2	49.2	53.0	53.5
Mean	54.8	51.7	53.3	51.8	50.9	50.3
- std	1.55	1.71	2.24	3.13	1.92	2.19
Structured	56.1	53.1	53.8	54.9	51.8	50.5
- std	1.02	1.31	2.48	0.648	1.23	1.6
Natural	53.4	50.3	52.8	48.7	50.1	50.0
- std	0.216	0.0471	1.84	0.368	2.1	2.62
Table 37: Accuracy per prompt template for RoBERTa-base.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	54.0	55.8	58.0	58.7	58.3	57.8
2	56.5	50.5	52.0	55.8	56.0	54.2
3	53.0	56.8	56.8	61.3	59.5	58.8
4	55.2	56.0	58.7	59.8	56.8	57.2
5	55.7	50.3	52.3	54.8	55.5	53.0
6	59.2	50.3	54.2	55.8	55.7	55.3
Mean	55.6	53.3	55.3	57.7	57.0	56.1
- std	1.97	2.93	2.65	2.38	1.47	2.05
Structured	54.1	56.2	57.8	59.9	58.2	57.9
- std	0.899	0.432	0.785	1.07	1.1	0.66
Natural	57.1	50.4	52.8	55.5	55.7	54.2
- std	1.5	0.0943	0.974	0.471	0.205	0.939
Table 38: Accuracy per prompt template for RoBERTa-large.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	57.7	50.2	62.0	64.7	64.7	60.5
2	46.7	53.3	58.5	64.2	61.2	55.7
3	60.8	54.8	64.5	62.8	61.8	59.5
4	66.2	50.3	64.0	59.0	57.0	58.2
5	46.7	53.3	58.8	63.5	60.5	56.5
6	46.7	55.5	59.3	60.0	60.8	52.3
Mean	54.1	52.9	61.2	62.4	61.0	57.1
- std	7.84	2.03	2.45	2.13	2.26	2.7
Structured	61.6	51.8	63.5	62.2	61.2	59.4
- std	3.51	2.15	1.08	2.37	3.18	0.942
Natural	46.7	54.0	58.9	62.6	60.8	54.8
- std	7.11e-15	1.04	0.33	1.84	0.287	1.82
Table 39: Accuracy per prompt template for GPT-2-medium.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.2	53.7	54.0	53.8	53.8	55.0
2	52.8	53.7	55.8	57.2	60.3	57.2
3	53.7	54.0	52.5	56.5	55.8	55.3
4	53.5	55.7	53.3	55.8	55.5	54.3
5	59.2	54.3	56.7	57.7	60.7	58.8
6	58.3	54.8	55.7	57.7	61.7	57.8
Mean	55.1	54.4	54.7	56.4	58.0	56.4
- std	2.6	0.706	1.5	1.36	3.03	1.63
Structured	53.5	54.5	53.3	55.4	55.0	54.9
- std	0.205	0.881	0.613	1.14	0.881	0.419
Natural	56.8	54.3	56.1	57.5	60.9	57.9
- std	2.83	0.45	0.45	0.236	0.589	0.66
Table 40: Accuracy per prompt template for GPT-2-large.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.3	53.3	54.5	53.5	55.3	56.2
2	47.5	56.7	57.5	57.8	60.8	61.0
3	55.0	53.8	55.7	54.0	54.8	56.0
4	54.0	53.7	56.2	53.5	54.8	56.7
5	47.2	54.5	56.7	58.8	61.2	60.8
6	47.0	53.3	57.2	59.5	60.3	60.8
Mean	50.7	54.2	56.3	56.2	57.9	58.6
- std	3.47	1.18	1.0	2.57	2.92	2.29
Structured	54.1	53.6	55.5	53.7	55.0	56.3
- std	0.698	0.216	0.713	0.236	0.236	0.294
Natural	47.2	54.8	57.1	58.7	60.8	60.9
- std	0.205	1.41	0.33	0.698	0.368	0.0943
Table 41: Accuracy per prompt template for GPT-2-xl.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.2	53.3	57.0	54.5	54.7	56.2
2	48.7	61.3	57.3	63.7	62.0	60.5
3	55.0	55.2	59.5	59.0	58.0	60.7
4	54.2	54.3	56.0	54.5	54.3	56.3
5	48.0	59.7	58.3	60.8	62.7	61.7
6	48.5	60.8	58.0	61.8	61.5	61.5
Mean	51.3	57.4	57.7	59.1	58.9	59.5
- std	2.92	3.25	1.1	3.5	3.43	2.32
Structured	54.1	54.3	57.5	56.0	55.7	57.7
- std	0.736	0.776	1.47	2.12	1.66	2.1
Natural	48.4	60.6	57.9	62.1	62.1	61.2
- std	0.294	0.668	0.419	1.2	0.492	0.525
Table 42: Accuracy per prompt template for EleutherAI-125M.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.3	53.7	52.7	56.2	56.2	54.0
2	52.2	50.0	47.5	53.5	55.7	53.3
3	53.3	53.8	51.2	55.8	54.8	52.8
4	53.7	52.5	51.2	53.8	55.8	53.2
5	50.7	50.2	47.3	53.8	56.2	53.8
6	48.2	49.8	47.5	53.2	57.5	53.5
Mean	51.9	51.7	49.6	54.4	56.0	53.4
- std	1.93	1.72	2.19	1.17	0.806	0.394
Structured	53.4	53.3	51.7	55.3	55.6	53.3
- std	0.189	0.591	0.707	1.05	0.589	0.499
Natural	50.4	50.0	47.4	53.5	56.5	53.5
- std	1.65	0.163	0.0943	0.245	0.759	0.205
Table 43: Accuracy per prompt template for EleutherAI-1.3B.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	54.3	53.7	54.8	57.5	57.2	56.2
2	51.8	56.8	57.5	59.0	55.8	54.7
3	58.0	55.5	59.5	58.0	61.5	57.5
4	53.2	57.5	56.8	55.2	56.5	54.7
5	49.7	55.2	57.5	58.7	57.2	56.7
6	51.8	55.7	56.5	58.7	56.5	56.2
Mean	53.1	55.7	57.1	57.8	57.4	56.0
- std	2.59	1.21	1.4	1.29	1.87	1.02
Structured	55.2	55.6	57.0	56.9	58.4	56.1
- std	2.05	1.55	1.93	1.22	2.21	1.14
Natural	51.1	55.9	57.2	58.8	56.5	55.9
- std	0.99	0.668	0.471	0.141	0.572	0.85
Table 44: Accuracy per prompt template for EleutherAI-2.7B.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	54.0	52.8	58.2	57.8	59.5	56.7
2	62.0	56.2	57.7	55.8	57.8	57.7
3	58.7	60.0	58.8	59.2	57.8	57.8
4	56.5	54.2	57.5	56.2	57.5	55.5
5	62.7	54.7	58.7	55.7	57.3	57.8
6	61.2	55.2	57.3	57.5	58.5	58.7
Mean	59.2	55.5	58.0	57.0	58.1	57.4
- std	3.13	2.25	0.576	1.26	0.741	1.02
Structured	56.4	55.7	58.2	57.7	58.3	56.7
- std	1.92	3.12	0.531	1.23	0.881	0.939
Natural	62.0	55.4	57.9	56.3	57.9	58.1
- std	0.613	0.624	0.589	0.826	0.492	0.45
Table 45: Accuracy per prompt template for EleutherAI-6B.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	57.5	58.8	52.7	53.0	52.5	51.3
2	57.7	51.8	63.2	62.7	64.3	65.3
3	56.2	58.2	57.2	53.0	54.7	54.5
4	52.8	55.5	53.3	52.2	54.0	53.8
5	56.8	52.7	62.7	63.2	65.2	64.2
6	57.2	52.8	61.3	61.8	62.2	63.3
Mean	56.4	55.0	58.4	57.6	58.8	58.7
- std	1.67	2.75	4.28	4.94	5.2	5.65
Structured	55.5	57.5	54.4	52.7	53.7	53.2
- std	1.98	1.44	1.99	0.377	0.918	1.37
Natural	57.2	52.4	62.4	62.6	63.9	64.3
- std	0.368	0.45	0.804	0.579	1.26	0.818
Table 46: Accuracy per prompt template for EleutherAI-20B.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.0	58.0	55.3	54.3	52.8	54.3
2	61.3	54.2	65.8	63.3	65.0	60.3
3	54.3	58.3	58.5	56.7	55.3	52.0
4	56.2	58.2	55.3	57.2	57.0	58.7
5	59.0	53.0	66.7	62.8	65.0	59.2
6	61.3	53.5	65.2	61.7	64.0	59.7
Mean	57.5	55.9	61.1	59.3	59.9	57.4
- std	3.25	2.33	4.9	3.42	4.98	3.09
Structured	54.5	58.2	56.4	56.1	55.0	55.0
- std	1.31	0.125	1.51	1.27	1.72	2.78
Natural	60.5	53.6	65.9	62.6	64.7	59.7
- std	1.08	0.492	0.616	0.668	0.471	0.45
Table 47: Accuracy per prompt template for BLOOM-560M.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	54.3	54.2	53.5	53.8	53.8	53.5
2	46.7	56.3	54.0	54.8	56.0	55.3
3	58.8	53.3	53.8	53.3	54.5	54.0
4	56.3	54.8	53.5	54.8	52.7	56.7
5	46.7	54.3	53.7	55.3	56.3	55.5
6	46.7	56.0	54.0	55.2	56.7	55.0
Mean	51.6	54.8	53.8	54.5	55.0	55.0
- std	5.05	1.04	0.206	0.734	1.45	1.04
Structured	56.5	54.1	53.6	54.0	53.7	54.7
- std	1.84	0.616	0.141	0.624	0.741	1.41
Natural	46.7	55.5	53.9	55.1	56.3	55.3
- std	7.11e-15	0.881	0.141	0.216	0.287	0.205
Table 48: Accuracy per prompt template for BLOOM-1B1.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.3	53.5	56.2	54.2	55.2	54.5
2	49.0	51.5	58.2	59.8	58.8	60.8
3	57.2	54.2	55.8	54.0	55.5	50.8
4	53.3	54.0	54.2	53.3	55.7	55.8
5	47.3	51.2	59.8	61.3	60.2	60.0
6	46.8	51.0	60.2	61.2	60.2	59.3
Mean	51.2	52.6	57.4	57.3	57.6	56.9
- std	3.75	1.36	2.18	3.51	2.19	3.53
Structured	54.6	53.9	55.4	53.8	55.5	53.7
- std	1.84	0.294	0.864	0.386	0.205	2.12
Natural	47.7	51.2	59.4	60.8	59.7	60.0
- std	0.942	0.205	0.864	0.685	0.66	0.613
Table 49: Accuracy per prompt template for BLOOM-1B7.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.5	54.7	53.8	54.0	55.7	56.5
2	57.7	52.2	56.3	55.5	55.8	52.0
3	54.7	53.2	53.8	51.0	54.5	54.0
4	54.5	53.8	54.5	51.2	55.5	50.3
5	50.0	51.2	54.3	53.2	54.7	50.0
6	51.3	51.8	53.8	54.0	54.7	50.8
Mean	53.6	52.8	54.4	53.1	55.1	52.3
- std	2.49	1.2	0.886	1.6	0.528	2.31
Structured	54.2	53.9	54.0	52.1	55.2	53.6
- std	0.525	0.616	0.33	1.37	0.525	2.55
Natural	53.0	51.7	54.8	54.2	55.1	50.9
- std	3.37	0.411	1.08	0.953	0.519	0.822
Table 50: Accuracy per prompt template for BLOOM-3B.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.0	54.0	56.8	59.5	60.0	58.2
2	62.5	58.0	58.2	59.7	57.5	60.0
3	53.5	54.0	57.2	58.7	59.2	58.2
4	54.8	55.3	55.7	59.0	58.2	55.8
5	58.5	57.5	58.0	59.7	58.8	60.2
6	59.0	56.8	57.3	59.8	58.5	59.5
Mean	56.9	55.9	57.2	59.4	58.7	58.6
- std	3.4	1.6	0.823	0.408	0.783	1.5
Structured	53.8	54.4	56.6	59.1	59.1	57.4
- std	0.759	0.613	0.634	0.33	0.736	1.13
Natural	60.0	57.4	57.8	59.7	58.3	59.9
- std	1.78	0.492	0.386	0.0471	0.556	0.294
Table 51: Accuracy per prompt template for BLOOM-7B1.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.2	55.2	55.2	52.0	53.0	52.7
2	61.2	59.0	53.7	58.3	58.8	61.7
3	58.7	53.3	53.0	53.3	53.0	52.8
4	53.5	53.5	55.2	52.8	54.3	53.5
5	62.0	61.0	55.3	60.3	58.5	62.5
6	63.5	60.0	54.7	59.8	56.3	62.5
Mean	58.7	57.0	54.5	56.1	55.7	57.6
- std	4.03	3.11	0.871	3.46	2.39	4.63
Structured	55.1	54.0	54.5	52.7	53.4	53.0
- std	2.52	0.852	1.04	0.535	0.613	0.356
Natural	62.2	60.0	54.6	59.5	57.9	62.2
- std	0.953	0.816	0.66	0.85	1.11	0.377
Table 52: Accuracy per prompt template for BLOOM-176B.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.8	58.8	58.5	57.7	55.7	56.7
2	55.8	60.8	68.0	65.7	64.2	62.7
3	53.5	66.7	69.3	71.8	71.7	69.8
4	54.3	59.8	64.8	62.2	60.7	61.3
5	52.3	61.3	66.2	61.8	58.8	57.5
6	55.5	59.2	65.7	61.7	60.3	58.3
Mean	54.2	61.1	65.4	63.5	61.9	61.1
- std	1.19	2.65	3.43	4.38	5.06	4.44
Structured	53.9	61.8	64.2	63.9	62.7	62.6
- std	0.33	3.51	4.43	5.88	6.68	5.43
Natural	54.5	60.4	66.6	63.1	61.1	59.5
- std	1.58	0.896	0.988	1.86	2.28	2.29
Table 53: Accuracy per prompt template for OPT-125M.
Template	k = 0	k = 1	k = 5	k = 10	k = 15	k = 30
1	53.3	55.2	54.0	55.2	54.2	55.0
2	49.5	50.5	47.5	52.7	50.5	48.2
3	53.5	55.5	53.0	55.0	53.7	56.0
4	53.3	54.5	54.2	53.8	54.3	53.8
5	48.5	50.5	46.3	50.7	49.5	48.0
6	47.3	50.2	46.3	50.0	49.0	48.0
Mean	50.9	52.7	50.2	52.9	51.9	51.5
- std	2.55	2.35	3.56	1.99	2.25	3.49
Structured	53.4	55.1	53.7	54.7	54.1	54.9
- std	0.0943	0.419	0.525	0.618	0.262	0.899
Natural	48.4	50.4	46.7	51.1	49.7	48.1
- std	0.899	0.141	0.566	1.14	0.624	0.0943