SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 22: Accuracy per label for 0-shot evaluation.
Model	Mean	Particularised	Generalised	Other
OPT-125m	50.92	49.43 +/- 5.52	55.07 +/- 21.10	50.56 +/- 1.33
OPT-350m	57.14	47.92 +/- 4.37	69.20 +/- 8.36	56.68 +/- 4.99
OPT-1.3b	60.36	51.52 +/- 6.81	74.64 +/- 2.05	59.65 +/- 3.51
OPT-2.7b	59.56	50.19 +/- 5.06	69.93 +/- 5.53	59.22 +/- 6.14
OPT-6.7b	60.33	52.27 +/- 5.90	75.36 +/- 2.71	60.77 +/- 6.13
OPT-13b	61.03	55.49 +/- 8.79	75.00 +/- 5.72	58.79 +/- 5.51
OPT-30b	61.47	54.55 +/- 4.15	71.38 +/- 5.94	61.11 +/- 2.12
OPT-66b	61.33	55.11 +/- 7.33	69.93 +/- 12.26	61.46 +/- 3.68
OPT-175b	55.33	54.17 +/- 7.70	58.33 +/- 18.51	55.12 +/- 4.21
BLOOM-560m	51.58	50.76 +/- 5.59	48.91 +/- 25.22	51.59 +/- 3.50
BLOOM-1b1	51.17	50.57 +/- 6.32	53.26 +/- 27.40	51.16 +/- 2.41
BLOOM-1b7	53.61	50.38 +/- 7.95	59.78 +/- 18.82	53.23 +/- 1.62
BLOOM-3b	56.89	51.70 +/- 8.27	67.39 +/- 10.35	56.46 +/- 4.39
BLOOM-7b1	58.67	46.59 +/- 2.86	79.35 +/- 2.74	57.11 +/- 4.03
BLOOM-176b	54.22	54.73 +/- 10.61	60.14 +/- 16.73	53.57 +/- 1.65
EleutherAI-125m	51.89	50.38 +/- 5.71	57.25 +/- 20.15	50.90 +/- 0.99
EleutherAI-1.3b	53.14	50.57 +/- 7.03	55.43 +/- 23.20	53.32 +/- 2.05
EleutherAI-2.7b	59.17	50.95 +/- 7.43	74.64 +/- 4.97	58.10 +/- 2.92
EleutherAI-6b	56.36	53.22 +/- 6.86	69.20 +/- 7.36	55.73 +/- 2.43
EleutherAI-20b	57.53	49.43 +/- 6.68	72.83 +/- 5.99	56.20 +/- 3.01
Cohere-409m	51.61	51.33 +/- 4.84	52.54 +/- 22.71	51.25 +/- 2.93
Cohere-6b	57.28	51.52 +/- 6.68	64.49 +/- 16.06	57.06 +/- 3.13
Cohere-13b	57.19	52.27 +/- 5.83	68.12 +/- 10.99	57.45 +/- 3.35
Cohere-52b	58.50	51.52 +/- 7.21	73.91 +/- 5.75	56.85 +/- 3.81
GPT-3-350m	51.47	50.76 +/- 6.96	52.90 +/- 24.07	51.29 +/- 1.63
GPT-3-1.3b	57.72	50.00 +/- 6.29	67.75 +/- 9.84	57.06 +/- 3.78
GPT-3-6.7b	54.83	52.65 +/- 8.16	63.41 +/- 15.14	54.26 +/- 2.12
GPT-3-175b	57.22	53.03 +/- 1.93	71.01 +/- 4.81	54.61 +/- 5.58
T0-3b	48.25	55.68 +/- 0.66	27.17 +/- 2.74	49.83 +/- 1.90
T0-11b	55.61	57.95 +/- 2.18	47.10 +/- 17.15	56.33 +/- 6.49
BlenderBot-90m	46.64	55.49 +/- 0.42	23.91 +/- 0.00	48.32 +/- 0.00
BlenderBot-3b	53.44	44.51 +/- 0.42	76.09 +/- 0.00	51.81 +/- 0.20
BlenderBot-9b	53.36	49.24 +/- 4.81	71.01 +/- 5.71	51.03 +/- 1.63
Flan-T5-780m	63.31	59.28 +/- 3.90	68.84 +/- 7.60	62.23 +/- 3.13
Flan-T5-3b	52.50	54.55 +/- 1.61	48.19 +/- 11.73	53.14 +/- 3.15
Flan-T5-11b	60.78	51.52 +/- 3.39	73.19 +/- 7.28	59.60 +/- 2.18
Cohere-command-6b	66.31	58.90 +/- 3.62	73.19 +/- 2.71	66.15 +/- 2.41
Cohere-command-52b	60.22	55.49 +/- 4.66	60.51 +/- 16.67	59.99 +/- 5.09
text ADA-001-unknown	56.50	52.65 +/- 3.86	61.59 +/- 15.96	56.24 +/- 5.74
text-babbage-001-unknown	64.47	56.25 +/- 2.52	72.46 +/- 9.86	63.87 +/- 1.55
text-curie-001-unknown	68.94	66.48 +/- 2.34	68.84 +/- 5.98	68.48 +/- 3.63
text-davinci-001-unknown	72.31	59.66 +/- 5.07	79.35 +/- 9.78	73.17 +/- 2.54
text-davinci-002-unknown	70.58	64.20 +/- 3.75	80.07 +/- 5.67	69.94 +/- 3.69
text-davinci-003-unknown	71.25	63.64 +/- 1.86	82.25 +/- 4.77	71.23 +/- 2.74
ChatGPT-unknown	72.08	68.75 +/- 2.99	69.57 +/- 11.16	71.66 +/- 5.79
GPT-4-unknown	81.78	71.59 +/- 3.47	89.86 +/- 2.05	81.35 +/- 1.66
Humans	86.23	83.18	92.17	84.86
Table 23: Accuracy per label for 1-shot evaluation.
Model	Mean	World knowledge	Idiom	Rhetorical question
OPT-125m	52.72	43.48 +/- 5.02	54.92 +/- 7.24	59.09 +/- 4.55
OPT-350m	52.92	39.86 +/- 1.62	48.11 +/- 2.76	59.09 +/- 6.94
OPT-1.3b	56.31	54.35 +/- 4.16	58.33 +/- 4.48	53.03 +/- 12.22
OPT-2.7b	56.83	64.49 +/- 15.35	64.39 +/- 2.51	66.67 +/- 4.29
OPT-6.7b	60.08	61.59 +/- 13.84	68.94 +/- 6.90	56.06 +/- 6.25
OPT-13b	60.56	68.84 +/- 7.70	69.70 +/- 6.11	54.55 +/- 15.75
OPT-30b	60.33	71.74 +/- 5.47	63.26 +/- 4.81	51.52 +/- 10.05
OPT-66b	63.19	70.29 +/- 11.06	62.50 +/- 2.86	48.48 +/- 13.55
OPT-175b	58.36	63.77 +/- 4.81	66.67 +/- 8.77	57.58 +/- 17.14
BLOOM-560m	54.83	50.00 +/- 5.47	64.02 +/- 4.62	63.64 +/- 0.00
BLOOM-1b1	52.56	56.52 +/- 9.05	59.47 +/- 2.04	59.09 +/- 4.55
BLOOM-1b7	52.81	54.35 +/- 6.52	60.98 +/- 3.57	63.64 +/- 5.25
BLOOM-3b	55.94	50.72 +/- 4.10	64.39 +/- 4.08	59.09 +/- 4.55
BLOOM-7b1	57.00	50.00 +/- 3.32	64.77 +/- 2.86	62.12 +/- 6.25
BLOOM-176b	61.11	77.54 +/- 3.90	66.67 +/- 6.11	50.00 +/- 6.94
EleutherAI-125m	51.67	44.93 +/- 6.95	50.76 +/- 4.29	57.58 +/- 4.29
EleutherAI-1.3b	55.72	47.10 +/- 4.64	55.68 +/- 4.50	50.00 +/- 11.44
EleutherAI-2.7b	55.50	54.35 +/- 5.47	67.42 +/- 4.67	65.15 +/- 9.70
EleutherAI-6b	54.97	57.25 +/- 5.84	60.23 +/- 4.30	53.03 +/- 8.16
EleutherAI-20b	55.86	69.57 +/- 4.35	62.88 +/- 4.85	53.03 +/- 6.25
Cohere-409m	51.89	42.75 +/- 6.84	51.89 +/- 4.62	54.55 +/- 5.25
Cohere-6b	57.86	58.70 +/- 12.74	67.05 +/- 5.83	68.18 +/- 11.44
Cohere-13b	61.78	71.74 +/- 11.43	67.42 +/- 8.47	37.88 +/- 9.70
Cohere-52b	62.97	66.67 +/- 6.48	70.08 +/- 4.43	62.12 +/- 14.29
GPT-3-350m	55.97	50.72 +/- 4.10	61.74 +/- 5.15	69.70 +/- 12.49
GPT-3-1.3b	60.75	58.70 +/- 4.16	65.53 +/- 4.23	54.55 +/- 5.25
GPT-3-6.7b	61.17	60.87 +/- 11.77	69.32 +/- 3.65	56.06 +/- 8.16
GPT-3-175b	65.72	76.81 +/- 3.24	73.48 +/- 2.51	57.58 +/- 16.32
T0-3b	48.89	54.35 +/- 2.17	42.80 +/- 2.04	36.36 +/- 0.00
T0-11b	47.78	52.17 +/- 0.00	40.53 +/- 2.43	36.36 +/- 0.00
BlenderBot-90m	49.94	55.07 +/- 6.48	47.73 +/- 9.99	51.52 +/- 13.55
BlenderBot-3b	53.31	47.83 +/- 0.00	61.36 +/- 0.00	63.64 +/- 0.00
BlenderBot-9b	52.53	50.72 +/- 9.61	57.20 +/- 12.95	66.67 +/- 6.78
Flan-T5-780m	62.89	64.49 +/- 7.70	67.42 +/- 13.03	46.97 +/- 8.16
Flan-T5-3b	52.75	65.22 +/- 15.47	55.30 +/- 9.34	45.45 +/- 12.86
Flan-T5-11b	57.44	59.42 +/- 3.24	61.36 +/- 12.17	48.48 +/- 12.49
Cohere-command-6b	65.00	71.74 +/- 6.99	71.59 +/- 3.15	36.36 +/- 0.00
Cohere-command-52b	72.83	83.33 +/- 3.90	83.33 +/- 2.51	71.21 +/- 6.25
text ADA-001-unknown	57.36	60.87 +/- 7.10	67.80 +/- 3.81	66.67 +/- 6.78
text-babbage-001-unknown	63.89	68.84 +/- 3.90	76.89 +/- 2.43	50.00 +/- 11.44
text-curie-001-unknown	64.39	66.67 +/- 5.98	68.56 +/- 9.94	56.06 +/- 6.25
text-davinci-001-unknown	72.72	93.48 +/- 4.16	80.68 +/- 2.18	57.58 +/- 12.49
text-davinci-002-unknown	75.61	91.30 +/- 2.51	87.12 +/- 2.51	56.06 +/- 8.16
text-davinci-003-unknown	74.31	90.58 +/- 5.28	82.20 +/- 1.56	54.55 +/- 7.42
ChatGPT-unknown	75.11	86.23 +/- 2.99	85.61 +/- 3.12	56.06 +/- 14.29
GPT-4-unknown	82.31	97.10 +/- 3.24	88.64 +/- 3.94	89.39 +/- 3.39
Humans	86.23	93.04	92.73	92.73
Table 24: Accuracy per label for 1-shot evaluation.
Model	Mean	Particularised	Generalised	Other
OPT-125m	52.72	48.30 +/- 1.83	60.87 +/- 13.04	52.89 +/- 1.04
OPT-350m	52.92	47.73 +/- 2.37	60.87 +/- 11.71	54.35 +/- 3.10
OPT-1.3b	56.31	53.41 +/- 2.71	52.17 +/- 9.88	57.41 +/- 1.38
OPT-2.7b	56.83	49.81 +/- 5.31	69.93 +/- 6.33	55.17 +/- 3.94
OPT-6.7b	60.08	52.65 +/- 7.44	73.55 +/- 3.18	59.09 +/- 5.85
OPT-13b	60.56	53.03 +/- 1.82	71.01 +/- 7.60	59.56 +/- 2.75
OPT-30b	60.33	55.87 +/- 3.31	70.65 +/- 8.01	59.26 +/- 4.61
OPT-66b	63.19	60.04 +/- 4.12	67.39 +/- 8.70	63.39 +/- 4.28
OPT-175b	58.36	56.63 +/- 3.96	59.42 +/- 7.06	57.28 +/- 7.30
BLOOM-560m	54.83	43.94 +/- 3.12	66.30 +/- 8.21	54.82 +/- 1.85
BLOOM-1b1	52.56	47.35 +/- 3.12	63.04 +/- 12.36	51.16 +/- 1.54
BLOOM-1b7	52.81	45.64 +/- 3.50	67.03 +/- 9.92	51.29 +/- 1.44
BLOOM-3b	55.94	45.27 +/- 1.21	76.09 +/- 1.26	55.12 +/- 1.93
BLOOM-7b1	57.00	49.62 +/- 4.08	77.17 +/- 1.66	55.56 +/- 3.57
BLOOM-176b	61.11	58.14 +/- 3.31	66.67 +/- 5.98	59.73 +/- 3.66
EleutherAI-125m	51.67	50.19 +/- 2.49	53.99 +/- 8.82	52.11 +/- 0.89
EleutherAI-1.3b	55.72	50.57 +/- 4.67	57.97 +/- 13.44	57.36 +/- 2.66
EleutherAI-2.7b	55.50	48.67 +/- 4.84	65.22 +/- 4.86	54.22 +/- 2.79
EleutherAI-6b	54.97	49.81 +/- 1.21	66.30 +/- 4.82	54.01 +/- 3.36
EleutherAI-20b	55.86	53.03 +/- 3.57	64.49 +/- 6.36	53.83 +/- 2.41
Cohere-409m	51.89	52.84 +/- 3.98	48.55 +/- 3.69	52.54 +/- 1.96
Cohere-6b	57.86	44.13 +/- 1.66	77.54 +/- 2.05	57.15 +/- 5.08
Cohere-13b	61.78	53.98 +/- 2.05	74.28 +/- 3.64	61.41 +/- 1.84
Cohere-52b	62.97	60.42 +/- 8.18	69.20 +/- 5.24	61.76 +/- 4.21
GPT-3-350m	55.97	50.76 +/- 1.82	73.91 +/- 7.94	54.31 +/- 1.92
GPT-3-1.3b	60.75	53.79 +/- 2.98	68.48 +/- 3.26	61.07 +/- 1.82
GPT-3-6.7b	61.17	55.49 +/- 6.83	72.10 +/- 2.64	60.29 +/- 4.09
GPT-3-175b	65.72	62.31 +/- 4.17	64.86 +/- 7.26	65.33 +/- 2.00
T0-3b	48.89	56.25 +/- 1.57	34.06 +/- 4.29	49.83 +/- 0.55
T0-11b	47.78	56.44 +/- 0.54	27.54 +/- 1.02	49.22 +/- 0.53
BlenderBot-90m	49.94	52.46 +/- 4.27	44.57 +/- 15.66	50.00 +/- 1.65
BlenderBot-3b	53.31	44.51 +/- 0.42	76.09 +/- 0.00	51.59 +/- 0.24
BlenderBot-9b	52.53	54.92 +/- 3.45	55.80 +/- 12.90	50.90 +/- 2.60
Flan-T5-780m	62.89	56.44 +/- 3.32	68.84 +/- 12.90	63.44 +/- 6.28
Flan-T5-3b	52.75	55.11 +/- 1.57	44.20 +/- 5.98	52.41 +/- 3.23
Flan-T5-11b	57.44	53.98 +/- 1.94	62.68 +/- 15.85	57.28 +/- 4.79
Cohere-command-6b	65.00	60.61 +/- 3.69	68.12 +/- 9.53	65.25 +/- 1.37
Cohere-command-52b	72.83	67.42 +/- 2.83	80.07 +/- 2.92	71.36 +/- 1.70
text ADA-001-unknown	57.36	46.97 +/- 2.76	74.64 +/- 2.99	55.90 +/- 3.11
text-babbage-001-unknown	63.89	58.52 +/- 1.43	63.41 +/- 7.26	63.70 +/- 1.10
text-curie-001-unknown	64.39	60.98 +/- 2.14	69.93 +/- 3.42	64.04 +/- 5.79
text-davinci-001-unknown	72.72	62.31 +/- 1.66	76.81 +/- 2.71	72.83 +/- 1.70
text-davinci-002-unknown	75.61	68.18 +/- 2.86	77.54 +/- 2.05	75.32 +/- 3.14
text-davinci-003-unknown	74.31	64.20 +/- 1.43	80.43 +/- 5.02	74.50 +/- 1.29
ChatGPT-unknown	75.11	70.08 +/- 4.38	78.99 +/- 7.50	74.46 +/- 1.19
GPT-4-unknown	82.31	74.43 +/- 2.60	86.96 +/- 3.32	81.70 +/- 1.94
Humans	86.23	83.18	92.17	84.86
Table 25: Accuracy per label for 5-shot evaluation.
Model	Mean	World knowledge	Idiom	Rhetorical question
OPT-125m	50.22	44.93 +/- 3.24	57.58 +/- 7.73	57.58 +/- 4.29
OPT-350m	51.47	53.62 +/- 4.81	58.71 +/- 1.56	45.45 +/- 0.00
OPT-1.3b	58.03	68.84 +/- 8.48	63.26 +/- 6.21	30.30 +/- 8.57
OPT-2.7b	57.33	57.97 +/- 4.81	66.67 +/- 4.67	71.21 +/- 3.39
OPT-6.7b	63.31	66.67 +/- 16.20	67.42 +/- 4.08	42.42 +/- 16.32
OPT-13b	67.39	80.43 +/- 4.86	68.94 +/- 4.29	39.39 +/- 6.78
OPT-30b	65.64	84.78 +/- 8.60	66.29 +/- 8.13	37.88 +/- 6.25
OPT-66b	61.50	75.36 +/- 8.20	55.30 +/- 6.90	36.36 +/- 7.42
OPT-175b	63.89	78.26 +/- 7.10	65.15 +/- 2.83	43.94 +/- 3.39
BLOOM-560m	53.75	44.20 +/- 2.99	65.91 +/- 3.94	54.55 +/- 5.25
BLOOM-1b1	57.39	49.28 +/- 6.95	65.15 +/- 4.85	66.67 +/- 6.78
BLOOM-1b7	54.44	61.59 +/- 5.84	56.06 +/- 1.69	43.94 +/- 6.25
BLOOM-3b	57.19	50.72 +/- 3.24	64.77 +/- 4.87	63.64 +/- 12.86
BLOOM-7b1	54.50	50.00 +/- 2.17	62.88 +/- 1.69	69.70 +/- 4.29
BLOOM-176b	65.42	76.09 +/- 6.02	69.32 +/- 4.87	43.94 +/- 3.39
EleutherAI-125m	49.56	50.00 +/- 3.32	50.38 +/- 4.43	34.85 +/- 3.39
EleutherAI-1.3b	57.11	55.07 +/- 5.98	63.64 +/- 4.15	37.88 +/- 11.03
EleutherAI-2.7b	58.03	71.74 +/- 4.16	59.85 +/- 3.12	43.94 +/- 14.29
EleutherAI-6b	58.39	67.39 +/- 6.99	56.82 +/- 6.01	42.42 +/- 18.68
EleutherAI-20b	61.14	65.22 +/- 8.70	64.77 +/- 11.11	30.30 +/- 8.57
Cohere-409m	53.39	47.83 +/- 5.61	59.47 +/- 5.32	31.82 +/- 6.94
Cohere-6b	60.89	65.94 +/- 8.48	66.67 +/- 10.05	45.45 +/- 9.09
Cohere-13b	62.47	81.88 +/- 8.10	62.88 +/- 10.71	34.85 +/- 11.03
Cohere-52b	65.14	73.91 +/- 5.61	67.80 +/- 3.05	51.52 +/- 6.78
GPT-3-350m	55.72	46.38 +/- 3.24	65.53 +/- 1.56	51.52 +/- 4.29
GPT-3-1.3b	62.64	72.46 +/- 10.55	69.70 +/- 4.48	37.88 +/- 12.22
GPT-3-6.7b	62.39	76.81 +/- 14.57	62.50 +/- 5.53	36.36 +/- 7.42
GPT-3-175b	68.72	82.61 +/- 4.35	71.59 +/- 2.54	60.61 +/- 13.55
T0-3b	46.67	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
T0-11b	47.00	52.17 +/- 0.00	39.02 +/- 0.85	36.36 +/- 0.00
BlenderBot-90m	46.58	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
BlenderBot-3b	53.36	47.83 +/- 0.00	61.36 +/- 0.00	63.64 +/- 0.00
BlenderBot-9b	52.81	47.83 +/- 4.35	60.98 +/- 0.85	63.64 +/- 0.00
Flan-T5-780m	61.03	61.59 +/- 4.64	70.08 +/- 9.59	42.42 +/- 4.29
Flan-T5-3b	54.89	62.32 +/- 7.39	60.61 +/- 8.26	34.85 +/- 3.39
Flan-T5-11b	61.64	68.84 +/- 6.84	67.80 +/- 8.03	43.94 +/- 8.16
Cohere-command-6b	68.56	77.54 +/- 9.86	78.79 +/- 5.36	39.39 +/- 4.29
Cohere-command-52b	75.42	87.68 +/- 3.90	84.09 +/- 1.31	74.24 +/- 9.70
text-ada-001-unknown	57.61	52.17 +/- 3.55	64.39 +/- 2.83	62.12 +/- 8.16
text-babbage-001-unknown	66.14	71.74 +/- 2.17	77.65 +/- 5.15	57.58 +/- 12.49
text-curie-001-unknown	71.33	76.09 +/- 2.17	70.08 +/- 6.07	43.94 +/- 3.39
text-davinci-001-unknown	74.53	88.41 +/- 3.24	78.03 +/- 5.97	66.67 +/- 12.49
text-davinci-002-unknown	79.56	90.58 +/- 1.62	89.02 +/- 2.04	69.70 +/- 6.78
text-davinci-003-unknown	79.67	89.13 +/- 2.17	86.36 +/- 2.27	74.24 +/- 11.03
ChatGPT-unknown	73.89	86.96 +/- 6.15	87.88 +/- 4.85	75.76 +/- 12.49
GPT-4-unknown	82.03	95.65 +/- 2.51	86.74 +/- 2.04	87.88 +/- 6.78
Humans	86.23	93.04	92.73	92.73
Table 26: Accuracy per label for 5-shot evaluation.
Model	Mean	Particularised	Generalised	Other
OPT-125m	50.22	47.35 +/- 3.12	60.87 +/- 13.28	48.84 +/- 3.71
OPT-350m	51.47	39.58 +/- 1.79	67.39 +/- 3.07	51.38 +/- 0.95
OPT-1.3b	58.03	56.06 +/- 3.63	57.61 +/- 4.12	58.01 +/- 2.84
OPT-2.7b	57.33	47.35 +/- 3.05	72.46 +/- 2.40	56.20 +/- 3.60
OPT-6.7b	63.31	56.63 +/- 6.93	71.01 +/- 6.48	63.74 +/- 3.18
OPT-13b	67.39	60.23 +/- 2.93	64.86 +/- 2.92	69.12 +/- 2.83
OPT-30b	65.64	59.85 +/- 1.42	59.42 +/- 7.70	67.27 +/- 4.36
OPT-66b	61.50	56.44 +/- 3.51	58.70 +/- 9.64	63.65 +/- 3.93
OPT-175b	63.89	61.55 +/- 2.22	52.54 +/- 5.53	65.33 +/- 2.44
BLOOM-560m	53.75	44.89 +/- 2.05	73.19 +/- 6.11	52.50 +/- 0.78
BLOOM-1b1	57.39	48.67 +/- 3.44	70.65 +/- 4.12	57.02 +/- 1.86
BLOOM-1b7	54.44	48.86 +/- 4.91	60.14 +/- 3.24	54.74 +/- 0.96
BLOOM-3b	57.19	50.00 +/- 3.35	72.46 +/- 2.40	56.24 +/- 0.89
BLOOM-7b1	54.50	46.02 +/- 2.25	72.10 +/- 4.24	53.10 +/- 1.04
BLOOM-176b	65.42	65.53 +/- 4.38	49.28 +/- 9.69	66.88 +/- 3.42
EleutherAI-125m	49.56	44.32 +/- 5.76	56.88 +/- 4.05	50.04 +/- 2.49
EleutherAI-1.3b	57.11	50.76 +/- 3.26	69.93 +/- 4.77	56.89 +/- 1.57
EleutherAI-2.7b	58.03	51.52 +/- 2.43	61.59 +/- 4.10	58.61 +/- 1.39
EleutherAI-6b	58.39	49.62 +/- 1.69	63.04 +/- 5.02	59.91 +/- 5.04
EleutherAI-20b	61.14	51.52 +/- 2.43	61.59 +/- 12.78	63.48 +/- 5.29
Cohere-409m	53.39	50.38 +/- 1.69	68.48 +/- 4.82	52.45 +/- 0.99
Cohere-6b	60.89	52.65 +/- 2.14	64.49 +/- 5.98	61.80 +/- 4.77
Cohere-13b	62.47	59.66 +/- 6.11	68.84 +/- 7.28	62.02 +/- 3.56
Cohere-52b	65.14	60.04 +/- 4.07	68.12 +/- 7.39	65.46 +/- 3.30
GPT-3-350m	55.72	44.70 +/- 1.26	74.28 +/- 6.07	55.47 +/- 1.59
GPT-3-1.3b	62.64	49.24 +/- 2.51	67.39 +/- 4.35	64.38 +/- 2.51
GPT-3-6.7b	62.39	51.70 +/- 2.25	64.86 +/- 7.68	64.38 +/- 2.30
GPT-3-175b	68.72	60.98 +/- 5.74	66.67 +/- 8.76	69.90 +/- 0.68
T0-3b	46.67	55.68 +/- 0.00	23.91 +/- 0.00	48.32 +/- 0.15
T0-11b	47.00	55.87 +/- 0.42	25.00 +/- 1.66	48.62 +/- 0.23
BlenderBot-90m	46.58	55.11 +/- 1.27	24.28 +/- 0.81	48.28 +/- 0.31
BlenderBot-3b	53.36	44.32 +/- 0.00	76.09 +/- 0.00	51.72 +/- 0.10
BlenderBot-9b	52.81	44.32 +/- 0.93	75.72 +/- 0.81	50.95 +/- 1.02
Flan-T5-780m	61.03	54.36 +/- 3.50	71.01 +/- 12.96	60.77 +/- 6.09
Flan-T5-3b	54.89	57.01 +/- 1.79	41.30 +/- 9.64	55.47 +/- 4.00
Flan-T5-11b	61.64	56.25 +/- 3.20	64.86 +/- 17.04	61.76 +/- 5.21
Cohere-command-6b	68.56	60.23 +/- 5.00	74.28 +/- 6.57	68.91 +/- 1.47
Cohere-command-52b	75.42	70.08 +/- 3.39	77.17 +/- 3.26	74.68 +/- 2.80
text ADA-001-unknown	57.61	48.86 +/- 2.18	72.83 +/- 1.66	57.11 +/- 3.74
text-babbage-001-unknown	66.14	57.01 +/- 2.82	71.74 +/- 2.81	66.06 +/- 0.85
text-curie-001-unknown	71.33	60.04 +/- 0.78	69.93 +/- 5.67	74.63 +/- 0.98
text-davinci-001-unknown	74.53	60.80 +/- 3.75	81.16 +/- 3.69	75.80 +/- 1.32
text-davinci-002-unknown	79.56	71.02 +/- 2.76	87.68 +/- 1.62	79.03 +/- 2.26
text-davinci-003-unknown	79.67	71.59 +/- 1.86	87.68 +/- 1.02	79.33 +/- 1.12
ChatGPT-unknown	73.89	69.51 +/- 4.80	73.91 +/- 11.64	72.44 +/- 6.16
GPT-4-unknown	82.03	71.21 +/- 2.91	87.32 +/- 3.64	82.30 +/- 2.31
Humans	86.23	83.18	92.17	84.86
Table 27: Accuracy per label for 10-shot evaluation.
Model	Mean	World knowledge	Idiom	Rhetorical question
OPT-125m	52.89	55.80 +/- 8.48	55.68 +/- 8.78	66.67 +/- 6.78
OPT-350m	56.72	57.97 +/- 4.10	60.61 +/- 3.86	65.15 +/- 16.11
OPT-1.3b	59.92	70.29 +/- 4.64	54.17 +/- 3.57	34.85 +/- 3.39
OPT-2.7b	58.03	52.17 +/- 2.51	65.53 +/- 2.76	63.64 +/- 5.25
OPT-6.7b	63.28	71.01 +/- 5.42	65.53 +/- 6.07	53.03 +/- 17.73
OPT-13b	65.75	77.54 +/- 6.84	63.26 +/- 4.23	50.00 +/- 10.16
OPT-30b	63.36	78.99 +/- 4.64	56.06 +/- 8.57	33.33 +/- 4.29
OPT-66b	60.81	71.01 +/- 7.80	54.55 +/- 9.46	36.36 +/- 0.00
OPT-175b	60.75	76.81 +/- 4.10	58.33 +/- 6.52	43.94 +/- 11.03
BLOOM-560m	54.56	49.28 +/- 3.24	64.39 +/- 3.12	63.64 +/- 0.00
BLOOM-1b1	57.31	55.07 +/- 5.42	60.61 +/- 5.36	59.09 +/- 8.70
BLOOM-1b7	53.14	68.12 +/- 6.48	45.45 +/- 2.62	59.09 +/- 11.44
BLOOM-3b	59.39	54.35 +/- 4.86	65.53 +/- 5.32	66.67 +/- 4.29
BLOOM-7b1	56.11	53.62 +/- 7.39	65.53 +/- 3.81	69.70 +/- 6.78
BLOOM-176b	63.47	75.36 +/- 5.98	68.18 +/- 9.00	42.42 +/- 4.29
EleutherAI-125m	54.39	58.70 +/- 6.02	52.27 +/- 5.41	83.33 +/- 19.93
EleutherAI-1.3b	57.83	67.39 +/- 5.47	60.61 +/- 5.19	62.12 +/- 11.03
EleutherAI-2.7b	57.03	73.19 +/- 2.99	55.68 +/- 5.37	66.67 +/- 13.55
EleutherAI-6b	57.64	64.49 +/- 3.90	51.14 +/- 11.04	56.06 +/- 16.94
EleutherAI-20b	59.33	67.39 +/- 5.47	62.12 +/- 10.47	37.88 +/- 6.25
Cohere-409m	53.92	63.04 +/- 5.47	46.21 +/- 6.90	51.52 +/- 11.34
Cohere-6b	58.72	66.67 +/- 9.61	63.26 +/- 14.46	50.00 +/- 12.59
Cohere-13b	60.36	76.81 +/- 6.48	56.06 +/- 9.52	34.85 +/- 3.39
Cohere-52b	63.31	72.46 +/- 5.98	68.18 +/- 4.55	51.52 +/- 10.05
GPT-3-350m	57.72	53.62 +/- 3.24	63.64 +/- 6.01	65.15 +/- 8.16
GPT-3-1.3b	60.92	73.19 +/- 7.28	59.47 +/- 5.78	48.48 +/- 8.57
GPT-3-6.7b	63.94	71.01 +/- 7.80	67.80 +/- 1.56	40.91 +/- 8.70
GPT-3-175b	67.28	76.81 +/- 9.61	68.56 +/- 5.32	81.82 +/- 10.50
T0-3b	46.67	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
T0-11b	46.72	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
BlenderBot-90m	46.67	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
BlenderBot-3b	53.25	47.83 +/- 0.00	61.36 +/- 0.00	63.64 +/- 0.00
BlenderBot-9b	53.36	42.03 +/- 4.10	62.12 +/- 3.12	63.64 +/- 0.00
Flan-T5-780m	60.19	63.04 +/- 2.17	68.56 +/- 10.77	40.91 +/- 4.55
Flan-T5-3b	55.14	61.59 +/- 6.84	58.71 +/- 10.37	36.36 +/- 0.00
Flan-T5-11b	60.56	67.39 +/- 7.43	70.83 +/- 10.45	40.91 +/- 4.55
Cohere-command-6b	68.22	78.99 +/- 5.28	74.62 +/- 5.32	36.36 +/- 0.00
Cohere-command-52b	75.64	88.41 +/- 3.24	84.85 +/- 2.51	66.67 +/- 8.57
text ADA-001-unknown	57.36	64.49 +/- 7.28	56.44 +/- 5.78	57.58 +/- 6.78
text-babbage-001-unknown	63.53	67.39 +/- 2.17	73.11 +/- 4.62	68.18 +/- 4.55
text-curie-001-unknown	70.17	83.33 +/- 1.62	76.14 +/- 2.86	45.45 +/- 5.25
text-davinci-001-unknown	74.97	89.13 +/- 2.17	83.33 +/- 2.83	59.09 +/- 6.94
text-davinci-002-unknown	79.56	93.48 +/- 2.17	88.26 +/- 0.85	66.67 +/- 8.57
text-davinci-003-unknown	79.00	94.93 +/- 3.90	85.61 +/- 3.39	66.67 +/- 4.29
ChatGPT-unknown	74.28	84.06 +/- 6.48	86.36 +/- 4.35	62.12 +/- 11.03
GPT-4-unknown	81.31	94.93 +/- 2.99	86.74 +/- 4.03	89.39 +/- 3.39
Humans	86.23	93.04	92.73	92.73
Table 28: Accuracy per label for 10-shot evaluation.
Model	Mean	Particularised	Generalised	Other
OPT-125m	52.89	51.33 +/- 4.52	57.97 +/- 15.30	51.68 +/- 1.58
OPT-350m	56.72	55.11 +/- 1.57	72.10 +/- 2.32	54.39 +/- 1.50
OPT-1.3b	59.92	60.23 +/- 0.93	61.23 +/- 2.32	60.34 +/- 2.77
OPT-2.7b	58.03	49.62 +/- 3.32	74.64 +/- 2.05	57.19 +/- 3.77
OPT-6.7b	63.28	58.33 +/- 3.97	73.19 +/- 3.48	62.70 +/- 3.22
OPT-13b	65.75	60.23 +/- 2.45	72.10 +/- 3.18	66.37 +/- 2.84
OPT-30b	63.36	62.31 +/- 2.40	65.22 +/- 6.28	64.17 +/- 3.73
OPT-66b	60.81	57.58 +/- 1.26	60.51 +/- 7.98	62.36 +/- 3.33
OPT-175b	60.75	60.98 +/- 2.24	56.16 +/- 2.92	60.94 +/- 3.42
BLOOM-560m	54.56	46.21 +/- 1.26	73.19 +/- 2.99	53.06 +/- 0.77
BLOOM-1b1	57.31	47.54 +/- 4.97	64.13 +/- 7.40	58.31 +/- 3.30
BLOOM-1b7	53.14	53.03 +/- 3.45	53.62 +/- 4.64	52.80 +/- 1.74
BLOOM-3b	59.39	55.49 +/- 2.22	69.57 +/- 5.89	58.35 +/- 0.41
BLOOM-7b1	56.11	49.62 +/- 5.07	71.38 +/- 2.92	54.35 +/- 3.34
BLOOM-176b	63.47	68.18 +/- 3.47	50.36 +/- 9.00	63.35 +/- 4.15
EleutherAI-125m	54.39	55.11 +/- 3.13	63.41 +/- 6.20	52.20 +/- 2.23
EleutherAI-1.3b	57.83	50.00 +/- 2.78	69.20 +/- 2.32	57.15 +/- 1.52
EleutherAI-2.7b	57.03	57.20 +/- 2.76	57.25 +/- 6.23	55.77 +/- 0.96
EleutherAI-6b	57.64	56.25 +/- 2.76	59.42 +/- 10.25	58.05 +/- 4.71
EleutherAI-20b	59.33	57.39 +/- 1.83	63.04 +/- 13.63	59.04 +/- 3.30
Cohere-409m	53.92	57.39 +/- 2.60	66.30 +/- 4.98	51.94 +/- 1.99
Cohere-6b	58.72	51.70 +/- 2.34	64.86 +/- 6.33	58.74 +/- 5.75
Cohere-13b	60.36	58.52 +/- 1.27	70.29 +/- 5.98	59.73 +/- 5.24
Cohere-52b	63.31	53.41 +/- 2.93	67.75 +/- 7.88	64.17 +/- 2.12
GPT-3-350m	57.72	50.95 +/- 2.89	73.91 +/- 1.26	56.59 +/- 1.79
GPT-3-1.3b	60.92	57.01 +/- 5.01	63.77 +/- 2.71	61.15 +/- 1.10
GPT-3-6.7b	63.94	58.52 +/- 4.19	66.67 +/- 5.28	64.56 +/- 1.31
GPT-3-175b	67.28	63.45 +/- 2.66	68.84 +/- 4.64	66.75 +/- 2.65
T0-3b	46.67	55.68 +/- 0.00	24.28 +/- 0.81	48.28 +/- 0.10
T0-11b	46.72	55.68 +/- 0.00	24.28 +/- 0.81	48.36 +/- 0.10
BlenderBot-90m	46.67	55.68 +/- 0.00	23.91 +/- 0.00	48.32 +/- 0.00
BlenderBot-3b	53.25	44.70 +/- 0.54	76.09 +/- 0.00	51.46 +/- 0.23
BlenderBot-9b	53.36	45.27 +/- 1.21	76.09 +/- 1.77	51.77 +/- 0.69
Flan-T5-780m	60.19	54.17 +/- 2.51	71.38 +/- 10.75	59.60 +/- 4.49
Flan-T5-3b	55.14	54.92 +/- 1.42	43.48 +/- 12.10	56.29 +/- 3.84
Flan-T5-11b	60.56	59.66 +/- 3.33	57.61 +/- 13.09	60.03 +/- 5.13
Cohere-command-6b	68.22	63.07 +/- 4.44	77.17 +/- 6.73	67.79 +/- 2.45
Cohere-command-52b	75.64	70.27 +/- 1.53	76.45 +/- 4.24	75.15 +/- 1.17
text ADA-001-unknown	57.36	49.24 +/- 3.32	61.96 +/- 5.14	58.23 +/- 1.53
text-babbage-001-unknown	63.53	56.63 +/- 2.22	65.22 +/- 5.47	63.35 +/- 1.49
text-curie-001-unknown	70.17	62.69 +/- 2.12	67.75 +/- 7.36	71.32 +/- 1.01
text-davinci-001-unknown	74.97	63.83 +/- 1.21	80.80 +/- 1.95	75.41 +/- 1.79
text-davinci-002-unknown	79.56	70.08 +/- 1.56	84.78 +/- 2.51	79.59 +/- 2.79
text-davinci-003-unknown	79.00	68.18 +/- 1.31	87.32 +/- 1.49	79.07 +/- 1.38
ChatGPT-unknown	74.28	68.37 +/- 4.37	75.36 +/- 11.06	73.90 +/- 4.82
GPT-4-unknown	81.31	70.83 +/- 4.29	86.96 +/- 2.81	81.31 +/- 3.82
Humans	86.23	83.18	92.17	84.86