SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Response: 'We are all sold out. We wont probably have any more in for a few weeks.'
Implicature: Yes. But we dont have any calculators to sell now.
This is an example that GPT-4 gets wrong for all templates 5-shot, probably because the response has a negative sentiment, and also it means they do not sell calculators right now. However, the implicature is yes, because they do sell them normally. The CoT completion is:
CoT completion: 'Alice says they are all sold out and won't have any more in for a few weeks. Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. Alice must be implying that they do sell calculators, but they currently don't have any in stock. The answer to Bob's question is yes. Answer: yes'
This example is similar to the one given in the main text in Section 4, and there are more like these in the dataset: the sentiment is negative, and an interpretation can be negative, but strictly the implicature is positive (yes).
Utterance: 'Do you know where he is?'
Response: 'I'll find him.'
Implicature: No.
GPT-4 gets this wrong for all templates 5-shot, and right for all of them with CoT:
CoT completion: 'Alice says she will find him. Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. If Alice knew where he is, she would have provided that information directly. Instead, she says she will find him, implying that she does not currently know his location. The answer to Bob's question 'Do you know where he is?' must therefore be no. Answer: no'
It's unclear why GPT-4 gets this one wrong 5-shot (all humans get this one right), but the CoT is very relevant and helpful.
Utterance: 'It seems Cathy is not happy at all with her new room mate. Did you ask her why?'
Response: 'Believe me. I tried to find out. But I simply could not pin it down.'
Implicature: Yes. But Cathy did not give specific reasons for her feelings.
Again an example where the sentiment of the response seems to hint at no. Of 5 human annotators, also only 2 interpret this response as "yes". With a CoT, GPT-4 gets it right six times:
CoT completion: 'Alice says she tried to find out but couldn't pin it down. Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. Alice must mean that she attempted to ask Cathy why she is unhappy with her new roommate, but was unable to get a clear answer. The answer to Bob's question 'Did you ask her why?' must therefore be yes. Answer: yes'
A helpful reasoning trace. All CoT completions by the models we have run CoT on are available in the GitHub: https://github.com/LauraRuis/do-pigs-fly.
K.1 Contrastive experiment
In this section we reframe the implicature resolution task to a contrastive one, allowing the model to contrast the coherent to the incoherent sentence in a single prompt.
Contrastive task. In the ranking task the model is required to assign higher likelihood to the coherent utterance than the incoherent one  $(p_{\theta}(\mathbf{x}) > p_{\theta}(\hat{\mathbf{x}}))$ . In assigning a likelihood to  $\mathbf{x}$ , the model has no knowledge of  $\hat{\mathbf{x}}$ , and vice-versa. We hypothesize that the task might become easier if we reformulate it as a contrastive task. Consider the following prompt  $\mathbf{p}$ .
Which of the following sentences is coherent:
A: Esther asked "Can you come to my party on Friday?" and Juan responded "I have to work", which means no.
B: Esther asked "Can you come to my party on Friday?" and Juan responded "I have to work", which means yes.
We can now evaluate the models' ability to understand which is the coherent sentence by evaluating whether it assigns  $p_{\theta}(A \mid \mathbf{p}) > p_{\theta}(B \mid \mathbf{p})$ . Note that this can again be framed in a ranking task of assigning a higher likelihood to the coherent prompt. If we finish the above prompt  $\mathbf{p}$  by adding "A" to make a coherent prompt  $\mathbf{x}$  and "B" to make an incoherent prompt  $\hat{\mathbf{x}}$  we can again formulate the task by  $p_{\theta}(\mathbf{x}) > p_{\theta}(\hat{\mathbf{x}})$ . The difference is that within both the coherent and the incoherent prompt, the model can contrast the coherent and incoherent utterance to each other. We randomise the assignment of A and B to the utterances.
We do a small experiment with the contrastive task with one of the best performing models overall, OpenAI's text-davinci-002, for  $k = \{0,1,5\}$ . We use two prompt templates and for each template try three different multiple choice answers: A and B like above, one and two, or the full text of the answer. For the last option the coherent prompt x would look as follows:
Answer: Esther asked "Can you come to my party on Friday?" and Juan responded "I have to work", which means no.
Table 11: Performance on the implicature task framed contrastively by OpenAI's text-davinci-002. The mean and standard deviation are reported over two different prompt templates (template 1 and 2).
k	Non-contrastive	Rank one, two	Rank A, B	Rank full text
0	71.3% ± 1.75	53.9% ± 0.9	59.3% ± 1.3	48.9% ± 0.6
1	76.1% ± 2.6	59.4% ± 1.6	63.2% ± 2.0	66.9% ± 0.9
5	80.5% ± 2.3	61.4% ± 1.3	64.0% ± 1.3	67.9% ± 2.1
In Table 11, perhaps surprisingly, we can see that the contrastive task is much more difficult than the original ranking task. For  $k = 0$ , the result is random except for the prompt where the multiple choice options are A and B. For  $k = \{1,5\}$  the full text ranking does best, but is still significantly worse than the original ranking setup. Because of these disappointing results, we did not evaluate the other models contrastively. Future work must establish whether the contrastive setup is worse across all model classes and sizes.
K.2 Variance over prompt ordering
As mentioned in Section 3 in the main text, models are sensitive to the ordering of the  $k$  examples in the prompt. Instead of marginalising over this random factor by evaluating all possible prompt orderings, we randomly sampled an ordered set of examples from the development set for each test example. Throughout experiments, we kept this randomly sampled order the same, meaning if you re-run the 5-shot evaluation you get exactly the same orderings. The reason for this is that we want evaluate each model equally. In this section we ask how the performance chances for the best performing model if we select another random order. We do this for the 5-shot evaluation, because the results show that adding more in-context examples barely helps performance.
Table 12: Variance over prompt ordering for 5-shot evaluation per prompt template (P.T.) for text-davinci-002
Seed	P. T. 1	P. T. 2	P. T. 3	P. T. 4	P. T. 5	P. T. 6	Mean
0	80.17	78.17	82.83	80.50	79.17	76.50	79.56
1	80.17	76.17	81.33	81.83	76.00	76.33	78.64
2	79.50	78.17	81.17	80.17	78.17	76.50	78.94
mean	79.94	77.50	81.78	80.83	77.78	76.44	-
std	0.31	0.94	0.75	0.72	1.32	0.08	-
Table 12 shows the results of this experiment. Some prompt templates seem to be more sensitive to prompt example ordering than others, but for none of them the variance is high enough to change any conclusions.
K.3 Different zero-shot instruction prompts
There is a narrative around large language models that if they fail a task, it might be that the prompt was not the right one (through works like Reynolds and McDonell (2021b); Kojima et al. (2022)). The idea is that they can be prompted to simulate almost anything, if you set them up correctly. Because implicature resolution is a ubiquitous result of learning language, we hold the view that a model should be able to do this task if a prompt is given in coherent natural language. Nonetheless, in an additional effort to find the "let's think step-by-step" (Kojima et al., 2022) of zero-shot implicature resolution we try three more prompt templates.
We evaluate a base large language model and two in-structable models: GPT-3-175B, text-davinci-001, and text-davinci-002. The prompts we use are taken from recent work that proposes a dialogue agent trained with human feedback (Glaese et al., 2022), but adapted to the task of implicature resolution. The full prompts are presented in Table 7 and Table 13 shows the results. The new templates do not improve performance for any of these models. The variance over the prompt templates for text-davinci-002 is
Table 13: Zero-shot accuracy over three additional prompt templates for a base LLM and two instructable models.
Model	Templates
GPT-3-175b	59.2% ± 4.5
text-davinci-001-?	66.1% ± 3.2
text-davinci-002-?	67.7% ± 9.6
high, and the best prompt template of these three does achieve a slightly higher accuracy than the others:  $74.5\%$ . These results do not change the picture sketched so far.
K.4 The effect of in-context examples on sensitivity to prompt wording
Figure 7 shows the relative performance increase due to in-context prompting broken down per prompt template. For text-davinci-001, most templates benefit similarly from more in-context examples, except for template 1. Perhaps surprisingly, we see that this template already achieves a performance of  $76.5\%$  at the zero-shot evaluation and does not improve much with few-shot prompting. For Cohore-52B and OPT-175B we see a clear grouping between the structured prompts (dashed lines) and natural prompts (dotted lines). Cohere struggles significantly more with the structured prompts than with the natural prompts in the zero-shot evaluation, and few-shot prompting can mitigate that, lowering the standard deviation over prompt templates to 1.89 at  $k = 30$  from 4 at  $k = 0$ . OPT benefits from prompting for the natural prompts, but not for the structured prompts.
Relative accuracy (w.r.t. 0-shot) due to in-context examples for all prompt templates.
Figure 7: Relative performance increase over 0-shot due to in-context prompting. Structured prompt templates are dashed lines (1, 3, 4) and natural prompt templates dotted lines (2, 5, 6).
K.5 Variance over API runs
In this section we comment on the reproducibility of research done using APIs. OpenAI and Cohere have their models behind an API, meaning we do not have control over what happens to the prompt before the model processes it. We run the zero-shot evaluation ten more times for two models of OpenAI and Cohere, text-davinci-002 and Cohere-52B. The results from this experiment are shown in Table 14 and 15. From this we can conclude that there is some stochasticity in the API that we have no control over, a bit more for OpenAI than for Cohere, but again we can be relatively confident that the conclusion will not be different because of it. The results from this work are therefore reproducible with access to the same models behind the API now. Unfortunately, when OpenAI or Cohere changes the models behind the API, these results are not exactly reproducible anymore.
For completeness, we add the timestamp that each result was obtained below (Appendix L).
Table 14: Results per prompt template (P.T.) for 10 different runs from text-davinci-002 for 0-shot evaluation.
Each evaluation has exactly the same text, so the variance in performance is due to API stochasticity.
API-run	P. T. 1	P. T. 2	P. T. 3	P. T. 4	P. T. 5	P. T. 6	Mean
0	73.50	68.83	73.00	71.17	67.17	68.83	70.42
1	73.83	69.00	72.83	71.50	67.67	68.33	70.53
2	73.67	68.67	73.17	71.33	67.50	68.50	70.47
3	73.83	68.17	73.17	71.00	67.67	68.17	70.33
4	73.67	68.83	73.33	71.17	67.00	68.33	70.39
5	73.83	68.50	73.00	71.00	67.00	68.17	70.25
6	73.67	69.00	73.00	71.17	67.33	68.50	70.44
7	73.67	68.67	72.83	71.33	67.50	68.67	70.44
8	73.83	69.17	72.83	71.17	67.33	68.00	70.39
9	73.50	68.50	72.83	71.00	67.50	68.67	70.33
10	73.67	69.50	73.00	71.33	67.50	68.50	70.58
mean	73.70	68.80	73.00	71.20	67.38	68.42	-
std	0.12	0.35	0.16	0.16	0.23	0.24	-
K.6 Experiment with random in-context labels
This paper presents the thesis that instruction-tuning at the example level ("Example IT") is important for pragmatic understanding in LLMs. However, the 0-shot result that one of the models in the Example IT group achieves is similar to that of base models; Cohore-command-52b obtains a zero-shot performance of  $60.2\%$ . From the sharp rise in performance observed for the  $k = 0$  to  $k = 1$  result (from  $60.2\%$  to  $72.8\%$ ) we hypothesise that the k-shot in-context examples in this task do not necessarily teach the model pragmatics in-context, but prime the model for the task format (namely,
Table 15: Results per prompt template (P.T.) for 10 different runs from Cohere-52B for 0-shot evaluation.
Each evaluation has exactly the same text, so the variance in performance is due to API stochasticity.
API-run	P. T. 1	P. T. 2	P. T. 3	P. T. 4	P. T. 5	P. T. 6	Mean
0	56.00	62.67	54.33	54.00	62.17	62.17	58.56
1	56.00	62.83	54.33	54.00	62.33	62.33	58.64
2	56.00	62.83	54.33	54.00	62.17	62.33	58.61
3	56.00	62.83	54.33	54.00	62.17	62.33	58.61
4	55.83	62.67	54.33	54.00	62.17	62.33	58.56
5	56.00	62.83	54.33	54.00	62.17	62.17	58.58
6	56.00	62.83	54.33	54.00	62.17	62.17	58.58
7	56.00	62.67	54.33	54.00	62.33	62.17	58.58
8	56.00	62.83	54.33	54.00	62.00	62.33	58.58
9	56.00	62.83	54.00	53.83	62.17	62.17	58.50
mean	55.98	62.78	54.30	53.98	62.18	62.25	-
std	0.05	0.08	0.10	0.05	0.09	0.08	-
outputting either "yes" or "no" as detailed in Section 3 in the main text). If this hypothesis is true, we would observe similar performance regardless of whether the labels given in the prompt for the few-shot examples are true. We test this empirically for two base models (GPT-3, Cohere-52b) and two Example IT models (text-davinci-001, Cohere-command-52b) for 1-shot and 5-shot evaluation. The results can be found in Table 16. We find that for the Example IT models in-context prompts with random labels obtain the same results (i.e. within confidence intervals) as the experiments with ground-truth labels in the in-context examples. For base models however we do observe a drop in performance; for GPT-3-175b at 5-shot, and Cohere-52b both at 1- and 5-shot. Taken together, we can conclude that for base models the content of the in-context prompt seems important, whereas for models in the example IT group the in-context examples mainly serve as a primer for the task structure.
Table 16: The results of the 1- and 5-shot experiment with random labels for the few-shot examples as opposed to the true labels. We find that performance does not degrade for the models in the Example IT group, which implies that for these models not the content of the examples is important for performance, but the structure.
Model	1-shot	1-shot rand labels	5-shot	5-shot rand labels
GPT-3-175b	65.7% ± 1.4	65.4% ± 1.2	68.7% ± 1.5	64.7% ± 1.9
Cohere-52b	63.0% ± 3.8	58.3% ± 3.3	65.1% ± 2.9	60.5% ± 1.9
text-davinci-001	72.7% ± 1.3	73.9% ± 1.7	74.5% ± 1.0	73.4% ± 1.2
Cohere-command-52b	72.8% ± 1.3	72.0% ± 1.6	75.4% ± 1.8	73.5% ± 2.7
Table 17: Results of the chain-of-thought (CoT) experiment for models in the base group. The numbers between brackets show the difference in performance with the number on the same row one column to the left. These models do not benefit from CoT-prompting. The reason Cohere-6b achieves such a low score for CoT-prompting is because it is not able to adhere to the correct output format (yes/no).
Model	0-shot	5-shot	5-shot CoT
GPT-3-350m	51.5% ± 3.0	55.7% ± 1.6 (+4.2%)	55.0% ± 3.5 (-0.7%)
GPT-3-1.3b	57.7% ± 3.1	62.6% ± 2.0 (+4.9%)	54.4% ± 5.8 (-8.2%)
GPT-3-6.7b	54.8% ± 1.9	62.4% ± 1.5 (+7.6%)	61.0% ± 2.3 (+4.0%)
GPT-3-175b	57.2% ± 4.4	68.7% ± 1.5 (+11.5%)	60.3% ± 4.2 (-8.4%)
Cohere-6b	57.3% ± 2.2	60.9% ± 4.1 (+3.6%)	29.2% ± 14.7 (-31.7%)
Cohere-52b	58.5% ± 4.0	65.1% ± 2.9 (+6.6%)	64.7% ± 3.2 (-0.4%)
K.7 Chain-of-thought on base models
In the main paper we do a CoT experiment on the models in the Example IT group. Base models also benefit from in-context examples, so it makes sense to also try CoT prompting on these models. After attempting this for two of the model classes in the group, we decided not to apply this prompting technique to the other models, because it decreases performance, sometimes significantly. See the results of the CoT experiment on the two base model classes in Table 17
K.8 Testing for spurious correlations
In this section, we do a small scale experiment to test whether the benchmark has spurious correlations. Specifically, we run the benchmark with only the utterance or only the response as input. Strictly, getting the implicature right from the response only does not always indicate spurious correlations, as some examples only need the response (e.g. rhetorical questions like 'do pigs fly?'). Utterance-only results do always indicate spurious correlations. We run this experiment for GPT-3.5-turbo and GPT-4 0-shot and 5-shot (see Table 18 and Table 19).
Table 18: Results of running the benchmark with only the utterance as input, to test for spurious correlations with the label.
Utterance-only	0-shot	5-shot
GPT-3.5-Turbo	54.3% ± 3.3	41.7% ± 12.4
GPT-4	48.9% ± 10.5	53.7% ± 0.5
Table 19: Results of running the benchmark with only the response as input, to test what part of the examples can be resolved without the utterance.
Response-only	0-shot	5-shot
GPT-3.5-Turbo	59.2% ± 4.7	58.3% ± 6.6
GPT-4	62.6% ± 1.7	65.5% ± 1.1
We find that models mostly perform random for utterance-only, so spurious correlations do not seem to be an issue. For response-only, GPT-4 5-shot gets  $65\%$  accuracy. Some examples it gets right are: "do fish swim?" and "let's hope so".
Table 20: An example from the dataset for each type of implicature found in the test set. The rightmost column shows the amount of that type we manually found in the test set.
Type	Example Utterance	Example Response	Impl.	#
Generalised	You know all these people?	Some.	No.	47
Particularised	Want to stay for a nightcap?	I’ve gotta get up early.	No.	94
World knowledge	Did you leave fingerprints?	I wore gloves.	No.	23
Idiom	Would he fire me?	He’s all bark and no bite.	No.	42
Rhetorical question	Can you drive that far?	Can fish swim?	Yes.	11
Other	-	-	-	383
K.9 Detailed results type label analysis
In the main paper we do an analysis of two types of examples that occur frequently in the dataset, namely generalised and particularised implicatures. Here, we detail the full taxonomy of types of examples occurring in the dataset and report detailed results for each type per model (see 21 until Table 34 below). In Table 20 the full taxonomy of the examples is shown, representing types of examples that occur frequently in the dataset. We manually labeled 217 examples of the 600 examples in the test set according to this taxonomy. The remaining 383 examples do not fall as clearly within a category and are grouped together as type other. Generalised implicatures require little or no context to be understood. They are the simplest type of example in the test set, and generally imply the same thing ("some" almost always implies "not all"). Particularised implicatures, by contrast, do require context to be resolved. For example, from Table 20, we need the context that it is undesirable to stay up late drinking when one has to get up early (see in Appendix D more on generalised vs. particularised). The type world knowledge requires knowledge of the physical world to be resolved. From the example in Table 20; we need to know that you cannot leave fingerprints when wearing gloves to resolve this implicature. Idiom types contain an idiom or a metaphor that one needs to know or understand to resolve the implicature, and finally Rhetorical question types contain a question like "Is the Pope Catholic?", often requiring factual knowledge to be resolved.
The following tables contain the detailed results broken down per example type: Table 21 - Table 34. The most interesting pattern in this data is that for almost all models, even the best model (GPT-4 30-shot in Table 32), there is a significant gap between human-level performance on the particularised examples. This gap is larger than the gap for the other labels usually. Few-shot prompting can often mitigate this (e.g. for GPT-3-175b, Cohore-52b, and text-davinci-002), but not always (e.g. for GPT-4 the gap remains large for  $k = 30$ ). However, for GPT-4, chain-of-thought can mitigate the gap as seen in Table 34. Where GPT-4 30-shot obtains  $71.97\%$  accuracy on the particularised examples (and humans  $83.18\%$ ), GPT-4 with 5-shot CoT achieves  $81.63\%$ , which is close to human-level. We find that the particularised examples mostly benefit from CoT prompting. Namely, for the generalised type of examples, GPT-4 30-shot already achieves  $86.23\%$  accuracy and CoT improves this to  $88.66\%$ , which is a much smaller improvement than for the particularised examples.
K.10 Detailed results per model
This section contains the results used for the zero-shot and few-shot evaluation in the main text in Section 4, broken down per prompt template. See Table 35 until Table 84.
Table 21: Accuracy per label for 0-shot evaluation.
Model	Mean	World knowledge	Idiom	Rhetorical question
OPT-125m	50.92	50.00 +/- 2.17	51.52 +/- 9.96	57.58 +/- 10.05
OPT-350m	57.14	57.97 +/- 10.25	64.77 +/- 3.65	65.15 +/- 3.39
OPT-1.3b	60.36	60.14 +/- 5.84	68.94 +/- 5.52	59.09 +/- 4.55
OPT-2.7b	59.56	60.87 +/- 6.15	67.05 +/- 2.18	69.70 +/- 6.78
OPT-6.7b	60.33	59.42 +/- 6.95	59.47 +/- 2.04	53.03 +/- 19.93
OPT-13b	61.03	63.77 +/- 14.78	73.86 +/- 7.51	66.67 +/- 16.32
OPT-30b	61.47	65.94 +/- 10.48	62.88 +/- 8.05	74.24 +/- 6.25
OPT-66b	61.33	69.57 +/- 13.75	60.23 +/- 4.30	59.09 +/- 18.74
OPT-175b	55.33	55.07 +/- 5.42	54.55 +/- 9.19	63.64 +/- 21.64
BLOOM-560m	51.58	54.35 +/- 5.47	54.92 +/- 16.72	50.00 +/- 13.64
BLOOM-1b1	51.17	50.00 +/- 2.17	50.38 +/- 11.77	53.03 +/- 12.22
BLOOM-1b7	53.61	52.17 +/- 6.15	53.79 +/- 8.77	68.18 +/- 6.94
BLOOM-3b	56.89	54.35 +/- 6.02	59.85 +/- 4.48	63.64 +/- 5.25
BLOOM-7b1	58.67	63.77 +/- 14.57	68.94 +/- 5.82	68.18 +/- 4.55
BLOOM-176b	54.22	55.07 +/- 7.39	50.38 +/- 11.01	62.12 +/- 9.70
EleutherAI-125m	51.89	56.52 +/- 9.72	52.65 +/- 8.84	63.64 +/- 5.25
EleutherAI-1.3b	53.14	51.45 +/- 3.90	53.03 +/- 11.19	62.12 +/- 3.39
EleutherAI-2.7b	59.17	60.14 +/- 13.38	65.91 +/- 3.94	68.18 +/- 4.55
EleutherAI-6b	56.36	57.25 +/- 7.28	56.06 +/- 8.87	50.00 +/- 17.99
EleutherAI-20b	57.53	51.45 +/- 3.90	67.80 +/- 5.93	72.73 +/- 5.25
Cohere-409m	51.61	52.17 +/- 4.35	53.41 +/- 11.94	54.55 +/- 12.86
Cohere-6b	57.28	55.80 +/- 5.28	60.23 +/- 5.98	72.73 +/- 9.09
Cohere-13b	57.19	59.42 +/- 4.81	54.55 +/- 10.82	48.48 +/- 10.05
Cohere-52b	58.50	60.14 +/- 13.61	65.15 +/- 3.86	74.24 +/- 11.03
GPT-3-350m	51.47	51.45 +/- 3.90	53.41 +/- 13.56	50.00 +/- 13.64
GPT-3-1.3b	57.72	61.59 +/- 11.06	64.39 +/- 4.08	65.15 +/- 3.39
GPT-3-6.7b	54.83	54.35 +/- 6.99	53.79 +/- 7.61	62.12 +/- 3.39
GPT-3-175b	57.22	55.80 +/- 7.28	68.94 +/- 5.19	77.27 +/- 8.70
T0-3b	48.25	54.35 +/- 4.86	42.42 +/- 4.29	36.36 +/- 0.00
T0-11b	55.61	60.14 +/- 6.84	54.92 +/- 14.93	36.36 +/- 0.00
BlenderBot-90m	46.64	52.17 +/- 0.00	38.64 +/- 0.00	36.36 +/- 0.00
BlenderBot-3b	53.44	47.83 +/- 0.00	61.36 +/- 1.31	63.64 +/- 0.00
BlenderBot-9b	53.36	52.17 +/- 6.64	60.98 +/- 4.81	63.64 +/- 0.00
Flan-T5-780m	63.31	72.46 +/- 4.10	71.97 +/- 5.82	54.55 +/- 13.89
Flan-T5-3b	52.50	50.72 +/- 6.95	51.89 +/- 4.23	42.42 +/- 8.57
Flan-T5-11b	60.78	65.94 +/- 5.84	72.35 +/- 7.59	65.15 +/- 6.25
Cohere-command-6b	66.31	72.46 +/- 7.80	78.41 +/- 4.30	37.88 +/- 3.39
Cohere-command-52b	60.22	66.67 +/- 10.85	63.64 +/- 10.33	77.27 +/- 6.94
text ADA-001-unknown	56.50	63.77 +/- 4.10	58.71 +/- 16.04	51.52 +/- 10.05
text-babbage-001-unknown	64.47	67.39 +/- 6.02	76.52 +/- 1.69	60.61 +/- 10.05
text-curie-001-unknown	68.94	76.81 +/- 3.24	76.89 +/- 2.76	54.55 +/- 12.86
text-davinci-001-unknown	72.31	84.78 +/- 7.43	78.79 +/- 4.08	59.09 +/- 13.64
text-davinci-002-unknown	70.58	82.61 +/- 9.05	75.38 +/- 3.05	57.58 +/- 16.32
text-davinci-003-unknown	71.25	86.96 +/- 13.28	72.35 +/- 7.35	48.48 +/- 8.57
ChatGPT-unknown	72.08	82.61 +/- 12.04	83.33 +/- 5.97	56.06 +/- 16.11
GPT-4-unknown	81.78	92.03 +/- 2.99	90.91 +/- 3.21	84.85 +/- 8.57
Humans	86.23	93.04	92.73	92.73