SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In later work, Potts (2006) describes how pragmatic pressures interacting with context cause conversational implicature to arise. He shows how sensitive conversational implicatures are to small changes in the context. Novel information about a speaker's belief state might completely change what is implied. There are many more models of implicature that aim to explain how humans understand language in context. Most notably, Frank and Goodman (2012) formalise the view that speakers produce utterances that are helpful and not longer than necessary with a Bayesian model called the rational speech act (RSA). Many variants on the RSA framework have since been proposed. For example, Goodman and Frank (2016) extend it to handle nonliteral uses of language, like irony, and metaphor. In the context of computational models, prior work uses insights from pragmatics to show that the use of certain words can make a language model produce biased completions (Patel and Pavlick (2021), e.g. saying someone "claimed" something rather than "said" something), and inform bias and sentiment classifiers (Greene and Resnik, 2009; Recasens et al., 2013).
In this work, we focus on conversational implicatures and not on conventional implicatures. All conversational implicatures are negotiable by context, but the way they depend on context can be different. Grice (1975) identifies generalised conversational implicatures and particularised conversational implicatures. The former require little or no context to be resolved. For example, "some athletes smoke" can imply "not all athletes smoke", but might also imply "I do not know whether all athletes smoke" when it is a response to the question "do you know whether all athletes
smoke?" (Davis, 2019). The latter only arise in certain contexts. For example, the response "I have an early morning" to the question "do you want to stay for a drink?".
F Detailed prompt templates
Table 5 contains the full prompt templates we used for the main evaluation and Table 7 contains the extra prompt templates.
Table 5: Ranking prompt templates. The six templates we wrap the test examples in to present to the models. Template 1, 3, and 4 are of the type structured, and 2, 5, and 6 of the type natural. Within the type of prompt template they only differ slightly in wording.
#	Prompt template
1	Does the following response to the question imply yes or no? question: <utterance> response: <response> implicature: <implicature>
2	Finish the following text: Esther asked "<utterance>" and Juan responded "<response>", which means <implicature>
3	Is the implied meaning of the following response yes or no: question: <utterance> response: <response> meaning: <implicature>
4	What is the intent of the following response, yes or no? question: <utterance> response: <response> intent: <implicature>
5	Finish the following text: Karen asked "<utterance>" and William responded "<response>", which means <implicature>
6	Finish the following text: Bob asked "<utterance>" and Alice responded "<response>", which means <implicature>
Table 6: Completion prompt templates. The six adjusted templates we wrap the test examples in to present to the models when we use completion instead of likelihood ranking. Template 1, 3, and 4 are of the type structured, and 2, 5, and 6 of the type natural. Within the type of prompt template they only differ slightly in wording.
#	Prompt template
1	Does the following response to the question imply yes or no? Only output ‘yes’ or ‘no’.
Even if you’re uncertain, choose either ‘yes’ or ‘no’.
question: <utterance>
response: <response>
implicature: <implicature>
2	Finish the following text. Only output ‘yes’ or ‘no’. Even if you’re uncertain, choose either ‘yes’ or ‘no’.
Esther asked "<utterance>" and Juan responded "<response>", which means <implicature>
3	Is the implied meaning of the following response yes or no. Only output ‘yes’ or ‘no’.
Even if you’re uncertain, choose either ‘yes’ or ‘no’.
question: <utterance>
response: <response>
meaning: <implicature>
4	What is the intent of the following response, yes or no? Only output ‘yes’ or ‘no’.
Even if you’re uncertain, choose either ‘yes’ or ‘no’.
question: <utterance>
response: <response>
intent: <implicature>
5	Finish the following text. Only output ‘yes’ or ‘no’. Even if you’re uncertain, choose either ‘yes’ or ‘no’.
Karen asked "<utterance>" and William responded "<response>", which means <implicature>
6	Finish the following text. Only output ‘yes’ or ‘no’. Even if you’re uncertain, choose either ‘yes’ or ‘no’.
Bob asked "<utterance>" and Alice responded "<response>", which means <implicature>
Table 7: The three additional templates we wrap the test examples in to present to the models, adapted from Glaese et al. (2022).
#	Prompt template
7	The following text shows an interaction between two humans called Esther and Juan. In the interaction, Esther will ask Juan a question, and Juan will give an answer that contains an implicature. An implicature is an utterance that means something other than the literal meaning of the words. The implicature of Juan's response is yes or no. You, the AI assistant, are asked to finish the text with yes or no. The task begins: Esther asked "<utterance>" and Juan responded "<response>", which means <implicature>
8	The following text shows an interaction between two humans called Esther and Juan. In the interaction, Esther will ask Juan a question, and Juan will give an answer that has a meaning besides the literal meaning of the words. That meaning is either yes or no. You, the AI assistant, are asked to finish the text with the correct meaning, either yes or no. The task begins: Esther asked "<utterance>" and Juan responded "<response>", which means <implicature>
9	The following text shows an interaction between two humans called Esther and Juan. In the interaction, Esther will ask Juan a question, and Juan will give an answer that has a meaning besides the literal meaning of the words. That meaning is either yes or no. You, a highly intelligent and knowledgeable AI assistant, are asked to finish the text with the correct meaning, either yes or no. The task begins: Esther asked "<utterance>" and Juan responded "<response>", which means <implicature>
Table 8: Chain-of-thought (CoT) prompt templates. One of the six chain-of-thought prompt templates we use for the CoT experiment. Note that this is a 5-shot prompt. Each prompt variation contains five CoT examples. The other five variations are separately added to the supplementary materials
#	Prompt template
1	Bob asks Alice a question, and Alice responds with an implicature. This means that Alice's response does not literally contain the answer to Bob's question, but implies an answer. Assuming that Alice is a cooperative conversational partner, what is the implicated answer to the question? For example:
Bob: You invented fire?
Alice: I told you that.
Implicature: Alice says 'I told you that'. Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. Alice must mean that she told Bob that she invented fire. Alice's response to Bob's question 'You invented fire?' is yes.
Answer: yes
Bob: That cake looks delicious. Aren't you going to have some with me?
Alice: But that was a huge meal we just had.
Implicature: Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. Alice must mean that the meal they just had was so huge she is not hungry anymore, and this must be relevant to Bob's question: 'Aren't you going to have some with me?'
Alice's response to the question must therefore be no.
Answer: no
Bob: Could you perform well?
Alice: Being bilingual would help put me ahead of the pack.
Implicature: Alice says being bilingual would help put her ahead of the pack. Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. Alice must be implying that she could perform well because she is bilingual. This means the response to Bob's question is yes.
Answer: yes
Bob: Have you any news for me?
Alice: I've made progress
Implicature: Alice says she has made progress. Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. If Alice would not have any news for Bob, Alice would not have said she would have made progress because that would be misleading. The answer to Bob's question 'Have you any news for me?' must therefore be yes.
Answer: yes
Bob: You looked out for him?
Alice: He looked out for me. He taught me.
Implicature: Bob asks Alice 'You looked out for him?' and Alice's response says that the person that is being referred to by 'him' here looked out for Alice. If Alice meant yes to Bob's question, Alice would have said something like 'he also looked out for me'. Stating the response like this implies that the answer to Bob's question is no.
Answer: no
Only output a 'yes' or 'no' as a final answer. Write your reasoning after 'Implicature:' and then output either 'Answer: yes' or 'Answer: no'.
Bob: <utterance>
Alice: <response>
Implicature:
G Model categorization
Table 9 contains details on the model classes that are a part of each group of models we evaluate, along with their model sizes.
Table 9: Model categorization for each of the models. DL stands for dialogue, FT for fine-tuning, BL for benchmark-level, EL for example-level, and IT for instruction-tuning.
Group	Model class	Model IDs	Model size	Instruct
Base	BERT	base uncased	110M	No
RoBERTa	base, large	125M, 355M	No
GPT-2	GPT-2 medium, large, xl	354M, 774M, 1.6B	No
EleutherAI	GPT-J, GPT-NeoX	125M, 1.3B, 2.7B, 6B, 20B	No
BLOOM	-	560M, 1B1, 3B, 7B1, 176B	No
OPT	-	125M, 350M, 1.3B, 13B, 30B, 66B, 175B	No
Cohere	small, medium, large, XL	409.3M, 6.067B, 13.12B, 52.4B	No
GPT-3	ada, babbage, curie, davinci	Est. 350M, 1.3B, 6.7B, 175B	No
DL FT	BlenderBot	-	90M, 2.7B, 9.4B	No
BL IT	T0	-	3B, 11B	Yes
Flan-T5	-	780M, 3B, 11B	Yes
EL IT	Cohere-command	medium, xlarge	6.067B, 52.4B	Yes
text-davinci-001	ada, babbage, curie, davinci-1	Unknown, left-to-right increasing in size	Yes
text-davinci-002	-	Unknown	Yes
text-davinci-003	-	Unknown	Yes
ChatGPT	gpt-3.5.turbo	Unknown	Yes
GPT-4	gpt-4	Unknown	Yes
H Human evaluation
The participants for the human evaluation in this paper were recruited using Prolific (www.prolific.co). The setup of the experiment is as follows. We divide the test set of 600 examples into four non-overlapping subsets of 150 examples. Each set of 150 examples was given to five unique annotators. This means each example in the test set is labeled five times by different people, and we have in total twenty annotators for the whole test set (five different ones for each of the four subsets). The only constraint for the annotators is that they are native English speakers. In Figure 5 the screen shown to potential participants on Prolific is shown. Participants are paid 15 pounds an hour, which was the living wage at the time of the experiment and more than the 12 dollars an hour Prolific recommends. The total amount spent on the human evaluation is 236 pounds. This amount came to be from four subsets, each costing about 30 minutes to label per annotators, and having 5 annotators per subset:  $15*4*0.5*5 = 150$ . The extra costs were for the annotator that didn't pass the attention check which we paid nonetheless, and for prolific as a platform.
A study about the implied meanings of responses
The aim of this study
We want to compare the performance of humans and language models (like GPT-3) on an important aspect of language.
What you'll do
You'll see a set of 150 sentences in a google form, and are asked to finish the sentence with yes/no, whichever makes more sense to you given the sentence. Please finish each of the 150 questions. If you don't know the answer, just take the best guess. To avoid your submission being rejected, make sure you enter your correct unique Prolific ID in the first question, fill out the correct email address to receive your completion code, finish all the 150 questions, and pass the attention test. We expect this task to take around 30 minutes. When you've submitted the form, we will check that everything has been filled out and send you the completion code that you can use to tell Prolific you've finished the task.
No sensitive information is required to give in this test and the answers are completely anonymised. We don't do anything with the information about you except match the prolific ID to the responses.
We will pay out the reward as quickly as possible after your submission, hopefully within a day, at least within a week.
Devices you can use to take this study:
Figure 5: A screenshot of how the experiment is presented to potential annotators on Prolific (www.prolific.co).
The 150 test examples are wrapped in prompt template 2 (see Table 5) and presented in a Google form. We opted to wrap all examples in prompt template 2 to make the full human study directly comparable to the model's results on template 2. If we had done a mix of all templates we either had to spent six times as much on the human evaluations (which was not within our budget) or subsample evaluations, making it less comparable to part of the model study. Although models have been shown to be very sensitive to prompt wording, humans are less likely to perform differently for different prompt templates. All templates are coherent natural language that any native English speaker will understand. That said, this is speculative, and to confirm this hypothesis future work should investigate the effect of different wordings on implicature resolution by humans. The participants are asked to choose the correct continuation, yes or no (see Figure 6a). As recommended by Prolific, we subject the participants to an attention test (see Figure 6b). At three random places in the form, we add a question that does not contain an implicature and obviously maps to "yes". In this way, if the participants fail at least two of these questions, we can conclude they were not paying attention and remove their answers from the result. In practice, this happened once and we decided to pay the participant regardless, but discard their results, which were close to random.
Table 10 shows the performance of each annotator on the subset they annotated. The average human performance across subsets and annotators is  $86.2\% \pm 2.3$ , the best performance is  $89.8\% \pm 2.2$ , and the worst performance is  $83.5\% \pm 1.5$ . The column "IAA" shows the average Cohen's Kappa coefficient which is the pairwise inter-annotator agreement for each annotator per subset. All agreements are substantial according to the interpretation guidelines for Cohen's Kappa (between 0.61–0.80).
Figure 6: Screenshots of the Google form participants fill out as part of the implicature study.
Table 10: The performance of the human annotators on the subsets of the test set. Subset 1 through 4 are non-overlapping and cover the whole test set. Annotator X for subset Y might be a different human than annotator X for subset Z. IAA is the average pairwise inter-annotator agreement (Cohen's kappa coefficient) between annotators per subset.
Annotator	1	2	3	4	5	Mean	Best	Worst	IAA
Subset 1	86.0%	92.0%	90.7%	90.6%	86.0%	89.1%	92.0%	86.0%	0.73
Subset 2	84.7%	83.3%	87.3%	86.0%	86.0%	85.5%	87.3%	83.3%	0.64
Subset 3	84.0%	85.3%	88.0%	86.0%	82.7%	85.2%	88.0%	82.7%	0.78
Subset 4	85.3%	82.7%	84.0%	82.0%	92.0%	85.2%	92.0%	82.0%	0.71
Total	-	-	-	-	-	86.2%	89.8%	83.5%	0.72
Std	-	-	-	-	-	2.3	2.2	1.5	0.1
Human source of disagreement with ground-truth. We do an analysis of the source of disagreement with the ground-truth. We explicitly do not call this error, as in some cases examples might allow multiple interpretations, and both could be right. In other cases, the ground-truth might be wrong.
Annotation errors and multiple interpretations: We analyse the examples for which most humans choose a different answer than the ground-truth. For 30 out of 600 examples in the test set, only one or zero people choose the same answer as the ground-truth. Of these examples, most are annotated wrongly (18 of 30). For example: 'Are you busy?', 'I'm drowning in work', implicature: 'no'. Some are examples that can have multiple different interpretations (12 of 18), and the ground-truth answer likely just chooses one that is unnatural to humans. For example: 'You don't remember them?', 'Leave me alone!', implicature: 'yes'. 6 of the 30 examples are particularised, and 1 is generalised.
Examples for which all humans agree with the ground-truth: There are 409 out of 600 examples that all humans get correct. This set of examples contains most of the generalised implicatures (39 out of 47). These contain 58 out of 94 particularised examples.
Examples most humans agree with the ground-truth: When we look at examples that 3 or more humans got correct, that comprises most of the test set (530 of 600), and all of the generalised examples (47 of 47). This subset has 78 of 94 particularised examples, so for 16 particularised examples 3 or more humans disagree with the ground-truth.
I Comparison with BIG-bench implicatures task
One of the BIG-bench tasks is related to the task in this work. It uses the same underlying dataset we use (George and Mamidi, 2020). With the below we aim to discuss our contribution in light of the BIG-bench result. To summarise; the methodology used by the BIG-bench task contributors has limitations, which call into question the validity of their claims. Further, some of the BIG-bench results are irreproducible due to missing details in the tech report and the use of proprietary models. Considering this, our work is an important contribution validating the BIG-bench results in a reproducible and methodologically sound way, and above that providing insight into what aspects of LLM training are crucial for the ability to do pragmatic inferences.
Limitations of the methodological approach of the task contributors in BIG-bench implicatures. Our benchmark has  $30\%$  more data, which the BIG-bench task contributors discard. In this section we motivate the crucial importance of that data for evaluating implicature understanding (Section I.1), and why BIG-bench in turn might be overestimating the performance of LLMs on implicature resolution (Section I.2). Moreover, the human performance on the BIG-bench task indicates low quality human annotation, which we will also elaborate upon below, noting that this is impossible to verify because the BIG-bench report does not detail how the evaluation was done for this task (Section I.3).
I.1 Discarding ambiguous examples
The BIG-bench task preprocesses the 1001 examples that George and Mamidi (2020) curate by keeping only yes/no questions, discarding any examples that are ambiguous according to the task contributors, and discarding remaining examples to create a 50/50 distribution of yes/no answers. This leaves them with 492 examples. Of these examples, 81 appear in our development set and the remaining 411 appear in our test set. Our test set has 600 examples, so BIG-bench effectively discarded 189 ambiguous examples compared to our test set; a bit more than  $30\%$  of the benchmark. To illustrate the importance of not discarding this data, we cherry picked a few examples that the BIG-bench authors discarded from the data.
- Utterance: "Can you lend me hundred dollars?", Response: "Is this supposed to be some kind of a joke?", Implicature: "No"
- Utterance: "Do you know, how long is Uncle Arthur staying with us?", Response: "Ask your father.", Implicature: "No"
Indeed, these examples are ambiguous. Asking whether the request for a hundred dollars is a joke does not literally mean you're saying no to the request. The response "ask your father" does not mean the speaker does not actually know, maybe they just do not want to respond. The humans in our study all infer the intended ground truth implicature. This shows a general property of implicatures; they are ambiguous, but often humans do infer the intended meaning. Ambiguity is not a discrete property. Some examples may be so vague that no one gets it. The following are examples the BIG-bench task discards that the humans in our study did struggle with:
- Utterance: "Got any more of those?", Response: "Nothing I'm at liberty to reveal here.", Implicature: "Yes"
- Utterance: "Have you finished sight-seeing?", Response: "Sorry. I should've come to see you first.", Implicature: "Yes"
In the first of these the implicature is "yes" because the person responding is implying that they do have more, they just cannot reveal them. Otherwise they would most likely simply say no. In the second example it feels more natural that someone says this when they are finished sight-seeing, otherwise they would've probably said something to the effect of "I'm still out, but I'm sorry.." In any case, humans in our study did not understand these responses like that. This illustrates another aspect of implicature; sometimes communication will go wrong over it. Removing implicatures that are ambiguous though, defeats the purpose of the task, as they are all ambiguous to a certain degree. The purpose of this study is to compare how humans resolve this type of non-literal language
<sup>9</sup>https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/ implications
compared to how models do it. The human baseline of  $86\%$  accuracy that humans achieve on our test set deals more naturally with examples that are too ambiguous for models to understand than discarding examples based on the subjective opinion of a few people.
I.2 Overestimation of performance on implicature understanding
On the overlapping part of our test set and theirs the humans in our study achieve  $92.8\%$  accuracy. The best model on the BIG-bench task is PaLM, achieving a zero-shot performance of  $64.4\%$ . Note that this performance is on their full test set (not the overlapping part) and hence not directly comparable. Nonetheless, the missing examples are randomly sampled for our development set, and we can be pretty confident this number indicates a large gap with human performance. Two-shot PaLM comes very close to human performance with  $91.7\%$  accuracy, but of course this does not take into account the 189 more challenging examples that are part of our benchmark. Humans achieve  $71.9\%$  performance on this subset of ambiguous data, indicating that these data are more difficult than average, but nonetheless performance is higher than random. Without access to the models used to evaluate the BIG-bench task we cannot say anything for certain, but we expect the performance of PaLM to be overestimated as it does not account for a large part of the type of implicatures found in natural discourse; ambiguous ones.
I.3 Other limitations
Poor quality human evaluation. The average human evaluator on BIG-bench implicatures achieves around  $82\%$  performance (where ours achieves on average  $86\%$  on a more challenging dataset), and their human best rater achieves  $100\%$  (where our human best is  $92\%$ ). This difference between human average and best hints at poor quality average rating. This is impossible to verify because there is no information in the BIG-bench tech report on how the human evaluation was done exactly, or even which examples where evaluated.
No fine-tuned models. BIG-bench uses only base LLMs and no SotA fine-tuning methods. A question that remains is therefore, what aspects of LLMs contribute to their performance on implicatures? In our work we find that implicature performance emerges at a much smaller scale in models instruction fine-tuned at the example level, and that scale and prompting techniques are important.
J Chain-of-thought completions by GPT-4
GPT-4 reaches human-level performance with CoT prompting. In this section we look at a few completions to help illuminate how CoT-prompting improves implicature resolution.
Specifically, we look at some of the examples for which GPT-4 5-shot gets  $0\%$  accuracy (i.e. wrong for all templates), and GPT-4 5-shot CoT gets  $100\%$  accuracy (i.e. right for all templates). This is a set of 10 examples. We only look at the first template, but all templates get a similar performance with CoT.
Utterance: 'Do you sell calculators?'