Title: The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
Venue: NeurIPS 2023
Accepted: yes
Keywords: large language models

pragmatics

natural language processing

communication

conversation

implicature

language model fine-tuning
Abstract: Despite widespread use of LLMs as conversational agents, evaluations of performance fail to capture a crucial aspect of communication: interpreting language in context---incorporating its pragmatics. Humans interpret language using beliefs and prior knowledge about the world. For example, we intuitively understand the response "I wore gloves" to the question "Did you leave fingerprints?" as meaning "No". To investigate whether LLMs have the ability to make this type of inference, known as an implicature, we design a simple task and evaluate four categories of widely used state-of-the-art models. We find that, despite only evaluating on utterances that require a binary inference (yes or no), models in three of these categories perform close to random. However, LLMs instruction-tuned at the example-level perform significantly better. These results suggest that certain fine-tuning strategies are far better at inducing pragmatic understanding in models. We present our findings as the starting point for further research into evaluating how LLMs interpret language in context and to drive the development of more pragmatic and useful models of human discourse.

The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs

Abstract

Despite widespread use of LLMs as conversational agents, evaluations of performance fail to capture a crucial aspect of communication: interpreting language in context -incorporating its pragmatics. Humans interpret language using beliefs and prior knowledge about the world. For example, we intuitively understand the response 'I wore gloves' to the question 'Did you leave fingerprints?' as meaning 'No'. To investigate whether LLMs have the ability to make this type of inference, known as an implicature , we design a simple task and evaluate four categories of widely used state-of-the-art models. We find that, despite only evaluating on utterances that require a binary inference (yes or no), models in three of these categories perform close to random. However, LLMs instruction-tuned at the example-level perform significantly better. These results suggest that certain fine-tuning strategies are far better at inducing pragmatic understanding in models. We present our findings as the starting point for further research into evaluating how LLMs interpret language in context and to drive the development of more pragmatic and useful models of human discourse.

1 Introduction

User: 'Have you seen my phone?'

GPT-3: 'Yes, I have seen your phone.'

GPT-3's response 2 is a perfectly fine answer to the question, but a human might answer differently. They might respond 'it's in your bag," bypassing the obvious follow-up question ('where is it?'). Giving such a helpful and efficient answer is an example of pragmatic language use that goes beyond the mere production of semantically plausible and consistent utterances. Meaning is not only determined by a combination of words, but also context, beliefs, and social institutions (Wittgenstein, 1953; Grice, 1975; Huang, 2017). Consider another exchange where Esther asks her friend Juan 'Can you come to my party on Friday?' and Juan responds 'I have to work'. We resolve Juan's response as him declining the invitation by using the contextual commonsense knowledge that having to work on a Friday night precludes attendance. Both these exchanges contain an implicature -utterances that convey something other than their literal meaning. 3 Implicatures illustrate how context contributes to meaning; distinguishing writing and speaking from communicating (Green, 1996). We cannot fully

----Example----

Utterance

---.Prompt template ----.

Want to stay for a

nightcap?

Esther asked "Want to stay for a nightcap?"

X

and Juan responded "I've gotta get up early"

Response

which means no.

.--Evaluation--.

pe(x) > pe(x)

I've gotta get up early.

Esther asked "Want to stay for a nightcap?"

X

and Juan responded "I've gotta get up early"

0LLM

which means yes.

Implicature

No.

understand utterances without understanding their implications. Indeed, the term 'communication' presupposes the speaker's implications are understood by the addressee. Being able to resolve completely novel implicatures and, more broadly, engage in pragmatic understanding constitutes an essential and ubiquitous aspect of our every day use of language.

Large language models (LLMs) have demonstrated remarkable ability on a variety of downstream tasks such as planning (Huang et al., 2022), commonsense reasoning (Kojima et al., 2022), information retrieval (Lewis et al., 2020; Kim et al., 2022) and code completion (Austin et al., 2021; Biderman and Raff, 2022), to name a few. When fine-tuned with human feedback, LLMs obtain higher ratings on desiderata like helpfulness (Ouyang et al., 2022; Bai et al., 2022), and are proposed as conversational agents (Thoppilan et al., 2022). Despite the widespread use of LLMs as conversational agents, there has been limited evaluation of their ability to navigate contextual commonsense knowledge.

This raises an important question: to what extent can large language models resolve conversational implicature? To answer this question we use a public dataset of conversational implicatures and propose an evaluation protocol on top of it (Figure 1). We evaluate a range of state-of-the-art models that can be categorised into four groups; large-scale pre-trained models, like OPT (Zhang et al., 2022), LLMs fine-tuned on conversational data, like BlenderBot (Ng et al., 2019), LLMs fine-tuned on common NLP benchmarks with natural instructions for each benchmark, like Flan-T5 (Chung et al., 2022), and LLMs fine-tuned on tasks with natural instructions for each example, e.g. versions of OpenAI's InstructGPT-3 series 4 . Our results show that implicature resolution is a challenging task for LLMs. All pre-trained models obtain close to random zero-shot accuracy (around 60%), whereas humans obtain 86%. However, our results suggest that instruction-tuning at the example level is important for pragmatic understanding. Models fine-tuned with this method perform much better than others, and analysis of different model sizes shows that they have the best scaling properties. We further push performance for these models with chain-of-thought prompting, and find that one model in the group (GPT-4) reaches human-level performance. In summary, we conclude that pragmatic understanding has not yet arisen from large-scale pre-training on its own , but scaling analysis shows that it might for much larger scale. Fine-tuning on conversational data or benchmark-level instructions does not produce models with pragmatic understanding. However, fine-tuning on instructions at the example-level is a fruitful path towards more useful models of human discourse.

The main contributions of this work are: i) we motivate implicature understanding as a crucial aspect of communication that is currently mostly missing from evaluations of LLMs, ii) we design an implicature resolution task and propose a comprehensive evaluation protocol on which we evaluate both humans and LLMs to find that it poses a significant challenge for SotA LLMs, and iii) we provide a thorough analysis of the results and identify one fine-tuning strategy (instruction-tuning at the example-level) as a promising method that produces models with more pragmatic understanding.

2 Related Work

LLMs have demonstrated remarkable performance on tasks for which they were not explicitly trained (Brown et al., 2020). Building on the hypothesis that these abilities arise due to implicit multitask learning (Radford et al., 2019), the recent works of Sanh et al. (2022) and Wei et al. (2022) explicitly train LLMs in a supervised multitask fashion, leading to models that are better zero-shot learners with fewer parameters. Besides rapidly saturating language understanding benchmarks (Kiela et al., 2021), these advancements make LLMs beneficial foundations for agents performing a plethora of tasks (Adolphs et al., 2022; Reed et al., 2022). The trend towards using these models as agents brings along with it increased urgency for alignment with human values (Kenton et al., 2021). However, larger models trained with next-word prediction are generally more toxic and unhelpful (Gehman et al., 2020; Bender et al., 2021; Lin et al., 2022). Recent work mitigates this with methods like prompting and finetuning on human-annotated outputs (Askell et al., 2021; Ouyang et al., 2022; Thoppilan et al., 2022). The produced models are more aligned on desiderata such as informativeness when evaluated by dedicated benchmarks and humans. We argue, however, that there is still something missing in these benchmarks. What is helpful and informative, as Kasirzadeh and Gabriel (2022) also point out, depends on the context in which a conversation is held. Consequently, any application that requires communicating with humans will rely on pragmatic communication skills-something that is not explicitly captured by the benchmarks used to evaluate the alignment of LLMs.

There is a large body of work that investigates the interplay between pragmatics and computational modeling (Cianflone et al., 2018; Schuster et al., 2020; Louis et al., 2020; Kim et al., 2021; Li et al., 2021; Jeretic et al., 2020; Parrish et al., 2021; Hosseini et al., 2023). Cianflone et al. (2018) introduce the task of predicting adverbial presupposition triggers, which are words like 'again' that trigger the unspoken presupposition that an event has happened before. Schuster et al. (2020) study the ability of computational models to do scalar inferences, finding that models use linguistic features to make pragmatic inferences. Kim et al. (2021) find that a substantial part of question-answering datasets contain questions that are unanswerable due to false presuppositions (i.e. 'which linguist invented the lightbulb'). Hosseini et al. (2023) present a dataset for selecting entities with indirect answers, and find that language models adapted for this task get reasonable accuracy, but that there is room for improvement. The difference with this body of work and ours is that we look at the emergence of pragmatic understanding from large-scale language modeling. Jeretic et al. (2020); Parrish et al. (2021) are early works investigating the emergence of pragmatic understanding in pretrained language models, but they only look at scalar implicatures and presuppositions. Zheng et al. (2021) are the first to evaluate pretrained language models on conversational implicatures. This is important pioneering work highlighting the difficulty of implicature for language models, but their evaluations require task-specific training and the models they evaluate are relatively small. In contrast, our evaluation protocol is applicable out-of-the-box and is much more comprehensive, evaluating models up to 176 billion parameters and using in-context prompting. Additionally, Zheng et al. (2021) benchmark synthetic data whereas this work evaluates performance on naturally occurring implicatures (George and Mamidi, 2020). We believe this to be a better representation of the true distribution of implicatures in natural dialogue.

The standard set of benchmarks LLMs are evaluated on covers many tasks, but even though implicature is one of the most important aspects of language pragmatics (Levinson, 1983), it is only evaluated as part of BIG-bench (Srivastava et al., 2022). Unfortunately, the methodology used by the BIG-bench implicature task contributors has limitations, which call into question the validity of their claims. Firstly, the task contributors discard a subset of the data that is ambiguous according to them. In our view this defeats the point of the benchmark. Implicatures are a type of non-literal, ambiguous language the intended meaning of which humans often easily interpret; comparing the way humans and models do this is precisely what we are interested in. In turn, we expect performance on the BIG-bench task to overestimate the ability of LLMs to resolve naturally occurring implicatures. We keep this challenging subset of the data and instead use human evaluation to deal with examples that are too ambiguous to understand. Secondly, the difference in performance between their average and best rater is 18%, whereas for our evaluations this difference is 6%. This indicates their human evaluation is of low quality, but it is impossible to verify because there are no details available on how the annotation is done. Finally, BIG-bench uses only base LLMs and no SotA fine-tuning methods. In summary, we use a more challenging dataset, and in turn at least six times more evaluations per model, we provide higher-quality human annotations, and evaluate four different categories of LLMs to investigate which aspects of LLMs contribute to their performance on implicature understanding.

3 Evaluation Protocol

Here we outline the evaluation protocol we use to answer the research question 'To what extent can LLMs resolve conversational implicature?'. We focus on binary implicatures that imply 'yes' or 'no' (see Figure 1). We say a model resolves an implicature correctly if it assigns higher likelihood to a coherent utterance than a similar but incoherent one, detailed below.

Zero-shot evaluation . Consider the example from the introduction packed into a single utterance:

Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means no.

We can transform this example to be pragmatically incoherent (in the sense that it will become pragmatically inconsistent with expected use) by replacing the word 'no' with 'yes':

Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means yes.

To resolve the implicature, the model should assign higher likelihood to the first of the two sentences above, namely the most coherent one. Importantly, both sentences have exactly the same words except for the binary implicature 'yes' or 'no', making the assigned likelihood scores directly comparable. Formally, let the coherent prompt be x and the augmented, incoherent prompt be ˆ x . A model outputs a likelihood p parameterized by weights θ . We say a model correctly resolves an example x when it assigns p θ ( x ) > p θ ( ˆ x ) . This is equivalent to evaluating whether the model assigns a higher likelihood to the correct continuation of the two options. Note that this is a more lenient evaluation protocol than sometimes used for language models, where models are evaluated on on their ability to generate the correct continuation, in this case 'no'. The greedy decoding approach (evaluating whether 'yes' or 'no' is generated) is also captured by our approach, but we additionally label an example correct if 'no' is not the highest assigned likelihood, but still higher than 'yes'. We did not opt for greedy decoding because 'no' is not the only coherent continuation here, and marginalising over all possible correct continuations is intractable. The more lenient evaluation does capture implicature resolution, because the choice of 'no' versus 'yes' is only determined by the resolution of the implicature. We guide the models to output 'yes' or 'no' explicitly in three of the six prompt templates with instructions, such that we can estimate the effect of this guidance on performance. For two model classes (i.e. GPT-3.5-turbo and GPT-4) we do not have access to likelihoods, and for these models we take the greedy decoding approach, guiding the model to output 'yes' or 'no' explicitly in all prompts (see Table 6 in Appendix F).

We use a dataset of conversational implicatures curated by George and Mamidi (2020) 5 . It contains implicatures that, like in Figure 1, are presented in utterance-response-implicature tuples. Of these, 718 are binary implicatures that we can convert into an incoherent sentence. We randomly sample 600 examples for the test set and keep the remaining 118 as a development set to improve implicature resolution after pre-training through in-context prompting or fine-tuning.

Few-shot in-context evaluation . We add k examples of the task to the prompt, e.g. with k = 2 :

Esther asked 'Have you found him yet?' and Juan responded 'They're still looking', which means no.

Esther asked 'Are you having fun?' and Juan responded 'Is the pope Catholic?', which means yes.

Finish the following sentence:

Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means no.

We evaluate the models' k -shot capabilities for k ∈ { 1 , 5 , 10 , 15 , 30 } by randomly sampling k examples from the development set for each test example. We opt for a random sampling approach to control for two sources of randomness. Firstly, examples have different levels of informativeness. Secondly, recent work found that the order in which examples are presented matters (Lu et al., 2022). Ideally, to marginalise over these random factors, we would evaluate each test example with all

permutations of k examples from the development set. This requires 118! (118 -k )! evaluations for each test example, which is intractable. Instead, we estimate performance per test example by randomly sampling from the development set. In this way we control for some of the variance in performance, but avoid extra evaluations. We ensure each model sees the same few-shot examples per test example.

Controlling for prompt sensitivity . It has been shown language models are sensitive to prompt wording (Efrat and Levy, 2020; Tan et al., 2021; Reynolds and McDonell, 2021a; Webson and Pavlick, 2021). To control for this factor of randomness we manually curate six different template prompts and measure performance across these. One of the templates has been presented above, namely 'Esther asked < utterance > and Juan responded < response >, which means < implicature >'. Another template is: 'Question: < utterance >, response: < response >, meaning: < implicature >'. The former we call natural prompts and the latter structured prompts. Each group has three templates that only differ slightly in wording. This grouping allows us to look at the variance due to slight changes in wording as well as performance difference due to a completely different way of presenting the example. The full list of prompts can be found in Appendix F.

4 Experiments

The set of large language model classes we evaluate can be grouped into four distinct categories:

Base models : large-scale pre-trained models; RoBERTa (Liu et al., 2019), BERT (Devlin et al., 2018), GPT-2 (Radford et al., 2019), EleutherAI (Wang and Komatsuzaki, 2021; Black et al., 2022), BLOOM (BigScience, 2022), OPT (Zhang et al., 2022), Cohere's base models, and GPT-3 (Brown et al., 2020)

Dialogue FT : LLMs fine-tuned on dialogue, BlenderBot (Ng et al., 2019).

Benchmark IT : LLMs fine-tuned on tasks with natural instructions for each benchmark or 'benchmark-level instruction-tuned models'; T0 (Sanh et al., 2022) and Flan-T5 (Chung et al., 2022).

Example IT : LLMs fine-tuned on tasks with natural instructions for each example or 'example-level instruction-tuned models'; a subset of OpenAI's API models and Cohere's API models).

For Benchmark IT models, annotators write a single instruction for an entire dataset. The models are then fine-tuned on each example from the dataset with the same instruction. We distinguish this from example-level IT; for that type of fine-tuning each example in a dataset gets a new instruction, resulting in a more diverse dataset. Each group contains model classes for which we evaluate a range of sizes. A detailed categorization of the models and their attributes can be found in appendix G. 6 We make use of the OpenAI and Cohere APIs as well as the pretrained models in the transformers library (Wolf et al., 2020) and EleutherAI's framework to evaluate them (Gao et al., 2021). All code used for this paper can be found on GitHub 7 and the dataset is made publicly available on HuggingFace 8 . Below, we present zero-shot and few-shot results, discussing patterns of performance of the models in the four different groups. We further look at the results for different model sizes of each model class and the variance over the prompt templates. We contrast the models' performance with human performance. To this end, each test example gets annotated by five humans. We split the test set in four and assign each annotator a subset, leaving us with twenty annotators in total. The average human performance is 86.2%, and the best performance is 92%. Some of the errors humans make uncover examples that have multiple interpretations, and others uncover annotation errors. The nature of the task of implicature resolution means we do not expect models to perform better than human best performance. Details on the human experiment can be found in the Appendix H (also containing an analysis of human errors), and detailed results per model and prompt template in Appendix K.10. We also test for spurious correlations present in the benchmark (like lexical cues the model can rely on), and find no indication (Appendix K.8).

Insight 1: Models instruction-tuned at the example level outperform all others. Table 1 shows the best 0-, 1-, and 5-shot accuracy each model class achieved on the implicature task. The best overall

accuracy is achieved by GPT-4 (the size of this model is unknown) at 82 . 3% ± 1 . 4 . This leaves a gap of 3.9% with human average performance. All models in the class Example IT perform significantly better than any of the other models for all k , except Cohere-command-52b at 0-shot. This result is more clearly seen in Figure 2, where we present the average accuracy for each model group. The performance for the other model classes across k ranges from 47.0% by BlenderBot-2.7b at k = 5 and 68.7% by GPT-3-175b at k = 5 . Even though base models benefit from few-shot examples, their performance remains mostly closer to random than to humans for all k , showing a gap of at least 17.5% with the average human. We observe a decrease in performance for k > 0 for the group Benchmark IT . This is not surprising, as these kind of models are specifically fine-tuned to be better at zero-shot generalisation (Sanh et al., 2022; Chung et al., 2022). BlenderBot, in the group Dialogue FT , performs barely better than random for all k . We hypothesise that the lower performance which Cohere-command-52b achieves 0-shot is not due to a lack of implicature understanding, but due to a failure to calibrate the yes/no likelihoods without examples. For this model, we observe a sharp rise in performance from k = 0 to k = 1 (see Table 1 or Figure 2). Since it is unlikely that one example of an implicature induces pragmatic understanding, we hypothesise that few-shot prompting mostly serves to clarify the task format. We test this hypothesis in Appendix K.6 by repeating the 1- and 5-shot experiment with random labels for Cohere-command-52B and text-davinci-001. We find that the performance does not degrade, which confirms that the few-shot examples mainly serve to prime the model towards producing outputs following the yes/no structure.

Insight 2: The results are robust to different prompt templates. As detailed in Section 3, each example in the test set is wrapped in six different prompt templates. The standard deviation in Table 1 and in Figure 2 shows the sensitivity to different prompt wording. The standard deviation ranges from 0.3 for BlenderBot to 7.0 for T0-11B. All in all, the sensitivity to prompt wording does not seem to be a problem for this task; when taking into account the confidence intervals the result remains that models in the group Example IT perform significantly better than all other models, but worse than humans. In Appendix K.4 another analysis is presented that shows how different prompt templates benefit from in-context examples. The takeaway from the analysis is that in-context prompting can mitigate the fact that some models are better at natural prompts and others better at structured prompts by improving performance on the type of prompt the model struggles with zero-shot. Again, when

The accuracy versus number of in-context examples (k)

withbarsshowinggroupmeans

100

Example IT

Base

Benchmark IT

Dialogue FT

80

text-davinci-002-*

text-davinci-003-*

text-davinci-001-*

60

ChatGPT-*

Cohere-command-52b

GPT-3-175b

40

OPT-13b

Flan-T5-11b

BLOOM-176b

20

EleutherAI-20b

BlenderBot-3b

T0-11b

0

Zero

One

Five

Ten

Fifteen

Thirty

Random chance

In-context examples (k)

only looking at the best prompt type for each model class (i.e. structured or natural), the results remain that models in the group Example IT perform best.

Insight 3: Models instruction-tuned at the example-level have the most favourable scaling properties, but some base models also show positive correlation with scale. Figure 3 shows the scaling behaviour of the model classes for which we know the number of non-embedding parameters. We highlight 0- and 5-shot results, because for k > 5 the accuracy of most models plateaus (see Figure 2). However, detailed results for other k can be found in Appendix K.10. Note that we do not know the number of parameters for OpenAI's 'text-<engine>-001'-series, but we do know the order of the engines in size, and we separately present its scaling results in Table 2. Except OpenAI's 'text-<engine>-001'-series, none of the models show significant performance increase with model size for the 0-shot evaluations. However, for k -shot evaluations with k ≥ 1 we observe significant positive correlation with size for the models in the Example IT class for which we have multiple sizes (Cohere-command and 'text-<engine>-001') as well as some models in the base model class. Not only do the models in the Example IT class exhibit higher performance for the same model size, these models also have a steeper performance increase with size than the base models. Comparing the scaling properties of the best base model (GPT-3) with Cohere-command, we see that the increase in performance from the second-largest to the largest model is 0.04% per billion parameters from GPT-3-6.7B to GPT-3-175B and 0.15% per billion parameters for Cohere-command-6B to Cohere-command-52b (exact numbers used to calculate the slope can be found in Appendix K.10). If performance is linearly extrapolated from this curve GPT-3 reaches human-level performance at 642b parameters where Cohere-command would need 125b parameters.

Insight 4: GPT-4 reaches average human-level performance with chain-of-thought prompting. For the model groups that benefit from in-context examples, we attempt to push performance further

Performance on the implicature task for k-shot evaluations

Zero-shot.

Five-shot.

100

80

Accuracy (%)

60

40

Best human

OPT

GPT-3

Avg.human

EleutherAI

TO

20

Cohere-command

BLOOM

BlenderBot

Flan-T5

Cohere

Random chance

0

108

109

1070

101

108

109

1070

10TT

with chain-of-thought prompting. We manually write a five-shot chain-of-thought prompt for all six prompt templates, and evaluate model performance using this prompt. One of the six chain-of-thought prompts can be found in Table 4 in Appendix F, and the other five are provided in the supplementary material. We only present the results for the group Example IT here, since CoT prompting did not improve performance for two of the base model classes we tried (see Appendix K.7). Consequently, we decided not to apply this experiment to the other models in the base group to save compute costs. The results of are shown in Table 3. We find that chain-of-thought prompting does not help for all models, but is nonetheless able to boost performance of GPT-4 to 86.5% ± 1.0. This is on-par with average human-level performance, and slightly below human best performance at 89.8%. To illustrate how explicit reasoning helps implicature understanding, we highlight a CoT generated by GPT-4 for an example from the dataset that models persistently get wrong. 'A: Is there a bus I can get to the station? B: You can't rely on it'. The implicature is yes, there is a bus, you just cannot rely on it. GPT-4 five-shot gets this wrong for all six templates. With CoT it gets it right for five of six templates. The generated CoT for one template is the following:

Alice says 'You can't rely on it.' Alice must be implying that there is a bus, but it may not be dependable or timely. This means the response to Bob's question is yes, but with a caution about reliability. Answer: yes

More completions can be found in Appendix J.

Accuracyfor twodifferent types ofimplicature.

NB:Y-axis starts at 45%(just below random performance).

100

Cohere-command-52B

90

Accuracy (%)

80

70

Particularised

Generalised

60

Humanparticularised

Human generalised

50

Randomchance

5

10

15

20

25

30

0

5

10

15

20

25

30

k

k

Insight 5: Models often struggle with the same type of examples humans struggle with. We manually labeled 217 examples of the 600 examples in the test set according to a taxonomy. The remaining 383 examples do not fall as clearly within a category and are grouped together as type other . In Table 4 the two types of examples that occur frequently in the dataset are exemplified. Generalised implicatures require little or no context to be understood. They are the simplest type of example in the test set, and generally imply the same thing ('some' almost always implies 'not all'). Particularised implicatures, by contrast, do require context to be resolved. For example, from Table 4, we need the context that it is undesirable to stay up late drinking when one has to get up early (see in Appendix E for more on generalised vs. particularised). In these type of examples, the context needed to resolve it is different every time. We label three other types of implicatures in the dataset, but since the analysis of these examples does not show significant patterns, we present it in Appendix K.9. We show the accuracy broken down per example type for two models from the Example IT group, as these patterns hold more broadly for almost all models evaluated (see the detailed results broken down per example type in Appendix K.9). Figure 4 shows that for lower k , the models often have a significantly worse performance for particularised examples than for generalised examples, just like humans do. For some, like Cohere-command-52b, this is mitigated by few-shot prompting, which brings particularised and generalised performance closer together (sometimes at the cost of generalised performance). For others, like GPT-4, the gap between particularised and generalised performance remains large for all k . From the bottom row in Figure 4 we observe that the edge GPT-4 has over Cohere-command-52b seems mostly driven by a higher accuracy on generalised examples. The accuracy on the particularised examples is comparable between those two models.

5 Discussion

In this study we use prompting to evaluate whether different groups of LLMs can resolve implicatures. In designing our experimental protocol, we carefully considered various alternatives, and here we discuss limitations of the chosen approach. Firstly, evaluating LLM competencies is inherently uncertain and sensitive to prompt choice. Nonetheless, we are confident our evaluation is comprehensive enough to assess implicature understanding: we apply six different prompt templates per test example, each used in three different prompting techniques (zero-shot, few-shot, chain-of-thought). Addition-

ally, in the appendix we present alternative zero-shot prompts and task specifications (Appendix K.3 and K.1 respectively), but since these did not improve performance they were not further considered. Another limitation is the fact that a subset of the models we evaluate are behind APIs. This means models are subject to change (affecting reproducibility) and certain details about these models are unknown. This affects the group instruction-tuned at the example-level, which is the group we find outperforms all others and has the most favourable scaling properties. How do we know instructiontuning at the example-level is the main driver behind these findings without controlled A/B testing? Unfortunately, due to the secrecy surrounding the exact implementation of these models we cannot be certain, but we can be relatively confident. We evaluated ten models across six model classes and two APIs in the group example-level instruction tuned. Within this group, models probably vary significantly in other training and architecture details (especially Cohere-command models versus OpenAI models). The most salient commonality they share with each other and none of the other models is multi-task instruction-tuning at the example level, making it likely that this is the driving factor of their performance. A further datapoint in favour of this conclusion can be seen in Figure 3 (right); base models at similar scales as Example IT models perform significantly worse. We see that Cohere-command 52B significantly outperforms Cohere-base 52B, and the only difference between those models is instruction-tuning at the example level (Cohere-command is fine-tuned from Cohere-base). In fact, Cohere-command 52B outperforms other base models more than 3 times the size by a large margin (e.g. GPT-3 175B, BLOOM-176B, OPT-175B). We are therefore confident that instruction-tuning at the example-level is important for pragmatic understanding, an insight which can guide the development of open-source models capable of pragmatic understanding. Investigating the exact effect of this type of instruction-tuning on pragmatic understanding in a controlled setting is an interesting future work direction (e.g. by isolating the effect of data diversity from instructions). Another limitation is that some evaluations are subject to API stochasticity, which we address in Appendix K.5. After running the zero-shot experiment ten times through each API we conclude there is some stochasticity, but it is too small to impact our conclusions. We publish exact timestamps at which we queried APIs in Appendix L. Further, a downside of doing a comprehensive analysis on many models is compute costs. In Appendix M we publish a list of exact compute used (time and hardware), as well as estimated carbon emissions for each of the models that are not behind an API. Finally, the likelihood ranking approach we take limits our study to implicatures with clear alternative. However, implicatures in natural language can entail more complex propositions. For example, imagine Esther now asking 'Can I use your stapler?' and Juan responding 'Here's the key to my office.'. Juan is implicating that (1) Esther can use the stapler, (2) the stapler is located in the office, and (3) the office is currently locked. This leaves ample room for the design of benchmarks with implicatures entailing multiple non-binary propositions.

6 Conclusion

LLMs have made remarkable progress on fluency and coherence in recent years. We argue however that a central aspect of language understanding is missing from evaluations. To understand language means to understand its pragmatics: its usage in a context that incorporates commonsense understanding, goals, objectives, and so on. We design a protocol that evaluates LLMs on binary implicature resolution and establish a significant gap with human understanding for SotA LLMs in three categories; large-scale pre-trained models, models fine-tuned on conversations, and models fine-tuned with benchmark-level instructions. By contrast, we find that models fine-tuned on example-level instructions perform significantly better. This group also exhibits the best correlation between accuracy and model size. Scaling analysis shows that for some large-scale pre-trained models accuracy also positively correlates with model size, but the best model in this group would need at least five times more parameters to reach similar performance. From these results, we conclude that instruction-tuning at the example level is important for pragmatic understanding. We hypothesise that there is something about the multi-task data diversity obtained from example-level instructions (i.e. each example a new task) that makes pragmatic understanding appear at smaller scale.

References

Adolphs, L., Börschinger, B., Buck, C., Huebscher, M. C., Ciaramita, M., Espeholt, L., Hofmann, T., Kilcher, Y., Rothe, S., Sessa, P. G., and Sestorain, L. (2022). Boosting search engines with interactive agents. Transactions on Machine Learning Research .

American Psychiatric Association, A. P. A. (2013). Diagnostic and statistical manual of mental disorders : DSM-5 . American Psychiatric Association Arlington, VA, 5th ed. edition.

Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T. B., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A general language assistant as a laboratory for alignment. CoRR , abs/2112.00861.

Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732 .

Bach, K. (1999). The myth of conventional implicature. Linguistics and Philosophy , 22(4):327-366.

Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 .

Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 610-623.

Biderman, S. and Raff, E. (2022). Fooling moss detection with pretrained language models. arXiv preprint arXiv:2201.07406 .

BigScience (2022). Bigscience language open-science open-access multilingual (bloom) language model.

Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. (2022). GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models .

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc.

Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. (2022). Scaling instruction-finetuned language models.

Cianflone, A., Feng, Y., Kabbara, J., and Cheung, J. C. K. (2018). Let's do it 'again': A first computational approach to detecting adverbial presupposition triggers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2747-2755, Melbourne, Australia. Association for Computational Linguistics.

Davis, W. (2019). Implicature. In Zalta, E. N., editor, The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, fall 2019 edition.

Davis, W. A. (1998). Implicature : intention, convention, and principle in the failure of Gricean theory / Wayne A. Davis. Cambridge studies in philosophy. Cambridge University Press, Cambridge England ; New York.

Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2018). BERT: pre-training of deep bidirectional transformers for language understanding. CoRR , abs/1810.04805.

Efrat, A. and Levy, O. (2020). The turking test: Can language models understand instructions? CoRR , abs/2010.11982.

Frank, M. C. and Goodman, N. D. (2012). Predicting pragmatic reasoning in language games. Science , 336:998 - 998.

Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. (2021). A framework for few-shot language model evaluation.

Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. (2020). RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3356-3369, Online. Association for Computational Linguistics.

George, E. J. and Mamidi, R. (2020). Conversational implicatures in english dialogue: Annotated dataset. Procedia Computer Science , 171:2316-2323. https://doi.org/10.1016/j.procs.2020.04.251.

Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Sanchez Elias, J., Green, R., Mokrá, S., Fernando, N., Wu, B., Foley, R., Young, S., Gabriel, I., Isaac, W., Mellor, J., Hassabis, D., Kavukcuoglu, K., Hendricks, L. A., and Irving, G. (2022). Improving alignment of dialogue agents via targeted human judgements.

Goodman, N. and Frank, M. (2016). Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences , 20.

Green, G. (1996). Pragmatics and Natural Language Understanding . Tutorial essays in cognitive science. Erlbaum.

Greene, S. and Resnik, P. (2009). More than words: Syntactic packaging and implicit sentiment. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pages 503-511, Boulder, Colorado. Association for Computational Linguistics.

Grice, H. P. (1975). Logic and conversation. In Cole, P. and Morgan, J. L., editors, Syntax and Semantics: Vol. 3: Speech Acts , pages 41-58. Academic Press, New York.

Hosseini, M. J., Radlinski, F., Pareti, S., and Louis, A. (2023). Resolving indirect referring expressions for entity selection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 12313-12335, Toronto, Canada. Association for Computational Linguistics.

Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. (2022). Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207 .

Huang, Y. (2017). The Oxford Handbook of Pragmatics . Oxford handbooks in linguistics. Oxford University Press.

Jeretic, P., Warstadt, A., Bhooshan, S., and Williams, A. (2020). Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 8690-8705, Online. Association for Computational Linguistics.

Kasirzadeh, A. and Gabriel, I. (2022). In conversation with artificial intelligence: aligning language models with human values.

Katsos, N., Roqueta, C. A., Estevan, R. A. C., and Cummins, C. (2011). Are children with specific language impairment competent with the pragmatics and logic of quantification? Cognition , 119(1):43-57.

Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V ., and Irving, G. (2021). Alignment of language agents. CoRR , abs/2103.14659.

Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stenetorp, P., Jia, R., Bansal, M., Potts, C., and Williams, A. (2021). Dynabench: Rethinking benchmarking in nlp. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4110-4124, Online. Association for Computational Linguistics.

Kim, N., Pavlick, E., Karagol Ayan, B., and Ramachandran, D. (2021). Which linguist invented the lightbulb? presupposition verification for question-answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3932-3945, Online. Association for Computational Linguistics.

Kim, S. Y., Park, H., Shin, K., and Kim, K.-M. (2022). Ask me what you need: Product retrieval using knowledge from gpt-3. arXiv preprint arXiv:2207.02516 .

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners.

Lepore, E. and Stone, M. (2014). Imagination and Convention: Distinguishing Grammar and Inference in Language . Oxford University Press.

Levinson, S. C. (1983). Pragmatics . Cambridge University Press, Cambridge, U.K.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459-9474.

Li, E., Schuster, S., and Degen, J. (2021). Predicting scalar inferences from 'or' to 'not both' using neural sentence encoders. In Proceedings of the Society for Computation in Linguistics 2021 , pages 446-450, Online. Association for Computational Linguistics.

Lin, S., Hilton, J., and Evans, O. (2022). TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3214-3252, Dublin, Ireland. Association for Computational Linguistics.

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized BERT pretraining approach. CoRR , abs/1907.11692.

Louis, A., Roth, D., and Radlinski, F. (2020). 'I'd rather just go to bed': Understanding indirect answers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7411-7425, Online. Association for Computational Linguistics.

Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. (2022). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL .

Ng, N., Yee, K., Baevski, A., Ott, M., Auli, M., and Edunov, S. (2019). Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) , pages 314-319, Florence, Italy. Association for Computational Linguistics.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. J. (2022). Training language models to follow instructions with human feedback. ArXiv , abs/2203.02155.

Parrish, A., Schuster, S., Warstadt, A., Agha, O., Lee, S.-H., Zhao, Z., Bowman, S. R., and Linzen, T. (2021). NOPE: A corpus of naturally-occurring presuppositions in English. In Proceedings of the 25th Conference on Computational Natural Language Learning , pages 349-366, Online. Association for Computational Linguistics.

Patel, R. and Pavlick, E. (2021). 'was it 'stated' or was it 'claimed'?: How linguistic bias affects generative language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 10080-10095, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Potts, C. (2005). The Logic of Conventional Implicatures . Oxford University Press UK.

Potts, C. (2006). Conversational implicatures via general pragmatic pressures. In Washio, T., Satoh, K., Takeda, H., and Inokuchi, A., editors, New Frontiers in Artificial Intelligence , pages 205-218, Berlin, Heidelberg. Springer Berlin Heidelberg.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners.

Recasens, M., Danescu-Niculescu-Mizil, C., and Jurafsky, D. (2013). Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1650-1659, Sofia, Bulgaria. Association for Computational Linguistics.

Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., Eccles, T., Bruce, J., Razavi, A., Edwards, A., Heess, N., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. (2022). A generalist agent.

Reynolds, L. and McDonell, K. (2021a). Prompt programming for large language models: Beyond the few-shot paradigm.

Reynolds, L. and McDonell, K. (2021b). Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , CHI EA '21, New York, NY, USA. Association for Computing Machinery.

Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Fevry, T., Fries, J. A., Teehan, R., Scao, T. L., Biderman, S., Gao, L., Wolf, T., and Rush, A. M. (2022). Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations .

Schaeken, W., Van Haeren, M., and Bambini, V. (2018). The understanding of scalar implicatures in children with autism spectrum disorder: Dichotomized responses to violations of informativeness. Frontiers in Psychology , 9.

Schuster, S., Chen, Y., and Degen, J. (2020). Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5387-5403, Online. Association for Computational Linguistics.

Sperber, D. and Wilson, D. (1986). Relevance: Communication and Cognition . Language and thought series. Harvard University Press.

Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Slone, A., Rahane, A., Iyer, A. S., Andreassen, A., Madotto, A., Santilli, A., Stuhlmüller, A., Dai, A., La, A., Lampinen, A., Zou, A., Jiang, A., Chen, A., Vuong, A., Gupta, A., Gottardi, A., Norelli, A., Venkatesh, A., Gholamidavoodi, A., Tabassum, A., Menezes, A., Kirubarajan, A., Mullokandov, A., Sabharwal, A., Herrick, A., Efrat, A., Erdem, A., Karaka¸ s, A., Roberts, B. R., Loe, B. S., Zoph, B., Bojanowski, B., Özyurt, B., Hedayatnia, B., Neyshabur, B., Inden, B., Stein, B., Ekmekci, B., Lin, B. Y., Howald, B., Diao, C., Dour, C., Stinson, C., Argueta,

C., Ramírez, C. F., Singh, C., Rathkopf, C., Meng, C., Baral, C., Wu, C., Callison-Burch, C., Waites, C., Voigt, C., Manning, C. D., Potts, C., Ramirez, C., Rivera, C. E., Siro, C., Raffel, C., Ashcraft, C., Garbacea, C., Sileo, D., Garrette, D., Hendrycks, D., Kilman, D., Roth, D., Freeman, D., Khashabi, D., Levy, D., González, D. M., Perszyk, D., Hernandez, D., Chen, D., Ippolito, D., Gilboa, D., Dohan, D., Drakard, D., Jurgens, D., Datta, D., Ganguli, D., Emelin, D., Kleyko, D., Yuret, D., Chen, D., Tam, D., Hupkes, D., Misra, D., Buzan, D., Mollo, D. C., Yang, D., Lee, D.-H., Shutova, E., Cubuk, E. D., Segal, E., Hagerman, E., Barnes, E., Donoway, E., Pavlick, E., Rodola, E., Lam, E., Chu, E., Tang, E., Erdem, E., Chang, E., Chi, E. A., Dyer, E., Jerzak, E., Kim, E., Manyasi, E. E., Zheltonozhskii, E., Xia, F., Siar, F., Martínez-Plumed, F., Happé, F., Chollet, F., Rong, F., Mishra, G., Winata, G. I., de Melo, G., Kruszewski, G., Parascandolo, G., Mariani, G., Wang, G., Jaimovitch-López, G., Betz, G., Gur-Ari, G., Galijasevic, H., Kim, H., Rashkin, H., Hajishirzi, H., Mehta, H., Bogar, H., Shevlin, H., Schütze, H., Yakura, H., Zhang, H., Wong, H. M., Ng, I., Noble, I., Jumelet, J., Geissinger, J., Kernion, J., Hilton, J., Lee, J., Fisac, J. F., Simon, J. B., Koppel, J., Zheng, J., Zou, J., Koco´ n, J., Thompson, J., Kaplan, J., Radom, J., Sohl-Dickstein, J., Phang, J., Wei, J., Yosinski, J., Novikova, J., Bosscher, J., Marsh, J., Kim, J., Taal, J., Engel, J., Alabi, J., Xu, J., Song, J., Tang, J., Waweru, J., Burden, J., Miller, J., Balis, J. U., Berant, J., Frohberg, J., Rozen, J., Hernandez-Orallo, J., Boudeman, J., Jones, J., Tenenbaum, J. B., Rule, J. S., Chua, J., Kanclerz, K., Livescu, K., Krauth, K., Gopalakrishnan, K., Ignatyeva, K., Markert, K., Dhole, K. D., Gimpel, K., Omondi, K., Mathewson, K., Chiafullo, K., Shkaruta, K., Shridhar, K., McDonell, K., Richardson, K., Reynolds, L., Gao, L., Zhang, L., Dugan, L., Qin, L., Contreras-Ochando, L., Morency, L.-P., Moschella, L., Lam, L., Noble, L., Schmidt, L., He, L., Colón, L. O., Metz, L., ¸ Senel, L. K., Bosma, M., Sap, M., ter Hoeve, M., Farooqi, M., Faruqui, M., Mazeika, M., Baturan, M., Marelli, M., Maru, M., Quintana, M. J. R., Tolkiehn, M., Giulianelli, M., Lewis, M., Potthast, M., Leavitt, M. L., Hagen, M., Schubert, M., Baitemirova, M. O., Arnaud, M., McElrath, M., Yee, M. A., Cohen, M., Gu, M., Ivanitskiy, M., Starritt, M., Strube, M., Sw˛ edrowski, M., Bevilacqua, M., Yasunaga, M., Kale, M., Cain, M., Xu, M., Suzgun, M., Tiwari, M., Bansal, M., Aminnaseri, M., Geva, M., Gheini, M., T, M. V., Peng, N., Chi, N., Lee, N., Krakover, N. G.-A., Cameron, N., Roberts, N., Doiron, N., Nangia, N., Deckers, N., Muennighoff, N., Keskar, N. S., Iyer, N. S., Constant, N., Fiedel, N., Wen, N., Zhang, O., Agha, O., Elbaghdadi, O., Levy, O., Evans, O., Casares, P. A. M., Doshi, P., Fung, P., Liang, P. P., Vicol, P., Alipoormolabashi, P., Liao, P., Liang, P., Chang, P., Eckersley, P., Htut, P. M., Hwang, P., Miłkowski, P., Patil, P., Pezeshkpour, P., Oli, P., Mei, Q., Lyu, Q., Chen, Q., Banjade, R., Rudolph, R. E., Gabriel, R., Habacker, R., Delgado, R. R., Millière, R., Garg, R., Barnes, R., Saurous, R. A., Arakawa, R., Raymaekers, R., Frank, R., Sikand, R., Novak, R., Sitelew, R., LeBras, R., Liu, R., Jacobs, R., Zhang, R., Salakhutdinov, R., Chi, R., Lee, R., Stovall, R., Teehan, R., Yang, R., Singh, S., Mohammad, S. M., Anand, S., Dillavou, S., Shleifer, S., Wiseman, S., Gruetter, S., Bowman, S. R., Schoenholz, S. S., Han, S., Kwatra, S., Rous, S. A., Ghazarian, S., Ghosh, S., Casey, S., Bischoff, S., Gehrmann, S., Schuster, S., Sadeghi, S., Hamdan, S., Zhou, S., Srivastava, S., Shi, S., Singh, S., Asaadi, S., Gu, S. S., Pachchigar, S., Toshniwal, S., Upadhyay, S., Shyamolima, D., Shakeri, S., Thormeyer, S., Melzi, S., Reddy, S., Makini, S. P., Lee, S.-H., Torene, S., Hatwar, S., Dehaene, S., Divic, S., Ermon, S., Biderman, S., Lin, S., Prasad, S., Piantadosi, S. T., Shieber, S. M., Misherghi, S., Kiritchenko, S., Mishra, S., Linzen, T., Schuster, T., Li, T., Yu, T., Ali, T., Hashimoto, T., Wu, T.-L., Desbordes, T., Rothschild, T., Phan, T., Wang, T., Nkinyili, T., Schick, T., Kornev, T., Telleen-Lawton, T., Tunduny, T., Gerstenberg, T., Chang, T., Neeraj, T., Khot, T., Shultz, T., Shaham, U., Misra, V., Demberg, V., Nyamai, V., Raunak, V., Ramasesh, V., Prabhu, V. U., Padmakumar, V., Srikumar, V., Fedus, W., Saunders, W., Zhang, W., Vossen, W., Ren, X., Tong, X., Zhao, X., Wu, X., Shen, X., Yaghoobzadeh, Y., Lakretz, Y., Song, Y., Bahri, Y., Choi, Y., Yang, Y., Hao, Y., Chen, Y., Belinkov, Y., Hou, Y., Hou, Y., Bai, Y., Seid, Z., Zhao, Z., Wang, Z., Wang, Z. J., Wang, Z., and Wu, Z. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.

Tan, Z., Zhang, X., Wang, S., and Liu, Y. (2021). Msp: Multi-stage prompting for making pre-trained language models better translators. arXiv preprint arXiv:2110.06609 .

Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C., Krivokon, I., Rusch, W., Pickett, M., Meier-Hellstern, K. S., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M.,

Kuzmina, V., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E. H., and Le, Q. (2022). Lamda: Language models for dialog applications. CoRR , abs/2201.08239.

Volden, J. (2017). Autism Spectrum Disorder , pages 59-83. Springer International Publishing, Cham.

Wang, B. and Komatsuzaki, A. (2021). GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax .

Webson, A. and Pavlick, E. (2021). Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247 .

Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V . (2022). Finetuned language models are zero-shot learners. In International Conference on Learning Representations .

Wittgenstein, L. (1921). Tractatus logico-philosophicus. London: Routledge, 1981 .

Wittgenstein, L. (1953). Philosophical Investigations . Basil Blackwell, Oxford.

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y ., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online. Association for Computational Linguistics.

Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. (2022). Opt: Open pre-trained transformer language models.

Zheng, Z., Qiu, S., Fan, L., Zhu, Y., and Zhu, S.-C. (2021). GRICE: A grammar-based dataset for recovering implicature and conversational rEasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 2074-2085, Online. Association for Computational Linguistics.

Appendices

Contents

A Contributions

Laura Ruis : project proposal and leadership, dataset development, code writing, human experiment, manual error analysis, paper writing and editing.

Akbir Khan : code writing, model evaluations, human experiment, paper writing and editing.

Stella Biderman : model evaluations, compute usage, paper writing and editing, advisor.

Sara Hooker : compute usage, paper writing and editing, advisor.

Tim Rocktäschel : paper writing and editing, advisor.

Edward Grefenstette : initial idea, manual error analysis, paper writing and editing, advisor.

B Reproducibility Statement

We share all the data, human annotations, code used for the evaluations, and the raw results in the supplementary material. Additionally, in Appendix K.5 we estimate the variance due to stochasticity in the API's of OpenAI and Cohere. Of course, if either OpenAI or Cohere decides to change the models behind the API, the results might look different. We publish the exact date and time each API was queried for the results in Appendix L. Finally, in Appendix K.2 we estimate the variance over the prompt order of the in-context examples. The compute used for the experiments is detailed in Appendix M. The remaining experiments were done with API credits received as a research grant from OpenAI and Cohere.

C Ethics Statement

In this work, we conduct a study with human subjects (see Appendix H for details). To get matched with participants, we used the platform Prolific. Prolific complies with ethical standards according to UK law (e.g. complying with the GDPR). We compensated participants with a UK living wage at 15 GBP an hour, which is 6 GBP an hour more than Prolific recommends at 9 GBP per hour. Implicature is an aspect of pragmatics, and pragmatic language impairments are universal in Autism Spectrum Disorder (ASD) (American Psychiatric Association, 2013). Difficulties in understanding scalar implicatures are claimed to be present in people with ASD (Volden, 2017), although the nature of the relation has proven hard to establish and has recently been debated (Katsos et al., 2011; Schaeken et al., 2018). For the purposes of this work, whether or not implicature understanding relates to ASD is not important. We took the following steps to make sure no sensitive data is collected or published. The human annotations we obtain are anonymous, related to a participant only by their Prolific ID for the purposes of compensation. In publishing the human annotations, we will not publish the Prolific ID of participants or anything else related to the participants. Additionally, we did not collect or request any personal or demographic characteristics of the participants apart from that they are all native English speakers.

Additionally, in this work we run a lot of compute-intensive experiments. We publish the estimated emissions per experiment in Appendix M. The total amount of GPU hours is estimated at maximally 966. How this is broken down per experiment can be seen in Appendix M.

D Opener example with InstructGPT

This quote was obtained through the OpenAI playground for text-davinci-002 on May 15th 2023. The model text-davinci-001 consistently generates better responses for the same prompt. GPT-3 itself (i.e. davinci) mainly gives nonsensical answers. In the following, the prompt is italic and the completion bold . The completion was generated with a maximum of 10 tokens and a temperature of 0:

User: 'Have you seen my phone?'

GPT-3: ' Yes, I have seen your phone.'

The opener example is used to introduce the problem we are looking at, and not to judge the model used to generate it. In fact, although text-davinci-002 sometimes completes conversations in a way that is unexpected according to pragmatic language usage, it is one of the better models when evaluated few-shot.

E Background on implicature

The first influential consideration of implicature is Grice (1975). In his work, Grice continues the trend of moving away from purely logical accounts of language started by Wittgenstein (1921) by hypothesising implicatures arise in conversation when some mutually agreed upon maxims seem to be violated. For example, if we agree on only making relevant contributions to conversation, Juan's response in the introduction seemingly violates this maxim-after all, he starts talking about work when Esther asks him about a party. However, because Juan agreed to be relevant he must be implying that having to work means he cannot come to the party. Grice contrasts conversational implicatures that arise through context with conventional implicatures. These are implicatures where the conventional meaning of the word determines what is implicated. An example given by Grice is the following sentence: 'he is an Englishman; he is therefore brave.'. Grice notes that this sentence does not literally state that an Englishman being brave is a direct consequence of him being English, but it's implied by the conventional meaning of the word 'therefore'.

Since then, issues with the Gricean cooperative principle have been pointed out by many (Levinson, 1983; Sperber and Wilson, 1986; Davis, 1998; Lepore and Stone, 2014). The most influential alternative theory is relevancy theory by Sperber and Wilson (1986). They do away with the cooperative principle and instead theorise implicatures arise because speakers try to produce utterances that are both as relevant as possible and require the least effort to process. Another point of contention is the incorporation of conventional implicatures on the pragmatics side. Bach (1999) argues that there is no such thing as conventional implicatures, and they are simply instances of something else. Based on a thorough treatment of what Grice calls conventional implicatures, Bach argues all examples of it can be filed under other concepts within semantics, like utterance modifiers (called 'utterance modifiers' instead of 'sentence modifiers' because they go against the semantic content of the rest of the sentence). Potts (2005) also argues that to explain conventional implicatures we can stay on semantic turf. Indeed, even Grice himself says conventional implicatures derive from the meaning of the words, not from conversational context. However, Potts does not claim conventional implicatures do not exist, but instead argues they arise by a combination of lexical meaning and novel ways of combining words-the latter being the well-known principle of compositionality, an important part of semantics, not of pragmatics. Potts provides us with an illuminating demarcation between conventional and conversational implicatures. Conventional implicatures are never negotiable by context, whereas conversational implicatures are context-dependent and can always be cancelled without causing incoherent discourse. Consider again the sentence 'he is an Englishman; he is therefore brave.' and the sentence 'Eddie has three bicycles' (implicating that Eddie has exactly three bicycles and not more). The former sentence can not be cancelled by new context without contradiction, whereas for the latter, if we continue saying 'In fact, Eddie has 10 bicycles, he is a bicycle junkie', we have cancelled the implicature. This demarcation clearly puts conventional implicatures on the semantic side, and conversational implicatures on the pragmatic side. Potts goes on by providing a formal theory for conventional implicatures.

In later work, Potts (2006) describes how pragmatic pressures interacting with context cause conversational implicature to arise. He shows how sensitive conversational implicatures are to small changes in the context. Novel information about a speaker's belief state might completely change what is implied. There are many more models of implicature that aim to explain how humans understand language in context. Most notably, Frank and Goodman (2012) formalise the view that speakers produce utterances that are helpful and not longer than necessary with a Bayesian model called the rational speech act (RSA). Many variants on the RSA framework have since been proposed. For example, Goodman and Frank (2016) extend it to handle nonliteral uses of language, like irony, and metaphor. In the context of computational models, prior work uses insights from pragmatics to show that the use of certain words can make a language model produce biased completions (Patel and Pavlick (2021), e.g. saying someone 'claimed' something rather than 'said' something), and inform bias and sentiment classifiers (Greene and Resnik, 2009; Recasens et al., 2013).

In this work, we focus on conversational implicatures and not on conventional implicatures. All conversational implicatures are negotiable by context, but the way they depend on context can be different. Grice (1975) identifies generalised conversational implicatures and particularised conversational implicatures. The former require little or no context to be resolved. For example, 'some athletes smoke' can imply 'not all athletes smoke', but might also imply 'I do not know whether all athletes smoke' when it is a response to the question 'do you know whether all athletes

smoke?' (Davis, 2019). The latter only arise in certain contexts. For example, the response 'I have an early morning' to the question 'do you want to stay for a drink?'.

F Detailed prompt templates

Table 5 contains the full prompt templates we used for the main evaluation and Table 7 contains the extra prompt templates.

#

Prompt template

1

Bob asks Alice a question, and Alice responds with an implicature. This means that Alice's response

does not literally contain the answer to Bob's question, but implies an answer. Assuming that Alice is a

cooperative conversational partner, what is the implicated answer to the question? For example:

Bob: You invented fire?

Alice: I told you that.

Implicature: Alice says 'I told you that'. Alice's response must be relevant to Bob's question because

Alice is a cooperative conversational partner. Alice must mean that she told Bob that she invented fire.

Alice's response to Bob's question 'You invented fire?' is yes.

Answer: yes

Bob: That cake looks delicious. Aren't you going to have some with me?

Alice: But that was a huge meal we just had.

Implicature: Alice's response must be relevant to Bob's question because Alice is a cooperative

conversational partner. Alice must mean that the meal they just had was so huge she is not hungry

anymore, and this must be relevant to Bob's question: 'Aren't you going to have some with me?'

Alice's response to the question must therefore be no.

Answer: no

Bob: Could you perform well?

Alice: Being bilingual would help put me ahead of the pack.

Implicature: Alice says being bilingual would help put her ahead of the pack. Alice's response must

be relevant to Bob's question because Alice is a cooperative conversational partner. Alice must be

implying that she could perform well because she is bilingual. This means the response to Bob's

question is yes.

Answer: yes

Bob: Have you any news for me?

Alice: I've made progress

Implicature: Alice says she has made progress. Alice's response must be relevant to Bob's

question because Alice is a cooperative conversational partner. If Alice would not have any

news for Bob, Alice would not have said she would have made progress because that would

be misleading. The answer to Bob's question 'Have you any news for me?' must therefore be yes.

Answer: yes

Bob: You looked out for him?

Alice: He looked out for me. He taught me.

Implicature: Bob asks Alice 'You looked out for him?' and Alice's response says that the person

that is being referred to by 'him' here looked out for Alice. If Alice meant yes to Bob's

question, Alice would have said something like 'he also looked out for me'. Stating the

response like this implies that the answer to Bob's question is no.

Answer: no

Only output a 'yes' or 'no' as a final answer. Write your reasoning after 'Implicature:'

and then output either 'Answer: yes' or 'Answer: no'.

utterance

>

response

>

Implicature:

G Model categorization

Table 9 contains details on the model classes that are a part of each group of models we evaluate, along with their model sizes.

H Human evaluation

The participants for the human evaluation in this paper were recruited using Prolific ( www.prolific. co ). The setup of the experiment is as follows. We divide the test set of 600 examples into four nonoverlapping subsets of 150 examples. Each set of 150 examples was given to five unique annotators. This means each example in the test set is labeled five times by different people, and we have in total twenty annotators for the whole test set (five different ones for each of the four subsets). The only constraint for the annotators is that they are native English speakers. In Figure 5 the screen shown to potential participants on Prolific is shown. Participants are paid 15 pounds an hour, which was the living wage at the time of the experiment and more than the 12 dollars an hour Prolific recommends. The total amount spent on the human evaluation is 236 pounds. This amount came to be from four subsets, each costing about 30 minutes to label per annotators, and having 5 annotators per subset: 15 * 4 * 0.5 * 5 = 150. The extra costs were for the annotator that didn't pass the attention check which we paid nonetheless, and for prolific as a platform.

UCL

Astudyabouttheimpliedmeaningsofresponses

7.50 ▪15.00/hr30 mins6 places

The aim of this study

We want to compare the performance of humans and language models (like GPT-3) on an important aspect of language

What you'll do

You'll see a set of 150 sentences in a google form, and are asked to finish the sentence with yes/no, whichever makes more sense to you given the sentence.

Please finish each of the 150 questions. If you don't know the answer, just take the best guess. To avoid your submission being rejected, make sure you enter your

correct unique Prolific ID in the first question, fill out the correct email address to receive your completion code, finish all the 150 questions, and pass the attention

test. We expect this task to take around 30 minutes. When you've submitted the form, we will check that everything has been filled out and send you the

completion code that you can use to tell Prolific you've finished the task.

Sensitive information

No sensitive information is required to give in this test and the answers are completely anonymised. We don't do anything with the information about you except

match the prolific ID to the responses.

Reward

We wil pay out the reward as quickly as possible after your submission, hopefully within a day, at least within a week.

Devices you can use to take this study:

Desktop MobileTablet

Openstudylinkinanewwindow

The 150 test examples are wrapped in prompt template 2 (see Table 5) and presented in a Google form. We opted to wrap all examples in prompt template 2 to make the full human study directly comparable to the model's results on template 2. If we had done a mix of all templates we either had to spent six times as much on the human evalyations (which was not within our budget) or subsample evaluations, making it less comparable to part of the model study. Although models have been shown to be very sensitive to prompt wording, humans are less likely to perform differently for different prompt templates. All templates are coherent natural language that any native English speaker will understand. That said, this is speculative, and to confirm this hypothesis future work should investigate the effect of different wordings on implicature resolution by humans. The participants are asked to choose the correct continuation, yes or no (see Figure 6a). As recommended by Prolific, we subject the participants to an attention test (see Figure 6b). At three random places in the form, we add a question that does not contain an implicature and obviously maps to 'yes'. In this way, if the participants fails at least two of these questions, we can conclude they were not paying attention and remove their answers from the result. In practice, this happened once and we decided to pay the participant regardless, but discard their results, which were close to random.

Table 10 shows the performance of each annotator on the subset they annotated. The average human performance across subsets and annotators is 86.2% ± 2.3, the best performance is 89.8% ± 2.2, and the worst performance is 83.5% ± 1.5. The column 'IAA' shows the average Cohen's Kappa coefficient which is the pairwise inter-annotator agreement for each annotator per subset. All agreements are substantial according to the interpretation guidelines for Cohen's Kappa (between 0.61-0.80).

Astudyabouttheimpliedmeaningsof

responses

Each of the responses in the following sentences means yes or no, choose the right one.

What is your unique Prolific ID?*

Which email can we send your Completion Code to once you finish this form? *

Esther asked “ls the dress perfect?” and Juan responded “it fits like a glove.”,

which means

OYes

ON

(a) The start of the Google form participants are asked to fill out for the human study.

Esther asked “Aren't you scared?” and Juan responded “Sometimes.",which

means

Yes

ON

Esther asked“Are you focused?” and Juan responded “Yes., which means *

Yes

ON

Esther asked “ls it a difficult test?” and Juan responded “lt would be a piece of

cake',whichmeans

O Yes

ON

(b) Part of the Google form the participants are asked to fill out. The second question in this image is part of the attention test. Juan's response does not contain an implicature but simply gives away the correct answer.

Humansource of disagreement with ground-truth . Wedoan analysis of the source of disagreement with the ground-truth. We explicitly do not call this error, as in some cases examples might allow multiple interpretations, and both could be right. In other cases, the ground-truth might be wrong.

Annotation errors and multiple interpretations : We analyse the examples for which most humans choose a different answer than the ground-truth. For 30 out of 600 examples in the test set, only one or zero people choose the same answer as the ground-truth. Of these examples, most are annotated wrongly (18 of 30). For example: 'Are you busy?', 'I'm drowning in work.', implicature: 'no'. Some are examples that can have multiple different interpretations (12 of 18), and the ground-truth answer likely just chooses one that is unnatural to humans. For example: 'You don't remember them?', 'Leave me alone!', implicature: 'yes'. 6 of the 30 examples are particularised, and 1 is generalised.

Examples for which all humans agree with the ground-truth : There are 409 out of 600 examples that all humans get correct. This set of examples contains most of the generalised implicatures (39 out of 47). These contain 58 out of 94 particularised examples.

Examples most humans agree with the ground-truth : When we look at examples that 3 or more humans got correct, that comprises most of the test set (530 of 600), and all of the generalised examples (47 of 47). This subset has 78 of 94 particularised examples, so for 16 particularised examples 3 or more humans disagree with the ground-truth.

I Comparison with BIG-bench implicatures task

One of the BIG-bench tasks is related to the task in this work 9 . It uses the same underlying dataset we use (George and Mamidi, 2020). With the below we aim to discuss our contribution in light of the BIG-bench result. To summarise; the methodology used by the BIG-bench task contributors has limitations, which call into question the validity of their claims. Further, some of the BIGbench results are irreproducible due to missing details in the tech report and the use of proprietary models. Considering this, our work is an important contribution validating the BIG-bench results in a reproducible and methodologically sound way, and above that providing insight into what aspects of LLM training are crucial for the ability to do pragmatic inferences.

Limitations of the methodological approach of the task contributors in BIG-bench implicatures . Our benchmark has 30% more data, which the BIG-bench task contributors discard. In this section we motivate the crucial importance of that data for evaluating implicature understanding (Section I.1), and why BIG-bench in turn might be overestimating the performance of LLMs on implicature resolution (Section I.2). Moreover, the human performance on the BIG-bench task indicates low quality human annotation, which we will also elaborate upon below, noting that this is impossible to verify because the BIG-bench report does not detail how the evaluation was done for this task (Section I.3).

I.1 Discarding ambiguous examples

The BIG-bench task preprocesses the 1001 examples that George and Mamidi (2020) curate by keeping only yes/no questions, discarding any examples that are ambiguous according to the task contributors, and discarding remaining examples to create a 50/50 distribution of yes/no answers. This leaves them with 492 examples. Of these examples, 81 appear in our development set and the remaining 411 appear in our test set. Our test set has 600 examples, so BIG-bench effectively discarded 189 ambiguous examples compared to our test set; a bit more than 30% of the benchmark. To illustrate the importance of not discarding this data, we cherry picked a few examples that the BIG-bench authors discarded from the data.

Utterance: "Can you lend me hundred dollars?", Response: "Is this supposed to be some kind of a joke?", Implicature: "No"

Utterance: "Do you know, how long is Uncle Arthur staying with us?", Response: "Ask your father.", Implicature: "No"

Indeed, these examples are ambiguous. Asking whether the request for a hundred dollars is a joke does not literally mean you're saying no to the request. The response 'ask your father' does not mean the speaker does not actually know, maybe they just do not want to respond. The humans in our study all infer the intended ground truth implicature. This shows a general property of implicatures; they are ambiguous, but often humans do infer the intended meaning. Ambiguity is not a discrete property. Some examples may be so vague that no one gets it. The following are examples the BIG-bench task discards that the humans in our study did struggle with:

Utterance: "Got any more of those?", Response: "Nothing I'm at liberty to reveal here.", Implicature: "Yes"

Utterance: "Have you finished sight-seeing?", Response: "Sorry. I should've come to see you first.", Implicature: "Yes"

In the first of these the implicature is 'yes' because the person responding is implying that they do have more, they just cannot reveal them. Otherwise they would most likely simply say no. In the second example it feels more natural that someone says this when they are finished sight-seeing, otherwise they would've probably said something to the effect of 'I'm still out, but I'm sorry..'. In any case, humans in our study did not understand these responses like that. This illustrates another aspect of implicature; sometimes communication will go wrong over it. Removing implicatures that are ambiguous though, defeats the purpose of the task, as they are all ambiguous to a certain degree. The purpose of this study is to compare how humans resolve this type of non-literal language

compared to how models do it. The human baseline of 86% accuracy that humans achieve on our test set deals more naturally with examples that are too ambiguous for models to understand than discarding examples based on the subjective opinion of a few people.

I.2 Overestimation of performance on implicature understanding

On the overlapping part of our test set and theirs the humans in our study achieve 92.8% accuracy. The best model on the BIG-bench task is PaLM, achieving a zero-shot performance of 64.4%. Note that this performance is on their full test set (not the overlapping part) and hence not directly comparable. Nonetheless, the missing examples are randomly sampled for our development set, and we can be pretty confident this number indicates a large gap with human performance. Two-shot PaLM comes very close to human performance with 91.7% accuracy, but of course this does not take into account the 189 more challenging examples that are part of our benchmark. Humans achieve 71.9% performance on this subset of ambiguous data, indicating that these data are more difficult than average, but nonetheless performance is higher than random. Without access to the models used to evaluate the BIG-bench task we cannot say anything for certain, but we expect the performance of PaLM to be overestimated as it does not account for a large part of the type of implicatures found in natural discourse; ambiguous ones.

I.3 Other limitations

Poor quality human evaluation . The average human evaluator on BIG-bench implicatures achieves around 82% performance (where ours achieves on average 86% on a more challenging dataset), and their human best rater achieves 100% (where our human best is 92%). This difference between human average and best hints at poor quality average rating. This is impossible to verify because there is no information in the BIG-bench tech report on how the human evaluation was done exactly, or even which examples where evaluated.

No fine-tuned models . BIG-bench uses only base LLMs and no SotA fine-tuning methods. A question that remains is therefore, what aspects of LLMs contribute to their performance on implicatures? In our work we find that implicature performance emerges at a much smaller scale in models instruction fine-tuned at the example level, and that scale and prompting techniques are important.

J Chain-of-thought completions by GPT-4

GPT-4 reaches human-level performance with CoT prompting. In this section we look at a few completions to help illuminate how CoT-prompting improves implicature resolution.

Specifically, we look at some of the examples for which GPT-4 5-shot gets 0% accuracy (i.e. wrong for all templates), and GPT-4 5-shot CoT gets 100% accuracy (i.e. right for all templates). This is a set of 10 examples. We only look at the first template, but all templates get a similar performance with CoT.

Utterance : 'Do you sell calculators?'

Response : 'We are all sold out. We wont probably have any more in for a few weeks.'

Implicature : Yes. But we dont have any calculators to sell now.

This is an example that GPT-4 gets wrong for all templates 5-shot, probably because the response has a negative sentiment, and also it means they do not sell calculators right now. However, the implicature is yes, because they do sell them normally. The CoT completion is:

CoT completion : 'Alice says they are all sold out and won't have any more in for a few weeks. Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. Alice must be implying that they do sell calculators, but they currently don't have any in stock. The answer to Bob's question is yes. Answer: yes'

This example is similar to the one given in the main text in Section 4, and there are more like these in the dataset: the sentiment is negative, and an interpretation can be negative, but strictly the implicature is positive (yes).

Utterance : 'Do you know where he is?'

Response : 'I'll find him.'

Implicature : No.

GPT-4 gets this wrong for all templates 5-shot, and right for all of them with CoT:

CoT completion : 'Alice says she will find him. Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. If Alice knew where he is, she would have provided that information directly. Instead, she says she will find him, implying that she does not currently know his location. The answer to Bob's question 'Do you know where he is?' must therefore be no. Answer: no'

It's unclear why GPT-4 gets this one wrong 5-shot (all humans get this one right), but the CoT is very relevant and helpful.

Utterance : 'It seems Cathy is not happy at all with her new room mate. Did you ask her why?'

Response : 'Believe me. I tried to find out. But I simply could not pin it down.'

Implicature : Yes. But Cathy did not give specific reasons for her feelings.

Again an example where the sentiment of the response seems to hint at no. Of 5 human annotators, also only 2 interpret this response as 'yes'. With a CoT, GPT-4 gets it right six times:

CoT completion : 'Alice says she tried to find out but couldn't pin it down. Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner. Alice must mean that she attempted to ask Cathy why she is unhappy with her new roommate, but was unable to get a clear answer. The answer to Bob's question 'Did you ask her why?' must therefore be yes. Answer: yes'

A helpful reasoning trace. All CoT completions by the models we have run CoT on are available in the GitHub: https://github.com/LauraRuis/do-pigs-fly .

K Additional results

K.1 Contrastive experiment

In this section we reframe the implicature resolution task to a contrastive one, allowing the model to contrast the coherent to the incoherent sentence in a single prompt.

Contrastive task . In the ranking task the model is required to assign higher likelihood to the coherent utterance than the incoherent one ( p θ ( x ) > p θ ( ˆ x ) ). In assigning a likelihood to x , the model has no knowledge of ˆ x , and vice-versa. We hypothesize that the task might become easier if we reformulate it as a contrastive task. Consider the following prompt p .

Which of the following sentences is coherent:

A: Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means no.

B: Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means yes.

Answer:

We can now evaluate the models' ability to understand which is the coherent sentence by evaluating whether it assigns p θ ( A | p ) > p θ ( B | p ) . Note that this can again be framed in a ranking task of assigning a higher likelihood to the coherent prompt. If we finish the above prompt p by adding 'A' to make a coherent prompt x and 'B' to make an incoherent prompt ˆ x we can again formulate the task by p θ ( x ) > p θ ( ˆ x ) . The difference is that within both the coherent and the incoherent prompt, the model can contrast the coherent and incoherent utterance to each other. We randomise the assignment of A and B to the utterances.

We do a small experiment with the contrastive task with one of the best performing models overall, OpenAI's text-davinci-002, for k = { 0 , 1 , 5 } . We use two prompt templates and for each template try three different multiple choice answers: A and B like above, one and two, or the full text of the answer. For the last option the coherent prompt x would look as follows:

Which of the following sentences is coherent:

A: Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means no.

B: Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means yes.

Answer: Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means no.

In Table 11, perhaps surprisingly, we can see that the contrastive task is much more difficult than the original ranking task. For k = 0 , the result is random except for the prompt where the multiple choice options are A and B. For k = { 1 , 5 } the full text ranking does best, but is still significantly worse than the original ranking setup. Because of these disappointing results, we did not evaluate the other models contrastively. Future work must establish whether the contrastive setup is worse across all model classes and sizes.

K.2 Variance over prompt ordering

As mentioned in Section 3 in the main text, models are sensitive to the ordering of the k examples in the prompt. Instead of marginalising over this random factor by evaluating all possible prompt orderings, we randomly sampled an ordered set of examples from the development set for each test example. Throughout experiments, we kept this randomly sampled order the same, meaning if you re-run the 5-shot evaluation you get exactly the same orderings. The reason for this is that we want evaluate each model equally. In this section we ask how the performance chances for the best performing model if we select another random order. We do this for the 5-shot evaluation, because the results show that adding more in-context examples barely helps performance.

Table 12 shows the results of this experiment. Some prompt templates seem to be more sensitive to prompt example ordering than others, but for none of them the variance is high enough to change any conclusions.

K.3 Different zero-shot instruction prompts

There is a narrative around large language models that if they fail a task, it might be that the prompt was not the right one (through works like Reynolds and McDonell (2021b); Kojima et al. (2022)). The idea is that they can be prompted to simulate almost anything, if you set them up correctly. Because implicature resolution is a ubiquitous result of learning language, we hold the view that a model should be able to do this task if a prompt is given in coherent natural language. Nonetheless, in an additional effort to find the 'let's think step-by-step' (Kojima et al., 2022) of zero-shot implicature resolution we try three more prompt templates. Table 13: Zero-shot accuracy over three additional prompt templates for a base LLM and two instructable models. Model Templates GPT-3-175b 59.2% ± 4.5 text-davinci-001-? 66.1% ± 3.2 text-davinci-002-? 67.7% ± 9.6 We evaluate a base large language model and two instructable models: GPT-3-175B, text-davinci-001, and textdavinci-002. The prompts we use are taken from recent work that proposes a dialogue agent trained with human feedback (Glaese et al., 2022), but adapted to the task of implicature resolution. The full prompts are presented in Table 7 and Table 13 shows the results. The new templates do not improve performance for any of these models. The variance over the prompt templates for text-davinci-002 is

high, and the best prompt template of these three does achieve a slightly higher accuracy than the others: 74.5%. These results do not change the picture sketched so far.

K.4 The effect of in-context examples on sensitivity to prompt wording

Figure 7 shows the relative performance increase due to in-context prompting broken down per prompt template. For text-davinci-001, most templates benefit similarly from more in-context examples, except for template 1. Perhaps surprisingly, we see that this template already achieves a performance of 76.5% at the zero-shot evaluation and does not improve much with few-shot prompting. For Cohere-52B and OPT-175B we see a clear grouping between the structured prompts (dashed lines) and natural prompts (dotted lines). Cohere struggles significantly more with the structured prompts than with the natural prompts in the zero-shot evaluation, and few-shot prompting can mitigate that, lowering the standard deviation over prompt templates to 1.89 at k = 30 from 4 at k = 0 . OPT benefits from prompting for the natural prompts, but not for the structured prompts.

Relative accuracy(w.r.t. O-shot) due to in-context examples for all prompt templates.

20

InstructGPT-3-175B

Cohere-52B

OPT-175B

Prompttemplate1

Prompttemplate2

15

Prompt template 3

（%)

Prompttemplate4

Relative Accuracy

Prompt template 5

10

Prompttemplate6

5

0

-10

01

5

10

15

30

01

5

10

15

30

01

5

10

15

30

k-shot

k-shot

k-shot

K.5 Variance over API runs

In this section we comment on the reproducibility of research done using APIs. OpenAI and Cohere have their models behind an API, meaning we do not have control over what happens to the prompt before the model processes it. We run the zero-shot evaluation ten more times for two models of OpenAI and Cohere, text-davinci-002 and Cohere-52B. The results from this experiment are shown in Table 14 and 15. From this we can conclude that there is some stochasticity in the API that we have no control over, a bit more for OpenAI than for Cohere, but again we can be relatively confident that the conclusion will not be different because of it. The results from this work are therefore reproducible with access to the same models behind the API now. Unfortunately, when OpenAI or Cohere changes the models behind the API, these results are not exactly reproducible anymore.

For completeness, we add the timestamp that each result was obtained below (Appendix L).

Each evaluation has exactly the same text, so the variance in performance is due to API stochasticity.

K.6 Experiment with random in-context labels

This paper presents the thesis that instruction-tuning at the example level ('Example IT') is important for pragmatic understanding in LLMs. However, the 0-shot result that one of the models in the Example IT group achieves is similar to that of base models; Cohere-command-52b obtains a zeroshot performance of 60.2%. From the sharp rise in performance observed for the k = 0 to k = 1 result (from 60.2% to 72.8%) we hypothesise that the k-shot in-context examples in this task do not necessarily teach the model pragmatics in-context, but prime the model for the task format (namely,

Each evaluation has exactly the same text, so the variance in performance is due to API stochasticity.

outputting either 'yes' or 'no' as detailed in Section 3 in the main text). If this hypothesis is true, we would observe similar performance regardless of whether the labels given in the prompt for the few-shot examples are true. We test this empirically for two base models (GPT-3, Cohere-52b) and two Example IT models (text-davinci-001, Cohere-command-52b) for 1-shot and 5-shot evaluation. The results can be found in Table 16. We find that for the Example IT models in-context prompts with random labels obtain the same results (i.e. within confidence intervals) as the experiments with ground-truth labels in the in-context examples. For base models however we do observe a drop in performance; for GPT-3-175b at 5-shot, and Cohere-52b both at 1- and 5-shot. Taken together, we can conclude that for base models the content of the in-context prompt seems important, whereas for models in the example IT group the in-context examples mainly serve as a primer for the task structure.

K.7 Chain-of-thought on base models

In the main paper we do a CoT experiment on the models in the Example IT group. Base models also benefit from in-context examples, so it makes sense to also try CoT prompting on these models. After attempting this for two of the model classes in the group, we decided not to apply this prompting technique to the other models, because it decreases performance, sometimes significantly. See the results of the CoT experiment on the two base model classes in Table 17

K.8 Testing for spurious correlations

In this section, we do a small scale experiment to test whether the benchmark has spurious correlations. Specifically, we run the benchmark with only the utterance or only the response as input. Strictly, getting the implicature right from the response only does not always indicate spurious correlations, as some examples only need the response (e.g. rhetorical questions like 'do pigs fly?'). Utterance-only results do always indicate spurious correlations. We run this experiment for GPT-3.5-turbo and GPT-4 0-shot and 5-shot (see Table 18 and Table 19).

We find that models mostly perform random for utterance-only, so spurious correlations do not seem to be an issue. For response-only, GPT-4 5-shot gets 65% accuracy. Some examples it gets right are: 'do fish swim?' and 'let's hope so'.

K.9 Detailed results type label analysis

In the main paper we do an analysis of two types of examples that occur frequently in the dataset, namely generalised and particularised implicatures. Here, we detail the full taxonomy of types of examples occurring in the dataset and report detailed results for each type per model (see 21 until Table 34 below). In Table 20 the full taxonomy of the examples is shown, representing types of examples that occur frequently in the dataset. We manually labeled 217 examples of the 600 examples in the test set according to this taxonomy. The remaining 383 examples do not fall as clearly within a category and are grouped together as type other . Generalised implicatures require little or no context to be understood. They are the simplest type of example in the test set, and generally imply the same thing ('some' almost always implies 'not all'). Particularised implicatures, by contrast, do require context to be resolved. For example, from Table 20, we need the context that it is undesirable to stay up late drinking when one has to get up early (see in Appendix D more on generalised vs. particularised). The type world knowledge requires knowledge of the physical world to be resolved. From the example in Table 20; we need to know that you cannot leave fingerprints when wearing gloves to resolve this implicature. Idiom types contain an idiom or a metaphor that one needs to know or understand to resolve the implicature, and finally Rhetorical question types contain a question like 'Is the Pope Catholic?', often requiring factual knowledge to be resolved.

The following tables contain the detailed results broken down per example type: Table 21 - Table 34. The most interesting pattern in this data is that for almost all models, even the best model (GPT-4 30-shot in Table 32), there is a significant gap between human-level performance on the particularised examples. This gap is larger than the gap for the other labels usually. Few-shot prompting can often mitigate this (e.g. for GPT-3-175b, Cohere-52b, and text-davinci-002), but not always (e.g. for GPT-4 the gap remains large for k = 30 ). However, for GPT-4, chain-of-thought can mitigate the gap as seen in Table 34. Where GPT-4 30-shot obtains 71.97% accuracy on the particularised examples (and humans 83.18%), GPT-4 with 5-shot CoT achieves 81.63%, which is close to human-level. We find that the particularised examples mostly benefit from CoT prompting. Namely, for the generalised type of examples, GPT-4 30-shot already achieves 86.23% accuracy and CoT improves this to 88.66%, which is a much smaller improvement than for the particularised examples.

K.10 Detailed results per model

This section contains the results used for the zero-shot and few-shot evaluation in the main text in Section 4, broken down per prompt template. See Table 35 until Table 84.

L Timestamps API calls

For reproducibility purposes, Table 88, 89, and 90 contain the dates and times the APIs from OpenAI and Cohere were queried for the results.

M Compute and Emissions

Find below in Table 91 until Table 94 the timestamps, durations, and emissions per experiment (calculated with the CodeCarbon library in Python). Find below in Table 95 until Table 98 the cpu-type and count and gpu-type and count per experiment. In terms of compute the following GPU hours can be estimated if we assume each run is entirely done on the GPU (which is not true in reality, but worst case):

NVIDIA A100-SXM4-40GB used for 926.4291392151515 hours.

Tesla V100-PCIE-32GB used for 29.282544113265143 hours.

Tesla V100-PCIE-16GB used for 11.462701331244574 hours.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a task to evaluate the ability of large language models (LLMs) to resolve conversational implicatures and go beyond the literal interpretation of the meaning of the language. The evaluation is based on a dataset of naturally occurring implicatures that are converted into a binary classification task using a set of templates. For instance, the dialogue _\"Can you come to my party on Friday?”_ , _\"I have to work\"_ is converted into two sentences: _Esther asked “Can you come to my party on Friday?” and Juan responded “I have to work” which means no_ and _Esther asked “Can you come to my party on Friday?” and Juan responded “I have to work”, which means yes._. The two sentences are inputed in the different models, considering the one maximizing the likelihood as the model choice.  \n\nFour categories of state of the art models are evaluated on the proposed task. The evaluation protocol takes into account potential variance induced by prompt templates. Different methodologies are also benchmarked to try to improve the performance of the models on the task, this includes:\n- Fine tuning on the task. \n- Instruction tuning-;\n- Chain of thought prompting\n\nResults demonstrate that the task is challenging for base models (BERT, RoBERTa, GPT2, GPT3, ...) with a performance slightly above random. Without any fine tuning, some models already reach a performance above 70% (text-davinci-00x, ChatGPT), with GPT4 only 5 points behind human performance. The improvement induced by instruction tuning and chain of thought prompting is further demonstrated for some models with GPT4 reaching human performance.\n\nSTRENGTHS:\nThe paper is well written with all the material provided to reproduce the results. The paper introduces an original evaluation protocol for the task of resolving implicatures and demonstrates that human performance can be reached by combining state of the art models (GPT-4) and the recently proposed chain-of-thought prompting.\n\nWEAKNESSES:\nThe task proposed is not new and the dataset proposed is the same as the one introduced in [BigBench](https://arxiv.org/abs/2206.04615). The results, in the mentioned paper, already demonstrated the ability of PaLM with k-shot-prompting to [perform](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/implicatures) above average human performance. The main differences between the two papers include the choice to keep ambiguous cases for the task and adding other models along with a chain of thoughts prompting to the benchmark. \nIn the benchmark, the variance in the performance of the models, displayed in Table 2, only takes into account the variability induced by different prompt wording. To compare the models, it would have been helpful to perform bootstrap on the test set to estimate to support the claim on the comparison of the performance of the different models.\n\nQUESTIONS:\nWhy has no fine tuning on the base models been performed? \n\nDo you have the increase in performance induced by chain of thoughts prompting for GPT4 for particularized (context heavy) examples?\n\nl107 _\"Our analysis is novel in its approach; using ambiguous data that humans can easily resolve, and scope\"_ Maybe reformulate, not very clear\n\nGPT4 doesn't seem to benefit from in context example. How do you explain that?\n\nDid you try to isolate the effect of data contamination? Especially for GPT-4, the model could have been trained on the data used in your evaluation protocol given that the original dataset was published in 2020\n\nLIMITATIONS:\nThe proposed evaluation framework focuses on conversational implicature. If implicature is key for natural language understanding, evaluating models on conventional implicature is also important. If outside of the scope of this well conducted evaluation, it could be of interest for future work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper evaluates several popular LLMs on a benchmark for evaluating pragmatic reasoning via resolving question-answer conversation snippets into the implied binary answer. The study explores the influence of different factors in LLM design, including training method (next-token prediction vs. instruction tuning), number of few-shot examples, model size, inference method, etc. Several conclusions are made about the results: instruction tuning seems to improve pragmatic reasoning, models are mostly robust to perturbations in the input prompts, scaling model size seems to have an effect in improving performance, CoT prompting is useful for some models, and remaining failures of the model appear to center on more difficult (particularised) implicatures.\n\nSTRENGTHS:\n* The paper offers a very comprehensive / thorough / careful evaluation of the LLMs being evaluated. Particularly, the try a number of different prompt variations, try different numbers of few-shot examples (as well as having a randomization process for choosing these examples, to avoid confounding results with some fixed set of examples)\n* The finding that few-shot examples are useful mostly for conveying format of the response, rather than as a way of defining the task itself, is very interesting. I would be interested to see a more in-depth study of this (perhaps another paper) as it is somewhat surprising (but perhaps less surprising for instruction-tuned models).\n\nWEAKNESSES:\nIn general, I wanted to see a bit more discussion and analysis, since this paper is focused on analysis.\n* Moving the analysis for other types of implicatures into the main paper\n* More examples of what the model is generating, particularly for CoT examples\n* More discussion / analysis of human \"errors\" here -- why might there be errors? Genuine noise in the annotation process, or different interpretations of the same example?\n* Discussion on why instruction tuning might be influential in improving pragmatic reasoning\n* Discussion / analysis on why there is still a gap for particularised implicatures, and what we can do to address it\n\nNits:\n* The first example (GPT's answer to the user's question about the phone) actually doesn't necessarily make sense, as it's definitely not literally true -- GPT can't have seen the user's phone\n* \"fine-tuning on instructions at the example-level\" is vague to me, particularly the \"example level\" part. What is this supposed to refer to? Is there a distinction between general \"instruction-tuning\" and \"instruction-tuning ad the example level\"?\n* Also, \"context\" in Insight 5 is vague -- I wouldn't think of this kind of commonsense/world knowledge as \"context\" necessarily, so I'd suggest renaming this\n* Formatting of Figure 2 is somewhat odd\n* I'd suggest adding a line for \"human performance\" in Figure 2\n* I'd suggest adding the few-shot CoT example (or some examples of generated CoT answers) into the main paper\n\nQUESTIONS:\n* Is there evaluation on this dataset with respect to an upper bound in performance given only the response, without the question? I'm curious whether answers like \"I've gotta get up early\" or \"Some\" have some spurious correlations with the labels, e.g., \"no\", in this dataset. \n* Is the accuracy actually decreasing relative to k in Figure 4? I.e. is this difference significant? If so, why might that be?\n\nMinor questions:\n* Are there cases where examples are ambiguous, and situational context might affect the label? Did you investigate cases where the human annotators were \"wrong\" to identify the source of those \"errors\"?\n\nLIMITATIONS:\nYes, mostly. More discussion on these non-binary implicatures (at the end of the discussion) would be interesting. I'd also be interested if the authors can perform analysis on the existing benchmark to see whether the labels can be predicted with only one half of the example (i.e., the response); i.e. whether there are spurious correlations within the benchmark.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyze the behavior of LLM pragmatic understanding (capability to add and omit imformation to efficiently communicate in context) using a new evaluation protocol. They compare LLM performance to human performance and argue that instruction fine-tuning with examples may improve pragmatic understanding.\n\nThey propose using the assignment of higher likelihood to coherent utterances than similar but incoherent utterances as \"resolving an implicature correctly.\" This includes both describing a situation where implicature understanding is necessary to essentially resolve an entailment question, whether an answer that on its face is irrelevant means \"yes\" or \"no,\" or whether its a sufficient answer. For example, `Esther asked “Can you come to my party on Friday?” and Juan responded “I have to work”, which means yes.` is a failure case because the answer is wrong; the response means no. A model preffering `...which means yes.` resolves the implicature.\n\nThey assess this problem is few and zero shot settings for LLMs. They control for prompt variation effects with 6 different prompt settings, testing a fairly comprehensive set of both generative proprietary LLMs and standard  open pretrained transformers like RoBERTa. They find no model can beat human performance on their resolution task, but GPT-4 comes closest.\n\nThey conclude with a detailed analysis, with 5 insights.\n\nSTRENGTHS:\nWell-defined and scoped problem. Out-of-community definitions (eg, explaining pragmatics) well-motivated. Citations are sufficiently comprehensive in my opinion.\n\nExciting direction to explore to further understnading and engineering of LLMs.\n\nFully-specified approach and sufficiently broad evaluation.\n\nClear statement of insights.\n\nOveral, I think this paper has clear value to the community.\n\nWEAKNESSES:\nI think it would be possible to use more than 6 \"curated prompt templates\" (163-172). The \"we control for prompt variability\" is really not central to the story of the paper but I don't really find the idea that averaging in essence over 6 prompts is sufficient to argue that that has happened. Perhaps claim of robustness should be weakened.\n\n~They claim task-specific instruction tuning improves implicature resolution, but there's no concrete A/B test for this. They don't instruction fine-tune a model themselves for the task (not that doing so would necessarily be possible given constraints). How do we know that IFT is really driving GPT4's superior performance and not just scale or some other factor? I find the evidence presented here unconvincing~\n\nEDIT: In light of the author's response, I do think the case for IFT driving the improvement to be a bit more convincingly made, or at least as convincingly made as can be given the inherent constraints (OpenAI's closedness for ex). I hope to see the authors response to this point integrated in the camera ready, with maybe some weakening of claims in light of the incomplete degree of knowability of this issue\n\nQUESTIONS:\nMaybe respond to the questions I raised implicitly in the weaknesses? I am open to being convinced.\n\nLIMITATIONS:\nI find limitations to be sufficiently addressed this time.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how well recent LLMs preform on resolving conversational implicatures. It presents a task on conversational implicature resolution built on top of the crowdsourced and human-annotated dataset of George and Mamidi (2020). The work presents an evaluation protocol that lays out how LLMs are evaluated on implicature resolution. The work further presents an array of experiments on various LLMs to understand whether they do well on this task and in what settings (e.g. 0-shot vs 5-shot). Results are analyzed in the form of insights and main takeaways that the authors highlight.\n\nSTRENGTHS:\n-\tAddresses an important problem, namely studying the competence of LLMs at pragmatic inferences—here specifically implicatures\n-\tExtensive experiments on a wide range of recent LLMs to validate how well these LLMs perform pragmatic inferences involving implicatures\n\nWEAKNESSES:\n-\tThe work is poorly situated with respect to prior work on computational modeling of pragmatics and the related work section misses an important line of work which I discuss in the comments.\n-\tFurther analysis is required in certain cases (see comments below)\n\nQUESTIONS:\nIntro:\n\nLine 28: “utterances that convey something other than their literal meaning”\nAt the first pass of this sentence and reading “other”, I read it as suggesting implicature conveys something else different/unrelated to the literal meaning, and I had to reread it to fit it within the definition of implicature. I suggest replacing “other” with “beyond” as I feel that fits better the description of implicature.\n\n\nSection 2:\n\nAs a work that is, in essence, motivated by investigating how good LLMs are at “interpreting language in context—incorporating its pragmatics”, I feel the related work section leaves out an important line of work on the computational modeling of pragmatic phenomena including implicature and the closely related phenomenon of presupposition and on understanding the limitations of modern NLP models in capturing these phenomena starting with the work of Cianflone et al 2018 on modeling adverbial presupposition, Schuster et al 2020 & Li et al 2021 on scalar implicature, the ImpPress dataset (Jeretic et al 2020) on presupposition and implicature in the context of NLI and related experiments on how good NLI models (including BERT-based models) are at capturing these phenomena, Kim et al 2021 on presupposition verifiability in the context of QA, Parrish et al 2021 with the NOPE presupposition corpus, Kabbara et al 2022 on the limitations of NLI models in resolving presupposition cases. These first come to mind. Despite these efforts focusing on (what is now) smaller LMs models (with the exception of the first mentioned work which precedes the BERT era), I believe they are important efforts that set the stage for the present work and need to be cited and imo are more relevant to the work than some of the information mentioned in the first paragraph of Section 2 (e.g., how some models are toxic, unhelpful, etc). Appendix D has a nice background on implicature from a linguistics/philosophy of language but still misses all of the work I mentioned here on modern computational modeling of pragmatic phenomena.\n\nSection 3:\n\nA general concern of mine is the benchmark contribution of the paper given that the same dataset has been introduced in BIG-bench. The authors are clear about how their dataset is different given that it encompasses instances that are discarded in BIG-bench (for being too “ambiguous”) but regardless the overlap between the two is obviously substantial and so I felt this is something that could benefit from a discussion here. \n\nSection 4:\n\nRegarding the random label experiment and looking at the related section in the appendix (I.6), two points/questions come to mind:\n1)\tGiven that there are two labels in question, did you verify how often the label was not changed? That could eclipse the results we’re seeing here. \n2)\tThis experiment and the close results in some cases (for 5-shot text-davinci-001 and cohere-command-52b) brings up the following question: to what extent the models are learning any notion of pragmatics here? Did you carry out some qualitative analysis in a systematic way to see whether the models are in some cases getting it right for the wrong reasons? For example, Kabbara el al 2022 showed that for presupposition-based NLI, RoBERTa and BERT were getting high accuracy performance on certain presupposition types but upon further investigation and adversarial testing, it turns out the models are often exploiting superficial cues that are not related to notions of pragmatics. I wonder to what extent this could be applicable for the task you’re exploring here.\n\nRegarding the analysis in Appendix I.4 (varying k in-context examples), apart from the conclusion that is in the body of the paper (Section 4, Insight 2), I somewhat feel the results are so noisy for Cohere-52B and OPT-175B that it’s hard to really make sense of the results. InstructGPT3-175B is the only case where the results follow our intuition: the more examples a model sees, the better it gets at understanding the notion of implicature and so the higher the accuracy. For Cohere, we see a very noisy trajectory across the board between k = 1 and 15 (before the results start stabilizing and moving upwards) and for OPT, the results are mostly on a downward trajectory except for a chunk in the middle and end up still going downward for k>15. Would like to see if you have any thoughts on this.\n\nTable 3: Regarding the 5-shot CoT performance \n\nWe see a variance between the text-davinci models. Generally, although specific details are not public, we can assume models got better as newer versions were released. So it makes sense to see the performance improve for the 5-shot and 5-shot COT scenarios across the 3 davinci-models. However, it’s still unclear why CoT would help marginally in one case (002) and in a rather noticeable way (003) but hurt the performance in the 001 case. Any insight there? This is even more pronounced when the drop is not trivial (7%) and is the only case among 6 (if we were to count the -0.1% as roughly not helping but not hurting)\n\nAppendix G: Human evaluation\n\nI’m somewhat confused by the choice of the authors to run human experiments using on prompt only. They could’ve simply chosen to run multiple experiments with at least a couple of different prompts. The authors justify this by the fact that humans are less likely (than models) to be sensitive to variations in the prompt. I feel this is at best speculative and, if anything, human experiments and pilot studies often show that human subjects can be primed to think or react in a certain way given certain wording or structure, etc. \n\nMinor presentation note:\n\nLine 25: I’m not sure if there’s a writing style guide that allows this form “I have to work.”. [essentially the two dots back-to-back]. Seems rather odd to me. I would think the one within the quotations need to be dropped. See here for more details:\nhttps://www.hamilton.edu/academics/centers/writing/style/essentials/punctuation-of-quotations#:~:text=The%20final%20period%20or%20comma,the%20period%20follows%20the%20citation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-08-01"}