SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Insight 2: The results are robust to different prompt templates. As detailed in Section 3, each example in the test set is wrapped in six different prompt templates. The standard deviation in Table 1 and in Figure 2 shows the sensitivity to different prompt wording. The standard deviation ranges from 0.3 for BlenderBot to 7.0 for T0-11B. All in all, the sensitivity to prompt wording does not seem to be a problem for this task; when taking into account the confidence intervals the result remains that models in the group Example IT perform significantly better than all other models, but worse than humans. In Appendix K.4 another analysis is presented that shows how different prompt templates benefit from in-context examples. The takeaway from the analysis is that in-context prompting can mitigate the fact that some models are better at natural prompts and others better at structured prompts by improving performance on the type of prompt the model struggles with zero-shot. Again, when
Figure 2: The few-shot accuracy for the best model of each class (e.g. the best performing model in the class Cohere-command is the 52b model, whereas the best model in the class OPT is the 13b model). The bars show the group means. Models fine-tuned on example-level instructions perform better than most other models, especially for  $k > 0$ . For all models there is a significant gap between best accuracy and human accuracy (which is  $86.2\%$ ). * means size unknown.
Table 2: Scaling results for OpenAI's text-<engine>-001-series, for which we do not know the number of non-embedding parameters but do know the ordering in terms of size. The colors indicate whether going up in size (from left-to-right) increases performance significantly or not.
Engine	Ada	Babbage	Curie	Davinci
0-shot	56.5% ± 5.8	64.5% ± 1.8 (+8.0%)	69.0% ± 2.9 (+4.5%)	72.3% ± 2.8 (+3.3%)
5-shot	57.6% ± 2.8	66.1% ± 0.3 (+8.5%)	71.3% ± 1.3 (+5.2%)	74.5% ± 1.0 (+4.0%)
only looking at the best prompt type for each model class (i.e. structured or natural), the results remain that models in the group Example IT perform best.
Insight 3: Models instruction-tuned at the example-level have the most favourable scaling properties, but some base models also show positive correlation with scale. Figure 3 shows the scaling behaviour of the model classes for which we know the number of non-embedding parameters. We highlight 0- and 5-shot results, because for  $k > 5$  the accuracy of most models plateaus (see Figure 2). However, detailed results for other  $k$  can be found in Appendix K.10. Note that we do not know the number of parameters for OpenAI's 'text<-engine>-001'-series, but we do know the order of the engines in size, and we separately present its scaling results in Table 2. Except OpenAI's 'text<-engine>-001'-series, none of the models show significant performance increase with model size for the 0-shot evaluations. However, for  $k$ -shot evaluations with  $k \geq 1$  we observe significant positive correlation with size for the models in the Example IT class for which we have multiple sizes (Cohere-command and 'text<-engine>-001') as well as some models in the base model class. Not only do the models in the Example IT class exhibit higher performance for the same model size, these models also have a steeper performance increase with size than the base models. Comparing the scaling properties of the best base model (GPT-3) with Cohere-command, we see that the increase in performance from the second-largest to the largest model is  $0.04\%$  per billion parameters from GPT-3-6.7B to GPT-3-175B and  $0.15\%$  per billion parameters for Cohere-command-6B to Cohere-command-52b (exact numbers used to calculate the slope can be found in Appendix K.10). If performance is linearly extrapolated from this curve GPT-3 reaches human-level performance at 642b parameters where Cohere-command would need 125b parameters.
Insight 4: GPT-4 reaches average human-level performance with chain-of-thought prompting. For the model groups that benefit from in-context examples, we attempt to push performance further
Figure 3: Scaling results for the model classes of which we know the number of non-embedding parameters. The error bars show standard deviation over prompt templates. Cohre's command models instruction-tuned at the example-level perform better than all other models. For all models there is still a significant gap between best accuracy and human accuracy.
Table 3: Results of the chain-of-thought (CoT) experiment for models in the group Example IT. The numbers between brackets show the difference in performance with the number on the same row one column to the left. Most models benefit from CoT-prompting, but not all. Additionally, GPT-4 reaches average human-level performance with CoT prompting.  $\star$  means size unknown.
Model	0-shot	5-shot	5-shot CoT
text-davinci-001-★	72.3% ± 2.8	74.5% ± 1.0 (+2.2%)	67.3% ± 2.6 (-7.2%)
text-davinci-002-★	70.6% ± 2.3	79.6% ± 2.0 (+9.0%)	80.1% ± 0.8 (+0.5%)
text-davinci-003-★	71.2% ± 2.8	79.7% ± 0.6 (+8.5%)	83.6% ± 0.6 (+4.0%)
ChatGPT-★	72.1% ± 6.0	73.9% ± 6.3 (+1.8%)	77.2% ± 1.0 (+3.3%)
GPT-4-★	81.8% ± 1.8	82.0% ± 1.7 (+0.2%)	86.5% ± 1.0 (+4.5%)
Cohere-command-52b	60.2% ± 5.2	75.4% ± 1.8 (+15.2%)	75.3% ± 0.5 (-0.1%)
with chain-of-thought prompting. We manually write a five-shot chain-of-thought prompt for all six prompt templates, and evaluate model performance using this prompt. One of the six chain-of-thought prompts can be found in Table 4 in Appendix F, and the other five are provided in the supplementary material. We only present the results for the group Example IT here, since CoT prompting did not improve performance for two of the base model classes we tried (see Appendix K.7). Consequently, we decided not to apply this experiment to the other models in the base group to save compute costs. The results of are shown in Table 3. We find that chain-of-thought prompting does not help for all models, but is nonetheless able to boost performance of GPT-4 to  $86.5\% \pm 1.0$ . This is on-par with average human-level performance, and slightly below human best performance at  $89.8\%$ . To illustrate how explicit reasoning helps implicature understanding, we highlight a CoT generated by GPT-4 for an example from the dataset that models persistently get wrong. "A: Is there a bus I can get to the station? B: You can't rely on it". The implicature is yes, there is a bus, you just cannot rely on it. GPT-4 five-shot gets this wrong for all six templates. With CoT it gets it right for five of six templates. The generated CoT for one template is the following:
Alice says 'You can't rely on it.' Alice must be implying that there is a bus, but it may not be dependable or timely. This means the response to Bob's question is yes, but with a caution about reliability. Answer: yes
More completions can be found in Appendix J.
Figure 4: The accuracy v. k for the generalised and particularised examples obtained by the Example IT models Cohere-command and GPT-4. Particularised (context-heavy) examples are often significantly more difficult than generalised (context-free) examples for both models and humans. For most models, in-context prompting can mitigate this, but for others (like GPT-4), a significant gap remains. We see that Cohere-command-52b achieves similar performance as GPT-4 on the particularised examples, but significantly lower on the generalised examples.
Table 4: An example from the dataset for two types of implicature found in the test set. The rightmost column shows the amount of that type we manually found in the test set.
Type	Example Utterance	Example Response	Impl.	#
Generalised	You know all these people?	Some.	No.	47
Particularised	Want to stay for a nightcap?	I’ve gotta get up early.	No.	94
Insight 5: Models often struggle with the same type of examples humans struggle with. We manually labeled 217 examples of the 600 examples in the test set according to a taxonomy. The remaining 383 examples do not fall as clearly within a category and are grouped together as type other. In Table 4 the two types of examples that occur frequently in the dataset are exemplified. Generalised implicatures require little or no context to be understood. They are the simplest type of example in the test set, and generally imply the same thing ("some" almost always implies "not all"). Particularised implicatures, by contrast, do require context to be resolved. For example, from Table 4, we need the context that it is undesirable to stay up late drinking when one has to get up early (see in Appendix E for more on generalised vs. particularised). In these type of examples, the context needed to resolve it is different every time. We label three other types of implicatures in the dataset, but since the analysis of these examples does not show significant patterns, we present it in Appendix K.9. We show the accuracy broken down per example type for two models from the Example IT group, as these patterns hold more broadly for almost all models evaluated (see the detailed results broken down per example type in Appendix K.9). Figure 4 shows that for lower  $k$ , the models often have a significantly worse performance for particularised examples than for generalised examples, just like humans do. For some, like Cohere-command-52b, this is mitigated by few-shot prompting, which brings particularised and generalised performance closer together (sometimes at the cost of generalised performance). For others, like GPT-4, the gap between particularised and generalised performance remains large for all  $k$ . From the bottom row in Figure 4 we observe that the edge GPT-4 has over Cohere-command-52b seems mostly driven by a higher accuracy on generalised examples. The accuracy on the particularised examples is comparable between those two models.
In this study we use prompting to evaluate whether different groups of LLMs can resolve implicatures. In designing our experimental protocol, we carefully considered various alternatives, and here we discuss limitations of the chosen approach. Firstly, evaluating LLM competencies is inherently uncertain and sensitive to prompt choice. Nonetheless, we are confident our evaluation is comprehensive enough to assess implicature understanding: we apply six different prompt templates per test example, each used in three different prompting techniques (zero-shot, few-shot, chain-of-thought). Addition-
LLMs have made remarkable progress on fluency and coherence in recent years. We argue however that a central aspect of language understanding is missing from evaluations. To understand language means to understand its pragmatics: its usage in a context that incorporates commonsense understanding, goals, objectives, and so on. We design a protocol that evaluates LLMs on binary implicature resolution and establish a significant gap with human understanding for SotA LLMs in three categories; large-scale pre-trained models, models fine-tuned on conversations, and models fine-tuned with benchmark-level instructions. By contrast, we find that models fine-tuned on example-level instructions perform significantly better. This group also exhibits the best correlation between accuracy and model size. Scaling analysis shows that for some large-scale pre-trained models accuracy also positively correlates with model size, but the best model in this group would need at least five times more parameters to reach similar performance. From these results, we conclude that instruction-tuning at the example level is important for pragmatic understanding. We hypothesise that there is something about the multi-task data diversity obtained from example-level instructions (i.e. each example a new task) that makes pragmatic understanding appear at smaller scale.
We would like to thank members of the UCL DARK lab, members of the UCL NLP group, Pontus Stenetorp, Sebastian Borgeaud, Philipp Jettkant, Robert Kirk, and Max Bartolo for fruitful discussions and comments on an earlier draft of this paper. We would also like to thank the anonymous reviewers for engaging actively in discussion and providing feedback that has improved this work. This work was supported by the EPSRC Grant EP/S021566/1 and UCL International Scholar Award for Doctoral Training Centres.
Tan, Z., Zhang, X., Wang, S., and Liu, Y. (2021). Msp: Multi-stage prompting for making pre-trained language models better translators. arXiv preprint arXiv:2110.06609.
Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C., Krivokon, I., Rusch, W., Pickett, M., Meier-Hellstern, K. S., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M.,
Kuzmina, V., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E. H., and Le, Q. (2022). Lamda: Language models for dialog applications. CoRR, abs/2201.08239.
Volden, J. (2017). Autism Spectrum Disorder, pages 59-83. Springer International Publishing, Cham.
Wang, B. and Komatsuzaki, A. (2021). GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
Webson, A. and Pavlick, E. (2021). Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247.
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. (2022). Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Wittgenstein, L. (1921). Tractatus logico-philosophicus. London: Routledge, 1981.
Wittgenstein, L. (1953). Philosophical Investigations. Basil Blackwell, Oxford.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. (2022). Opt: Open pre-trained transformer language models.
Zheng, Z., Qiu, S., Fan, L., Zhu, Y., and Zhu, S.-C. (2021). GRICE: A grammar-based dataset for recovering implicature and conversational rEasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2074-2085, Online. Association for Computational Linguistics.
A Contributions 18
B Reproducibility Statement 18
C Ethics Statement 18
D Opener example with InstructGPT 19
E Background on implicature 20
F Detailed prompt templates 22
G Model categorization 26
H Human evaluation 27
I Comparison with BIG-bench implicatures task 29
I.1 Discarding ambiguous examples 29
I.2 Overestimation of performance on implicature understanding 30
I.3 Other limitations 30
J Chain-of-thought completions by GPT-4 31
K Additional results 32
K.1Contrastive experiment 32
K.2 Variance over prompt ordering 33
K.3 Different zero-shot instruction prompts 33
K.4 The effect of in-context examples on sensitivity to prompt wording 33
K.5 Variance over API runs 34
K.6 Experiment with random in-context labels 34
K.7 Chain-of-thought on base models 36
K.8 Testing for spurious correlations 36
K.9 Detailed results type label analysis 37
K.10 Detailed results per model 37
L Timestamps API calls 69
M Compute and Emissions 72
Laura Ruis: project proposal and leadership, dataset development, code writing, human experiment, manual error analysis, paper writing and editing.
Akbir Khan: code writing, model evaluations, human experiment, paper writing and editing.
Stella Biderman: model evaluations, compute usage, paper writing and editing, advisor.
Sara Hooker: compute usage, paper writing and editing, advisor.
Tim Rocktaschel: paper writing and editing, advisor.
Edward Grefenstette: initial idea, manual error analysis, paper writing and editing, advisor.
We share all the data, human annotations, code used for the evaluations, and the raw results in the supplementary material. Additionally, in Appendix K.5 we estimate the variance due to stochasticity in the API's of OpenAI and Cohere. Of course, if either OpenAI or Cohere decides to change the models behind the API, the results might look different. We publish the exact date and time each API was queried for the results in Appendix L. Finally, in Appendix K.2 we estimate the variance over the prompt order of the in-context examples. The compute used for the experiments is detailed in Appendix M. The remaining experiments were done with API credits received as a research grant from OpenAI and Cohere.
In this work, we conduct a study with human subjects (see Appendix H for details). To get matched with participants, we used the platform Prolific. Prolific complies with ethical standards according to UK law (e.g. complying with the GDPR). We compensated participants with a UK living wage at 15 GBP an hour, which is 6 GBP an hour more than Prolific recommends at 9 GBP per hour.
Implicature is an aspect of pragmatics, and pragmatic language impairments are universal in Autism Spectrum Disorder (ASD) (American Psychiatric Association, 2013). Difficulties in understanding scalar implicatures are claimed to be present in people with ASD (Volden, 2017), although the nature of the relation has proven hard to establish and has recently been debated (Katsos et al., 2011; Schaeken et al., 2018). For the purposes of this work, whether or not implicature understanding relates to ASD is not important. We took the following steps to make sure no sensitive data is collected or published. The human annotations we obtain are anonymous, related to a participant only by their Prolific ID for the purposes of compensation. In publishing the human annotations, we will not publish the Prolific ID of participants or anything else related to the participants. Additionally, we did not collect or request any personal or demographic characteristics of the participants apart from that they are all native English speakers.
Additionally, in this work we run a lot of compute-intensive experiments. We publish the estimated emissions per experiment in Appendix M. The total amount of GPU hours is estimated at maximally 966. How this is broken down per experiment can be seen in Appendix M.
D Opener example with InstructGPT
This quote was obtained through the OpenAI playground for text-davinci-002 on May 15th 2023. The model text-davinci-001 consistently generates better responses for the same prompt. GPT-3 itself (i.e. davinci) mainly gives nonsensical answers. In the following, the prompt is italic and the completion bold. The completion was generated with a maximum of 10 tokens and a temperature of 0:
The opener example is used to introduce the problem we are looking at, and not to judge the model used to generate it. In fact, although text-davinci-002 sometimes completes conversations in a way that is unexpected according to pragmatic language usage, it is one of the better models when evaluated few-shot.
E Background on implicature
The first influential consideration of implicature is Grice (1975). In his work, Grice continues the trend of moving away from purely logical accounts of language started by Wittgenstein (1921) by hypothesising implicatures arise in conversation when some mutually agreed upon maxims seem to be violated. For example, if we agree on only making relevant contributions to conversation, Juan's response in the introduction seemingly violates this maxim—after all, he starts talking about work when Esther asks him about a party. However, because Juan agreed to be relevant he must be implying that having to work means he cannot come to the party. Grice contrasts conversational implicatures that arise through context with conventional implicatures. These are implicatures where the conventional meaning of the word determines what is implicated. An example given by Grice is the following sentence: "he is an Englishman; he is therefore brave." Grice notes that this sentence does not literally state that an Englishman being brave is a direct consequence of him being English, but it's implied by the conventional meaning of the word 'therefore'.
Since then, issues with the Gricean cooperative principle have been pointed out by many (Levinson, 1983; Sperber and Wilson, 1986; Davis, 1998; Lepore and Stone, 2014). The most influential alternative theory is relevancy theory by Sperber and Wilson (1986). They do away with the cooperative principle and instead theorise implicatures arise because speakers try to produce utterances that are both as relevant as possible and require the least effort to process. Another point of contention is the incorporation of conventional implicatures on the pragmatics side. Bach (1999) argues that there is no such thing as conventional implicatures, and they are simply instances of something else. Based on a thorough treatment of what Grice calls conventional implicatures, Bach argues all examples of it can be filed under other concepts within semantics, like utterance modifiers (called "utterance modifiers" instead of "sentence modifiers" because they go against the semantic content of the rest of the sentence). Potts (2005) also argues that to explain conventional implicatures we can stay on semantic turf. Indeed, even Grice himself says conventional implicatures derive from the meaning of the words, not from conversational context. However, Potts does not claim conventional implicatures do not exist, but instead argues they arise by a combination of lexical meaning and novel ways of combining words—the latter being the well-known principle of compositionality, an important part of semantics, not of pragmatics. Potts provides us with an illuminating demarcation between conventional and conversational implicatures. Conventional implicatures are never negotiable by context, whereas conversational implicatures are context-dependent and can always be cancelled without causing incoherent discourse. Consider again the sentence "he is an Englishman; he is therefore brave." and the sentence "Eddie has three bicycles" (implicating that Eddie has exactly three bicycles and not more). The former sentence can not be cancelled by new context without contradiction, whereas for the latter, if we continue saying "In fact, Eddie has 10 bicycles, he is a bicycle junkie", we have cancelled the implicature. This demarcation clearly puts conventional implicatures on the semantic side, and conversational implicatures on the pragmatic side. Potts goes on by providing a formal theory for conventional implicatures.