The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
Authors: Laura Eline Ruis
Akbir Khan
Stella Biderman
Sara Hooker
Tim Rocktäschel
Edward Grefenstette
Venue: NeurIPS 2023 spotlight (2023)
Keywords: large language models
pragmatics
natural language processing
communication
conversation
implicature
language model fine-tuning
TL;DR: Of four categories of large language models evaluated the only type of model that understands implicatures is those fine-tuned on example-level instruction data.

Abstract: Despite widespread use of LLMs as conversational agents, evaluations of performance fail to capture a crucial aspect of communication: interpreting language in context---incorporating its pragmatics. Humans interpret language using beliefs and prior knowledge about the world. For example, we intuitively understand the response "I wore gloves" to the question "Did you leave fingerprints?" as meaning "No". To investigate whether LLMs have the ability to make this type of inference, known as an implicature, we design a simple task and evaluate four categories of widely used state-of-the-art models. We find that, despite only evaluating on utterances that require a binary inference (yes or no), models in three of these categories perform close to random. However, LLMs instruction-tuned at the example-level perform significantly better. These results suggest that certain fine-tuning strategies are far better at inducing pragmatic understanding in models. We present our findings as the starting point for further research into evaluating how LLMs interpret language in context and to drive the development of more pragmatic and useful models of human discourse.

The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
Laura Ruis" University College London
Akbir Khan University College London
Stella Biderman EleutherAI, Booz Allen Hamilton
Sara Hooker Cohere for AI
Tim Rocktäschel
Edward Grefenstette University College London
University College London
Abstract
Despite widespread use of LLMs as conversational agents, evaluations of perfor- mance fail capture crucial aspect of communication: interpreting language in contexi-incorporating its pragmatics. Humans interpret language using beliefs and prior knowledge about the world. For example, we intuitively understand the response "I wore gloves" to the question "Did you leave fingerprints?" as meaning "No". To investigate whether LLMs have the ability make this type of inference, known as an implicature, we design simple task and evaluate four categories of widely used sate-of-the-aa models. We find that, despite only evalu- ating on utterances that require a binary inference (yes or no), models in three of these categories perform close to random. However, LLMs instruction the example-level perform significantly better. These results suggest that certain fine-tuning strategies are far better at inducing pragmatic understanding in models. We present our findings the starting point for further research into evaluating how LLMs interpret language in context and to drive the development of more pragmatic and useful models of human discourse.
Introduction
User: "Have you seen my phone?" GPT-3: "Yes, have seen your phone."
GPT-3's response is a perfectly fine answer to the question, but a human might answer differently. They might respond "it's in your bag," bypassing the obvious follow-up question ("where is it?"). Giving such a helpful and efficient answer an example of pragmatic language use that goes beyond the mere production of semantically plausible and consistent utterances. Meaning is not only determined by a combination of words, but also context, beliefs, and social institutions (Wittgenstein 1953 Grice 1975 Huang 2017). Consider another exchange where Esther asks her friend Juan "Can you come to my party on Friday?" "d and Juan responds "I have to work". We resolve Juan's 's response as him declining the invitation by using the contextual commonsense knowledge that having to work on a Friday night precludes attendance. Both these exchanges contain an implicature utticcttrrr-ttttteeeeee that convey something other than their literal meaning Implicatures illustrate how context contributes to meaning; distinguishing writing and speaking from communicating (Green 1996). We cannot fully
Coorreppnndence to laura. ruis. 21 @ucl. ac. AAppendix D contains details on how this completion obtained from text-davinci-002 In Appendix B we present an introduction to implicature.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

EEammple
Utterunce
Prompt template
Want stay nightcap?
Esther asked "Want to stay for a nightcap?" and Juan responded "I've gotta get early" which means no.
X
LLM
Response
Evaluation polx) > po(x)
I've gotta get up carly.
&
Esther asked "Want to stay for nightcap?" and Juan responded "I've gotta get up early" which means yes.
LLM
Implicature
No.
Figure I: A schematic depiction of the protocol we propose to evaluate whether language models can resolve implicatures. Each example in the test set gets wrapped in templates and transformed into an incoherent example by swapping "yes" and "no". The model is said to resolve the implicature if it assigns higher likelihood to the coherent text than the incoherent text.
understand utterances without understanding their implications. Indeed, the term "communication" presupposes the speaker's ' implications are understood by the addressee. Being able to resolve completely novel implicatures and, more broadly, engage in pragmatic understanding constitutes an essential and ubiquitous aspect of our every day use of language.
Large language models (LLMs) have demonstrated remarkable ability variety of downstream tasks such as planning (Huang et al. (022). commonsense reasoning (Kojima et al. (2022), information retrieval (Lewis et al. 20m0|KKmm Kim et al. [2022] and code completion (Austin al.] 1 Biderman and
Rafi 2002), to name a few. When fine fine-tuned with human feedback, LLMS obtain higher ratings on
desiderata like helpfulness (Ouyang et al. 2022 Bai et al. 2022), and proposed as conversational agents (Toppplaan et al. 2022). Despite the widespread use of L.MMs as conversational agents, there has been limited evaluation of their ability to navigate contextual commonsense knowledge.
This raises an important question: to what extent can large language models resolve conversational implicature? To answer this question we use public dataset of conversational implicatures and propose an evaluation protocol on top of it (Figure [.. We evaluate a range of state-of-the that can be categorised into four groups: large-scale pre-trained models, like OPT (Zhang al. 2022), LLMs fine-tuned on conversational data, like BlenderBot [Ng et al. 2019), LLMs fine-tuned common NI.P benchmarks with natural instructions for each benchmark, like Flan-T5 (Chung eet al. 2022), and L.Ms fine-tuned on tasks with natural instructions for each example, e.g. versions of OpenAl's InstructGPT-3 serie Our results show that implicature resolution is challenging task for LLMs. All pre-trained models obtain close to random zero-shot accuracy (around 60%), whereas humans obtain 86%. However, our results suggest that instruction-tuning at the example level is important for pragmatic understanding. Models fine-tuned with this method perform much better than others, and analysis of different model sizes shows that they have the best scaling properties. We further push performance for these models with chain-of ofo---f-thhhee of-thought prompting, and find that one model in the group (GPT-4) reaches human-leve performance. In summary, we conclude that pragmatic understanding has not yet arisen from large-scale pre-trate vee-rriniig on its own, but scaling analysis shows that it might for much larger scale. Fine-tuning on conversational data or benchmark- instructions does not produce models with pragmatic understanding. However, fine-tuning on instructions at the example-level a fruitful path towards more useful models of human discourse.
models
The main contributions of this work are: i) we motivate implicature understanding as a crucial aspect of communication that currently mostly missing from evaluations of LLMs, ii) we design an implicature resolution task and propose a comprehensive evaluation protocol on which we evaluate both humans and LLMs to find that it poses a significant challenge for SotA LLMs. and iii) we provide a thorough analysis of the results and identify one fine-tuning strategy (instruction-tunii) at the example-level) as a promising method that produces models with more pragmatic understanding.
'The precise method is unpublished and differs from the original instructGPT (Ouyang et al. 2022

---

2
Related Work
LLMs have demonstrated remarkable performance on tasks for which they were not explicitly trained
(Brown al. 2020). Building on the hypothesis that these abilities arise due to implicit multitask learning (Radford et al. 2019), the recent works of Sanh et al. (2022) and Wei et al. (2022) explicitly train LLMs in supervised multitask fashion, leading to models that are better zero-shot learners with fewer parameters. Besides rapidly saturating language understanding benchmarks (Kiela et al. 2022])
these advancements make LLMs beneficial foundations for agents performing a plethora of tasks (Adolphs et al. 2022, Reed et al. 2022). The trend towards using these models as agents brings along with it increased urgency for alignment with human values (Kenton et al. (02T). However, larger models trained with next-word prediction are generally more toxic and unhelpful (Gehman et al.)
2020; Bender et al.) [202T] Lin et al., (022). Recent work mitigates this with methods like prompting and finetuning on human- anntaaaddd outputs (Askell et al. [021 Ouyang et al. [022 Thoppilan et al. 2022). The produced models are more aligned on desiderata such as informativent veness when evaluated by dedicated benchmarks and humans. We argue, however, that there is still something missing in these benchmarks. What is helpful and informative, as Kasirzzdeh and Gabriel (2022) also point out,
depends the context in which conversation is held. Consequently, any application that requires communicating with humans will rely on pragmatic communication Saills-something that is not explicitly captured by the benchmarks used to evaluate the alignment of LLMs.
There is large body of work that investigates the interplay between pragmatics and computational modeling (Cianflone et al. (chuutSr SSchuster et al. [2020] Louis et al. 2020 Kim et al. 2021 Li et al.
202T, Jerettc al., [2020] Parrish et al.) 2021] Hosseiii al. 2023). Cianflone al. (2018) introduce the task of predicting adverbial presupposition triggers, which are words like "again" that trigger
the unspoken presupposition that an event has happened before. [Schuster et al. (2020) study the
ability of computational models to do scalar inferences, finding that models use linguistic features to make pragmatic inferences. Kim et al. (202T) find that a substantial part of question-answeringg datasets contain questions that are unanswerable due to false presuppositions (i.e. "which linguist invented the lightbulb"). Hosseini et al. (2023) present a dataset for selecting entities with indirect answers, and find that language models adapted for this task get reasonable accuracy, but that there is room for improvement. The difference with this body of work and ours is that we look at the emergence of pragmatic understanding from large-scale language modeling. [Jeretic et al. (2020); Parrish et al. (02TI) are early works investigating the emergence of pragmatic understanding in pretrained language models, but they only look scalar implicatures and presuppositions. [Zheng eet al. 2021) are the first to evaluate pretrained language models on conversational implicatures. This important pioneering work highlighting the difficulty of implicature for language models, but their evaluations require task-specific training and the models they evaluate are relatively small. In contrast, our evaluation protocol is applicable out-of-the-box and is much more comprehensive, evaluating models up to 176 billion parameters and using in-context prompting. Additionally, /Zheng et al. (2021) benchmark synthetic data whereas this work evaluates performance on naturally occurring implicatures (George and Mamidi, (020). We believe this to be better representation of the true distribution of implicatures in natural dialogue.
The standard set of benchmarks L.MSs are evaluated on covers many tasks, but even though im- plicature is one of the most important aspects of language pragmatics (Levinson) 1983). it is only evaluated as part of BIG-bench (Srivastava et al. 2022). Unfortunately, the methodology used by the BIG-bench implicature task contributors has limitations, which call into question the validity of their claims. Firstly, the task contributors discard a subset of the data that is ambiguous according to them. In our view this defeats the point of the benchmark. Implicatures are type of non-literal, ambiguous language the intended meaning of which humans often easily interpret; comparing the way humans and models do this precisely what we are interested in. In turn, we expect performance on the BIG-bench task to overestimate the ability of LLMs to resolve naturally occurring implicatures. We keep this challenging subset of the data and instead use human evaluation to deal with examples that are too ambiguous to understand. Secondly, the difference in performance between their average and best rater is 18%, whereas for our evaluations this difference is 6%. This indicates their human evaluation is of low quality, but it is impossible to verify because there are no details available on how the annotation is done. Finally, BIG-bench uses only base LLMS and no SotA fine-tuning methods. In summary, we use a more challenging dataset, and in turn at least six times more evaluations per model, we provide higher-quality human annotations, and evaluate four different categories of L.LMs to investigate which aspects of LLMs contribute to their performance on implicature understanding.
3

---

3
Evaluation Protocol
Here we outline the evaluation protocol we use to answer the research question "To what extent can LLMs resolve conversational implicature?". We focus on binary implicatures that imply "yes" or "no" (see Figure J]. We say a model resolves an implicature correctly if it assigns higher likelihood to a coherent utterance than a similar but incoherent one, detailed below.
Zero-shot evaluation. Consider the example from the introduction packed into a single utterance:
Esther asked "Can you come to my party on Friday?" and Juan responded "I have to work", which means
We can transform this example be pragmatically incoherent (in the sense that it will become pragmatically inconsistent with expected use) by replacing the word "no" with "yes":
Esther asked "Can you come to my party on Friday?" and Juan responded "I have to work", which means yes.
To resolve the implicature, the model should assign higher likelihood to the first of the two sentences above, namely the most coherent one. Importantly, both sentences have exactly the same words except for the binary implicature "yes" "no", making the assigned likelihood scores directly comparable. Formally, let the coherent prompt be x and the augmented, incoherent prompt be & model outputs a likelihood p parameterized by weights We say a model correctly resolves an example x when it assigns pe (x) > Pe (8). This is equivalent to evaluating whether the model assigns higher likelihood to the correct continuation of the two options. Note that this is a more lenient evaluation protocol than sometimes used for language models, where models are evaluated on on their ability to generate the correct continuation, in this case "no". The greedy decoding approach (evaluating whether "yes" or "no" is generated) is also captured by our approach. but we additionally label an example correct if "no" is not the highest assigned likelihood, but still higher than "yes". We did not opt for greedy decoding because "no" is not the only coherent continuation here, and marginalising over all possible correct continuations is intractable. The more lenient evaluation does capture implicature resolution, because the choice of "no" versus "yes"" is only determined by the resolution of the implicature. We guide the models to output "yes" or "no" explicitly in three of the six prompt templates with instructions, such that we can estimate the effect of this guidance on performance. For two model classes (i.e. GPT-3.5-turbo and GPT-4) we do not have access to likelihoods, and for these models we take the greedy decoding approach, guiding the model to output "yes" or "no" explicitly in all prompts (see Table 6 in Appendix (F).
We use a dataset of conversational implicatures curated by George and Mamidi 2020 It contains implicatures that, like in Figure [ are presented 71 718 are binary implicatures that we can convert into an incoherent sentence. We randomly sample 600 examples for the test set and keep the remaining 118 as a development set to improve implicature resolution after pre-training through in-context prompting or fine-tuning.
tuples. Of these,
Few-shot in-context evaluation. We add k examples of the task to the prompt, e.g. with k: 2:
Esther asked "Have you found him yet?" and Juan responded "They're still looking", which means no.
Esther asked "Are you having fun?" and Juan responded "Is the pope Catholic?", which means yes.
Finish the following sentence:
Esther asked "Can you come to my party on Friday?" and Juan responded "I have to work", which means no.
We evaluate the models' k-shot capabilities for & E [1,5, 10, 15, 30) by randomly sampling examples from the development set for each test example. We opt for random sampling approach to control for two sources of randomness. Firstly, examples have different levels of informativeness. Secondly, recent work found that the order in which examples are presented matters (Lu et al. 2022). Ideally, to marginalise over these random factors, we would evaluate each test example with all
'Published under CC BY 4.0 license.

---

permutations of l examples from the development set. This requires (118-k) evaluations for each test example, which is intractable. Instead, we estimate performance per test example by randomly sampling from the development set. In this way we control for some of the variance in performance, but avoid extra evaluations. We ensure each model sees the same few-shot examples per test example.
118!
Controlling for prompt sensitivity. It has been shown language models are sensitive to prompt wording (Efrat and Levy 2020, Tan et al. 2021 Reynolds and McDonell aa Webson and Pavlick 202T). To control for this factor of randomness we manually curate six different template prompts and measure performance across these. One of the templates has been presented above, namely "Esther asked <utterance> and Juan responded <response>, which means <implicatures". Another template is: "Question: <htterance>, response: <response>, meaning: <implicature>" The former we call natural prompts and the latter structured prompts. Each group has three templates that only differ slightly in wording. This grouping allows us to look at the variance due to slight changes in wording as well as performance difference due to a completely different way of presenting the example. The full list of prompts can be found in Appendix
4 Experiments
The set of large language model classes we evaluate can be grouped into four distinet categories:
1. Base models:
large-scale pre-trained models; RoBERTa (Liu al., 2019). BERT (evlin
et al.
2018),
GPT-2 (Radford et al. 2019), FleutherAl (Wang and Komatsuzaki 2021 Black
et al. 2022).
BLOOM (BigScience] 2022), OPT (Zhang et al. 2022). Cohere's ' base models,
and GPT-3
[Bronn et al.) 2020)
2. Dialogue FT: LLMs fine-tuned on dialogue, BlenderBot (Ng et al. 2019).
3. Benchmark IT: LLMs fine-tuned on tasks with natural instructions for each benchmark or "benchmark-level instruction-tuned models"; TO (Sanh et al. 2022) and Flan-T5 (Chung
ct al. 2022).
4. Example IT: LLMs fine-tuned on tasks with natural instructions for each example or "example-level instruction-tunet models";; a subset of OpenAl's API models and Cohere's API models).
For Benchmark IT models, annotators write a single instruction for an entire dataset. The models then fine-tuned on each example from the dataset with the same instruction. We distinguish this from example-level IT; for that type of fine-tuning each example in dataset gets new instruction, resulting more diverse dataset. Each group contains model classes for which we evaluate a range of sizes. A detailed categorization of the models and their attributes can be found in appendix G We make use of the OpenAl and Cohere APIs as well as the pretrained models in the transformers library (Wolf et al., 2020) and EleutherAl's framework to evaluate them (Gao et al., (021). All code used for this paper can be found on GitHut and and the dataset made publicly available on HuggingFacd Below, we present zero-shot and few-shot results, discussing patterns of performance of the models in the four different groups. We further look at the results for different model sizes of each model class and the variance over the prompt templates. We contrast the models' performance with human performance. To this end, each test example gets annotated by five humans. We split the test set in four and assign each annotator subset, leaving with twenty annotators in total. The average human performance is 86.2%. and the best performance is 92%. Some of the errors humans make uncover examples that have multiple interpretations, and others uncover annotation errors. The nature of the task of implicature resolution means we do not expect models to perform better than human best performance. Details the human experiment can be found in the Appendix H (also containing an analysis of human errors), and detailed results per model and prompt template in Appendix K0 We also test for spurious correlations present in the benchmark (like lexical cues the model can rely on), and find no indication (A (Appendix K.8).
Insight I: Models instruction-tuned at the example level outperform all others. Table I shows the best 0-, 1-, and 5-shot accuracy each model class achieved on the implicature task. The best overall
'Nte that there are several important aspects unknown for models behind APIs, like OpenAl's model sizes.
5

---

Table 1: The k-shot accuracy (k 1,,1,5)) for the best performing model of each class. For each model, we select the model size to show by choosing the one that achieves the best 5-shot performance. The std is over prompt templates for the models and over annotators for humans. FT stands for fine-tuning and IT for instruction-tuninn obtain significantly higher performance than all others. means size unknown.
We find that the models in the Example IT class
Model
0-shot
1-shot
5-shot
Random
50%
Baselines and Toplines
Human avg.
86.2% 1 2.3
BERT-110M
54.8% 1.6
51.7% + 1.7
53.3% + 2.2
RoBERTa-355M
55.6% 2.0
54.1%
0.9
57.1%
1.5
GPT-2-xl
51.3% + 2.9
57.4%
3.3
57.7%
1.1
Base models
EleutherAT-20B
57.5% 3.3
55.9% 2.3
61.1%
4,9
BLOOM-1766
54.2% 1.2
61.1% + 2.7
65.4%
+ 3,4
OPT-13B
61.0% 5.5
60.6% 2.7
67.4% 2.1
Cohere-52B
58.5% 4.0
63.0%
+ 3.8
65.1% 2.9
GPT-3-175B
57.7% 4.4
65.7% 1.4
68.7% 1.5
Dialogue FT
BlenderBot-2.78
53.4% 0.3
53.3% 0.1
53.3% 0.1
TO-11B
55.6% 7.0
47.8%
0.5
47.0% 0.2
Benchmark IT
TBB
60.8% 2.4
57.4% + 5.0
61.7% 4.8
text-davinci-001->
72.3% + 2.8
72.7%
1.3
74.5% 1.0
Example IT
lext-davinci-002++
70.6% 2.3
75.6%
2.8
79.6%
2.0
lext-davinci-003-x
71.2% 2.8
74.3%
1 1.4
79.7% 0.6
ChatGPT-*
72.1% 5.9
75.1%
1 1.5
73.9% 6.3
GPT-4-*
81.8%
82.3%
1.4
82.0%
1.7
Sohere-command-555
60.2% 5.2
72.8%
1.3
75.4% 1.8
accuracy is achieved by GPT-4 (the size of this model is unknown) at 82.3% + 1.4. This leaves a gap of 3.9% with human average performance. All models in the class Example IT perform significantly better than any of the other models for all k. except Cohere-command-520 at O-shot. This result is in Figure 2 where present the average accuracy for each model group. The
more clearly performance for the other model classes across ranges from 47.0% by BlenderBot-2 and 68.7% by GPT-3-175b k 5. Even though base models benefit from few-shot examples, their performance remains mostly closer to random than to humans for all k. showing a gap of at least 17.5% with the average human. We observe a decrease performance for k: >> 0 for the group Benchmark IT. This is not surprising, as these kind of models are specifically fine-tuned to be better zero-shot generalisation (Sanh et al., [2022] Chung et al.) 2022). BlenderBot. in the group Dialogue FT, performs barely better than random for all k. We hypothesise that the lower performance which coberr-command-526 achieves O-shot is not due lack of implicature understanding, but due to failure to calibrate the yes/no likelihoods without examples. For this model, we observe sharp rise in performance from k of an implicature induces pragmatic understanding, we hypothesise that few-shot prompting mostly serves to clarify the task format. We test this hypothesis Appendix K.6 by repeating the and 5-shot experiment with random labels for Cohere-command-5221 and text-davinci-001. We find that performance does not degrade, which confirms that the few-shot examples mainly serve to prime the model towards producing outputs following the yes/no structure.
at k
(see Table T or 2). Since is unlikely that one example
Insight 2:
The results are robust different prompt templates. As detailed in Section 3. each example in the test set is wrapped in six different prompt templates. The standard deviation in Table the sensitivity different prompt wording. The standard deviation ranges from 0.3 for BlenderBot to 7.0 for TO-1 IB. All in all, the sensitivity to prompt wording does not seem be a problem for this task; when taking into account the confidence intervals the result remains that models in the group Example IT perform significantly better than all other models, but worse than humans. Appendix K. another analysis is presented that shows how different prompt templates benefit from in-context examples. The takeaway from the analysis is that in-context prompting can mitigate the fact that some models are better at natural prompts and others better at structured prompts by improving performance the type of prompt the model struggles with shot. Again, when
Mand in Tigure]showe

---

The accuracy number of in-context examples (k)
with bars showing group means
100
Example
Base
Benchmark
Dialogue FT
GPT-4-
text 1exi-davinci-002-
text-davinci-003--
text-divinci-001-a
Accuracy (%)
60
CharGPT.*
GPT-3-1756
Cohere-52b
OPT-13b
Flan-TS-1
BLOOM-176b
EleutherAl-200
BlenderBat-3b
TO-11b
Thiry
Randorn chance
In-context examples (k)
Figure 2: The few-shot accuracy for the best model of each class (c.g. the best performing model in the class Cohere-comment model). The bars show the group means. Models fine-tuned on example-level instructions perform better than most other models, especially for k 0. For all models there is a significant gap between best accuracy and human accuracy (which is 86.2%).
is the 52b model, whereas the best model in the class OPT is the 13b
means size unknown.
Table 2: Scaling results for OpenAl's cext=<engine~-001sseeee for which we do not know the number of non-embedding parameters but do know the ordering in terms of size. The colors indicate whether going up in size (from left-to-right) increases performance significantly or not.
Engine
Ada
Babbage
Curie
Davinci
O-shot
56.5% + 5.8
64.5% - 1.8 (+8.0%)
69.0% + 2.9 (+4.5%)
72.3% + 2.8 (+3.3%)
5-shot
57.6% + 2.8
66.1 + 0.3 (48.5%)
71.3% + 1.3 (45.2%)
74.5% + 1.0 (44.0%)
only looking at the best prompt type for each model class (i.e. structured or natural), the results remain that models in the group Example IT perform best.
Insight 3: Models instruction-tuned at the example-level have the most favourable scaling proper- ties, but some base models also show positive correlation with scale. Figure FigureBssows the scaling behaviour of the model classes for which we know the number of son-embedding parameters. We highlight 0- and 5-shot results, because for le the accuracy of most models plateaus (see Figure 2). However, detailed results for other k can be found in Appendix K.10 Note that we do not know the number of parameters for OpenAl's text-<engine>-00) -seiies, but we do know the order of the engines in size, and we separately present its scaling results in Table 2 Except OpenAl's text-<engine>-0011 -series, none of the models show significant performance increase with model size for the 0-shot evaluations. However, for k-shot evaluations with A: 2 we observe significant positive correlation with size for the models in the Exumple IT class for which we have multiple sizes (Cohere-commant
and "eext-cengine>-001 as well as some models in the base model class. Not only do the models in the Example IT class cxhibit higher performance for the same model size, these models also have steeper performance increase with size than the base models. Comparing the scaling properties of the best base model (GPT-3) with Cohere-command, we see that the in- crease in performance from the second-larges to the largest model is 0.04% per billion parameters from GPT-3-6.78 to GPT-3-175B and 0.15% per billion parameters for taterr-command-6f1 to conereecommand--ae (exact numbers used to calculate the slope be found in Appendix 10. performance is linearly extrapolated from this curve GPT-3 reaches human-level performance at 642b parameters where Cohere-comman would need 125b parameters.
Insight 4: GPT-4 reaches average human-level performance with chain-of-thought prompting. For the model groups that benefit from in-context examples, attempt to push performance further
7

---

with ohfiouhf prompting. We manually write a five-shot chain-of-thougee prompt for all six prompt templates, and evaluate model performance using this prompt. One of the six chain-of-thour prompts can be found in Table 4 in Appendix F and the other five are provided in the supplementary material. We only present the results for the group Example IT here, since CoT prompting did not improve performance for two of the base model classes we tried (see Appendix K. Consequently, we decided not apply this experiment the other models in the base group save compute costs. The results of are shown in Table 3 We find that chain-of thain-of-thoue of-thought prompting does not help for all models, but is nonetheless able boost performance of GPT-4 86.5% = 1.0. This on-par with average human-level performance, and slightly below human best performance at 89.8%. To illustrate how explicit reasoning helps implicature understanding, we highlight CoT generated by GPT-4 for an example from the dataset that models persistently get wrong. "A: Is there a bus can get to the station? You can't 't rely on it". The implicature is yes, there bus, you just cannot rely on GPT-4 five-shot gets this wrong for all six templates. With CoT it gets it right for five of six templates. Performance the implicature task for k-shot evaluations
Zero-shot.
Five-shot.
100
Accuracy (%)
60
Best human
OPT
GPT-3
Avg. human
EleutherAI
TO
20
Cohere-command
BLOOM
BlenderBoo
Flan-TS
Cohere
Random chance
10"
10" 10"
10
10"
10"
Model Parameters (Non-Embedding)
Model Parameters (Non-Embedding)
Figure 3: Scaling results for the model classes of which we know the number of non-embedding parameters. The eтror bars show standard deviation over prompt templates. Cohere's ' command models instruction-tume there is still a significant gap between best accuracy and human accuracy.
at the example-level perform better than all other models. For all models
Table 3: Results of the chain-of- -ftthughtt (CoT) experiment for models in the group Example IT. The numbers between brackets show the difference in performance with the number the same row one column to the left. Most models benefit from CoT-prompting. but not all. Additionally, GPT-4 reaches average human-level performance with CoT prompting. means size unknown.
Model
0-shot
5-shot
5-shot CoT
text-davinci-000 000eiii-00--x----
72.3% + 2.8
74.5% + 1.0 (+2.2%)
67.3% + 2.6 (-7.2%)
text-davinci-00r
70.6% = 2.3
79.6% 2.0 (+9.0%
80.1% 0.8 (+0.5%)
text-davinci-003-.
71.2% 2.8
79.7% = 0.6 (+8.5%)
83.6% = (+4.0%)
ChatGPT-*
72.1%
6.0
73.9% + -6.3 (+1.8%)
77.2%
(+3.3%)
GPT-4-+
81.8% + 1.8
82.0% + 1.77 (+0.2%)
86.5% = 1.0 (+4.5%
Cohere-command-55h
60.2% + 5.2
75.4% + 1.8 (+15.2%%
75.3% + 0.5 (-0.1%)
The generated CoT for one template is the following:
Alice says 'You can't rely it.' Alice must be implying that there is bus, but it may not be dependable or timely. This means the response to Bob's 's question is yes, but with a caution about reliability, Answer: yes
More completions can be found in Appendix J

---

Accuracy different types implicature
NB: Y-axis sturts 45% (just below random performance).
Cohere-ommmdd-
Accumcy (%)
Paricslerisol
Genenlisod
Uummon perticibiised
gencralised
Random chance
The accuracy k for the generalised and particularised examples obtained by the Example IT models Cohere-command and GPT-4. Particularised context-heavy) examples are often significantly more difficult than generalised (context-free) examples for both models and humans. For most models, in-context prompting can mitigate this, but for others (like GPT-4), a significant gap remains. We that Cohere-command-522) achieves similar performance GPT-4 the particularised examples, but significantly lower on the generalised examples.
Figure 4:
Table 4: An example from the dataset for two types of implicature found in the test set. The rightmost column shows the amount of that type we manually found in the test set.
Type
Example Utterance
Example Response
Impl.
Generalised
You know these people?
Some.
No.
Particularised
Want to stay for nightcap?
I've gotta get up early.
No.
94
Insight 5: Models often struggle with the same type of examples humans struggle with. We manually labeled 217 examples of the 600 examples in the test set according to a taxonomy. The remaining 383 examples do not fall clearly within category and are grouped together type other. In Table 4 the two types of examples that occur frequently in the dataset are exemplified. Generalised implicatures require little no context to be understood. They the simplest type of example in the test set, and generally imply the same thing ("some" almost always implies "not all"). Particularised implicatures, by contrast, do require context to be resolved. For example, from Table 4 we need the context that it is undesirable to stay up late drinking when one has to get up early (see in Appendix for more on generalised vs. particularised). In these type of examples, the context needed to resolve it is different every time. We label three other types of implicatures in the dataset, but since the analysis of these examples does not show significant patterns, we present it in Appendix K.9 We show the accuracy broken down per example type for two models from the Example IT group, as these patterns hold more broadly for almost all models evaluated (see the detailed results broken down per example type in Appendix K.9.. Figure 4shows that for lower k; the models often have a significantly worse performance for particularised examples than for generalised examples, just like humans do. For some, like Cohere-command-522 this is mitigated by few-shot prompting. which brings particularised and generalised performance closer together (sometimes at the cost of generalised performance). For others, like GPT-4, the gap between particularised and generalised performance remains large for all k. From the bottom row in Figure 4 we observe that the edge GPT-4 has Cohere-commande
seems mostly driven by higher accuracy on generalised examples. The accuracy on the particularised examples is comparable between those two models.
Discussion
In this study we use prompting to evaluate whether different groups of LLMs can resolve implicatures. In designing our experimental protocol, we carefully considered various alternatives, and here we discuss limitations of the chosen approach. Firstly, evaluating LLM competencies inherently uncer- tain and sensitive to prompt choice. Nonetheless, we are confident our evaluation is comprehensive enough to assess implicature understanding: we apply six different prompt templates per test example, each used in three different prompting techniques (zero-shot, sho-shot, chain-of-thoue ofothhghhh).......oooooo......). Addition-

---

ally, in the appendix we present alternative zero-shot prompts and task specifications (Appendix K. I but since these did not improve performance they were not further considered. Another limitation is the fact that subset of the models we evaluate are behind APIs. This means models are subject to change (affecting reproducibility) and certain details about these models are unknown. This affects the group instruction-tunee at the example-level, which is the group we find outperforms all others and has the most favourable scaling properties. How do we know instruction- tuning at the example-level is the main driver behind these findings without controlled A/B testing? Unfortunately, due to the secrecy surrounding the exact implementation of these models we cannot be certain, but can be relatively confident. We evaluated ten models across six model classes and two APIs in the group example-leve instruction tuned. Within this group, models probably vary significantly in other training and architecture details (especially Cohere-commane OpenAl models). The most salient commonality they share with each other and none of the other models is multi-task instruction-tuniig factor of their performance. A further datapoint in favour of this conclusion can be seen in Figure 3 (right); base models at similar scales as Example IT models perform significantly worse. We see that Cohere-command 52B significantly outperforms Cohere-base 52B, and the only difference between those models is instruction-tuning the example level (Cohere-command is fine-tuned from Cohere-base). In fact, Cohere-command 52B outperforms other base models more than 3 times the size by a large margin (e.g. GPT-3 175B, BLOOM- 176B, OPT-175B). We are therefore confident that instruction tonnnnntttnnnnggn at the example-leve is important for pragmatic understanding, an insight which can guide the development of open-source models capable of pragmatic understanding. Investigating the exact effect of this type of instruction-tuninn on pragmatic understanding in a controlled setting is an interesting future work direction (e.g. by isolating the effect of data diversity from instructions). Another limitation is that some evaluations are subject to API stochasticity, which we address in Appendix K.5] After running the zero-shot experiment ten times through each API we conclude there is some stochasticity, but it is too small to impact our conclusions. We publish exact timestamps at which we queried APIs in Appendix [ Further, a downside of doing a comprehensive analysis on many models is compute costs. In Appendix M we publish list of exact compute used (time and hardware), as well as estimated carbon emissions for each of the models that are not behind an API. Finally, the likelihood ranking approach alternative. However, implicatures in natural language can entail more complex propositions. For example, imagine Esther now asking "Can I use your stapler?" and Juan responding "Here's 's the key to my office.". ". Juan is implicating that (1) Esther can use the stapler, (2) the stapler is located in the office, and (3) the office is currently locked. This leaves ample room for the design of benchmarks with implicatures entailing multiple non-binary propositions.
models versus
at the example level, making it likely that this is the driving
take limits study to implicatures with clear
6 Conclusion
LLMs have made remarkable progress on fluency and coherence in recent years. We argue how- ever that central aspect of language understanding is missing from evaluations. To understand language means to understand its pragmatics: its usage in a context that incorporates commonsense understanding, goals, objectives, and so on. We design protocol that evaluates LLMs on binary implicature resolution and establish a significant gap with human understanding for SotA LLMs in three categories; surge-scale pre-trained models, models fine-tuned on conversations, and mod- els fine-tuned with benchmark-Ioo example-level instructions perform significantly better. This group also exhibits the best correlation between accuracy and model size. Scaling analysis shows that for some large-scale pre-trained models accuracy also positively correlates with model size, but the best model in this group would need at least five times more parameters to reach similar performance. From these results, we conclude that cntructtion-unnngg at the example level is important for pragmatic understanding. We hypothesise that there is something about the multi-task data diversity obtained from example-level instructions (i.e. each example new task) that makes pragmatic understanding appear at smaller scale.
instructions. By contrast, we find that models fine-tuned on
A Acknowledgements
We would like to thank members of the UCL DARK lab, members of the UCL NLP group, Pontus Stenetorp, Sebastian Borgeaud, Philipp Jettkant, Robert Kirk, and Max Bartolo for fruitful discussions and comments on an earlier draft of this paper. We would also like to thank the anonymous reviewers for engaging actively in discussion and providing feedback that has improved this work. This work was supported by the EPSRC Grant EP/S021 566/1 and UCL International Scholar Award for Doctoral Training Centres.
10

---

References
Adolphs, L.. Börschinger, B., Buck, C., Huebscher, M. C., Ciaramita, M., Espeholt, L.,, Hofmann, T., Kilcher, Y., Rothe, S., Sessa, P. G., and Sestorain, L. (2022). Boosting search engines with interactive agents. Transactions on Machine Learning Research.
American Psychiatric Association, A. A. (2013). Diagnostic and statistical manual of mental disorders DSM-5. American Psychiatric Association Arlington, VA, 5th ed. edition.
Askell, A., Bai, Y.. Chen, A., Drain, D., Ganguli, D.. Henighan, T., Jones, A., Joseph, N., Mann, B.,
DasSarma, N., Elhage, N., Hlatieed-Dodde Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T. B.. Clark, J., McCandlish, S., Olah, C.. and Kaplan, J. (2021). A general language assistant as a laboratory for alignment. CoRR, abs/21 12.00661.
Austin, J., Odena, A., Nye. M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021). Program synthesis with large language models. arXiv preprint 108.0:2/08.07732.
Bach, K. (1999). The myth of conventional implicature. Linguistics and Philosophy, 22(4):327-366.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022). Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXhv:2204.05862
Bender, E. M., Gebru, T., McMillan-Man Majorr A., and Shmitchell, (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transpurency, pages 610-623.
Biderman, S. and Raff, E. (2022). Fooling moss detection with pretrained language models. arXiv preprint arXiv:2201.0744
BigScience (2022). Bigscience language open-science open-access multilingual (bloom) language model.
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L.., Golding, L.,, He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L.., Tow, J., Wang, B., and Weinbach, S. (2022). GPT-NeoX-205B An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H.. editors, Advances Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Chung, H. W., Hou, L., Longpre, S.. Zoph, B., Tay, Y., Fedus, W., Li, E., Wang. X., Dehghami, M.. Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. (2022). Scaling instruction finiionn-f language models.
Cianflone, A., Feng, Y., Kabbara, J., and Cheung, J. C. K. (2018). Let's do it "again": A first computational approach to detecting adverbial presupposition triggers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2747-2755, Melbourne, Australia. Association for Computational Linguistics.
Davis, W. (2019). Implicature. In Zalta, N., editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, fall 2019 edition.

---

Davis, W. A. (1998). Implicature: intention, convention, and principle in the failure of Gricean theory / Wayne A. Davis. Cambridge studies philosophy. Cambridge University Press, Cambridge England: New York.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2018). BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.0480/
Efrat, A. and Levy, O. (2020). The turking test: Can language models understand instructions? CoRR, abs/2010.11982
Frank, M. C. and Goodman, N. D. (2012). Predicting pragmatic reasoning in language games. Science, 336:998- - 998.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L.,, Hsu, J., McDonell, K., Muennighoff, N., Phang. J., Reynolds, L.., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. (2021). A framework for few-shot language model evaluation.
Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. (2020). RealToxicityPromee Evaluating neural toxic degeneration in language models. Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 3356-3369, Online. Association for Computational Linguistics.
George, E. J. and Mamidi, R. (2020). Conversational implicatures in english dialogue: Annotated dataset. Procedia Computer Science, 171:2316-2323, https:/doi..or//10 0. 1016/0001 ptt.://....g/...101../ooo2222220 propp
Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M.. Weidinger, L., Chadwick, M., Thacker, P., Campbell-Cillingham,, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang. F., See, A., Dathathri, S., Greig, R., Chen, C.,, Fritz, D., Sanchez Elias, J., Green, R., Mokrá, S., Fernando, N., Wu, B., Foley, R., Young, S., Gabriel, I., Isaac, W., Mellor, J., Hassabis, D., Kavukcuoglu, K., Hendricks, L. A., and Irving, G. (2022). Improving alignment of dialogue agents via targeted human judgements.
Goodman, N. and Frank, M. (2016). Pragmatic language interpretation as probabilistic inference.
Trends in Cognitive Sciences, 20.
Green, G. (1996). Pragmatics and Natural Language Understanding. Tutorial essays in cognitive science. Erlbaum.
Greene, S. and Resnik, P. (2009). More than words: Syntactic packaging and implicit sentiment. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 503-511, Boulder, Colorado. Association for Computational Linguistics.
Grice, H. P. (1975). Logic and conversation. In Cole, P. and Morgan, J. L., editors, Syntax and Semanties: Vol. 3: Speech Acts, pages 41 41-58. Academic Press, New York.
Hosseini, M. J., Radlinski, F., Pareti, S.,, and Louis, A. (2023). Resolving indirect referring expressions for entity selection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12313-12335, Toronto, Canada. Association for Computational Linguistics.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. (2022). Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.077 077077.
Huang, Y. (2017). The Oxford Handbook of Pragmatics. Oxford handbooks in linguistics. Oxford University Press.
Jeretic, P., Warstadt, A., Bhooshan, S., and Williams, A. (2020). Are natural language inference models IMPPRESSiVe? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690-8705 Online. Association for Computational Linguistics.
Kasirzadeh, A. and Gabriel, I. (2022). conversation with artificial intelligence: aligning language models with human values.
12

---

Katsos, N., Roqueta, C. A., Estevan, R. A. C., and Cummins, C. (2011). Are children with specific language impairment competent with the pragmatics and logic of quantification? Cognition. (19(1):43-77.
Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I,, Mikulik, V., and Irving, G. (2021). Alignment of language agents. CoRR, abs/2103.14659)
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stenetorp. P., Jia, R., Bansal, M., Potts, C., and Williams, A. (2021). Dynabench: Rethinking benchmarking in nip. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis- lies: Human Language Technologies, pages 41 4100-4124 Online, Association for Computational Linguistics.
Kim, N., Pavlick, E., Karagol Ayan, B., and Ramachandran, D. (2021). Which linguist invented the lightbulb? presupposition verification for question-answeringg In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 3932-3955, Online. Association for Computational Linguistics.
Kim, S. Y., Park, H., Shin, K., and Kim, K.-M. (2022). Ask what you need: Product retrieval using knowledge from gpt-3. arXiv preprint arXiv:2207.02516
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners.
Lepore, E. and Stone, M. (2014). Imagination and Convention: Distinguishing Grammar and Inference in Language. Oxford University Press.
Levinson, S. C. (1983). Pragmatics. Cambridge University Press, Cambridge, U.K.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., al. (2020). Retrieval-augmenned generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474.
Li, E., Schuster, S., and Degen, J. (2021). Predicting scalar inferences from "or" to "not both" using neural sentence encoders. In Proceedings of the Society for Computation in Linguistics 2021, pages 446-450, Online. Association for Computational Linguistics.
Lin, S., Hilton, J., and Evans, O. (2022). TruthfulQA: Measuring how models mimic human false- hoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 3214-3252, Dublin, Ireland. Association for Computational Lin- guistics.
Liu, Y. Out, M., Goyal, N., Du J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized BERT pretraining approach. CoRR, abs//990.11669
Louis, A., Roth, D., and Radlinski, F. (2020). "I'd rather just go bed": Understanding indirect answers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7411-7425, Online. Association for Computational Linguistics.
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp. P. (2022). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL.
Ng, N., Yee, K., Baevski, A., Ott, M., Auli, M., and Edunov, S. (2019). Facebook FAIR's s WMT19 news translation task submission. Proceedings of the Fourth Conference on Machine Trans- lation (Volume 2: Shared Task Papers, Day 1), pages 31 314-319, Florence, Italy. Association for Computational Linguisties.
Ouyang, L., Wu, J., Jiang, X., Almeida, D.,, Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. J. (2022). Training language models to follow instructions with human feedback. ArXiv, abs/2203,.mm 05515
13

---

Parrish, A., Schuster, S., Warstadt, A., Agha, O., Lee, S.-H., Zhao, Z.. Bowman, S. R., and Linzen, T. (2021). NOPE: A corpus of occuurrig presuppositions in English. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 349-366, Online. Association for Computational Linguistics.
Patel, R. and Pavlick, E. (2021). "was it "stated" or was it "claimed" How linguistic bias affects generative language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10080-10095, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Potts, C. (2005). The Logic of Conventional Implicatures. Oxford University Press UK.
Potts, C. (2006). Conversational implicatures via general pragmatic pressures. In Washio, T., Satoh, K. Takeda, H., and Inokuchi, A., editors, New Frontiers in Artificial Intelligence, pages 205-218, Berlin, Heidelberg. Springer Berlin Heidelberg.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners.
Recasens, M., Prreossu-Niculescu--ee
C., and Jurafsky, D. (2013). Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1650-1659, Sofia, Bulgaria. Association for Computational Linguistics.
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., Eccles, T., Bruce. J., Razavi, A., Edwards, A., Heess, N.. Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. (2022). A generalist agent.
Reynolds, L. and McDonell, K. (2021a). Prompt programming for large language models: Beyond the few-shot paradigm.
Reynolds, L. and McDonell, K. (2021b). Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHII Conference Human Factors in Computing Systems, CHI EA '21, New York, NY, USA. Association for Computing Machinery.
Sanh, V., Webson, A., Raffel, C.,, Bach, S., Sutawika, L.,, Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J.. Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X.,
Pandey, H., Bawden, R., Wang, T., Neeraj. T., Rozen, J., Sharma, A., Santilli, A., Fevry, T., Fries, J. A., Teehan, R., Scao, T. L.,, Biderman, S., Gao, L., Wolf, T., and Rush, A. M. (2022). Multitask prompted training enables zero-shot task generalization. International Conference on Learning Representations.
Schaeken, W., Van Haeren, M., and Bambini, V. (2018). The understanding of scalar implicatures in children with autism spectrum disorder: Dichotomized responses to violations of informativeness. Frontiers in Psychology, 9.
Schuster, S., Chen, Y., and Degen, J. (2020). Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387-5403, Online. Association for Computational Linguistics.
Sperber, D. and Wilson, D. (1986). Relevance: Communication and Cognition. Language and thought series. Harvard University Press.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro,
A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowyez, A., Agarwal, A., Power, A., Ray, A..
Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Slone, A., Rahane, A., Iyer, A. S., Andreassen, A., Madotto, A., Santilli, A., Stuhlmüller, A., Dai, A., La, A., Lampinen, A., Zou, A., Jiang, A., Chen, A., Vuong, A., Gupta, A., Gottardi, A., Norelli, A., Venkatesh, A., Gholamidavoodi, A., Tabassum, A., Menezes, A., Kirubarajan, A., Mullokandov, A., Sabharwal, A., Herrick, A., Efrat, A., Erdem, A., Karakaş, A., Roberts, B. R., Loe, B. S., Zoph, B., Bojanowski, B., Özyurt, B., Hedayatnia, B., Neyshabur, B., Inden, B.. Stein, B., Ekmekci, B., Lin, B. Y., Howald, B., Diao, C., Dour, C.. Stinson, C., Argueta,
14

---

C., Ramírez, C. F., Singh, C.. Rathkopf, C., Meng, C., Baral, C., Wu, C., Callison- -Brrhh, C.. Waites, C., Voigt. C.. Manning, C. D., Potts, C., Ramirez, C., Rivera, C. E., Siro, C., Raffel, C.. Ashcraft, C., Garbacea, C., Sileo, D., Garrette, D., Hendrycks, D., Kilman, D., Roth, D., Freeman, D., Khashabi, D., Levy, D., González, D. M., Perszyk, D., Hernandez, D., Chen, D., Ippolito, D., Gilboa, D. Dohan, D.. Drakard, D., Jurgens. D., Datta, D., Ganguli, D., Emelin, D., Kleyko, D., Yuret, D. Chen, D., Tam, D., Hupkes, D., Misra, D., Buzan, D., Mollo, D. C., Yang. D., Lee, D.-H. Shutova, E., Cubuk, E. D., Segal, E., Hagerman, E., Barnes, E., Donoway, E.,, Pavlick, E., Rodola, E., Lam, E., Chu, E., Tang, E., Erdem, E., Chang, E., Chi, E. A., Dyer, E.,. Jerzak, E., Kim, E., Manyasi, E. E., Zheltonozhskii, E., Xia, F., Siar, F., Martinez-Plumeet F., Happé, F., Chollet, F., Rong, F., Mishra, G., Winata, G. I., de Melo, G., Kruszewski, G., Parascandolo, G., Mariani, G., Wang. G., Jaimovitch-- López, G., Betz. G., Gur-Ari, G., Galijasevic, H., Kim, H., Rashkin, H., Hajishirzi, H., Mehta, H., Bogar, H., Shevlin, H., Schütze, H., Yakura, H., Zhang, H., Wong, H. M., Ng. I., Noble, I., Jumelet, J., Geissinger, J., Kernion, J., Hilton, J., Lee, J., Fisac, J. F., Simon, J. B., Koppet, J., Zheng, J., Zou, J. Kocoń, J., Thompson, J., Kaplan, J., Radom, J., Sohl- J., Phang. J., Wei, J., Yosinski, J., Novikova, J., Bosscher, J., Marsh, J. Kim, J., Taal, J., Engel, J., Alabi, J., Xu, J.. Song. J., Tang, J.. Waweru, J., Burden, J., Miller, J., Balis, J. U., Berant, J., Frohberg. J., Rozen, J., Hernandez-Orallo, J., Boudeman, J., Jones, J., Tenenbaum, J. B., Rule, J. S., Chua, J.. Kanclerz, K., Livescu, K., Krauth, K., Gopalakrishnan, K., Ignatyeva, K., Markert, K.. Dhole, K. D., Gimpel, K., Omondi, K., Mathewson, K., Chiafullo, K. Shkaruta, K., Shridhar, K., McDonell, K., Richardson, K., Reynolds, L.. Gao, L., Zhang. L., Dugan, L.. Qin, L., Contreras-Ochando, L., Morency, L.-P. Moschella, L., Lam, L.,, Noble, L.. Schmidt, I.., He, L.., Colón, I.. O., Metz, I.., Şenel, I.. K., Bosma, M., Sap. M. ter Hoeve, M., Farooqi, M., Faruqui, M., Mazeika, M., Baturan, M., Marelli, M., Maru, M., Quintana, M. J. R., Tolkiehn, M., Giulianelli, M., Lewis, M., Potthast, M., Leavitt, M. L., Hagen, M., Schubert, M., Baitemirova, M. O., Arnaud, M., McElrath, M.. Yee, M. A., Cohen, M., Gu, M., Ivanitskiy, M., Starritt, M., Strube, M., Swędrowski, M., Bevilacqua, M., Yasunaga, M.. Kale, M., Cain, M., Xu, M., Suzgun, M., Tiwari, M., Bansal, M., Aminnaseri, M., Geva, M., Gheini, M., T. M. V., Peng. N., Chi, N., Lee, N., Krakover, N. G.-A., Cameron, N., Roberts, N. Doiron, N., Nangia, N., Deckers, N., Muennighoff, N., Keskar, N. S., Iyer, N. S., Constant, N., Fiedel, N., Wen, N., Zhang. O.. Agha, O., Elbaghdadi, O., Levy, O.,, Evans, O., Casares, P. A. M., Doshi, P., Fung, P., Liang, P. P., Vicol, P., Alipoormolabashi. P., Liao, P., Liang, P., Chang. P., Eckersley, P., Htut, M., Hwang, P., Milkowski, P., Patil, P., Pezeshkpour, P., Oli, P., Mei, Q., Lyu, Q., Chen, Q., Banjade, R., Rudolph, R. E., Gabriel, R., Habacker, R., Delgado, R. R., Millière, R., Garg, R., Barnes, R., Saurous, R. A., Arakawa, R., Raymaekers, R., Frank, R., Sikand, R., Novak, R., Sitelew, R., LeBras, R., Liu, R., Jacobs, R., Zhang, R., Salakhutdinov, R., Chi, R., Lee, R., Stovall, R., Teehan, R., Yang, R., Singh. S., Mohammad, S. M., Anand, S., Dillavou, S., Shleifer, S., Wiseman, S., Gruetter, S., Bowman, S. R., Schoenholz, S. S., Han, S., Kwatra, S., Rous, S. A., Ghazarian, S., Ghosh, S., Casey, S., Bischoff, S., Gehrmann, S., Schuster. S., Sadeghi, S., Hamdan, S., Zhou, S., Srivastava, S., Shi, S., Singh, S., Asaadi, S., Gu, S. S., Pachchigar, S.. Toshniwal, S., Upadhyay, S., Shyamolima, D., Shakeri, S., Thormeyer. S., Melzi, S., Reddy, S., Makini. S. P., Lee, S.H.., Torene. S., Hatwar, S., Dehaene, S., Divic. S., Ermon, S., Biderman, S.. Lin, S., Prasad, S., Piantadosi, S. T., Shieber, M., Misherghi, S., Kiritchenko, S. Mishra, S., Linzen, T., Schuster, T., Li, T., Yu, T., Ali, T., Hashimoto, T., Wu, T.-L.. Desbordes, T., Rothschild, T., Phan, T., Wang. T., Nkinyili, T., Schick, T., Kornev, T., Teatonn T., Tunduny, T., Gerstenberg. T., Chang, T., Neeraj, T. Khot, T., Shultz, T., Shaham, U., Misra, V. Demberg, V., Nyamai, V., Raunak, V., Ramasesh, V., Prabhu, V. U., Padmakumar, V., Srikumar, V., Fedus, W., Saunders, W., Zhang, W., Vossen, W., Ren, X., Tong, X., Zhao, X., Wu, X., Shen, X., Yaghoobzadeh, Y., Lakretz, Y., Song, Y., Bahri, Y., Choi, Y., Yang, Y., Hao, Y., Chen, Y., Belinkov, Y., Hou, Y., Hou, Y., Bai, Y., Seid, Z.,, Zhao, Z., Wang, Z., Wang, Z. J., Wang, Z., and Wu, Z. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
Tan, Z., Zhang, X., Wang, S., and Liu, Y. (2021). Msp: Multi-stage prompting for making pre-trained language models better translators. arXiv preprint 2
Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C., Krivokon, I., Rusch, W., Picketr, M., Meier-
K. S., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K. Molina, A., Nofffann-ohnn. E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M.,
15

---

Kuzmina. V., Fenton, J.. Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Areas, B., Cui, C., Croak, M.. Chi, E. H., and Le, Q. (2022). Lamda: Language models for dialog applications. CoRR. 035/22011082233
Volden, J. (2017). Autism Spectrum Disorder, pages 59-83. Springer International Publishing, Cham.
Wang, B. and Komatsuzzki, A. (2021). GPT-J-6B: 6 Billion Parameter Autoregressive Language Model.
htt.:/tww.cco///iggoooo///////aa tranhtorrarffffrrrrrr jax
Webson, A. and Pavlick, R. (2021). Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247.
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. (2022). Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Wittgenstein, L. (1921). Tractatus logico-phiiossophicus. London: Routledge, 1981.
Wittgenstein, L. (1953). Philosophical Investigations. Basil Blackwell, Oxford.
Wolf, T., Debut, I.., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., Platen, P., Ma, C., Jernite, Y., Plu, Xu, C.,, Le Seao, T., Gugger, S., Drame, M., Lhoest, Q.. and Rush, A. (2020). Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguisties.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang. T. and Zettlemoyer, L. (2022). Opt: Open pre-trained transformer language models.
Zheng, Z., Qiu, S., Fan, L., Zhu, Y., and Zhu, S.-C. (2021). GRICE: grammar- based dataset for recovering implicature and conversational rEasoning. In Findings of the Association for Computa- tional Linguistics: ACL-IJCNLP 202/, pages 2074-2085, Online, Association for Computational Linguistics.
16

---

Appendices
Contents
Contributions
18
Reproducibility Statement
18
Ethies Statement
18
Opener example with InstructGPT
19
Background implicature
20
Detailed prompt templates
22
Model categorization
26
Human evaluation
27
Comparison with BIG-bench implicatures task
29
Discarding ambiguous examples)
29
1.2
Overestimation of performance on implicature understanding
30
Other limitations
30
J
Chain-of-thought completions by GPT-4
31
K
Additional results
32
K.I
Contrastive experiment
32
K.2
Variance over prompt ordering
33
K.3
Different zero-shot instruction prompts
33
K.4
The effect of in-context examples sensitivity to prompt wording
33
Variance over API runs
34
K.6
Experiment with random in in-context labels
34
K.7
Chain-of-thought on base models
36
K.8
Testing for spurious correlations
36
K.9
Detailed results type label analysis
37
K. 10 Detailed results per model
37
Timestamps API calls
69
M Compute and Emissions
72
17

---

A
Contributions
Laura Ruis: project proposal and leadership, dataset development, code writing, human experiment, manual error analysis, paper writing and editing.
Akbir Khan: code writing, model evaluations, human experiment, paper writing and editing.
Stella Biderman: model evaluations, compute usage, paper writing and editing, advisor.
Sara Hooker: compute usage, paper writing and editing, advisor.
Tim Rocktäschel: paper writing and editing, advisor.
Edward Grefenstelte: initial idea, manual error analysis, paper writing and editing, advisor.
B Reproducibility Statement
We share all the data, human annotations, code used for the evaluations, and the raw results in the supplementary material. Additionally, in Appendix K_5 estimate the variance due to stochasticity in the API's of OpenAT and Cohere. Of course, if either OpenAl Cohere decides to change the models behind the API, the results might look different. We publish the exact date and time each API Finally, in Appendix K.2 we estimate the variance
was queried for the results in Appendix L the prompt order of the in-context examples. The compute used for the experiments is detailed in Appendix M The remaining experiments were done with API credits received research grant from OpenAl and Cohere.
C Ethics Statement
In this work, we conduct a study with human subjects (see Appendix H for details). get matched with participants, we used the platform Prolific. Prolific complies with ethical standards according to UK law (e-g. complying with the GDPR). We compensated participants with UK living wage at 15 GBP an hour, which is 6 GBP an hour more than Prolific recommends at GBP per hour.
Implicature is an aspect of pragmatics, and pragmatic language impairments are universal in Autism Spectrum Disorder (ASD) (American Psychiatric Association 2013). Difficulties in understanding scalar implicatures are claimed to be present in people with ASD (Volden 2017). although the nature of the relation has proven hard to establish and has recently been debated (Katsos et al. 2011 Schaeken et al.) 2018). For the purposes of this work, whether or not implicature understanding relates ASD not important. We took the following steps to make sure sensitive data is collected or published. The human annotations we obtain are anonymous, related to a participant only by their Prolific ID for the purposes of compensation. In publishing the human annotations, we will not publish the Prolific ID of participants or anything else related to the participants. Additionally, did not collect or request any personal demographie characteristics of the participants apart from that they are all native English speakers.
Additionally, in this work we run lot of compute-intensive experiments. We publish the estimated emissions per experiment in Appendix M The total amount of GPU hours is estimated at maximally 966. How this broken down per experiment can be seen in Appendix M
18

---

D Opener example with InstructGPT
This quote was obtained through the OpenAl playground for text-davinci-002 on May 15th 2023, The model text-davinci-001 consistently generates better responses for the same prompt. GPT-3 itself (i.e. davinci) mainly gives nonsensical answers. In the following, the prompt italic and the completion bold. The completion was generated with maximum of 10 tokens and a temperature of O:
User: "Have you seen phone?" GPT-3: "Yes, have seen your phone."
The opener example is used introduce the problem looking at, and judge the model used to generate it. In fact, although text-davinci-002 sometimes completes conversations in way that unexpected according to pragmatic language usage, is one of the better models when evaluated few-shot.
19

---

E Background on implicature
The first influential consideration of implicature is Grice (975). In his work, Grice continues the trend of moving away from purely logical accounts of language started by |Witteenteinn (921) by hypothesising implicatures arise in conversation when some mutually agreed upon maxims seem to be violated. For example, if we agree on only making relevant contributions to conversation, Juan's response in the introduction seemingly violates this maxim-after all, he starts talking about work when Esther asks him about party. However, because Juan agreed to be relevant he must be implying that having to work means he cannot come to the party. Grice contrasts conversational implicatures that arise through context with conventional implicatures. These implicatures where the conventional meaning of the word determines what is implicated. An example given by Grice is the following sentence: "he is an Englishman: he is therefore brave.". " Grice notes that this sentence does not literally state that an Englishman being brave is a direct consequence of him being English, but it's implied by the conventional meaning of the word 'therefore'.
Since then, issues with the Gricean cooperative principle have been pointed out by many (Levinson 1983; SSeereer and Wilson 1986 Davis 1998 [epore and Stone, 2014). The most influential alternative theory is relevancy theory by Sperber and Wilson (1986). They do away with the cooperative principle and instead theorise implicatures arise because speakers try to produce ufterances that are both as relevant as possible and require the least effort to process. Another point of contention is the incorporation of conventional implicatures is no such thing as conventional implicatures, and they are simply instances of something else. Based on thorough treatment of what Grice calls conventional implicatures, Bach argues all examples of it can be filed under other concepts within semantics, like utterance modifiers (called "utterance modifiers" instead of "sentence modifiers" because they go against the semantic content of the rest of the sentence). (Pottss (2005) also argues that to explain conventional implicatures we can stay on semantic turf. Indeed, even Grice himself says conventional implicatures derive from the meaning of the words, not from conversational context. However, Potts does not claim conventional implicatures do not exist, but instead argues they arise by combination of lexical meaning and novel ways of combining words- the latter being the well-known principle of compositionality. an important part of semantics, not of pragmatics. PPott provides us with an illuminating demarcation between conventional and conversational implicatures. Conventional implicatures are never negotiable by context, whereas conversational implicatures are context-dependen and can always be cancelled without causing incoherent discourse. Consider again the sentence "he is an Englishman; he is therefore brave." " and the sentence "Eddie has three bicycles" (implicating that Eddie has exactly three bicycles and not more). The former sentence can not be cancelled by new context without contradiction, whereas for the latter, if we continue saying "In fact, Eddie has 10 bicycles, he is bicycle junkie", we have cancelled the implicature. This demarcation clearly puts conventional implicatures on the semantic side, and conversational implicatures on the pragmatic side. Potts goes on by providing formal theory for conventional implicatures.
the pragmatics side. [Bach| (1999) argues that there
In later work. Potts (2006) describes how pragmatic pressures interacting with context cause conver- sational implicature to arise. He shows how sensitive conversational implicatures are to small changes in the context. Novel information about a speaker's belief state might completely change what is implied. There are many more models of implicature that aim to explain how humans understand language in context. Most notably. Frank and Goodman (201) formalise the view that speakers produce utterances that are helpful and not longer than necessary with Bayesian model called the rational speech act (RSA). Many variants on the RSA framework have since been proposed. For example, Goodman and Frank (2016) extend it to handle nonliteral uses of language, like irony, and metaphor. In the context of computational models, prior work uses insights from pragmatics to show that the use of certain words make language model produce biased completions (Patel and Pavlick (02T)). e.g. saying someone "claimed" something rather than "said" something), and inform bias and sentiment classifiers (Greene and Resnik 2009; Recasens et al., 2013).
In this work, we focus on conversational implicatures and not on conventional implicatures. All conversational implicatures are negotiable by context, but the way they depend on context can be different. Grice (1775) identifies generalised conversational implicatures and particularised conversational implicatures. The former require little or no context to be resolved. For example, "some athletes smoke" can imply "not all athletes smoke", but might also imply "I do not know whether all athletes smoke" when it is a response to the question "do you know whether all athletes
20

---

smoke?" (Davis 2019). The latter only arise in certain contexts. For example, the response "I have an early morning to the question "do you want to stay for a drink?".
21

---

Detailed prompt templates
Table contains the full prompt templates we used for the main evaluation and Table contains the extra prompt templates.
Table 5: Ranking prompt templates. The six templates wrap the test examples in to present the models. Templlte 3, and are of the type structured, and 2, 5, and 6 of the type natural. Within the type of prompt template they only differ slightly in wording.
Prompt template
Does the following response to the question imply yos or no?
question: <utterance>
response: <response>
implicature: <implicature>
Finish the tollowing text:
Esther asked "<ufterance>" and Juan responded "<response>", which means <implicature> Is the implied meaning of the following response no:
question:
<utterance>
response: <response>
meaning: <ómplicature>
What is the intent of the following response, yes no?
question: <utterance>
response: <response>
intent: <implicature>
Finish the following text:
Karen asked "<utterance>" and William responded "cresponse>" which means <implicature>
Finish the following text:
Bob asked "<utterances" Alice responded "<response>", which means <implicature>
22

---

Table 6: Completion prompi templates. The six adjusted templates wrap the test examples in to present to the models when we use completion instead of likelihood ranking. Template 1, 3, and are of the type structured, and 2, 5, and 6 of the type natural. Within the type of prompt template they only differ slightly in wording.
#
Prompt template
Does the following response the question imply yes no? Only output "yes" 'no'. Even you'l uncertain, choose either 'yes' 'no'.
question: <utterance>
response: <response>
implicature: <implicature>
Finish the following text. Only output 'yes' or 'no', Even if you' re uncertain, choose either "yes" or "no'.
2
Esther asked "<utterance" and Juan responded "<response>", which means <implicature> Is implied meaning the following response yes or no. Only output "yes" or 'no". Even you'te re uncertain, choose either 'yes' or 'no'.
question: <utterance>
response: <response>
meaning: <implicature?
What is the intent of the following response, yes or no? Only output "yes" or 'no".
Even if you're re uncertain, choose either 'yes" or 'no'.
4
question: <utterance>
response: <response>
intent: <implicature>
Finish the following Only output 'yes' 'no'. Even if you 'urre uncertain, choose either "yes' 'no'.
Karen asked "<utterance>" and William responded "<response>", which means <implicature> Finish the following text. Only output 'yes' 'no'. Even if you're uncertain, choose either "yes' or 'no'.
Bob asked "<utterance?" and Alice responded "<response>", which means <implicature>
23

---

Table 7:
The three additional templates wrap the test examples in to present the models, adapted from Glaese et al. (2022). #
Prompt template
The following text shows an interaction between two humans called Esther and Juan. In the interaction, Esther will ask Juan a question, and Juan will give answer that contains an implicature.
An implicature an utterance that means something other than the literal meaning of the words. The implicature of Juan's 's response is yes or no. You, the Al assistant, asked finish the text with yes or no.
7
The task begins:
Esther asked "sutterance>" and Juan responded "cresponse>", which means <implicature>
The following text shows an interaction between two humans called Esther and Juan. In the interaction, Esther will ask Juan question, and Juan will give answer that has meaning besides the literal meaning of the words.
That meaning is either yes You, the Al assistant, are asked to finish the text with the correct meaning, either yes or no. The task begins:
8
Esther asked "sutterance>" and Juan responded "<response>", which means <implicature> The following text shows an interaction between two humans called Esther and Juan. In the interaction, Esther will ask Juan a question, and Juan will give an answer that has meaning besides the literal meaning of the words. That meaning either yes or no. You, highly intelligent and knowledgeable Al assistant, are asked to finish the text with the correct meaning, either yes or no. The task begins:
9
Esther asked "sutterance>" and Juan responded "<response>", which means <implicature>
24

---

Table 8: Chain-of-thought (CoT) prompt templates. One of the six chain-of-thouge tf-thuughtt prompt templates we use for the CoT experiment. Note that this is 5-shot prompt. Each prompt variation contains five CoT examples. The other five variations are separately added to the supplementary materials
Prompt template
Bob asks Alice question, and Alice responds with implicature. This means that Alice's response does not literally contain the answer to Bob's question, but implies an answer. Assuming that Alice a cooperative conversational partner, what is the implicated answer to the question? For example:
Bob: You invented fire?
Alice: told you that.
Implicature: Alice says 'I told you that'. Alice's response must be relevant Bob's question because Alice is cooperative conversational partner. Alice must mean that she told Bob that she invented fire. Alice's response Bob's question 'You invented fire?' is yes.
Answer: yes
Bob: That cake looks delicious. Aren't 't you going to have some with me?
Alice: But that was huge meal we just had.
Implicature: Alice's 's response must be relevant to Bob's 's question because Alice is cooperative conversational partner. Alice must mean that the meal they just had was so huge she is not hungry anymore, and this must be relevant to Bob's s question: 'Aren't you going to have some with me?" Alice's response the question must therefore be no.
Answer:
Bob: Could you perform well?
Alice: Being bilingual would help put me ahead of the pack.
Implicature: Alice says being bilingual would help put her ahead of the pack. Alice's response must relevant to Bob's 's question because Alice is cooperative conversational partner. Alice must be implying that she could perform well because she is bilingual. This means the response to Bob's question is yes. Answer: yes
Bob: Have you any news for me?
Alice: I've made progress
Implicature: Alice says she has made progress. Alice's response must be relevant Bob's question because Alice is cooperative conversational partner. If Alice would not have any news for Bob, Alice would have said she would have made progress because that would be misleading. The answer to Bob's question 'Have you any news for me?' must therefore be yes. Answer: yes
Bob: You looked out for him?
Alice: He looked out for me. He taught me.
Implicature: Bob asks Alice 'You looked out for him?' and Alice's response says that the person that is being referred to by 'him' here looked out for Alice. If Alice meant yes to Bob's question, Alice would have said something like 'he also looked out for me'. Stating response like this implies that the answer to Bob's 's question is no. Answer: no
Only output a 'yes" or 'no' as a final answer. Write your reasoning after 'Implicature." and then output either 'Answer: yes' or 'Answer: no'.
Bob: <utferance>
Alice: <response> Implicature:
25

---

G Model categorization
Table 9 contains details on the model classes that are part of each group of models evaluate, along with their model sizes.
Table 9:
Model categorization for each of the models. DL stands for dialogue, FT for fine-tuning, BL for benchmark-level, EL for example-level, and IT for instruction-tuning
Group
Model class
Model IDs
Model size
Instruct
BERT
base uncased
110M
No
RoBERTa
base, large
125M, 355M
No
GPT-2
GPT-2 medium, large,
354M, 774M, 1,6B
No
EleutherAI
GPT-J, GPT-NcoX
125M, 1.3B, 2.7B, 20B
No
Base
BLOOM
560M, 1B1, 3B, 7B1, 176B
No
OPT
125M, 350M, 1.3B, 13B, 30B, 66B, 175B
No
Cohere
small, medium, large, ada, babbage, curie, davinci
409.3M, 6.067B, 13.12B, 52.4B
GPT-3
Est. 350M, 1.3B, 6.7B, 175B
No
DL FT
BlenderBot
90M, 2.7B, 9.4B
No
TO
11B
Yes
BL. IT
Flan-T5
780M. 3B, 11B
Yes
Cohere-command
medium, xlarge
6,067B, 52.4B
IL
text-davinci-001
ada, babbage, curic, davinci-1
Unknown, left-to-right increasing in size
Yes
text-davinci-002
Unknown
Yes
text-davinci-003
Unknown
Yes
ChatGPT
gpt-3.5.turbo
Unknown
Yes
GPT-4
gpt-4
Unknown
Yes
26

---

H Human evaluation
The participants for the human evaluation in this paper were recruited using Prolific www..prow prolific co). The setup of the experiment is as follows. We divide the test set of 600 examples into four non- overlapping subsets of 150 examples. Each set of 150 examples This means each example in the test set is labeled five times by different people, and we have in total twenty annotators for the whole test set (five different ones for each of the four subsets). The only constraint for the annotators is that they are native English speakers. In Figure the screen shown to potential participants
given to five unique annotators.
Prolific is shown. Participants are paid 15 pounds an hour, which was the living wage at the time of the experiment and more than the 12 dollars an hour Prolific recommends. The total amount spent on the human evaluation is 236 pounds. This amount came to be from four subsets, each costing about 30 minutes to label per annotators, and having 5 annotators per subset: 15 * 4 * 0.5 * 5 = 150. The extra costs were for the annotator that didn't pass the attention check which we paid nonetheless, and for prolific as a platform.
study about the implied meanings of responses
COLOOMN
The ain of thk skaly
compas pertenance of hamana bing rocilu dim GPF-)) inimptrtnnt ispect
given
Please Seat
5GB questices. you dan't know anowee the
subriiasion being rejcted,
coree ariusn Prolko aD tho fect cuestion, all coeoot evall acdimo methi compaition code fresh the 100 milioon, pasoo the othe kist We exset fil losk tunc anoudd vo iuhoa ten you'ne sute sblud the form will clack thet evaylhing hos been filee out and band you co tha cou usa thattic you Blsshwed tack.
Soeetiive antormatio
No aaro Sva information la rosiied to give in this bust and the arovers are complesaay anoeynt We dan't do anything with the information about except
malk the prodis the rerpo
Rewerd
the sevand pessiito other your repprilll withr day at best with
Desinso
stady
Denano
Mesale
Tatler
Open sludy in now window
Figure 5: A screenshot of how the experiment is presented to potential annotators on Prolific (www. prolific.co).
The 150 test examples are wrapped in prompt template (see Table and presented in a Google form. We opted wrap all examples in prompt template 2 to make the full human study directly comparable to the model's lsss results on template 2. If we had done mix of all templates we either had to spent six times much on the human evalyations (which was not within our budget) or subsample evaluations, making it less comparable to part of the model study. Although models have been shown to be very sensitive to prompt wording, humans are less likely to perform differently for different prompt templates. All templates are coherent natural language that any native English speaker will understand. That said, this is speculative, and to confirm this hypothesis future work should investigate the effect of different wordings on implicature resolution by humans. The participants are asked to choose the correct continuation, yes or no (see Figure (ag. As recommended by Prolific, we subject the participants to an attention test (see Figure 6b). At three random places in the form, we add a question that does not contain an implicature and obviously maps to "yes". In this way, if the participants fails at least two of these questions, we can conclude they were not paying attention and remove their answers from the result. In practice, this happened once and we decided to pay the participant regardless, but discard their results, which were close to random.
Table TO shows the performance of each annotator on the subset they annotated. The average human performance across subsets and annotators is 86.2% + 2.3, the best performance is 89.8% + 2.2, and the worst performance 83.5% + 1.5. The column "IAA" shows the average Cohen's Kappa coefficient which is the pairwise inter-annotator agreement for each annotator per subset. All agreements are substantial according to the interpretation guidelines for Cohen's Kappa (between 0.61-0.80).
27

---

A study about the implied meanings of responses
Eather osked Areef you scaredf and Juan resper ded "So nedinell whvch mears
Each the reperoee talluing cantenoes maans prc chocos right one
Yet
your uniquu
asked you 1nd Juan responded
Which amail can sand Completion Code once farm?*
Eatter saked tait dihiau teath and Jaan would whihh
Eather saked ta the dreas perfect?" and Juan responded it fir l glove:
No
whlch meens
(b) Part Google form participants are asked to fill out. The second question in this image is part of the attention test. Juan's response does not contain implicature but simply gives away the correct answer.
No
(a) The start of the Google form participants are asked to out for the human study.
Figure 6: Screenshots of the Google form participants fill out part of the implicature study.
Table 10: The performance of the human annotators on the subsets of the test set. Subset through 4 are non-overlapping and cover the whole test set. Annotator X for subset Y might be different human than annotator X for subset Z. IAA the average pairwise inter-annotator agreement (Cohen's kappa coefficient) between annotators per subset.
Annotator
1
2
3
4
5
Mean
Best
Worst
IAA
Subset 1
86.0%
92.0%
90.7%
90.6%
86.0%
89.1%
92.0%
86.0%
0.73
Subset 2
84.7%
83.3%
87.3%
86.0%
86.0%
85.5%
87.3%
83.3%
0.64
Subset 3
84.0%
85.3%
88.0%
86.0%
82.7%
85.2%
88.0%
82.7%
0.78
Subset 4
85.3%
82.7%
84.0%
82.0%
92.0%
85.2%
92.0%
82.0%
0.71
Total
86.2%
89.8%
83.5%
0.72
Std
2.3
2.2
1.5
0.1
Human souree of disagreement with ground-truth. We do an analysis of the source of disagreement with the ground-truth. We explicilly do not call this error, as in some cases examples might allow multiple interpretations, and both could be right. In other cases, the ground-trutt might be wrong.
Annotation errors and multiple interpretations: We analyse the examples for which most humans choose a different answer than the ground-truth. For 30 out of 600 examples in the test set, only one or zero people choose the same answer as the ground-trutt truth. Of these examples, most are annotated wrongly (18 of 30). For example: 'Are you busy?", 'T m drowning in work.', implicature: 'no'. Some are examples that can have multiple different interpretations (12 18), and the ground-truth answer likely just chooses one that is unnatural to humans. For example: 'You don't 't remember them?', II "Leave me alone!', implicature: 'yes'. of the 30 examples are particularised, and is generalised.
Examples for which all humans agree with the ground-truth: There are 409 out of 600 examples that all humans get correct. This set of examples contains most of the generalised implicatures (39 out of 47). These contain 58 out of 94 particularised examples.
Examples most humans agree with the ground-truth: When we look examples that 3 more humans got correct, that comprises most the test set (530 of 600), and all of the generalised examples (47 of 47). This subset has 78 of 94 particularised examples, for 16 particularised examples 3 or more humans disagree with the ground-truth.
28

---

I Comparison with BIG-bench implicatures task
One of the BIG-bench tasks is related the task in this work It uses the same underlying dataset we use (George and Mamidi, 2020). With the below we aim to discuss our contribution in light of the BIG-bench result. To summarise; the methodology used by the BIG-bench task contributors has limitations, which call into question the validity of their claims. Further, some of the BIG- bench results are irreproducible due to missing details in the tech report and the use of proprietary models. Considering this, our work is an important contribution validating the BIG-bench results in reproducible and methodologically sound way, and above that providing insight into what aspects of L.M training are crucial for the ability to do pragmatic inferences.
Limitations of the methodological approach of the task contributors in BIG-bench implicatures. Our benchmark has 30% more data, which the BIG-bench task contributors discard. In this section we motivate the crucial importance of that data for evaluating implicature understanding (Section I.I. and why BIG-bench in turn might be overestimating the performance of LLMs on implicature resolution (Section I.2] Moreover, the human performance on the BIG-bench task indicates low quality human annotation, which we will also elaborate upon below, noting that this is impossible to verify because the BIG-bench report does not detail how the evaluation was done for this task (Section L33
I.1 Discarding ambiguous examples
The BIG-bench task preprocesses the 1001 examples that George and Mamidi (022)) curate by keeping only yes/no questions, discarding any examples that are ambiguous according to the task contributors, and discarding remaining examples to create 50/50 distribution of yes/no answers. This leaves them with 492 examples. Of these examples, 81 appear in our development set and the remaining 411 appear in our test set. Our test set has 600 examples, so BIG-bench effectively discarded 189 ambiguous examples compared to our test set; a bit more than 30% of the benchmark. To illustrate the importance of not discarding this data, we cherry picked a few examples that the BIG-bench authors discarded from the data.
Utterance:
"Can you lend me hundred dollars?", Response: "Is this supposed be some kind of a joke?", Implicature: "No"
"Do you know, how long is Uncle Arthur staying with us?", Response: "Ask your father.". ", Implicature: "No"
Utterance:
Indeed, these examples are ambiguous. Asking whether the request for a hundred dollars is a joke does not literally mean you're saying to the request. The response "ask your father" does not mean the speaker does not actually know, maybe they just do not want to respond. The humans in our study all infer the intended ground truth implicature. This shows a general property of implicatures; they are ambiguous, but often humans do infer the intended meaning. Ambiguity is not discrete property, Some examples may be so vague that no one gets it. The following are examples the BIG-bench task discards that the humans in our study did struggle with:
Utterance: "Got any more of those?", Response: "Nothing I'm at liberty to reveal here.", Implicature: "Yes"
"Have you finished sight-seeing?", Response: "Sorry. I should've 'e come to see you first.", ", Implicature: "Yes"
Utterance:
In the first of these the implicature is "yes" because the person responding is implying that they do have more, they just cannot reveal them. Otherwise they would most likely simply say no. In the second example it feels more natural that someone says this when they are finished sight-seeing, otherwise they would' ve probably said something to the effect of m still out, but I'm sorry... In any case, humans in our study did not understand these responses like that. This illustrates another aspect of implicature; sometimes communication will go wrong over it. Removing implicatures that are ambiguous though, defeats the purpose of the task, they are all ambiguous to a certain degree. The purpose of this study is to compare how humans resolve this type of non-literal language
htg.a//cw...ootoooooooo....oaane taskss
implicatures
29

---

compared to how models do it. The human baseline of 86% accuracy that humans achieve on our test set deals more naturally with examples that are too ambiguous for models understand than discarding examples based on the subjective opinion of a few people.
1.2
Overestimation of performance on implicature understanding
On the overlapping part of our test set and theirs the humans in study achieve 92.8% accuracy. The best model on the BIG-bench task is PaLM, achieving zero-shot performance of 64.4%. Note that this performance is on their full test (not the overlapping part) and hence not directly comparable. Nonetheless, the missing examples are randomly sampled for our development set, and we can be pretty confident this number indicates large gap with human performance. Two-shot PaLM comes very close to human performance with 91. 91.7% accuracy, but of course this does not take into account the 189 more challenging examples that are part of our benchmark. Humans achieve 71.9% performance on this subset of ambiguous data, indicating that these data are more difficult than average, but nonetheless performance is higher than random. Without access to the models used to evaluate the BIG-bench task we cannot say anything for certain, but expect the performance of PaLM be overestimated as does not account for large part the type of implicatures found in natural discourse; ambiguous ones.
I.3 Other limitations
Poor quality human evaluation. The average human evaluator on BIG-bench implicatures achieves around 82% performance (where ours achieves on average 86% more challenging dataset), and their human best rater achieves 100% (where our human best 92%). This difference between human average and best hints poor quality average rating. This is impossible to verify because there no information the BIG-bench tech report on how the human evaluation was done exactly, or even which examples where evaluated.
No fine-tuned models. BIG-bench uses only base LLMs and no SotA fine-tuning methods. A question that remains is therefore, what aspects of LLMs contribute to their performance on implicatures? In our work we find that implicature performance emerges much smaller scale in models instruction fine-tuned at the example level, and that scale and prompting techniques are important.
30

---

J Chain-of-thought completions by GPT-4
GPT-4 reaches human-level performance with CoT prompting. In this section we look at few completions to help illuminate how CoT-prompting improves implicature resolution.
Specifically, we look at of the examples for which GPT-4 5-shot gets 0% accuracy (i.e. wrong for all templates), and GPT-4 5-shot CoT gets 100% accuracy (i.e. right for all templates). This is of 10 examples. We only look the first template, but all templates get similar performance with COT.
Utterance: 'Do you sell calculators?"
Response: "We are all sold out. We wont probably have any more in for a few weeks." Implicature: Yes. But we dont have any calculators to sell now.
This is an example that GPT-4 gets wrong for all templates 5-shot, probably because the response has negative sentiment, and also it means they do not sell calculators right now. However, the implicature is yes, because they do sell them normally. The CoT completion is:
COT completion: 'Alice says they are all sold out and won't have any more in for a few weeks. Alice's response must be relevant Bob's 's question because Alice is cooperative conversational partner. Alice must be implying that they do sell calculators, but they currently don't 't have any in stock. The answer Bob's question is yes. Answer: yes'
This example is similar to the one given in the main text in Section 4, and there are more like these in the dataset: the sentiment is negative, and interpretation can be negative, but strictly the implicature positive (yes).
Utterance: 'Do you know where he is?" Response: 'I'll find him.'
Implicuture: No.
GPT-4 gets this wrong for all templates 5-shot, and right for all of them with CoT:
COT completion: 'Alice says she will find him. Alice's response must be relevant to Bob's s question because Alice is cooperative conversational partner. If Alice knew where he is, she would have provided that information directly. Instead, she says she will find him, implying that she does not currently know his location. The answer to Bob's 's question 'Do you know where he is?" must therefore be no. Answer: no'
It's unclear why GPT-4 gets this one wrong 5-shot (all humans get this one right), but the CoT is very relevant and helpful.
Utterance: 'It seems Cathy is not happy at all with her new room mate. Did you ask her why?' Response: 'Believe me. tried find out. But simply could not pin it down.'
Implicature: Yes. But Cathy did not give specific reasons for her feelings.
Again example where the sentiment of the response seems to hint at Of 5 human annotators, also only 2 interpret this response as "yes". With a CoT, GPT gets right six times:
CoT completion: 'Alice says she tried to find out but couldn't 't pin it down. Alice's response must be relevant to Bob's 's question because Alice a cooperative conversational partner. Alice must mean that she attempted to ask Cathy why she is unhappy with her new roommate, but was unable to get clear answer. The answer to Bob's question 'Did you ask her why?' must therefore be yes. Answer: yes'
helpful reasoning trace. All CoT completions by the models we have run CoT on are available in the GitHub: https:://ithhn com/ aat.arraaiii/ooppii.ss.s..
31

---

K
Additional results
K.1
Contrastive experiment
In this section we reframe the implicature resolution task to a contrastive one, allowing the model to contrast the coherent the incoherent sentence in single prompt.
Contrastive task. the ranking task the model required to assign higher likelihood to the coherent utterance than the incoherent one (pa (x) > pe (&)). In assigning a likelihood to the model has no knowledge &, and vice-versa. We hypothesize that the task might become easier we reformulate it contrastive task. Consider the following prompt
Which of the following sentences is coherent:
A: Esther asked "Can you come my party Friday?" and Juan responded "I have to work", which means no.
B: Esther asked "Can you come to my party on Friday?" and Juan responded "I have to work", which means yes.
Answer:
We can now evaluate the models' ability to understand which is the coherent sentence by evaluating whether it assigns pe (A p) > pe (B p). Note that this can again be framed in a ranking task of assigning higher likelihood to the coherent prompt. If we finish the above prompt by adding "A" to make a coherent prompt x and "B" to make an incoherent prompt & we can again formulate the task by pe (x) > pa (8). The difference is that within both the coherent and the incoherent prompt, the model can contrast the coherent and incoherent utterance to each other. We randomise the assignment of A and B to the utterances.
We do a small experiment with the contrastive task with one of the best performing models overall, OpenAl's I's -0viniinni,- for k = {0, 5,). We use two prompt templates and for each template try three different multiple choice answers: and B like above, one and two, or the full text of the answer. For the last option the coherent prompt x would look as follows:
Which of the following sentences is coherent:
A: Esther asked "Can you come to my party on Friday?" and Juan responded "I have to work", which means no.
B: Esther asked "Can you come to my party Friday?" and Juan responded "I have to work", which means yes.
Answer: Esther asked "Can you come to my party on Friday?" and Juan responded "I have to work", which means no.
Table 11: Performance on the implicature task framed contrastively by OpenAl's text-davinci-002. The mean and standard deviation are reported over two different prompt templates (template I and 2).
k
Non-contrastive
Rank one, two
Rank A, B
Rank full text
0
71.3% 1.75
53.9% + 0.9
59.3% + 1.3
48.9% = 0.6
1
76.1% = 2.6
59.4% + 1.6
63.2% + 2.0
66.9% = 0.9
5
80.5% + 2.3
61.4% + 1.3
64.0% + 1.3
67.9% = 2.1
In Table I1 perhaps surprisingly, we can see that the contrastive task is much more difficult than
the original ranking task. For & = 0. the result is random except for the prompt where the multiple choice options are and B. For k = (1, 5} the full text ranking does best, but still significantly worse than the original ranking setup. Because of these disappointing results, we did not evaluate the other models contrastively. Future work must establish whether the contrastive setup is worse across all model classes and sizes.
32

---

K.2 Variance over prompt ordering
As mentioned in Section 3 in the main text, models are sensitive the ordering of the k examples in the prompt. Instead of marginalising over this random factor by evaluating all possible prompt orderings, we randomly sampled an ordered set of examples from the development set for each test example. Throughout experiments, we kept this randomly sampled order the same, meaning you re-run the 5-shot evaluation you get exactly the same orderings. The reason for this is that want evaluate each model equally. In this section we ask how the performance chances for the best performing model if select another random order. We do this for the 5-shot evaluation, because the results show that adding more in-context examples barely helps performance.
Table 12: Variance over prompt ordering for 5-shot evaluation per prompt template (P.T.) for text- davinci-002
Seed
P. T. 1
P. T. 2
P. T. 3
P. T. 4
P. T. 5
P. T. 6
Mean
80.17
78.17
82.83
80.50
79.17
76.50
79.56
80.17
76.17
81.33
81.83
76.00
76.33
78.64
79.50
78.17
81.17
80.17
78.17
76.50
78.94
mean
79.94
77.50
81.78
80.83
77.78
76.44
std
0.31
0.94
0.75
0.72
1.32
0.08
Table 12 shows the results of this experiment. Some prompt templates seem be more sensitive to prompt example ordering than others, but for none of them the variance is high enough to change any conclusions.
K.3 Different zero-shot instruction prompts
There is narrative around large language models that if they fail task, it might be that the prompt was not the right one (through works like Reynolds and McDonell (2021b); Kojima et al. (2022)). The idea is that they can be prompted to simulate almost anything, if you set them up correctly. Because implicature resolution is a ubiquitous result learning language, we hold the view that model should be able to do this task if prompt is given in coher- ent natural language. Nonetheless, in an additional effort to find the "let's think step-by-step" (Kojima et al.] 2022) of zero-shot implicature resolution We evaluate a base large language model and two in- structable models: GPT-3-175B, text-davinci-001, and text- davinci-002. The prompts we use are taken from recent work that proposes dialogue agent trained with human feedback (Glaese et al. 2022), but adapted to the task of implicature resolution. The full prompts are presented in Table 7 and Table 13 shows the results. The new templates do not improve performance for any of these models. The variance over the prompt templates for text-davinci-oU0 high, and the best prompt template these three does achieve a slightly higher accuracy than the others: 74.5%. These results do not change the picture sketched so far.
try three more prompt templates. Table 13: Zero-shot accuracy over three additional prompt templates for a base LLM and two instructable models.
Model
Templates
GPT-3-175b
59.2% 4.5
text-davinci-001-2
66. 1% 3.2
text-davinci-002-?
67.7% + 9.6
is
K.4
The effect of in-context examples on sensitivity to prompt wording
Figure the relative performance increase due to in-context prompting broken down per prompt template. For text-davinci-001, most templates benefit similarly from more in-context examples, except for template 1. Perhaps surprisingly, we see that this template already achieves a performance of 76.5% at the zero-shot evaluation and does not improve much with few-shot prompting. For Cohere-52B and OPT-175B we see a clear grouping between the structured prompts (dashed lines) and natural prompts (dotted lines). Cohere struggles significantly more with the structured prompts than with the natural prompts in the zero-shot evaluation, and few-shot prompting can mitigate that, lowering the standard deviation over prompt templates to 1.89 = 30 from benefits from prompting for the natural prompts, but not for the structured prompts.
= OPT
33

---

Relative accuracy (w.r.t. 0-shot) in-context examples prompt templates.
0P7:1758
Prompt tompiate
15
Prompt tomplste
Prompt template
Relative Accuracy
10-
Frompt template
-5
15
k-shot
k-shot
k-shot
Figure 7: Relative performance increase over 0-shot due to in-context prompting, Structured prompt templates dashed lines (1, 3, 4) and natural prompt templates dotted lines (2, 5, 6).
K.S Variance over API runs
In this section we comment on the reproducibility of research done using APIs OpenAl and Cohere have their models behind an API, meaning we do have control over what happens the prompt before the model processes it. We run the zero-shot evaluation ten more times for two models of shown
OpenAl and Cohere, text-davinci-002 and Cohere-52B. The results from this experiment in Table 14 and [5| From this we can conclude that there is some stochasticity in the API that we have no control over, bit more for OpenAI than for Cohere, but again can be relatively confident that the conclusion will not be different because of it. The results from this work are therefore reproducible with access the same models behind the API now. Unfortunately. when OpenAI or Cohere changes the models behind the API, these results are not exactly reproducible anymore.
For completeness, add the timestamp that each result obtained below (Appendix D
Table 14: Results per prompt template (P.T.) for 10 different from text-davinci-002 for O-shot evaluation.
Each evaluation has exactly the text, so the variance in performance due to API stochasticity.
API-run
P. T.
P. T.2
P. T.3
P.T.4
P. T.5
P. T. 6
Mean
0
73.50
68.83
73.00
71.17
67.17
68.83
70.42
1
73.83
69.00
72.83
71.50
67.67
68.33
70.53
2
73.67
68.67
73.17
71.33
67.50
68.50
70.47
3
73.83
68.17
73.17
71.00
67.67
68.17
70.33
4
73.67
68.83
73.33
71.17
67.00
68.33
70.39
5
73.83
68.50
73.00
71.00
67.00
68.17
70.25
6
73.67
69.00
73.00
71.17
67.33
68.50
70.44
7
73.67
68.67
72.83
71.33
67.50
68.67
70.44
8
73,83
69.17
72.83
71.17
67.33
68.00
70.39
9
73.50
68.50
72.83
71.00
67.50
68.67
70.33
10
73.67
69.50
73.00
71.33
67.50
68.50
70.58
mean
73.70
68.80
73.00
71.20
67.38
68.42
std
0.12
0.35
0.16
0.16
0.23
0.24
K.6
Experiment with random in-context labels
This paper presents the thesis that instruction-tuning at the example level ("Example IT"" is important for pragmatic understanding in LLMs. However, the 0-shot result that one of the models in the Example group achieves is similar to that of base models; Cohere-command-522 obtains a zero- shot performance of 60.2%. From the sharp rise in performance observed for the k 0 to A = 1 result (from 60.2% to 72.8%) we hypothesise that the k-shot in-context examples in this task do not necessarily teach the model pragmatics in-context, but prime the model for the task format (namely,
34

---

Table 15: Results per prompt template (P.T.) for 10 different runs from Cohere-52B for O-shot evaluation.
Each evaluation has exactly the same text, so the variance performance due to API stochasticity.
API-run
P. T. 1
P. T.2
P. T.3
P. T.
P. P.T.5
P. T. 6
Mean
0
56.00
62.67
54.33
54.00
62.17
62.17
58,56
1
56.00
62.83
54.33
54.00
62.33
62.33
58.64
2
56.00
62.83
54.33
54.00
62.17
62.33
58.61
3
56.00
62.83
54.33
54.00
62.17
62.33
58.61
4
55.83
62.67
54.33
54.00
62.17
62.33
58.56
5
56.00
62.83
54.33
54.00
62.17
62.17
58.58
6
56.00
62.83
54.33
54.00
62.17
62.17
58.58
7
56.00
62.67
54.33
54.00
62.33
62.17
58.58
8
56.00
62.83
54.33
54.00
62.00
62.33
58.58
9
56.00
62.83
54.00
53.83
62.17
62.17
58.50
mean
55.98
62.78
54.30
53,98
62.18
62.25
std
0.05
0.08
0.10
0.05
0.09
0.08
outputting either "yes" "no" as detailed in Section in the main text). this hypothesis is true, we would observe similar performance regardless of whether the labels given in the prompt for the few-shot examples are true. We test this empirically for two base models (GPT-3, Cohere-52b) and two Example ITT models (text-davinci-001 Cohere-command-5200 for 1-shot and 5-shot evaluation. The results can be found in Table [6] We find that for the Example IT models in-context prompts with random labels obtain the same results (i.e. within confidence intervals) the experiments with ground-truth labels the in-context examples. For base models however we do observe drop in performance; for GPT-3-175b 5-shot, and Cohere-52b both at 1- and 5-shot. Taken together, can conclude that for base models the content of the in-context prompt seems important, whereas for models in the example IT group the in-context examples mainly serve primer for the task structure.
Table 16:
The results of the 1- and 5-shot experiment with random labels for the few-shot examples as opposed to the the true labels. We find that performance does not degrade for the models in the Example IT group, which implies that for these models not the content of the examples is important for performance, but the structure.
Model
1-shot
I-shot rand labels
5-shot
5-shot rand labels
GPT-3-175h
65.7% + 1.4
65.4% + 1.2
68.7% + 1.5
64.7% + 1.9
Cohere-52b
63.0% + 3,8
58.3% 3.3
65.1%
+ 2.9
60.5% + 1.9
text-davinci-001
72.7% + 1.3
73.9% + 1.7
74.5%
+ 1.0
73.4% + 1.2
Cohere-command-52b
72.8% + 1.3
72.0% + 1.6
75.4% + 1.8
73.5% 2.7
35

---

Table 17: Results of the chain-of- of-tthught (CoT) experiment for models in the base group. The numbers between brackets show the difference in performance with the number on the same row one column to the left. These models do not benefit from CoT prommpptn The reason Cohere- 6b achieves such a low score for CoT-prompting is because it is not able to adhere to the correct output format (yes/no).
Model
0-shot
5-shot
5-shot CoT
GPT-3-350m
51.5% + 3.0
55.7% 1: 1.6 (+4.2%%
55.0% 1 3,5 (-0.7%)
GPT-3-1.3b
57.7%
3.1
62.6% 2.0 (+4.9%)
54.4% 5.8 (-8.2%)
GPT-3-6.7b
54.8%
+ 1.9
62.4%
1.5 (+7.6%)
61.0% 1 2.3 (44.0%)
GPT-3-175b
57.2%
+ 4.4
68.7%
1.5 (+11.5%)
60.3% 4.2 (-8.4%)
Cohere-6b
57.3%
+ 2.2
60.9%
+ 4.1 (+3.6%)
29.2% + 14.7 (-31.%%)
Cohere-52b
58.5%
+ 4.0
65.1% + 2.9 (+6.6%)
64.7% + 3.2 (-0.4%)
K.7
Chain-of-thought on base models
In the main paper we do CoT'' experiment on the models in the Example IT group. Base models also benefit from in-context examples, so it makes sense also try CoT prompting on these models. After attempting this for two of the model classes in the group, we decided not to apply this prompting technique the other models, because it decreases performance, sometimes significantly. See the results of the CoT experiment on the two base model classes in Table
K.8 Testing for spurious correlations
In this section, we do small scale experiment to test whether the benchmark has spurious correlations. Specifically, we run the benchmark with only the utterance or only the response input. Strietly, getting the implicature right from the response only does not always indicate spurious correlations, as some examples only need the response (e.g. rhetorical questions like 'do pigs fly?"). Utterance-only results do always indicate spurious correlations. We run this experiment for GPT-3.5-turke and GPT-4 O-shot and 5-shot (see Table 18 and T9).e[19.
Table 18: Results of running the benchmark with only the utterance as input, to test for spurious correlations with the label.
Utterance-only
0-shot
5-shot
GPT-3.5-Turbo
54.3% + 3.3
41.7% +: 12.4
GPT-4
48.9% 10.5
53.7% 0.5
Table Results of running the benchmark with only the response as input, to test what part the examples be resolved without the utterance.
Response-only
0-shot
5-shot
GPT-3,5-Turbo
59.2% + 4.7
58.3% 6.6
GPT-4
62.6% + 1.7
65.5% + 1.1
We find that models mostly perform random for utterance-only. so spurious correlations not seem to be an issue. For response-only, GPT-4 5-shot gets 65% accuracy. Some examples gets right are: "do fish swim?" and "let's hope so".
36

---

Table 20: An example from the dataset for each type of implicature found in the test set. The rightmost column shows the amount of that type we manually found in the test set.
Type
Example Utterance
Example Response
Impl.
Generalised
You know all these people?
Some. I've gotta get up early.
No.
Particularised
Want to stay for nightcap?
No.
94
World knowledge
Did you leave fingerprints?
wore gloves.
No.
23
Idiom
Would he fire me?
He's all bark and no bite.
No.
42
Rhetorical question
Can you drive that far?
Can fish swim?
Yes.
11
Other
383
K.9
Detailed results type label analysis
In the main paper we do an analysis of two types of examples that occur frequently in the dataset, namely generalised and particularised implicatures. Here, we detail the full taxonomy of types of examples occurring in the dataset and report detailed results for each type per model (see (see[2] until Table 34 below). In Table 20 the full taxonomy of the examples is shown, representing types of examples that occur frequently in the dataset. We manually labeled 217 examples of the 600 examples in the test set according to this taxonomy. The remaining 383 examples do not fall as clearly within category and are grouped together as type other. Generalised implicatures require little or no context to be understood. They are the simplest type of example in the test set, and generally imply the same thing ("some" almost always implies "not all"). Particularised implicatures, by contrast, do require context be resolved. For example, from Table we need the context that it is undesirable to stay up late drinking when one has to get up early (see in Appendix more on generalised vs. particularised). The type world knowledge requires knowledge of the physical world to be resolved. From the example in Table 20 we need to know that you cannot leave fingerprints when wearing gloves resolve this implicature. Idiom types contain an idiom metaphor that needs know or understand to resolve the implicature, and finally Rhetorical question types contain a question like "Is the Pope Catholic?", often requiring factual knowledge be resolved.
The following tables contain the detailed results broken down per example type: Table 21 Table 34
The most interesting pattern in this data is that for almost all models, even the best model (GPT-4 30-shot in Table there a significant gap between human-level performance on the particularised examples. This gap is larger than the gap for the other labels usually. Few-shot prompting can often mitigate this (c.g. for GPT-3-175b, Cohere-52b, and daxt-iavinc)-002), but not always (c.g. for GPT-4 the gap remains large for k 30). However, for GPT-4, chain-of-thouge
mitigate the gap as seen in Table 34 Where GPT-4 30-shot obtains 71.97% accuracy on the particularised examples (and humans 83.18%) GPT-4 with 5-shot CoT achieves 81.63% which is close human-level. We find that the particularised examples mostly benefit from CoT prompting. Namely, for the generalised type of examples, GPT-4 30-shot already achieves 86.23% accuracy and CoT improves this to 88.66%. which is a much smaller improvement than for the particularised examples.
K.10 Detailed results per model
This section contains the results used for the zero-shot and few-shot evaluation in the main text in Section 4, broken down per prompt template. See Table 35 until Table 84
37

---

Table 21: Accuracy per label for O-shot evaluation.
Model
Mean
World knowledge
Idiom
Rhetorical question
OPT-125m
50.92
50.00 +/- 2.17
51.52 +/- 9.96
57.58 +/- 10.05
OPT-350m
57.14
57.97 +/- 10.25
64.77 3/65
65.15 +/- 3.39
OPT-1.3b
60.36
+/-144/-5.88
68.94 5.52
59.09 4.55
OPT-2.7b
59.56
60.87 6.15
67.05 +/-
69.70 +/- 6.78
OPT-6.7b
60.33
59,42 6.95
59.47 +-47
53,03 +/- 19.93
OРT-13b
61.03
63.77 +/- 14.78
7/-86+
66.67 +/- 16.32
OPT-30b
61.47
+/- 6/.94 10.48
+/-.8
74.24+4-6.22
OPT-66b
61.33
60.23
59.09 18.74
OPT-175b
55.33
5/.07
54.55+/-9.19
63.64 21.64
BLOOM-560m
51.58
54.92
50.00 13.64
BLOOM-Ib1
51.17
50.00+/-2.11
50.38+/-11.7
53.03 5/-03+4-1222
BLO0M-1177
53.61
52.17
83.79
68.18 6/-184/66.94
BLOOM-3b
56.89
54.35 6/02
4/-448
63.64
BLO0M-7b1
58.67
63.77 4/-77+/-144 14.57
+/
6/.18+
BLOOM-176b
54.22
55.07 7.39
50-38
62.12+/-9.70
EleutherA1-125m
51.89
56-52
52.65 8.84
5/-24
EleutherA1-1.3b
53.14
41-95
53.03 +/-11/-11.99
6/
EleutherAl-2.7b
59.17
60.14 13.38
65.91 3/94
68.18 +/- 4.55
EleutherAl-6b
56.36
57.25
56.06+-8..8
50.00 +/- 17.99
EleutherAl-206
57.53
3
07.80+5555
72.73 +/- 5.25
Cohere-409m
51.61
52.17 +2.17+44.35
54.55 +/- 12.86
Cohere-6b
57.28
55.80 +/-
60.23 5.988
72.73 +/-9,09
Cohere-13b
57.19
59.42 4/-881
10-1122
48.48 +/- 10.05
Cohere-52b
58.50
60.14 13.61
4/-15
74.24 +/- 11.03
GPT-3-350m
51.47
51.45+/33.90
53-41
50.00 +/- 13.64
GPT-3-1.3b
57.72
61.59+111.0
64.39 4/
65.15 +/ 3.39
GPT-3-6.7b
54.83
54.35+/66.99
4-.79
0/-12+/-3.33
GPT-3-175b
57.22
4-288
68.944-5.19
77.27+-8..1 8.70
TO-3b
48.25
54.35
42.42+/-4.29
36.36 0.00
TO-1lb
55.61
60.14 6.84
54.92+-14499
36.36 0.00
BlenderBot-90m
46.64
52.17
38.64
36.36+/00.00
BlenderBot-3h
53.44
47.83 +/.33/--000
61.36+-11.3
63.64
BlenderBot-9b
53.36
52.17 52.17+/-6.64
+/-881
63.64+/0.00
63.31
72.46 +/-4.10
4/
F5an-T5-3b
52.50
50.72 +/-
51.89 4/-4.23
+/-57
Flan-T5-11b
60.78
65.94+/-55
65.15 6/2
Cohere-command--6
66.31
72.46 7.80
78.41 4.3.
37.88 3/33
Cohere-command-526
60.22
66.67 +/- 10.85
63.64 10.33
77.27 +/- 6.94
text-ada-001--unnnown
56.50
63.77 4.10
58.71 4/-1+-110 16.04
51.52 +/- 10.05
eext-babbage-001-umnnon
64.47
67.39 +/- 6.02
76.52 +-52++111 1.69
60.61 +/- 10.05
text-curie-001-uuknnww
68.94
76.81 3.24
76.89 2.76
54.55 +/- 12.86
eext-daving--001-unnnown
72.31
84.78 7.43
78.79 +/
59.09 +/- 13.64
text-davinci-002-umnnoww
70.58
82.61 +/ 9.05
75.38 3.05
57.58 +/- 16.32
text-davine--003-uunnown
71.25
86.96 +/966+110 13.28
72.35 +/3
48.48 +/- 8.57
ChaIGPT-unknown
72.08
82.61 +/- 12.04
83.33 5.97
56.06 +/- 16.11
GPT-4-unknown
81.78
92.03 +/- 2.99
90.91 +/-
84.85 8.57
Humans
86.23
93.04
92.73
92.73
38

---

Table 22: Accuracy per label for O-shot evaluation.
Model
Mean
Particularised
Generalised
Other
OPT-125m
50.92
49.43 +/- 5.52
55.07 +/- 21.10
50.56 +/- 1.33
OPT-350m
57.14
47.92 4/-924/-4.37
69.20 4.36
8.36
56.68 4.99
OPT-1.3b
60.36
51-52+4/6.88
74.64 +/- 2.05
59.65
+/-1
OPT-2.7b
59.56
50.19
69.93 +/-
59.22 +/-6.4
OPT-6.7b
60.33
52.27+45555
75.36 +/- 2.71
60.77
OPT-13b
61.03
5/.494/-8.79
75.00 5/72
5/-79
OPT-30b
61.47
54.55+/4.115
71.38 +/94
61.11
OPT-66b
61.33
55.11+/77.33
6/-93
61.46+/-3.68
OPT-175b
55.33
54.17+-7.70
-8.33+-18.51
55.12+/-4.21
BLOOM-560m
51.58
50.76+/-5.59
48.91+/-25.22
5/.59
BLOOM-1b1
51.17
6..622
53.26+/227.40
51.16+-2.41
BLO0M-1677
53.61
5/-39
59.78+/-18.82
53.23+/-1.5
BLOOM-3b
56.89
5/.70
6/.39
36.46+/-4.33
BLO0M-7b1
58.67
46.594-2..6
7/.35
57.11 +/-4.33
BLOOM-1766
54.22
54.73+/-1066
6/.14
53.57+/-1.
EleutherA1-125m
51.89
5/.38+/-5.71
57-25
50.90+/-0.99
EleutherAl-1.3b
53.14
4/
55.43
53.32
EleutherAl-2.7b
59.17
50.95+/-7.43
74.64 4/9
58.10
EleutherAl-6b
56.36
5/.22
69.20 +/200+-7.36
55.73
EleutherAl-200
57.53
4/.43
72.83 +/-
56.20 +/20++33.00
Cohere-409m
51.61
4/-83
5/-544/
Cohere-6b
57.28
51-524/-6.66
64.494--16.06
57.06 +/13
Cohere-13h
57.19
52.27+/-5.88
68.12+/-10.99
57.45 3/35
Cohere-52b
58.50
5/.52+/77.21
+/-575
56.85 3/81
GPT-3-350m
51.47
50.76+/-6.96
52.90+/-24.07
4/-294/-15
GPT-3-1.3b
57.72
50.00+/6::29
67.75
57.06+/3.77
GPT-3-6.7b
54.83
8/-651
6/-4144-15.14
54.26+/-2.12
GPT-3-175b
57.22
53.03
71.01 +/-4.81
54.61+-5555
TO-3b
48.25
0.666
27.17
49.83+/11.90
TO-11b
55.61
4-.95
47.10+-11.15
56.33
BlenderBot-90mm
46.64
55.49+/-0.2
23.91 +/- 0.00
48.32
BlenderBot--b
53.44
44.514
76.09 +/00
51.81 #1-11/-00.20
BlenderBot-9h
53.36
49.244/-481
71.01 +/-
51.03 1.66
Flan- -lan-75-780m
63.31
3/-990
68.84
62.23 +-23+/-3.11
Flan-T5-3b
52.50
1.-
48.19
Flan-T5-11b
60.78
51.324/33.99
+/-128
Cohere-command-6h
66.31
58.90+/33.62
7/-194/-2.7
66-15+/-2.4
Cohere-command-520
60.22
55.494/-4.66
16.67
+/-99
text-adaa-011-umnnown
56.50
52.65 3/83.86
61.59 15.96
56.24+/-5.7
text-babbage-001-uunnnwww
64.47
56.25
72.46+--9.8
text-curie-001-uuknovw
68.94
66.48/-22.3
68.84 +/
08.48+/30
text-davinei-001-unnnown
72.31
59.66+/-5.01
79.35+/-9.
73.17
textcdavinci-022-umnnoonn
70.58
64,20 +/-3.75
80.07 5.67
19.94+/330
text-davinci-003--unnoww
71.25
6/-644/-1.8 1.86
82.25 +/-547
71.23 2/74
CharGPT-unknown
72.08
68.75 +/ 2.99
69.57 +/-
71.66 +/- 5.79
GPT-4-unknown
81.78
71.59 3/47
89.86 +/05
81.35 +/- 1.66
Humans
86.23
83.18
92.17
84.86
39

---

Table 23: Accuracy per label for 1-shot evaluation.
Model
Mean
World knowledge
Idiom
Rhetorical question
OPT-125m
52.72
43.48 +/- 5.02
54.92 +/- 7.24
59.09 +/- 4.55
OPT-350m
52.92
39.86 +/- 1.62
48.11 2.76
59.09 +/-
OPT-1.3b
56.31
54.35 4/16.
58.33 4.48
53.03 +/- 12.22
OPT-2.7b
56.83
64.49 +/- 15.35
+/-39---2.51 2/-1
66.67 +/-4.29
OPT-6.7b
60.08
61.59 +/- 13.84
+-.94+660
56.06 6.25
OРT-13b
60.56
68.84
6/-70+/-6.11
54.55 +/- 15.75
OPT-30b
60.33
4/-74
63.26
51.52 +/- 10.05
OPT-66b
63.19
70.29 +/- 11.06
2/
48.48 +/- 13.55
OPT-175b
58.36
63.77 4.8.1
66.67 4-7.7
57.58 +/- 17.14
BLOOM-560m
54.83
4/.02
63.644/-0.0
BLOOM-Ib1
52.56
4/.52+
59.47+/22.04
59.09
BLO0M-1177
52.81
54.35
60.984/-3.57
6/-64
BLOOM-3b
55.94
50.72 +/4.10
6/.39
59.09+/-4.5
BLOOM-7b1
57.00
50.00 +/33
6/-77
62.12+/-6:25
BLOOM-176b
61.11
77.54
6-.67
50.00+/-6.9
EleutherA1-125m
51.67
44.93 6/66
50.76+/-4.29
57.58 +/-4.29
EleutherA1-1.3b
55.72
50.00 11.44
EleutherAl-2.7b
55.50
54.35 +/- 5,47
67.42 4.67
65.15 +/15+
EleutherAl-6b
54.97
+/-
60.23
53.03 +/- 8.16
EleutherAl-206
55.86
69.57 4/3355
62.88 +/- 4.85
53.03 +/- 6.25
Cohere-409m
51.89
42.75 6.84
6.84
54.55 +/- 5.25
Cohere-6b
57.86
58.70 +/- 12.74
67.05 5.83
68.18 +/- 11.44
Cohere-13b
61.78
7/-74 11.43
37.88 9/-9.70
Cohere-52b
62.97
66.67 6.48
70.08 +/4..3
62.12 +/- 14.29
GPT-3-350m
55.97
50.72+/44.10
61-744/-5.15
69.70 +/- 12.49
GPT-3-1.3b
60.75
38.70+/441
+/5.55
GPT-3-6.7b
61.17
49-324-3..6
56.06+/-88
GPT-3-175b
65.72
76.81+/-322
57.58 +/- 16.32
TO-3b
48.89
54.35+/-
36.36
TO-1lb
47.78
52.17+/-0.00
40.53
+/-000000.0
BlenderBot-90m
49.94
55.07+/-6.2
47.73+/-92
41-52/-133 13.55
BlenderBot-3h
53.31
4/-83+/-000
61.36+-0.00
+/.00
BlenderBot-9b
52.53
50.72+/-9.61
57.20 12.95
66.67
62.89
+/494077.70
67.42 +/2 13.03
46.97 +/-
Flan-T5-3b
52.75
15.47
55.30 9.34
45.45 12.86
Flan-T5-11b
57.44
59.42 4/3.24
4/-48+/-12.4
Cohere-command--6
65.00
71.74 6.99
+/-594-3.115
36.36
Cohere-command-526
72.83
4/-33+/3.59
text-ada-001--unnnown
57.36
60.87 7.10
67.80 +/- 3.81
66.67 +/- 6.78
(ext-babbagg-0001uunnoow
63.89
6/-84+13.90
+-293
50.00 +/- 11.44
text-curie-001-unknnww
64.39
66.67 +/
68.56 9.94
36.06+-66.2
eext-daving--001-unnnown
72.72
93.48 +/-4.16
80.68 +/-2.8
57.58 +/- 12.49
text-davinci-022-umnnoww
75.61
91.30 +/52.51
87.12
+/-816
text-davine--003-uunnovn
74.31
90.58 5/2
82.20 +/- 1.56
54.55 54-554/-7.42 +/-
CharGPT-unknown
75.11
86.23 +/
85.61 +/
56.06 +/- 14.29
GPT-4-unknown
82.31
97.10 +/- 3.24
88.64 3/94
89,39 +/- 3.39
Humans
86.23
93.04
92.73
92.73
40

---

Table 24: Accuracy per label for 1-shot evaluation.
Model
Mean
Particularised
Generalised
Other
OPT-125m
52.72
48.30 +/- 1.83
60.87 +/- 13.04
52.89 +/- 1.04
OPT-350m
52.92
47.73 2/37
60.87 +/- 11.71
54.35 +/-3.10
OPT-1.3b
56.31
53.41 +/-2.71
52.17 +/-9.88
57.41 +/- 1.38
OPT-2.7b
56.83
49.81+/-55
+/-6633
55.17 +/994
OPT-6.7b
60.08
52.65
73.55
59.09 +/-
OPT-13b
60.56
13.034/11.82
71.01+-77.0
2/-56
OPT-30b
60.33
5/.87+133.11
70.65+/-8.01
59.26+/-4.61
OPT-66b
63.19
60.044/-4.12
4/-797
63.39
OPT-175b
58,36
+/3.96
4/.42
57.28 7.30
BLOOM-560m
54.83
43.944/3.112
66.30+/-8.21
54.82 1.85
BLOOM-Ib1
52.56
47.35+/3.11
63.04
51.16+/-15,
BLO0M-1b77
52.81
3/-55
67.03
51,29
BLOOM-3b
55.94
4/-274-11.21
55.12
BLOOM-7bl
57.00
49.62
77.17+/-1.66
55.56+/-3.57
BLOOM-176b
61.11
58.144/-3.31
+/-598
59.73
EleutherAl-125m
51.67
20.19
53.99+/88.82
+/-11/00.89
BleutherA1-1.3b
55.72
4/.67
57.97
5/.36+/-2.0 2/62
EleutherA1-2.7b
55.50
48.67 +/-4.4
65.22
54.22 2.79
EleutherAl-6b
54.97
1/-111-1.21
6/.30+/-4.82
54.01 +/336
EleutherAl-20b
55.86
53-03+/-355
6/.494/-6.36
53.83 2/42
Cohere-409m
51.89
5/-84
4/-659
5/-54 1.96
Cohere-6b
57.86
4/-166
2/-54
57.15 +/-5.08
Cohere-13h
61.78
74.28 +/-3.64
61.41 +1.41+/-1.8 1.84
Cohere-52b
62.97
6/-42/-8..18
5/-2044
6/-4.1
GPT-3-350m
55.97
10.76+/-1.82
73.91+/-7.99
54.31 1.92
GPT-3-1.3b
60.75
53.79+/22.88
6/.48-
61.07 +/-
GPT-3-6.7b
61.17
55.494/-6.83
72.10+/-2.64
60.29+/-4.00
GPT-3-175b
65.72
62.314/4.11
6/.86+
6/-
TO-3b
48.89
56.25
34.06+/-4.29
49.83
TB-1lb
47.78
56.44/-0.54
2/.54+/-1.02
49.22+/00553
BlenderBot-90m
49.94
52.46+/-4.27
14.57+/11566
50.00
BlenderBot-3b
53.31
44.514-00.47
76.09 +/-0.0
5/.59
BlenderBot-9h
52.53
+4.924
15.80+/-12.9
50.90+/-2.00
Flan- -
62.89
56.44/-3.32
0/.844-12.999
63.44
Flan-T5-3b
52.75
1/-11
44-20+/-5.98
52.41 +/-3
1-
57.44
1/-984/11994
+/-18885
57.28 +/-87
Cohere-command--h
65.00
60.61 3/-6.9
6/-
65.25 +/- 1.37
Cohere-command-522
72.83
6/-4244-2.88
+/-2.2
71.36 +/-66+-11 1.70
cext-ada-001-umknowe
57.36
46.97 +/7.
2/-69
55,90 3.11
text-
Nost-babbage-001-umnnne
63,89
38-52//-1..
7/-22
63.70 +/-0+111.10
text-curi--001-umnnoww
64.39
69.93 +/-3.2
64.04+/-5.79
text-davini-001-amnnoon
72.72
1/-166
76.81 +/-2.1
72.83 +/- 1.70
text-avvini-002-uunnoon
75.61
68.18 +/-2.86
2/-
75.32 +/13.14
eex-daavinci-00--amknoon
74.31
+/-044
80.43 +/-5.02
74.50 +/- 1.29
ChaiGPT-unknown
75.11
70.08 +/
78.99 +/5
74.46 1.19 1.19
GPT-4-unknown
82.31
74.43 +/- 2.60
86.96 +/-3.22
81.70 +/- 1.94
Humans
86.23
83.18
92.17
84.86
41

---

Table 25: Accuracy per label for 5-shot evaluation.
Model
Mean
World knowledge
Idiom
Rhetorical question
OPT-125m
50.22
44.93 +/- 3.24
57.58 +/- 7.73
57.58 +/- 4.29
OPT-350m
51.47
53.62 4.81
58.71 +/- 1.56
45.45 +/- 0.00
OPT-1.3b
58.03
68.84 +/- 8.48
63.26 4/266+-6600 6.21
30.30 8.57
OPT-2.7b
57.33
57.97 4/-4.81
66.67 4/-4.67
71.21 +/-3.39
OPT-6.7b
63.31
66.67 +/- 16.20
67.42
1-.2++16..33
OРT-13b
67.39
80.43
4/-944/-4.299
39.39+/-6.7
OPT-30b
65,64
34.784-88.
66.29
37.88+-66.25
OPT-66b
61.50
75.36+/-8.20
5/-30
36.36+/-7.42
OPT-175b
63.89
78.26+/-..1
65.15
4/.94+/-3.39
BLOOM-560m
53.75
44.20 +/-
+/-914
5/.55
BLOOM-Ib1
57.39
49.28 6/95
65.15
66.67 +-674
BLO0M-1177
54.44
61-594/-50
56.06
4/-94
BLOOM-3b
57.19
50.72 3/22
6/-64 1/4 12.86
BLO0M-7b1
54.50
50.00 2/12
6/.88+/1..9
69.70
BLOOM-176b
65.42
16.09+/6.00
4-882
43.94+/-33
EleutherA1-125m
49.56
50.00 +/33
50.38 +/4844
34.85 3/33
EleutherA1-1.3b
57.11
55.07 +/- 5.98
63.64
37.88 +/- 11.03
EleutherA1-2.7b
58.03
71.74 4.16
59.85 +/122
43.94 +/- 14.29
EleutherAl-6b
58.39
67.39 6.99
56.82
42.42 +/- 18.68
EleutherAI-209
61.14
65.22 8/70
4/-774-111.11
3/.30+/-8.5 8.57
Cohere-409m
53.39
47-83
59.47 4-322
+/- 3/-82 6.94
Cohere-6b
60.89
65.94 +/-8.88
66.67 10.05
45.45 +/-
Cohere-13b
62.47
81.88 +/-8.00
62.88 62.88+/-10.7 10.7
34.85 34.85+/-11.0
Cohere-52b
65.14
73.91 +/-5.61
4-.005
51.52
GPT-3-350m
55.72
46.38 +/-224
65-53+-1155
51.52+/-4.29
GPT-3-1.3b
62.64
72.46 10.55
6/.704
3/-88 1/.22
GPT-3-6.7b
62.39
76.81 +/- 14.57
6/-50
36.36
GPT-3-175b
68.72
82.61 4/3.35
7/.594-2.54
60.61 13.55
TO-3b
46.67
52.17 0.00
+/.00
36.36+/-0.00
TO-1lb
47.00
+/170
39.02
36.36
BlenderBot-90m
46.58
52.17+/-0.00
38.644/-0.00
36.36+/-0.00
BlenderBot-3h
53.36
47-83
0/-0.00
63.644-0.00
BlenderBot-9b
52.81
77-33/-4.33.
63.64+40.00
61.03
61.59 4/-4.4
70.08 +/-8+
42.42+/4.22
F5an-T5-3b
54.89
22-324/7.33
60.61 8/26
34.85 +/33
Flan-T5-11b
61.64
4/-844/66884
67.80
43.94 8.16
Cohere-commmnd--6h
68.56
77.54 9/88
78-79+/-5.5
39.39 +/-4.29
Cohere-command-526
75.42
87.68 3.90
44.09+-1.33
74.24 +/- 9.70
text-ada-001-uunnnow
57.61
52.17 +/- 3.55
64.39 +/-
62.12 8.16
(ext-babbage-001-mknnon
66.14
7/-74+/-22 2.17
+/-55
57.58 +/- 12.49
text-curie-001-umknnww
71.33
76.09 +/- 2.17
70.08 6/0
43.94 +/3.
eext-daving--001-unnnown
74.53
88.41 +/-
78.03 5/-
66.67 +/- 12.49
text-davinci-002-umnnoww
79.56
90.58 1.62
89.02 2.04
69.70 +/- 6.78
text-davine--003-umnnown
79.67
89.13 +/-2
74.24 +/- 11.03
CharGPT-unknown
73.89
86.96 4/-1.5
87.88 +/8.85
75.76 +/- 12,49
GPT-4-unknown
82.03
+/-
86.74 +/- 2.04
87.88 +/- 6.78
Humans
86.23
93.04
92.73
92.73
42

---

Table 26: Accuracy per label for 5-shot evaluation.
Model
Mean
Particularised
Generalised
Other
OPT-125m
50.22
47.35 +/- 3.12
60.87 +/- 13.28
48.84 +/- 3,71
OPT-350m
51.47
39.58 +/- 1.79.79
67.39 +/-3.07
51.38 +/-0.95
OPT-1.3b
58.03
36.06+/-3.08
57.61 +/-4.12
58.01 +/- 2.84
OPT-2.7b
57.33
47.35+/-3..
72.46
56.20 +/-360
OPT-6.7b
63.31
4/6.93
7/-6144-6.48
63.74
OPT-13b
67.39
60.23
64.86+/-2.92
6/
OPT-30b
65.64
59.85+
4/-42+
67.27+--4..
OPT-66b
61,50
56.444/33.51
58.70+/-9.64
63-65+/-3.99
OPT-175b
63.89
6-.55
5/.54+/-5.53
65.33
BLOOM-560m
53.75
2-.89
73.19+/-6.11
52.50
BLOOM-Ib1
57.39
48.67 +/3.4
7/.65
57.02 1.86
BLO0M-1b77
54.44
48.86
6/.14
54.74
BLOOM-3b
57.19
50.00
7/-444
5/-24
BLOOM-7bl
54.50
2--2.55
72.10+1-4:24
53.10+/-1.00 1.04
BLOOM-176b
65,42
49.28 +/-9.9
66.88 +/-3.22
EleutherAl-125m
49.56
44-32+/-5.76
56.88 4/-4.05
50.04+/-22
EleutherA1-1.3b
57.11
30.76+/-3.26
56.89 +/57
EleutherA1-2.7b
58.03
$1.52/--.44
61.59
58.61 +/- 1.39
EleutherAl-6b
58.39
49.62 4.6199
63-044/-5.00
59.91 5/04
EleutherAl-20b
61.14
4/-524-2.44
6/-59 12.78
63.48 5/22
Cohere-409m
53.39
68.48 +/-.82
Cohere-6b
60.89
52.65 2/12.14
6/.49
61.80+/-4.7
Cohere-13h
62.47
4/-661
6/-844
62.02
Cohere-52b
65.14
4/-007
68.12+/77.39
65.46+/-3.30
GPT-3-350m
55.72
1..70+/11.26
74.28+/-6.00
55.47+/-155
GPT-3-1.3b
62.64
49.24+/2.551
+/-393
64.38 +/-2.1
GPT-3-6.7b
62.39
21.70+
64.86+/-7.68
6/-38
GPT-3-175b
68.72
60.98+/-577
8/8.7
69.90
TO-3b
46.67
5/.68
23.91+/-0.00
48.32+-00115
TB-1lb
47.00
55.87
25.00
48.62 +/223
BlenderBot-90m
46.58
55.11+/-1.27
24-28+/-0.81
48,28 83.28+--02. 0.31
BlenderBot-3b
53.36
4/.324
76.09//-0.00
51.72
BlenderBot-9h
52.81
44.32+/00.93
7/.72+
50.95 1.02
Flan-T5 -an-0m-7780m
61.03
54.36+/33.50
71.01+-12.0 12.96
60.77 6.009
Flan-T5-3b
54.89
57.01 1.79
41-304/
55.47 4/-000
1-
61.64
+/-3.00
64.86//-17.00
61-76+/-5.2
Cohere-command-6h
68.56
60.23 5/05.00
74.28 6/-6.57
68.91 1.47
Cohere-command-522
75.42
+/-3.9
+/-.26
74.68 2.80
cext-ada-001-umkmowe
57.61
48.86+/-2.1
72.83 1.66
57.11 3.74
text-
Nost-aabbage-001-umnnne
66.14
+/-282
+/-281
66.06 0,8
text-curi--001-umnnoww
71.33
60.04//0..7
69.93 +/6..67
74.63 +/98
text-davini--01-amnnoon
74.53
60.80+/-3.7
+/-3.99
75.80 1.33
text-avvini--02-uunnovn
79.56
71.02 +/-2.76
87.68 1/-1
79.03 2/26
eex-ddavinci-00--amknoon
79.67
71.59 11-594/-1.8 1.8
87.68 +/-
79.33 +/- 1.12
ChaiGPT-unknown
73.89
69.51 +/8.800
73.91 +199++11..66 11.64
72.44
+/ 6.16
GPT-4-unknown
82.03
71.21 4/-2.91
87.32 3/-3
82.30 2.31
Humans
86.23
83.18
92.17
84.86
43

---

Table 27: Accuracy per label for 10-shot evaluation.
Model
Mean
World knowledge
Idiom
Rhetorical question
OPT-125m
52.89
55.80 +/- 8.48
55.68 +/- 8.78
66.67 +/- 6.78
OPT-350m
56.72
57.97 4/-4.10
60.61 3/86
65.15 +/- 16.11
OPT-1.3b
59.92
70.29 +/-
54.17 3/57
34.85 +/- 3.39
OPT-2.7b
58.03
52.17
63.64 +/-544
OPT-6.7b
63.28
71.01 4/
+/-53.4-6..7
53,03 17.73
OРT-13b
65.75
7/
63.26+/-4.23
50.00 10.0041-10.16
OPT-30b
63.36
78.99 4/4.64
4-.06
OPT-66b
60.81
71.01 4/87
5/.55
36.36+/-0.00
OPT-175b
60.75
76.81 4/14.10
58.33
BLOOM-560m
54.56
19.28+/-33
64.39+/3.12
63-644/-0.00
BLOOM-Ib1
57.31
4/-074
60.61
39.09+/88.77
BLO0M-1177
53.14
68.12+/-6.48
4-.45
59.09 +/- 1/.44
BLOOM-3b
59.39
54.35 +/-.66
6-.33
66.67+/-4.7
BLO0M-7b1
56.11
53.62 +/
65.53
69.70+/-6.78
BLOOM-176b
63.47
75.36+/-550
68.18
4/.42
EleutherA1-125m
54.39
58.70 +/-00/6.00
52.27 5.41
83.33 +/- 19,93
EleutherA1-1.3b
57.83
67.39 5,47
4-5.1
62.12 +/- 11.03
EleutherAA-2.7b
57.03
73.19 +/199/-2.99
55.68 5/337
66.67 +/- 13.55
EleutherAl-6b
57.64
64.49 4.990
51.14
56.06 +/- 16.94
EleutherAl-206
59.33
67.39
62.12
3/-88+/662 6/22
Cohere-409m
53.92
4/-044/55.44
46.21
51.52 +/- 11.34
Cohere-6b
58.72
66.67 +/-9.61
63.26
50.00 +/- 12.59
Cohere-13b
60.36
76.81 6/-6.48
56-06+/99.5
34.85 +/-399
Cohere-52b
63.31
72.46+/-525
4/-58
51.52 +/- 10.05
GPT-3-350m
57.72
53.62
63.644/-6.01
8/
GPT-3-1.3b
60,92
73.19 4-288
5/
48-48+/-8..
GPT-3-6.7b
63.94
71.01 7.80
6/.80+/1..56
GPT-3-175b
67.28
76.81 9/6.611
68.56
8/ 10.50
TO-3b
46.67
52.17
38.644/-0.00
36.36+/-0.00
TO-1lb
46.72
+2.174-0.00
38.64440.00
36.36
BlenderBot-90m
46.67
52.17+/-0.00
38.644/0..00
36.36+/-0.00
BlenderBot-3h
53.25
4/-06+
63.64+/0..0
BlenderBot-9b
53.36
+/.03+/-4.10
62.12+/3.112
63.64+/0.00
60.19
+3.04+/2.11
68.56 10.77
40.91+/44.
Flan-T5-3b
55.14
4/-594
58-71+/-10.33
0/-3600-0000
Flan-T5-11b
60.56
4/-39//-..4
40.91+/-4.55
Cohere-command-6h
68.22
78.99 5/25
74.62+-5.33
36.36
Cohere-command-526
75.64
88.41 3/24
84.85+42.55
66.67 +/-
text-ada-001--unnnown
57.36
64.49 7/28
4/.444-5.78
57.58 6.78
(ext-babbagg-0001uumnoon
63.53
+/-
73.11+-4.66
+/-184-4.55
text-curie-001-uuknnww
70.17
+3-33411155
76.14 4-44/2250
+/-255
eett-daving--001-unnnoon
74.97
2/12
83.33 +/-23
59.09 6.94
text-davinci-022-umnnoww
79.56
93.48 2.17
38.26+/-035
66.67 +/- 8.57
text-davine--003-umnnown
79.00
+/-339
85.61 +/3
66.67 +/2
CharGPT-unknown
74.28
84.06 +/- 6.48
86.36 +/4.6544.35
62.12 +- 11.03
GPT-4-unknown
81.31
94.93 2.99
86.74
89.39 +/- 3.39
Humans
86.23
93.04
92.73
92.73
44

---

Table 28: Accuracy per label for 10-shot evaluation.
Model
Mean
Particularised
Generalised
Other
OPT-125m
52.89
51.33 +/- 4.52
57.97 +/- 15.30
51.68 +/- 1.58
OPT-350m
56.72
55.11 1.57
72.10 2/-2.2
54.39 +/- 1.50
OPT-1.3b
59.92
60.23 +/-0.93
+0.34+/-2..
OPT-2.7b
58.03
49.62+/-33.
74.64
5/-19+/-3.77
OPT-6.7b
63.28
58.33+4-399
+/-1944-3.48
+/3222
OPT-13b
65.75
60.23
72.10+/-3.18
OPT-30b
63.36
2-2440
6/-22
64.17+/33.7
OPT-66b
60.81
57.58+/-1.2
60.51
62.36+/-3.33
OPT-175b
60.75
+-2224
56.16+/-2.22
BLOOM-560m
54.56
46.21
73.19
53.06+/-0.7
BLOOM-Ib1
57.31
4/.94
4/.13+
58.31 +/-330
BLO0M-1b77
53.14
33.03+/33.4
53.624/-4.64
5/.80+--1.74
BLOOM-3b
59.39
35.494/2222
09.57+/-5.88
BLOOM-7bl
56.11
49.62
71.38+/-2.92
54.35+/-3..
BLOOM-176b
63.47
68.18+/-3.44
50.36
03-35
EleutherAl-125m
54.39
+/-113
+3.41-/66.00
52.20
BleutherA1-1.3b
57.83
50.00+/-2.78
6/.20+/-2.32
+/-15
EleutherA1-2.7b
57.03
57.20+/-2.76
$7.25+/-622
55.77 +/-
EleutherAl-6b
57.64
2
59-42+/-10.22
58.05 +/74.71
EleutherAl-20b
59.33
+/.463
59.04 +/33
Cohere-409m
53.92
57.39/-224
6/-30
51.94 1.99
Cohere-6b
58.72
31.70+/-2.34
6/.86
48-74+/-5.77
Cohere-13h
60.36
58-524/-1.21
70.29
59.73 +/-
Cohere-52b
63.31
53.41+--2.99
6/-75
64.17 4/17+/-2.12
GPT-3-350m
57.72
50.95+/2..9
73.91
56.59 +/-99+-11 1.79
GPT-3-1.3b
60.92
37.014-5.00
+/2271
61.15 +/1
GPT-3-6.7b
63.94
58.524/-4.19
66.67
64.56+/-1.31
GPT-3-175b
67.28
6/.45
6/.844/-4.64
66.75
TO-3b
46.67
55.68+/00.00
2/-28
48.28+/00110
T0-1lb
46.72
55.68
24.28+-0.88
4/.36
BlenderBot-90m
46.67
55.68
2/-910/-0.00
4/-32+/00.00
BlenderBot-3b
53.25
44.70+/-0.54
76.09//-0.00
BlenderBot-9h
53.36
45.27+/-1.21
76.09
51.77 31.77+/-00 0/6
Flan- -
60.19
44-15
11-38+-10.77 10.75
59.60 4/44
Flan-T5-3b
55.
54.92/--.44
43.48 1/.10
56.29 4/84.84
1-
60.56
59.664/-3.33
57.61
1/-.09
60.03 5/13
Cohere-command-6h
68.22
63.07 4/-4.44
77.17+/-6.75
67.79 2.45
Cohere-command-522
75.64
+/-274-1155
76.45 +/-4.24
75.15 +/- +/-15 1.17
cext-ada-001-umknowe
57.36
4/.24+/3332
6/-564
58,23 +/- 1.53
text-
Nost-aabbage-001-umnnn
63.53
4/-2.22
+/-547
63.35 +/55+/11.44 1.49
text-curie-001-umnnoww
70.17
2.112
71.32 +/-
text-davini-001-amnnoon
74.97
63.83 1.-111
80.80
75.41 +/- 1.79
text-avvini--02-uunnoon
79.56
70.08 1.56
2/-2.11
79.59 +/-
eex-daavinci-00--amknoon
79.00
+/-181
87.32
79.07 +/- 1.38
ChaiGPT-unknown
74.28
68.37 4/3.377
75.36 11.06 11.06
73.90
+/-.882
GPT-4-unknown
81.31
70.83 4/-4.29
4/-2.1
81.31 3,82
Humans
86.23
83.18
92.17
84.86
45

---

Table 29: Accuracy per label for 15-shot evaluation.
Model
Mean
World knowledge
Idiom
Rhetorical question
OPT-125m
51.86
44.93 +/- 4.10
53,41 +/- 8.48
43.94 +/- 19.93
OPT-350m
55.42
48.55+/-2 2.99
48.48 +/- 2.51
42.42 +/- 6.78
OPT-1.3b
61.61
64.49 +/-
68.94 4.67
42.42 +/- 6.78
OPT-2.7b
59.53
55.80 +/-
2/-208
60.61 +/-4.29
OPT-6.7b
64.72
55.80
60.61 16.32
OРT-13b
65.17
64.49
66.67
OPT-30b
64.06
68.84
6/.23
43.94+/-8.16
OPT-66b
61.83
65.94
8/.20
39.39
OPT-175b
64.78
16-09
67.05
50,00 6,94
BLOOM-560m
55.00
2/-
2/
62.12+/-3.38
BLOOM-Ib1
57.58
50.00 4/4.66
53.03
57.58 4.29
BLO0M-1177
55.14
60.144/112.4 12.40
5/.38+/-4.98
53.03 +/- 16.94
BLOOM-3b
58.69
44.93 +/3.24
61.36+/66.4
57.58 +/- 6.78
BLO0M-7b1
55.67
55.07 +/-07
6/
56.06 8/16.16
BLOOM-176b
61.89
+/-54+99.88
70.08+1755
37.88 +/ 3.39
EleutherA1-125m
56.03
6/.14+/-7.70
42.80+/-4.66
59.09 +/- 13.64
EleutherA1-1.3b
57.44
49.28 +/- 2.05
51.52
39.39 +/- 15,45
EleutherAl-2.7b
58.08
53.62 4.81
57.20 +/- 4.81
56.06 +/- 11.03
EleutherAl-6b
58.81
58.70 +/ 10.87
56.06 10.47
5/.06+/-633 6.25
EleutherAl-206
59.86
55.80 2.99
63.64 9.19
4/-42+/-4.1 4.29
Cohere-409m
55.19
50.00 4/166
50.76
4/.42 6.78
Cohere-6b
60.44
65.94 9/-9.19
67.05 +/-988
50.00 +/- 17.99
Cohere-13b
62.83
67.39 +/- 13.92
64.77+/-555
43.94+/-99 63.64 +/- 13.89
Cohere-52b
64.72
63.04 +/-622
69.32
GPT-3-350m
58.83
50.00 7/
53.41 +-114
42.42 +/- 13.55
GPT-3-1.3b
62.86
53.62 +/7.39
35.91+33.77
50,00 14.61
GPT-3-6.7b
65.17
2/-222
51.52 1/- 51-52 10.05
GPT-3-175b
68.31
78.26+/-500
66.67+/4.44
9/.06+-11100
TO-3b
46.67
52.17+/-0.00
38.644/-0.00
36.36+/-0.00
TO-1lb
46.81
52.17+/0.00
38.64440.00
36.36
BlenderBot-90m
46.56
52.17+/-000
38.644/00.00
34.85+/-3.39
BlenderBot-3h
53.14
47.83 0.000
61.06+00000
63.64
BlenderBot-9b
53.19
45.65 +/-55/-33.2
+/5.33
65.15
61.50
65.94 4/44//5..8
6/ 10.71
42.42+/4.22
F5an-T5-3b
55.08
66.67 +/- 10.55
60.234/-11.6
36.36 +/-0.00
Flan-T5-11b
60.83
6/-94
68.56
45.45 +/-.22
Cohere-command-66
70.03
80.43
78.41
45.45 +/- 10.50
Cohere-command-526
75.39
8/-13
8/.33
+/-334 5.25
text-ada-001--unnnown
58.28
55.07 5,98
56.06 5.67
63.64 +/- 13.89
(ext-babbagg-0001uumnoon
65.19
+/.04//4.886
77.27 3.7.
68.18 6.94
text-curie-001-unknnww
69.92
79.71 +/- 2.05
73.11 +/- 1.56
45.45 +/- 10.50
text-daving--001-unnnoon
75.31
88.41 2.05
82.95 2
57.58 8/58
text-davinci-022-umnnoww
79.06
94.93 +/- 1.62
85.23 +/ 2.86
72.73 +/- 15.75
text-davine--003-uunnown
79.03
91.30 0.00
85.61 +/- 1.69
69.70 +/- 15.45
CharGPT-unknown
75.56
86.23 +/ 4.64
86.74 +/
60.61 +- 10.05
GPT-4-unknown
82.08
95.65 2.51
81.44 1/444+-220 2.43
90.91 +.00
Humans
86.23
93.04
92.73
92.73
46

---

Table 30: Accuracy per label for 15-shot evaluation.
Model
Mean
Particularised
Generalised
Other
OPT-125m
51.86
50.95 +/- 4.32
55.80 +/- 20.34
51.94 +/- 0.84
OPT-350m
55.42
55.30 4/39
60.51 4/-4.24
56.29 +/-
OPT-1.3b
61.61
57.39 4/95
63.77 +/- 6.60
61.76+/-3.50
OPT-2.7b
59.53
49.24
75.00 +/-00+11.66
59.78 +/-
OPT-6.7b
64.72
+/-5.4
65.12
OPT-13b
65.17
64.02+/-2.34
65.22+/-7.32
65.50
OPT-30b
64.06
62.50+
+/.594
6/.42
OPT-66b
61.83
58.71+/-..21
57.61+/-5.14
64.25+/-3.20
OPT-175b
64.78
6/.20
59.78+/33.92
64.90+/-3.66
BLOOM-560m
55.00
4--1
7/.64
54.78+/-117
BLOOM-Ib1
57.58
+/
4/-13
6/.42+/22.44
BLO0M-1b77
55.14
52.27+/-5.2
+/-8.0
5/-73
BLOOM-3b
58.69
4/-211
+/-2.22
59.30
BLOOM-7bl
55.67
48.67+/-3.7
7/.29+/-3.90
54.78 +/-3.3
BLOOM-176b
61.89
64.02
46-38+/-11..
62.02 +/4.53
EleutherAl-125m
56.03
3--82
5/-39
57.11 0/6
BleutherA1-1.3b
57.44
51.144/33.1
+/-59
59.95 2/42
EleutherA1-2.7b
58.08
2/-2244
61.96+/-6.49
58.27 +/- 1.50
EleutherA1-6b
58,81
54.17+/-6607
66.67+/-8..
59.35
EleutherAl-20b
59.86
5/-11+/-3.98
6/-68
60.85 +/-4.33
Cohere-409m
55.19
61.23
2/
Cohere-6b
60.44
49.62
71.01 4/-6.88
60.85 +/-338
Cohere-13h
62.83
57.77
71.01 71.01+/-5.98 +/-5.98
Cohere-52b
64.72
2/-69
71.01 +/-.81
65.16+/-1.07
GPT-3-350m
58.83
55.684/-2.54
6/-22
60.29 +/-
GPT-3-1.3b
62.86
5/5.67
6/-044
64.86 1/-6++-11, 1.78
GPT-3-6.7b
65.17
58.334/-4.62
73.55
66.37+/-24
GPT-3-175b
68.31
64.77+/-4.10
68.60 +/3.07
TO-3b
46.67
55.68+/00.00
23.91+/-0.00
48.32+/-0.0
T0-1lb
46.81
+/0.00
25.00+-11.09
48.41 +/1122
BlenderBot-90m
46.56
55.68+/-0.00
2/-910/-0.00
48.19+/-0.15
BlenderBot-3b
53.14
44.32+/-0.00
7/-36+/-1.00
51.46+/-0.2
BlenderBot-9h
53.19
44.134/11.79
7/-724/-1.49
51-72+/-0.74
Flan- -an-0m-7780m
61.50
36.63-/22.22
71.74+-11.33 11.30
60.90
Flan-T5-3b
55.08
36.82/-1.33
45,65 4/.92
55.04 3/39
1b
60.83
37.01/-3..3
Cohere-command-6h
70.03
60.80/-4..7
72.83 +/-
70.84 +/-1..66
Cohere-command-522
75.39
69.89 4/-2.3
76.81 +/-3.0
74.76
cext-ada-001-umkmowe
58.28
+/-374
69.20
58.57 2/0
text-
Nost-aabbage-001-umnnn
65.19
5/.33
67.03 +/-3.42
65.12
text-curie-001-umnnoww
69.92
62.50+/11.47
6/-772
71.40 +/-933
text-davini--01-amnnoon
75.31
64.58 2/-2.1
8/.70
text-avvini--02-uunnoon
79.06
72.92 1.02
9/-3.77
77.99 2.77
eex-ddavinci-00--amknoon
79.03
69.32 4/-3.77
87.68
1/61
78.94 +/- 1.12
ChaiGPT-unknown
75.56
72.16 1116+-4.888
77.54 +/
74.63 +/4,,41
GPT-4-unknown
82.08
72.92 +/- 1.02
86.23 +/-
82.69 3.87
Humans
86.23
83.18
92.17
84.86
47

---

Table 31: Accuracy per label for 30-shot evaluation.
Model
Mean
World knowledge
Idiom
Rhetorical question
OPT-125m
51.50
55,80 +/- 5,28
54.55 +/- 9.28
54.55 +/- 9.09
OPT-350m
54.61
49.28 +/-
56.82 +/- 1.86
37.88 +/- 9.70
OPT-1.3b
61.67
71.01 3.24
67.80 +/- +/- 5.48
28.79 +/- 11.03
OPT-2.7b
59.86
58.70 +/-70+--100 10.87
71.21 +/- 8.37
46.97 11.03
OPT-6.7b
63.61
62.32 +/-
67.05 2.86
46.97 19.93
OРT-13b
63.39
6/-14+4-5.28
+/-98+
6:2225
OPT-30b
65,47
71-74
02.88+-7..38
37.88+-33339
OPT-66b
60.83
60.14+/33.90
51.524/-11.79
6:-
OPT-175b
62.44
10.77
6/.50 13.43
60.61 +/14//4.22
BLOOM-560m
55.00
47.10+/-
2-
62-12+/-3.5
BLOOM-Ib1
56.89
49.28 +/
54.92
4 8.16
+/-
BLO0M-1177
52.28
52.90+/-7.2
47.35+/-9.
36.36 +/- 16.60
BLOOM-3b
58.64
50.72 2.05
59.09+/-699 6.94
BLO0M-7b1
57.61
10.72+/-5
41.74+/33.8
54.55 9/-9.09
BLOOM-176b
61.06
73.19 +/19--88855
48.48 +/ 4.29
EleutherA1-125m
53.44
47.10/-44.66
47.73 6/44
39.39 +/- 15.45
EleutherA1-1.3b
55.97
44.93 4/-4.81
37.88 +/- 6.25
EleutherAl-2.7b
57.36
4/32+/-599
53.41 2.86
37.88 +/- 11.03
EleutherAl-6b
58.75
59.42 +- 12.21
52.27 +/- 14.43
36.36 +/-
EleutherAl-206
57.36
57.97 +/- 5,42
60.61 +/- 10.30
31.82 +/- 8.70
Cohere-409m
57.17
47.83 3.55
53.41+-3.15
53.03 +/- 3.39
Cohere-6b
60.36
58.70 +/- 13.92
02.50+/-10.
54.55 +/- 10.50
Cohere-13b
64.81
70.29 4/-21.65
45.91+-7.07
45.45 +/-555
Cohere-52b
65.72
67.39 +/-
+/555
53.03 +/- 11.03
GPT-3-350m
60.25
55.07
57.95
51.52 +/- 10.05
GPT-3-1.3b
19.19
61-59
54.92
+/-,70
GPT-3-6.7b
62.86
56.52
6/-33
50.00+/-6.99
GPT-3-175b
68.31
67.39 5.47
7-.73+-2..2
75.76 4.29
TO-3b
46.67
52.17+/-0.0 0.00
38.644/
36.36
TO-1lb
46.75
52.17+/0.00
38.64440.00
36.36
BlenderBot-90m
46.67
51.45+/
38.644/-0.00
36.36+/00.00
BlenderBot-3h
53.25
47.83
61.36
63.64
BlenderBot-9b
53.72
46.38+/-4.10
63.26+/3..22
63.64+/0.00
61.50
67.39 +/
70.83
42.42+/4.229
Flan-T5-3b
56.11
45-22+/77.11
62.50 13.04
+/-000
Flan-T5-11b
62.11
+/-9211..21
+/- 10.66
51.52 +/- 15.45
Cohere-command--6
70.44
81.164-3.22
78.03 2.83
46.97 8.16
Cohere-command-526
75.00
85.51+/-2.08
78.41 +/- 1.14
78.79 +/- 6.78
text-ada-001--unnnown
55.58
50.72
57.58 4.29
57.58 8.57
(ext-babbage-001-uknnvn
66.00
67.39+/-50
71.59 +--9
4/.644/-5.25 5.25
text-curie-001-unknnww
70.33
75.36 +5-664-3.24
76.52 +/ 5.03
60.61 +/
eext-daving--001-unnnown
75.83
85.51 2.05
84.09 +/- 1.86
4/-156/88.16
text-davinci-002-mmnnowww
80.64
97.83 +/-
87.50 +/-218
text-davine--003-uunnown
79.53
94.93 1.62
+/-399
91-82+/-90
CharGPT-unknown
75.64
87.68 +/
89.02 +-0.4
83.33 +/
GPT-4-unknown
82.17
95.65 +/- 3.55
87.12 4/1122
90.91 +/- 0.00
Humans
86.23
93.04
92.73
92.73
48

---

Table 32: Accuracy per label for 30-shot evaluation.
Model
Mean
Particularised
Generalised
Other
OPT-125m
51.50
50.38 +/- 4.29
52.17 +/- 23.95
50,99 +/- 2.42
OPT-350m
54.61
55.30 +/- 1.26
49.28 +/-3.48
55.51 +/- 1.92
OPT-1.3b
61.67
2/-54//2.76
6/-14+/-5.41 4/.164
OPT-2.7b
59.86
5/.199/-2.89
69.57+/-4.11
59.95 +/-553
OPT-6.7b
63.61
5-.904
72.10+/-8.08
+/-44.0
OPT-13b
63.39
29
6/.86+/-6.07
6/-07+/-2.24
OPT-30b
65.47
6/.64
63.04
6/-93
OPT-66b
60.83
63.07
55.43+/-9.29
4/-498
OPT-175b
62.44
64.394
51.09+/-5.72
63.18 2,85
BLOOM-560m
55.00
27.324-2.31
71.01+/-6.23
54.18 +.812
BLOOM-Ib1
56.89
+9.814-4.99
56.88 +/-.46
+/-,11
BLO0M-1b77
52.28
55.494/-1.77
4/-8.2
4/-83+--1.25
BLOOM-3b
58.64
52.46
+/-349
+/- 1.44
BLOOM-7bl
57.61
54.36+/-8.95
70.29
56.76+/-4.
BLOOM-176b
61.06
67.42//11.9
+/-905
60.08 +/08/1
EleutherAl-125m
53.44
59-47+/-2.44
4/-.44
54.18 44.18+/111 1.21
BleutherA1-1.3b
55.97
5/-70
56.52 10.80
58.40 +/- 1.35
EleutherA1-2.7b
57.36
$7.39+/-2.2
4/-314
58.79 +/- 0,89
EleutherA1-6b
58.75
4/-722
58.33 +/-9.2
60.38 +/-84
EleutherAl-20b
57.36
$1.5244-400
59.42+/-14551
58.79 2/39
Cohere-409m
57.17
51.14
4/-557
58.66 +/-255
Cohere-6b
60.36
2/-52
1/.29
61.20 +/-5.15
Cohere-13h
64.81
2/.144/-2.96
68-12
65.98 +/-3.77
Cohere-52b
65.72
6/.964/-4.17
2/-3322
6/.50+-22001
GPT-3-350m
60.25
2--5
65.22+//6.28
GPT-3-1.3b
60.19
4-6.6
57.61
+/-.022
GPT-3-6.7b
62.86
61.17
63.41+/-7.98
63.65 +/-2.2
GPT-3-175b
68.31
6/5.8
70.29+/66.23
68-17+-2.00
TO-3b
46.67
5/.68+
23.91+/-0.00
48-324/0000
TB-1lb
46.75
+-0.4
23.91+/00000
48.41 0/10
BlenderBot-90m
46.67
55.68+/00.00
23.55+/-0.81
48.41 4/-1+/00.24
BlenderBot-3b
53.25
44.32+/-0.00
76.09//00.00
51.55 +/-200
BlenderBot-9h
53.72
45.08+/-1.88
7/.36+/-1102
52.11 +/11+
Flan-T5 -
61.50
57.01+/3.88
73.19+-8.67
60.16
Flan-T5-3b
56.11
1.-44
47.83+/-11.7
56.29 +/-4.18
1b
62.11
61-36
57.25 1/-254/-16.6
61.58 +/-88
Cohere-command-6h
70.44
64.96
78.62+/-5.5
69.81 +/-
Cohere-command-522
75.00
7/-78+/-14 +/- 1.66
75,00
74.55 +/-55/
vext-ada-001-umknowe
55.58
56.52//-433
55.86 +/-
text-
Nost-aabbage-001-ummnn
66.00
62.32
66.88 +/32
text-curi--001-umnnoww
70.33
60.42+/-5.00
74.28 6/-6.20
71.32 4-22+11..44 1.42
text-davini--01-amnnoon
75.83
67.05 1/-
83.33 +/-3.8
75.67 +/- 1.02
textdaavini--02-uunnoon
80.64
74.43 1.83
2777.4
79.76 +/- 1.44
eex-ddavinci-00--amkkoon
79.53
72.92
4/-2.99
8/.59//-1.99
78.55 +/- 1.20
ChaiGPT-unknown
75.64
67.99
+/ 2.74
78.26+/-6.15
74.55 3/90
GPT-4-unknown
82.17
71.97 2.83
86.23 +/-
82.34 +/-
Humans
86.23
83.18
92.17
84.86
49

---

Table 33: Accuracy per label for model group Example IT for 5-shot chain-of-thought evaluation.
Model
Mean
World knowledge
Idiom
Rhetorical question
Cohere-command-1
69.14
72.46 +/- 5.98
78.03 +/ 3.12
62. 12 +/- 16
Johere-command-521
75.28
78.99 +/- 1.62
84.47 +/53
51.52 +/- 8.57
text-ada-001-unkme
15.33
11.59 +/- 8.20
1/-42
10.61 +/- 9.70
text-babbage-001-unke
47.67
47.83 +/- 11.50
55.30+/-16:: 16.21
42.42 8.57
text-curie-0001-unnovvn
68.22
69.57 6.64
79.17 +-.77+00885
69.70 +/- 10.05
text-davinii-001--unrrvvn
67.25
69.57 7.10
71.59 3/65
60.61 11.34
text-davinii-002--unnnnn
80.06
92.03 1.62
88.26 +/3.77
46.97 +/- 24.29
exx-davvnni--00--mkkoon
83.61
93.48 +/- 2.17
93.18 +/- 0.00
69.70 +/- 10.05
ChatGPT-unkmown
77.19
89.86 4.10
87.88 +- 3.63
65.15 +/- 9.70
GPT-4-unknown
86.47
93.48 +/- 3.32
93.18 +/- 2.93
87.88 +/- 4.29
Humans
86.23
93.04
92.73
92.73
Table 34: Accuracy per label for model group Example IT for 5-shot chain-of chain-of-thought evaluation.
Model
Mean
Particularised
Generalised
Other
Sthere-command-
69.
58.33 +/- 1.93
81.52 +/- 2.08
69.04 +/- 2.04
Sthere-command-..
75.28
68.94 3.19
77.17 +/- 2.08
75.88 +/- 0.53
text-ada--01--unkmn
15.33
1/.53 7.73
44-86+/-9..0
15.50 +/- 8.33
exx-babbbage-001-unnmm
47.67
45.27 4/-27+/1113 11.94
40.94+/-119 19.34
48.19 +/- 14.11
text-curie-00)-unnnoon
68.22
+/-155
+/-7.88
68.04 +/- 1.75
lext-davinei-001-unnnooo
67.25
64.58 3.85
64.13+/-5.11
07-92
text-davine-0022uunnnoww
80.06
75.95 3.68
80.07
80.23 60-23+/-14. 1.07
text-davinci-00---unnown
83.61
77.46 +/- 1.02
+/-318
83.25 +/- 0.96
ChaiGPT-unknown
77.19
72.35 +/- 1.56
80.43 5/47
76.23 +/- 1.11
GPT-4-unknown
86.47
81.63 +/- 2.58
88.77 4/05
86.05 +/- 1.17
Humans
86.23
83.18
92.17
84.86
Table 35: Accuracy per prompt template for BERT-cased.
Template
=
k=
10
=
=
47.3
48.8
50.5
49.8
46.7
46.7
2
46.8
50.3
45.5
50.2
46.7
46.5
3
57.3
51.5
50.0
50.0
47.0
46.7
48.8
51.0
49.5
48.5
46.8
46.7
5
46.7
50.3
44.5
47.7
46.7
46.7
6
46.7
50.3
45.8
47.8
46.8
46.7
Mean
48.9
50.4
47.6
49.0
46.8
46.7
std
3.81
0.832
2.42
1.04
0.107
0.0745
Structured
51.1
50.4
50.0
49.4
46.8
46.7
std
4.4
1.17
0.408
0.665
0.125
7.11e-15
Natural
46.7
50.3
45.3
48.6
46.7
46.6
std
0.0471
7.11e-15
0.556
1.16
0.0471
0.0943
50

---

Table 36: Accuracy per prompt template for BERT-uncased.
Template
==0
=
k
5
10
k 15
k 30
1
57.0
53.2
51.8
55.2
51.7
49.3
2
53.7
50.3
54.0
48.7
49.0
49.3
3
54.7
54.7
57.3
55.5
53.3
52.8
4
56.7
51.5
52.3
54.0
50.3
49.5
5
53.2
50.2
50.2
48.3
48.2
47.2
6
53.3
50.3
54.2
49.2
53.0
53.5
Mean
54.8
51.7
53.3
51.8
50.9
50.3
std
1.55
1.71
2.24
3.13
1.92
2.19
Structured
56.1
53.1
53.8
54.9
51.8
50.5
std
1.02
1.31
2.48
0.648
1.23
1.6
Natural
53.4
50.3
52.8
48.7
50.1
50.0
std
0.216
0.0471
1.84
0.368
2.1
2.62
Table 37: Accuracy per prompt template for RoBERTa base.
Template
==0
k =
k ==
k 10
k 15
k 30
54.0
55.8
58.0
58.7
58.3
57.8
2
56.5
50.5
52.0
55.8
56.0
54.2
53.0
56.8
56.8
61.3
59.5
58.8
4
55.2
56.0
58.7
59.8
56.8
57.2
55.7
50.3
52.3
54.8
55.5
53.0
6
59.2
50.3
54.2
55.8
55.7
55.3
Mean
55.6
53.3
55.3
57.7
57.0
56.1
std
1.97
2.93
2.65
2.38
1.47
2.05
Structured
54.1
56.2
57.8
59.9
58.2
57.9
std
0.899
0.432
0.785
1.07
1.1
0.66
Natural
57.1
50.4
52.8
55.5
55.7
54.2
std
1.5
0.0943
0.974
0.471
0.205
0.939
Table 38: Accuracy per prompt template for RoBERTa-
Template
k=0 0
k=
== 5
k 10
k = 15
k= 30
57.7
50.2
62.0
64.7
64.7
60.5
2
46.7
53.3
58.5
64.2
61.2
55.7
3
60.8
54.8
64.5
62.8
61.8
59.5
4
66.2
50.3
64.0
59.0
57.0
58.2
5
46.7
53.3
58.8
63.5
60.5
56.5
6
46.7
55.5
59.3
60.0
60.8
52.3
Mean
54.1
52.9
61.2
62.4
61.0
57.1
std
7.84
2.03
2.45
2.13
2.26
2.7
Structured
61.6
51.8
63.5
62.2
61.2
59.4
std
3.51
2.15
1.08
2.37
3.18
0.942
Natural
46.7
54.0
58.9
62.6
60.8
54.8
- std.
7.11e-15
1.04
0.33
1.84
0.287
1.82
51

---

Table 39; Accuracy per prompt template for GPT-2-medium.
Template
==0
k
10
k 15
k = 30
53.2
53.7
54.0
53.8
53.8
55.0
52.8
53.7
55.8
57.2
60.3
57.2
53.7
54.0
52.5
56.5
55.8
55.3
53.5
55.7
53.3
55.8
55.5
54.3
59.2
54.3
56.7
57.7
60.7
58.8
58.3
54.8
55.7
57.7
61.7
57.8
Mean
55.1
54.4
54.7
56.4
58.0
56.4
std
2.6
0.706
1.5
1.36
3.03
1.63
Structured
53.5
54.5
53.3
55.4
55.0
54.9
std
0.205
0.881
0.613
1.14
0.881
0.419
Natural
56.8
54.3
56.1
57.5
60.9
57.9
std
2.83
0.45
0.45
0.236
0.589
0.66
Table 40: Accuracy per prompt template for GPT-2--arre.
Template
k=0 0
k 1
k 5
k = 10
k == 15
k 30
53.3
53.3
54.5
53.5
55.3
56.2
2
47.5
56.7
57.5
57.8
60.8
61.0
55.0
53.8
55.7
54.0
54.8
56.0
4
54.0
53.7
56.2
53.5
54.8
56.7
47.2
54.5
56.7
58.8
61.2
60.8
6
47.0
53.3
57.2
59.5
60.3
60.8
Mean
50.7
54.2
56.3
56.2
57.9
58.6
std
3,47
1.18
1.0
2.57
2.92
2.29
Structured
54.1
53.6
55.5
53.7
55.0
56.3
std
0.698
0.216
0.713
0,236
0.236
0.294
Natural
47.2
54.8
57.1
58.7
60.8
60.9
std
0.205
1.41
0.33
0.698
0.368
0.0943
Table 41: Accuracy per prompt template for GPT-2-xl.
Template
k=0
k=
k
k= 10
k= 15
k 30
53.2
53.3
57.0
54.5
54.7
56.2
48.7
61.3
57.3
63.7
62.0
60.5
55.0
55.2
59.5
59.0
58.0
60.7
54.2
54.3
56.0
54.5
54.3
56.3
48.0
59.7
58.3
60.8
62.7
61.7
48.5
60.8
58.0
61.8
61.5
61.5
Mean
51.3
57.4
57.7
59.1
58.9
59.5
std
2.92
3.25
1.1
3.5
3.43
2.32
Structured
54.1
54.3
57.5
56.0
55.7
57.7
std
0.736
0.776
1.47
2.12
1.66
2.1
Natural
48.4
60.6
57.9
62.1
62.1
61.2
std
0.294
0.668
0.419
1.2
0.492
0.525
52

---

Table 42: Accuracy per prompt template for Eleuther A1-MMM.
Template
==0
k=
k=5
k 10
= 15
k 30
53.3
53.7
52.7
56.2
56.2
54.0
2
52.2
50.0
47.5
53.5
55.7
53.3
3
53.3
53.8
51.2
55.8
54.8
52.8
53.7
52.5
51.2
53.8
55.8
53.2
5
50.7
50.2
47.3
53.8
56.2
53,8
48.2
49.8
47.5
53.2
57.5
53.5
Mean
51.9
51.7
49.6
54.4
56.0
53.4
std
1.93
1.72
2.19
1.17
0.806
0.394
Structured
53.4
53.3
51.7
55.3
55.6
53.3
std
0.189
0.591
0.707
1.05
0.589
0.499
Natural
50.4
50.0
47.4
53.5
56.5
53.5
std
1.65
0.163
0.0943
0.245
0.759
0.205
Table 43: Accuracy per prompt template for EleutherAl-1.38
Template
k=0
==1
k=5
k= 10
k= 15
k= 30
54.3
53.7
54.8
57.5
57.2
56.2
2
51.8
56.8
57.5
59.0
55.8
54.7
58.0
55.5
59.5
58.0
61.5
57.5
4
53.2
57.5
56.8
55.2
56.5
54.7
49.7
55.2
57.5
58.7
57.2
56.7
6
51.8
55.7
56.5
58.7
56.5
56.2
Mean
53.1
55.7
57.8
57.4
56.0
std
2.59
1.21
1.4
1.29
1.87
1.02
Structured
55.2
55.6
57.0
56.9
58.4
56.1
std
2.05
1.55
1.93
1.22
2.21
1.14
Natural
51.1
55.9
57.2
58.8
56.5
55.9
std
0.99
0.668
0.471
0.141
0.572
0.85
Table 44: Accuracy per prompt template for Eleuther AI-2.7B.
Template
k=0
k=1
k=
10
k 15
k 30
54.0
52.8
58.2
57.8
59.5
56.7
62.0
56.2
57.7
55.8
57.8
57.7
58.7
60.0
58.8
59.2
57.8
57.8
56.5
54.2
57.5
56.2
57.5
55.5
62.7
54.7
58.7
55.7
57.3
57.8
61.2
55.2
57.3
57.5
58.5
58.7
Mean
59.2
55.5
58.0
57.0
58.1
57.4
- std
3.13
2.25
0.576
1.26
0.741
1.02
Structured
56.4
55.7
58.2
57.7
58.3
56.7
std
1.92
3.12
0.531
1.23
0.881
0.939
Natural
62.0
55,4
57.9
56.3
57.9
58.1
std
0.613
0.624
0.589
0.826
0.492
0.45
53

---

Table 45: Accuracy per prompt template for Eleuther AI-6B.
Template
k=0
==1
k=5
k 10
k= 15
k 30
57.5
58.8
52.7
53.0
52.5
51.3
2
57.7
51.8
63.2
62.7
64.3
65.3
3
56.2
58.2
57.2
53.0
54.7
54.5
52.8
55.5
53.3
52.2
54.0
53.8
5
56.8
52.7
62.7
63.2
65.2
64.2
6
57.2
52.8
61.3
61.8
62.2
63.3
Mean
56.4
55.0
58.4
57.6
58.8
58.7
std
1.67
2.75
4.28
4.94
5.2
5.65
Structured
55.5
57.5
54.4
52.7
53.7
53.2
std
1.98
1.44
1.99
0.377
0.918
1.37
Natural
57.2
52,4
62.4
62.6
63.9
64.3
std
0.368
0.45
0.804
0.579
1.26
0.818
Table 46: Accuracy per prompt template for EleutherAI-20BB.
Template
k=0
k=
k=5
k= 10
k= 15
k= 30
53.0
58.0
55.3
54.3
52.8
54.3
2
61.3
54.2
65.8
63.3
65.0
60.3
54.3
58.3
58.5
56.7
55.3
52.0
4
56.2
58.2
55.3
57.2
57.0
58.7
59.0
53.0
66.7
62.8
65.0
59.2
6
61.3
53.5
65.2
61.7
64.0
59.7
Mean
57.5
55.9
61.1
59.3
59.9
57.4
std
3.25
2.33
4.9
3.42
4.98
3.09
Structured
54.5
58.2
56.4
56.1
55.0
55.0
std
1.31
0.125
1.51
1.27
1.72
2.78
Natural
60.5
53.6
65.9
62.6
64.7
59.7
std
1.08
0.492
0.616
0.668
0.471
0.45
Table 47: Accuracy per prompt template for BLOOM-560M.
Template
k= 0
==1
k=5
10
k 15
30
54.3
54.2
53.5
53.8
53.8
53.5
46.7
56.3
54.0
54.8
56.0
55.3
3
58.8
53.3
53.8
53.3
54.5
54.0
56.3
54.8
53.5
54.8
52.7
56.7
46.7
54.3
53.7
55.3
56.3
55.5
46.7
56.0
54.0
55.2
56.7
55.0
Mean
51.6
54.8
53.8
54.5
55.0
55.0
std
5.05
1.04
0.206
0.734
1.45
1.04
Structured
56.5
54.1
53.6
54.0
53.7
54.7
std
1.84
0.616
0.141
0.624
0.741
1.41
Natural
46.7
55.5
53.9
55.1
56.3
55.3
std
7.11e-15
0.881
0.141
0.216
0.287
0.205
54

---

Table 48: Accuracy per prompt template for BLOOM-IB1.
Template
k=0
k=1
k=5
k 10
k 15
k 30
53.3
53.5
56.2
54.2
55.2
54.5
49.0
51.5
58.2
59.8
58.8
60.8
57.2
54.2
55.8
54.0
55.5
50.8
53.3
54.0
54.2
53.3
55.7
55.8
47.3
51.2
59.8
61.3
60.2
60.0
46.8
51.0
60.2
61.2
60.2
59.3
Mean
51.2
52.6
57.4
57.3
57.6
56.9
std
3.75
1.36
2.18
3.51
2.19
3.53
Structured
54.6
53.9
55.4
53.8
55.5
53.7
1.84
0.294
0.864
0.386
0,205
2.12
Natural
47.7
51.2
59.4
60.8
59.7
60.0
std
0.942
0.205
0.864
0.685
0.66
0.613
Table 49: Accuracy per prompt template for BLOOM-IBS 1B7.
Template
k=0
k=
k=5
k= 10
k= 15
k 30
53.5
54.7
53.8
54.0
55.7
56.5
57.7
52.2
56.3
55.5
55.8
52.0
54.7
53.2
53.8
51.0
54.5
54.0
54.5
53.8
54.5
51.2
55.5
50.3
50.0
51.2
54.3
53.2
54.7
50.0
6
51.3
51.8
53.8
54.0
54.7
50.8
Mean
53.6
52.8
54.4
53.1
55.1
52.3
- std
2.49
1.2
0.886
1.6
0.528
2.31
Structured
54.2
53.9
54.0
52.1
55.2
53.6
std
0.525
0.616
0.33
1.37
0.525
2.55
Natural
53.0
51.7
54.8
54.2
55.1
50.9
std
3.37
0.411
1.08
0.953
0.519
0.822
Table 50: Accuracy per prompt template for BLOOM-3B.
Template
k=0
k=
k=5
k 10
k 15
k 30
53.0
54.0
56.8
59.5
60.0
58.2
2
62.5
58.0
58.2
59.7
57.5
60.0
3
53.5
54.0
57.2
58.7
59.2
58.2
54.8
55.3
55.7
59.0
58.2
55.8
5
58.5
57.5
58.0
59.7
58.8
60.2
59.0
56.8
57.3
59.8
58.5
59.5
Mean
56.9
55.9
57.2
59.4
58.7
58.6
- std
3.4
1.6
0.823
0.408
0.783
1.5
Structured
53.8
54.4
56.6
59.1
59.1
57.4
std
0.759
0.613
0.634
0.33
0.736
1.13
Natural
60.0
57.4
57.8
59.7
58.3
59.9
std
1.78
0.492
0.386
0.0471
0.556
0.294
55

---

Table 51: Accuracy per prompt template for BLOOM-7B1.
Template
k=0
k=
=
k
k 15
k
53.2
55.2
55.2
52.0
53.0
52.7
61.2
59.0
53.7
58.3
58.8
61.7
58.7
53.3
53.0
53.3
53.0
52.8
53.5
53.5
55.2
52.8
54.3
53.5
62.0
61.0
55.3
60.3
58.5
62.5
63.5
60.0
54.7
59.8
56.3
62.5
Mean
58.7
57.0
54.5
56.1
55.7
57.6
std
4.03
3.11
0.871
3.46
2.39
4.63
Structured
55.1
54.0
54.5
52.7
53.4
53.0
std
2.52
0.852
1.04
0.535
0.613
0.356
Natural
62.2
60.0
54.6
59.5
57.9
62.2
std
0.953
0.816
0.66
0.85
1.11
0.377
Table 52: Accuracy per prompt template for BLOOM- 176B.
Template
k 0
k
k 5
k
10
k 15
k 30
53.8
58.8
58.5
57.7
55.7
56.7
2
55.8
60.8
68.0
65.7
64.2
62.7
3
53.5
66.7
69.3
71.8
71.7
69.8
4
54.3
59.8
64.8
62.2
60.7
61.3
52.3
61.3
66.2
61.8
58.8
57.5
6
55.5
59.2
65.7
61.7
60.3
58.3
Mean
54.2
61.1
65.4
63.5
61.9
61.1
- std
1.19
2.65
3,43
4.38
5.06
4.44
Structured
53.9
61.8
64.2
63.9
62.7
62.6
std
0.33
3.51
4.43
5.88
6.68
5,43
Natural
54.5
60.4
66.6
63.1
61.1
59.5
std
1.58
0.896
0.988
1.86
2.28
2.29
Table 53: Accuracy per prompt template for OPT- 125M.
Template
k=0 0
k=
k=5
k
10
k= 15
= 30
53.3
55.2
54.0
55.2
54.2
55.0
2
49.5
50.5
47.5
52.7
50.5
48.2
53.5
55.5
53.0
55.0
53.7
56.0
53.3
54.5
54.2
53.8
54.3
53.8
48.5
50.5
46.3
50.7
49.5
48.0
47.3
50.2
46.3
50.0
49.0
48.0
Mean
50.9
52.7
50.2
52.9
51.9
51.5
std
2.55
2.35
3.56
1.99
2.25
3.49
Structured
53.4
55.1
53.7
54.7
54.1
54.9
std
0.0943
0.419
0.525
0.618
0.262
0.899
Natural
48.4
50.4
46.7
51.1
49.7
48.1
std
0.899
0.141
0.566
1.14
0.624
0.0943
56

---

Table 54: Accuracy per prompt template for OPT-350M.
Template
k 0
==1
k=5
= 10
k 15
k 30
53.3
53.8
51.5
56.5
54.2
54.7
60.5
50.3
50.8
56.5
55.2
54.0
53.3
56.3
52.8
58.7
55.0
56.2
53.7
56.3
52.0
55.2
55.2
56.3
62.3
50.3
50.8
57.0
56.5
53.5
59.7
50.3
50.8
56.5
56.5
53.0
Mean
57.1
52.9
51.4
56.7
55.4
54.6
std
3.78
2.71
0.752
1.04
0.826
1.26
Structured
53.4
55.5
52.1
56.8
54.8
55.7
0.189
1.18
0.535
1.44
0.432
0.732
Natural
60.8
50.3
50.8
56.7
56.1
53.5
std
1.09
7.11e-15
7.11e-15
0.236
0.613
0.408
Table 55: Accuracy per prompt template for OPT-1.3B.
Template
k=0
k=1
k=5
k= 10
k= 15
k 30
57.8
56.2
55.5
60.2
59.8
62.7
2
62.2
57.0
61.2
61.8
64.8
67.2
3
60.8
59.5
57.2
59.7
60.3
58.2
4
54.8
55.8
59.2
56.5
57.0
54.7
5
62.5
56.2
59.3
61.7
65.0
64.5
6
64.0
53.2
55.8
59.7
62.7
62.8
Mean
60.4
56.3
58.0
59.9
61.6
61.7
std
3.13
1.85
2.05
1.76
2.86
4.11
Structured
57.8
57.2
57.3
58.8
59.0
58.5
std
2.45
1.66
1.51
1.64
1.45
3.27
Natural
62.9
55.5
58.8
61.1
64.2
64.8
.sdd
0.787
1.64
2.24
0.967
1.04
1.81
Table 56: Accuracy per prompt template for OPT-2.7B.
Template
k=0
k=1
k=5
k= 10
k= 15
30
54.7
53.0
53.2
53.8
54.3
53.7
64.0
60.3
60.2
60.3
61.3
64.5
55.8
53.3
55.2
55.8
57.0
56.5
54.5
53.3
54.8
55.5
56.8
57.0
64.8
60.7
60.7
62.2
64.3
64.3
63.5
60.3
60.0
60.5
63.3
63.2
Mean
59.6
56.8
57.4
58.0
59.5
59.9
- std
4.58
3.62
3.02
3.11
3.68
4.28
Structured
55.0
53.2
54.4
55.0
56.0
55.7
std
0,572
0.141
0.864
0.881
1.23
1.45
Natural
64.1
60.4
60.3
61.0
63.0
64.0
std
0.535
0.189
0.294
0.852
1.25
0.572
57

---

Table 57: Accuracy per prompt template for OPT-6.7B.
Template
k=0
==1
k=5
= 10
k= 15
k 30
55.7
54.3
60.8
61.2
61.2
58.5
2
64.2
68.0
66.8
65.7
66.3
66.3
3
54.2
53.5
59.5
61.2
63.3
60.5
58.8
56.3
61.8
62.2
63.5
63.2
5
64.2
65.2
66.0
65.2
67.7
67.5
6
65.0
63.2
64.8
64.3
66.3
65.7
Mean
60.4
60.1
63.3
63.3
64.7
63.6
std
4.34
5.62
2.73
1.84
2.23
3.23
Structured
56.2
54.7
60.7
61.5
62.7
60.7
std
1.92
1.18
0.942
0.471
1.04
1.93
Natural
64.5
65.5
65.9
65.1
66.8
66.5
std
0.377
1.97
0.822
0.579
0.66
0.748
Table 58: Accuracy per prompt template for OPT-13B.
Template
k=0
k=1
k=5
k= 10
k= 15
k= 30
54.7
64.0
69.8
68.2
67.8
62.2
2
68.2
57.8
69.5
68.0
66.8
63.7
54.3
62.2
65.2
63.2
64.3
66.3
4
58.3
63.3
64.3
63.7
63.5
64.0
66.0
58.5
67.2
65.3
63.7
62.7
6
64.7
57.5
68.3
66.2
64.8
61.5
Mean
61.0
60.6
67.4
65.8
65.1
63.4
std
5.51
2.68
2.06
1.92
1.6
1.55
Structured
55.8
63.2
66.4
65.0
65.2
64.2
std
1.8
0.741
2.41
2.25
1.87
1.68
Natural
66.3
57.9
68.3
66.5
65.1
62.6
std
1.44
0.419
0.939
1.12
1.28
0.899
Table 59: Accuracy per prompt template for OPT-30B.
Template
k=
k= I
k=5
k= 10
k 15
k= 30
62.2
62.7
66.0
65.2
65.5
65.0
2
62.0
58.7
69.0
65.7
66.3
69.0
3
60.3
63.5
62.7
60.8
60.5
61.5
4
65.0
66.8
57.8
57.2
57.2
56.2
5
60.3
55.8
70.0
66.0
67.2
71.0
6
59.0
54.5
68.3
65.3
67.7
70.2
Mean
61.5
60.3
65.6
63.4
64.1
65.5
- std
1.92
4.37
4.24
3.27
3.87
5.28
Structured
62.5
64.3
62.2
61.1
61.1
60.9
std
1.93
1.77
3.37
3.27
3,41
3,62
Natural
60.4
56.3
69.1
65.7
67.1
70.1
- std
1.23
1.76
0.698
0.287
0.579
0.822
58

---

Table 60; Accuracy per prompt template for OPT-66B.
Template
k=0
k=
k=5
k= 10
k= 15
k=
59.3
56.2
56.7
56.5
55.7
54.3
66.5
67.3
65.3
64.2
67.2
65.2
56.5
64.3
55.5
55.0
56.2
52.2
62.0
61.5
66.5
63.0
61.7
63.7
62.5
66.0
64.8
63.7
65.7
65.0
61.2
63.8
60.2
62.5
64.7
64.7
Mean
61.3
63.2
61.5
60.8
61.9
60.8
std
3,06
3.61
4.3
3.65
4.5
5.43
Structured
59.3
60.7
59.6
58.2
57.9
56.7
std
2.25
3.36
4.93
3.47
2.72
5.0
Natural
63.4
65.7
63.4
63.5
65.9
65.0
std
2.26
1.44
2.3
0.713
1.03
0.205
Table 61: Accuracy per prompt template for OPT-175B.
Template
k=0 0
k
k 5
k = 10
k= 15
k 30
56.7
58.0
64.8
61.0
65.0
62.3
52.7
53.3
67.3
63.2
68.0
65.8
54.5
68.5
60.0
55.3
57.8
56.7
64.0
66.7
61.5
58.0
62.0
58.7
52.0
52.0
65.0
63.8
67.8
65.2
52.2
51.7
64.7
63.2
68.0
66.0
Mean
55.3
58.4
63.9
60.8
64.8
62.4
std
4.19
6.87
2.42
3.13
3.79
3.62
Structured
58.4
64.4
62.1
58.1
61.6
59.2
std
4.06
4.58
2.0
2.33
2.95
2.32
Natural
52.3
52.3
65.7
63.4
67.9
65.7
std
0.294
0.694
1.16
0.283
0.0943
0.34
Table 62: Accuracy per prompt template for Cohere-409.33 (Cohere-small),
Template
k=
k=1
k=5
k= 10
k= 15
k 30
54.2
49.7
52.7
51.7
53.5
56.0
47.5
50.7
52.7
53.2
55.8
57.8
57.2
55.5
55.2
55.5
55.7
57.0
54.8
53.8
54.5
56.8
54.8
54.5
48.5
50.7
52.8
52.7
56.0
58.8
47.5
51.0
52.5
53.7
55.3
58.8
Mean
51.6
51.9
53.4
53.9
55.2
57.2
std
3.91
2.05
1.05
1.72
0.847
1.54
Structured
55.4
53.0
54.1
54.7
54.7
55.8
std
1.3
2.43
1.05
2.16
0.903
1.03
Natural
47.8
50.8
52.7
53.2
55.7
58.5
std
0.471
0.141
0.125
0.408
0.294
0.471
59

---

Table 63: Accuracy per prompt template for Cohere-6.0677 (Cohere-medium)
Template
==0
k=
k=5
k 10
k 15
k 30
54.7
54.2
55.3
51.8
56.3
55.3
2
61.8
62.8
64.3
63.8
65.2
64.7
3
57.2
53.3
58.5
55.3
57.8
55.3
56.0
53.3
57.0
53.2
55.8
56.7
5
57.8
60.7
64.0
64.2
64.7
64.2
6
56.2
62.8
66.2
64.0
62.8
66.0
Mean
57.3
57.9
60.9
58.7
60.4
60.4
std
2.24
4.32
4.11
5.38
3.92
4.65
Structured
56.0
53.6
56.9
53.4
56.6
55.8
std
1.02
0.424
1.31
1.44
0.85
0.66
Natural
58.6
62.1
64.8
64.0
64.2
65.0
std
2.36
0.99
0.974
0.163
1.03
0.759
Table 64: Accuracy per prompt template for Cohere- 13.12B (Cohere-large).
Template
k=0
k=
k= 5
k= 10
k= 15
k= 30
55.3
57.3
56.3
55.0
58.5
59.0
2
59.2
64.2
68.0
66.3
64.7
69.5
3
57.2
62.8
61.0
59.0
64.2
62.3
4
55.5
61.3
56.3
54.0
59.0
59.8
56.8
64.3
66.7
64.2
65.7
69.8
6
59.2
60.7
66.5
63.7
65.0
68.3
Mean
57.2
61.8
62.5
60.4
62.9
64.8
std
1.56
2.41
4.88
4.69
2.94
4.55
Structured
56.0
60.5
57.9
56.0
60.6
60.4
std
0.852
2.32
2.22
2.16
2.58
1.41
Natural
58.4
63.1
67.1
64.7
65.1
69.2
std
1.13
1.67
0.665
1.13
0.419
0.648
Table 65: Accuracy per prompt template for Cohere-52B (Cohere-xI)
Template
k=
k=1
k=5
k= 10
k 15
k= 30
56.0
60.7
70.3
65.3
66.3
68.7
2
62.8
65.0
64.3
64.2
65.0
64.3
3
54.0
65.3
62.8
60.2
64.0
63.5
4
53.8
55.5
61.8
64.8
64.3
64.7
5
62.2
65.7
67.3
63.0
63.7
65.3
62.2
65.7
64.2
62.3
65.0
67.8
Mean
58.5
63.0
65.1
63.3
64.7
65.7
- std
3.97
3.77
2.87
1.72
0.855
1.89
Structured
54.6
60.5
65.0
63.4
64.9
65.6
std
0.993
4.0
3.79
2.3
1.02
2.22
Natural
62.4
65.5
65.3
63.2
64.6
65.8
- std
0.283
0.33
1.44
0.785
0.613
1.47
60

---

Table 66: Accuracy per prompt template for GPT-3-350M (ada).
Template
k=0
k=1
k=5
k 10
k 15
k 30
55.3
57.2
58.3
57.5
58.2
60.5
46.7
56.8
56.3
59.5
59.2
61.7
54.0
54.5
53.3
54.0
56.5
56.7
53.5
52.8
54.7
56.7
58.8
59.7
49.8
57.3
55.3
58.5
58.8
61.8
49.5
57.2
56.3
60.2
61.5
61.2
Mean
51.5
56.0
55.7
57.7
58.8
60.3
std
3.02
1.72
1.55
2.04
1.48
1.75
Structured
54.3
54.8
55.4
56.1
57.8
59.0
0.759
1,81
2.11
1.5
0.974
1.64
Natural
48.7
57.1
56.0
59.4
59.8
61.6
std
1.4
0.216
0.471
0.698
1.19
0.262
Table 67: Accuracy per prompt template for GPT-3-1.3B (babbage).
Template
k=
k=1
k=5
k= 10
k= 15
k= 30
55.7
60.7
61.0
59.0
60.7
57.8
2
63.0
62.5
65.7
61.7
63.0
59.3
3
56.2
59.0
60.5
59.3
64.8
61.0
4
53.3
59.7
60.7
62.5
65.0
66.7
59.2
62.5
63.7
61.8
61.5
58.7
6
59.0
60.2
64.3
61.2
62.2
57.7
Mean
57.7
60.8
62.6
60.9
62.9
60.2
std
3.1
1.33
2.01
1.31
1.6
3.11
Structured
55.1
59.8
60.7
60.3
63.5
61.8
std
1.27
0.698
0.205
1.58
1,98
3.68
Natural
60.4
61.7
64.6
61.6
62.2
58.6
std
1.84
1.08
0.838
0.262
0.613
0.66
Table 68: Accuracy per template for GPT-3-6.7B (curie).
Template
k=0 0
k=1
k=5
k= 10
k= 15
k= 30
53.3
58.3
63.0
64.8
67.7
64.0
2
57.5
65.2
63.2
65.3
65.8
65.2
3
57.0
54.2
59.2
61.2
60.8
59.3
4
53.3
61.7
62.8
63.8
64.7
60.7
5
55.3
64.2
62.5
64.5
65.8
63.7
6
52.5
63.5
63.7
64.0
66.2
64.3
Mean
54.8
61.2
62.4
63.9
65.2
62.9
std
1.92
3.83
1.48
1.32
2.14
2.12
Structured
54.5
58.1
61.7
63.3
64.4
61.3
std
1.74
3.07
1.75
1.52
2.82
1.97
Natural
55.1
64.3
63.1
64.6
65.9
64.4
- std
2.05
0.698
0.492
0.535
0.189
0.616
61

---

Table 69: Accuracy per prompt template for GPT-3-175B (davinci).
Template
= 0
k= 1
k=5
k 10
k 15
k 30
61.2
67.3
66.3
62.7
66.7
66.2
2
53.7
65.3
68.8
69.3
71.0
69.7
3
58.7
65.8
68.2
64.7
65.0
65.3
64.0
62.8
71.3
68.7
66.2
67.8
5
54.2
66.3
69.0
70.0
70.0
70.8
6
51.7
66.7
68.7
68.3
71.0
70.0
Mean
57.2
65.7
68.7
67.3
68.3
68.3
std
4.4
1.44
1.46
2.65
2.43
2.03
Structured
61.3
65.3
68.6
65.4
66.0
66.4
std
2.16
1.87
2.06
2.49
0.713
1.03
Natural
53.2
66.1
68.8
69.2
70.7
70.2
std
1.08
0.589
0.125
0.698
0.471
0.464
Table 70: Accuracy per prompt template for BlenderBot-90n.
Template
k=0
k=
k=
k= 10
k= 15
k= 30
46.7
51.5
46.7
46.7
46.5
46.5
2
46.7
51.3
46.5
46.7
46.7
46.7
3
46.7
46.7
46.7
46.7
46.3
46.8
4
46.7
46.7
46.7
46.7
46.5
46.7
5
46.7
50.0
46.7
46.7
46.7
46.7
6
46.5
53.5
46.3
46.7
46.7
46.7
Mean
46.7
49.9
46.6
46.7
46.6
46.7
std
0.0745
2.52
0.153
7.11e-15
0.149
0,0898
Structured
46.7
48.3
46.7
46.7
46.4
46.7
std
7.11e-15
2.26
7.11e-15
7.11e-15
0.0943
0.125
Natural
46.6
51.6
46.5
46.7
46.7
46.7
.tdd
0.0943
1.44
0.163
7.1le-15
7.11e-15
7.11e-15
Table 71: Accuracy per prompt template for Blender
Template
k=0
==
k=5
k= 10
k
k 30
54.0
53.2
53.3
53.0
52.8
53.3
53.3
53.3
53.3
53.3
53.3
53.3
3
53.2
53.2
53.3
53.2
53.2
53.2
53.5
53.5
53.5
53.3
52.8
53.0
5
53.3
53.3
53.3
53.3
53.3
53.3
6
53.3
53.3
53.3
53.3
53.3
53.3
Mean
53.4
53.3
53.3
53.2
53.1
53.2
std
0.269
0.1
0.0745
0.111
0.227
0..11
Structured
53.6
53.3
53.4
53.2
52.9
53.2
std
0.33
0.141
0.0943
0.125
0.189
0.125
Natural
53.3
53.3
53.3
53.3
53.3
53.3
- std
7.1le-15
7.1le-15
7.1le-15
7.11e-15
7.11e-15
7.11e-15
62

---

Table 72: Accuracy per prompi template for BlenderBott9.4B
Template
==0
k
10
k= 15
= 30
53.7
51.5
53.0
53.0
53.0
54.0
53.2
53.8
54.2
52.5
52.2
52.2
53.3
49.7
52.0
54.0
54.2
55.5
54.0
55.3
52.5
54.0
53.5
53.7
53.3
52.8
53.5
53.2
53.5
53.3
52.7
52.0
51.7
53.5
52.8
53.7
Mean
53.4
52.5
52.8
53.4
53.2
53.7
std
0.407
1.77
0.859
0.537
0.63
0.978
Structured
53.7
52.2
52.5
53.7
53.6
54.4
std
0.287
2.33
0.408
0.471
0.492
0.787
Natural
53.1
52.9
53.1
53.1
52.8
53.1
0.262
0.736
1.05
0.419
0.531
0.634
Table 73: Accuracy per prompt template for TO-3B.
Template
k 0
k= 1
k = 5
k 10
k 15
k 30
1
48.7
49.5
46.5
46.7
46.7
46.7
2
46.7
47.5
46.7
46.7
46.7
46.7
3
49.2
48.3
46.7
46.7
46.7
46.7
4
51.7
49.0
46.7
46.7
46.7
46.7
5
46.7
49.2
46.7
46.7
46.7
46.7
6
46.7
49.8
46.8
46.7
46.7
46.7
Mean
48.3
48.9
46.7
46.7
46.7
46.7
std
1.84
0.773
0.0898
7.11e-15
7.11e-15
7.11e-15
Structured
49.9
48.9
46.6
46.7
46.7
46.7
std
1.31
0.492
0.0943
7.11e-15
7.11e-15
7.11e-15
Natural
46.7
48.8
46.7
46.7
46.7
46.7
std
7.11e-15
0.974
0.0471
7.11e-15
7.11e-15
7.11e-15
Table 74: Accuracy per prompt template for TO-1IB.
Template
k=0
k=
k= 5
k=
10
k= 15
k= 30
1
57.5
47.7
47.3
46.8
46.7
46.7
2
49.3
47.5
46.7
46.7
46.8
46.7
3
65.3
48.8
47.3
46.7
46.7
46.7
4
63.8
48.0
47.0
46.7
46.7
46.7
5
48.0
47.2
46.7
46.7
47.0
46.8
6
49.7
47.5
47.0
46.8
47.0
47.0
Mean
55.6
47.8
47.0
46.7
46.8
46.8
std
7.04
0.515
0.245
0.0471
0.134
0.111
Structured
62.2
48.2
47.2
46.7
46.7
46.7
std
3.38
0.464
0.141
0.0471
7.11e-15
7.11e-15
Natural
49.0
47.4
46.8
46.7
46.9
46.8
std.
0.726
0.141
0.141
0.0471
0.0943
0.125
63

---

Table 75: Accuracy per prompt template for Flan-TS-780M.
Template
0
k
k ==3 5
15
k = 30
64.5
63.3
62.2
60.7
61.5
60.2
2
66.5
65.8
65.3
62.8
65.5
65.0
3
61.7
60.2
58.8
60.8
59.8
59.7
4
58.0
50.2
50.7
51.3
52.3
54.8
5
63.8
69.0
64.3
63.2
65.2
65.5
6
65.3
68.8
64.8
62.3
64.7
63.8
Mean
63.3
62.9
61.0
60.2
61.5
61.5
std
2.79
6.44
5.1
4.08
4.61
3.73
Structured
61.4
57.9
57.2
57.6
57.9
58.2
std
2.66
5.59
4.82
4.45
4.0
2.44
Natural
65.2
67.9
64.8
62.8
65.1
64.8
std
1.1
1.46
0.408
0.368
0.33
0.713
Table 76: Accuracy per prompt template for Flan-T5-33
Template
==
k
k 5
k = 10
k 15
k 30
54.7
58.8
56.8
56.7
57.5
60.0
2
51.2
50.8
59.0
59.2
59.0
59.7
3
54.8
51.3
49.7
49.0
48.7
48.5
4
55.3
50.0
48.0
49.0
49.3
50.8
51.0
54.3
57.2
58.0
58.0
57.8
6
48.0
51.2
58.7
59.0
58.0
59.8
Mean
52.5
52.7
54.9
55.1
55.1
56.1
- std
2.65
3.02
4.37
4.42
4.33
4.67
Structured
54.9
53.4
51.5
51.6
51.8
53.1
std
0.262
3.88
3.81
3.63
4.01
4.97
Natural
50.1
52.1
58.3
58.7
58.3
59.1
std
1.46
1.56
0.787
0.525
0.471
0.92
Table 77: Accuracy per prompt template for Flan-T5-11 TBB.IB.
Template
k=0
k=
k= 5
k= 10
k 15
k 30
64.3
61.0
63.7
65.0
62.5
64.3
61.5
59.7
63.2
62.3
64.0
68.0
56.5
63.0
60.2
57.3
56.7
56.8
61.7
47.7
51.7
50.3
50.3
49.5
61.5
55.8
64.8
64.7
65.5
66.3
59.2
57.5
66.3
63.7
66.0
67.7
Mean
60.8
57.4
61.7
60.5
60.8
62.1
std
2.42
4.94
4.82
5,25
5.62
6.78
Structured
60.8
57.2
58.5
57.5
56.5
56.9
std
3.24
6.79
5.04
6.0
4.98
6.04
Natural
60.7
57.7
64.8
63.6
65.2
67.3
std
1.08
1.6
1.27
0.984
0.85
0.741
64

---

Table 78: Accuracy per prompt template for Cohere-command-Gu
Template
==0
k= 1
k= 5
= 10
k 15
k= 30
65.0
63.2
71.7
70.2
71.3
70.3
2
64.8
64.2
66.8
67.5
69.8
71.7
3
68.0
65.5
69.2
65.5
66.8
68.2
4
70.0
68.5
69.2
71.2
71.7
73.2
5
66.3
65.0
66.8
67.5
70.5
69.8
6
63.7
63.7
67.7
67.5
70.0
69.5
Mean
66.3
65.0
68.6
68.2
70.0
70.5
std
2.13
1.74
1.71
1.9
1.59
1.61
Structured
67.7
65.7
70.0
69.0
69.9
70.6
std
2.05
2.17
1.18
2.49
2.22
2.05
Natural
64.9
64.3
67.1
67.5
70.1
70.3
std
1.07
0.535
0.424
0.0
0.294
0.974
Table 79: Accuracy per prompt template for Cohere-command-522
Template
k=0 0
k =
k = 5
k = 10
k = 15
k 30
65.2
74.2
77.8
75.8
75.5
76.0
61.7
72.0
73.5
75.3
75.2
74.5
56.7
74.5
77.3
77.2
76.5
75.0
68.2
70.7
76.0
74.8
75.3
75.3
54.8
72.7
74.8
76.2
74.8
74.7
54.8
73.0
73.0
74.5
75.0
74.5
Mean
60.2
72.8
75.4
75.6
75.4
75.0
std
5.19
1.29
1.8
0.903
0.546
0.529
Structured
63.4
73.1
77.0
75.9
75.8
75.4
std
4.87
1.72
0.759
0.984
0.525
0,419
Natural
57.1
72.6
73.8
75.3
75.0
74.6
std
3.25
0.419
0.759
0.694
0.163
0.0943
Table 80: Accuracy per prompt template for text-ada- 001 1ww.0000--n.mmm.....mmmmmmmmmm.m
Template
k=0
k=1
k=5
k= 10
15
30
60.8
62.8
60.8
59.0
58.7
58.8
2
50.7
56.3
54.8
56.0
57.7
52.7
3
63.7
58.5
60.8
59.0
56.7
57.5
4
61.8
56.3
59.3
58.3
61.0
56.7
5
53.3
55.5
55.2
55.7
58.0
54.3
6
48.7
54.7
54.7
56.2
57.7
53.5
Mean
56.5
57.3
57.6
57.4
58.3
55.6
std
5.82
2.7
2.75
1.43
1.34
2.22
Structured
62.1
59.2
60.3
58.8
58.8
57.7
std
1.2
2.7
0.707
0.33
1.76
0.865
Natural
50.9
55.5
54.9
56.0
57.8
53.5
std
1.88
0.653
0.216
0.205
0.141
0.653
65

---

Table 81: Accuracy per prompt template for text-babbage- 001-unnnownn
Template
k=
k=1
k=5
k 10
k 15
k 30
67.5
64.0
66.3
63.0
64.0
64.7
2
63.0
62.5
66.2
64.2
66.5
68.2
3
65.3
65.2
66.0
63.2
64.7
64.5
65.2
63.5
65.7
62.7
63.0
64.8
5
61.8
64.3
66.5
64.0
66.3
67.8
64.0
63.8
66.2
64.2
66.7
66.0
Mean
64.5
63.9
66.1
63.6
65.2
66.0
std
1.82
0.815
0.25
0.605
1.4
1.5
Structured
66.0
64.2
66.0
63.0
63.9
64.7
1.06
0.713
0.245
0.205
0.698
0.125
Natural
62.9
63.5
66.3
64.1
66.5
67.3
0.899
0.759
0.141
0.0943
0.163
0.957
Table 82: Accuracy per prompt template for text-curie-001 -0w.00oo0n..........mmmmmmo
Template
k=0
k=1
k=5
k= 10
k= 15
k= 30
70.7
70.2
72.5
70.8
70.8
70.7
66.5
59.3
70.3
69.7
68.3
71.2
73.2
70.2
73.5
69.7
71.8
69.7
71.3
68.0
71.0
69.8
71.0
69.0
65.5
58.8
70.0
70.2
68.5
70.7
6
66.5
59.8
70.7
70.8
69.0
70.8
Mean
69.0
64.4
71.3
70.2
69.9
70.4
- std
2.9
5.14
1.25
0.478
1.35
0.754
Structured
71.7
69.5
72.3
70.1
71.2
69.8
std
1.07
1.04
1.03
0.497
0.432
0.698
Natural
66.2
59.3
70.3
70.2
68.6
70.9
std
0.471
0.408
0.287
0.45
0.294
0.216
Table 83: Accuracy per prompt template for text-davinci-001 -01--noow...
Template
k=0 0
k=
k=5
k= 10
k= 15
k= 30
76.5
73.7
75.7
75.7
76.3
76.8
2
72.0
72.5
74.3
75.2
76.0
75.3
3
74.8
74.2
75.7
77.2
75.8
76.8
4
68.0
70.2
72.8
72.8
73.3
75.0
5
72.5
73.2
74.3
74.3
75.3
75.7
6
70.0
72.7
74.3
74.7
75.0
75.3
Mean
72.3
72.7
74.5
75.0
75.3
75.8
std
2.82
1.28
0.991
1.34
0.986
0.724
Structured
73.1
72.7
74.7
75.2
75.1
76.2
std
3.67
1.78
1.37
1.83
1.31
0.849
Natural
71.5
72.8
74.3
74.7
75.4
75.4
- std
1.08
0.294
0.0
0,368
0.419
0.189
66

---

Table 84: Accuracy per prompt template for text-davinci-005 100002unnnoww.....m
Template k=
k
=
= 10
k= 15
k 30
73.7
76.2
80.2
79.5
79.8
80.7
2
69.5
73.5
78.2
78.5
76.7
79.8
3
73.0
78.7
82.8
82.8
82.7
82.8
4
71.3
79.7
80.5
80.8
82.0
81.5
5
67.5
72.5
79.2
79.2
77.0
79.8
6
68.5
73.2
76.5
76.5
76.2
79.2
Mean
70.6
75.6
79.6
79.5
79.1
80.6
std
2.28
2.79
1.96
1.94
2.6
1.22
Structured
72.7
78.2
81.2
81.0
81.5
81.7
std
1.01
1.47
1.16
1.36
1.24
0.865
Natural
68.5
73.1
78.0
78.1
76.6
79.6
std
0.816
0.419
1.11
1.14
0.33
0.283
Table 85: Accuracy per prompt template for texx-davvinii-OO---nnooon
Template
k 0
k
5
k 10
15
=
74.3
71.7
79.8
80.2
80.7
80.3
2
71.8
75.0
80.2
78.8
78.2
78.3
3
71.8
73.7
79.7
79.5
79.2
81.2
4
65.2
74.2
78.5
78.2
79.7
79.5
5
72.2
75.3
80.2
78.5
78.2
78.8
72.2
76.0
79.7
78.8
78.3
79.0
Mean
71.2
74.3
79.7
79.0
79.0
79.5
std
2.84
1.38
0.57
0.666
0.929
0.975
Structured
70.4
73.2
79.3
79.3
79.9
80.3
3.84
1.08
0.591
0.829
0.624
0.694
Natural
72.1
75.4
80.0
78.7
78.2
78.7
0.189
0.419
0.236
0.141
0.0471
0.294
Table 86: Accuracy per prompt template ChaiGPT-unknown.
Template
k 10
k 15
k 30
77.8
73.3
72.7
72.7
74.2
74.5
2
73.2
76.2
78.7
78.2
79.7
79.2
72.7
74.0
74.3
74.7
75.0
74.8
4
59.3
73.7
60.8
63.5
66.0
68.0
74.7
76.8
77.8
77.7
79.3
78.8
6
74.8
76.7
79.0
79.0
79.2
78.5
Mean
72.1
75.1
73.9
74.3
75.6
75.6
std
5.94
1.48
6.29
5.29
4.79
3.9
Structured
69.9
73.7
69.3
70.3
71.7
72.4
std
7.8
0.287
6.02
4.88
4.07
3.14
Natural
74.2
76.6
78.5
78.3
79.4
78.8
std
0.732
0.262
0.51
0.535
0.216
0.287
67

---

Table 87: Accuracy per prompt template for GPT-4-unknown.
Template
= 0
k=5
k= 10
k 15
k 30
83.3
82.7
84.0
84.2
85.5
84.5
81.8
80.8
80.8
78.0
79.3
79.7
84.7
83.7
84.2
85.3
85.5
85.3
80.5
84.3
82.5
84.3
83.3
83.7
79.5
81.0
80.8
77.0
79.0
79.0
80.8
81.3
79.8
79.0
79.8
80.8
Mean
81.8
82.3
82.0
81.3
82.1
82.2
std
1.76
1.36
1.67
3.37
2.81
2.44
Structured
82.8
83.6
83.6
84.6
84.8
84.5
std
1.75
0.66
0.759
0.497
1.04
0.653
Natural
80.7
81.0
80.5
78.0
79.4
79.8
std
0.942
0.205
0.471
0.816
0.33
0.741
68

---

L Timestamps API calls
For reproducibility purposes, Table 88.89 and Cohere were queried for the results.
90 contain the dates and times the APIs from OpenAl
Table 88: Timestamp each evaluated through OpenAl's API (1/2).
model
timestamp
GPP-3-ada/O-shot
2022-009-2 13:13:29
GPT-3-ada/I-sho
2022-09-22 15:11:13
GPT-3-ada/5-shot
2022-09-22 15:40:12
GPT:3-ada/IO-shot
2022-09-22 18:14:18
GPT-3-ada/15-shot
2022-09-22 19:15:29
GPT-3-ada/3O-shot
2022-09-22 22:47:58
GPT-3-babbxgc00sshtt
2022-09-22 23:19:05
GPT-s-babbage//shhott
8022-09-22 22 23:39:53
GPT:3-baahaageS--hhtt
2022-09-23 00:01:32
GPT:3--hahbagellO=shott
2022-09-23 00:24:27
GP7-3-hahbageel5-Shhott
2022-09-23 00:49:13
GPT:3-babbbae/30-shho
2022-09-23 01:15:44
GPT-3-curie/O-shot
2022-09-22 14:04:32
GPT-3-curie/I-shot
2022-09-23 02:09:14
GPT-3-curic/5-shot
2022-09-23 02:32:20
GPT:3-curie/1O-shot
2022-09-23 02:56:43
GPT-3-curie/1-shhot
2022-09-23 03:23:19
GPT:3-curie/3O-shot
2022-09-23 03:52:30
GPT:3-davinniiO-sshot
2022-09-22 12:21:48
GPT:3-davinci/I-shot
2022-09-23 14:27:15
GPT:3-davinnc/5-shot
2022-09-23 15:10:40
GPT-3-davvicc/11--shot
2022-09-23 16:04:53
GPT:3-davinci/l5-shoo
0022-09-23 17:17:04
GPT:3-daviini//0--shhtt
2022-09-23 18:36:38
OpenAAlrexttaddd-00110hhhth
2022-08-17 16:59:45
OpenAlHetttaaaa-00111hhho
2022-08-17 18:23:12
OpeuaAllexttadd--0015--hhoo
2022-08-17 19:16:48
OpemaAliaattaaad000/100hhoooo
2022-08-17 20:24:16
OpeuaAlteatt-aaaa001//55hhh
2022-08-17 21:21:46
2022-08-17 22:44:47
2022-08-17 11:50:44
OpenAA/roxthaabbgg-00///1ssoo
2022-08-17 12:22:08
2022-08-17 12:50:59
2022-08-17 13:27:52
2022-08-17 14:57:43
2022-08-17 15:45:16
2022-08-18 04:39:55
OpeunAlLeextccnie-00///1-hhe
2022-08- 05:10:17
OpenAllnecttcuiicc-001/--hhh
2022-08-18 05:40:56
2022-08-18 06:15:28
OpenaAlreat-aniie-0011//55-hh
2022-08-18 06:53:09
2022-08- 18 07:35:40
2022-08-26 20:26:21
OpennALLeattccvvvii--0l///hoooo
2022-08-26 21:02:31
2022-08-26 21:35:19
Opennll-zzatddaviici--0///hhhhh
2022-08-27 07:14:02
2022-08-27 07:58:25
OpenrAl--ecttdavvnn-00////-----
2022-08-27 08:44:42
69

---

Table 89: Timestamp each was evaluated through OpenAl's API continued (2/2).
model
timestamp
2022-08-10211:4:55
2022-08-11 10:04:17
OpenAkHeexxdavvicc-0222-httt
2022-08-12 15:41:45
2022-08-12 16:41:14
2022-08-16 12:11:43
2022-08-16 14:35:38
2023-03-15 11:35:23
2023-04-04 13:12:05
2023-03-15 12:30:39
2023-04-04 14:01:03
2023-04-04 15:23:29
2023-04-06 15:08:38
2023-04-05 13:33:09
0panAl-gnt-3..5.ubb//1lhttt
2023-04-05 16:36:45
0panAllggt:33.5turb//::lle
2023-04-06 08:46:09
2023-04-06 09:54:07
OpanAl--gtt3.....rrbb11--5---b
2023-04-06 10:57:18
OpenAl-gn-3..ttrh//SSshhh
2023-04-06 12:03:59
OpenAl-gnt-400-shott
2023-04-06 17:38:16
OpenAl-ggt-441lshot
2023-04-06 19:41:59
OpenAAl-gpt-4/-shoot
2023-04-06 22:56:31
OpenAllgpt-4/10-shhe
2023-04-08 12:06:03
-pe
2023-04-08 17:32:04
OpenAl--gt-4330-shot
2023-04-08 19:56:26
70

---

Table 90: Timestamp each model was evaluated through Cohere's API.
model
timestamp
shaere-smallOO-sho
2022-08- 16 22:22:17
Cohere-small/I-sho
2022-08-17 08:22:43
Cohere-small/5--ee
2022-08-17 09:19:57
Cohere-small/10--hho
2022-08-17 10:43:53
Cohere-small//5--hot
2022-08-17 12:53:02
Cohere-smmall33--soot
2022-08-17 13:46:08
Coherre-medium/O-shot
2022-08-17 15:14:02
Cohere-meddun//l-hho
2022-08-17 16:00:21
Coherr--mediun/--shot
2022-08-17 18:23:38
Cohere- medium/1O-shot
2022-08-17 19:16:00
Cohere- Cohere-medium//1 medium/15-shot
2022-08-17 20:24:12
Cohere-medium/3004
2022-08-17 21:20:28
Cohere-larec/O-shoo
2022-08-17 22:47:49
Cohere-large/II-ee
2022-08-17 23:27:00
Joheerelargee55-sho
2022-08-18 00:10:08
Cohere-large/lO-shoo
2022-08-18 --88 00:56:55
Coherr-llagee/15-shot
2022-08-18 01:48:30
Coherrelargee3O-ssoot
2022-08-18 02:47:14
Cohere-xt/O-shot
2022-07-29
Cohere-xl/1-shot
2022-07-31
Cohere-xl/5-shot
2022-08-02
Cohere-xl/10-shot
2022-08-02 15:16:45
Cohere-xl/15-shot
2022-08-07 13:55:44
Cohere-xl/30-shot
2022-08-16 19:51:08
Coherr-comman--mdddum----ooo
2023-04-04 09:54:27
Colere-commann-mmddiui/i--iio
2023-04-04 11:51:07
Cohert--ommmanncmmdduun-----oo
2023-04-04 13:03:07
Colerr-commaan--mefffun//0--0-
2023-04-04 13:31:47
Colerr-commmaanc-cddddnn/Shhhh
2023-04-04 14:06:10
Coterr-commman--neiiiunoO------
2023-04-04 14:42:13
Cotert-ccommmanddz//--ttttttt
2023-04-04 10:25:30
Cohere-commmand--//--htttt
2023-04-04 15:27:01
Cohere-coommanddzz/---hhttt
2023-04-04 15:59:47
hahher-ccommandxx/l0---hhh
2023-04-04 16:36:22
Cohere-commanne
2023-04-04 17:22:58
Cohere-commnn
2023-04-04 18:16:54
71

---

Table 91: Timestamp, duration, and emissions per experiment with non-APl models. (1/4)
model
timestamp
duration
0leutherAll1125m-0-hhott
2022-09-01T211:3::4
8549.649220
2022-09-02T000110:3
640.861 120
EleutherAl-125m---hht
2022-09--2T00::26:27
982.369876
EleuterrA-1125-110-tttt
2022-09-02T00:55::4
1495.525381
HeutherAI-1255-115-tt0
2022-09--02701:29.0
BletthrrAt11255-330shhhh
2022-09-002109:04::3
37298.375266
FleutherAl-22.7--00shoot
2022-09--33T00:36:1
3752.897448
JleutherAl-2.77-I-hhht
2022-09-03T002:04:1
6279.884696
JeutherAl--.70-5-shho
2022-09--03T04:28:19
86416655116
JeutherrAl-2.7--11hhhhh
2022-09-037088:18:13
13792.592126
JeutherrAf-277--155hhh
2022-09--03T11:33::25
18909.55 38909.551123
BleuhherAl-2..--30-hhho
2022-09-03T22247::0
33219.682098
EleutherAl-2bb-O-hhott
1378.197924
JloutherA--2b-I-hhhtt
2022-08-25T08:15:23
807.702344
FleutherAl-220---shhtt
2022-08-255115:39:51
859.585535
JleutherAl-20b--0-shho
2022-08-25T16:18.55
1175.128651
EleutherAl-20b-15-shho
2022-08-257116:47:33
1713.266182
BeutherAl-220--3--hhhl
2022-08-25T17::4522
3469.8 81 1664
EleutherAl-6b-O-shhott
2022-08-24122:29:3
1287.62748 627453
JeetthrAl--b---shht
2022-08-24T23:22:30
1831.554774
EleutherAl-Gb---shot
2022-08-25T00:16:57
3255.128955
EleutherAl--G--1--hhott
4022-08-25T01:223
3971.650572
EleutherA1-66-15-shor
2022-08-25T02:26:23
3772.113814
BeeuthrrAL----30-hhott
4022-08-25T044118:3
6719.419030
EleutherAl-11.3--0-hhhoo
3000.666020
HeutherrAU-1..b-I-shhot
3142.207699
EleutherA1-1.3b---shhot
2022-09-027122:55:2
5933.046596
DeeutherAt-133b-11-hhhod
2022-09--0211::39.00
8509. 257493
EleutherAl-1.3b-155
2022-09-02718::00:99
11615.289366
MeeutherAI-1.3b-3000
2022-09-02722:33339
19978.306457
M
Compute and Emissions
Find below Table 91 until Table 94 the timestamps, durations, and emissions per experiment (calcu- lated with the CodeCarbon library Python). Find below in Table 95 until Table 98lee9811 the cpu-type and count and gpu-type and count per experiment. In terms of compute the following GPU hours can be es- timated if we assume each run is entirely done the GPU (which is not true reality, but worst case):
NVIDIA A100-SXM4 400BX used for 926.4291392151515 hours.
Tesla V100-PCIE-32GB used for 29.282544113225144 hours.
Tesla V100-PCIE-I6GB used for 1.462701331244574 hours.
72

---

Table 92: Timestamp, duration, and emissions per experiment with non-API models. (2/4)
model
timestamp
duration
BLOOM-3b-O-shot
2022-08-31112::5:377
$178.369790
BLOOM-3b-1-S3 1-shoo
2022-08-31T14:39:32
6292.560350
B1.00M-3b-55shot
2022-08-31111::3:299
10675.230701
BI.0OM-33-10-shot
2022-08-31721.59:27
15715.744792
BL.0OM-b--15-shot
2022-09-011703:41::02
20492.823278
BLOOMM-3--30-shtt
2022-09-011115:47.21
43577.882399
BLOOM-7b1-0-shht
2022-08-25700456:35
625,931470
BLOOM-7bI-1-shot
2022-08-25T05:07:13
630.628939
BLOOM-7b1-5-shot
2022-08-23705:24:22
1022. 138932
BLOOM-7bl-10-shot
200
1471.008220
BLO0M-7b1-115-hht
BLOOM-7b1-30-shot
0022-08-25T07:29946
3972. 772039
B1.00M-560m--0shot
2022-08-29T15:35:52
248956
BL.00M-560m-1sshor
2022-00--9918855::6
2532. 794568
BL000M-560m-5-shot
2022-08-29T20:16:11
5038.547060
Bl000M-550mm100shhtt
0022-08-29T22:1::3
7285. 239875
BL00M-5660::15-sho
2022-08-30T00:38:23
8438.09652
BL00M-5660-330ssht
2022-08-30T04:38:44
14419.44711
BLOOM-1b1-0-shot
2022-08-30T05:
2398.822 828856
BLOOM-IbI-I-shot
4359.4
BLOOM-IbI-5-shot
5352.607075
BLOOM-IbI-1O-shot
2022-08-30T11015:02
9541.535419
BL.0MM-111-15-shtt
2022-08-30711::22:22
11257 17577077128
BL.0MM-1b1-30-shot
2022-00--30118.8:155
17131,797610
BLOOM-1766-OOshot
2022-10-14T12:5::11
3015.240235
BLOOM-176b-I-shot
2022-10-14T13:55:53
3906.461752
BL.0MM-176---shht
2022-10-14T20:41110
7411.725385
BLO0M--176-11-shho
2022-10-23122::3:211
14462.201855
BL000M-7766-15-shot
2022-10-24T01:14:10
12609 026676
BLO0M-176b-30-shho
2022-10-14720:47:02
33159.499966
73

---

Table 93: Timestamp, duration, and emissions per experiment with non-APl models. (3/4)
model
timestamp
duration
OPT-13b-0-shot
2022-08-25T007:07::8
878.202579
OPT-13b-1-shot
2022-08--5700:311:0
OPT-13b-5-shot
4022-08-25T00737::5
578.308507
OPT-13b-10-she
2022-08-25T08:01::
821.158826
OPT-13b-15-shot
0022-08-25T00::0:49
1131.479665
OPT-13b-30-shot
2235.869414
OPT-350m-O-shot
1022-09-16117:26:1
389.173900 173905
OPT:350m-1-shot
2022-09-16117:33:42
424.832551
OPT-350m-5-shot
2022-09-16T18:00:14
1583.824094
OPT-350m-10-shot
2022-09-16T18::2:12
1908.822462
OPT-350m-15-shot
2022-09-167119:3:22
1863.625027
OPT-350m-30-shor
2637.811867
OPT-125m-0-shot
2022-09-16111:15::6
273.178967
OPT-125m-I-shot
2022-09-16T15::0:22
259.680856
OPT-125m-55shhtt
2022-09-16115::41:37
1259.801 1OS
OPT:125m-10-shot
2022-09-166116:09555
1693.598805
OPT-125m-155shot
2022-09-161116:41:46
1899.415318
OPT-125m-30-shot
2022-09-16117:19:51
2276.441314
OPT-6.76-0-shot
2022-08-24T23:03:0
1140,485014
OPT-6.7b-1-shot
2022-08-24T23:1755
872.225225
0PT-6.76-5-shot
2022-08-24T23:33:44
995.894396
OPT-6.7b-10-shot
0102-08-24423:55::2
1252.956499
OPT-6.7b-15-shot
0822-08-25T00:2333
1627.749039
OPT-6.7b-30-shot
2022-08-255
2553.054289
OPT-2.7b-0-shot
1082:00-18
686. 197892
OPT-2.7b-1-shot
2022-09-18711::5::1
593.508211
OPT-2.7b-5-shot
2022-09-18T17:1221
1613.313387
OPT-2.7b-10-shot
2022-09-18117:44448
1949 1949.808232
OPT-2.7b-15-sher
2022-09-18T18:22:11
2225.927837
OPT-2.7b-30-shot
2022-09-18T19:09::0
2815.327871
OPT-3Ob-O-shot
2022-08-25T19:003
591.665447
OPT-30b-I-sho
645,923823
OPT-30b-5-shot
1022-08-25T16:4441
1825.821606
OPT-30b-10-shu
1372.752916
OPT-30b-15-shot
2015.00610 104
OPT-30b-30-shot
3859.078056
OPT-1.3b-O-shot
595.193443
OPT-1.3b-I-shot
2022-09-17718::3:45
579.367790
OPT-1.3b-5-shot
2022-09-17T18:33:11
1739.103432
OPT-1.3b-10-shot
2022-09-17719:112:1
2327.300123
OPT-1.3b-15-shot
2022-09-17T19:48:32
2161.637401
OPT-1.3b-30-shot
2022-09-11722::3:00
2893.829010
OPT-175b-O-shot
2022-10-19T155:0::56
2387.104187
OPT-175b-1-shot
2022-10-19T16:34:06
1589.972279
OPT-175b-5-shot
2022-10-19717:25:55
3072.59117
10T-175b-10-shot
2022-10-19711:33:11
6211.692086
OPT-175b-15-shen
2022-10-19T21:299
8019.585246
OPT-175b-30-shot
2022-10-19T21:33:55
19901.470347
OPT-66b-0-shot
0022-08-25718:58:
2834.901372
OPT-66b-1-shot
2022-08-25T19:222
1427.806986
OPT-66b-5-shot
9922-09-25719:47:399
1521.168440
19T-66b-10-shor
2022-08-25T20:24::
2228.407874
157-666-15-shtt
3370.689256
OPT-66b-30-shot
2022-08-26T00:311
6816.312183
74

---

Table 94: Timestamp, duration, and emissions per experiment with non-API models. (4/4)
model
timestamp
duration
BlenderBott2.76---shoo
2022-09-04TT08:09:56
3656.381540
-lenderBBot-277b--shho
2022-09-122115:58:01
4051.858188
BlenderBBot-2.7b----hee
2022-09-12211::1::30
4696.628979
-henderBot-2277-100--3
2022-099-12718:33::5
4772.083818
BlenderBoot2.77-155--h
2022-09-12271::54:13
4698. 638356
BlemderBBot2.76-335shho
2022-09-12122::10:34
4479.460884
BlenderBoo-9.44-00-hhh
2022-10-22T04:004:4
614.201131
BlenderBoot-.44----hhhl
2022-10-22T11:17:22
659.975971
BlenderBot--944-55-hhe
2022-10-22T17:31:488
839.336277
2022-10-22T11:46:18
843.852691
BlenderBot-.44--15--5
2022-10-22T17:53:41
1262.038660
BlenderrBot--4bb-30---
2022-10-22711:23::5
853.334728
BlenderBBot-90m-00shht
2022-09-144115:11:44
273.13477 134403
BlenderBBot-90m-1sshot
2022-09--44115:1::38
351.542638
BlenderBoot900m-5-hhot
2022-09-144115:29:55
730.774348
BlenderBoo-90m-11-shho
2022-09-144115:44:22
1050.647882
BlenderBor-90m--5--hhot
2022-09-14411:007277
BlenderBoot-00m-33-hhh
2022-09-144116:28:55
1285.91366
T0-3b-0-shot
2022-10-21T11:33:36
348. 245298
T0-3b-1-shot
2022-10-24T23:20:57
350.730799
T0-3b-5-shot
2022-10-24T23:29:22
474.378557
T0-sb--0-shot
2022-10-257115:5655
676. 166.111559 1759
TO-3b-15 15-shot
2022-10-25T116:2::55
928.215524
T0-3b-30-shot
2022-10-24123:30:1
1961.897054
T0-11b-O-shot
2022-10-21715:38:13
2289.81522
TO-Lob-1-shot
2022-10-22T19:18:25
814.872760
TO-11b-5-shot
2022-10-22T11:44::55
1368.644314
T0-11b-10-shot
2022-10-22T20:17:30
2112.628515
TO-11b-15-shot
2022-10-22122:06::0
1904.655213
TO-11b330-shot
2022-10-22T22:41:16
Flan-T5-3b-O-shot
2022-10-24T111:00:6
617.820384
Flan-T5-3b-1-shot
2022-10-257112:9:59
348.405589
Flan-T5-3b-55ssot
2022-10-257112338:2
474.872964
Flan-T5-3b-10-shot
2022-10-25T12:50:00
665.592482
Flan-T5-3b-15-sshot
2022-10-255113:05:34
902. 197151
Flan-T5-3b-30-shot
2022-10-25T11:37:14
1864.885266
Fl
2022-10-24T111:5::00
160.503411
Flan-T5-770mm11shhot
2022-10-25T14:41:2
3816.321305
Flan-T5 -
2022-10-25T14:46:09
251.699700
Flam-1557700m1100hhot
2022-10-25T14:52.09
331.340966
Flan-T5- -57m777577mmmm
2022-10-25T14:59:00
381.107934
2022-10-25T15:11::1
705.711192
-lan-So--1b-O-shoot
2022-10-24T10:25:0
1111.283857
Flan-T5--1b-11shoot
2022-10-24T10:55::22
654.411412
-ann-So-ll----shot
2022-10-25T11:26:50
1403.159768
2022-10-25T11:29:599
3756.529085
Tlan-T5--1b-11-shoot
2022-10-25T19:21::5
3042.271478
Flan-T5--1b--30-shot
2022-10-25T20:57:13
5722.244579
75

---

Table 95: Compute used per experiment with non-API models. (1/4)
model
cpus
cpu model
gpu model
EleutherAl-125m-0-dee
10
Apple MI Max
BleuherAA--125m11-55
10
Apple M1 Max
EleutherAl-125m--shtee
10
Apple MI Max
10
Apple
FleutherA1-1155m-155hhh
10
Apple M1 Max
EleutherA1-125mm30-55
10
Apple M1 Max
EleutherAl-2777----hoot
10
Apple MI Max
EleutherAll2.77--sshoo
10
Apple MI
Eleuthe:Al-2..6-5-hhtt
10
Apple MI Max
BletterAAl-2776-11-shee
10
Apple MI Max
EleutherAll-.77-1155hhtt
10
Apple MI Max
EleutherAl-22.7-30--hhot
10
Apple MI Max
FleutherAl-20--O-shott
Intel(R) Xeon(R) Platinum 8275CI CPU 3.00GHZ
NVIDIA A A100-40GB
FleutherAl-200---shhtt
1
Intel(R) Xcon(R) Platinum 8275CI @ 3.00GHz
8 NVIDIA A 100-40GB
FleutherA120b---sshtt
48
Intel(R) Xeon(R) Platinum 8275CI @ 3.00GHZ
8 NVIDIA A A100-40GB
EleutherAl-200--I--hhot
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHZ
NVIDIA A110-40CI
EleutherAl-20b-1--hhot
Intel(R) Xeon(R) Platinum 8275CL CPU 3.00GHz
NVIDIA A100-40GB
BleutherAl-20b-3--hhot
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 NVIDIA A100-40GB
EleutherAl-Gb-D-shoot
Intel(R) Xcon(R) Platinum 8275CI CPU @ 3.00GHZ
8 NVIDIA A100-40GB
F
lleuterAA-6b-1-shoot
Intel(R) Xcon(R) Platinum 8275CL CPU @ 3.00GHZ
8 NVIDIA A100-40GB
FleutherAl-Gb-S-shott
Intel(R) Xcon(R) Platinum 8275CL CPU 3.00GHz
8 NVIDIA A100-40GB
EleutherAll-Gb-1O-hhoo
Intel(R) Xeon(R) Platinum 8275CL CPU 3.00GHZ
8 NVIDIA A100-40GB
EleutherAl-Gb-1S-shhot
I
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
x NVIDIA A100-40GB
EleutherA1--bb-3-shhot
I
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHZ
8 NVIDIA A100-40GB
EleutherAl-1..b---hhot
10
Apple M1 Max
EleutherAl-1.3---shhot
10
Apple M1 Max
EleutherAl-1.3--Sshhot
10
Apple MI Max
FleutherAl-113---10shhot
10
Apple MI Max
MleutherAA-133b-15---h
10
Apple M1 Max
BleutherAl-113b-300-11
10
Apple MI Max
76

---

Table 96: Compute used per experiment with non-API models. (2/4)
model
cpus
cpu model
gpu model
1300M-3b-0-500
10
Apple MI Max
BLOOM-3b-I--3
10
Apple MI Max
BLOOM-30- --o
Apple MI Max
BL.0OM3h-- 10-shot
Apple MI Max
BI.0OM-3--15 -3h-15-shot
10
Apple M1 Max
#1.00M-3--00-10
10
Apple M1 Max
BLOOM-7bl-O-shot
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHZ
8 NVIDIA A100-40GB
BLOOM-7bl-I-shot
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHZ
NVIDIA A100-40GB
BLOOM-77b--Sshhot
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHIz
NVIDIA A100-40GB
BL00M-771-10-ssot
Intel(R) Xcon(R) Platinum 8275CL CPU @ 3.00GHz
x NVIDIA A100-40GB
BLOOM-7bl-11-ssht
Intel(R) Xcon(R) Platinum 8275CL CPU @ 3.00GHz
8 NVIDIA A100-40GB
BL00M-7b1-330ssot
Intel(R) Xcon(R) Platinum 8275CL CPU @ 3.00GHz
X A100-40GB
BLO0MM-560m----hot
Apple MI Max
BL00MM-560m-1-shot
10
Apple MI Max
B1.00M-560m--5shot
10
Apple M1 Max
9L00M-5600-11-shoot
10
Apple MI Max
BL00M-5600n-15sshot
Apple MI Max
BL00M-566m--30-hott
Apple MI Max
BL.00M-1b1-D-ssht
10
Apple M1 Max
BL.0OM-1b1-1-sho
10
Apple MI Max
BL.00M-1b1-5-shot
Apple M1 Max
BLOOM-Ib1-1O-shot
Apple MI Max
BLOOM-1b1-11-shot
10
Apple MI Max
BLOOM--1l--3-shot
10
Apple M1 Max
BL00M--176-OIshot
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 NVIDIA A100-40GB
BL00M--160-I-shot
Intel(R) Xeon(R) CPU 2.20GHz
16 NVIDIA A100-40GB
1L00M-176b-5-shot
96
Intel(R) Xeon(R) CPU 2.20GHz
16 x NVIDIA A100-40GB
B1.00M-17766-0--shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 x NVIDIA A A100-00GB
B1.00M-1766-15-shor
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 NVIDIA A A100-40GB
11.00M--766-30-ssot
Intel(R) Xeon(R) CPU 2.20GHz
16 NVIDIA A100-40GB
77

---

Table 97: Compute used per experiment with non-API models. (3/4)
model
cpus
cpu model
gpu model
OPT-13h-0-shot
48
Intel(R) Xeon(R) Platinum 8275CI CPU @ 3.00GHz
8 NVIDIA A100-40GB
OPT-13b-I-shot
48
Intel(R) Xeon(R) Platinum 8275CI CPU @ 3.00GHz
8 x NVIDIA A100-40GB
OPT-13b-5-shot
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 x NVIDIA A100-40GB
OPT-13b- 10-
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3,00GHz
8 NVIDIA A100-40GB
OPT-13b-15-shee
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 X NVIDIA A100-40GB
OPT-13b-30-shan
48
Intel(R) Xeon(R) Platinum 8275CL CPU 3.00GHz
X NVIDIA A100-40GB
OPT-350m-O-shot
24
Intel(R) Xcon(R) Gold 5118 CPU @ 2.30GHz
4 X Tesla V100-PCIE-32GE
OPT-350m-1-shot
24
Intel(R) Xcon(R) Gold 5118 CPU @ 2.30GHz
4x Tesla V100-PCIE-32GE
OPT-350m-5-shot
24
Intel(R) Xcon(R) Gold 5118 CPU @ 2.30GHz
4x Tesla V100-PCIE-32GB
10-shot
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 X Tesla V100-PCIE-32GB
0PT-350m-15-shot
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 X Tesla V100-PCIE-32GB
OPT-350m-30-shot
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 X Tesla V100-PCIE-32GB
OPT-125m-O-shoo
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4x Tesla V100-PCIE-3200
OPT-125m-1-she
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 Tesla V100-PCIE-32GB
OPT-125m-5-shot
24
Intel(R) Xeon(R) Gold 5118 CPU 2.30GHz
4 X Tesla V100-PCIE-32GH
OPT-125m-10-shmn
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 x Tesla V100-PCIE-32GB
199-125m-15-shor
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 x Tesla V100-PCIE-32GB
OPT-125m-30-shot
24
Intel(R) Xcon(R) Gold 5118 CPU @ 2.30GHz
4 x Tesla V100-PCIE-32GB
OPT-6.7b-D-shot
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHZ
8 x NVIDIA A100-40GB
0PT-6.7b-1-shot
Intel(R) Xeon(R) Platinum 8275CL CPU 3.00GHZ
8. x NVIDIA A100-40GB
OPT-6.7b-5-shot
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 X NVIDIA A100-40GB
OPT-6.7b- 70-sshhot
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 X NVIDIA A100-40GB
OPT-6.7b-15 15-shot
Intel(R) Xcon(R) Platinum 8275CL CPU @ 3.00GHz
8 X NVIDIA A100-40GB
OPT-6.76-30-shon
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
X NVIDIA A100-40GB
OPT-2.7b-O-shot
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 X Tesla VI00-PCIE-32GB
OPT-2.7b-1-shot
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 X Tesla V100-PCIE-32GB
OPT-2.7b-5-shot
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 x Tesla V100-PCIE-32GB
0PT-2.76-10-sh
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4x Tesla V100-PCIE-32GH
OPT-2.76-15--25
24
Intel(R) Xeon(R) Gold 5118 CPU 2.30GHz
4 Tesla V100-PCIE-32GB
OPT-2.7b-30-shot
24
Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
4 x Tesla V100-PCIE-3200
OPT-30h-O-shot
48
Intel(R) Xcon(R) Platinum 8275CL CPU @ 3.00GHz
8 x NVIDIA A100-40GB
OPT-30b-1-shot
48
Intel(R) Xcon(R) Platinum 8275CL CPU @ 3.00GHz
8 x NVIDIA A100-40GB
OPT-30b-5-shot
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 x NVIDIA A100-40GB
OPT-30b-10-shot
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHZ
8 X NVIDIA A100-40GB
OPT-30b-15-shol
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 X NVIDIA A100-40GB
OPT-30b-30-shot
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 X NVIDIA A100-40GB
OPT-1.3b-O-shot
40
Intel(R) Xcon(R) 4114 @ 2.20GHz
4 Tesla VI00-PCIE-16GB
OPT-1.36-I-shot
40
Intel(R) Xeon(R) Silver 4114 CPU 2.20GHz
4 x Tesla "100-PCIE-16GB
OPT-1.3b-5-shor
40
Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
4 x Tesla V100-PCIE-I6GE
OPT-1.3b-10-shot
40
Intel(R) Xeon(R) Silver CPU @ 2.20GHz
4 x Tesla V100-PCIE-16GBB
OPT-1.3b-15-shot
40
Intel(R) Xeon(R) Silver CPU @ 2.20GHZ.
4x Tesla V100-PCIE-16GB
OPT-1.3b-30-shot
40
Intel(R) Xeon(R) Silver CPU @ 2.20GHz
4 x Tesla V100-PCTE-16GB
OPT-175b-0-shot
96
Intel(R) Xeon(R) CPU 2.20GHz
16 x NVIDIA A100-40GB
OPT-175b-1-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 x NVIDIA A100-40GB
OPT-175b-5-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 x NVIDIA A100-40GB
OPT-175b-10-shh
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 NVIDIA A100-40GB
19T-1756-15-she
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 NVIDIA A100-40GB
0PT-1756-30-shod
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 NVIDIA A100-40GB
OPT-66b-0-shol
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 x NVIDIA A100-40GB
0PT-66b-1-shol
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 x NVIDIA A100-40GB
0PT-66b-5-shot
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 x NVIDIA A100-40GB
1PT-66b-10-shol
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 x NVIDIA A100-40GB
0PT-66b-15-65
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
8 X NVIDIA A100-40GB
OPT-66b-300
30-shoot
48
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3,00GHz
8 X NVIDIA A100-40GB
78

---

Table 98: Compute used per experiment with non-API models. (4/4)
model
cpus
epu model
gpu model
BlenderBott-2.7b---hh
10
Apple MI Max
DennderBot-2.7b-I-hhhe
10
Apple MI Max
BlennlerBot2.77-5-shhott
10
Apple MI Max
BlenderBot-2.765 10-soot
10
Apple MI Max
-lenderBot-2.7b-15--2
10
Apple MI Max
10
Apple MI Max
(annderBott-,4b----shee
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16x NVIDIA A100-40GB
BlenderBot--.44b- 4b-1-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 x NVIDIA A100-40GB
B BlenderBot-9, 4.b----hhot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 x NVIDIA A A100-40GB
BlenderBot 4b-10-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
BlenderBot-9 9.4b-15-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 NVIDIA A100-40GB
BlenderBot-9 9.4--t---9b------0
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
BlenderBot-90m- 90m-0-shot
10
Apple MI Max
BlenderBot-90mm 1-shot
10
Apple M1 Max
-enneeroot-90m-55-hh 90m-5-shot
10
Apple MI Max
BlenderBot-90m--l 10-shot
10
Apple MI Max
BlenderBot-9Ont: -hhh
10
Apple MI Max
BlenderBott-9mm-30--5
10
Apple MI Max
TO-3b-O-shot
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16x NVIDIA A A100-40GB
TO-3b-1-shor
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
T0-3b-5-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
T0-3b-10-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
T0-3b-15-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A A100-40GB
TO-3h-30-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 NVIDIA A A100440GB
TO-1Ib-O-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 NVIDIA A100-40GB
T0-11b-I-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
T0-11b-5-shot
96
Intel(R) Xeon(R) CPU @ 2,20GHz
16 X NVIDIA A100-40GB
T0-11b-10-shot
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 X NVIDIA A100440GB
T0-11b-15-shot
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 NVIDIA A100-40GB
T0-11b330-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
Flan-TS-3b-O-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
Flan--5-3b-I-shot
96
Intel(R) Xeon(R) CPU @ 2,20GHz
16 X NVIDIA A100-40GB
-lan--t-3b-S-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
-1an-15-3b-10--1
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 NVIDIA A100-40GB
-lan-15-3b-15--
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
Flan -lan-T5-3b-300 T5-3--30-shot
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
Plan-T5-770m--Oshho
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A00-40GB
Flan-T55770m-11shhot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 NVIDIA A00-400GB
Flan-T5-7700m-5-hoot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
F
96
Intel(R) Xeon(R) CPU @ 2,20GHz
16 X NVIDIA A100-40GB
Flan-T5-7800 750-000-5o-
96
Intel(R) Xeon(R) CPU @ 2,20GHz
16 X NVIDIA A100-40GB
T5n--55-780m-30-51
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
-la---5--1
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 NVIDIA A A100-40GB
Flan--t-I1b--Ishoot
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
96
Intel(R) Xcon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
Tlan--ot-1bbtOO-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
Tlan-Sh--1b-11-shhot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16 X NVIDIA A100-40GB
Flan--5--1b--30-shot
96
Intel(R) Xeon(R) CPU @ 2.20GHz
16x NVIDIA A100-40GB
79


Reviews:
[{"review_id": "xrwToo35xh", "reviewer": "Reviewer_SeJL", "summary": "This paper investigates how well recent LLMs preform on resolving conversational implicatures. It presents a task on conversational implicature resolution built on top of the crowdsourced and human-annotated dataset of George and Mamidi (2020). The work presents an evaluation protocol that lays out how LLMs are evaluated on implicature resolution. The work further presents an array of experiments on various LLMs to understand whether they do well on this task and in what settings (e.g. 0-shot vs 5-shot). Results are analyzed in the form of insights and main takeaways that the authors highlight. ", "questions": "Intro:\n\nLine 28: “utterances that convey something other than their literal meaning”\nAt the first pass of this sentence and reading “other”, I read it as suggesting implicature conveys something else different/unrelated to the literal meaning, and I had to reread it to fit it within the definition of implicature. I suggest replacing “other” with “beyond” as I feel that fits better the description of implicature.\n\n\nSection 2:\n\nAs a work that is, in essence, motivated by investigating how good LLMs are at “interpreting language in context—incorporating its pragmatics”, I feel the related work section leaves out an important line of work on the computational modeling of pragmatic phenomena including implicature and the closely related phenomenon of presupposition and on understanding the limitations of modern NLP models in capturing these phenomena starting with the work of Cianflone et al 2018 on modeling adverbial presupposition, Schuster et al 2020 & Li et al 2021 on scalar implicature, the ImpPress dataset (Jeretic et al 2020) on presupposition and implicature in the context of NLI and related experiments on how good NLI models (including BERT-based models) are at capturing these phenomena, Kim et al 2021 on presupposition verifiability in the context of QA, Parrish et al 2021 with the NOPE presupposition corpus, Kabbara et al 2022 on the limitations of NLI models in resolving presupposition cases. These first come to mind. Despite these efforts focusing on (what is now) smaller LMs models (with the exception of the first mentioned work which precedes the BERT era), I believe they are important efforts that set the stage for the present work and need to be cited and imo are more relevant to the work than some of the information mentioned in the first paragraph of Section 2 (e.g., how some models are toxic, unhelpful, etc). Appendix D has a nice background on implicature from a linguistics/philosophy of language but still misses all of the work I mentioned here on modern computational modeling of pragmatic phenomena.\n\nSection 3:\n\nA general concern of mine is the benchmark contribution of the paper given that the same dataset has been introduced in BIG-bench. The authors are clear about how their dataset is different given that it encompasses instances that are discarded in BIG-bench (for being too “ambiguous”) but regardless the overlap between the two is obviously substantial and so I felt this is something that could benefit from a discussion here. \n\nSection 4:\n\nRegarding the random label experiment and looking at the related section in the appendix (I.6), two points/questions come to mind:\n1)\tGiven that there are two labels in question, did you verify how often the label was not changed? That could eclipse the results we’re seeing here. \n2)\tThis experiment and the close results in some cases (for 5-shot text-davinci-001 and cohere-command-52b) brings up the following question: to what extent the models are learning any notion of pragmatics here? Did you carry out some qualitative analysis in a systematic way to see whether the models are in some cases getting it right for the wrong reasons? For example, Kabbara el al 2022 showed that for presupposition-based NLI, RoBERTa and BERT were getting high accuracy performance on certain presupposition types but upon further investigation and adversarial testing, it turns out the models are often exploiting superficial cues that are not related to notions of pragmatics. I wonder to what extent this could be applicable for the task you’re exploring here.\n\nRegarding the analysis in Appendix I.4 (varying k in-context examples), apart from the conclusion that is in the body of the paper (Section 4, Insight 2), I somewhat feel the results are so noisy for Cohere-52B and OPT-175B that it’s hard to really make sense of the results. InstructGPT3-175B is the only case where the results follow our intuition: the more examples a model sees, the better it gets at understanding the notion of implicature and so the higher the accuracy. For Cohere, we see a very noisy trajectory across the board between k = 1 and 15 (before the results start stabilizing and moving upwards) and for OPT, the results are mostly on a downward trajectory except for a chunk in the middle and end up still going downward for k>15. Would like to see if you have any thoughts on this.\n\nTable 3: Regarding the 5-shot CoT performance \n\nWe see a variance between the text-davinci models. Generally, although specific details are not public, we can assume models got better as newer versions were released. So it makes sense to see the performance improve for the 5-shot and 5-shot COT scenarios across the 3 davinci-models. However, it’s still unclear why CoT would help marginally in one case (002) and in a rather noticeable way (003) but hurt the performance in the 001 case. Any insight there? This is even more pronounced when the drop is not trivial (7%) and is the only case among 6 (if we were to count the -0.1% as roughly not helping but not hurting)\n\nAppendix G: Human evaluation\n\nI’m somewhat confused by the choice of the authors to run human experiments using on prompt only. They could’ve simply chosen to run multiple experiments with at least a couple of different prompts. The authors justify this by the fact that humans are less likely (than models) to be sensitive to variations in the prompt. I feel this is at best speculative and, if anything, human experiments and pilot studies often show that human subjects can be primed to think or react in a certain way given certain wording or structure, etc. \n\nMinor presentation note:\n\nLine 25: I’m not sure if there’s a writing style guide that allows this form “I have to work.”. [essentially the two dots back-to-back]. Seems rather odd to me. I would think the one within the quotations need to be dropped. See here for more details:\nhttps://www.hamilton.edu/academics/centers/writing/style/essentials/punctuation-of-quotations#:~:text=The%20final%20period%20or%20comma,the%20period%20follows%20the%20citation.\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "-\tAddresses an important problem, namely studying the competence of LLMs at pragmatic inferences—here specifically implicatures\n-\tExtensive experiments on a wide range of recent LLMs to validate how well these LLMs perform pragmatic inferences involving implicatures\n", "weaknesses": "-\tThe work is poorly situated with respect to prior work on computational modeling of pragmatics and the related work section misses an important line of work which I discuss in the comments.\n-\tFurther analysis is required in certain cases (see comments below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review and for the in-depth suggestions. We are glad to read that the reviewer thinks our work _“addresses an important problem”_ and contains _“extensive experiments on a wide range of recent LLMs”_, rating the soundness, presentation, and contribution as good. Below we address the reviewers comments. To summarise;\n\n- we discuss the line of work on computational modeling of pragmatics, which we will incorporate in our manuscript\n- we motivate our contribution further by addressing that the BIG bench result cannot be built upon because their method raises serious questions and is not discussed in any paper.\n- we discuss that models are likely doing pragmatic inferences, but not learning to do so in-context, and present results motivating that models are not relying on spurious correlations.\n\nWe hope that this will lead to strengthening your recommendation or letting us know what still stands in the way, so that we may further improve the submission.\n\n### Situating our work\nThank you for pointing out these works, we are happy to incorporate them. Specifically, we will expand the related work by discussing those that look into emergence of pragmatic understanding from language modeling (Jeretic et al (2020) and Parrish et al (2021)). We will cite the papers that look into explicit computational modeling for pragmatic understanding in the the related work, saying they laid the groundwork for the idea that pragmatics is difficult for computational models, and expand upon each in the background section (Cianflone et al. (2018), Schuster et al. (2020), Kim et al., (2021)). We couldn't find Li et al. (2021), could you share the title?\n\n### Contribution in relation to the BIG-bench\nThanks for raising this point of discussion, which we address in the [global response above](https://openreview.net/forum?id=5bWW9Eop7l&noteId=RgN3e2U0c5), as reviewer VB9K mentioned it as well.\n\n### Random labels\nWe aren't sure we understand the question about the random label experiment (_”did you verify how often the label was not changed?”_). We take it to mean; how often did the random labelling assign the correct label to the in-context examples? In the 5-shot case the label is wrong 1443 times and right 1557 times. For 1-shot, the label is wrong 265 times and right 335 times. We hope this clarifies and if not, please let us know.\n\n### Are models learning pragmatics?\nWe believe they are, though not from the in-context examples. It's an interesting question what other spurious correlations the model might rely on. Kabbara et al. (2022) point out cues that arise because the data is synthetic, like the presence of certain tokens (“exactly”) in many of the examples or similarity between premise and hypothesis (”Rene might have hidden” and “Rene hid”). Similar lexical cues will not be a part of our dataset, as they are naturally occurring and have little/no similarity between question and response. This also means the adversarial testing method Kabbara et al. use is less applicable.\nInstead, we ran an analysis to determine potential spurious correlation between question and label. If the models know what the implicature is without even looking at the response, that indicates spurious correlations. The result is as follows:\n\n|Question-only|0-shot|5-shot|\n|---|---|---|\n|ChatGPT|54.3% +/- 3.3|41.7% +/- 12.4|\n|GPT-4|48.9% +/- 10.5|53.7% +/- 0.5|\n\nFrom this we can conclude there are probably no spurious correlations in the questions that can be used without fine-tuning, and hence it’s unlikely the models in our study uses them.\n\n### Noise w.r.t. increasing k (I.4)\nWe agree with the reviewer that the results for the separate prompt templates w.r.t. k barely show a pattern, apart from what's mentioned in the paper. We know from the random labels experiment that the models that can do pragmatic inferences only use the in-context examples for the format. We think the reason for this noise is that giving the format of a problem doesn’t help if the failure wasn’t formatting but understanding in the first place. In such case, a model might give entirely different answers when given different in-context examples.\n\n### 5-shot CoT performance\nWe have asked ourselves this question as well. What we came up with is the following (albeit speculative):\n\n- CoT doesn’t help/hurts for Dav-1 and Dav-2 and Cohere-command-52B\n- CoT does help for Dav-3, ChatGPT, and GPT-4\n\nWhat distinguishes the first and the second category is RLHF. Only ChatGPT, Dav-3, and GPT-4 have undergone RLHF fine-tuning. Perhaps this enables CoT reasoning for this task.\n\n### Human evaluation\nThis is a fair point. Indeed, assuming humans are not sensitive to prompt template is speculative, but we opted for this to make the full human study _directly_ comparable to the model's results on template 2. If we had done a mix of all templates we either had to spent 6x as much on the human evals (which was not within our budget) or subsample evals, making it less comparable to part of the model study. We hope the reviewer also sees value in this design choice over the alternative. We will clarify in the paper that there is a speculative aspect to this choice.\n\nThanks for the notes on line 25, and 28; we will update both!\n\nWe would like to thank the reviewer for the thoughtful questions and we are confident that we can improve the manuscript as a result of this discussion. Specifically; we will update the related work and background. We’ll add the spurious correlation analysis in the appendix. We’ll clarify that the design choice of using only one prompt template for the human eval is speculative. Finally, we’ll update the related work and summarise why this work is a contribution in light of the BIG bench results. We hope this adequately addresses the mentioned weaknesses. If there’s discussion points left that would prevent the reviewer from raising their score, please let us know and we will further address them."}, {"review_id": "hIfEHc88k7", "reviewer": "Reviewer_LHbR", "summary": "The authors analyze the behavior of LLM pragmatic understanding (capability to add and omit imformation to efficiently communicate in context) using a new evaluation protocol. They compare LLM performance to human performance and argue that instruction fine-tuning with examples may improve pragmatic understanding.\n\nThey propose using the assignment of higher likelihood to coherent utterances than similar but incoherent utterances as \"resolving an implicature correctly.\" This includes both describing a situation where implicature understanding is necessary to essentially resolve an entailment question, whether an answer that on its face is irrelevant means \"yes\" or \"no,\" or whether its a sufficient answer. For example, `Esther asked “Can you come to my party on Friday?” and Juan responded “I have to work”, which means yes.` is a failure case because the answer is wrong; the response means no. A model preffering `...which means yes.` resolves the implicature.\n\nThey assess this problem is few and zero shot settings for LLMs. They control for prompt variation effects with 6 different prompt settings, testing a fairly comprehensive set of both generative proprietary LLMs and standard  open pretrained transformers like RoBERTa. They find no model can beat human performance on their resolution task, but GPT-4 comes closest.\n\nThey conclude with a detailed analysis, with 5 insights.", "questions": "Maybe respond to the questions I raised implicitly in the weaknesses? I am open to being convinced.", "limitations": "I find limitations to be sufficiently addressed this time.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Well-defined and scoped problem. Out-of-community definitions (eg, explaining pragmatics) well-motivated. Citations are sufficiently comprehensive in my opinion.\n\nExciting direction to explore to further understnading and engineering of LLMs.\n\nFully-specified approach and sufficiently broad evaluation.\n\nClear statement of insights.\n\nOveral, I think this paper has clear value to the community.", "weaknesses": "I think it would be possible to use more than 6 \"curated prompt templates\" (163-172). The \"we control for prompt variability\" is really not central to the story of the paper but I don't really find the idea that averaging in essence over 6 prompts is sufficient to argue that that has happened. Perhaps claim of robustness should be weakened.\n\n~They claim task-specific instruction tuning improves implicature resolution, but there's no concrete A/B test for this. They don't instruction fine-tune a model themselves for the task (not that doing so would necessarily be possible given constraints). How do we know that IFT is really driving GPT4's superior performance and not just scale or some other factor? I find the evidence presented here unconvincing~\n\nEDIT: In light of the author's response, I do think the case for IFT driving the improvement to be a bit more convincingly made, or at least as convincingly made as can be given the inherent constraints (OpenAI's closedness for ex). I hope to see the authors response to this point integrated in the camera ready, with maybe some weakening of claims in light of the incomplete degree of knowability of this issue", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time, and for the supportive and positive review. We are happy to see the reviewer believes the soundness and presentation are excellent and the contribution is good, stating the paper *“has clear value to the community”*. Below, we address the questions raised. To summarise; \n\n- we argue that six templates to control for sensitivity to prompt wording is significantly more than the standard in LLM literature (namely 1) and that more can become prohibitively expensive.\n- we highlight the results in the submission that show that instruction-tuning at the example-level is important for pragmatic understanding; most notably the sharp increase in performance between Cohere-base-52B and Cohere-command-52B, which is only due to instruction tuning at the example level. However, we agree that verifying this in a controlled study is an interesting future work direction (albeit outside our computational budget).\n\nWe hope the reviewer will find the answers satisfactory, and will seek to make clarifying tweaks to the writing to anticipate such questions arising for future readers of the paper.\n\n### Only six prompt templates\n\nWe appreciate this suggestion, as we agree with the reviewer that in essence we still cannot be sure there is not some prompt that the model will fail on (or perform better on), and that this is an important shortcoming of LLM evaluations more generally. With the below response we’d like to show that we use many more prompt templates than is the standard in LLM literature (namely 6 instead of only 1 template), and hence believe the claims of likely robustness hold up. Additionally, using more templates quickly becomes prohibitively expensive, which is why most papers likely don’t use more than one or two.\n\nWe agree with the reviewer that six prompt templates will not cover the full scope of potential wordings of the problem, we still believe it shows robustness to prompt wording and our claims are not overstated. This is mainly because:\n\n1. We find in general a low variance w.r.t. prompt wording for models that perform well, showing that variability to wording is most likely not going to be an issue.\n2. The standard in LLM evaluation papers is still to only report results on a single prompt template (e.g. see; many of the tasks in BIG-bench, or the highly cited zero-shot reasoning paper (Kojima et al., 2022) which uses 2 templates differing only slightly in wording, or the recent LLM MPT-7b presented by MosaicML in a blogpost, or Hu et al. 2023 Appendix A (”a fine-grained comparison …”) who use one prompt template per task, etc.).\n3. This common pattern of using relatively few prompt templates is likely because more is prohibitively expensive (we times all our evals by 6 for each model, so instead of 600 evaluations per model if we had 1 template per example, we have 3600 evaluations per model. Taking into account that we have evaluated 17 model classes each with multiple model sizes totalling 49 different models, adding more templates runs into high costs pretty quickly). A pragmatic balance between exploring the effect of prompt templates and affording to run experiments on a budget needed to be struck.\n\nFurthermore, because our 600 examples in the dataset are naturally occurring, they already span a broad coverage of different topics and writing styles, showing that a model being able to achieve human-level performance on them is likely to be able to generalise to different examples. A small selection of 3 examples from the test set:\n\n*A: You were a smoker? B: Two packs a day.*\n\n*A: Do you think we were right? B: I think you’d lose.*\n\n*A: Do you have any ketchup left? B: We are swimming in it.*\n\nWe hope this adequately addresses the concern of the reviewer regarding claims of robustness, and if not we’d be happy to consider caveating claims of robustness in the paper if the reviewer can elaborate on this still being necessary in light of our arguments above.\n\n### Claiming task-specific instruction-tuning improves implicature resolution\n\nThanks for bringing up a valid point of discussion here. We agree with the reviewer that A/B testing is important, but as the reviewer notes as well, this is out of scope given our computational constraints. We believe our results do sufficiently convincingly show that instruction-tuning at the example level is important. For example, Figure 3 (right) shows that base models at similar scales as IFT models perform significantly worse. We see that Cohere-command 52B significantly outperforms Cohere-base 52B, and the only difference between those models is instruction-tuning at the example level (Cohere-command is fine-tuned from Cohere-base). In fact, Cohere-command 52B outperforms other base models more than 3 times the size by a large margin (e.g. GPT-3 175B, BLOOM-176B, OPT-175B).\n\nThen the question remains, how do we know example-level IT is the driving factor for GPT-4 and the other OpenAI models? The answer is that we can’t be sure because of OpenAI’s policy of secrecy, but we can be pretty confident: We evaluated 10 models across 6 model classes and two APIs in the group example-level instruction tuned (of which GPT-4 is one). Within this group, models probably vary significantly in other training and architecture details (especially Cohere-command models versus OpenAI models). This means the most significant difference the Example IT models have with other model groups is instruction-tuning at the example level, making it likely that this factor is the driving factor in their performance. Nonetheless, we agree with the reviewer; an important future work direction would be to verify our findings by a controlled study looking into the effect of instruction fine-tuning on implicature resolution.\n\nWe thank the reviewer again for taking the time, and hope the questions raised in the weaknesses section have been adequately addressed. If not, please let us know, and we will gladly look into addressing them."}, {"review_id": "D5njdJlDUZ", "reviewer": "Reviewer_FPTW", "summary": "This paper evaluates several popular LLMs on a benchmark for evaluating pragmatic reasoning via resolving question-answer conversation snippets into the implied binary answer. The study explores the influence of different factors in LLM design, including training method (next-token prediction vs. instruction tuning), number of few-shot examples, model size, inference method, etc. Several conclusions are made about the results: instruction tuning seems to improve pragmatic reasoning, models are mostly robust to perturbations in the input prompts, scaling model size seems to have an effect in improving performance, CoT prompting is useful for some models, and remaining failures of the model appear to center on more difficult (particularised) implicatures.", "questions": "* Is there evaluation on this dataset with respect to an upper bound in performance given only the response, without the question? I'm curious whether answers like \"I've gotta get up early\" or \"Some\" have some spurious correlations with the labels, e.g., \"no\", in this dataset. \n* Is the accuracy actually decreasing relative to k in Figure 4? I.e. is this difference significant? If so, why might that be?\n\nMinor questions:\n* Are there cases where examples are ambiguous, and situational context might affect the label? Did you investigate cases where the human annotators were \"wrong\" to identify the source of those \"errors\"? ", "limitations": "Yes, mostly. More discussion on these non-binary implicatures (at the end of the discussion) would be interesting. I'd also be interested if the authors can perform analysis on the existing benchmark to see whether the labels can be predicted with only one half of the example (i.e., the response); i.e. whether there are spurious correlations within the benchmark. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper offers a very comprehensive / thorough / careful evaluation of the LLMs being evaluated. Particularly, the try a number of different prompt variations, try different numbers of few-shot examples (as well as having a randomization process for choosing these examples, to avoid confounding results with some fixed set of examples)\n* The finding that few-shot examples are useful mostly for conveying format of the response, rather than as a way of defining the task itself, is very interesting. I would be interested to see a more in-depth study of this (perhaps another paper) as it is somewhat surprising (but perhaps less surprising for instruction-tuned models).", "weaknesses": "In general, I wanted to see a bit more discussion and analysis, since this paper is focused on analysis.\n* Moving the analysis for other types of implicatures into the main paper\n* More examples of what the model is generating, particularly for CoT examples\n* More discussion / analysis of human \"errors\" here -- why might there be errors? Genuine noise in the annotation process, or different interpretations of the same example?\n* Discussion on why instruction tuning might be influential in improving pragmatic reasoning\n* Discussion / analysis on why there is still a gap for particularised implicatures, and what we can do to address it\n\nNits:\n* The first example (GPT's answer to the user's question about the phone) actually doesn't necessarily make sense, as it's definitely not literally true -- GPT can't have seen the user's phone\n* \"fine-tuning on instructions at the example-level\" is vague to me, particularly the \"example level\" part. What is this supposed to refer to? Is there a distinction between general \"instruction-tuning\" and \"instruction-tuning ad the example level\"?\n* Also, \"context\" in Insight 5 is vague -- I wouldn't think of this kind of commonsense/world knowledge as \"context\" necessarily, so I'd suggest renaming this\n* Formatting of Figure 2 is somewhat odd\n* I'd suggest adding a line for \"human performance\" in Figure 2\n* I'd suggest adding the few-shot CoT example (or some examples of generated CoT answers) into the main paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the supportive review, saying the paper offers a _“very comprehensive evaluation”_ and that _“the finding that few-shot examples are useful mostly for conveying format [..] is very interesting”_. Below, we address questions. To summarise, we:\n- present new results showing that spurious correlations are likely not an issue\n- show the source of human errors\n- present CoT generations and how those address the gap for particularised examples\n- speculate on why instruction tuning is important\n\nWe are confident the reviewer’s suggestions will contribute to a better manuscript and hope that our responses below will lead to strengthening your recommendation or letting us know what still stands in the way, so that we may further improve the submission.\n\n### Potential spurious correlations\n\nThis is an interesting suggestion, so we ran versions of the benchmark with only the question or response. Getting the implicature right from the response only does not always indicate spurious correlations, as some examples only need the response (e.g. rhetorical questions like ‘do pigs fly?’). Question-only results do always indicate spurious correlations.\n\n|Response-only|0-shot|5-shot|\n|---|---|---|\n|ChatGPT|59.2% +- 4.7|58.3% +- 6.6|\n|GPT-4|62.6% +- 1.7|65.5% +- 1.1|\n\n|Question-only|0-shot|5-shot|\n|---|---|---|\n|ChatGPT|54.3% +- 3.3|41.7% +- 12.4|\n|GPT-4|48.9% +- 10.5|53.7% +- 0.5|\n\nModels mostly perform random for question-only, so spurious correlations don't seem to be an issue. For response-only, GPT-4 5-shot gets 65%. Some examples it gets right are: “do fish swim?” and \"let's hope so\".\n\n### CoT generations + gap on particularised examples\n\nFor this analysis, we propose to add an appendix section that looks at the examples that GPT-4 got right with CoT and wrong with 5-shot. This answers the question about what can fill the gap on particularised examples; CoT closes it for GPT-4 (see Table 28). Explicit reasoning helps, and an example can clarify how (which we will add to the main text). We find models usually get the following wrong:\n\n> A: Is there a bus I can get to the station?\n> B: You can’t rely on it\n> Implicature: yes\n\nGPT-4 5-shot gets this wrong for all 6 templates. With CoT it gets it right for 5 of 6 templates.\n\nThe CoT GPT-4 generates is useful:\n\n> Alice says 'You can't rely on it.' Alice must be implying that there is a bus, but it may not be dependable or timely. This means the response to Bob’s question is yes, but with a caution about reliability. Answer: yes\n\n### Source of human error\n\nWe agree this can be added to the main text. We will add that the score we expect a model to achieve is not 100\\%, but human best. Some examples can be interpreted in different ways, so there is no true right answer, only the answer most people give. We find that part of the errors in our human eval are different interpretations of the same example, and a few annotation errors. One example that people disagree on is:\n\nA: “Was that easy to negotiate?” B: “That is as easy as shooting fish in a barrel.” Implicature: yes\n\nIt depends on whether you think it’s easy to shoot fish in a barrel. We propose to look into the examples that all humans get wrong (likely annotation errors) and examples that humans disagree on (likely multiple interpretations), and summarise the results in the main paper.\n\n### Why instruction tuning (IT) at the example-level is influential\n\nTo answer the reviewer’s question on why we distinguish between benchmark-level and example-level IT; the former is where annotators write a single instruction for an entire dataset. The models are then fine-tuned on each example from the dataset with the same instruction. By contrast, in example-level IT each example in a dataset gets a new instruction, resulting in a more diverse dataset. We will clarify the distinction further in the paper.\nWe think example-level IT is important for pragmatics because it provides examples of pragmatic inferences (if the instruction is ambiguous, the annotators are asked to infer the intent; see sec. 3.6 in [1]). It also provides diversity; each example is a new task with a tailored instruction (see Appendix A.2.1 on p. 26 in [1]). In the discussion we suggest future work should look into the effect of data diversity on pragmatic inference (line 297-298); we will work this out in more detail.\n\n[1] Ouyang et al., 2022\n\n### Analysis for other types of implicatures into the main paper\n\nWe appreciate this suggestion, and also find the other types interesting, but the patterns we find are not significant (overlapping confidence intervals), so it’s difficult to say anything about this in the main paper.\n\n### Discussion on non-binary implicatures\n\nCould the reviewer clarify what they would like discussed? Much discussion is out of scope, because we focus on binary implicatures, but we could for example discuss methods for evaluating non-binary implicatures.\n\n### Other questions\n*decreasing accuracy w.r.t to k in Fig 4?”*\nThis decrease is not significant w.r.t. k=0. Reasons for no increase w.r.t k could be that the examples only clarify structure, which can be clear at k=1, and more examples might cause high variance.\n\nOn using the word “context”; we respectfully disagree that we should rename this. It's meant to encompass things like commonsense and world knowledge. We'll clarify in the paper.\n\nWe will clarify what human performance is in Fig 2.\n\n### Concluding remarks\n\nWe thank the reviewer for all suggestions. We propose to add to the paper:\n\n- the spurious correlations experiment\n- analysis on the source of human errors\n- a CoT completion with a discussion on how this fills the gap for particularised examples\n- why example IT might be important\n\nWe believe the paper will be stronger after adding these, and thank the reviewer for suggestions. We hope you feel in a position to support the publication more strongly as a result. If not, we will gladly attempt to address additional concerns."}, {"review_id": "dtHXbUy5cc", "reviewer": "Reviewer_VB9K", "summary": "The paper introduces a task to evaluate the ability of large language models (LLMs) to resolve conversational implicatures and go beyond the literal interpretation of the meaning of the language. The evaluation is based on a dataset of naturally occurring implicatures that are converted into a binary classification task using a set of templates. For instance, the dialogue _\"Can you come to my party on Friday?”_ , _\"I have to work\"_ is converted into two sentences: _Esther asked “Can you come to my party on Friday?” and Juan responded “I have to work” which means no_ and _Esther asked “Can you come to my party on Friday?” and Juan responded “I have to work”, which means yes._. The two sentences are inputed in the different models, considering the one maximizing the likelihood as the model choice.  \n\nFour categories of state of the art models are evaluated on the proposed task. The evaluation protocol takes into account potential variance induced by prompt templates. Different methodologies are also benchmarked to try to improve the performance of the models on the task, this includes:\n- Fine tuning on the task. \n- Instruction tuning-;\n- Chain of thought prompting\n\nResults demonstrate that the task is challenging for base models (BERT, RoBERTa, GPT2, GPT3, ...) with a performance slightly above random. Without any fine tuning, some models already reach a performance above 70% (text-davinci-00x, ChatGPT), with GPT4 only 5 points behind human performance. The improvement induced by instruction tuning and chain of thought prompting is further demonstrated for some models with GPT4 reaching human performance.", "questions": "Why has no fine tuning on the base models been performed? \n\nDo you have the increase in performance induced by chain of thoughts prompting for GPT4 for particularized (context heavy) examples?\n\nl107 _\"Our analysis is novel in its approach; using ambiguous data that humans can easily resolve, and scope\"_ Maybe reformulate, not very clear\n\nGPT4 doesn't seem to benefit from in context example. How do you explain that?\n\nDid you try to isolate the effect of data contamination? Especially for GPT-4, the model could have been trained on the data used in your evaluation protocol given that the original dataset was published in 2020", "limitations": "The proposed evaluation framework focuses on conversational implicature. If implicature is key for natural language understanding, evaluating models on conventional implicature is also important. If outside of the scope of this well conducted evaluation, it could be of interest for future work.\n\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written with all the material provided to reproduce the results. The paper introduces an original evaluation protocol for the task of resolving implicatures and demonstrates that human performance can be reached by combining state of the art models (GPT-4) and the recently proposed chain-of-thought prompting.", "weaknesses": "The task proposed is not new and the dataset proposed is the same as the one introduced in [BigBench](https://arxiv.org/abs/2206.04615). The results, in the mentioned paper, already demonstrated the ability of PaLM with k-shot-prompting to [perform](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/implicatures) above average human performance. The main differences between the two papers include the choice to keep ambiguous cases for the task and adding other models along with a chain of thoughts prompting to the benchmark. \nIn the benchmark, the variance in the performance of the models, displayed in Table 2, only takes into account the variability induced by different prompt wording. To compare the models, it would have been helpful to perform bootstrap on the test set to estimate to support the claim on the comparison of the performance of the different models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to thoroughly review, rating the soundness and presentation as good. We are happy the reviewer believes our evaluation protocol is original and all the material is provided to reproduce it. We justify below why _our contribution is especially important in light of the BIG bench result the reviewer mentions_, which we believe is not a sound nor reproducible empirical result.\n\nTo summarise our argument, further detailed below;\n\n1. The method of the implicature task contributors in BIG bench raises serious questions as to the validity of their claims\n2. The result is based on base LLMs; no models with SotA fine-tuning methods are evaluated\n3. It is not discussed in a peer-reviewed (or any) paper\n\nAdditionally, we discuss the bootstrap estimates of the std error, why data contamination is unlikely, and answers to other questions. We hope that this will lead to strengthening your recommendation or letting us know what still stands in the way, so that we may further improve our paper.\n\n### Clarifying misunderstandings in the reviewer’s summary of the submission\n\nBefore we address the reviewer’s comments in detail, we’d like to clarify what we believe is a misunderstanding in the reviewer's summary. We don't fine-tune models on the task. We instead look at _emergent_ pragmatic understanding from different domain-general SotA LLM training and fine-tuning methods. Further, the reviewer says we perform “_Instruction tuning: adding k examples from the training set in the prompt;”._ We apologise for the confusion, but instruction tuning is a different method from adding k examples to the prompt. The latter is a prompting technique requiring no weight updates to the model, whereas the former is a SotA fine-tuning method. This is mentioned in line 47-49 and line 179-182. We will make sure to update these explanations to prevent future confusion by other readers.\n\n### The BIG bench result is not an empirically sound or reproducible result\n\nWe respectfully disagree with the reviewer's assertion that our work lacks contribution due to the existence of the BIG bench task using the same dataset and presenting similar results, although we can understand that at first glance this might seem to be the case. For a detailed argument, please refer to the [global author rebuttal](https://openreview.net/forum?id=5bWW9Eop7l&noteId=RgN3e2U0c5).\n\n### Bootstrap on the test set to compare different models\n\nBootstrap estimates of standard error is actually a feature of the evaluation library that we used (see line 207-233 on GitHub in the file `lm-evaluation-harness/lm_eval/metrics.py`, function `bootstrap_stderr()`), so we do have these numbers. We didn't include them because the variation was always much lower than the variation across prompts, which we felt would be more interesting to most NLP researchers. Once we upload all result files, the bootstrap estimates will also be included.\n\n### Conventional implicature is out of scope for this work\n\nThe reviewer says _“evaluating models on conventional implicature is also important. If outside of the scope of this well conducted evaluation, it could be of interest for future work.”_ We agree that studying whether conventional implicature is interesting, but indeed, it is out of scope of what is being studied here. We look at how models can resolve implicatures that require context to be resolved, whereas conventional implicatures are resolved by the conventional meaning of the word. In fact, linguists sometimes argue conventional implicatures are part of semantics, not pragmatics (see appendix D).\n\n### Answers to questions\n\n_Why no fine tuning on the base models?_\n\nWe believe the reviewer suggest an interesting avenue for future work by this question that is out of scope for our current work (we do not have the computational resources), and hope someone with the available computes picks up this question.\n\n_Increase in performance from CoT for GPT4 for particularized examples?_\n\nOn particularised examples GPT-4 achieves 81.6\\% accuracy with CoT prompting (from Table 28), and 71.2\\% 5-shot (from Table 20), so the improvement is ~10\\%.\n\n_Why does GPT4 not benefit from in context examples_\n\nIt’s a good question but it’s hard to say because we don’t know details of GPT-4. We think this is due to more extensive instruction tuning allowing it to better respond to instructions zero-shot. This is partly corroborated by our results in appendix I.6 that show that the models mostly benefit from in-context examples because of the formatting.\n\n_Effect of data contamination_\n\nWe believe data contamination is not an issue for this benchmark for multiple reasons. When we evaluate GPT-4 with only the question, the performance is close to random (e.g. question-only 0-shot is ~49\\% and 5-shot is ~54%). We can add these results to the appendix to clarify this in the paper. Additionally, when we search for the dataset in the large-scale internet-scraped datasets, we do not find any matches. This can be verified with the Gaia search tool on HuggingFace. Finally, the pattern of performance that GPT-4 achieves maps closely onto human performance. If it would’ve memorised the dataset it should be higher than human performance, as our human evals are not public yet.\n\n### Concluding remarks\n\nWe hope this adequately addresses the reviewer’s concerns. We will make the following changes to the manuscript to clarify the arguments laid out above:\n\n1. adjust the related work to summarise the argument about BIG bench.\n2. rewrite Appendix H to get across the above arguments in more detail.\n\nWe would like to thank the reviewer for raising these issues, as we believe clarifying the presentation issues above will make the paper’s contribution clearer. We hope you consider raising your score, and if there are any outstanding concerns that would inhibit you from doing so, we will gladly address them."}]

Author rebuttal:
This response is for **reviewer VB9K** and **reviewer SeJL**, who ask about our contribution in light of the BIG bench results. With the below we aim to motivate in further detail why we believe the BIG bench result cannot be built upon in a scientific way. Hence, our work is an important contribution validating the BIG bench result in a reproducible and methodologically sound way, and above that providing insight into what aspects of LLM training are crucial for the ability to do pragmatic inferences. 

1. **The methodological approach of the task contributors in BIG bench implicatures raises serious questions as to the validity of their claims.** The reason for this is twofold:
    1. The BIG bench result likely overestimates implicature resolution performance. They show that base LLMs at a certain scale achieve above human average performance on a _subset of our dataset: the unambiguous examples_. This likely overestimates performance on our more challenging _ambiguous_ subset, which we show humans in our study resolve at significantly above chance level (72\%). Hence, the question remains; can LLMs resolve _inherently ambiguous implicatures like humans can_? We find the answer to be “no” for some type of models, and yes for other types, and believe this is an important scientific finding to be shared with the research community.
    2. We believe the human evaluation of the BIG bench task is of low quality (noting that this is impossible to fully verify because there is no information available on how the evaluation was done exactly). The average human evaluator on BIG bench implicatures achieves around 82\% performance (where ours achieves on average 86\% on a more challenging dataset), and their human best rater achieves 100\% (where our human best is 92\%). This difference between human average and best hints at poor quality average rating.
2. **the BIG bench task uses only base LLMs and no state-of-the-art fine-tuning methods**, which are the standard in almost all recent published LLMs and considered crucial to performance (e.g. [1], [2]). So even if we could take their results at face value, a question remains; _what aspects of LLMs contributes to their performance on implicatures?_ In our work we find that implicature performance emerges at a much smaller scale in models instruction fine-tuned at the example level, and that scale and prompting techniques are important.
3. **the BIG bench implicatures contribution is not a scientifically scrutinised and peer-reviewed result, as it is not described and discussed in a peer-reviewed (or any) paper** (the BIG bench tech report does not mention it apart as an entry in a list of tasks), and therefore our result reproducibly validating their result with a sound protocol is—on its own—an important contribution.

We thank both reviewers to raising this point of discussion, and we will make the above clearer in the related work section as well as in Appendix Section H.

[1] “Llama 2: Open Foundation and Fine-Tuned Chat Models”, Touvron et al., 2023

[2] “GPT-4 Technical Report”, OpenAI, 2023

Meta-review: All reviewers have agreed that this paper presents a comprehensive evaluation providing interesting insights on the ability of LLMs to resolve conversational implicatures.

The reviewer who gave a lower score was mainly concerned about the paper's novelty because a non-peer-reviewed paper BIG BENCH has investigated the direction. In the authors' rebuttal, they explained why BIG BENCH did not satisfy the goal, and they made an additional effort (human annotation) to better position the problem. I am convinced by the authors about the paper's novelty, and all reviewers agreed that this paper provides informative insights for future LLM investigation. Therefore, I suggest accept this paper.

Given the interesting problem, carefully annotated data, and detailed discussion about conversational implicatures, I suggest this paper be presented as a spotlight paper.
Decision: Accept (spotlight)