Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Main contributions:
1) Identify implicature understanding as a useful benchmark of LLM's communication ability.
2) Design a protocol for comparing implicature understanding of LLMs versus that of humans.
3) Roll-out the protocol on a host of existing LLMs. Providing a deeper analysis of (a) how different properties of LLMs (e.g., instrucability or size) affect their performance, (b) which types of implicatures are harder for LLMs to understand, and (c) the effects of multi-shot learning on performance.

Broadly speaking, implicature refers to the act of saying one thing that implies another. The authors observe that any entity (or model) that is claimed to "understand language" or "be able to communicate" must, in particular, understand implicature. Continuing the wide range of works aimed at understanding and evaluating LLMs, the authors design a protocol for evaluating an LLM's ability to understand implicature.

The protocol entails prompting an LM (or human) with text of an implicature in the form of a question and answer; the task is to classify whether the answer means "yes" or "no" (logits of any token other than yes/no are masked, I think). Performing better-than-random (50%) at this task necessitates implicature understanding. The authors use six prompt templates, some *structured* (like script) and some *natural* (like prose).

Using this protocol, the authors compare the performance of various LLMs and benchmark these against humans. Overall, the authors find that the zero-shot performance of most LLMs is fairly close to random (around 60%), with the best models achieving around 72% accuracy -- compared to the mean 86% of humans. From this, the authors conclude that LLMs fail at zero-shot implicature understanding. Few-shot evaluation sees an increase to about 80%.

In addition to these main findings, the authors perform several refined anlyses:
 - breakdown by prompt type, showing that structured prompts are easier than natural prompts, but the overall ranking among models is not sensitive to prompt type
 - breakdown by implicatures type, showing that generalized implicature is easier than particularized implicature.
 - in *k*-shot evaluation, how performance scales with *k*, showing diminishing returns for $k \geq 5$.
 - how performance scales with model size (the conclusion here was unclear to me).
strength_and_weaknesses: **Pros**
 - Rigorously evaluating implicature understanding is an interesting contribution to a timely and important area, namely, LLMs understanding of language and communication abilities.
 - The protocol is extremely simple. This might seem like a Con, but often times identifying simple ideas and executing them correctly requires much heavy lifting. The authors put in the work to present their idea clearly: the paper is well-written and concepts are clearly defined and demarcated.
 - Overall, the empirical evaluations are easy to understand and fairly complete (Cons 2-5, below, notwithstanding). The in-depth breakdowns shed light on the general claim that LLMs fail at zero-shot implicature understanding.
 - I commend the authors for identifying a subset of their experiments that can be run without access to paid APIs or large compute.
 - The authors survey existing literature on implicature.
 - Cons 6-8 aside, the findings of empirical evaluations are presented in a careful and complete manner. The experiments seem reproducible (though I do not have API access or compute to attempt a full reproduction).

**Cons & suggestions for improvement**
Items (6-8) refer to the code and reproducibility, and items (9-14) are minor comments.

 1. The title is too general -- it's not necessary to replace the task of implicature understanding with the much more general task of "communication". If anything, you can drop "Zero-Shot". Concretely, I'd suggest something along the lines of "Large Language Models Do Not Understand Implicature" or, less accurate but more accessible, "Large Language Models Do Not Understand Implication".
 2. The perfomance gaps of LLMs and humans are about 14% in zero-shot evaluation, which decreases to 6% in few-shot evaluation. Can you quantify what these gaps mean? How much better is 6% than 14%? Given that humans perform imperfectly in this task (86%), I imagine there is a some *x%* threshold past which perfomance is "essentially" human. How close is this threshold to $x=86$? This likely requires a new set of experiments, but seems an important step forward that I would suggest as future work. For example, a Turing test in which a distinguisher prompts a black box with an implicature (or sequence of these) and must distinguish an LLM from another human.
 3. In Table 1, the differences between Davinci-001 and Davinci-002 are within each others' standard deviation. Does this affect the statistical significance / validity of the claim that Davinci-001 outperforms Davinci-002 (page 6)? Not much changes in the paper if you say that the Davincis are indistinguishable, but I find this glaring and worth a brief discussion in the paper.
 4. The identity of the asker/answerer among the pairs (Esther,Juan), (Karen,William) and (Bob,Alice) may affect the performance of the models: for example, a question asked by a Karen may be interpreted differently than a question asked by a William. I suggest adding a small ablation permuting the order of the pairs to test this.
 5. Why were human subjects only shown Prompt #2, rather than a mixture of all six prompts? As the authors find, structured prompts are easier for LLMs than natural ones. It is likely too late to test the effect of multiple vs. single prompt types on human performance, but I would expect to find an explicit discussion of this choice in the text.
 6. **Code and reproducibility.** No Python version is specified in the readme, and the code is incompatible with any version other than the intended one. Which version should be used for reproducing the results? I managed to infer that it’s 3.9 using binary search:
    - Your use of the walrus operator (’:=’) implies Python≥3.8.
    - Your requirement of torch==1.10.2 implies Python<3.10.
    - Your use of union (’|’) operator implies Python≥3.9.
 7. Even after finding the right Python version, the code does not run out of the box. Specifically, I had to manually downgrade protobuf to 3.20 (pip install -Iv protobuf==3.20). You may want to add this to your requirements.txt.
 8. You should add a lightweight and easy-to-run sanity test. For example, an argument-less script that that trains a model on tiny data, validates on the same data, and asserts very high accuracy. While the small experiment from the readme did reproduce for me, it took several minutes to run; it's good to have something that fails faster than that in case things aren't working.
 9. **Minor comments.** In the paragraph entitled "On Prompting" (page 8), the authors state that "There is a narrative around large language models that if they fail a task, it might that the prompt was not the right one." This statement should be supported by citations (just a few examples would suffice).
 10. A footnote next to a punctuation mark comes after the mark, rather than before it. In page 5, move footnote 3 to come after the period.
 11. Although Wittgenstein is cited in the appendix, I expected to find him cited alongside Grice and Huang in page 1, supporting the statement that meaning is determined by contexts, beliefs and social institutions. Not a big deal, I'll leave this decision up to the authors.
 12. Consider changing the title of Appendix B to "Background on Implicatures" or just "Implicatures".
 13. On page 19, the authors write that Bach argues that "conventional implicatures are [...] instances of something else". Can you give an example of what this something might be?
 14. It would be nice to conclude Appendix B with a paragraph explaining in what way this work is informed by the literature mentioned in this section. What is the connection between conventional/conversational implicatures (in the appendix) and generalized/particularized implicatures? If conventional/conversational implicatures do not make an appearance in the rest of the work, why the long discussion of them?
clarity,_quality,_novelty_and_reproducibility: These are all covered in the sections above and below.
summary_of_the_review: This paper offers a significant contribution to an important area of research. Overall, it is written well, its novelties are well-defined and explored, and the main claims are sufficiently supported by empirical findings. Do not take the many suggestions I made above as a serious negative sign; only a handful of them noticeably impact the paper. I strongly suggest the authors to make the experiments more easily reproducible by fixing Cons 6-8.

EDIT: The authors' rebuttal adequately addresses my concerns (primarily reproducibility and the title) and I have increased my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper evaluates whether or not large language models understand implicature. They create (automatically) a dataset that allows them to evaluate pre-trained models’ understanding of this phenomenon. They evaluate LLMs pre-trained on large text corpora and also ones that are fine-tuned with RL feedback to see the differences in models.

strength_and_weaknesses: Strengths

1. This paper is well-motivated and the evaluation is formulated clearly and concretely.
2. The authors spend time evaluating all models in different ways (e.g., relative accuracy, comparison across model sizes and classes, measuring the effect of in-context examples and so on) which is more comprehensive than most evaluations and helps understand model performance
3. The authors report variance with the accuracy results which is good to see—however it looks like the variance is exceptionally high? 
4. The authors also measure the effect of a larger number of samples (k), whether or not the wording of samples in the prompt affects performance, and they break down performance by example type which allows a nice and comprehensive analysis

Weaknesses

1. The framing of this paper (and the title) are misleading. This paper focuses on evaluating implicature, which is a very important phenomena for communication, however is not everything that is needed for (zero-shot) communication and the title implies that this paper focuses on all aspects of communication. There are other aspects of communication/intent (e.g., assertives, hedges, subjective words) and so on that are nuances aspects of communication, as well as other important pragmatic effects, and this paper does not touch upon any of those.
2. The title also does not currently contain information/the word “implicature” which is misleading, so I urge the authors to reframe/change it to remove “zero-shot communicators” (since this is not something the paper tackles) and add “evaluating implicatures” (which is what the paper is largely about)
3. There are several citations re: language models and implicature that are missing. They are: “Linguistic models for analyzing and detecting biased language. Recasens, M., Danescu-Niculescu-Mizil, C., and Jurafsky, D. (2013)”, “More than words: Syntactic packaging and implicit sentiment. ​​Greene, S. and Resnik, P. (2009)” “Was it “said” or was it “claimed”? How linguistic bias affects generative language models. Patel and Pavlick (2021)”

clarity,_quality,_novelty_and_reproducibility: This paper is well written and seems reproducible (assuming the dataset is released). Small clarification questions and typos are added below.

Clarification questions:

1. Page 4. “..the order in which these examples are presented matters.” ← can the authors clarify why this is the case, since there is no explanation given.
2. On naming “natural” vs. “structured” prompts. This seems somewhat non-intuitive–can the authors clarify this naming strategy?
3. Why are the fine-tuned models called “instructable” models?

Small typos:

Page 4. “.. change in protocol allows us to control for two two sources..”


summary_of_the_review: This paper studies an interesting problem however the framing of the paper is misleading.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors construct a test to examine LLM's ability to understand certain types of conversational implicatures by asking the models to convert non-direct answer to yes/no questions to yes/no. They found that intractable GPT-3 models perform much better than other baselines they compared but still a lot worse than human performance. They also showed that scaling doesn't help on this task. In-context learning helps the task up to 5 examples. 
strength_and_weaknesses: # Strength:

1. It is interesting to understand how LLM understand implicatures which is acquired through social learning for humans. The contrastive task design is neat and effective. 
2. The experiments make sense and the results are informative. 

# Weakness

1. Given the small size of the dataset, some of the results for smaller categories may not be significant. It would be nice if the authors could calculate the uncertainty for Figure 3 & 4. 
2. The title is too eye-catching-oriented and a little disconnected to the main claim of this paper. I would suggest use "Measuring implicature understanding" or similar ones to more clearly state the purpose of this paper. Communication is a much broader concepts than implicature. Not being able to understand implicature than average humans doesn't mean LLMs cannot communicate. 
clarity,_quality,_novelty_and_reproducibility: This task and experiment design in this paper is very straightforward, and it is presented clearly. Given access to the models used in this paper, I think the results should be reproducible. 
summary_of_the_review: Good paper on analyzing implicature resolution of LLMs.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors evaluate pretrained large LMs on the binary implicature resolution. The results reveal that there is still a big gap between human performance and LM's on this task, indicating the challenge to interpret the language in context of the LMs. They also show that increasing the model size and prompt size (through few-shot prompting) is unlikely to close the gap, especially for the samples that require more context understanding.
strength_and_weaknesses: Strength:
The paper presents an interesting result of pretrained LMs performance on the binary implicature resolution task. While we are seeing promising performance with the large LMs, especially with the newest ones like OpenAI's, there is still a gap with human performance on this implicature resolution task.

Weaknesses:
The claim that large LMs are not zero-shot communicators is a little strong because it is only based on a single dataset. There will be questions regarding whether this dataset is representative? the dataset size (600 samples) is enough or not? and even the quality of the samples/labels?

For the samples that require context understanding ("Particularised"), authors can show more case studies and analyze the predictions on these samples. Also, I think it is better to plot the absolute accuracy scores in Figure 3 to illustrate whether the performance on this type of samples will increase with the model size.

The claim that "increasing the model size and prompt size is unlikely to close the gap" seems not true for the recent model from OpenAI. Evidence in Figures 2 and 4 show that we are seeing performance gain when going to bigger model or having more few-shot examples in the prompt.

Furthermore, the current paper does not have enough analysis to understand why the current LM models fail and hence suggest directions for improvement.
clarity,_quality,_novelty_and_reproducibility: The idea is clearly presented, and the paper is easy to read. The appendix section provides more details about the evaluation results. The result seems reproduceable without much effort.
summary_of_the_review: Although this work reveals an interesting result when evaluating LMs for conversation context understanding, this evaluation is on a single test set, hence it is not strong enough to make a claim on the general effectiveness LMs. It is also expected to see more analysis to understand why the current LMs model fail on this test and hence have some potential directions for improvements.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This is an interesting paper about what large language models understand. The focus is "implicatures," an important aspect of communication. For example, if you ask if "Did you take out the trash?" and someone says "All 12 bins" the implication is "Yes." The present paper designs a protocol and evaluates several language models on their understanding of implicatures both zero-shot and above. 

The reviewers liked this question and found the execution reasonable but felt that the overall framing and discussion of related work was inadequate. Also, there were concerns about the inter-rater reliability though these were less significant.

In particular, the original submission title "Large language models are not zero-shot communicators" makes a overly strong claim not convincingly established in the paper, and is at the same time not specific to what this paper is about. Furthermore, it is not clear which language models are being referred to: all language models, past, present and future? all present language models? all language models that the authors tried out? The authors propose changing the title to a cute new title that focuses on implicatures, which is good, but it is still carries a tone that makes a vague claim about the ability of (all?) large language models. Important to the decision was the fact that one of the papers that the authors cited a well-known paper that pointed to 14-month-old benchmark for implicatures on large language models. In fact, while the benchmark itself has not yet been peer reviewed, the data it has claims that a few-shot 540B language model outperforms average humans. Whether or not this is true, its existence highlights the problems with the current paper's framing and claims that it is unlikely that scaling will work.

The authors may consider reframing the paper as discussing the *evaluation* of language models on the axis of implicatures, and what is novel/interesting about that.
justification_for_why_not_higher_score: Two major problems: 1. The well-known Big-Bench has a 14-month-old LLM benchmark on exactly the same task, using much larger language models, that the authors were unaware of, even though they themselves cite it. 2. The framing of the paper is sensational and unscientific.
Decision: Reject