PUB: A Pragmatics Understanding Benchmark for Assessing LLMs’ Pragmatics Capabilities (2024)

Abstract: LLMs have demonstrated remarkable capability for understanding semantics, but their understanding of pragmatics is not well studied. To this end, we release a Pragmatics Understanding Benchmark (PUB) dataset consisting of fourteen tasks in four pragmatics phenomena, namely, Implicature, Presupposition, Reference, and Deixis. We curate high-quality test sets for each task, consisting of Multiple Choice Question Answers (MCQA). PUB includes a total of 28k data points, 6.1k are newly annotated. We evaluate nine models varying in the number of parameters and type of training. Our study reveals several key observations about the pragmatic capabilities of LLMs: 1. chat-fine-tuning strongly benefits smaller models, 2. large base models are competitive with their chat-fine-tuned counterparts, 3. there is a huge variance in performance across different pragmatics phenomena, and 4. a noticeable performance gap between human capabilities and model capabilities. We hope that PUB will enable comprehensive evaluation of LLM’s pragmatic reasoning capabilities.

heading: 1 Introduction
text: Pragmatics, within linguistics, examines how context shapes language understanding in communication (Grice, 1975). It centers on real-life language use, considering context, speaker intentions, presuppositions, and implied meanings to derive interpretations beyond literal words. Human’s proficiency in pragmatics stems from their inherent cognitive skills and social awareness. Our minds adeptly process not only spoken words but also context and implied messages. In Natural Language Processing (NLP), Large Language Models (LLMs) (Brown et al., 2020; Scao et al., 2022; Chowdhery et al., 2022; Touvron et al.,
*First author †Equal contribution in coding and experiments
2023) have emerged as a transformative force in recent years. LLMs have shown remarkable abilities on many downstream tasks like Natural Language Understanding (Wang et al., 2019b; Williams et al., 2018), text generation (Paperno et al., 2016; Merity et al., 2016), code synthesis (Chen et al., 2021; Hendrycks et al., 2021), question answering (Mihaylov et al., 2018; Kwiatkowski et al., 2019; Rajpurkar et al., 2018) and reasoning (Wang et al., 2019a; Cobbe et al., 2021; Geva et al., 2021; Clark et al., 2018), etc. While semantics involves the study of words and their meanings in a language, pragmatics extends this inquiry by considering word’s meanings within the context in which they are used.
Given LLMs’ increased interaction with humans via practical, real-world applications like chatbots,
12075
search engines, and web browsers, the following research question arises: Do LLMs understand pragmatics in conversations?
Most benchmarks until now deal only with abilities like problem-solving (Cobbe et al., 2021) or semantic understanding (Wang et al., 2019b; Srivastava et al., 2022) where LLMs have started approaching human performance. However, due to lack of reliable benchmarks, it is still unclear whether an LLM understands pragmatics or not. To facilitate this research, we propose a Pragmatic Understanding Benchmark (PUB)1 over four major pragmatic phenomena: 1. Implicature: understanding what is implied in a statement even though it is not literally expressed. 2. Presupposition: an implicit assumption that is taken for granted before the use of a statement. 3. Deixis: a phenomenon in which certain phrases within a sentence rely on contextual cues, such as the speaker, the listener, or the surrounding context, to convey their meaning effectively. 4. Reference: how language points to things, people, place, time, etc. in accordance with the content and structure outlined in the Handbook of Pragmatics (Horn and Ward, 2004).
PUB includes 22,000 examples, leveraging existing data, and introduces three new datasets with 6,100 newly annotated examples. Human evaluation of a subset of these datasets is conducted to assess performance against established LLMs. The benchmark comprises fourteen tasks that evaluate pragmatics as an MCQA task since MCQA evaluation is more closely related to question-answering abilities in conversations (Robinson and Wingate, 2023). We carefully curate the existing datasets to balance them and formulate prompts for these tasks, which are more natural and better suited to evaluate LLMs. Following ((Brown et al., 2020), (Robinson and Wingate, 2023)), we evaluate the pragmatic abilities of LLMs using Multiple Choice Prompting (MCP) and Cloze prompting (CP). To validate the model’s confidence in its choices we also calculate the Proportion of Plurality Agreement (PPA) on 3 tasks similar to (Robinson and Wingate, 2023).
Our contributions are: (1) a comprehensive and
1The benchmark is available at huggingface.co/datasets/cfilt/PUB
unified dataset for 14 distinct tasks in pragmatics (Figure: 3), containing 28k data points; to the best of our knowledge this is the first datasetlinguistically motivated and well-grounded- to test pragmatic capabilities of LLMs. (2) a systematic evaluation of 6 variations of llama-2, t5, flan-t5, and GPT-3.5, on the fourteen mentioned tasks. (3) a study of human performance on a sample of the dataset to highlight the performance gap between LLMs and humans. (4) insight emerging from (3) to uncover strengths and weaknesses of LLMs vis-a-vis humans. These contribution points- we hope- will assist researchers in improving the interactive abilities of LLMs.

heading: 2 Related Work
text: Pragmatics is very crucial in the domain of linguistics, where it plays a critical role in understanding meaning (Allwood, 1981). In linguistic terms, pragmatics deals with the study of context-dependent aspects of meaning that are systematically abstracted away from, in the construction of content or logical form (Horn and Ward, 2004). Some of the basic subfields of pragmatics include implicature, presupposition, speech acts, reference, deixis, definiteness, and indefiniteness.
Over the years, many researchers have devoted their research to studying such pragmatic phenomena for machine learning. To study implicatures, Louis et al. (2020) employ indirect answers in polar questions, Zheng et al. (2021) utilize hierarchical grammar models for understanding implicature and deictic reference in simple conversations, Jeretic et al. (2020) employ Natural Language Inference (NLI) to grasp scalar implicatures, Deng et al. (2014) leverage implicature rules for optimizing sentiment detection, and Lahiri (2015) develop a sentence-level corpus with implicature ratings. Whereas for presupposition, Kim et al. (2022) use search engine queries that may contain questionable assumptions that are closely related to presupposition. Kabbara and Cheung (2022) also reveals that Transformer models exploit specific structural and lexical cues as opposed to performing some kind of pragmatic reasoning. Cianflone et al. (2018) introduced a computational approach to detecting adverbial presupposition triggers, emphasizing the importance of pragmatics in language understanding. Schuster et al. (2019)
focused on predicting scalar inferences using linguistic signals, while Li et al. (2021) explored neural sentence encoders for predicting scalar inferences. Kim et al. (2021) investigated presupposition verification in question-answering, and Kabbara and Cheung (2023) studied the impact of pre-finetuning BERT models on NLI involving presuppositions. These studies collectively highlight the advancements in computational pragmatics.
A recent comparison of pragmatic understanding between humans and models, conducted by Hu et al. (2023), shows that language models struggle to comprehend humor, irony, and conversational maxims (Grice, 1975). In the most recent work, Ruis et al. (2023) have studied implicature recovery in polar questions and answers. These approaches have offered only a restricted understanding of the shortcomings exhibited by these models by either evaluating only a single phenomenon or using a smaller number of samples to make it quantifiable. Other existing works (Deng et al., 2014; Sileo et al., 2022; Qi et al., 2023) do not comprehensively cover all important domains of pragmatics to evaluate LLMs. To the best of our knowledge, we are the first ones to combine major aspects of pragmatics to create a quantifiable benchmark.

heading: 3 Datasets and Tasks
text: We describe the datasets used and curated for creating PUB in section 3.1. Various tasks for evaluation of LLMs is introduced in section 3.2.

heading: 3.1 Datasets
text: With the help of language experts, we selected existing datasets covering important pragmatic aspects. Specifically, we select Circa (Louis et al., 2020), GRICE (Zheng et al., 2021), FigQA (Liu et al., 2022), FLUTE (Chakrabarty et al., 2022), IMPPRES (Jeretic et al., 2020), and NOPE (Parrish et al., 2021). We adapted datasets for various tasks (in MCQA format) with necessary changes and also made new ones where needed for specific purposes. Annotation details are discussed in Appendix A.
Overview of newly annotated datasets is discussed below: 1. CircaPlus is a newly annotated dataset contain-
ing 2.5k human written implied meanings based
on the indirect responses present in Circa dataset (Louis et al., 2020). 2. DialogAssumptions is a new dataset containing 2.5k pairs of expert-annotated presuppositions based on a subset of dialogues from the Dailydialog dataset (Li et al., 2017). While current presupposition datasets are built around trigger words present in sentences, to our understanding, there hasn’t been a resource addressing presuppositions in conversational contexts where trigger words are absent. Hence, we developed this dataset specifically to fill this gap. 3. MetoQA is a novel dataset comprising 1100 multiple-choice questions based on the linguistic phenomenon called metonymy. Metonymy is a figure of speech in which one word or phrase is substituted with another word or phrase with which it is closely associated or related. Unlike a metaphor, where one thing is said to be another (e.g., “Life is a journey"), in metonymy, the substitution is based on a real, often contiguously related, connection between the two terms (e.g., “These are my hired guns").

heading: 3.2 Tasks
text: Each task incorporated within PUB is structured to evaluate distinct domains of pragmatics. Owing to the importance of Implicature in pragmatics, this benchmark includes a greater focus on Implicature, with ten tasks designed to thoroughly evaluate models’ abilities. Presupposition and Reference are covered through two tasks each. Figure 3 contains details and illustrations for each task introduced in PUB. Additional description for all tasks is given in Appendix B.

heading: 3.3 Discussion about Data Leakage
text: LLMs have been trained on a vast amount of openly available data. However, this abundance of data raises concerns about the evaluation sets, as they can yield biased results when exposed to similar data during testing. We assess a wide range of models, which introduces the risk of data leakage. While we cannot conduct exhaustive collision checks with the training corpora of all these models due to their immense size, we have performed several studies to reduce the risk of data leakage in their fine-tuning datasets. Firstly, we have identified that Circa, Imppres, and DailyDialog are components of instruction-tuning datasets, such as Super Natural Instructions (Wang et al., 2022) and Flan (Wei et al.), on which flan-t5
is fine-tuned. Secondly, despite the potential for data leakage, flan-t5 demonstrates competitive performance on datasets it has never encountered before, such as Task 14, which is an entirely new dataset.
Since these datasets are available on public websites, it is likely that some part of the data might be seen in the pertaining corpora of these models, but we suspect the following reasons why data leakage does not affect our results for other models. First, we see that the models perform consistently on new data, and we do not notice a surge in numbers for a particular model on these tasks. Secondly, similar to Robinson and Wingate (2023), we see that shuffling candidate answers does not cause a dip in PPA performance , and if data leakage would have impacted our results then we would see more probability assigned to the correct answer regardless of the order of options as claimed by Robinson and Wingate (2023).

heading: 4 Evaluation strategy
text: We have selected two evaluation methods namely length normalized cloze prompting (Brown et al., 2020) and Multiple Choice Prompting (MCP) (Robinson and Wingate, 2023). Since MCP is also dependent on the multiple choice symbol binding ability of LLMs, we have computed the Proportion of Plurality Agreement (PPA) (Robinson and Wingate, 2023) to ensure the model’s consistency across possible orders of answer options. The mod-
els under investigation include flan-t5-xxl (Chung et al., 2022); llama-2 : 7b, 7b-chat, 13b, 13b-chat, 70b, 70b-chat (Touvron et al., 2023); t5 (Raffel et al., 2020) and GPT-3.5 (Brown et al., 2020)

heading: 4.1 Prompting LLMs
text: We do a zero-shot and a 3-shot evaluation for each of the above mentioned strategies. The OpenAI model is evaluated only using MCP. For Zero-shot prompts, all the instances of the data were used as is. For Few-shot prompts, a dev set of 20 examples was created for each task. These 20 examples were selected to ensure a balanced representation of options. These examples were randomly selected from the entire dataset for tasks with unique options for each question. Three samples were randomly selected from this dev set for 3-shot evaluation. The remaining instances of the data, other than the dev set, were used to evaluate the model. Details about evaluation strategy are presented in Appendix D. Prompts for each task are given in Appendix C.

heading: 4.2 Human evaluation
text: To compare the performance of these LLMs with humans, we selected 100 examples from the complete evaluation set for each task. We employed three human evaluators to answer these 100 samples from each task, resulting in a total of 4,200 human evaluations. The evaluators are fluent English speakers and have graduated from a technical university where English is the medium of instruction. It is important to note that the human evaluation does not reflect expert human reference but rather the performance of a random human on complex pragmatic tasks. These evaluators are presented with the same prompt as the 0-shot MCP presented to the LLMs.

heading: 5 Results and Analysis
text: The results of our experiments are presented in Figures 4, 5. Only the maximum across all evaluation strategies is reported in these figures. Detailed results are given in Appendix E. The results for PPA are presented in Figure 2. Based on these results, the following section aims to answer the questions regarding the pragmatic capabilities of LLMs.

heading: 5.1 Main Results
text: How much do LLMs understand what humans mean during conversations? To evaluate how
well LLMs understand implied meanings of conversations, implicature and reference tasks offer pertinent insights. We observe that the models perform moderately in classifying a response as direct or indirect (Task 1). They also struggle to interpret the meaning of the indirect response (Task 2). However, llama-70b-chat is an exception to this trend. Similar to humans, a noticeable increase in performance is observed when a hint is provided for indirect response interpretation (Task 3). The performance of models is indifferent to both polar and non-polar question answers in
resolving implicatures (Task 2 vs. Task 4). Even though NLI is an established task in NLP, it is observed that models perform poorly in making pragmatic inferences (Task 5). Finally, as in Figure 1, the average performance on implicature and reference is similar, suggesting that these models do not fully interpret human conversations.
Despite operating on the same dataset, do LLMs demonstrate varying task sensitivity? While it’s known that LLMs are sensitive to the wording of prompts (Webson and Pavlick, 2021), this investigation aims to explore their task sensitivity. Specifically, we want to understand how altering the order of speakers, asking a different question or giving a different hint impacts the model’s performance. Although derived from the same dataset, LLMs demonstrate stronger performance in agreement detection (Task 5) over sarcasm detection (Task 6) (on average, there is a 13% performance gap in models ≥ 13b parameters). The tasks designed on flute dataset (Chakrabarty et al., 2022) shed light on the model’s susceptibility to distractions. We can observe that with a change in the hint from positive (Task 8) to contrastive (Task 9), there is a drastic decrease (on an average of 20%) in the accuracy levels. Interestingly, the inclusion of a positive hint, which has a higher lexical overlap with the correct answer, seems to boost the performance of the model. However, the model’s performance decreases when a contrastive hint is introduced. This observed pattern brings into question the pragmatic abilities of these models, suggesting that their understanding and interpretation of language may be more
significantly influenced by the presence and nature of linguistic cues than by inherent logic.
Does a Model’s Scale Correlate with Its Pragmatic Abilities? The performance shown in Figure 1 hints at a possible correlation between a model’s scale and its pragmatic capabilities. However, given the model’s vulnerability to task sensitivity, even the largest models display perplexity, as previously discussed. Consequently, concluding that pragmatics is an emergent ability might be premature due to observed inconsistencies, even among models at the extremes of the scale.

heading: Do LLMs that are optimized for dialogue use cases exhibit superior pragmatic abilities?
text: From the experiments, it is evident that the chatoptimized variants of llama slightly outperform
the base models on most of the tasks. There is a notable performance gap between models like t5-11b and flan-t5-xxl, with the instruction-tuned flan-t5-xxl model approaching near-human-level performance in many of the tasks. This suggests that instruction tuning can significantly enhance a model’s ability to handle complex language tasks, bringing it closer to human-like language comprehension.
How do the pragmatic abilities of LLMs compare concerning world knowledge involvement? All implicature tasks (except Tasks 1 and 4) involve a certain degree of world knowledge. In Reference, while the metonymy task (Task 14) requires world knowledge, the Deixis task (Task 13) does not. The model’s below-par performance is not primarily due to a lack of world knowledge. Instead, it appears to stem from a deficiency in their innate pragmatic abilities. This is evident because even in tasks not reliant on world knowledge, like Deixis, the model’s performance isn’t on par with tasks involving world knowledge. It suggests that the challenge lies more in the model’s pragmatic processing than its knowledge base.

heading: Do they understand the same implied meaning and make the same assumptions as humans?
text: The models demonstrate relatively stronger per-
formance in tasks related to implicature and reference, both of which involve inferred meanings from the speaker. However, the models exhibit shortcomings in capturing the speaker’s assumptions, known as presuppositions. On average, there is a performance gap of ∼ 15% between humans and the best-performing model on these tasks. Notably, the model’s sensitivity to hints and task variations is also an important aspect. Human performance remains consistent across sarcasm detection and agreement detection tasks, whereas the models show significant performance discrepancies in these tasks (with an average difference of 13%). Similarly, this gap is observed in tasks concerning figurative language understanding, with models showing an average gap of ∼ 25% and human performance only differs by 1%.

heading: 5.2 Error Analysis
text: In this section, we analyse cases where LLMs fall short in simple pragmatic understanding tasks that humans do with ease. Specifically, we consider the llama-2-70b base model due to its consistently high performance across various tasks. We compare mistakes of humans and LLMs to see if there is any correlation in pragmatic understanding and if so, is it significant?
To see this correlation, we report ϕ (Matthew’s correlation coefficient) in Table 1 comparing
human-LLMs (llama-2-70b-base) and human-GT correlation values. ϕ ranges from -1 to 1 where 1 means total agreement, 0 means the predictions are random with respect to the actual values, and -1 means total disagreement. For most tasks, the human-LLM correlation values are above random. This suggests that models make some mistakes similar to humans, but this is far from a human-ground truth correlation. For instance, the performance of LLMs is comparable to humans for response classification with implied meaning (Figure 4 - Task 3), but the correlation values say otherwise. This is further supported by Figure 6 showing that LLMs do make different mistakes than humans during classification.
Now, we present examples for each pragmatic phenomenon to understand the pragmatic abilities of LLMs qualitatively. For response classification (Figure 7), the model selects that the response is true given some conditions are met, unlike humans, who consider the context only as auxiliary information (Example 1). We also encounter examples where Y’s response is what we call a “polite decline” since there isn’t a direct no in the response but an implied No in a tactful manner (Example 2).
For understanding implicature in figurative language, we often see responses where metaphors, hyperbole, and tautological statements exist but are in agreement with the speaker. Figure 8 shows that the model often confuses agreements with figurative language as sarcastic disagreement (Task 5) but can correctly differentiate sarcastic statements from statements that agree with the speaker (Task 6).
Using distractors in figurative-language understanding tasks shows LLMs’ vulnerability in their pragmatic abilities. Humans are robust to see that the hint is contrasting and helps distinguish the options in the context and choose the correct one (Figure 9).
In instances of presupposition, we observe a recurring pattern where the model erroneously interprets negatives as positives. In Figure 10, Speaker A expresses frustration about the unsanitary condition of the room, attributing it to the presence of cockroaches. However, the model incorrectly dismisses the notion that being "knee-deep in cockroaches" signifies unhygienic conditions, deeming it an invalid presupposition.
Although llama-2 achieves better results than humans in Metonymy understanding, it makes trivial mistakes where humans get it right. Humans fail, too, when a reference is one that they are not familiar with, but LLMs, due to access to vast and diverse sources of texts, get it right. This task requires common sense and world knowledge to un-
derstand references that humans learn over time. Figure 11 shows examples where the LLM takes the semantic meaning of the reference instead of the pragmatic one.
This error analysis shows that LLMs don’t make the same mistakes as humans. Importantly, LLMs fail in trivial cases where humans easily understand the underlying pragmatic answer. More insight into why LLMs fail in such cases is in the scope of our future research work.

heading: 6 Conclusion
text: In this study, we introduce the Pragmatic Understanding Benchmark (PUB) designed to assess pragmatic comprehension in LLMs. We offer a detailed analysis, providing insights into various aspects of pragmatic understanding within LLMs. Our observations reveal that pragmatic understanding in LLMs can be enhanced through instructiontuning of these models. Interestingly, even without specific fine-tuning, language models at scale exhibit equivalent performance. Notably, smaller models, particularly the instruction-tuned variants, outperform their base counterparts, but this advantage diminishes as models scale up, with base and instruction-tuned models showing comparable performance. Despite advancements, LLMs are yet to attain human-level performance, especially in tasks requiring a deep understanding of language context. The observed variability in model performance
across different tasks within the same dataset highlights the complexity of achieving human-like pragmatic understanding in LLMs. The PUB benchmark thus provides a clear indication of where LLMs currently stand and the strides still needed to reach human parity in language understanding. We hope that this benchmark will aid researchers in improving LLMs’ conversational abilities with humans.
Limitations
Our work addresses an important benchmark that can be used to understand and improve the chat capabilities of language models. While we carefully put together a benchmark for evaluation, it’s important to note that there might be biases present that may show up in evaluations. Furthermore, we employed different sampling techniques to avoid evaluation bias for different classes. Although we tried our best to evaluate the models consistently, the models are sensitive to prompt wordings. For the same prompts too, the models are not consistent with the answers when changed the order of options as mentioned in PPA. Therefore there can be slight variations in the performances when trying to reproduce the results. The human evaluation scores reported in the paper are done by graduate students who are proficient in English and language understanding, the results may vary for different sets of human evaluators. The inconsistency of language models is another issue for MCQA results (Robinson and Wingate, 2023), since inconsistency in answers can lead to false results but until better evaluation methods arrive, we rely on the methods currently used in the paper.
Ethics Statement
This study adhered to the ACL Ethics Policy. All annotators and human evaluators received fair compensation. Our datasets solely serve the purpose of evaluating the pragmatic comprehension of LLMs. We make our dataset available for research and educational purposes, with no expectation of it being misused for malicious intent.

heading: A Annotation details
text: For CircaPlus, Considering the subjectivity inherent in implicature, we employ two expert English linguists for the annotation process and implement double-blind checking for the annotations. For DialogAssumptions, all the conversations from the DailyDialog dataset were given to 2 linguistic experts. These experts were asked to add presuppositions to random dialog turns from the datasets. The annotators were also instructed to create false presuppositions and mark them as invalid. Only those examples that are in agreement of both the experts are considered. The Metonymy dataset is curated by four graduates of Literature and two linguist experts from a reputable university. The annotators are given basic examples from Wikipedia and a list of metonymic words as references. We encouraged the annotators to discover new metonymic words in order to avoid repetition in the data. They create these examples from scratch while referring to the provided instructions and examples. All the examples were verified by the experts.

heading: B Tasks
text: B.1 Implicature
Implicature, an unspoken aspect of a speaker’s meaning, extends beyond the literal content in a speaker’s message. Understanding implicature is crucial for LLMs, as it allows them to interpret context, discern implied messages, and produce responses that surpass literal text, ensuring more contextually suitable, human-like, and meaningful interactions. Owing to the importance of implicature in pragmatics we have designed ten tasks that thoroughly test the LLM’s abilities to capture this phenomenon. Task 1 - Direct/Indirect classificationThis task evaluates language models’ capability to distinguish between direct and indirect responses, crucial for understanding user intentions in dialogue systems. The model receives context, a question, and a response (that can be direct or indirect) and then selects between two options: A) Direct answer and B) Indirect answer. We utilized a label-balanced set of 2,500 data points sourced from the Circa dataset for this purpose. Task 2 and 3 - Response classification without implied meaning and with implied meaning: Task 2 involves categorizing indirect answers using five labels. The model receives context, a question, and an indirect answer and must choose the most fitting label from options A) Yes, B) No, C) Yes, subject to conditions, D) In the middle, neither yes nor no, E) Other. This task evaluates LLMs’ ability to comprehend indirect responses, specifically within polar Question and Answer scenarios, utilizing the Circa dataset. Task 3, an extension of Task 2, introduces implied meanings as additional cues to assist LLMs in interpreting indirect answers. The implied meaning acts as a chain-of-thought prompt for understanding indirect responses, assessed using the CircaPlus dataset. Both tasks involve evaluating 2,500 data points. Task 4 - Implicature recovery Task 4 differs from tasks 2 and 3 by focusing on implicature recovery in non-polar Question and Answer contexts. In this task, we present the conversation which is a sequence of QAs (Q1, A1), (Q2, A2), .., (Qn, An) and four choices for the implied meaning of An. The task for the model is to select an appropriate choice that resolve’s the implicature to its explicit form, i.e., to perform implicature recovery. We use 2000 data points from the Grice dataset for this task.
While prior tasks have focused on understanding implied meanings in conversations devoid of figurative language, it’s important to note that figurative language is a common feature in human communication (Lakoff and Johnson, 2008). Understanding the underlying meanings when such language is used in dialogue is crucial. Therefore, to provide a comprehensive benchmark, we are introducing tasks that focus on understanding implied meanings in conversations where figurative language is present. Task 5 and 6 - Agreement detection and Understanding sarcasm Task 5, "Agreement Detection", and Task 6, "Understanding Sarcasm", are both designed to evaluate a language model’s ability to comprehend and interpret figurative language within a dialogue. In Task 5, the model is given a conversation between two speakers, a question, and two options: A: Agrees and B: Disagrees. Speaker 1 uses figurative language, and Speaker 2 responds either in agreement or disagreement. The model’s objective is to accurately determine if the second speaker concurs with the first. Task 6 flips the roles from Task 5. Here, Speaker 1 makes a statement, and Speaker 2 responds with ’yes’, but continues the sentence using figurative language to either agree or disagree (refer to Figure 3 for examples). The model is then tasked with correctly determining if the second speaker is in agreement with the first or is being sarcastic. Modifications are applied to the (Liu et al., 2022) dataset to accommodate both tasks. The evaluation involves 2000 data points for each of the tasks. Task 7, 8 and 9 - Figurative language understanding using positive and contrastive hints Tasks 7, 8, and 19 are formulated based on the FLUTE dataset (Chakrabarty et al., 2022). The FLUTE dataset consists of sentences or premises in figurative language and their corresponding hypotheses in simple language. For each premise, there are two types of hypotheses: one that entails and another that contradicts. Additionally, the dataset includes separate explanations for the entailment and contradiction. In Task 7, the objective is to test if the figurative language is correctly understood. The model must choose between an entailed sentence or a contradictory sentence as the meaning of the premise. In Task 8, the model is provided with an explanation of the entailment, which is referred to as a positive hint as it explains why the entailment option is the correct meaning of the premise. In Task 9, an explanation of the contradictory statement is provided, along with an explanation of why it is not the correct meaning of the figurative sentence. This is considered a contrastive hint. Through these tasks, we aim to test if the models understand the task or if their responses rely on the semantic overlap with the positive hint. The evaluation involves 1770 data points for each of the tasks. Task 10 - Implicature NLI Given that Natural Language Inference (NLI) is a well-established task in the training and evaluation of language models, we have incorporated the NLI task to assess whether the models are capable of making inferences when implicatures are involved. We use 2100 data points from IMPRESS(Jeretic et al., 2020) dataset for this task. B.2 Presuppositions Presuppositions in a sentence are the underlying assumptions or facts that are implicitly accepted as true by the speaker when making a statement. Task 11 - Presupposition NLI In this task, we approach presupposition verification by framing it as Natural Language Inference (NLI), with an objective akin to that of task 10. We use 1800 data points from IMPRESS (Jeretic et al., 2020) NOPE (Parrish et al., 2021) dataset for this task. Task 12 - QA over presupposition This task aims to test the ability of the language models on how well they can capture the speaker’s assumptions in a dialog. We provide the model with a conversation (set of dialogues between two people), presupposition on the conversation, and two options A. Valid and B. Invalid. The task for the model is to determine if the given presupposition is valid or invalid based on the conversation. We use 2500 data points from the newly annotated DialogAssumptions dataset for this task. B.3 Reference Deixis, which involves the act of pointing through language, encompasses expressions that are often among the earliest spoken by very young children. These expressions, such as person deixis (’me’, ’you’), spatial deixis (’here’, ’there’), or temporal deixis (’now’, ’then’) (Yule, 1996), are indicative of individuals, locations, or times. Deixis is a type of reference closely linked to the speaker’s context. Task 13 - Diectic QA This task is designed to access the model’s capabilities in resolving references where deictic terms are used. The model is provided with a conversation containing deictic
expressions, a polar question regarding reference resolution, and two answer options: A. "Yes" and B. "No.". The model’s objective is to accurately determine and provide the correct response to the polar question within the context of the conversation. We selected all the questions and corresponding conversations from the GRICE dataset (Zheng et al., 2021) that have Yes/No answers. These questions were then filtered using a manually curated list of deictic terms. A total of 2000 data points are used for this task. Task 14 - Referential metonymy The task aims to test the model’s abilities to understand language use that involves referring to a target object/individual in terms of a distinctive or saliently associated feature. The model is presented with a context featuring metonymic references, along with a question and four possible options. The task requires the model to choose the most suitable option that correctly resolves the reference in response to the question. We use 1100 data points from the newly annotated MetoQA dataset for this task.

heading: C Prompts used for each task
text: In this section we provide prompts used for each task. Any typos in the shown examples are present in the datasets they are drawn from. The examples presented here are Multiple Choice Prompts (MCPs). Cloze Prompts (CPs) can be obtained by removing the options from the MCPs.
Your task is to label the 'Response' as an Indirect or Direct answer based on the Context and Question:
Context: X wants to know what activities

heading: Y likes to do during weekends. Question: Are you a fan of bars?
text: Response: I love to drink beer at pubs. Options:

heading: A: Direct answer
text: B: Indirect answer Correct option=
Figure 12: Prompt example for Task 1
Your task is to interpret Y's answer to X's question into one of the options:

heading: A: Yes
text: B: No

heading: C: Yes, subject to some conditions
text: D: In the middle, neither yes nor no

heading: E: Other
text: Context: X and Y are childhood neighbours who unexpectedly run into each other at a cafe.

heading: X: Would you like to exchange numbers?
text: Y: I'll get my contacts open here. Options:

heading: A: Yes
text: B: No

heading: C: Yes, subject to some conditions
text: D: In the middle, neither yes nor no

heading: E: Other Correct option=
text: Figure 13: Prompt example for Task 2
Your task is to interpret Y's answer to X's question into one of the options:

heading: A: Yes
text: B: No

heading: C: Yes, subject to some conditions
text: D: In the middle, neither yes nor no

heading: E: Other
text: Context: X and Y are childhood neighbours who unexpectedly run into each other at a cafe.

heading: X: Would you like to exchange numbers?
text: Y: I'll get my contacts open here. Implied meaning: He likes to exchange numbers Options:

heading: A: Yes
text: B: No

heading: C: Yes, subject to some conditions
text: D: In the middle, neither yes nor no

heading: E: Other Correct option=
text: Figure 14: Prompt example for Task 3
Your task is to understand the implied meaning in Speaker_2's last response and give the explicit meaning:
Speaker_1: did Liam leave the watermelons in the attic Speaker_2: no, he didn't Speaker_1: did Jackson leave the watermelons there Speaker_2: he said he was not there Speaker_1: where can I get them Speaker_2: the watermelons are in the bathroom or the laundry Speaker_1: what about the cherries Speaker_2: they are in the kitchen Speaker_1: did you see the cabbages Speaker_2: there is a blue bathtub in the bathroom Speaker_1: did you place the cabbages there Speaker_2: no, I didn't Speaker_1: are all of them there Speaker_2: some are there Speaker_1: how many cherries are in the kitchen Speaker_2: there are at least one there Speaker_1: did Liam put the cherries there Speaker_2: he put them there and walked to the bathroom Options:

heading: A: Liam put the cherries in the kitchen and then walked to the bathroom
text: B: Liam didn't put the cherries in the kitchen C: I put the cherries in the kitchen

heading: D: Liam put the cherries in the kitchen Correct option=
text: Figure 15: Prompt example for Task 4
Your task is to decide if Speaker_2 Agrees or Disagrees with Speaker_1 in the conversation:
Speaker_1: The chair was comfortable like a pillow. Speaker_2: The chair was uncomfortable. Options:

heading: A: Agrees

heading: B: Disagrees
text: Correct option=
Figure 16: Prompt example for Task 5
Your task is to decide if Speaker_2 Agrees or is being Sarcastic with Speaker_1 in the conversation:
Speaker_1: The chair was uncomfortable. Speaker_2: Yeah, The chair was comfortable like a pillow. Options:

heading: A: Agrees

heading: B: Sarcastic Correct option=
text: Figure 17: Prompt example for Task 6
Your task is to identify the correct meaning of the figurative sentence:
Sentence : To add insult to injury, a boy was leading a handsome sheep on a string behind him. Options:

heading: A: To make things worse, a boy was leading a handsome sheep on a string behind him.
text: B: In order to make things a lot better, a boy was leading a handsome sheep on a string behind him. Correct option=
Figure 18: Prompt example for Task 7
Your task is to identify the correct meaning of the figurative sentence from the given hint:
Sentence : To add insult to injury, a boy was leading a handsome sheep on a string behind him. Hint : To add insult to injury means to make a bad situation worse, and in this sentence the boy leading the sheep makes the situation worse. Options:

heading: A: To make things worse, a boy was leading a handsome sheep on a string behind him.
text: B: In order to make things a lot better, a boy was leading a handsome sheep on a string behind him. Correct option=
Figure 19: Prompt example for Task 8
Your task is to identify the correct meaning of the figurative sentence from the given hint:
Sentence : To add insult to injury, a boy was leading a handsome sheep on a string behind him. Hint : To add insult to injury means to make a bad situation worse, but in this sentence the boy leading the sheep makes the situation better. Options:

heading: A: To make things worse, a boy was leading a handsome sheep on a string behind him.
text: B: In order to make things a lot better, a boy was leading a handsome sheep on a string behind him. Correct option=
Figure 20: Prompt example for Task 9
Premise: Amy could prevent Stephen from hiding. Hypothesis: Amy couldn't prevent Stephen from hiding. Options:

heading: A: Hypothesis is definitely true given premise

heading: B: Hypothesis might be true given premise

heading: C: Hypothesis is definitely not true given premise Correct option=
text: Figure 21: Prompt example for Task 10
Premise: Natalie hasn't discovered where Tracy worries. Hypothesis: Tracy doesn't worry. Options:

heading: A: Hypothesis is definitely true given premise

heading: B: Hypothesis might be true given premise

heading: C: Hypothesis is definitely not true given premise Correct option=
text: Figure 22: Prompt example for Task 11
Your task is to deduce if the Assumption is valid or invalid based on the conversation:
Conversation: A: Say , Jim , how about going for a few beers after dinner? Assumption: Jim exists. Options: A: Valid B: Invalid Correct option=
Figure 23: Prompt example for Task 12
Your task is to answer the given question based on the conversation:
Conversation: Speaker_1: did you go to the basement Speaker_2: I walked to the cellar Speaker_1: did you see the beans Speaker_2: I have no idea Speaker_1: what about the pumpkin Speaker_2: it is in the hallway Speaker_1: did you see the celeries Speaker_2: there is a green pantry in the cellar Speaker_1: did Mason place the celeries there Speaker_2: he placed them there and walked to the hallway Speaker_1: did he put the peaches in the cellar Speaker_2: no, he didn't Speaker_1: did Lily place them in the cellar Speaker_2: no, she didn't Speaker_1: where can I get the melons Speaker_2: there is a red bottle in the cellar Speaker_1: are all of them there Speaker_2: yes Speaker_1: where are the peaches Speaker_2: the peaches are in the basement Question: are the melons in the cellar? Options:

heading: A: yes
text: B: no Correct option=
Figure 24: Prompt example for Task 13
Your task is to answer the Question based on the given Context:
Context: She is attracted to blue jacket Question: What does "blue jacket" refer to? Options:

heading: A: Colour

heading: B: Jacket

heading: C: Sailor

heading: D: Sea
text: Correct option=
Figure 25: Prompt example for Task 15

heading: D Details about Evaluation Strategy
text: D.1 Cloze prompting (CP) In the cloze prompting approach, a question is given to an LLM, and the model independently scores each potential answer. The answer with the highest probability is selected by the model. Brown et al. (2020) acknowledged that the probabilities of answers could be affected by particularly frequent or rare tokens or sequences of different lengths, so they employed two normalization methods. One method involves normalizing the probability of a sequence for its length by taking the nth root; P (x1, x2, . . . , xn) = n √∏n
i=1 P (xi). The length normalization strategy requires N forward passes through LLMs as compared to 2N forward passes in the other normalization strategy. Since our primary goal is to evaluate the pragmatic abilities and not the normalization strategy, in this paper, we follow length normalization for all the evaluations involving the cloze prompting approach.
D.2 Multiple Choice Prompting (MCP) In Multiple Choice Prompting, a question and its candidate answers, each associated with a symbol, are combined into a single prompt for an LLM. The model is structured to predict only one token (e.g., "A", "B", etc.). The model’s answer is the answer choice corresponding to the token with the highest probability. Consequently, the probabilities of these symbols act as a substitute for the probabilities of each answer. A notable limitation of this evaluation method is that models exhibiting suboptimal performance in the context of multiple choice symbol binding (MCSB) tend to yield inferior results (Robinson and Wingate, 2023). Therefore we also perform the Proportion of Plurality Agreement (PPA) experiments for all the models to estimate the MCSB abilities of these models.
D.3 Proportion of Plurality Agreement (PPA) When presenting a multiple-choice question, the potential answers must be arranged in a specific sequence. In general, human responses to such questions exhibit order-invariance, meaning that the order of the options does not affect the answer selection. (Robinson and Wingate, 2023) have proposed a method to verify if LLMs exhibit the same
characteristics. Given a question with n answer options, there are n! different ways these options can be associated with an ordered, fixed set of symbols. To compute PPA, the model is presented with the question using each unique permutation of the answer options. For every permutation, the model assigns a probability to each answer, and the answer with the highest probability is recorded. Subsequently, the PPA for the question is calculated as the ratio of permutations that selected the plurality answer to the total number of permutations. PPA measures order invariance irrespective of the model’s ability to perform a task. A model with consistent answers across possible orders of answer options will have a high PPA, even if it performs poorly on the task. For a dataset with n answer choices per question, the baseline PPA is 1/n.

heading: E Results
text: In this section, we presented the results of all evaluation strategies in both 0-shot and 3-shot settings in tables 2,3,4,5,6,7,8,9,10,11,12,13,14,15

heading: Future Work
text: More insight into why LLMs fail in such cases is in the scope of our future research work