Diplomat: A Dialogue Dataset for Situated PragMATic Reasoning
Authors: Hengli Li
Song-Chun Zhu
Zilong Zheng
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Pragmatic Reasoning
Implicature
Conversation
Dataset
Inference

Abstract: The ability to discern and comprehend pragmatic meanings is a cornerstone of social and emotional intelligence, referred to as pragmatic reasoning. Despite the strides made in the development of Large Language Models (LLMs), such as ChatGPT, these models grapple with capturing the nuanced and ambiguous facets of language, falling short of the aspiration to build human-like conversational agents. In this work, we introduce a novel benchmark, the **DiPlomat**, which delves into the fundamental components of conversational pragmatic reasoning, encompassing situational context reasoning, open-world knowledge acquisition, and unified figurative language understanding. We start by collecting a new human-annotated dialogue dataset, composed of 4,177 multi-turn dialogues and a vocabulary of 48,900 words. Along with the dataset, two tasks are proposed to evaluate machines' pragmatic reasoning capabilities, namely, Pragmatic Reasoning and Identification(PIR) and Conversational Question Answering (CQA). Furthermore, we probe into a zero-shot natural language inference task, where the significance of context in pragmatic reasoning is underscored. Experimental findings illustrate the existing limitations of current prevailing LLMs in the realm of pragmatic reasoning, shedding light on the pressing need for further research to facilitate the emergence of emotional intelligence within human-like conversational agents.

DiPlomat::
A Dialogue Dataset for Situated Pragmatic Reasoning
Hengli Li
Song-Chun 1,34,5 F.c. zhu@pku. edu. en
Zilong Zheng 103 zlzheng@bi maiiiigi.a.......
lihengli@stu, pku. edu.
National Key Laboratory of General Artificial Intelligence, BIGAI Yuanpei College, PKU IIstitute for Artificial Intelligence, PKU School of Intelligence Science and Technology. PKU DDppartment of Automation, THU
Corresponding author.
https://diplomat-ee htt./ddapplomt-ddaassett dataset github.
we are nepotiating public, we're not really negotiating all.
Which turn of the conversation hus prugmatic meaning?
choose
conrect. out Senate three years and hall; but I was there for 36 years.
Well, I doubt that, Senator
Becaue
[A] being polite respectful towards /B)
Much of my work was budgeting: You would have had me on the TV many rimes, and you would have anymore.
b. [A]'s mind, politiaal figures always tall lies, therefore be questions [B]ss words.
said don't it want
[A] doubts whether would have had senaice the TV.
Well, doubs that, Senator.
d. [A] thinks that he will not only be tized [D)'s words but
Figure I: Illustration DiPlomat dataset. Left: Example of pragmatic conversation. Right: Prag- matic Identification and Reasoning task.
Abstract
The ability to discern and comprehend pragmatic meanings is a cornerstone of social and emotional intelligence, referred to as pragmatic reasoning. Despite the strides made in the development of Large Language Models (LLMs), such as Chat- GPT, these models grapple with capturing the nuanced and ambiguous facets of language, falling short of the aspiration build human-like conversational agents. In this work, introduce novel benchmark, the DiPlomat, which delves into the fundamental components of conversational pragmatic reasoning, encompass- ing situational context reasoning, open-world knowledge acquisition, and unified figurative language understanding. We start by collecting a new human-annotated dialogue dataset, composed of 4,177 multi-turn dialogues and vocabulary of 48,900 words. Along with the dataset, two tasks are proposed to evaluate machines' pragmatic reasoning capabilities, namely, Pragmatic Reasoning and Identifica- tion(PIR) and Conversational Question Answering (CQA). Furthermore, we probe into zero-shot natural language inference task, where the significance of context in pragmatic reasoning is underscored. Experimental findings illustrate the existing limitations of current prevailing LLMs in the realm of pragmatic reasoning, shed- ding light on the pressing need for further research to facilitate the emergence of emotional intelligence within human-like conversational agents.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
The fabric of human sociality is made up of complicated relations that evolve through different dimensions of interaction and communication channels [1, 2]. Social consensuses, such as social norms and values, are thereby formed between humans that convey meanings of individual minds, including beliefs, intentions and desires [3]. In a process of effective negotiation and social con- versation, particularly, such social behaviors are only partly driven by literal meanings that are objective, rational and explicir [4, 5]. Instead, these behaviors are commonly governed by affective pragmatic meanings of dialogue utterances that refer to the emotional and cultural meanings of conversational partners and are subjective, emotional and implicit. For instance in Fig. I, the lady responds with "I doubt that" rather than "I am not tired of you" to express sense of humor and politeness. The competency of perceiving such pragmatic meanings is crucial to social and emotional intelligence (EI) [6] and is referred to as pragmatic reasoning.
The rapid developments of large language models (LLMs), such as ChatGPT and InstructGPT [7], have set off wave of the next generation of conversational AI over the recent years. Despite the inspiring capabilities of language generation [8] and reasoning [9] achieved with massive computa- tional resources and tremendous natural language data, LLMs barely show convincing communicative skills [10], i... they fail to capture pragmatic and ambiguous meanings of input prompts [11, 10, 12]. Critically, current neural generative models are trained to be objective with safe and satisfiable responses [13, 14], which largely deviates from the long-standing goal of building a human-like agent. Recently, Meta Research Team al. [5] introduce ChatBot that demonstrates human-level play in language board game Diplomacy where lying and misleading commonly occur. However, their main focus is on game policy learning rather than pragmatic reasoning.
What are the core components of real-life conversational pragmatic reasoning? Motivated by theories of cognitive linguistics [1-3] and conversational modeling [4, 5, 15, 16), we anticipate it to be fhree-fold:
Situational Context Reasoning. Understanding pragmatic meaning requires a detailed understand- ing of conversational contexts. Consider the utterance "You are making the rest of us looking bad", under different situations of praise and sarcasm, the sentence may convey completely opposite meanings. Furthermore, typical conversational reasoning challenges such as coreference resolution and intention prediction are largely dependent on the success of situated context modeling.
Open-world Knowledge Acquisition. The open-world knowledge includes commonsense knowl- edge (e.g., social ethics) that can be learned from diffferent domains of dialogue corpus and domain-specific knowledge (e.g., American histories). Successful pragmatic reasoning requires the acquisition of open-world knowledge and joint reasoning the conversation.
Unified figurative language understanding. Figurative language is one of the most frequently used tricks for conveying implicit meanings with subjective emotions. Previous works treat different forms of figurative language understanding as individual tasks, such as metaphors [17], idioms [18, 19], pun [20], etc. Pragmatic reasoning provides a feasible unified perspective that considers all these tasks as recovering their literal meanings.
In order step towards general human-like communicative agent, in this work, we introduce DiPlomat, a real-life Dialogue dataset that focuses on Pragmatic reasoning. DiPlomat stems from an interview dataset [2 and experiences three steps of curation: automatic selection, fine-grained manual annotation and human refinement (Sec. 3). Our dataset comprises 4, 177 dialogues and covers vocabulary of 48, 900 words. More than that, human-annotated answers reach the amount of 6, 494 and hold a vocabulary size of 20, 000. Along with the dataset, we propose two tasks, Pragmatic Identification and Reasoning (PIR) and Conversational Question Answering (CQA), to benchmark machines' pragmatic reasoming capabilities (Sec. 4). The CQA task possesses 19, 482 questions concerning the content of collected dialogues and the answers the questions are written by humans. We run extensive experiments on previous state-of- offte----th-arr models on DiPlomat (Sec. 5). The best model achieves less than 0.70 accuracy score in PIR, and none of the models achieve more than 0.50 accuracy score for CQA. Moreover, we test previous pre-trained LLMs' (including ChatGPT) zero- shot reasoning capability with a natural language inference (NLI) task. Regarding the experimental results provided, the significance of pragmatic reasoning speaks for itself. Throughout a thorough analysis of the limitations of current models, we aim to shed light on future research toward building general conversational agents.

---

Table 1: Comparisons on similar datasets and our dataset. QA: Question Answering. NUP: Next Utterance Prediction. NLE: Natural Language Inference. PI: Plausible Inference. IR: Implicature
Recovery. PIR: Pragmatic Identification and Reasoning. Manually: the dataset being checked or collected by humans. Diverse: the dataset lies in several specific domains. Open: the dataset doens't fall into particular domains.
Dataset
Domain
Manually
Task
Implicature
Reasoning
Multi-Turey
Ubuntu (015) [22]
Tochnique
NUP
RACE [EMNLP 2017] [23]
Open
X
X
ARC [ArXiv 2018] [24]
Science
X
QA
X
MNLI 2018 [25]
Open
NLI
X
Persona-Chat (018) [26]
Persona
NUP
X
X
SWAG [EMNLP 2018]
Movic
X
X
Cosmos QA JEMNLP 2019] [28]
Persona
QA
X
CoQA INAACL 2019] [29]
Diverse
QA
X
DREAM (019) [30]
Open
QA
X
Dialogue NLI [ACL [31]
Persona
X
NLI
X
X
DROP [019] [32]
Open
X
QA
X
MuTual |ACL [33]
Open
NUP
IMPPRES (ACL 2020 [34]
Open
X
NLI
GRICE [ACL 2021] [12]
Daily
X
IR & QA
DiPlomat
Open
PIR & QA
V
2
Related Work
Conversational Dataset Tab. provides a comparative analysis of our dataset with similar conversational datasets. The Dream [30] dataset formalizes dialogues from English exams into question
crestion-answernng answering task with focus on in-depth dialogue comprehension. With question- question-answer pairs compiled by two annotators, CoQA [29] focuses on reasoning conversation understanding. By utilizing English listening comprehension tests, MuTual [33] is built to address the issue of general dialogue reasoning. In contrast these preceding datasets, the GRICE [12] dataset represents significant advancement in the field of pragmatic reasoning as it incorporates implicature and reasoning. However, both MuTual and GRICE exhibit a shared limitation in that they not possess data that closely resembles real-world interactions, leading to a lack of diversity in their respective datasets. Additionally, other relevant datasets, including commonsense [28], reasoning [23], and nat- ural language inference (NLI) [31, 25], are also included in Tab. 1. By examining various dimensions such as domain, manual annotation, task variety, implicature, reasoning, and multi-turn interactions, dataset offers unique advantages in addressing the challenge of pragmatic reasoning in dialogues.
Language Models for Dialogue Generation Conversational AI has emerged as a prominent research area within the field of natural language processing (NLP), attracting significant attention and interest. Numerous pre-trained models have been proposed tackle dialogue generation tasks, such as DialogGPT [35]. GODEL [36], LaMDA [37], and Meena [38], and they have achieved marvelous results on competitions [39]. Furthermore, a pivotal milestone has been achieved with the
advent of ChatGPT, garnrring widespread interest and stimulating further investigation in the domain of conversational AI. ChatGPT, built upon the principles transformer models [40], undergoes training on extensive corpus of data, resulting in its profound efficacy. Notably, this system boasts an impressive magnitude billions of parameters. a notable study conducted by OpenAl [13], has been demonstrated that the augmentation of parameters, referred to the Scaling Law, substantially enhances the model's 's capabilities. Also, the enlargement of the number of parameters triggers the emergence of miraculous ability. After the success of ChatGPT, more models such PaLM 2 [41] appeared in the field.
Pragmatic Reasoning Pragmatic reasoning is significant subject within the field of pragmaties, attracting considerable attention from linguists. The Gricean maxims, which one of the most important achievements, serves a foundational theory within the domain of pragmatics. This theoretical framework comprises four distinct maxims: (1) The Maxim Quality, (2) The Maxim of Quantity, (3) The Maxim of Relevance, and (4) The Maxim of Manner [15, 16). In contrast to rigid

---

Reasoning Type
Dialogue Context
Pragmatic Identification
A: What's remakmble about the Edenton Tea Party that it was all women
2u/ of PersonA.
Contexual
possibly the women-driven polirical protest in U.S. history. And guess we can keep beating up British just moment here. A: Why not?
Rationale: This is humorous response indicating agreement continuing to criticize the British.
Reasoning (48.7%)
Figuralive Language Reasoning (26.6%)
A: How has technology changed the way that vaccines are made? Is it easier to sort see responses in living things as they hapgen now? Yes, tochnology improved by leaps bounds.
I turn of PersonB.
Rationale: The implied meaning that
technology improved by large margin.
A: But what's the week been like?
1 of PersonB.
Commonscnse Reasoning (16.4%)
B: You know,
for people in country that's not used hest and doese't Does chateau have fall your head
RRationale: implied meaning that
have much air conditioning. see that the earth, Finnce included, getting hotter?
PersonB emphasizing the weather is getting hotter obviondly.
Extemal Knowledge Reasoning (12.9%)
Poland recently has gone step further, legislating how talk about of the darest periods in its history.
turn of PersonB.
Rationale: punishing those who
B: We are in moment when we decide about how we're going to teach about and remember the Holocause the future.
I suggest Poland complicit
stifting of discussion ahout atrocities.
A: I am getting reports on Twirter. [-.]] That's nothing we have that's verifiahle
I- tuturn of Personf
Others (8.3%)
or his been verified yct.
Rationale: The implied meaning is that PersonB suggests not trust Twitter as relisble source information.
B: We have confirmed those reports. And Twitter, meon, It's good source of information, but we can't report unril we confirm it. A: It's sometimes a good source of misinformation too, isn't it?
Figure 2: DiPlomat dataset samples. Each row illustrates an exemplar case with its reasoning type, dialogue context, pragmatic turn, and the corresponding rationale. Evidence that support the pragmatic identification are marked in orange.
rules or theorems, the Gricean maxims, which capture the prevalent dynamics of conversations, susceptible to frequent breaches in the context human communication. These breaches, stemming from the intricacies of real-world interaction, notably manifest in the violation of one more of these maxims. Such breaches, aligned with the cooperative principle, give rise to pragmatic phenomena that necessitate the engagement of pragmatic reasoning by recipients of the communication [15]. In the field of natural language processing, previous work tries to model the problem, but pre- dominantly focus specific types of phenomena. For example, EPIE [18], PIE [19] center around idiomatic expressions, while MOVER [42] emphasizes hyperbole, and MERMAID [17] investigates metaphor usage. However, paronomasia much more under-studied [20], with researchers frequently intertwining it with humor [43]. In related vein, GRICE [12] endeavors to study implicature in unified manner, but its data does not originate from real-world contexts, thereby lacking diversity and exhibiting conspicuous patterns, Insights into the comprehension of figurative language are provided by Stowe et al. [44] and Chakrabarty et al. [45], with the former specifically delving into the realm of metaphors and idioms, and the latter investigating idioms and similes. The successful completion of our task needs the incorporation of commonsense knowledge. Extensive scholarly efforts have been dedicated to addressing the challenge of leveraging commonsense in various works [33, 46-48).
3
The DiPlomat Dataset
3.1 Data Source Construction
The DiPlomat dataset stems from the Interview dataset [21], which consists of two subsets, a two- party subset comprising 23,714 dialogues and a multi-party subset with 105,848 dialogues. Given our specific focus on conversations involving only two communicators, we exclusively utilize the two-party subset for our research purposes. The Interview dataset itself is real-world collection of NPR radio transcripts, spanning a period of 20 years of NPR programs. The curation process for our dataset involved several stages, including automatic selection, fine-grained annotation, and human refinement. Details are provided as follows.
Step I. Automatic Selection. The extensive size of the source dataset introduces redundancy, and thus requires automatic measures to alleviate the burden of human annotation. Therefore, we employ algorithms and models to perform an initial filtering process. In order to establish unified framework.

---

we consider various types of pragmatic phenomena and utilize different techniques to extract relevant instances from the source dataset. For instance, we utilize the EPIE list [18] for a suringmmacching method to identify idioms in dialogues, and we train RoBERTa [49] on Hypo-XI. [50] for hyperbole detection; refer to Appendix B.
Topic Segmentation Topic segmentation is small operation taken after automatic selection. The original dialogues employed in our study consist of lengthy and multi-turn exchanges, which are ill- suited for our research objectives. Consequently, we implement segmentation process to break down these dialogues into shorter units. To achieve this, we employ two techniques, namely BERTScore [51] and TextTiling [52]. The segmentation procedure starts with computing the BERTScore between adjacent turns and subsequently applying the TextTiling algorithm to the generated BERTScores.
Step II. Fine-grained Annotation. We leverage Amazon Mechanical Turk (AMT) to conduct detailed annotation of pragmatic turns within our dialogues. Workers participating in the annotation task are instructed to select all turns that exhibit a divergence between their literal meaning and their intended meaning. Due to the subjective nature of pragmatic reasoning, we request the workers to provide confidence scores along with reasons for their choices and for each dialogue two workers are recruited. All annotators shall meet the following criteria: (i) from English-speaking countries; (ii) Completion of a minimum of 1,300 tasks with a 98% approval rate. We also present them with detailed instructions and four examples so that workers are clear about our objectives and requirements. The instructions part outlines the step-by-step procedures for accomplishing the assigned tasks and highlights some key points that workers should pay attention to. The four examples offered are representative of classical pragmatic conversations drawn from the field of linguistics, serving as practical references for the workers. To mitigate the intricacies arising from the identities of dialogue communicators, a simplified representation is adopted, whereby the speakers are denoted as PersonA and PersonB. To ensure the reasonableness and quality of the data, we manually examined 30 data samples and blocked workers who are unqualified. As a result, a total of 5,869 dialogues are selected; refer to Appendix B for details.
Step III. Human Refinement. In this process, tasks for workers are formulated as multiple- choice questions. Previously collected human- annotated reasons are transformed into choices, utilizing template format: [turn {turn_ _i]d: [reason]]. In addition, to mitigate the impact of careless workers, we introduce a distractor choice for each gold choice. These distractor choices are generated using BERTScore [51] by selecting the reason with the highest score from other unrelated dialogues. Of note, for each dialogue, equal number of gold choices and distractor choices are provided. Workers are requested to select all reasonable choices for each conversation and warned of the presence of distractor choices. Workers who frequently select distractor choices are blocked, and their work is rejected. Through this refinement process, 1.692 dialogues are filtered out, while 4,177 dialogues are preserved, ensuring the integrity and reliability of our dataset; refer to Appendix B for more details.
Table 2: Statistical feature of DiPlomat.
Dialegues
4.17 x 10
Avg. Turns per Dialogue
4.10
Avg. Words per Turn
12,80
Avg. Human Reason per Dialogue
1.56
Avg. Words per Human Annotated Reason
25.31
Vocabulary Size (dialogue) Vocabulary Size (human-annotated reasons)
4.89 x 10
2.00 10*
3.2 Dataset Statistics
The DiPlomat dataset comprises a total of 4,177 multi-turn dialogues, with each dialogue averaging 4.1 turns. On average, there are 1.56 pragmatic turns per dialogue. The distribution of dialogues with different quantities of pragmatic turns is illustrated in Fig. 4; see Tab. 2 for detailed dataset statistics.
With respect to the motivation introduced in Sec. 1, categorize the process of transitioning from dialogue to human-annotated rationales into five reasoning types:
The comprehension of the context is paramount for this reasoning process. Figurative Language Reasoning: Proficiency in understanding figurative language, such as idioms and metaphors, is indispensable for advancing this type of reasoning.
Contextual Reasoning:
Commonsense Reasoning: The utilization of commonsense knowledge, such as recognizing that chatcau cannot fall from the sky, is vital for this category.
5

---

Pane
Pragmatic Turns
(h) Questions
Tam
Figure 3: Distribution of trigram words of pragmatic turns and questions.
Figure 4: Distribution of dia-
logues w.e.. # of pragmatic turns.
This form of reasoning necessitates knowledge that extends beyond commonsense and is not explicitly mentioned in the dialogue. Others: This category includes pragmatic dialogues that fit none of the above.
External Knowledge Reasoning:
Fig. 2 demonstrates the proportion of each type. The prevalence of data within the context partition prove the significance of context in pragmatic reasoning of real life. Fig. 3a depicts sunburst visualization illustrating the distribution of trigram words within pragmatic turns. The diverse range of trigram words indicates that the DiPlomat dataset enjoys the rich diversity from real-life corpora. and covers wide array of topics. In addition, the recurring occurrence of the words "president" and "world" is observed, demonstrating DiPlomat's slight bias polities and world-wide events.
Task Definition
We propose 2 distinct tasks for our dataset: (i) Pragmatics Identification and Reasoning and (ii) Conversational Question Answering. The former task focuses assessing the capability of models to identify the presence of pragmatic phenomena and their ability to select suitable answer for such identification. The latter task aims to evaluate the models' adeptness in employing pragmatic reasoning by presenting them with carefully designed questions.
4.1 Task 1: Pragmatics Identification and Reasoning (PIR)
In this task, models are provided with dialogues and are required to identify turns whose actual meanings deviate from their literal interpretations, commonly referred selections are accurate, a set of rationales is presented and they are expected to choose the most plausible reason for each pragmatic turn. For each turn, there are 5 candidate reasons available, comprising one gold choice and four disturbing choices. The model's success in this task depends the precise execution of both steps. We consider three diagnostic settings test the machine's capability on pragmatic understanding:
pragmatic turns. If their
Conversation - Pragmatic Turn (C P). For each instance, models are presented with dialogue and specific turn extracted from that dialogue. They are then required to determine whether the given turn qualifies pragmatic turn. Consequently, the dataset is flattened to a total of 17,129 instances, with each instance corresponding to single queried turn. It's important to highlight that turns without pragmatic meanings are also extracted for evaluation.
Conversation + Pragmatic Turn - Rationale (CP + R). In this subtask, we offer the model both the dialogue and the pragmatic turn and it needs to choose the most plausible rationale out of five candidate choices.
Conversation -> Pragmatic Turn + Rationale (C PR) In this subtask, models pre-trained the previous two subtasks are combined to infer the final results. Specifically, the model obtained from the first subtask is utilized for determining pragmatic turns, while its version finetuned on the second subtask is employed for selecting the most suitable rationale. is worth noting that, in contrast C - P. this subtask, extracted turns are limited to pragmatic turns only.

---

Pragmatic Turns and Gold Choices Recall that in our data collecting procedure outlined in Sec. 3.1, besides asking workers to select pragmatic turns, they are also instructed to fill reasons to explain their choices. To simplify the evaluation process, the selected turns are directly utilized as answers for the first subtask, while the reasons provided by the workers serve as the designated correct choices (referred to as "gold choices") for the second subtask.
Distractor Choice As a result of the time-consuming nature of BERTScore [51], an alternative approach is adopted for measuring sentence similarity using Pentence-Transfommeee [53], which is a significantly faster method. In our methodology. for each gold choice, four alternative choices with high similarity scores are selected from the pool of gold choices to serve as distracting answers. De- spite their high similarity scores, upon careful examination within the given context, it's apparent that the distracting answers convey entirlly different meanings from the gold answer. This characteristic makes them appropriate components to build our task.
4.2 Task 2:
Conversational Question Answering (CQA)
The ability to apply pragmatic reasoning crucial for effective communication and achieving thorough grasp of the natu- ral language system of human beings. To address this, we propose conversational question-answeringg. wherein multiple ques- tions are formulated for each dialogue, and an example shown in Fig. 5. The ques- tions focus on delving deeper into dia- logues, often needs insights into intended meanings to answer. ChatGPT plays piv- otal role in question generation and thanks to AMT, we can ensure the collection of high-quality answers. Ultimately, 19, 482 question-answer pairs are assembled.
A: Finally, Ron, lots of talk about Congress releasing the second half of this $700 billion bailout this week. Where do we stand with that? B: Quite possible that Congress will get that done this week now that Burack Obama has asked George Bush, has asked the current president, to formally put a request for that money. Congress has got lot of questions about how this money is going be spent, as has questions about how the first half of the money was spent.
Rationale: Barack Obama's request for the $700 billion bailout may expedite the process.
Q: What may expedite the process?
A1: Request
Q2: Which president- -llceeeeeteeeetteeee requested the S700 billion bailout
to be released?
Agt Obama
Question Generation ChatGPT is em- ployed to generate questions with prompts consisting of dialogues and human- annotated reasons. We task it to generate questions challenging for individuals who are unaware of the dialogues' intended meanings. More than that, for the convenience of evaluation, the question is also asked be able answer within ChatGPT to start the questions with "What", "Which" or "How". A preliminary assessment is carried out by sampling a few examples out of the question pool guarantec quality.
Figure 5: Conversational Question Answering example.
two words. Furthermore, ensure diversity, instruct
Answer Collection The answers to the questions are obtained through the utilization of AMT. Each worker is provided with a dialogue along with several associated questions and it is requested to answer the questions in one single word. To minimize the potential for misinterpretation, we offer an example coming from dataset, which is annotated by the author itself. Through the process of human annotation, we consistently evaluate the collected data and reject unqualified answers as well as block workers who fail to meet our standards.
Statistical features of Questions Fig. 3b showcases the diverse range of our questions. These questions encompass variety of sentence structures, starting with interrogative words: What, Which, and How, and possessing a large diversity of the words that follow the interrogative ones. Apart from the questions, the answer set holds vocabulary size of 8,179, which is also of great diversity and raises a challenge for models.
5 Experiment
5.1
Pragmatics Identification and Reasoning
For C ++ P. we partitioned the dataset into distinct subsets for training, validation, and testing. The training set consists of 13,708 examples, surpassing the 1,361 instances in the validation set and the 2,060 instances in the test set in terms of size. Models are trained on the training set and their
7

---

performance is evaluated on the validation set after each epoch to determine the optimal check- point. The best checkpoint is subsequently loaded for the final evaluation on the test dataset. The evaluation metric employed is the accuracy score, calculated as the ratio of correct predic- tions to the total number of instances. Simi- larly, for the task CP - R, the dataset is also partitioned into training, validation, and test subsets. The respective sizes of these sub- sets 5,188, 244, and 1,062 examples. The training and evaluation procedures are identi- cal those of the previous subtask. For C -
Table Pragmatics Identification and Reasoning Results, The numerical results are accuracy scores in their percentage.
C P
CP - R
C
PR
Random
50
20
10
BERT baae
63.2 2: 1.1
91.3 0.7
50.2
RoBERTabos
64.4 1.3
92.0
50.0 = 11.28
RoBERTalage
0.0
60.8
0.0 0.0
GPT--2
6.74
13.06 1.1
DialoGPT
PaaloGPTacddam Teedium
0.6
24.5
3.8
DeBERTa Tatone
64.9
92.6 + 0.6
43.9 1.2
ALBERT buse
65.1 0.4
90.6 + 0.2
34.9 1.8
PR, the test sets in previous subtasks are taken for evaluation, and it's worth noting that their test sets consist of exactly the same instances. This design ensures the prevention of any leakage of the test set into the training set, thereby maintaining the integrity of the evaluation process.
Results and Analysis We present four key ob- servations for this task:
context
& The primary performance bottleneck lies in the subtask C- As shown in Tab. 3, the best model achieves accuracy score of 92.0% in the CP + subtask, indicating that models possess the ca- pability to reason to some extent. However, when comes to the C PR task, the best-performing model achieves only 50.2% accuracy, while the highest accuracy achieved in the C - task 65.0%. The substantial difference between the score of 92.0% and the score of 65.0% suggests that the difficulty in accomplishing the task pri- marily stems from the
other
external
figurative
common
P subtask.
Accumulated variance in the C PR subtask. The models exhibit significant variance in the re- sults of the third subtask, which can be attributed to the variance introduced by its constituent tasks.
Figure 6: Models performances of different rea- soning types. The taxonomy of reasoning types comes from Fig. 2.
The significance of pragmatic awareness in lan- guage models. Both the C-> P and C PR subtasks require pragmatic awareness, and the poor performances of the models on these subtasks highlight their limitations in accurately determining the optimal timing for deploying reasoning abilities.
As depicted in Fig. 6, we assess the performance of existing models across five dimensions by leveraging the taxonomy outlined in Fig. 2. It becomes evident that models exhibit nearly uniform performance across these dimensions, leading us to the inference that pragmatic reasoning constitutes cohesive task, and segregated approach is ill-suited for its treatment.
5.2 Conversational Question Answering
Similarly to the previous task, the question-answering dataset is divided into training, validation, and sets, comprising 15,585, 1,559, and 2,338 instances, respectively. Experimental subjects includes BART [54]. T5 [55], UnifiedQA [56], and mT5 [57]. The metric adopted is also accuracy score. Given CharGPT's impressive performance in MMLU and A12 Reasoning Challenge [13], we further examine ChatGPT's capability in the context of CQA by prompting it to provide one-word answers to questions. However, due its uncontrollable nature, the generated answers may not always align with our desired settings. Hence, we introduce two evaluation metrics for ChatGPT: (1) em (exact match), which requires ChatGPT to produce the exact same word as our answer, and (2) pm (partially match). where we consider ChatGPT to be correct as long as our answer appears in its generated output. Two configurations are employed for evaluation. the first configuration, models receive dialogues and questions while remaining blind to human-annotate annotated rationales. The

---

Table 4: CQA task results with/without human annotated rationales. The numerical results are accuracy scores in their percentage. em: exact match, pm: partially match.
BART-buse
T5-small
mT3-small
UniliedQA-bose
UnilicddAA-lage
CharGPP (em)
ChatGPT (pm)
wro rationale
20.2 # 1.1
24.9 + 0.7
19.7 04
28.9 =
32.8
1.0
40,6
w/ rationale
29.6 + 0,6
34.1 + 1.4
29.8 =008 0.8
38.8 = 0.2
42.4 0.6
1.5
45.1 +5 (12.35)
+9.4 (46.5%)
49.0 (38.8%)
+10.1 (51.3%)
+9.9 (34.1%)
(29,69)))
+0.5 (60%)
second configuration is contrasting experiment with configuration, we run each model three times using different random seeds and report the mean and variance of their results as the final outcomes.
rationales provided. For each
Results and Analysis The experimental results are presented in Tab. 4. Our observations can be categorized into three main aspects. Firstly, the importance of pragmatic meaning is proved. As shown in Tab. 4, there exists a notable disparity between the results of models that have access to human- annotated answers and those that do not. On average, the model performances improve by 38.47% after the introduction of human-annotated rationales. Even the lowest-pert pereoorminno model in the initial experiment, - demonstrates 9.4% increase in accuracy. The substantial discrepancy in results between the two configurations underscores the significance of elucidating intended meanings in the development of effective communicators. Second, the models display deficiencies in applying pragmatic reasoning. Our questions are designed to demand a deeper understanding of conversations, however, the models struggle to perform well on our task. The best-performing model, ChatGPT, achieves an accuracy score of 40.6%. It is worth noting that our questions are generated by ChatGPT itself, and our source dataset, Interview. was proposed prior to the emergence of ChatGPT, which means that ChatGPT may have encountered our text during training. These characteristics render its result unsatisfactory. Third, generalization across different types of pragmatic reasoning poses challenges. In this analysis, we focus exclusively on models other than ChatGPT due to the lack of clarity regarding its training process. As demonstrated in Tab. 4, these models showcase substantial improvement in performance following the inclusion of human- annootted rationale. The extent of this improvement exhibits slight fluctuations among the various models, suggesting shared obstacle that hinders their overall performance. Noticed that the models are fine-tuned on a training set that is 5.2 times larger than the test set, can conclude that achieving effective generalization from one pragmatic reasoning process to another remains formidable and challenging task.
5.3 Zero-Shot Natural Language Inference
We conduct the natural language inference (NLI) task [60] to evaluate the model's comprehension of language and to emphasize the importance of context in pragmatic rea- soning. Different from previous tasks, zero-shot NLI task sheds light models' initial ability as they are tested without finetuning. The task involves providing two sen- tences: premise and hypothesis, and models are quired to determine the relationship between the two sen- tences, which can be entailment, contradiction, neutral. As there are no negative samples in our dataset, we sim- plify the task by asking the model judge whether there is entailment. In this task, models are presented with a dialogue, turn of the dialogue, and an intended meaning, they need to judge whether the turn entails the intended meaning. Noticed that collected data as described in See. 3.1 consists of reasons and implied mean- ings, to better fit our task, we abandon the reasons and preserve the implied meanings. Models are tested under zero-shot setting, which means that they are not allowed train on any data before testing. Thus the innate abilities of models play a decisive role. Baseline models include T5 [55], DeBERTa [61], and ChatGPT. It's important to note that ChatGPT and the other two models are tested on different settings. ChatGPT is tested with the whole dialogue and the implied meaning as prompt. However, to inspect the significance of context, the other two models are only provided with the pragmatic turn and the corresponding pragmatic meaning. ChatGPT is evaluated using two types
Table 5: Results of Natural Language
Inference Task. : T5-XXL fine-tuned on true NLI mixure [58]. f; DeBERTa- v3 trained on MNLI [25] and SNLI [59].
Method
Acc.
Random
50.0
DeBERTa-Large
44.3
T5-XXL
45.3
ChatGPT
85.7
ChaiGPT w/ CoT
63.8

---

of prompts: with and without step- steppp instructions, "Let's think step by step" is prompt discovered [62] improve the model's reasoning ability; refer to Appendix C for more details.
Results and Analysis Results are listed in Tab. 5. As this task shares similar settings as binary classification, randomized answer accuracy is expected be 50%. We observe below randomized performance on some previous SOTA models. Note that, each of the data is annotated by two humans, thus it's reasonable view human performance as 100%. ChatGPT achieves the highest result but still shows huge gap compared with human annotations. The outcomes highlight the imperfectness of the models' reasoning abilities. For Ta-large and DeBERTa, context blinded, but for ChatGPT, it is reachable. Hence, the performance gap among Ta-large, DeBERTa, and ChatGPT shows the importance of context in our task. Interestingly. CoT doesn't offer help ChatGPT but is harmful to ChatGPT's performance.
6 Discussions and Future Work
In this paper, we propose DiPlomat, a highh-quality manually annotated multi-turn dataset of pragmatic reasoning in conversations. Along with the dataset, propose two tasks and baselines. Comparing experimental results, we emphasize the nonnegligible impact of contexts and reasoning on building perfect communicators. We also highlight the importance of pragmatic awareness and its bottleneck effect on our tasks. There still a significant disparity between current performances and established standards.
Memorization vs. Reasoning Noticed that models exhibit outstanding performance on CP R of Sec. 5.1. On the contrary, for C - PR, models achieve poor results. Since the underlying knowledge is consistent for both tasks, the disparity performance is hypothesized be attributed to memorizution. Instead of truly understanding the knowledge, the models tend to memorize patterns.
Subjectiveness vs. Objectiveness Ji al. [63] emphasize the importance of modeling distribu- tion that encompasses a diverse range of possibilities, rather than solely relying on single "best" prediction. During the annotation process, we observe phenomenon that different workers hold diverse opinions regarding pragmatic turns and their intended meanings. Their annotations often exhibit significant variations, sometimes even presenting completely opposing interpretations. We maintain the possibility of subjectiveness with careful task metric design (Sec. 4.1).
Larger Models Smaller Models Conventional wisdom suggests that larger models, endowed with a greater number of parameters, generally exhibit superior performance in comparison to their smaller counterparts. However, an intriguing observation comes the forefront within the context of the Pragmatic Identification and Reasoning (PIR) experiment, as elucidated in Table 3: RoBERTaaas surpasses RoBERTalage in performance. We posit that this phenomenon can be attributed to a dual- fold rationale. First and foremost, divergence emerges between the domain of pragmatic data and the domain of pretraining data for language models. Consequently, larger models exhibit a more substantial and consistent influence of their pretraining data, rendering them less adaptable to domain shifts towards pragmatic data. Secondly, the current landscape of pragmatic data characterized by its inherent diversity and relative scarcity. This inherent diversity poses a particular challenge for larger models in adapting to such heterogeneity; refer to Appendix
Future Work Achieving generalization across multiple types of pragmatic reasoning processes poses significant challenges. Consequently, we propose that the construction of proficient communi- cator necessitates the incorporation of methods beyond purely data-driven approaches. Furthermore, the availability of comprchensive evaluation data is of utmost importance. As result, we target more high-quality datasets and new methods other than data-driven for the problem.
Acknowledgements
The authors thank Jiaming Yu at Oxford for the preliminary survey, Dr. Xue Feng and Dr. Lifeng Fan at BIGAI for helpful discussions data collection, Ms. Zhen Chen at BIGAI for designing the teaser figure, Junpeng Li and Dr. Zixia Jia at BIGAI for application experiments. This work presented herein is supported by the National Key R&D Program of China (0222D0114900) and the National Natural Science Foundation China (62376031).
10

---

References
[1] Jens Ambrasat, Christian von Scheve, Markus Conrad, Gesche Schauenburg, and Tobias Schröder. Con- sensus and stratification in affective meaning of human sociality, Proceedings of the National Academy of Sciences (PNAS), 111022::001-80066 2014.
[2] Alan P Fiske. The four clementary forms of sociality: framework for a unified theory of social relations.
Psychological review, (9(4))689, 1992. 2
[3] John A Bargh and Tanya L Chartrand. The unbearable automaticity of being. American psychologist. 54 (7):462, 1999. 2
[4] Edward Finegan. Language: Its structure and use. Cengage Learning. 2014.
[5] Meta Fundamental AI Research Diplomacy Team (FAIR)t, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty. Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human- level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378 106-4))107710074, 2022. 2
[6] Silke Anders, Roos de Jong. Christian Beck, John-Dylan Haynes, and Thomas Ethofer. neural link between affective understanding and interpersonal attraction. Proceedings of the National Academy of Sciences (PNAS), (1311))E2248-E222
2016. 2
[7] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback, Advances Neural Information Processing Systems (NeurIPS), 35:27730-2777 277444. 2022.
[8] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2
[9] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fd Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[10] Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. Large language models are not zero-shot communicators. arXiv preprint arXiv:2210. 14986, 2022. 2
[11] Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swahha Swayamdipta, Noah A. Smith, and Yejin Choi. We're afraid language models aren't modeling ambiguity. 2023. 2
[12] Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. GRICE: grammar-bascd dataset for recovering implicature and conversational rEasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2074-2085, Online, August 2021. Association for Computational Linguistics. 2, 3,4
[13] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303. 2023. 2,3.8
[14] Amelia Glaese, Nat McAleese, Maja Tıębacz, John Aslanides, Vlad Firoiu, Timo Ewalds. Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et Improving alignment of dialogue agents via targeted human judgements. arXiv preprint (tti::2209.137777 2022. 2
[15] Alan Cruse. Mcaning in language: An introduction to semantics and pragmatics. 2, 3, 4, 24
[16] Herbert P Grice. Logic and conversation. 1975. 2, 3,
[17] Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. Mermaid: Metaphor generation with symbolism and diseriminative decoding. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021. 4
[18] Prateck Saxena and Soma Paul. Epic dataset: A corpus for possible idiomatic expressions. In Text, Speech, and Dialogue 23rd International Conference, TSD 2020, Brno, Czech Republic, September 8-11, 2020, Proceedings, 2020. 2, 4,5, 17
11

---

[19] Tosin P. Adewumi, Roshanak Vadoodi, Aparajita Tripathy, Konstantina Nikolaidou, Foteini Liwicki, and Marcus Liwicki. Potential idiomatic expression (pie)-english: Corpus for classes of idioms. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, 2022. 2, 4
[20] Issa Annamoradnejad and Gohar Zoghi. Colhert: Using bert sentence embedding in parallel neural networks for computational humor, 2022. 2, 4, 17
[21] Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. Interview: Large-scale modeling of media dialog with discourse patterns and knowledge grounding. In Annual Conference on Empirical Methods Natural Language Processing (EMNLP), pages 1129-8141, Online, November 2020. Association for Computational Linguistics. 2,4
[22] Ryan Lowe, Nissan Pow, lulian Serban, and Joelle Pineau. The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-tum dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285-294, Prague, Czech Republic, September 2015. Association for Computational Linguisties. 3
[23] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Annual Conference Empirical Methods in Natural Language Processing (EMNLP), 2017.
[24] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. 3
[25] Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018, 3.9
[26] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston, Personalizing dialogue agents: have dog, do you have pets too? In Annual Meeting the Association for Computational Linguistics (ACL), 2018. 3
[27] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
[28] Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qu: Machine reading comprehension with contextual commonsense reasoning. In Annual Conference on Empirical Methods Natural Language Processing (EMNIP), 2019.
[29] Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. In Transactions of the Association for Computational Linguistics (TACL), 2019. 3
[30] Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. Dream: A challenge dataset and models for dialogue- based reading comprehension. In Transactions of the Association for Computational Linguistics (TACL), 2019.
[31] Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. Dialogue natural language inference. In Annual Meeting of the Association for Computational Linguistics (ACL). 2019. 3
[32] Dheeru Dua, Yizhong Wang. Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HLT)), 2019. 3
[33] Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. Mutual: A dataset for multi-turn dialogue reasoning. In Anmual Meeting of the Association for Computational Linguistics (ACL), 2020. 3, 4
[34] Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. Are natural language inference models imppressive? learning implicature and presupposition. In Annual Meeting of the Association for Computational Linguistics (ACL), 2020. 3
[35] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2020. 3, 19
[36] Baolin Peng. Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu. Bill Dolan, and Jianfeng Gao. Godel: Large-scale pre-training for goal-directed dialog. arXiv preprint arXiv:2206.11 11309, 2022.
12

---

[37] Romal Thoppilan, Daniel De Freitas. Jamie Hall, Noam Shazeer. Apoorv Kulshreshtha, Heng-Tze Tze Cheng. Alicia Jin, Taylor Bos, Leslie Baker, Yu Du. YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng. Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin. Dehao Chen. Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Chine Chang. Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier- Hellstem, Mcredith Ringel Morris, Tulsce Doshi, Renelito Delos Santos. Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak. Ed Chi, and Quoc Le. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.00 082399. 2022.
[38] Daniel Adiwardana, Minh- Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang. Apoory Kulshreshtha. Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards human-lite like open-domain chatbot. arXiv preprint arXiv:2001. 009977, 2020.
[39] Chulaka Gunasckara, Scokhwan Kim, Luis Femaando D'Haro, Abhinav Rastogi, Yun-Nung Chen, Mihail Eric, Behnam Hedayatmia, Karthik Gopalakrishnan, Yang Liu, Chao-Wei Huang, Dilck Hakkani- Tür, Jinchao Li, Qi Zhu, Lingxiao Luo, Lars Liden, Kaili Huang. Shahin Shayandeh, Runze Liang, Baolin Peng. Zheng Zhang, Swadheen Shukla, Minlie Huang, Jianfeng Gao, Shikib Mehri, Yulan Feng, Carla Gordon, Seyed Hossein Alavi, David Traum, Maxine Eskenazi, Ahmad Beirami, Eunjoon, Cho, Paul A. Crook, Ankita De, Alborz Geramifard, Satwik Kottur, Seungwhan Moon, Shivani Poddar, and Rajen Subba. Overview of the ninth dialog system technology challenge: Dstc9, 2020.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 3
[41] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Fric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang, Kathy Meier-Hellsterm. Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder. Yi Tay, Kefan Xiao. Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha. James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette- Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Scbastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Itycheriah,, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li. Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Magginni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham. Eric Ni. Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Sacta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuczhi Wang, Pidong Wang, Zirui Wang, Tao Wang. John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu. Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
[42] Yunxiang Zhang and Xiaojun Wan. MOVER: Mask, over-generate and rank for hyperbole generation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 6018-6030. Seattle, United States, July 2022. Association for Computational Linguistics. 4, 17
[43] Orion Weller and Kevin Seppi. The rJokes dataset: large scale humor collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6136-6141, Marseille, France, May 2020. European Language Resources Association. 4
[44] Kevin Stowe, Prasetya Utama, and Iryna Gurevych. IMPLI: Investigating NLI models' performance on figurative language. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 5375-5388, Dublin, Ireland, May 2022. Association for Computational Linguistics. 4
[45] Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. It's not rocket science Interpreting ligurative language in narratives. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022. 4
[46] Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. Cicero: A dataset for contextualized commonsense inference in dialogues. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022. 4
13

---

[47] Deepanway Ghosal, Pengfei Hong, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. Cider: Commonsense inference for dialogue explanation and reasoning. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2021.
[48] Pei Zhou, Pegah Jandaghi, Bill Yuchen Lin, Justin Cho, Jay Pujara, and Xiang Ren. Prohing commonsense explanation in dialogue response generation. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP). 2021.
[49] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zeullmoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 2019.
5, 17, 18
[50] Yunxiang Zhang and Xiaojun Wan. Mover: Mask, over-generate and rank for hyperbole generation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022. 5
[51] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In Annual Meeting of the Association for Computational Linguistics (ACL), 2020. 5,
7, 18
[52] Marti A. Hearst. Texttiling: Segmenting text into multi- paragraph subtopic passages. Comput. Linguist.. 23(1):33-64, 64, mar 1997. ISSN 0891-2017.
[53] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2019. 7
[54] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-ssequence pre-training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019. 8
[55] Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In Journal of Machine Learning Research (JMLR), 2020. 8,
[56] Daniel Khashabi, Sewon Min. Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. In Annual Conference on Empirical Methods in Natural Lunguage Processing (EMNLP), 2020. 8
[57] Linting Xue, Noah Constant, Adam Roherts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In North American Chupter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 483 498, Online, June 2021. Association for Computational Linguistics. 8
[58] Or Honovich, Roce Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy. Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re-evaluating factual consistency evaluation. In North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (NAACI-HI.T), pages 3905-3920, Seattle, United States, July 2022. Association for Computational Linguistics. 9
[59] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. In Annual Conference on Empirical Methods Natural Language Processing (EMNLP), 2015. 9
[60] Bill MacCartney and Christopher D. Manning. Modeling semantic containment and exclusion natural language inference. In International Conference on Computational Linguistics (COLING), pages 521-528, Manchester, UK, August 2008, Coling 2008 Organizing Committee. 9
[61] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: serodinn--nhaaneee bert with disentangled attention. In International Conference on Learning Representations (ICLR), 2021. 9, 18
[62] Takeshi Kojima, Shixiang Shane Gu. Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
10
[63] Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr. Wai Keen Vong, Robert Hawkins, and Yoav Artzi. Abstract visual reasoning with tangram shapes. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 582-601 601, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. 10
14

---

[64] Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In Proceedings Machine Learning Research, volume 202, pages 10421- 10430, 23-29 Jul 2023.
[65] Timo Schick and Hinrich Schützze It's not just size that matters: Small language models also few-shot learners. In North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies (NAACL-HLI) pages Linguistics. 17
Online, June 2021. Association for Computational
[66] Rishabh Misra. News headlines dataset for sarcasm detection, 2022. 17
[67] Lennart Wachowiak, Dagmar Gromann, and Chao Xu. Drum up SUPPORT: Systematic analysis of image-schematic conceptual metaphors. In Proceedingy of the 3rd Workshop on Figurative Language Processing (FLP), pages 44-53, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. 17
[68] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre- Pre-training of deep bidirec- tional transformers for language understanding. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019. 18
[69] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: lite bert supervised learning of language representations. In International Conference on Learning Representations (ICLR), 2020. 18
[70] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodci, and llya Sutskever. Language models are unsupervised multitask learners. 2019. 18
15

---

Appendices
Contents
A Hypothetic Explanation for Experimental Results of RoBERTabes outperforms RoBERTaarge
17
B Annotation Details
17
B.1 Details For Automatic Selection
17
B.2 Details For Fine-grained Annotation
17
B.3 Details on Human Refinements
18
Experimental Detail
18
C.1 Pragmatic Identification and Reasoning (PIR)
18
C.2 Conversational Question Answering (CQA)
19
C.3 Zero-Shot Natural Language Inference
19
D More Detail on DiPlomat
20
Grice Maxims and Pragmatic Reasoning
24
Computational Resources
24
Limitations Negative Societal Impacts
24
H Ethics Concern
24
I Responsibility & Dataset Liscence
25
J Datasheets for Our Dataset
25
J.1
Motivation
25
J.2 Composition
25
J.3
Collection Process
26
J.4
Preprccesinggccaanigg/abiinng
27
J.5
Uses
28
J.6 Distribution
28
J.7
Maintenance
29
16

---

A Hypothetic Explanation for Experimental Results of RoBERTa outperforms RoBERTajarge
hasse
Conventional wisdom suggests that larger models, with more parameters, tend to outperform smaller models. However, indicated in Table notable phenomenon emerges in the context of the PIR experiment: RoBERTa outperforms RoBERTaange This unexpected observation leads us to formulate two-fold hypothesis:
1. domain discrepancy between pragmatic data and LM's pretraining data: Larger models exhibit consistent influence of their pretraining phase, whereas smaller models are more susceptible to adaptation during the fine-tuning process. Importantly, beyond the application of semantic knowledge, our pragmatic data introduces deeper challenges for language models, leading smaller models to potentially "superficially" memorize patterns from the fine-tuning data. In contrast, larger LMs tend to adhere more closely to their original pretraining understanding. Similar trends have been observed in other domains, such as multi-step math reasoning [64] and few-shot learning [65].
2. Low-resource and diversity of pragmatic data: Pragmatic data is both scarce and diverse. Effectively tackling this data needs language models move beyond mere semantic pattern memorization and engage in more nuanced reasoning over contextual information, social common sense, and even theory-of-mind reasoning. Such low-resource conditions challenge the conventional fine-tuning requirements for large language models, which typically benefit from substantial data for achieving superior performance.
To empirically substantiate these hypotheses, we undertake zero-shot version of Task 1 (PIR). The results of this zero-shot experiment are reported in Tab. 6, and when compared to the results presented Table 3, it becomes evident that the improvement in performance of RoBERTa following fine-tuning is considerably more pronounced than that of RoBERTa
Table 6: Zero-Shot PIR
C-P
CP
+
R
C
PR
RoBERTanrge
61.1
21.8
1.7
RoBERT Taguse
57.2
21.5
5.1
B Annotation Details
B.1 Details For Automatic Selection
Different methodologies are employed to address various pragmatic phenomena. To leverage prior advancements in the field, we begin by segmenting each dialogue into individual utterances. Sub- sequently, employ two distinct approaches, namely string matching and pretrained model clas- sification, to identify these phenomena within our source data. In the case of scalar implicature, which exhibits noticeable pattern characterized by word pairs such as (some, all) appearing in adjacent turns of dialogues, we employ string matching to annotate instances of scalar implicature in conversations. Similarly, for popeq implicature, which often features continuous question mark, we utilize this characteristic as a means of detection. With regards to idioms, which exhibit more evident patterns, we employ the idiom set proposed by Saxena and Paul [18] conduct searches. For other types of phenomena that lack obvious patterns, we leverage a pretrained RoBERTa base model [49], and fine-tune it for our specific task. The sarcasm dataset by Misra [66] is used for finetuning the sarcasm model, the MOVER dataset by Zhang and Wan [42] for hyperbole and the ColBERT dataset by Annamoradnçjad and Zoghi [20] for paronomasia. Several models have been proposed for metaphor detection, thus we utilize an existing model [67] specifically designed for metaphor identification.
B.2 Details For Fine-grained Annotation
AMT' integral to our process. To ensure clarity and consistency, we provide explicit instructions to the workers. Additionally, to further elucidate the objectives of our study, offer illustrative
17

---

examples. The task itself is presented below the instructions and examples, with the dialogue and corresponding turn numbers provided for workers select. Furthermore, as workers check checkbox, we prompt them to select confidence score and provide a rationale. In order to strike balance between our budget, the quality of annotations, and the speed of annotation, we have determined the compensation of $0.1 per completed task. After the annotation process, we collect responses that are assigned with confidence score of 4 or higher.
Specifically, we surveyed 10 users to accomplish our task. All users can complete a single task within 45 seconds, leading to a wage pay of around dollars per hour, which is about a dollar higher than the federal minimum hourly wage of the United States.
B.3 Details on Human Refinements
Disturbing choices chosen based on the BERTScore metric [51]. The rationale with the highest similarity, as determined by other dialogues, is selected and included the pool of candidate options. The instructions provided to the workers align with those used for Fine-grained Annotation, wherein they are also instructed to assign a confidence score to their responses. The remuneration for workers set at $0.05 per task.
AMT Workers Requirements order to guarantee the quality of annotated data, the qualification rules for workers are strict and can be found in Tab. 7.
Table 7: AMT workers requirements
Country
United States, Canada, Great Britain, Australia, Singapore, Ire- land, New Zealand
Tasks approveddooeerhaaaaaarrr
1300
Tasks approved Rafe@eenerrhannttiaaaf
95%
C Experimental Detail
C.1 Pragmatic Identification and Reasoning (PIR)
BERT base [68] BERT (Bidirectional Encoder Representations from Transformers) is revolutionary language representation model that has had a significant impact on natural language processing (NLP) tasks. It has achieved remarkable performance across various NLP benchmarks, including question answering, sentiment analysis, named entity recognition, and many others. Its birth brings profound influence on pretrained language models.
RoBERTa & RoBERTa [49] RoBERTa improves upon BERT by incorporating enhance- ments such as larger and more diverse training data, longer pretraining duration, dynamic masking, and advanced training strategies. These improvements enable RoBERTa to achieve even better performance on a wide range of NLP benchmarks. While BERT paved the way for contextualized rep- resentations in NLP, RoBERTa further refines and pushes the boundaries of language understanding, making it a powerful and preferred choice for many researchers and practitioners in the field.
ALBERT base & ALBERT large [69] ALBERT (A Lite BERT) is highly efficient and compact variant of the BERT model that addresses the computational limitations of the original architecture. It incorporates pardutier-rrdduction techniques to alleviate training time constraints and achieve improved performance compared to BERT.
DeBERTapase [61] DeBERTa (Decoding -enhinne-e BERT with Disentangled Attention) is state- of-the-art language representation model that builds upon the BERT architecture and introduces several key innovations, including disentangled attention mechanism. The performance of DeBERTa has been demonstrated to surpass that of BERT on a wide range of NLP tasks.
Leveraging transformers decoder, Radford et al. [70] proposed GPT2. It represents a significant breakthrough natural language processing and generation. One of the most notable features of GPT-2 is its ability to generate coherent and contextually relevant text. Through unsu- pervised pretraining on a large corpus of internet text, GPT-2 learns to predict the next word in sequence of text, enabling it to generate human-like responses.
GPT2base [70]
18

---

Table 8: Hyperparameters for models on CP -
Model
learning rate
batch size
weight decay
epochs
BERT
5e-5
12
0.001
50
BERT large
5e-5
12
0.001
50
ALBERT base
5c-5
12
0.001
50
ALBERT large
5c-5
0.001
50
DeBER Tabeee
5c-5
12
0.001
50
RoBERTan Thhese
5e-5
12
0.001
50
RoBERTanng Tatone
5e-5
12
0.001
50
GPT2
0.001
0.01
50
DialoGPT mediuum
0.001
2
0.01
50
Table 9: Batch size for models C --> P
Model
Batch Size
BERT base
ALBERT base
24
ALBERTI Turge
24
DeBERT ABBERTaaoo Tabuss
24
RoBERTapo Taboor
RoBERTaung
24
GPT2
24
DialoGPT meddum medium
medium [35] DialogGPT is dialogue-oriented GPT. It builds upon the GPT architecture and extends it to support interactive conversations. DialogGPT is trained in supervised manner using a dialogue dataset, which allows to understand and generate responses in a conversational context.
DialogGPT mediuu
The PIR task encompasses three distinct settings: C P. R, and C PR. In the C setting, models are trained for 20 epochs, employing batch size indicated in Tab. 9, a learning rate 2e - 5, and weight decay of 0.01. As for CP R. the hyperparameters adopted are listed in Tab. 8. For the - PR setting, there is training required; instead, simply load the best checkpoint obtained from the previous training for this task. The concrete implementation as follows: we initially flatten the test dataset of C P, ensuring that each instance contains both dialogue and pragmatic turn extracted from the same dialogue. As for the test dataset CP - R, no modifications made. It should be noted that, following the processing steps, both datasets the same dialogues and corresponding pragmatic turns, resulting in identical instance numbers. For instance be deemed correct, the models must successfully accomplish both component tasks iee. succeed in Identification and Reasoning.
C.2 Conversational Question Answering (CQA)
CQA ChatGPT was instructed to generate questions for our tasks. The prompt template that starts the questions with "Which" is depicted in Tab. 10. Through this methodology, collected total of 19,482 questions. To ensure the reliability of the answers provided to these questions, AMT is utilized. In our experiment, the hyperparameters adopted are illustrated in Tab. 11. To assess the performance of ChatGPT, we conducted testing using the template outlined Tab. 13.
C.3 Zero-Shot Natural Language Inference
Details are provided as follows. T5-XXL, and DeBERTa-v3 are tested with the pragmatic turn as premise and implied meaning hypothesis. The context is out of reach for these models. In contrast, as shown in Tab. 10, ChatGPT is given the context, and the red line labeled "Think step by step" represents two distinct configurations: with trep-by-ster and one without it.
19

---

Table 10: ChatGPT question generation template: using "Which" to start the question.
You are sensitive and always view others' words as having some implied meanings.
For the dialogue between "A" and "B" this task, have offered a statement that is the implied meaning of turn, please only offer one reading comprehension question that can be answered with only one word based the dialogue and mostly focuses on the turn the statement mentions.
The question will be tested by only by viewing the dialogue, so please make the question hard enough that it's impossible to answer without viewing the statement.
Use "Which" to ask the question!
Following is dialogue: (dialogue)
Following is the statement: (statement)
Use "Which" ask the question! And please make the question hard enough that it's impossible to answer without viewing
Table 11: Hyperparameters for models on CQA.
Training Epoch
Learning Rate
5.6e
Batch Size
24
Weight Decay
0.001
Table 12: Test ChatGPT: answer questions with only one word.
For the dialogue between "A" and "B" in this task, please answer question according to the dialogue with only word Following is the dialogue:
(dialogue)
Following is the question: {question}
Table 13: ChatGPT test template of Zero-Shot CoT
This is natural language inference taak. Given the dialogue context: (context) Does (pragmatic turn) entails (implied meaning)? Reply 'entails' 'not entails'.
Think step by step.
D More Detail on DiPlomat
In this section, we will propose more examples of our dataset in Tab. 14, Tab. 15, Tab. 16, Tab. 17, and Tab. 18.
20

---

Table 14: Contextual reasoning examples of DiPlomat
Yeah. They say that he's fastest pitcher there ever was. It's just he really couldn't t find home plate. I mean, some of the stories you learn about this guy, it reads like fiction. When he was think this is around 1960, He's pitching in the minor leagues, and he pitched fast he ripped the man's car off. B: Oh. A: Yeah.
Rationale: The literal
meaning is a simple ex- pression of agreement, while the implied mean- ing that the speaker is amazed by the story Steve Dalkowski's feats.
B: We're talking about 8 million people. Has the rise of temporary workers figured into, at least, the statistical improvement of the U. S. economy for some people?
Rationale: The implied meaning of this turn is reflect our reliance on temporary workers in our day-to-day lives.
A: has. Overall, about one seventh of the total job growth has been in the temp sector. The temp sector is growing nine times faster than the overall private sector as a whole. And the 2. 9 million workers represents a record number, both in the number of temp workers and in the percentage of the economy that they make up.
"Harvest Shame," Edward Murrow very famously said, the people we're re showing you in this documentary have picked your Thanksgiving bounty with their bare hands, and this is how they live.
A: And so I got up and ran. And it wasn't 't too far. But just that moment, I thought, don't want to be shot in the back, and need to find some And there's 's really no place to hide. But there are these B: You found a little, like, alcove that you could duck into. A: There was little alcove, ycah. And I just made myself as small as I could in that little corner.
Rationale: The speaker tried to protect itself from danger.
A: Well, there's big argument in the United States about this. There's group folks who think that engagement policy failed. We engaged with China from 1979 until about 2013 when Xi Jinping came into power. And the idca of engagement was that cocvolution was in the American interest as well as in China's 's interest. And you could bring China along to be responsible player some degree.
Rationale: The situa- tion not necessarily an 'cither/or' between China and the United States.
A: Many hardliners in United States government and outside and including in the expert community now claim that engagement was a sucker's 's game and that we have raised up tiger which could now devour us. But there are different schools of thought about this, and many of think that still need engage with China, albeit more strategically.
That image raising tiger that will devour us is very dramatic.
21

---

Table Figurative language reasoning examples of DiPlomat
A: Thank you. How you?
Rationale: Stuck like glue mcans be tached something.
B: I'm pretty good. Thank you. You must be stuck like glue this, but, you know, you's played in three World Cups, including one of wins the team in 1999. How would you describe what it's like to be out there on that field in that final game?
which is a particular is- sue person.
B: So in terms of what to do about it, we've said Twitter and Facebook have shut down these accounts, which prompts me wonder does shutting down fake account do that much? Can't the Chinese government, it's determined go down this path, just open up two new ones in place the one that was closed? A: It is cat
Rationale: Mice are constantly trying get away from cats and are constantly trying to catch mice. In the same way. the Chinese gov- ernment will always be trying escape restric- tions on social media ac- counts and media com- panies will always be trying to find fake counts.
game, and the companies are constantly trying to get ahead of it. [+.>] J As you said, they can always set up new accounts.
A: I really didn't feel safe because the Turkish government is very famous for hunting down those who oppose Erdogan. So, mean, I. just didn't tt want to really risk my life by going to Europe. But, you know, I talked to my team. told them all, like, how many times I want to come because I want to be with you guys there, and want to get win with you guys. And then, later on, they came back news
Rationale: He is imply- ing that he is still not safe.
said, you know what?I think the best decision is if you don't come. Let's just not risk it for one game.
B: Do you feel safe in New York and elsewhere in the U. S.?
A: I have been getting last two, three days hundreds death threats, but think I feel safe America. But anywhere else the world, wouldn's 't really feel safe.
B: Yeah Table 16: Commonsense reasoning examples of DiPlomat
mayor from Tallahassce.
Rationale: "Purplish" states are not really col- ored. They refer to US states that are neither clearly Republican (red) nor Democrat (blue) their voting.
A: Yes. So this sort a test of whether real progressive candidates can win in these sort of purplish states. [...]
B: He wrote of letters hy hand, didn't he?
Rationale: tons of let- ters implies a very large numher and not to full
A: He wrote tons of letters. het there are a hundred thousand hundreds out theref...]
B: Well, Pluto's official designation is dwarf planct. And have to tell you the people who sent this probe all the way Pluto little angry about that because when they launched decade ago, Pluto was still a planet.
Rationale: A is express- ing sympathy for the people who sent the probe, showing that they understand why they feel so disappointed.
A: (Laughter)
B: It got downgraded in the intervening years.
A: That seems unfair.
22

---

Table 17: External knowledge reasoning examples of DiPlomat
B: Inside of his house, family pictures decorate the walls and the fridge. Les has 15 great grandchildren. He grew up in an orphanage, and he couldn't ' wait to leave to join the military. And so in early 1944, he boarded ship and crossed the Atlantic Ocean go to the frontline.
Sailing across the ocean during wartime was a perilous experience.
Rationale:
A: I loved that sailing on, of course. It was so dramatic. You could all these ships bobhing up and down on the ocean. And destroyers were weaving in and out of them to make sure they uncovered any mines or anything.
A: equivalent to a nuclear bomb?
Rationale:
The energy released in the explo- sion is incredibly pow- erful.
B: Well, it's about- its equivalent- - the energy in that explosion is about 10 times the energy in the first atomic bomb...
B: So in your polling, your research, do you find that it's going come down to maybe couple thousand votes from these unaffiliated voters and on what issues?Or will they vote?
Rationale:
The turn suggesting that the county of Jefferson in Colorado could be key factor in deciding the election, despite the fact that it only one of many counties in the state and there are other swing states in the elec- tion.
A: It is likely at the moment to be very narrow victory. President Bush won in 2004 with five percent. That 100,000 votes. In other words, if it is percent, that would be 20,000 votes, and right now, the polls are moving around in just single percentage points. So it could he that narrow.
B: Now, I have read that Colorado is going to he this year's Florida and Ohio, that this is going to be the state that decides the election.
A: I think it could be, and the interesting thing is that Obama and Palin were both
in Jefferson County couple days ago, indicating that there may be actually even a county that could be looked at to be beyond an entire state.
Table 18:
Others examples of DiPlomat
There's that feeling 1 mean, many of us have parents in the industry. I mean, that's what this region is about, especially around Detroit, and Wayne State's 's in Detroit, the heart of Detroit. So, it's nerve-racking. Everyone is nervous. Everyone docsn't 't know what's 's going to happen next. We're all watching the news very closely. But at the same time, it's interesting, because with my generation, almost seem to, kind of, not be as directly impacted. I mean, our family is, puts stress on us, but the day to day of the university and the day to day at school doesn't I seem to have changed that much.
A:
Rationale: A believes that the auto industry will not vanish despite the current situation
B: I understand you have friends there who are engincering majors. Do they have any sense of what their future looks like, and will be it there in Michigan? A: Everybody is secure their choices and secure in their decision. Everybody thinks that the industry will come around, especially now with the news that GM is getting money from the government. And everybody is more hopeful, and I mean, the auto industry has always been one of the largest industries and staple in America, and to think that that industry just going vanish, nobody is willing to concede that.
B: In the meantime, what more have you learned in your reporting about the death of Carlos Hernandez Vasquez?
Rationale: The death of Carlos Hernandez Vasquez could have been prevented if he had been taken to the hospital.
A: Well, a couple of things. One thing that really stands out is that Carlos Hernandez Vasquez died in Border Patrol station. The previous migrant children who died were taken to the hospital first; Hernandez Vasquez was even though immigration authorities elearly knew that he was sick. He was diagnosed with the flu by a nurse practitioner.
B: So, how do you and the retired general, James Jones, know each other?
Rationale: A has high opinion of James Jones' character and career.
A: My gosh, think- I can't II even remember when first met him. It's been so long ago. I'm sure I met him when he was head of the legislative liaison over the Senate. But I really became acquainted with him when he became brigadier general, and, of course, followed his career. Of course, he served very ably as a commandant in the marine corps and then as the European commander, just been with him from time to time. And I just consider him a very good friend.
23

---

E Grice Maxims and Pragmatic Reasoning
The Gricean maxims have garnered substantial attention as foundational theory within the domain of pragmatics. This theoretical framework comprises four distinct maxims: (1) The Maxim of Quality, (2) The Maxim of Quantity, (3) The Maxim of Relevance, and (4) The Maxim of Manner [16, 15]. In contrast to rigid rules or theorems, the Gricean maxims, which capture the prevalent dynamics of conversations, are susceptible to frequent breaches in the context of human communication. These breaches, stemming from the intricacies of real-world interaction, notably manifest in the violation of one or more of these maxims. Such breaches, aligned with the cooperative principle, give rise to pragmatic phenomena that necessitate the engagement of pragmatic reasoning by recipients of the communication [15].
Computational Resources
For our experiment, we utilized two A100s and one 3090. The majority of our experiments were conducted on the A A100s, while for practical reasons, only Unified-QA-basee MART-base, and T5- small were tested on the 3090. It is important to mention that each experiment was run on a single GPU. We record the training time of models in Appendix F.
Model
C+P
CP
Device
BERT base
0. 0.Sinn/pooch
0.9min/epoch
A100
RoBERTa Taaase
0.8min/epoch
0. 0.9min/epoch
A100
RoBERTan Tajagg
2. 2.5min/epoch
2.8min/epoch
A100
GPT-2
5. 5.8min/epoch
6. 6.2mih/epoch
A100
DialoGPT medium
2. 2.4min/cpoch
4. 4.2mih//poch
A100
DeBERTan TeRTTaaaee "aase
0.9min/epoch
0. 0.9min/epoch
A100
ALBERT, base
0.5min/epoch
0.8min/epoch
A100
Table 19: Training Time of Models
G Limitations & Negative Societal Impacts
We acknowledge two limitations in our study: bias and subjectivity. Since our dialogues primarily stem from an interview dataset, a considerable focus placed on political topics. This reasonable, pragmatic phenomena frequently emerge in the statements of politicians to advance their specific goals. However, this focus introduces a certain degree of bias into our dataset. The second limitation relates to the absence of subjectivity. In our methodology, the data undergoes two stages of human annotation, ensuring higher quality and objectivity. However, pragmatic reasoning is inherently subjective, and prioritizing objectivity compromises the preservation of subjectivity, resulting in limitation in terms of subjectivity coverage. Our dataset exhibits minimal negative societal impacts. This is primarily due the fact that our dialogues are transcriptions of publicly available TV shows, which inherently limits the potential for negative effects.
H Ethics Concern
Were any ethical review processes conducted (e.g., by an institutional review board)? Our dataset does not involve human subjects. The dataset has undergone the institute's internal ethical review process.
Does the dataset contain data that might be considered confidential? No, the data source derives from an existing public interview dataset.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why. Few of the dialogues may talk about offensive topics.
Does the dataset identify subpopulations (e.g., by age or gender)? Not explicitly.

---

Is it possible to identify individuals (i.e., one more natural persons) directly or indirectly (i.e., in combination with other data) from the dataset? Yes, our data contains the names of celebrities.
I Responsibility & Dataset Liscence
We bear all responsibility in case of violation of rights and our dataset is under the license of CC BY-NC-SA Aariburio-NonCCmmeeriil---aaaaa
J Datasheets for Our Dataset
J.1 Motivation
1. For what purpose was the dataset created? (Was there a specific task in mind? Was there specific gap that needed to filled? Please provide a description.) This dataset was created to study pragmatic reasoning in dialogues, a specific gap is men- tioned above in Appendix G.
2. Who created this dataset (e.g., which team, research group) and on behalf of which entity (c.g., company, institution, organization)?
This dataset was created by the authors of this paper.
3. Who funded the creation of the dataset? (If there is an associated grant, please provide the name of the grantor and the grant name and number.)
The institute of the authors funded the creation the dataset.
4.
Any other comments?
None.
J.2 Composition
1. What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? (Are there multiple types of instances (e.g., movies, users, and ratings: people and interactions between them; nodes and edges)? Please provide description.) An instance of dataset represent piece of dialogue. Description is provided in paper.
2. How many instances are there in total (of each type, if approppriate??
Our dataset owns 4,177 dialogues.
3. Does the dataset contain all possible instances or it sample (not necessarily random) of instances from a larger set? (If the dataset is sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If is not representative of the larger set, please describe why not (e.g., to cover more diverse range of instances, because instances were withheld unavailable).)
It is sample of all possible cases. As pragmatic phenomena aren't proved be limited, we can't 't guarantee full sampling of them.
4. What data does each instance consist of?
We mention it in paper.
5. Is there label or target associated with each instance? If so, please provide description.
Yes. The description is in our paper.
6. Is any information missing from individual instances? (If so, please provide a description, explaining why this information is missing (c.g.. because was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.)
No. We leverage the original dialogues.
7. Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? ( If so, please describe how these relationships are made explicit.)
No. Instances weakly related, but focus on the same phenomenon.
25

---

8. Are there recommended data splits (e.g.. training, development/validationn testing)? (If so, please provide a description of these splits, explaining the rationale behind them.)
Yes. We provide it.
9. Are there any errors, sources of noise, redundancies in the dataset? (If so, please provide a description.)
Yes. Some workers try to finish the work as quickly as possible, therefore when we ask them offer a rationale for choosing certain turn a pragmatic turn, they simply type an "a" in the box. However, the situation is rare, and we blocked the workers and clean the data out of our dataset.
10. Is the dataset self-contained, or does it link or otherwise rely external resources (e.g.. websites, tweets, other datasets)? (If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e.. including the external resources as they existed at the created);
time the dataset any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, well as links other access points, as appropriate.) It's self-contained.
there any restrictions (e.g., licenses, fees) associated with
11. Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor- doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? (If so, please provide a description.)
No.
12. Does the dataset contain data that, if viewed directly, might offensive, insulting, threaten- ing, or might otherwise cause anxiety? (If so, please describe why.)
Yes. Some of the topic are big events, they may be offensive for some people. However, we consider our datasct's 's offensiveness to be limited, for the source dataset is a TV show transcript.
13. Does the dataset relate to people? (If not, you may skip the remaining questions in this section.)
Yes.
14. Does the dataset identifyy any subpopulations (e.g., g.. by age, gender)? (If so, please describe how these subpopulations are identified and provide a description of their respective distri- butions within the dataset.)
No. This is not explicitly identified
15. Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? (If so, please describe how.)
Yes; their names are given in running text.
16. Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial ethnic origins, sexual orientations, religious belief's, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of govermment identification, such as social security numbers; criminal history)? (If so, please provide description.)
Yes. Our dataset may have dialogues talking about religious, politics and so on.
17. Any other comments? None.
J.3 Collection Process
1. How was the data associated with each instance acquired? (Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verifiedd If so, please describe how.)
The data all comes from interview dataset already published. (See paper)
26

---

2. What mechanisms procedures were used to collect the data (e.g.. hardware apparatus or sensor, manual human curation, software program, software API)? (How were these mechanisms or procedures validated??
Software program and manual human curation (2 times). See paper for details. 3. If the dataset is a sample from larger set, what was the sampling strategy (c.g., deterministic, probabilistic with specific sampling probabilities)?
Randomly.
4. Who was involved the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g.. how much were crowdworkers paid)? Crowdworkers. They are paid nicely. See Appendix for detail.
5. Over what timeframe was the data collected? (Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.)
The dataset was collected the early Spring of 2023, which does not necessarily reflect the timeframe of the data collected.
6. Were any ethical review processes conducted (e-g-. by an institutional review board)? (If so, please provide a description of these review processes, including the outcomes, as well as link or other access point to any supporting documentation.)
No review processes were conducted with respect to the collection and annotation this data (though review was done for other aspects of this work; see the paper linked at the top of the datashect).
7. Does the dataset relate people? (If not, you may skip the remaining questions in this section.)
Yes.
8. Did you collect the data from the individuals in question directly, obtain it via third parties or other sources (e.g., websites)?
Other sources. By curating a published dataset.
9. Were the individuals in question notified about the data collection? (If so, please describe (or show with screenshots or other information) how notice was provided, and provide link or other access point to, or otherwise reproduce, the exact language of the notification itself.) No.
10. Did the individuals in question consent to the collection and use of their data? (If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link other access point to, otherwise reproduce, the exact language to which the individuals consented.)
No. All data are public.
11. If consent obtained, revoke their consent in the future or for certain uses? (If so, please provide description, as well link or other access point the mechanism (if appropriate).) N/A.
the consenting individuals provided with mechanism to
12. Has analysis of the potential impact of the dataset and its use on data subjects (e.g.. data protection impact analysis) been conducted? (If so, please provide a description of this analysis, including the outcomes, as well as link or other access point to any supporting documentation.)
No. We consider our dataset having limited negative effect, for all of our data has been published for more than year.
13. Any other comments? None.
J.4 Preprocsiingclaanngghbbbiiggggg
Was any preprcesing/teeaing/leeiigg tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, pro- cessing of missing values)? (If so, please provide a description. If not, you may skip the remainder of the questions in this section.)
the data done (e.g., discretization or bucketing,
27

---

No.
J.5
Uses
1. Has the dataset been used for any tasks already? (If so, please provide description.)
Yes. See paper for details.
2. Is there repository that links to any or all papers or systems that use the dataset? (If so, please provide link other access point.)
No.
3. What (other) tasks could the dataset be used for?
Many more. Such as generation of implied meanings.
4. Is there anything about the composition of the dataset or the way was collected and pepposesseddclemeedllaeeedd that might impact future uses? (For example, is there anything that future user might need to know to avoid uses that could result in unfair treatment of individuals groups (c.g., stereotyping, quality of service issues) other undesirable harms (e.g.. financial harms, legal risks) so, please provide a description. there anything future could do to mitigate these undesirable harms?)
No.
5. Are there tasks for which the dataset should not be used? (If so, please provide description.) No.
6. Any other comments? None.
J.6
Distribution
1. Will the dataset be distributed to third parties outside the entity (c.g., company, institution, organization) on behalf of which the dataset was created? (If so, please provide description.)
Yes, the dataset is freely available.
2. How will the dataset will be distributed (c.g., tarball on website, API, GitHub)? (Does the dataset have a digital object identifier (DOI)?)
On our website.
3. When will the dataset be distributed?
It's already been distributed.
4. Will the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms use (ToU)? (If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.)
The dataset is licensed under CC license.
5. Have any third parties imposed IP-based or other restrictions on the data associated with the instances? (If so, please describe these restrictions, and provide link other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.)
Not to our knowledge.
6.
Do any export controls or other regulatory restrictions apply to the dataset or individual instances? (If so, please describe these restrictions, and provide a to, or otherwise reproduce, any supporting documentation.)
link or other access point
Not to our knowledge.
7. Any other comments? None.
28

---

J.7 Maintenance
1. Who is supportinghassinn//ainninnnngggg the dataset?
The authors.
2. How the owner/curatorr/annger of the dataset be contacted (e.g., email address)?
We will post our email address.
3. Is there an erratum? (If so, please provide a link other access point.) Currently, no. As errors are encountered, future versions of the dataset may be released (but will be versioned). They will all be provided in the same location.
4. Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances')? (If so, please describe how often, by whom, and how updates will be communi- cated to users (e.g., mailing list, GitHub)?)
Yes. Howevr, the frequency isn' It determined, and we'll publish the updated dataset on the same website if renewal occurs, and we'll anounce it on the website.
5. If the dataset relates to people, are there applicable limits the retention of the data associated with the instances (e.g. were individuals in question told that their data would be retained for fixed period of time and then deleted)? (If so, please describe these limits and explain how they will be enforced.)
No.
6. Will older versions of the dataset continue to be supportechostedmmaanaancctt (If so, please describe how. If not, please describe how its obsolescence will be communicated to users.)
Yes. The older versions of the dataset will be available on the website.
7. If others want extend/augment/bule them to do so? (If so, please provide description. Will these contributions be validated/ver- ified? If so, please describe how. If not, why not? Is there process for communicating/dis- tributing these contributions to other users? If so, please provide a description.) Yes. They can email us.
on/contribute the dataset, there a mechanism for
8. Any other comments?
None.
29


Reviews:
[{"review_id": "t2PHrPFmpO", "reviewer": "Reviewer_xhK2", "summary": "", "questions": "", "limitations": "There are no negative societal impacts, as for limitations authors identified focus on politics and absence of subjectivity and discussed them. A possible way of addressing these limitations is included in the opportunities for improvement section but in short larger and more diverse dataset is the main thing that could help.", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "The main strength of the submission is sharing with the broader research community a new valuable, open and free dataset that was created by fairly paying annotators at AMT.\nAnother strong suit is drawing the research community's attention to the importance of pragmatic reasoning in tasks that deal with dialogues and successfully demonstrating this gap in experiments.\nSubmission with primarily supplementary material proves beyond the shadow of a doubt the high quality of the research.\nThe ethical and social implications of a paper are positive. The research community will gain new datasets to experiment with and materials build upon.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "FvKXnPYZ7n", "reviewer": "Reviewer_CPTz", "summary": "", "questions": "", "limitations": "The authors adequately address the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Generally well-written paper, good flow.\n- Pragmatic reasoning is an important ability for conversational agents to have, and is not sufficiently addressed by SOTA models.\n- The full dataset and benchmark code is available.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9i5k0KmJdd", "reviewer": "Reviewer_r1EM", "summary": "", "questions": "", "limitations": "I couldn't see anything critical missing.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "I liked the focus on dialogues from real-world interviews.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "MaaHJeWAol", "reviewer": "Reviewer_tasC", "summary": "", "questions": "", "limitations": "Same as above. ", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- **Relevant and Underexplored Problem**: The paper addresses the problem of pragmatic reasoning, which is of significant interest to the NLP community. It sheds light on an underexplored area, highlighting its importance for advancing natural language understanding.\n- **Rich Diversity and Real-life Coverage**: The Diplomat dataset demonstrates its robustness through rigorous filtering, showcasing a wide array of topics and rich diversity sourced from real-life corpora. This comprehensive coverage enhances the dataset's applicability and relevance to real-world scenarios.\n- **Emphasizing Context Understanding**: The experimental results presented in the paper emphasize the critical importance of context understanding. They highlight the limitations of current powerful language models, such as ChatGPT, in this aspect, paving the way for further advancements.\n- **Clarity and readability**: The paper is thoughtfully structured and presented in a manner that makes it mostly easy to follow. Its clear organization and coherence facilitate a smoother understanding of the research and its implications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "XsMZ471rwG", "reviewer": "Reviewer_hCbW", "summary": "", "questions": "", "limitations": "I have specific suggestions and questions.\n\n1. Clarify the dataset name.\n2. Explicitly state how the NLI task proposed in this study contributes to pragmatic reasoning and differs from previous research. \n3. In Tables 3 and 5, the chance-level performance is presented as “Random”. Are the number of cases balanced?\n4. I could not follow the details of the Human Refinement section. Why are their multiple gold choices? Is it because two different workers provided reasons for the same dialogue, and if there was a careless worker providing the reason, the refinement accuracy would be lower?\n5. When referring to the Supplementary material, clearly identify the relevant sections or tables to facilitate easy access and comprehension for readers.\n\n---\n\nThis suggestion would be out of scope for this paper and perhaps more suitable for social science venues. However, it’d be fascinating to see human performance on the PIR and CQA tasks, particularly from non-native English speakers who may encounter challenges due to different contexts and cultural backgrounds.\n", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset, PIR task, and CQA task each possess significant merits.\n\nThe authors' meticulous effort in dataset creation is commendable, as evidenced by the extensive review of prior works and the rigorous curation process. I was able to download and inspect the dataset, and the authors provided the detailed explanations in the Appendix.\n\nThe PIR task has informative subtasks, which were able to highlight the limitations of current models in understanding pragmatic awareness. This task offers valuable insights that can guide future research in the field of pragmatic reasoning.\n\nThe CQA task presents an intriguing challenge for agents, and even for non-native English speakers like myself. It offers promising research opportunities to delve into detailed dialogues and process implied meanings with language models.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents a dataset designed to measure pragmatic reasoning in models, an interesting challenge area for current language models.  Many weaknesses from reviewers were addressed and this should be an interesting addition to the Benchmarks track.
Decision: Accept (Poster)