SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DiPlomat:
A Dialogue Dataset for Situated Pragmatic Reasoning
Hengli  $\mathbf{L}\mathbf{i}^{1,2}$
lihengli@stu.pku.edu.cn
Song-Chun Zhu $^{1,3,4,5}$
s.c.zhu@pku.edu.cn
Zilong Zheng
zlzheng@bigai.ai
$^{1}$  National Key Laboratory of General Artificial Intelligence, BIGAI
$^{2}$  Yuanpei College, PKU  $^{3}$  Institute for Artificial Intelligence, PKU
$^{4}$  School of Intelligence Science and Technology, PKU  $^{5}$  Department of Automation, THU  $^{6}$  Corresponding author.
https://diplomat-dataset.github.io
Figure 1: Illustration of DiPlomat dataset. Left: Example of a pragmatic conversation. Right: Pragmatic Identification and Reasoning task.
If we are negotiating in public, we're not really negotiating at all.
That's correct. I'm out of the Senate three years and a half, but I was there for 36 years.
Much of my work was budgeting. You would have had me on the TV so many times, and you would have said I don't want to see you anymore.
Well, I doubt that, Senator.
Which turn of the conversation has a pragmatic meaning?
Ichoose
a. [A] is being polite and respectful towards [B].
b. [A]'s mind, political figures always tell lies, therefore he questions [B]'s words.
c. [A] doubts whether he would have had the senator on the TV.
d. [A] thinks that he will not only be tired of [B]'s words but also of the [B]'s face.
The ability to discern and comprehend pragmatic meanings is a cornerstone of social and emotional intelligence, referred to as pragmatic reasoning. Despite the strides made in the development of Large Language Models (LLMs), such as ChatGPT, these models grapple with capturing the nuanced and ambiguous facets of language, falling short of the aspiration to build human-like conversational agents. In this work, we introduce a novel benchmark, the DiPlomat, which delves into the fundamental components of conversational pragmatic reasoning, encompassing situational context reasoning, open-world knowledge acquisition, and unified figurative language understanding. We start by collecting a new human-annotated dialogue dataset, composed of 4,177 multi-turn dialogues and a vocabulary of 48,900 words. Along with the dataset, two tasks are proposed to evaluate machines' pragmatic reasoning capabilities, namely, Pragmatic Reasoning and Identification(PIR) and Conversational Question Answering (CQA). Furthermore, we probe into a zero-shot natural language inference task, where the significance of context in pragmatic reasoning is underscored. Experimental findings illustrate the existing limitations of current prevailing LLMs in the realm of pragmatic reasoning, shedding light on the pressing need for further research to facilitate the emergence of emotional intelligence within human-like conversational agents.
The fabric of human sociality is made up of complicated relations that evolve through different dimensions of interaction and communication channels [1, 2]. Social consensuses, such as social norms and values, are thereby formed between humans that convey meanings of individual minds, including beliefs, intentions and desires [3]. In a process of effective negotiation and social conversation, particularly, such social behaviors are only partly driven by literal meanings that are objective, rational and explicit [4, 5]. Instead, these behaviors are commonly governed by affective or pragmatic meanings of dialogue utterances that refer to the emotional and cultural meanings of conversational partners and are subjective, emotional and implicit. For instance in Fig. 1, the lady responds with "I doubt that" rather than "I am not tired of you" to express a sense of humor and politeness. The competency of perceiving such pragmatic meanings is crucial to social and emotional intelligence (EI) [6] and is referred to as pragmatic reasoning.
The rapid developments of large language models (LLMs), such as ChatGPT and InstructGPT [7], have set off a wave of the next generation of conversational AI over the recent years. Despite the inspiring capabilities of language generation [8] and reasoning [9] achieved with massive computational resources and tremendous natural language data, LLMs barely show convincing communicative skills [10], i.e., they fail to capture pragmatic and ambiguous meanings of input prompts [11, 10, 12]. Critically, current neural generative models are trained to be objective with safe and satisfiable responses [13, 14], which largely deviates from the long-standing goal of building a human-like agent. Recently, Meta Research Team et al. [5] introduce a ChatBot that demonstrates human-level play in a language board game Diplomacy where lying and misleading commonly occur. However, their main focus is on game policy learning rather than pragmatic reasoning.
What are the core components of real-life conversational pragmatic reasoning? Motivated by theories of cognitive linguistics [1-3] and conversational modeling [4, 5, 15, 16], we anticipate it to be three-fold:
- Situational Context Reasoning. Understanding pragmatic meaning requires a detailed understanding of conversational contexts. Consider the utterance "You are making the rest of us looking bad", under different situations of praise and sarcasm, the sentence may convey completely opposite meanings. Furthermore, typical conversational reasoning challenges such as coreference resolution and intention prediction are largely dependent on the success of situated context modeling.
- Open-world Knowledge Acquisition. The open-world knowledge includes commonsense knowledge (e.g., social ethics) that can be learned from different domains of dialogue corpus and domain-specific knowledge (e.g., American histories). Successful pragmatic reasoning requires the acquisition of open-world knowledge and joint reasoning over the conversation.
- Unified figurative language understanding. Figurative language is one of the most frequently used tricks for conveying implicit meanings with subjective emotions. Previous works treat different forms of figurative language understanding as individual tasks, such as metaphors [17], idioms [18, 19], pun [20], etc. Pragmatic reasoning provides a feasible unified perspective that considers all these tasks as recovering their literal meanings.
In order to step towards a general human-like communicative agent, in this work, we introduce DiPlomat, a real-life Dialogue dataset that focuses on Pragmatic reasoning. DiPlomat stems from an interview dataset [21], and experiences three steps of curation: automatic selection, fine-grained manual annotation and human refinement (Sec. 3). Our dataset comprises 4,177 dialogues and covers a vocabulary of 48,900 words. More than that, human-annotated answers reach the amount of 6,494 and hold a vocabulary size of 20,000. Along with the dataset, we propose two tasks, Pragmatic Identification and Reasoning (PIR) and Conversational Question Answering (CQA), to benchmark machines' pragmatic reasoning capabilities (Sec. 4). The CQA task possesses 19,482 questions concerning the content of collected dialogues and the answers to the questions are written by humans. We run extensive experiments on previous state-of-the-art models on DiPlomat (Sec. 5). The best model achieves less than 0.70 accuracy score in PIR, and none of the models achieve more than 0.50 accuracy score for CQA. Moreover, we test previous pre-trained LLMs' (including ChatGPT) zero-shot reasoning capability with a natural language inference (NLI) task. Regarding the experimental results provided, the significance of pragmatic reasoning speaks for itself. Throughout a thorough analysis of the limitations of current models, we aim to shed light on future research toward building general conversational agents.
Table 1: Comparisons on similar datasets and our dataset. QA: Question Answering. NUP: Next Utterance Prediction. NLI: Natural Language Inference. PI: Plausible Inference. IR: Implicature Recovery. PIR: Pragmatic Identification and Reasoning. Manually: the dataset is being checked or collected by humans. Diverse: the dataset lies in several specific domains. Open: the dataset doesn't fall into particular domains.
Dataset	Domain	Manually	Task	Implicature	Reasoning	Multi-Turn
Ubuntu [ACL 2015] [22]	Technique	X	NUP	X	X	✓
RACE [EMNLP 2017] [23]	Open	X	QA	X	✓	X
ARC [ArXiv 2018] [24]	Science	X	QA	X	✓	X
MNLI [NAACL 2018] [25]	Open	✓	NLI	X	✓	X
Persona-Chat [ACL 2018] [26]	Persona	✓	NUP	X	X	✓
SWAG [EMNLP 2018] [27]	Movie	X	PI	X	✓	X
Cosmos QA [EMNLP 2019] [28]	Persona	✓	QA	X	✓	X
CoQA [NAACL 2019] [29]	Diverse	✓	QA	X	✓	✓
DREAM [ACL 2019] [30]	Open	✓	QA	X	✓	✓
Dialogue NLI [ACL 2019] [31]	Persona	X	NLI	X	X	✓
DROP [ACL 2019] [32]	Open	X	QA	X	✓	X
MuTual [ACL 2020] [33]	Open	✓	NUP	X	✓	✓
IMPPRES [ACL 2020] [34]	Open	X	NLI	✓	✓	X
GRICE [ACL 2021] [12]	Daily	X	IR & QA	✓	✓	✓
DiPlomat	Open	✓	PIR & QA	✓	✓	✓
Conversational Dataset Tab. 1 provides a comparative analysis of our dataset with similar conversational datasets. The Dream [30] dataset formalizes dialogues from English exams into question-answering task with a focus on in-depth dialogue comprehension. With question-answer pairs compiled by two annotators, CoQA [29] focuses on reasoning in conversation understanding. By utilizing English listening comprehension tests, MuTual [33] is built to address the issue of general dialogue reasoning. In contrast to these preceding datasets, the GRICE [12] dataset represents a significant advancement in the field of pragmatic reasoning as it incorporates implicature and reasoning. However, both MuTual and GRICE exhibit a shared limitation in that they do not possess data that closely resembles real-world interactions, leading to a lack of diversity in their respective datasets. Additionally, other relevant datasets, including commonsense [28], reasoning [23], and natural language inference (NLI) [31, 25], are also included in Tab. 1. By examining various dimensions such as domain, manual annotation, task variety, implicature, reasoning, and multi-turn interactions, our dataset offers unique advantages in addressing the challenge of pragmatic reasoning in dialogues.
Language Models for Dialogue Generation Conversational AI has emerged as a prominent research area within the field of natural language processing (NLP), attracting significant attention and interest. Numerous pre-trained models have been proposed to tackle dialogue generation tasks, such as DialogGPT [35], GODEL [36], LaMDA [37], and Meena [38], and they have achieved marvelous results on competitions [39]. Furthermore, a pivotal milestone has been achieved with the advent of ChatGPT, garnering widespread interest and stimulating further investigation in the domain of conversational AI. ChatGPT, built upon the principles of transformer models [40], undergoes training on an extensive corpus of data, resulting in its profound efficacy. Notably, this system boasts an impressive magnitude of billions of parameters. In a notable study conducted by OpenAI [13], it has been demonstrated that the augmentation of parameters, referred to as the Scaling Law, substantially enhances the model's capabilities. Also, the enlargement of the number of parameters triggers the emergence of miraculous ability. After the success of ChatGPT, more models such as PaLM 2 [41] appeared in the field.
Pragmatic Reasoning Pragmatic reasoning is a significant subject within the field of pragmatics, attracting considerable attention from linguists. The Gricean maxims, which is one of the most important achievements, serves as a foundational theory within the domain of pragmatics. This theoretical framework comprises four distinct maxims: (1) The Maxim of Quality, (2) The Maxim of Quantity, (3) The Maxim of Relevance, and (4) The Maxim of Manner [15, 16]. In contrast to rigid
Figure 2: DiPlomat dataset samples. Each row illustrates an exemplar case with its reasoning type, dialogue context, pragmatic turn, and the corresponding rationale. Evidence that support the pragmatic identification are marked in orange.
rules or theorems, the Gricean maxims, which capture the prevalent dynamics of conversations, are susceptible to frequent breaches in the context of human communication. These breaches, stemming from the intricacies of real-world interaction, notably manifest in the violation of one or more of these maxims. Such breaches, aligned with the cooperative principle, give rise to pragmatic phenomena that necessitate the engagement of pragmatic reasoning by recipients of the communication [15]. In the field of natural language processing, previous work tries to model the problem, but predominantly focus on specific types of phenomena. For example, EPIE [18], PIE [19] center around idiomatic expressions, while MOVER [42] emphasizes hyperbole, and MERMAID [17] investigates metaphor usage. However, patronomasia is much more under-studied [20], with researchers frequently intertwining it with humor [43]. In a related vein, GRICE [12] endeavors to study implicature in a unified manner, but its data does not originate from real-world contexts, thereby lacking diversity and exhibiting conspicuous patterns. Insights into the comprehension of figurative language are provided by Stowe et al. [44] and Chakrabarty et al. [45], with the former specifically delving into the realm of metaphors and idioms, and the latter investigating idioms and similes. The successful completion of our task needs the incorporation of commonsense knowledge. Extensive scholarly efforts have been dedicated to addressing the challenge of leveraging commonsense in various works [33, 46-48].
3 The DiPlomat Dataset
3.1 Data Source Construction
The DiPlomat dataset stems from the Interview dataset [21], which consists of two subsets, a two-party subset comprising 23,714 dialogues and a multi-party subset with 105,848 dialogues. Given our specific focus on conversations involving only two communicators, we exclusively utilize the two-party subset for our research purposes. The Interview dataset itself is a real-world collection of NPR radio transcripts, spanning a period of 20 years of NPR programs. The curation process for our dataset involved several stages, including automatic selection, fine-grained annotation, and human refinement. Details are provided as follows.
Step I. Automatic Selection. The extensive size of the source dataset introduces redundancy, and thus requires automatic measures to alleviate the burden of human annotation. Therefore, we employ algorithms and models to perform an initial filtering process. In order to establish a unified framework,
we consider various types of pragmatic phenomena and utilize different techniques to extract relevant instances from the source dataset. For instance, we utilize the EPIE list [18] for a string-matching method to identify idioms in dialogues, and we train RoBERTa [49] on Hypo-XL [50] for hyperbole detection; refer to Appendix B.
Topic Segmentation Topic segmentation is a small operation taken after automatic selection. The original dialogues employed in our study consist of lengthy and multi-turn exchanges, which are ill-suited for our research objectives. Consequently, we implement a segmentation process to break down these dialogues into shorter units. To achieve this, we employ two techniques, namely BERTScore [51] and TextTiling [52]. The segmentation procedure starts with computing the BERTScore between adjacent turns and subsequently applying the TextTiling algorithm to the generated BERTScores.
Step II. Fine-grained Annotation. We leverage Amazon Mechanical Turk (AMT) to conduct detailed annotation of pragmatic turns within our dialogues. Workers participating in the annotation task are instructed to select all turns that exhibit a divergence between their literal meaning and their intended meaning. Due to the subjective nature of pragmatic reasoning, we request the workers to provide confidence scores along with reasons for their choices and for each dialogue two workers are recruited. All annotators shall meet the following criteria: (i) from English-speaking countries; (ii) Completion of a minimum of 1,300 tasks with a  $98\%$  approval rate. We also present them with detailed instructions and four examples so that workers are clear about our objectives and requirements. The instructions part outlines the step-by-step procedures for accomplishing the assigned tasks and highlights some key points that workers should pay attention to. The four examples offered are representative of classical pragmatic conversations drawn from the field of linguistics, serving as practical references for the workers. To mitigate the intricacies arising from the identities of dialogue communicators, a simplified representation is adopted, whereby the speakers are denoted as PersonA and PersonB. To ensure the reasonableness and quality of the data, we manually examined 30 data samples and blocked workers who are unqualified. As a result, a total of 5,869 dialogues are selected; refer to Appendix B for details.
Step III. Human Refinement. In this process, tasks for workers are formulated as multiple-choice questions. Previously collected human-annotated reasons are transformed into choices, utilizing a template format: [turn {turn_id} : {reason}]. In addition, to mitigate the impact of careless workers, we introduce a distractor choice for each gold choice. These distractor choices are generated using BERTScore [51] by selecting the reason with the highest score from
Table 2: Statistical feature of DiPlomat.
# Dialogues	4.17 × 103
Avg. Turns per Dialogue	4.10
Avg. Words per Turn	42.80
Avg. Human Reason per Dialogue	1.56
Avg. Words per Human Annotated Reason	25.31
Vocabulary Size (dialogue)	4.89 × 104
Vocabulary Size (human-annotated reasons)	2.00 × 104
other unrelated dialogues. Of note, for each dialogue, an equal number of gold choices and distractor choices are provided. Workers are requested to select all reasonable choices for each conversation and are warned of the presence of distractor choices. Workers who frequently select distractor choices are blocked, and their work is rejected. Through this refinement process, 1,692 dialogues are filtered out, while 4,177 dialogues are preserved, ensuring the integrity and reliability of our dataset; refer to Appendix B for more details.
The DiPlomat dataset comprises a total of 4,177 multi-turn dialogues, with each dialogue averaging 4.1 turns. On average, there are 1.56 pragmatic turns per dialogue. The distribution of dialogues with different quantities of pragmatic turns is illustrated in Fig. 4; see Tab. 2 for detailed dataset statistics.
With respect to the motivation introduced in Sec. 1, we categorize the process of transitioning from dialogue to human-annotated rationales into five reasoning types:
- Contextual Reasoning: The comprehension of the context is paramount for this reasoning process.
- Figurative Language Reasoning: Proficiency in understanding figurative language, such as idioms and metaphors, is indispensable for advancing this type of reasoning.
- Commonsense Reasoning: The utilization of commonsense knowledge, such as recognizing that a chateau cannot fall from the sky, is vital for this category.
(a) Pragmatic Turns
(b) Questions
Figure 4: Distribution of dialogues w.r.t. # of pragmatic turns.
Figure 3: Distribution of trigram words of pragmatic turns and questions.
- External Knowledge Reasoning: This form of reasoning necessitates knowledge that extends beyond commonsense and is not explicitly mentioned in the dialogue.
- Others: This category includes pragmatic dialogues that fit none of the above.
Fig. 2 demonstrates the proportion of each type. The prevalence of data within the context partition proves the significance of context in pragmatic reasoning of real life. Fig. 3a depicts a sunburst visualization illustrating the distribution of trigram words within pragmatic turns. The diverse range of trigram words indicates that the DiPlomat dataset enjoys the rich diversity from real-life corpora, and covers a wide array of topics. In addition, the recurring occurrence of the words “president” and “world” is observed, demonstrating DiPlomat's slight bias to politics and world-wide events.
4 Task Definition
We propose 2 distinct tasks for our dataset: (i) Pragmatics Identification and Reasoning and (ii) Conversational Question Answering. The former task focuses on assessing the capability of models to identify the presence of pragmatic phenomena and their ability to select a suitable answer for such identification. The latter task aims to evaluate the models' adeptness in employing pragmatic reasoning by presenting them with carefully designed questions.
4.1 Task 1: Pragmatics Identification and Reasoning (PIR)
In this task, models are provided with dialogues and are required to identify turns whose actual meanings deviate from their literal interpretations, commonly referred to as pragmatic turns. If their selections are accurate, a set of rationales is presented and they are expected to choose the most plausible reason for each pragmatic turn. For each turn, there are 5 candidate reasons available, comprising one gold choice and four disturbing choices. The model's success in this task depends on the precise execution of both steps. We consider three diagnostic settings to test the machine's capability on pragmatic understanding:
- Conversation  $\rightarrow$  Pragmatic Turn ( $\mathbf{C} \rightarrow \mathbf{P}$ ). For each instance, models are presented with a dialogue and a specific turn extracted from that dialogue. They are then required to determine whether the given turn qualifies as a pragmatic turn. Consequently, the dataset is flattened to a total of 17,129 instances, with each instance corresponding to a single queried turn. It's important to highlight that turns without pragmatic meanings are also extracted for evaluation.
- Conversation + Pragmatic Turn → Rationale (CP → R). In this subtask, we offer the model both the dialogue and the pragmatic turn and it needs to choose the most plausible rationale out of five candidate choices.
- Conversation  $\rightarrow$  Pragmatic Turn + Rationale ( $\mathbf{C} \rightarrow \mathbf{PR}$ ) In this subtask, models pre-trained on the previous two subtasks are combined to infer the final results. Specifically, the model obtained from the first subtask is utilized for determining pragmatic turns, while its version finetuned on the second subtask is employed for selecting the most suitable rationale. It is worth noting that, in contrast to  $\mathbf{C} \rightarrow \mathbf{P}$ , in this subtask, extracted turns are limited to pragmatic turns only.
Pragmatic Turns and Gold Choices Recall that in our data collecting procedure outlined in Sec. 3.1, besides asking workers to select pragmatic turns, they are also instructed to fill in reasons to explain their choices. To simplify the evaluation process, the selected turns are directly utilized as answers for the first subtask, while the reasons provided by the workers serve as the designated correct choices (referred to as "gold choices") for the second subtask.
Distractor Choice As a result of the time-consuming nature of BERTScore [51], an alternative approach is adopted for measuring sentence similarity using Sentence-Transformers [53], which is a significantly faster method. In our methodology, for each gold choice, four alternative choices with high similarity scores are selected from the pool of gold choices to serve as distracting answers. Despite their high similarity scores, upon careful examination within the given context, it's apparent that the distracting answers convey entirely different meanings from the gold answer. This characteristic makes them appropriate components to build our task.
4.2 Task 2: Conversational Question Answering (CQA)