Title: DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning
Benchmark: DiPlomat
Summary: This paper introduces Diplomat, a new dataset/benchmark for conversational pragmatic reasoning in LLMs. It has 4177 multi-turn dialogues annotated by humans. The authors propose two tasks - Pragmatic Identification and Reasoning (PIR) and Conversational Question Answering (CQA), to evaluate models' capabilities in understanding "nuanced and ambiguous language in context".
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Conversational pragmatic reasoning.
Phenomenon defined by authors: Yes
Phenomenon definition: "The ability to discern and comprehend pragmatic meanings is a cornerstone of social and emotional intelligence, referred to as pragmatic reasoning."  It involves understanding affective or pragmatic meanings of dialogue utterances that are subjective, emotional, and implicit, rather than just literal meanings.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The benchmark contains two tasks: 
1. Pragmatic Identification and Reasoning (PIR) (models identify pragmatic turns and their rationales),
2. Conversational Question Answering (CQA) (models answer questions based on dialogue context).
Task item definition: A single item consists of a dialogue excerpt and a question or prompt requiring the model to identify pragmatic meaning or provide an answer based on context.
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 2,060 (for PIR) and 2,338 (for CQA)
Dataset size notes: Training: 13,708 (for PIR), 15,585 (for CQA)  Validation: 1,361 (for PIR), 1,559 (for CQA)
Dataset metadata available: Yes
Dataset metadata detail: Reasoning Type (Contextual, Figurative Language, Commonsense, External Knowledge, Others)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: Simple mean/sum
Metric subscores: Yes
Metric subscores detail: Scores are provided for different reasoning types (Contextual, Figurative Language, Commonsense, External Knowledge, Others) for the PIR task.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Authors discuss the limitations of current models based on their performance on the proposed tasks. They highlight the gap between model and human capabilities in pragmatic reasoning. They also analyse performance across different reasoning types and observe a nearly uniform performance, suggesting pragmatic reasoning is a cohesive task.
Taxonomy: User Interaction / 
New taxonomy: General Capability / User Interaction