Title: DialogBench: Evaluating LLMs as Human-like Dialogue Systems
Benchmark: DialogBench
Summary: DialogBench is a benchmark designed to evaluate LLMs as human-like dialogue systems. It focuses on their ability to understand context, use relevant knowledge, detect emotions and personality, as well as generate coherent, friendly, and contextually appropriate responses. Benchmark includes 12 dialogue tasks generated using GPT-4, with evaluations conducted on 26 LLMs. This paper reveals that while instruction tuning does improve human likeness to some extent, there are also significant gaps in emotional perception and understanding of daily life.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: User Interaction, reasoning, natural language understanding
Phenomenon defined by authors: Yes
Phenomenon definition: Human-likeness covers correctly understanding the dialogue context, making reasonable use of relevant knowledge, detecting the user’s emotions and personality when necessary, and generating friendly, coherent, and consistent responses.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task requires LLMs to answer multi-choice questions based on a given multi-turn dialogue context and a test question relevant to a specific dialogue task.
Task item definition: A single item in the dataset consists of a multi-turn dialogue, potentially external information (like knowledge or personality), a test question, candidate options for the answer, and the correct label. The format is typically JSON.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: On average around 800 instances per task and there are 12 tasks
Dataset metadata available: Yes
Dataset metadata detail: Task, Abbreviation, Average Dialogue Turns, Number of Instances, Domain, Speaker Personalities, Speaker Emotions (for Emotion Detection), Relation (for Relation Classification and Dialogue NLI), Offensive (for Offensive Detection), Persona (for Personality-grounded Response Generation), Knowledge (for Knowledge-grounded Response Generation).
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean/sum
Metric subscores: Yes
Metric subscores detail: Subscores include accuracy on coherence, consistency, correctness, and safety tasks
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Authors present results showing that removing bias mitigation and data filtering steps leads to a drop in accuracy for GPT-4, which they interpret as validation of the effectiveness of these components in creating a more robust benchmark. They also compare LLM performance to a human baseline.
Taxonomy: User Interaction / 
New taxonomy: General Capability / User Interaction