Title: ToMBench: Benchmarking Theory of Mind in Large Language Models
Benchmark: ToMBench
Summary: ToMBENCH is a bilingual benchmark that evaluates language models' Theory of Mind (ToM) capabilities through classic psychology tasks measuring distinct social cognition abilities in a multiple-choice format. Built entirely from scratch to avoid data contamination, the benchmark reveals that even advanced models like GPT-4 still significantly underperform humans in understanding and attributing mental states, particularly when subjected to coherence stress tests. Their aim with ToMBENCH is to enable an efficient and effective evaluation of LLMs’ ToM capabilities, thereby facilitating the development of LLMs with inherent social intelligence.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: reasoning, Theory-of-mind
Phenomenon defined by authors: Yes
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Bilingual to avoid training-data leakage and enable the unbiased evaluation.
Task definition: Given a short social story, a question, and 4 answer options, the model is expected to choose a single correct option.
Task item definition: story, question, option A to D, correct answer
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task source detail: scenarios from social media posts
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 2,860 questions (934 stories) per English/Chinese
Dataset metadata available: Yes
Dataset metadata detail: language, task type, ability dimension, story length, annotator agreement
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean, Weighted Mean
Metric statistics: simple mean with percentage point
Metric subscores: Yes
Metric subscores detail: 8 classic psychology tasks, 31 distinct social cognition abilities
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: contamination checks, coherence stress test
Taxonomy: Theory of Mind / 
New taxonomy: General Capability / Theory of Mind