Title: Understanding Social Reasoning in Language Models with Language Models
Benchmark: BigToM
Summary: The paper introduces BigToM, a comprehensive benchmark containing 5,000 Theory-of-Mind scenarios created through procedural generation using GPT-4-populated causal templates and validated by human raters. This benchmark addresses key limitations in existing ToM evaluations of large language models, specifically the inconsistent results across previous studies and methodological concerns about evaluation validity. This benchmark aims to provide a more rigorous framework for assessing how well AI systems can understand human mental states.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Theory-of-mind, (social) reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: Theory of Mind (ToM) is defined as the ability to attribute latent mental states (beliefs, desires, knowledge, emotions) to agents and use them to explain or predict behavior.  By representing ToM scenarios as causal graphs, we can systematically intervene on variables, generate control conditions, and probe different aspects of an LLM’s ToM capabilities.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Multiple‑choice comprehension questions where a model is expected to infer an agent’s belief or action given a short story generated from a causal template.
Task item definition: Story, Question, 2 answer options, Correct answer, Condition category.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Synthetic narratives rather than real‑world usage, designed for controlled probing.
Dataset size: 5000 items
Dataset metadata available: Yes
Dataset metadata detail: Condition category (e.g., True/False Belief in terms of theory-of-mind)
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean with 95% Confidence Interval 
Metric subscores: Yes
Metric subscores detail: Condition category
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: expert review + participant rating studies, coherence test
Taxonomy: Theory of Mind / 
New taxonomy: General Capability / Theory of Mind