Title: FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Benchmark: FANToM
Summary: FANTOM is a comprehensive benchmark with 10,000 questions designed to evaluate theory-of-mind reasoning capabilities in LLMs by tracking beliefs across multi-party conversations with information asymmetry. The benchmark reveals that even leading LLMs struggle significantly with belief tracking compared to humans, demonstrating only "illusory" theory-of-mind abilities that break down when tested systematically.
Contribution: (1) First conversation-based theory-of-mind benchmark, (2) explicit test for consistency checks across diverse question formats, (3) 256 multiparty conversations around a certain topic 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Theory of mind, conversational question-answering
Phenomenon defined by authors: Yes
Phenomenon definition: The goal of FANTOM is to effectively measure how well models can track the belief of multiple characters in conversations where some information may be inaccessible to some participants. Our aim is to design questions at different levels that evaluate a model’s capability for a coherent understanding of others’ mental states. In doing so, we are particularly interested in identifying instances of illusory ToM, which we define as
situations where a model may answer some questions correctly but fails to answer others that require the same type of ToM reasoning.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Models read a multiparty conversation (short or full) and answer six types of questions (free‑form, multiple‑choice, list, yes/no) about participants’ beliefs or answerability.

Model responses are expected to be "yes", "knows", "does know", or similar type of answers.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Conversations autogenerated with davinci‑003, 
inaccessible info & QA generated with GPT‑4, 
all sets manually validated by 32 MTurk workers
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The reframed dataset contains questions asking how the person beliefs about certain information when the character randomly joins/leaves the conversation to create natural information gaps.
Dataset size: f 256 conversations with 1,415 (Theory of mind) belief questions, 703 fact-checking questions, and 2689 answerability questions.
Dataset metadata available: No
Dataset metadata detail: belief questions (theory of mind; how the characters beliefs of something), fact-checking questions, answerability questions
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Response format detail: Open text, binary or multiple choice types
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: BELIEFQ[Dist.] judged by sentence‑BERT cosine vs. two references and token‑F1; others via strict match
Metric aggregation: Simple Mean
Metric statistics: Means, comparisons with percentage point gaps.
Metric subscores: Yes
Metric subscores detail: Sub‑scores are broken out by each question family — BELIEF (choice, distance, token‑F1), ANSWERABILITY (list accuracy, Y/N F1, “All”), plus FACT token‑F1. Additional splits cover first‑ vs. second‑order (cyclic/acyclic) beliefs and short/full conversation contexts.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: conversation grounding (reporting bias), , adversarial false answers (wrong options are overlapped with the words in context to rule our surface matching), multi-format question sets (illusory success test by consistency checks across different formats), Manual MTurk validation (conversation-answer coherence and correctness)
Taxonomy: Theory of Mind / 
New taxonomy: General Capability / Theory of Mind
Procedural notes: https://hyunw.kim/fantom/