Title: MMToM-QA: Multimodal Theory of Mind Question Answering
Benchmark: MMToM‑QA
Summary: The paper introduces MMToM-QA, the first benchmark that evaluates Theory-of-Mind (ToM) reasoning across multimodal inputs (video and text), containing diverse test questions and synthetic training videos. The authors propose BIP-ALM, a novel approach that combines Bayesian inverse planning with language models to extract unified representations from multimodal data, demonstrating that while current large language and multimodal models lack robust ToM capacity, their method narrows the gap to human-level performance.
Contribution: New multimodal Theory-of-Mind benchmark (text + video)
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Theory-of-Mind, (social) reasoning, multi-modal
Phenomenon defined by authors: Yes
Phenomenon definition: While the recent ToM benchmarks provide welldesigned, cognitively informed tools, they share several notable limitations. One such limitation is the dependence on massive training data, which raises the concern that these models work by finding data patterns in a way that deviates from humanlike ToM reasoning. But ToM reasoning goes beyond merely text comprehension or video understanding. Hence the benchmark targets to evaluate how a model can infer mental states from either words or vision separately or fuse the separate information to form a single coherent mental scene.
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: For each item, the model receives  a short household video clip and a textual description of the scene and actions, then asked to decide which of two candidate mental‑state hypotheses is more plausible.
Task item definition: One clip (RGB‑D frames) + accompanying textual scene/action description + 1 question with two options.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Dataset items are entirely synthetic: avatar videos rendered in simulator; scene & action narratives drafted via templates and polished by GPT‑4. 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: High controllability, annotation (but lacks real‑world sensor noise), human test set for generalisation.
Dataset size: 1000 videos
Dataset size detail: Test (600 Questions over 134 videos), Validation (not separately mentioned)
Dataset metadata available: Yes
Dataset metadata detail: Question type, timestamps, ground‑truth goals/beliefs, scene graphs, RGB‑D, segmentation, 3‑D poses, camera parameters
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: binary options (overall chance = 50%)
Metric aggregation: Simple Mean
Metric statistics: simple mean, statistical tests
Metric subscores: Yes
Metric subscores detail: True/False belief, short/long-term, Goal inference given true/false/updated beliefs or given future actions
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Qualitative analysis and ablations to show why their method (BIP-ALM) succeeds where LLMs fail.
Taxonomy: Theory of Mind / 
New taxonomy: General Capability / Theory of Mind