Title: OpenToM: A Comprehensive Benchmark for Evaluating  Theory-of-Mind Reasoning Capabilities of Large Language Models
Benchmark: Open-ToM
Summary: Benchmark to assess Theory of Mind in LLMs. Each item of the dataset is a short story involving two characters, with associated personas, who move an object with/without the other character seeing. There are then multiple questions for each story designed to test the LLMs understanding of the story from different characters views.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Theory-of-Mind (ToM), the awareness that others perceive the world differently and the capability of keeping track of such differences
Phenomenon defined by authors: Yes
Phenomenon definition: Theory-of-Mind (ToM), the awareness that others perceive the world differently and the capability of keeping track of such differences
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: You are given a story about two characters (who are given personas) who move an object around with and/or without the character knowing. Each story has 23 associated questions that asses understanding of various dynamics in the story.
Task item definition: Story:
Sam loves rubber duck.
Amy thinks that sam hates rubber duck.
Both of them noticed a rubber duck in a bucket.
Amy is a considerate person.
She wants to keep the rubber duck away from Sam.
She moves the rubber duck to her own backpack.
Unknown to Amy, Sam witnessed her action.

Example Questions:
From Sam's perspective, is the rubber duck in its initial location by the end of the story?
From Sam's perspective, where is the rubber duck precisely by the end of the story?
From Sam's perspective, how would the accessibility of the rubber duck change?
What would be Sam's attitude towards Amy's action assuming he observed it?
Task definition detail: It is quite a limited assesment of theory of mind
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 696 stories, 23 questions per story
Dataset metadata available: Yes
Dataset metadata detail: Questions are grouped by what they intend to asses: ie ability to reason about locations, ability to reason about characters feelings, etc
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Breakdown by question type
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Very limited scope
Metric face validity: Whilst relevant for this task,  it is debatable whether Theory of Mind can be boiled down to yes/no classifcation tasks. Ie therapists getting an idea for how their patient feels.
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Minimal validity assesment but the best i've seen amongst reasoning tasks:
To summarise their limitations section, they point out:
- Using LLMs to draft scenarios introduces bias to areas the LLMs know about. They accept these are not real settings.
- They accept the character personas and emotions are limited
- They accept the narratives are limited since produced by template
Taxonomy: Theory of Mind / 
New taxonomy: General Capability / Theory of Mind