Title: WhodunitBench: Evaluating Large Multimodal
Agents via Murder Mystery Games
Benchmark: WhodunitBench
Summary: The paper evaluates LLMs ability to participate in (and answers questions about) murder mystery games. In the arena component (agents play as either detective or murderer in a multi-agent setting), the agents are tested on win rate against the other models. The QA component is split based on capability categories (Perception, Role-Play, Decision-making and Cognition)
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: The authors evaluate four distinct capabilities: multi-modal perception, interaction, reasoning and goal achievement.
Phenomenon defined by authors: Yes
Phenomenon definition: • Multi-modal Perception is the most basic ability for LMAs, which requires LMAs to perceive
information from the multimodal environment (e.g., vision and language).
• Interaction requires LMAs, whether through role-playing or direct engagement, to communicate
with the environment or other agents to gather essential information for task completion.
• Reasoning requires LMAs to combine their internal knowledge with newly gathered information
to perform long-chain, multi-step reasoning.
• Decision Making and Goal Achievement requires LMAs to establish clear goals and make
independent decisions in response to environmental changes. This autonomous decision-making is
crucial for effectively navigating and completing tasks in dynamic settings.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Since the benchmarks evaluates many things, the level of detail differs between the constructs.
Task definition: The agent arena component is based on "winning" in a murder mystery game, whereas the Chain-of-Evaluation component is based on a QA format. 
Task item definition: In the arena setting, each task item is a single murder mystery game with a winner. In the CoE, each task is a multiple-choice question.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task source detail: The arena is based on a script, and the questions are manually annotated. The murder game scripts ccome from real sources.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: It is based on a pure "fictional" game, with the hope that capabilities are general enough to transfer.
Dataset size: 3000
Dataset size notes: Only reported approximately
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice, Extended interaction (e.g. conversation, calling an API and processing the response)
Response format detail: CoE is multiple choice, arena is extended interaction
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Win rate
Metric definition detail: Some metrics (e.g., Role-playing) is entirely LLM-derived
Metric aggregation: Simple Mean
Metric statistics: Simple mean (no variance or standard reported)
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Repo without any code is provided.
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Agents / 
New taxonomy: General Capability / Core Agentic Capabilities