Title: MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration
Benchmark: MAgIC
Summary: The paper presents a benchmark called MAGIC  that is designed to evaluate Large Language Models (LLMs) in multi-agent settings. It evaluates LLMs' capabilities in multi-agent environments through competition-based scenarios and defines seven metrics to measure.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multi-agent capabilities of LLMs 
Phenomenon defined by authors: Yes
Phenomenon definition:  "the essential capabilities of LLMs (Wooldridge, 2009; Minsky, 1988)" in multi-agent systems, which they categorize into: "(1) Judgment and reasoning form the core cognition of agents, crucial for accurate information estimation in uncertain scenarios. (2) Self-awareness and deception are key to enhanced adaptability in agents, vital for multi-agent system. (3) Rationality serves as a metric to gauge the efficiency of an agent's behavior. It directs agents toward making decisions with the aim of optimizing their benefits by considering the potential actions of other agents rather than resorting to impulsive or uninformed actions. (4) Cooperation and coordination are two facets of collaboration, essential for effective teamwork in multi-agent systems."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Capabilites: judgment, reasoning, deception, self-awareness, cooperation, coordination, and rationality
Task definition: competition-based scenarios requiring the specified capabilities
Task item definition: A single item consists of a specific game scenario (e.g., Prisoner's Dilemma) with defined roles, rules, etc where the LLM must interact with other agents to achieve its objectives.


Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 103
Dataset metadata available: Yes
Dataset metadata detail:  scenario type, roles, topic settings, game rules, win conditions
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: Exact Match (accuracy, F1, precision, recall), Win rate
Metric aggregation: Simple Mean
Metric statistics: simple mean to aggregate performance over scenarios and roles
Metric subscores: Yes
Metric subscores detail: capability, performance, role
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They show the  correlation between the area of the radar charts and the win rates, indicating their metrics effectively capture capabilities relevant for success in multi-agent settings.
Taxonomy: Agents / 
New taxonomy: General Capability / Core Agentic Capabilities