Title: LLMARENA: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
Benchmark: LLMArena
Summary: LLMArena is a comprehensive benchmark for multi-agent LLM games. It proposes seven game environments that test a wide array of capabilities. The games range from adversarial games like Poker to collaborative games like Hanabi (a personal favourite of this reviewer).
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration (taken directly from the paper)
Phenomenon defined by authors: No
Phenomenon definition: The phenomena are only defined through which game they are measured by and how they are evaluated within the game context (e.g., Numerical reasoning -> difference between score and nash equalibrium in `bid` game). 
Definition scope: Comprehensive
Definition integrity: Authors' description is unclear
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Each of the (many) phenomena are measured seperately. 
Task definition: The tasks are different multiplayer game environments where all players are LLMs. The games are TicTacToe, ConnectFour, Texas Hold'em, Bid, Bargain, and Undercover 
Task item definition: A single instance of a game with instructions and a full interaction.
Task definition detail: There is a lot of variation between the games.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: The games are fairly typical, but the exact implementation comes from the authors. 
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: All games are "real" insofar they are played by humans, but having LLMs compete is quite constructed. 
Dataset size detail: The benchmark is entirely dynamic and is played until the scores converge. 
Dataset metadata available: No
Sampling method: Unknown
Response format: Extended interaction (e.g. conversation, calling an API and processing the response)
Response format detail: The format of each is a multi-turn game. 
Metric definition: TrueScore, win rates, reward (game specific)
Metric aggregation: Normalized score (to best model)
Metric subscores: Yes
Metric subscores detail: Each game
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: It really depends on the phenomena and task
Metric face validity: Depends on the game
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Author validity detail: Benchmark is only for ConnectFour (human alwas wins)
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning