Title: EVALUATING LANGUAGE MODEL AGENCY THROUGH
NEGOTIATIONS
Summary: The paper introduces a dynamic framework for evaluating LLMs using negotiation games in self-play and cross-play settings. They find that only closed-source models are able to successfully complete the task and that stronger LLMs don't always win over weaker opponents.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Alignment
Phenomenon defined by authors: Yes
Phenomenon definition: Alignment metrics of interest are internal and external faithfulness as defined in Section 2.3, and the ability to follow instructions. [...] We measure instruction-following behavior of staying within the maximum number of words allowed to generate notes/messages (note/msg instruct) and the ability to correctly format internal offer indications using valid JSON (format instruct). [... (from 2.3)...]. . In natural language processing (NLP), faithfulness is a concept used to describe how accurately a model’s reasoning explains its answers/actions. To measure internal faithfulness, agents are asked to summarize acceptable offers for each Issue in their mental notes. [...] If Alice makes an offer to Bob for fewer slices than she stated as acceptable, we register this as an instance of internal unfaithfulness.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: The paper is a bit unfocused in what it measures. The title says "Agency", the authors mainly note "Alignment" as motivation, and there is also a degree of "Negotiation skill" and "Theory of Mind". 
Task definition: The task is a series of negotiation games, where LLMs are given rules, a persona, protocols, and goals. Agents do both internal deliberation and external negotiation, and the game ends when a completion criteria is reached. 
Task item definition: A single task is one round of a negotiation game that is either self-play or against another model. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: The authors generate a list of Games, Issues. It seems these were crafted manually
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The tasks simulates agent negotiations (so no humans involved)
Dataset metadata available: Yes
Dataset metadata detail: prompts, game settings, issues
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Extended interaction (e.g. conversation, calling an API and processing the response)
Response format detail: Negotiation
Metric definition: Exact Match (accuracy, F1, precision, recall), Number of rounds completted
Metric aggregation: Simple Mean
Metric statistics: mean with variance
Metric subscores: Yes
Metric subscores detail: Scores are reported for different types of games. 
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Partially
Metric face validity: Partially
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: It is an entirely constructed scenario (no available realistic setting)
Human baseline: No
Authors discuss validity: No
Taxonomy: Alignment / Alignment
New taxonomy: General Capability / Alignment
Other notes: This "benchmark" defines too many phenomena to fit neatly in the framework