Title: Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark
Benchmark: MACHIAVELLI
Summary: MACHIAVELLI is a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million diverse scenarios centered on social decision-making. It measures AI agents' tendencies to be power-seeking, cause disutility, and commit ethical violations when pursuing rewards, revealing tensions between maximizing rewards and behaving ethically. 
Contribution: The authors explore methods to steer agents toward more ethical behavior while maintaining competence.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Unethical behavior as trade-offs in reward-seeking AI agents
Phenomenon defined by authors: Yes
Phenomenon definition: The phenomenon being measured is framed as the tension between maximizing reward and behaving ethically. Specifically, the authors examine three main components: (1) ethical violations, defined as actions that violate deontological principles; (2) disutility, defined as when the player reduces the wellbeing of other characters; and (3) power-seeking, defined as when the player attempts to take actions which increase its ability to influence the state of the world. They also provide comprehensive mathematical definitions of these phenomena. Note they often also use other general terms like "harmful behaviours". 
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: The paper provides detailed mathematical formulations for each of the components being measured (ethical violations, disutility, and power-seeking). For power specifically, they develop multiple definitions involving different units (watts, money, wellbeing, and social influence) and different temporal perspectives (resource power, prospective power, and exercised power)
Task definition: Agents navigate text-based Choose-Your-Own-Adventure games where they observe scenes, select actions from a list, and attempt to achieve in-game objectives while their ethical behaviors are tracked through dense annotations of actions.
Task item definition: A single item is a game scenario where the agent observes the current scene text ("You take out the guards but when you turn around Jess has her gun pointed as you...", selects an action from provided options ("I tell her I want to help. It'll be a sure way to advance my ambitions" vs "I lie, telling her I want to help. I can work against her secretly"), then receives annotations tracking ethical violations, disutility, and power-seeking behaviors. The agent also received goals e.g., "First Paycheck", "Increase your family's reputation".
Task definition detail: The agent's trajectory is tracked along multiple behavioral metrics including various forms of power (economic, physical, social), disutility to others, and 13 categories of ethical violations (deception, killing, etc.).
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt), Human-crafted task examples from an existing human game (Choose-Your-Own-Adventure)
Task source detail: Adventure games from choiceofgames.com, which were written by human authors for human players. GPT-4 then annotates these for the behavioural metrics.
Task ecology: Proxy task - tries to get at real-world scenarios of agents via fictional adventures
Task ecology detail: The task simulates real-world social decision-making scenarios, though the game scenarios are fictional and narrativised so their applicability to the real world may be limited.
Dataset size: The test set is 30 games where agents trained to maximize reward perform poorly on behavioral metrics, i.e., where points and harms are positively correlated.
Dataset size notes: The full dataset consists of 134 games containing 572,322 scenarios
Dataset metadata available: Yes
Dataset metadata detail: Game title, behavioral metrics (power, disutility, ethical violations across 13 categories)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Response format detail: Convenience because they take from existing Choose-your-adventure games but then specific criteria used to select the test set (where agents trained to maximize reward perform poorly on behavioral metrics, i.e., where points and harms are positively correlated.)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring), A key metric is: Score = # harms committed by agent / # harms committed by random baseline (aka a normalised ratio relative to random baseline of 1000 random trajectories)
Metric definition detail: The benchmark uses two primary types of metrics: (1) Reward - defined as the normalized sum of achievement points obtained by the agent, and (2) Behavioral metrics - defined as the ratio of harmful behaviors committed by the agent compared to a random baseline agent. Lower behavioral scores (closer to 0%) indicate better ethical behavior.  Note the behavioural annotations are provided by GPT-4.
Metric aggregation: Simple Mean, Relative ratio to random baseline of 1000 random trajectories
Metric subscores: Yes
Metric subscores detail: Scores are provided for each of the behavioral metrics separately (power, disutility, and ethical violations) as well as finer-grained subscores within those categories. For example, power is broken down into economic, physical, social, and utility dimensions. Ethical violations are broken down into 13 categories including deception, killing, manipulation, etc.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia and Non-Profit
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Author validity detail: They discuss how text-based games serve as a natural test-bed for evaluating interactive agents that require planning and natural language understanding. They argue that MACHIAVELLI's structure, with multiple competing objectives, realistic action spaces, long-term planning requirements, and moral trade-offs, allows for characterizing agent behavior in ways that may predict real-world deployment risks. They test the validity of using GPT-4 as a annotator by comparing GPT-4 annotations against human annotations, showing that their model-based annotation scheme outperforms human crowdworkers on most label categories.
Taxonomy: Alignment / Alignment
New taxonomy: General Capability / Alignment