# POWERplay: An open-source toolchain to study AI power-seeking

We're open-sourcing **POWERplay**, a research toolchain you can use to study power-seeking behavior in reinforcement learning agents. POWERplay was developed by [Gladstone AI](https://www.gladstone.ai/) for internal research.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ba1082658b4753ba6101d7f39142790d78f408a293e0fb62.png)

POWERplay's main use is to estimate the **instrumental value** that a reinforcement learning agent can get from a state in an [MDP](https://en.wikipedia.org/wiki/Markov_decision_process). Its implementation is based on a definition of instrumental value (or "POWER") first proposed by [Alex Turner et al.](https://arxiv.org/pdf/1912.01683.pdf) We've extended this definition to cover [certain tractable multi-agent RL settings](https://www.alignmentforum.org/posts/cemhavELfHFHRaA7Q/misalignment-by-default-in-multi-agent-systems#2__Multi_agent_POWER__human_AI_scenario), and built an implementation behind a simple Python API.

We've used POWERplay previously to obtain some suggestive early results in [single-agent](https://www.alignmentforum.org/posts/pGvM95EfNXwBzjNCJ/instrumental-convergence-in-single-agent-systems) and [multi-agent](https://www.alignmentforum.org/posts/cemhavELfHFHRaA7Q/misalignment-by-default-in-multi-agent-systems) [power-seeking](https://www.alignmentforum.org/posts/nisaAr7wMDiMLc2so/instrumental-convergence-scale-and-physical-interactions). But we think there may be [more low-hanging fruit](https://www.alignmentforum.org/posts/nisaAr7wMDiMLc2so/instrumental-convergence-scale-and-physical-interactions#5_2_Suggestions_for_future_work) to be found in this area.

Beyond our own ideas about what to do next, we've also received some [interesting conceptual questions](https://www.alignmentforum.org/posts/pGvM95EfNXwBzjNCJ/instrumental-convergence-in-single-agent-systems?commentId=QJmvsKCnupCJbmdhd) in connection with this work. A major reason we're open-sourcing POWERplay is to lower the cost of converting these conceptual questions into real experiments with concrete outcomes, that can support or falsify our intuitions about instrumental convergence.

Ramp-up
-------

We've designed POWERplay to make it as easy as possible for you to get started with it. Follow the [installation](https://github.com/gladstoneai/POWERplay#installation-setup-and-testing) and [quickstart](https://github.com/gladstoneai/POWERplay#quickstart) instructions to get moving quickly. Use the [replication API](https://github.com/gladstoneai/POWERplay#replicating-figures) to trivially reproduce any figure from any post in our [instrumental convergence sequence](https://www.alignmentforum.org/s/HBMLmW9WsgsdZWg4R). Design [single-agent and multi-agent MDPs and policies](https://github.com/gladstoneai/POWERplay#setting-up-the-environment), launch [experiments](https://github.com/gladstoneai/POWERplay#launching-an-experiment) on your local machine, and [visualize results with clear figures and animations](https://github.com/gladstoneai/POWERplay#visualizing-experiment-results).

POWERplay comes with "batteries included", meaning all the code samples in the documentation should just work out-of-the-box if it's been installed successfully. It also comes with pre-run examples of experimental results, so you can understand what "normal" output is supposed to look like. While this does make the repo weigh in at about 500 MB, it's worth the benefits of letting you immediately start playing around with [visualizations](https://github.com/gladstoneai/POWERplay#visualizing-experiment-results) on preexisting data.

If we've done our job right, a smart and curious grad student (with a bit of Python experience) should be able to start reproducing our previous experiments within an hour, and to have some new — and hopefully interesting! — results within a week.

We're looking forward to seeing what people do with this. If you have any questions or comments about POWERplay, feel free to reach out to Edouard at edouard@gladstone.ai.

[**Clone POWERplay on GitHub.**](https://github.com/gladstoneai/POWERplay)