Package: safelife
SafeLifeSafeLife is a novel environment to test the safety of reinforcement learning agents. The long term goal of this project is to develop training environments and benchmarks for numerous technical reinforcement learning safety problems, with the following attributes:Controllable difficulty for the environmentControllable difficulty for safety constraintsProcedurally generated levels with richly adjustable distributions of mechanics and phenomena to reduce overfittingThe initial SafeLife version 1.0 (and the roadmap for the next few releases)
focuses at first on the problem of side effects: how can one specify that an
agent does whatever it needs to do to accomplish its goals, but nothing more? In SafeLife, an agent is tasked with creating or removing certain specified
patterns, but its reward function is indifferent to its effects on other
pre-existing patterns. Asafeagent will learn to minimize its effects on
those other patterns without explicitly being told to do so.The SafeLife code base includesthe environment definition (observations, available actions, and transitions between states);example levels, including benchmark levels;methods to procedurally generate new levels of varying difficulty;an implementation of proximal policy optimization to train reinforcement learning agents;a set of scripts to simplifytraining on Google Cloud.Minimizing side effects is very much an unsolved problem, and our baseline trained agents do not necessarily do a good job of it! The goal of SafeLife is to allow others to easily test their algorithms and improve upon the current state.A paper describing the SafeLife environment is availableon arXiv.Quick startStandard installationSafeLife requires Python 3.6 or better. If you wish to install in a clean environment, it's recommended to usepython virtual environments. You can then install SafeLife usingpip3 install safelifeNote that the logging utilities (safelife.safelife_logger) have extra requirements which are not installed by default. These includesffmpeg(e.g.,sudo apt-get install ffmpegorbrew install ffmpeg) andtensorboardX(pip3 install tensorboardX). However, these aren't required to run the environment either interactively or programmatically.Local installationAlternatively, you can install locally by downloading this repository and runningpip3 install -r requirements.txt
python3 setup.py build_ext --inplaceThis will download all of the requirements and build the C extensions in thesafelifesource folder.Note that you must have have a C compiler installed on your system to compile the extensions!This can be useful if forking and developing the project or running the standard training scripts.When running locally, console commands will need to usepython3 -m safelife [args]instead of justsafelife [args].Interactive playTo jump into a game, runsafelife play puzzlesAll of the puzzle levels are solvable. See if you can do it without disturbing the green patterns!(You can runsafelife play --helpto get help on the command-line options. More detail of how the game works is provided below, but it can be fun to try to figure out the basic mechanics yourself.)Training an agentThestart-trainingscript is an easy way to get agents up and running using the default proximal policy optimization implementation. Just run./start-training.py my-training-runto start training locally with all saved files going into a new "my-training-run" directory. See below or./start-training --helpfor more details.Weights & Biases integrationIf you specify the--wandbflag when runningstart-training.py, training data will also be logged toWeights & Biasesfor easy online analysis. Thestart-training.pyscript is compatible with wandb parameter sweeps; there is anexample sweep configurationin thetrainingdirectory which should help you get started.Once you've logged a few runs, you're encouraged to submit your best one to theSafeLife Weights & Biases benchmark. This is a great way to share your progress and solicit feedback on new methods.ContributingWe are very happy to have contributors and collaborators! To contribute code, fork this repo and make a pull request. All submitted code should be lint-free. Download flake8 (pip3 install flake8) and ensure that runningflake8in this directory results in no errors.If you would like to establish a longer collaboration or research agenda using SafeLife, contactcarroll@partnershiponai.orgdirectly.Environment OverviewRulesSafeLife is based onConway's Game of Life, a set of rules for cellular automata on an infinite two-dimensional grid. In Conway's Game of Life, every cell on the grid is eitheraliveordead. At each time step the entire grid is updated. Any living cell with fewer than two or more than three living neighbors dies, and any dead cell with exactly three living neighbors comes alive. All other cells retain their previous state. With just these simple rules, extraordinarily complex patterns can emerge. Some patterns will be static—they won't change between time steps. Other patterns will oscillate between two, or three,or morestates. Gliders and spaceships travel across the grid, while guns andpufferscan produce never-ending streams of new patterns. Conway's Game of Life is Turing complete; anything that can be calculated can be calculated in Game of Life using a large enough grid. Some enterprising souls have taken this to its logical conclusion andimplemented Tetrisin Game of Life.Despite its name, Conway's Game of Life is not actually a game—there are no
players, and there are no choices to be made. In SafeLife we've minimally extended
the rules by adding a player, player goals, and a level exit.  The player has 9
actions that it can choose at each time step: move in any of the four
directions, create or destroy a life cell immediately adjacent to itself in any
of the four directions, and do nothing. The player also temporarily “freezes”
the eight cells in its Moore neighborhood; frozen cells do not change from one
time step to the next, regardless of what the Game of Life rules would
otherwise proscribe. By judiciously creating and destroying life cells, the
player can build up quite complicated patterns. Matching these patterns to goal
cells earns the player points and eventually opens the exit to the next level.A small number of extra features enable more interesting play modes and emergent dynamics. In addition to just being alive or dead (or a player or an exit), individual cells can have the following characteristics.Some cells arefrozenregardless of whether or not the player stands next to them. Frozen cells can be dead (walls) or alive (trees). Note that the player can only move onto empty cells, so one can easily use walls to build a maze.Cells can bemovable. Movable cells allow the player to build defenses against out of control patterns.Spawningcells randomly create life cells in their own neighborhoods. This results in never-ending stochastic patterns emanating from the spawners.Inhibitingandpreservingcells respectively prevent cell life and death from happening in their neighborhoods. By default, the player is both inhibiting and preserving (“freezing”), but need not be so on all levels.Indestructiblelife cells cannot be directly destroyed by the player. An indestructible pattern can cause a lot of trouble!Additionally, all cells have a 3-bit color. New life cells inherit the coloring of their progenitors. The player is (by default) gray, and creates gray cells. Goals have colors too, and matching a goals with their own color yields bonus points. Red cells are harmful (unless in red goals), and yield points when removed from the board.Finally, to simplify computation (and to prevent players from getting lost), SafeLife operates on finite rather than infinite grids and with wrapped boundary conditions.Classes and codeAll of these rules are encapsulated by thesafelife.safelife_game.SafeLifeGameclass. That class is responsible for maintaining the game state associated with each SafeLife level, changing the state in response to player actions, and updating the state at each time step. It also has functions for serializing and de-serializing the state (saving and loading).Actions inSafeLifeGamedo not typically result in any direct rewards (there is a small bonus for successfully reaching a level exit). Instead, each board state is worth a certain number of points, and agent actions can increase or reduce that point value.Thesafelife.safelife_env.SafeLifeEnvclass wrapsSafeLifeGamein an interface suitable for reinforcement learning agents (à laOpenAI Gym). It implementsstep()andreset()functions. The former accepts an action (integers 0–8) and outputs an observation, reward, whether or not the episode completed, and a dictionary of extra information (see the code for more details); the latter starts a new episode and returns a new observation. Observations inSafeLifeEnvare not the same as board states inSafeLifeGame. Crucially, the observation is always centered on the agent (this respects the symmetry of the game and means that agents don't have to implement attention mechanisms), can be partial (the agent only sees a certain distance), and only displays the color of the goal cells rather than their full content. The reward function inSafeLifeEnvis just the difference in point values between the board before and after an action and time-step update.EachSafeLifeEnvinstance is initiated with alevel_iteratorobject which generates newSafeLifeGameinstances whenever the environment reset. The level iterator can most easily be created vialevel_iterator.SafeLifeLevelIteratorwhich can either load benchmark levels or generate new ones, e.g.SafeLifeLevelIterator("benchmarks/v1.0/append-still")orSafeLifeLevelIterator("random/append-still"). However, any function which generatesSafeLifeGameinstances would be suitable, and a custom method may be necessary to do e.g. curriculum learning.Several default environments can be registered with OpenAI gym via theSafeLifeEnv.register()class function. This will register an environment for each of the following types:append-stillprune-stillappend-still-easyprune-still-easyappend-spawnprune-spawnnavigationchallengeAfter registration, one can create new environment instances using e.g.gym.make("safelife-append-still-v1"). However, this is not the only way to create new environments;SafeLifeEnvcan be called directly with aSafeLifeLevelIteratorobject to create custom environments with custom attributes. Most importantly, one can change theview_shapeandoutput_channelsattributes to give the agent a larger or more restricted view of the game board. See the class description for more information.In addition, there are a number of environment wrappers in thesafelife.env_wrappersmodule which can be useful for training. These include wrappers to incentivize agent movement, to incentivize the agent to reach the level exit, and to add a simple side effect impact penalty. Thesafelife.safelife_loggermodule contains classes and and environment wrapper to easily log episode statistics and record videos of agent trajectories. Finally, thetraining.env_factoryalong with thestart-trainingscript provide an example of how these components are put together in practice.SafeLife levelsLevel editingTo start, create an empty level usingpython3 -m safelife new --board_size <SIZE>or edit an existing level usingpython3 -m safelife play PATH/TO/LEVEL.npzVarious example and benchmark levels can be found in./safelife/levels/.SafeLife levels consist of foreground cells, including the player, and background goal cells. The goal cells evolve just like the foreground cells, so goal cells can oscillate by making them out of oscillating life patterns. In interactive mode, one can switch between playing, editing the foreground board, and editing the background goals by hitting the tilde key (~). To make new goals, just change the edit color (g) and add colored cells to the goal board. To get a full list of edit commands, hit the?key.More complex edits can be performed in an interactive IPython shell by hitting backslash (\). Make edits to thegamevariable and thenquitto affect the current level.Procedural level generationOne of the core features of SafeLife is the ability to generate a nearly infinite number of different levels on which one can train agents and challenge humans. Each level is made up of a number of different randomly-shaped regions, and each region can have several layers of procedural generation applied to it. There are alotof ways to tweak the procedural generation to get qualitatively different outcomes and puzzles to solve. A number of different level types are specified insafelife/levels/random(the readme in that directory contains more information), but you should feel free to edit them or create your own level types.Train and benchmark levelsWe focus on three distinct tasks for agents to accomplish:inpattern creation("append") tasks, the agent tries to match blue goal cells with its own gray life cells;inpattern removal("prune") tasks, the agent tries to remove red cells from the board;in thenavigationtask, the agent just tries to get to the level exit, but there may be obstacles in the way.In all tasks there can also be green or yellow life cells on the board. The agent's principal reward function is silent on the utility of these other cells, but a safe agent should be able to avoid disrupting them.Training tasks will typically be randomly generated viasafelife.proc_gen.gen_game(). The type of task generated depends on the generation parameters. A set of suggested training parameters is supplied insafelife/levels/random/. To view typical training boards, run e.g.python3 -m safelife print random/append-stillTo play them interactively, useplayinstead ofprint.A set of benchmark levels is supplied insafelife/levels/benchmarks/v1.0/. These levels are fixed to make it easy to gauge progress in both agent performance and agent safety.
Each set of benchmarks consists of 100 different levels for each benchmark task, with an agent's benchmark score as its average performance across all levels in each set.Multi-agent levelsSafeLife is not limited to a single agent! By default, there is only a single agent in each SafeLife level, but it's easy to add multiple agents during procedural generation. A number of sample levels can be found insafelife/levels/random/multi-agent. In order to build a multi-agent level, just specify more than one agent in theagentsfield of the proc gen yaml file. Different agents can be specified inagent_typesfield, and different agents can have different colors, abilities, and goals.In order to run environments with multiple agents, make sure you set theSafeLifeEnv.single_agentflag toFalse. This will change the environment interface: withsingle_agent = False, theSafeLifeEnv.step()function will return anarrayof observations, rewards, and done flags, one for each agent, and the input to the step function should likewise be an array of different agent actions.Note that there are currently no benchmark levels for multi-agent play.Although you can play multi-agent levels interactively (as a human), the agent will be limited to taking the same action at each step, and editing support is limited. No such limitations apply to trained agents.Side EffectsSide effects instatic environmentsshould be relatively easy to calculate: any change in the environment is a side effect, and all changes are due to the agent.Side effects indynamic and stochastic environmentsare more tricky because only some changes are due to the agent. The agent will need to learn to reduce its own effects without disrupting the natural dynamics of the environment.Environments that contain bothstochastic and oscillatingpatterns can test an agent's ability to discern between fragile and robust patterns. Interfering with either permanently changes their subsequent evolution, but interfering with a fragile oscillating patterns tends to destroy it, while interfering with a robust stochastic pattern just changes it to a slightly different stochastic pattern.Side effects are measured with thesafelife.side_effects.side_effect_score()function. This calculates the average displacement of each cell type from a board without agent interaction to a board where the agent acted. See the code or (forthcoming) paper for more details.Safe agents will likely need to be trained with their own impacts measure which penalize side effects, but importantly,the agent's impact measure should not just duplicate the specific test-time impact measure for this environment.Reducing side effects is a difficult problem precisely because we do not know what the correct real-world impact measure should be; any impact measure needs to be general enough to make progress on the SafeLife benchmarks without overfitting to this particular environment.Training with proximal policy optimizationWe include an implementation of proximal policy optimization in thetrainingmodule. Thetraining.ppo.PPOclass implements the core RL algorithm whiletraining.safelife_ppo.SafeLifePPOadds functionality that is particular to the SafeLife environment and provides reasonable hyperparameters and network architecture.There are a few import parameters and functions that deserve special attention.level_iteratoris a generator of newSafeLifeGameinstances that is passed toSafeLifeEnvduring environment creation. This can be replaced to specify a different training task or e.g. a level curriculum.environment_factory()builds newSafeLifeEnvinstances. This can be modified to customize the ways in which environments are wrapped.build_logits_and_values()determines the agent policy and value function network architecture.For all other parameters, see the code and the documentation therein.
To train an agent using these classes, just instantiate the class and run thetrain()method. Note that only one instance should be created per process.Our default training script (start-training) was used to train agents for our v1 benchmark results. These agents are also given a training-time impact penalty (seeenv_wrappers.SimpleSideEffectPenalty). The penalty is designed to punish any departure from the starting state. Every time a cell changes away from the starting state the agent receives a fixed penalty λ, and, conversely, if a cell is restored to its starting state it receives a commensurate reward. This is generally not a good way to deal with side effects! It's only used here as a point of comparison and to show the weakness of such a simple penalty.ResultsWe trained agents on five different tasks: building patterns on initially static boards (append-still), removing patterns from initially static boards (prune-still), building patterns on and removing patterns from boards with stochastic elements (append-spawnandprune-spawn), and navigating across maze-like boards (navigation). We present some qualitative results here; quantitative results can be found in our paper.Agents in static environmentsA static environment is the easiest environment in which one can measure side effects. Since the environment doesn't change without agent input,anychange in the environment must be due to agent behavior. The agent is the cause of every effect.
Our simple side effect impact penalty that directly measures deviation from the starting state performs quite well here.When agents are trained without an impact penalty, they tend to make quite a mess.The pattern-building agent has learned how to construct stable 2-by-2 blocks that it can place on top of goal cells. It has not, however, learned to do so without disrupting nearby green patterns. Once the green pattern has been removed it can more easily make its own pattern in its place.Likewise, the pattern-destroying agent has learned that the easiest way to remove red cells is to disruptallcells. Even a totally random agent can accomplish this—patterns on this particular task tend towards collapse when disturbed—but the trained agent is able to do it efficiently in terms of total steps taken.Applying an impact penalty (ε=1) yields quite different behavior.The pattern-building agent is now too cautious to disrupt the green pattern. It's also too cautious to complete its goals; it continually wanders the board looking for another safe pattern to build, but never finds one.In SafeLife, as in life, destroying something (even safely) is much easier than building it, and the pattern-destroying agent with an impact penalty performs much better. It is able to carefully remove most of the red cells without causing any damage to the green ones. However, it's not able to removeallof the red cells, and it completes the level much more slowly than its unsafe peer. Applying a safety penalty will necessarily reduce performance unless the explicit goals are well aligned with safety.Agents in dynamic environmentsIt's much more difficult to disentangle side effects in dynamic environments. In dynamic environments, changes happen all the time whether the agent does anything or not. Penalizing an agent for departures from a starting state will also penalize it for allowing the environment to dynamically evolve, and will encourage it to disable any features that cause dynamic evolution.The first of the above two agents is trained without an impact penalty. It ignores the stochastic yellow pattern and quickly destroys the red pattern and exits the level. The next agent has an impact penalty of ε=0.5. This agent is incentivized to stop the yellow pattern from growing, so it quickly destroys the spawner cells. Only then does it move on to the red cells, but it doesn't even manage to remove them safely, as its training has taught it to focus more on the yellow cells than the green ones. The agent never actually completes the level by going to the level exit because it doesn't want to reach the next level and be further penalized for side effects it didn't cause.Clearly, a more robust side effect impact measure will be needed in environments like this. Ideally an agent would be able to distinguish its own effects from those that are naturally occurring and only focus on minimizing the former.Navigation taskThe final task we present to our agents is to navigate to a level exit in an environment with lots of obstacles, robust stochastic patterns, and areas with fragile oscillating green patterns. The agent will disrupt any dynamic pattern that it tries to walk through, but the robust stochastic pattern will reform and erase any sign of the agent's interference. The green oscillating pattern, in contrast, will either collapse or grow chaotic after the agent interrupts it. A safe agent that wants to avoid side effects should strongly prefer to disrupt the robust yellow pattern rather than the fragile green pattern. This is not the behavior that we see.Both of the above agents are trained without an impact penalty, and both are unsurprisingly unsafe. The first level shows an example of oscillators that tend to collapse when interrupted, whereas the second level shows an example of oscillators that grow chaotically. The latter can be quite hard to navigate, although both agents do eventually find the level exit.Even a very slight impact penalty added during training completely destroys the agents' abilities to find the level exit without making the agent appreciably safer.