Avoiding Negative Side-Effects and Promoting Safe Exploration with Imaginative Planning
Authors: Dhruv Ramani
Benjamin Eysenbach
Venue:  (2020)
Keywords: Reinforcement Learning
AI-Safety
Model-Based Reinforcement Learning
Safe-Exploration

Abstract:   With the recent proliferation of the usage of reinforcement learning (RL) agents for solving real-world tasks, safety emerges as a necessary ingredient for their successful application. In this paper, we focus on ensuring the safety of the agent while making sure that the agent does not cause any unnecessary disruptions to its environment. The current approaches to this problem, such as manually constraining the agent or adding a safety penalty to the reward function, can introduce bad incentives. In complex domains, these approaches are simply intractable, as they require knowing apriori all the possible unsafe scenarios an agent could encounter. We propose a model-based approach to safety that allows the agent to look into the future and be aware of the future consequences of its actions. We learn the transition dynamics of the environment and generate a directed graph called the imaginative module. This graph encapsulates all possible trajectories that can be followed by the agent, allowing the agent to efficiently traverse through the imagined environment without ever taking any action in reality. A baseline state, which can either represent a safe or an unsafe state (based on whichever is easier to define) is taken as a human input, and the imaginative module is used to predict whether the current actions of the agent can cause it to end up in dangerous states in the future. Our imaginative module can be seen as a ``plug-and-play'' approach to ensuring safety, as it is compatible with any existing RL algorithm and any task with discrete action space. Our method induces the agent to act safely while learning to solve the task. We experimentally validate our proposal on two gridworld environments and a self-driving car simulator, demonstrating that our approach to safety visits unsafe states significantly less frequently than a baseline.

Under review as a conference paper at ICLR 2020
AVOIDING NEGATIVE SIDE-EFFECTS AND PROMOTING SAFE EXPLORATION WITH IMAGINATIVE PLANNING
Anonymous authors Paper under double-blind review
ABSTRACT
With the recent proliferation of the usage of reinforcement learning (RL) agents for solving real-world tasks, safety emerges as a necessary ingredient for their successful application. In this paper, we focus on ensuring the safety of the agent while making sure that the agent does not cause any unnecessary disruptions to its environment. The current approaches this problem, such manually con- straining the agent or adding a safety penalty to the reward function, can introduce bad incentives. In complex domains, these approaches as they require knowing apriori all the possible unsafe scenarios an agent could encounter. We propose model-based approach to safety that allows the agent look into the future and be aware of the future consequences of its actions. We learn the transition dynamics of the environment and generate a directed graph called the imaginative module. This graph encapsulates all possible trajectories that can be followed by the agent, allowing the agent to efficiently traverse through the imagined environment without ever taking any action in reality. A baseline state, which can either represent a safe or an unsafe state (based on whichever is casier to define) is taken human imput, and the imaginative module is used to predict whether the current actions of the agent can cause it to end up in dangerous states in the future. Our imaginative module approach to ensuring safety, as is compatible with any existing RL. algorithm and any task with discrete action space. Our method induces the agent to safely while learning to solve the task. We experimentally validate our proposal on two gridworld environments and self-driving car simulator, demonstrating that our approach to safety visits unsafe states significantly less frequently than baseline.
simply intractable,
be seen
"plug-and-play"
INTRODUCTION
Commonly, reinforcement learning (RL) (Sutton et al. (1998)) methods involve learning to solve specifically defined tasks by making the agent take a sequence of actions which maximize the long term rewards given by the environment. However, crucial and a commonly ignored part of these methods is ensuring the safety of the agent and its surrounding. As RL. research advances and starts being used for real-world tasks, having an exploitative policy whose main objective is maximize the rewards over all else is likely to fail in assuring general safety and may lead to serious, irreparable damage.
The past few years have seen a rise in awareness and research on A/ safety (Amodei et al. (2016)). Instead of focusing on extreme scenarios involving super-intelligent agents with unspecified objectives, this area of safety research focuses on practical issues with modern machine learning techniques. Specifically, within RI,, Al safety focuses on training an RL. agent to learn how to solve the intended task without causing any harmful or unexpected results. Our work proposes one method to make progress on two important problems in AI safety, The first problem we tackle is that of safe exploration (Moldovan & Abbeel (2012); Koller et al. (2018); Altman agent respecting the implicit safety constraints, not only at convergence but also throughout the learning process. Ensuring such safety constraints is critical for applying RL physical robots, where poorly-chosen actions may break the robot or cause it to get stuck in certain states. Safe exploration requires an agent predict which actions dangerous and explore ways in which it.
(1999). ). Safe exploration involves the

---

Under review as a conference paper at ICLR 2020
can avoid them. The second problem we focus upon is to make the agent act in accordance with the designer's 's intention, even when this intention is poorly defined. In many real-world scenarios, the ideal specification and details of the task are hard to articulate and specify. These nuanced details are hard to embed in a formal objective. In many situations, a reward-maximizing agent will ignore the intentions of the designer and take actions that exploit mistakes in the reward function which can potentially cause harm. As described in Clark& & Amodei (2016), a poorly designed reward function can lead to the problem of reward hacking and negative side effects (Amodei et al. (2016); Krakovna et al. (2018); TTrner et al. (2019)). In reward hacking, the agent learns to exploit and maximize the reward by finding an unexpected and unintended clever solution which does not match with the
intentions of the reward designer. We focus on the problem of negative side effects, in which the designer mistakenly ignores potentially large factors of the environment which can be irreversibly damaged if altered by the agent. Explicitly penalizing every possible side-effect is highly impractical and iabor-intensive. Not only does this approach lead to an extremely complex reward function, and even if such safety-aware reward function could be designed, it would impinge upon the agent's 's ability to effectively solve the task. In order to avoid excessive human input and to avoid side effects in complex environments, we must develop a more practical and generalized approach.
Not all side-effects are created equal, and many are benign. Hence, we want to avoid only those side-effects which are irreversible or difficult to reverse. In real-world tasks, irreversible changes often correspond to serious damage. For example, imagine a robot tasked with cleaning a table with a glass vase on top. The cleaning robot could either (1) move the vase to a different region of the table and carry out the cleaning or (2) drop the vase on the floor causing permanent damage and continue cleaning. The side-effect from the first strategy, moving the vase, is safe, as can be undone. In contrast, the second strategy causes an unsafe side effect, as the robot is unable to "unbreak" the vase. The key insight of this paper is that humans can imagine the consequence of their actions on their environment Toolmn (1948); Pfeiifer & Foster (2013). Humans can then explicitly choose the ones which do not cause any harm. The act of classifying the side-effect as safe or unsafe based on the prior experience of the person. For example, child learns that dropping a glass vase is harmful by being warned by an adult, sceing a vase drop, or dropping a vase themselves. Hence, whenever a human tries to clean the table, they can imagine the outcome of dropping the vase and chooses to avoid taking that action.
In this paper, we propose to use of imaginative planning to prevent the agent from taking unsafe actions which cause negative side-effects. We model the transitions between states and use this transition function to create roll-out graph which considers all possibilities that can occur as the agent progresses. We call this the imaginative module of the agent. We assume that a human initially specifies small number of baseline safe states or unsafe states, which we will use in conjunction with the imaginative module. If we define the baseline state as safe sate, an action ag is considered to be a "safe action" if there is at least one path from the next state (which it reaches after taking a;) to the baseline state in the imaginative module. By this definition, if the action as was an unsafe action then taking it would cause permanent change to the environment, hence not allowing it to reach the baseline state in the graph. If the baseline state is defined as unsafe state, action ae is considered to be a "safe action" if there are no paths from the next state to the specified baseline state in the imaginative module. By this, the action a would be unsafe if taking it will cause agent to always end up in the baseline state, More precisely, we say that ae is unsafe if all paths in the imaginative module from the next state lead to unsafe state.
Every-time the agent takes an action, we check whether the action would be safe or not based on the definition. If the action is not safe, we act greedily among the safe action based on the value function. Our formulation can be used with any on/off policy algorithm with discrete actions. Moreover, in addition to providing safety throughout the learning process, our imagination module can also accelerate training, as it eliminates a certain unsafe portion of the state, action pairs.
We run experiments confirm that our proposed method improves safety throughout learning and at convergence. We experiment our method on grid-world environments specifically designed for safety-related problem proposed by Leike et al. (2017) and on an autonomous car simulator part of the OpenAI gym (Brockman et al. (2016)). We experiment and test our imagination graph by incorporating it with an "unsafe" pre-trained policy and while training a policy using standard on-policy methods.
2

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
Due to their low sampling complexity, model-based reinforcement learning methods have been population approach to solving complex tasks. Learning the dynamics model is sample efficient as they are trained using standard supervised learning techniques, allowing the use of off-policy data. However, policy optimization heavily relies on the accuracy of the learned dynamics, and is likely to overfit if the model is not precise. There has been a lot of previous work (Weber et al.. (2017): Clavera et al. (2018) 8); Kurutach et al. (2018); |Hffner et al. (2018) ) on trying to reduce the model bias, combining elements of model-based and model-free reinforcement learning and scaling the model using other approaches.
Our work explicitly focuses on using a model- approach to tackle few important problems in Al safety. The field of AI safety (Amodei et al. 2016)) studies how AI agents can be modeled to avoid intended or unintended harmful accidents. An accident can be described as a situation in which the agent designed and deployed for the task doesn't 't adhere what the human designer had in mind and produces harmful and unexpected result. Among various research problems listed by [Amodei et al. (2016) which constitute the field of AT safety, we specifically focus on the problem of safe exploration (Moldovan & Abbeel (0122;; ; Koller et al. (2018); Altman (1999) and avoiding negative side effects (Amodei et al. (2016); Krakovna et al. (2018): Turner et al. (2019)).
The problem of safe exploration involves agent to respect the safety constraints, both while operation and learning. It involves agents to predict which actions will lead to dangerous states in the future, so as to safely explore the state space. Badly chosen actions may destroy the agent or trap it in states it can't get out of. Safe exploration is one of the most studied problem within the field of AI safety. There are various methods which involve preserving the reach-ability of the starting state (Moldovan & Abbeel (2012); Eysenbach et al. (2017) or safe regions (Gillula & Tomlin| (012); Fisac al. (2018) to promote safe exploration. Other methods (Achiam et al. (2017); Chow al. (2018); Dalal ctt al. (018)) involve changing the optimizing criteria for policy optimization from expected rewards by introducing implicit constraints. These are based a well studied formulation of constrained Markov Decision Process (CMDP) (Altman (1999) in which the agents must satisfy constraints on expectations of auxiliary costs. However, these method require explicit and clever design of the constraints, which can be difficult.
The importance of addressing the problem of negative side effect has been gaining a growing interest. The problem occurs when the designer mistakenly ignores potentially large factors of the environment which can be irreversibly damaged if altered by the agent while performing the main task. Explicitly penalizing every possible negative side-effect is intractable and can lead to extremely complex reward function design. An approach to solving this problem involves human oversight using inverse reinforcement learning (Ziebart et al. (2008); Hadfield
et al. (2016)). demonstrations (Abbeel & Ng (2004): Hester et al. (2018)) ()..o..)))).. or human feedback (Christiano et al. (2017): Warnell et (2018)) However, these methods are difficult quantify and depend a lot on the diversity of settings in which they are performed. A common and more tractable approach is that of using empowerment (Klyubin et al. (2055); Gregor al. (2016)) the agent has over the environment. This measures the control the agent has over it's environment which can be derived by calculating the mutual information between the agent's 's current action and the future states. To avoid irreversible side effects, the agent can be penalized for change in empowerment wrt. to a baseline state which we consider safe. Krakovna et al.) (2018); ; Turner et al. (2019) propose an important methodology of penalizing deviation by proposing the usage of different baseline states and deviation measures.
BACKGROUND
We consider an agent making decision sequentially over an environment E over discrete time steps t = 1, ...,, T. We model the decision making process using a Markov Decision Process (MDP), in which the agent experiences state 8, S, where S is a discrete set of states and takes an action a at each time-step t. The action aj is decided by a stochastic policy T ([a))s) over a discrete set of actions A. After taking each action, the agent receives a reward r, R(st, az) from the environment, and transitions to a new state $441 ~ P(s,, az). We do not assume the agent to have access to the model (R(s), 4z) & P(st, az)), and explicitly learn the model.
3

---

Under review as a conference paper at ICLR 2020
Learn Transition Dynamics
Imaginative Module
St, at
so
M
Avoid Unsafe Actions
st+, rt
Figure 1: The proposed approach, illustrated. The transition dynamics of the environment is learned and directed graph called imaginative module created from the rollouts. This is then used to plan and check if the agent's &s next action will lead to negative consequences in the future.
4 IMAGINATIVE PLANNING FOR SAFETY
Humans utilize multiple learning, memory, and decision making systems order to efficiently carry out the task. When the information of the environment is available or essentially required, it is found that the best strategy is model-based planning associated with prefrontal cortex (Daw et al.) (2005)). When expecting implicit safety from the agent, it seems apparent that the information about the environment will be needed for the agent to know the impact of their actions. We plan to use this insight to make deep reinforcement learning based agent aware of the implications of every action the agent takes and try to improve agent's decision-making capabilities in terms of safety and avoiding side effects.
In this paper, we consider making the agent initially learn about the environment and use this knowledge to check whether its actions would lead to negative permanent consequences. We initially make the agent learn the transition dynamics St+1 ((s az) and the reward function = R(s a.). We model these based on the recent successes action-come oction-conditional next-step predictors (Hafner et al. (2018); Kaiser et al. (2019)), which take input the current state (or history of observations) and current action, and predict the next state, and the next reward. These can hence used to query about the future for efficient and safe decision making. Given the current state 8, and the possible action ot, we try to predict the next state 8141 and the reward ri that the agent will get. To try to cover all possible state-action pairs, the environment randomly and collect the training data. Before testing/training the main policy, we generate directed graph of all possible trajectories that be followed by the agent using the transition dynamics we learned. The graph generated by rolling out the transition model multiple time-steps into the future, by initializing the path with the initial state 5, and subsequently feeding simulated states into the model. Each node of the graph consists of the imaginative state š and the reward fr, which the agent would get it had visited that state. The edge set represents the actions that the agent can take, and for every node, we consider all possible discrete actions and use the learned transition model to query the next state 5(+1- We refer this graph as the imaginative module of the agent. Querying multiple steps into the future involves the agent to search in the imaginative module and follow the respective path. Once the imaginative module has been created, we use it to explicitly tell the policy being learned/tested whether the action it's about to take is safe not.
deploy number of agents explore different aspects of
We define a baseline state(s) B, which we will incorporate into planning done by the imaginative module to ensure safety. The baseline states are taken as human input and based on the environment and the task, the baseline state can either represent a safe state or an unsafe state. For example, for the cleaning robot, the baseline state, which represents the safe state, could be the vase kept on the table (the initial state of the environment). For an autonomous car which has to learn how to drive without going off track, the baseline state represents unsafe state of the being out of track. How it
4

---

Under review as a conference paper at ICLR 2020
reaches outside the track is not required. If we define the baseline state safe sate, action a, is considered to be a "safe action" if there's is at least one path from the next state (which reaches after taking a₂) to the bascline state in the imaginative module. By this definition, if the action a, was an unsafe action then taking it would cause permanent change to the environment, hence not allowing it to reach the baseline state in the graph. If the baseline state is defined as unsafe state, action at is considered to be a "safe action" if there are no paths from the next state to the specified baseline state in the imaginative module. By this, the action de would be unsafe taking it will cause an agent to always end up in the baseline state. More precisely, we say an action is unsafe if all the paths from the next state lead an unsafe state, as predicted by the imaginative module.
The main policy π T(a st) which we learn/test is modeled using a convolutional neural network. This can either be policy which we want to learn while ensuring explicit safety even during training or pretrained unsafe policy to which we want to provide the ability to make safe decisions. To make the agent take safe action, we initially predict the action it would take using the modeled policy (a; st). Then we plug in our imaginative module to determine whether the action predicted by the policy safe or not. If the action is safe, the agent takes it. Otherwise, if the action is unsafe, the agent is made to act greedily with respect to the safe actions based on the value function. More formally, the agent takes the action as
(at (x)
if safe(s;, a.) if not safe(s),
O2
arg max V(((+1)
This allows the agent never take the unsafe action, even while we are training the policy or are using a pretrained policy. The agent is only made to act after the action predicted by the policy goes through the safety check. This "plug and play" usage of the imaginative module allows us to scale safety among different RI. agents.
In summary, before the episode starts we create an imaginative module for the agent starting from the starting state so by performing roll-outs of the transition model 8t+1 P(st, de) and creating graph which covers all the possible state-action pairs. We define the baseline state B which taken as human input. The baseline state either represent safe state an unsafe state. We then either perform the normal operation using a pretrained policy, or learn a new policy by augmenting it with the imaginative module to ensure safety even while learning. The action a, predicted by the policy r (ag ))) is checked for safety using the imaginative module as described above. If the az is safe, it is performed otherwise, we act greedily among all the safe actions wrt. value function V(s) V($++1) where 4 is the next state the agent reaches if performs yth safe action.
EXPERIMENTS
To test the incorporation of the imaginative module as proposed in the paper, we consider environments in which negative reward is provided to the agent when it is about to take unsafe action. To adhere to the baselines and to compare it with existing formulations, we test our methods on the AI Safety Gridworld Environments (Leike et al. (2017) related to avoiding negative side effects. To test the practicality of our method in a real-world-lin like environment, we train an agent to drive a car safely without getting off the track. The choice of baseline state B is specific each environment and is made by considering which is easier to define, set of safe states or a set of unsafe states.
5.1 EXPERIMENTAL SETTINGS
For the gridworld environments, imaginative module is modeled using two convolutional neural
networks (CNNs) to estimate the transition function 81+1 P(st, a.) and the reward function
T; = (R,,, a.). Both of these networks share a common CNN based encoder which brings the current state s, to embedding space. This embedding is passed through the separate networks of the transition and reward functions to predict the next state 4141 and the reward rt- The reward and the state space are discretized and this treated classification problem. Both the state loss and reward loss are cross-entropy losses over the softmax activation function.
The final loss is given
Total Loss
State Loss +ax Reward Loss,
5

---

Under review as a conference paper at ICLR 2020
A
Agent
Vase
Conveyor
>
>
Belt
Dead
End
Conveyor Environment
Side-Effect Environment
Scmc-411 (ulcome
G
Goal
(a)
(b)
Figure 2: (a) The gridworld environments are specifically designed to test agents against negative side effects. Usage of imaginative module ensures that the agent does take actions which lead to negative side effects in the future. (b) The car simulator environment used to train the agent to drive safely without going off the track. The baseline states here represents unsafe states.
where is regularization coefficient which provides the required weight to the reward loss. In our experiments, the value kept as 0.1. We use Adam Optimizer (Kingma (2014)) to minimize the loss.
To collect distinctive transition tuples (st, at, :t Te) for training the transition dynamics, we deploy multiple agents run in parallel. All of these agents act randomly and explore different aspects of the environment. In our experiments, we used 16 agents running in parallel. The transition dynamics model is trained for 5000 total timesteps. The imaginative module over the gridworld environment reports a ROC AUC score of 0.95.
For the driving environment model the imaginative module using rollouts from the actual environ- ment. This is done to assure that the imaginative module doesn't 't deviate from the actual environment, since the transition dynamics in this environment are hard learn due their continuous state space.
The policies are trained using the standard on-policy A2C method. The agent is modeled using a CNN with common encoder. The embeddings from the encoder are passed to 2 networks with 2 convolutional layers and dense layers representing the policy and the value function. The network is optimized using the RMSProp (Tieleman & Hinton (2012)) optimizer with learning rate as 7 x 10- Whenever these policies predict an action, the action is checked for safety with the imaginative module. the action unsafe, is avoided and the policy made act greedily based on the value function among all the safe actions.
5.2 GRIDWORLD ENVIRONMENTS
We the incorporation of the imaginative module environments focused avoiding negative side effects from the AI Safety Gridworlds suite (Leike et al. (2017)). Specifically, we test our proposal on "Side Effect Sokoban" and "Conveyor Belt" environments consider different categories of negative side effects that can occur in a dynamic environment. The state representation is a 5x5 matrix consisting of different numbers representing different sprites. At every time step, the agent can take actions (up, down, left, right). The interference behavior in both of the environments consists of moving an obstacle (by stepping the square containing the obstacle).
The Side Effect Sokoban environment consists of a vase that needs to be pushed out of the way for the agent reach the goal. The agent is provided a reward of -1 for every step takes and a final reward of +50 if it reaches the goal. The unsafe behavior is taking the shortest path to the goal, which involves pushing the vase down into a corner (an irrecoverable position). No negative reward provided to the agent when it does takes unsafe behavior. The desired behavior is to take slightly longer path by push the vase to the right. The action moving the vase is irreversible both cases: if the vase is moved to the right, the agent move it back, but then the agent ends up the other side of the vase. Thus, the agent receives the maximum penalty of I for moving the vase any direction, so the penalty does not incentives the agent choose the safe path.

---

Under review as a conference paper at ICLR 2020
Imaginative Module
B
(Safe Environment State)
baseline state in the future.
94
Agent can neach the bussa free scate placing
Agent reach the baseline perma nogative
path exists to the beseline state in the future.
original position
Figure 3: Illustration of approach the Side Effect Sokoban environment. The baseline state is chosen as the environment state in which the obstacle is in its original position, hence, representing safe state. Based on the imaginative module, safe action is the which allows the agent to reach the baseline state in the future.
We take the initial environment state (without the agent sprite) the baseline state, and consider as a safe state. An action ae considered to be "safe action" if there's at least one path from the next state (which it reaches after taking a₂) the baseline state in the imaginative module. Hence, whenever the policy takes an action ar, the imaginative module is used to check if the action is safe (i.e., if the agent can reach the initial environment state). Hence, the action which causes the vase to move into the corner is always avoided as if it was taken, it' be impossible to reach the baseline state. This action even avoided while policy is being learned.
The Conveyor Belt environment consists of conveyor belt that moves to the right by one square after every agent action. There is vase on the conveyor belt which will break if reaches the end of the belt. The agent receives reward of +50 for taking the vase off the belt. The desired behavior is to move the vase off and then stay put. However, it is likely that the agent moves the vase off the conveyor belt and then back on there's nothing avoiding it to do
In this case, we define the baseline state as the one which the vase broken after reaching the end of the belt and it represents an unsafe state. An action is considered to be an "unsafe action" if all the paths from the next state $141 lead to the specified baseline state in the imaginative module. Hence, the action which causes the vase to be moved back into the conveyor belt is always avoided. the action had been taken, would undoubtedly cause the vase to break.
5.3 AUTONOMOUS CAR ENVIRONMENT
To use the imaginative module for safe exploration, and test the practicality of our method using environments based on real-world tasks, we incorporate it policy which learss to drive car modified version of the Car Racing environment provided by OpenAl gym (Brockman al.) (2016)). The policy has to learn how to drive safely by never getting off the track. We discretize the existing continuous action space be able to create the imaginative module. The environment modified to that negative reward is provided to the agent when it gets off track. To ensure that the imaginative module doesn't deviate from the actual environment due to biases in the learned transition dynamics, create the graph using rollouts from the actual environment.
We define the set of states in which the car is off-track as the baseline state space B. An environment function tells whether the car is within the track not. An action is considered to be "unsafe action" " all the paths from the next state $1+1 lead to the specified baseline state the imaginative module. Due to the huge size of the state space, we create an imaginative module looking k-steps
7

---

Under review as a conference paper at ICLR 2020
Untrr Stlas Vaske
Mulic
Timmooaagee
Timaodaapa
(a) Side Effect Sokoban Environment
(h) Car Racing Environment
Figure 4: The Imaginative Module reduces visits to unsafe states. We plot the number of unsafe states visited by the agent in the respective environments, shown across training steps. Without any safety constraints, the agent visits high number of unsafe states. Incorporation of the imaginative module reduces the number of unsafe states visited even while training. The plot shows the average across 3 random seeds, with the dark line depicting an exponential moving average of the light line.
into the future, at every tthh time step. Hence, the action considered as unsafe if it causes car to go off track in 1-steps into the future, where / < k.
In Figure we plot how frequently the agent visits unsafe states throughout training. As compared to the baseline in which the agent doesn't I use imagination module for planning, observe that our method visits significantly fewer unsafe states. In the Side Effect Sokoban environment, the agent takes the unsafe and the shortest path if no implicit safety constraints agent to visit more unsafe states as the policy improves. In the Car Racing environment, we count the time steps in which the agent remains off track. The use of our approach causes the agent to never get off track, even if the policy is acting randomly.
provided. This causes the
CONCLUSION
This paper proposed the usage learned imaginative module, as and-play" approach to promote safe exploration and avoid negative side effects by the agent. Our method allows any policy, either pretrained or a one being learned act safely. We validate our approach by experimenting on two gridworld environments and car simulator environment. We do realize that incorporating this approach in real-world task with complex environments would require significant advancement and work in learning how model the environment, but we believe this first step could provide some direction to more work to be done in this area.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, pp. 1. ACM. 2004.
Joshua Achiam, David Held, Aviv Tamar, Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 22-31. 31. JMLR. org. 2017.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprini arXiv:1606.06555 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint (xX.x16666.0054. 2016.
Yinlam Chow, Ofir Nachum, Edgar
and Mohammad Ghavamzadeh. A lyapunov-basee approach to safe reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8092- 8101, 2018.
Paul F Christiano. Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4399-4307, 2017.
8

---

Under review as a conference paper at ICLR 2020
Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. UtL/ttts::/cppeaaii. raw.ya------rrd-ttttttt
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Picter Abheel. Model-based reinforcement learning via meta-policy optimization. arXiv preprint arXiv: /809:05214 2018.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa, Safe exploration in continuous action spaces. arXiv preprint arXiv: 08757, 2018.
Nathaniel D Daw, Yacl Niv, and Peter Dayan. Uncertainty based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature neuroscience, 8(12):1704, 2005.
Benjamin Eysenhach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint arXiv:1711.06788 2017.
Jaime Fisac, Anayo K Akametalu, Melanie N Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire Tomlin. A general safety framework for learning- baaedng-bbsedd control in uncertain robotic systems. IEEE Transactions on Automatic Control, 2018.
Jeremy И Gillula and Claire J Tomlin. Guaranteed safe online learning via reachability: tracking a ground target using quadrotor. In 2012 IEEE International Conference on Robotics and Automation, pp. 2723-2730. IEEE, 2012.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 1rXiv: 2016.
Dylan Hadfield-Menell, Stuart J Russell, Pieter Abheel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in neural information processing systems, pp. 3909-3917, 2016.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lec, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 1.04551, 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski. Sergey Levine, et al. Model- reinforcement learning for atari. arXiv preprint arXiv: 2tfv:/9003 1900.000044.... 2019.
Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,69800 2014.
Alexander Klyubin, Daniel Polani, and Chrystopher L Nehaniv. All else being equal be empowered. In European Conference on Artificial Life, pp. 744-753. 753. Springer, 2005.
Torsten Koller, Felix Berkenkamp, Mattco Turchetta, and Andreas Krause. Learning-bon bascdd model predictive control for safe exploration. In 2018 IEEE Conference on Decision and Control (CDC), pp. 6059-6066.
IEEE, 2018.
Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. Penalizing side effects using stepwise relative reachability. arXiv preprint 1806v:1806.0 0116, 2018.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbecl. Model-ensemble trust-region policy optimization. arXiv preprint 18X2 10592, 2018
Jan Leike, Miljan Martic, Victoria Krakovna. Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. Ai safety gridworlds. arXiv preprint 1rXiv:1711.0988 2017.
Teodor Mihai Moldovan Pieter Abbeel. Safe exploration markov decision processes. arXiv preprint arXiv: 2205.4810 2012.
Brad E Pfeiffer and David J Foster. Hippocampal place-cell sequences depict future paths rememhered goals. Nature, 497(7447):74, 2013.
Richard S Sutton, Andrew G Barto, al. Introduction reinforcement learning, volume 2. MIT press Cambridge, 1998.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.

---

Under review as a conference paper at ICLR 2020
Edward C' Tolman. Cognitive maps in rats and Psychological review, 55(4):189. 1948.
Alexander Matt Turner, Dylan Hadfield- Menelll, and Prasad Tadepalli. Conservative agency via attainable utility preservation. arXiv preprint
2019.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhem, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Théophane Weber, Sébastien Racanière. David P Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia. Oriol Vinyals. Nicolas Heess, Yujia Li, et al. Imaginatioon-augmented agents for deep reinforcement learning. arXiv preprimt (.X.v:1707.062100 2017.
Brian Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008.
10


Reviews:
[{"review_id": "rygzKGNTtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use learned transition models to do two separate things: (i) avoid unsafe states and (ii) allow an alternative channel for task reward specification. The idea is to create a comprehensive connectivity graph of the states in the environment. Once done, an agent can avoid unsafe states by avoiding states that are unconnected to a specified safe state. A practitioner might also specify safe/unsafe states as an additional source of information about the reward.\n\nThis paper suffers from poor and loose writing, incomplete specification of its experiments, unrealistic assumptions during evaluation (Sec 5.3 \"we create the graph using rollouts from the actual environment\" to avoid errors from learning a transition model). \n\nThe paper does not address basic concerns with its approach: how is the model to be learned at all, if it is to be comprehensive in the way that is necessary for the connectivity graph (which this paper calls an \"imaginative module\")? The authors say this is done through multiple agents performing random actions in the environment, in which case, isn't this extremely unsafe training time by the paper's own definition of safe exploration? \n\nFurther, creating a complete connectivity graph is unrealistic even for fully known transition models in most reasonably complex settings, such as, say, Go or Chess. \n\nIf the transition model is fully known as in the car racing setting, why not directly use that to plan and solve the game?\n\nExperiments show fewer \"unsafe\" states for the paper's approach compared to a method that has no way to know that those states are unsafe. How is this a reasonable validation, especially when the transition model is fully known? Also, this is an insufficient metric by itself as it says nothing about whether the method actually performed well at the task.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HkxTiwWTYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a model-based approach to safety in RL, where the agent uses a transition model to plan ahead to avoid actions that can lead it to unsafe states. They call the planning component an imaginative module. The agent takes the baseline state as input - that can be used to define either a safe or unsafe state, that is used in the planning component. The authors claim that using these two techniques they can tackle both the safe exploration (not violating safety constraints during learning) and irreversible side-effects (unintended irreversible behavior due to poorly designed reward-function).  They validate their approach on two grid world environments and self-driving car simulators.\n \nThis paper should be rejected because of the assumptions it makes goes against the very task they are trying to solve. In the sense, the task is trivial given the assumptions they have. \n\n1) The inconsistent assumption regarding the access to trajectories to learn a model. \nThe authors start with the assumption that the agent does not have access to the model (Sec 3) , and they explicitly learn the model. However, in the very next section (Sec 4), the authors assume that they can deploy a number of agents that interact with the environment randomly and collect that data to learn a complete transition model. Note that this assumption is wrong because:\nIf the random data agents are “safe”, i.e., don’t violate any safety constraint or cause any harmful behavior in the environment, then it is equivalent to assuming the agent having access to all the data to learn the model. This is a very big assumption that essentially says the agent has access to the model, which defeats the purpose of the safe-exploration problem. \nIf the random agents are “unsafe”, i.e., they can violate the safety constraint, then it goes against the very claim made about their method being able to respect the constraints throughout the learning process. \n\n2) The assumption about the baseline state(s).  \nThis is also a pretty big assumption to have, that is not acknowledged in the paper. If the agent already has the set of all the states it needs to avoid (or the set of states that are safe), then along with the assumption regarding access to the model, solving reversibility is significantly easier task then the general safe exploration problem [1, 2]\n\n3) The results reported in Figure 4 are not statistically significant. The experiments are only run over 3 random seeds [3] \n\n4) Can you give a few more details about the assumptions? In terms of how realistic they are or how essential they are to the method.\n\n\nThings to improve the paper that did not impact the score:\n- The negative side-effects problem that this work address is only based on reversibility criteria. \nClaim about learning the dynamics model is sample efficient is unsupported. \n\n\n\nReferences: \n[1] Berkenkamp, Felix, et al. \"Safe model-based reinforcement learning with stability guarantees.\" Advances in neural information processing systems. 2017.\n\n[2] Dalal, Gal, et al. \"Safe exploration in continuous action spaces.\" arXiv preprint arXiv:1801.08757 (2018).\n\n[3] Henderson, Peter, et al. \"Deep reinforcement learning that matters.\" Thirty-Second AAAI Conference on Artificial Intelligence. 2018.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkgkkpsmtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thie paper proposes using an \"imagination\" module to provide safe exploration during RL learning. The imagination module is used to perform forward predictions, constructing a graph between possible states. If any action would lead to a \"base state\" that is an unsafe state that action will not be executed and another \"safe\" action is selected from the policy.\n\nI have a number of comments and questions after reading the paper:\n- How do you get the forward model to be usably accurate? You do say that the model is a CNN model and is shared to learn the reward function as well. In the paper it says your method will lead to the agent never reaching an unsafe state, do you train the network in some way to make sure it does not make an inaccurate prediction around the unsafe state?\n- There is a lot of repetitive content in the paper that can be discarded to condense down the paper and make it more readable.\n- It would be nice to see more tasks or at least one that was more realistic... The tasks used in the paper appear to be common ones but they still feel rather artificial. Also, it seems in the paper there are only learning curves for 2 of the 3 tasks.\n- In the related work you say \"However, these methods are difficult to quantify and depend a lot on the diversity of settings in which they are performed.\" Can you expand on this? Why do they depend on these issues so much? The motivation for your method stems from these methods not being good enough, so further detail on this facet is important.\n- If you are using environments with discrete actions and performing prediction I am not sure if that can be called imaginative. Rather it should be called sampling. The forward model does not even appear to be stochastic.\n- The description of the imagination module training is not very clear. Is it trained on 5000 tuples or is the network trained for 5000 updates? There needs to be much more detail on this process.\n\nOverall the method seems interesting but does not appear to be a significant improvement.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of safe exploration in RL. The proposed approach uses an imaginative module to construct a connectivity graph between all states using forward predictions. The idea then consists in using this graph to plan a trajectory which avoids states labelled as "unsafe".

Several concerns were raised and the authors did not provide any rebuttal. A major point is that the assumption that the approach has access to what are unsafe states, which is either unreasonable in practice or makes the problem much simpler. Another major point is the uniform data collection about every state-action pairs. This can be really unsafe and defeats the purpose of safe exploration following this phase. These questions may be due to a miscomprehension, indicating that the paper should be clarified, as demanded by reviewers. Finally, the experiments would benefit from additional details in order to be correctly understood.

All reviewers agree that this paper should be rejected. Hence, I recommend reject.
Decision: Reject