Safety Aware Reinforcement Learning (SARL)

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms have seen great research advances in recent years , both in theory and in their applications to concrete engineering problems . The application of RL algorithms extends to computer games ( Mnih et al. , 2013 ; Silver et al. , 2017 ) , robotics ( Gu et al. , 2017 ) and recently real-world engineering problems , such as microgrid optimization ( Liu et al. , 2018 ) and hardware design ( Mirhoseini et al. , 2020 ) . As RL agents become increasingly prevalent in complex real-world applications , the notion of safety becomes increasingly important . Thus , safety related research in RL has also seen a significant surge in recent years ( Zhang et al. , 2020 ; Brown et al. , 2020 ; Mell et al. , 2019 ; Cheng et al . ; Rahaman et al . ) . 1.1 SIDE EFFECTS IN REINFORCEMENT LEARNING ENVIRONMENTS . Our work focuses specifically on the problem of side effects , identified as a key topic in the area of safety in AI by Amodei et al . ( 2016 ) . Here , an agent ’ s actions to perform a task in its environment may cause undesired , and sometimes irreversible , changes in the environment . A major issue with measuring and investigating side effects is that it is challenging to define an appropriate sideeffect metric , especially in a general fashion that can apply to many settings . The difficulty of quantifying side effects distinguishes this problem from safe exploration and traditional motion planning approaches that focus primarily on avoiding obstacles or a clearly defined failure state ( Amodei et al. , 2016 ; Zhu et al. , 2020 ) . As such , when learning a task in an unknown environment with complex dynamics , it is challenging to formulate an appropriate environment framework to jointly encapsulate the primary task and side effect problem . Previous work on formulating a more precise definition of side effects includes work by Turner et al . ( 2019 ) on conservative utility preservation and by Krakovna et al . ( 2018 ) on relative reachability . These works investigated more abstract notions of measuring side effects based on an analysis of changes , reversible and irreversible , in the state space itself . While those works have made great progress on advancing towards a greater understanding of side effects , they have generally been limited to simple grid world environments where the RL problem can often be solved in a tabular way and value function estimations are often not prohibitively demanding . Our work focuses on expanding the concept of side effects to more complex environments , generated by the SafeLife suite ( Wainwright and Eckersley , 2020 ) , which provides more complex environment dynamics and tasks that can not be solved in a tabular fashion . Turner et al . ( 2020 ) recently extended their approach to environments in the SafeLife suite , suggesting that attainable utility preservation can be used as an alternative to the SafeLife side metric described in Wainwright and Eckersley ( 2020 ) and Section 2 . The primary differentiating feature of SARL is that it is metric agnostic , for both the reward and side effect measure , making it orthogonal and complimentary to the work by Turner et al . ( 2020 ) . In this paper , we make the following contributions which , to the best of our knowledge , are novel additions to the growing field of research in RL safety : • SARL : a flexible , metric agnostic RL framework that can modulate the actions of a trained RL agent to trade off between task performance and a safety objective . We utilize the distance between the action probability distributions of two policies as a form of regularization during training . • A generalizeable notion of safety that allows us to train a safe agent independent of specific tasks in an environment and port it across multiple complex tasks in that environment . We provide a description of the SafeLife suite in Section 2 , a detailed description of our method in Section 3 , our experiments and results for various environments in Section 4 and Section 5 respectively , as well as a discussion in Section 6 . 2 THE SAFELIFE ENVIRONMENT . The SafeLife suite ( Wainwright and Eckersley , 2020 ) creates complex environments of systems of cellular automata based on a set of rules from Conway ’ s Game of Life ( Gardner , 1970 ) that govern the interactions between , and the state ( alive or dead ) of , different cells : • any dead cell with exactly three living neighbors becomes alive ; • any live cell with less than two or more than three neighbors dies ( as if by under- or overpopulation ) ; and • every other cell retains its prior state . In addition to the basic rules , SafeLife enables the creation of complex , procedurally generated environments through special cells , such as a spawner , that can create new cells and dynamically generated patterns . The agent can generally perform three tasks : navigation , prune and append which are illustrated in Figure 1 taken from Wainwright and Eckersley ( 2020 ) . The flexibility of SafeLife enables the creation of still environments , where the cell patterns do not change over time without agent interference , and dynamic environments , where the cell patterns do change over time without agent interference . The dynamic environments create an additional layer of difficulty , as the agent now needs to learn to distinguish between variations in the environment that are triggered by its own actions versus those that are caused by the dynamic rules independent of its actions . As described in Section 4 , our experiments focus on the prune and append tasks in still and dynamic environments : prune-still , prune-dynamic , append-still , append-dynamic . 2.1 SAFELIFE SIDE EFFECT METRIC . There are two separate side effect metrics that we use in this paper : a training-time side effect metric that can easily be calculated at every frame , and a separate end-of-episode side effect metric used to validate overall agent safety . These metrics are identical to the side effect metrics described in Wainwright and Eckersley ( 2020 ) . The side effect metric used in training is found by comparing any given state to the starting state of that episode . Every cell in the grid that is different from the corresponding cell in the starting state ( excluding differences due to agent movement or goal completion ) is marked as having a side effect of +1 . The total side effect of a state is the sum of side effects of individual cells . The difference in side effects between subsequent states is then to train safe agents as described in Section 3 . At the end of each episode , we perform a separate side effect calculation that is more robust to stochastic and chaotic dynamics observed in SafeLife . We prepare a counterfactual inaction baseline state at time tend , defined as the state that would have been achieved had the agent followed an inaction policy for tend time steps . We then evolve both the primary state and the inaction state for an additional n = 1000 time steps and store the average occupancy of green life cells at each point in the grid , producing two distributions . The total episodic side effect is then given the earth-mover distance between the two distributions using a modified Manhattan distance metric as described in Wainwright and Eckersley ( 2020 ) . The normalized episodic side effect metric , shown in the figures in Section 5 , is defined as the total episodic side effect divided by the number of green cells present at the start of the episode , showing the percentage of structures in each episode that are disrupted . 3 METHOD . 3.1 TRAINING FOR REGULARIZED SAFE RL AGENT . Our method relies on regularizing the loss function of the RL agent with the distance of the task agent , Aθ , from the virtual safe actor , Zψ , as shown in Figure 2 . More formally , the general objective of the task agent Aθ can be expressed as : FA ( θ ) = Lθ + β ∗ Ldist ( Pπθ , Pπψ ) ( 1 ) where β is a regularization hyperparameter , Pπθ represents the probability of taking a given action given by Aθ , and Pπψ represents the probability distribution of taking a given action according to Zψ . As shown in Equation 1 , the actor loss Lθ is regularized by the distance between the actions suggested by the task agent and the virtual safe agent . The gradient of the objective in Equation 1 expressed as the expectation of rewards of task agent actions α taken from a distribution of policies Pπθ is then given by : ∇θFA ( θ ) = ∇θEα∼Pπθ [ L ( θ ) ] + β ∗ ∇θEα∼Pπθ , ζ∼Pπψ [ Ldist ( α , ζ ) ] ( 2 ) where ∇θ is independent from the virtual safe agent actions ζ given that Zψ is only dependent on ψ . This formulation enables training Zψ independently from Aθ , thereby abstracting the notion of safety away from the task . The gradient formulation underscores the importance for a distance metric Ldist that is differentiable to ensure that gradients update the task agent parameters θ from both terms of the augmented loss functions . 3.2 DISTANCE METRICS FOR LOSS REGULARIZATION . The primary objective of the regularization term is to express a notion of distance between a purely reward based action and a purely safety motivated action , thereby penalizing Aθ for taking a purely reward motivated action . We model the regularization term as the distance between probability distributions Pπθ and Pπψ . Given this formulation , previous work ( Nowozin et al. , 2016 ; Arjovsky et al. , 2017 ; Huszár , 2015 ) has provided a number of choices for distance metrics in supervised learning problems with various advantages or shortfalls . One common method of measuring the difference in probability distributions is the KL Divergence , DKL ( p‖q ) = ∫ x p ( x ) log p ( x ) q ( x ) dx , where p and q are probability distributions described by probability density functions . The KL Divergence , however , has some significant disadvantages – the most significant one being that the KL Divergence is unbounded when probability density functions to express the underlying distribution can not be easily described by the model manifold ( Arjovsky et al. , 2017 ) . Furthermore , the KL divergence is not symmetric given that DKL ( p‖q ) 6= DKL ( q‖p ) , and also does not satisfy the triangle inequality . One alternative to the KL Divergence is the Jensen-Shannon distanceDJS ( p‖q ) = 1 2DKL ( p‖m ) + 1 2DKL ( q‖m ) withm = 1 2 ( p+q ) , which is symmetric , satisfies the triangle inequality and is bounded : 0 ≤ DJS ≤ log ( 2 ) . These advantages make DJS a good choice for the SARL algorithm , but as discussed extensively in Arjovsky et al . ( 2017 ) , DJS also has notable disadvantages , the most important being that DJS is not guaranteed to always be continuous and differentiable in low manifold settings . Another alternative to DJS is the Wasserstein Distance . As discussed in Arjovsky et al . ( 2017 ) , the Wasserstein Distance is generally better suited for calculating distances for low-dimensional manifolds compared to DJS and other variants of the KL divergence . In its analytical form the Wasserstein Distance W ( p , q ) = ( infJ∈J ( p , q ) ∫ ‖x − y‖pdJ ( x , y ) ) 1 p , however , is intractable to compute in most cases leading many researchers to establish approximations of the metric . A common way of approximating the Wasserstein Distance is to re-formulate the calculation as an optimal transport problem of moving probability mass from p to q , as shown in Cuturi ( 2013 ) and Pacchiano et al . ( 2019 ) . The dual formulation based on behavior embedding maps of policy characteristics described in Pacchiano et al . ( 2019 ) is particularly applicable for the SARL algorithm , leading us to adapt it as an additional alternative to the Jensen-Shannon Distance . In this formulation , policy characteristics are converted to distributions in a latent space of behavioral embeddings on which the Wasserstein Distance is then computed . For our experiments in Section 4 , we apply both DJS and the dual formulation of the Wasserstein Distance described in Pacchiano et al . ( 2019 ) to compute the distance between Pπθ and Pπψ .

Summary: The paper aims to reduce the unwanted side effects of the actions of a reward-maximizing reinforcement-learning (RL) agent. The authors study a framework in which the environment issues a metric that measures the total side effects of the agent's actions at the end of each episode. The work's proposed solution trains an agent who focuses on the total reward and another agent who minimizes the total side effects. The authors then empirically investigate the effectiveness of combining the two agents via a distance measure between the policies that unifies them into one.