Safety Aware Reinforcement Learning (SARL)
Authors: Santiago Miret
Somdeb Majumdar
Carroll Wainwright
Venue:  (2021)
Keywords: Reinforcement Learning
Safe RL
Probabilistic Distance Metrics

Abstract: As reinforcement learning agents become increasingly integrated into complex, real-world environments, designing for safety becomes a critical consideration. We specifically focus on researching scenarios where agents can cause undesired side effects while executing a policy on a primary task. Since one can define multiple tasks for a given environment dynamics, there are two important challenges. First, we need to abstract the concept of safety that applies broadly to that environment independent of the specific task being executed. Second, we need a mechanism for the abstracted notion of safety to modulate the actions of agents executing different policies to minimize their side-effects. In this work, we propose Safety Aware Reinforcement Learning (SARL) - a framework where a virtual safe agent modulates the actions of a main reward-based agent to minimize side effects. The safe agent learns a task-independent notion of safety for a given environment. The main agent is then trained with a regularization loss given by the distance between the native action probabilities of the two agents. Since the safe agent effectively abstracts a task-independent notion of safety via its action probabilities, it can be ported to modulate multiple policies solving different tasks within the given environment without further training. We contrast this with solutions that rely on task-specific regularization metrics and test our framework on the SafeLife Suite, based on Conway's Game of Life, comprising a number of complex tasks in dynamic environments. We show that our solution is able to match the performance of solutions that rely on task-specific side-effect penalties on both the primary and safety objectives while additionally providing the benefit of generalizability and portability. 

Under review as a conference paper at ICLR 2021
SAFETY AWARE REINFORCEMENT LEARNING (SARL)
Anonymous authors
Paper under double-bling blind review
ABSTRACT
As reinforcement learning agents become increasingly integrated into complex, real-world environments, designing for safety becomes critical consideration. We specifically focus on researching scenarios where agents can cause undesired side effects while executing policy a primary task. Since one can define multiple tasks for given environment dynamics, there are two important challenges. First, we need abstract the concept of safety that applies broadly to that environment independent of the specific task being executed. Second, we need a mechanism for the abstracted notion of safety to modulate the actions of agents executing different policies to minimize their side-effects. In this work, we propose Safety Aware Reinforcement Learning (SARL) framework where a virtual safe agent modulates the actions of a main wward-based task agent to minimize side effects. The safe agent learns a fask-independent notion safery for a given environment. The task agent is then trained with regularization loss given by the distance between the native action probabilities of the two agents. Since the safe agent effectively abstracts task-independent notion of safety via its action probabilities, it can be ported to modulate multiple policies solving different tasks across different environments without further training. We contrast this with solutions that rely on task-specific regularization metrics and test our framework on the SafeLife Suite, based on Conway's 's Game of Life, comprising a number of complex tasks dynamic environments. We show that our solution is able to match the performance of solutions that rely on task -psc-iic side-effect penalties on both the primary and safety objectives while additionally providing the benefit of generalizability and portability.
INTRODUCTION
Reinforcement learning (RL) algorithms have great research advances in recent years, both in theory and in their applications to concrete engineering problems. The application of RI. algorithms extends computer games (Mnih et al. 2013) Silver et al. (017), robotics (Gu et al. 2017) and
recently real-world engineering problems, such as microgrid optimization (Liu et al. 2018) and hardware design (Mirhoseini al. 2020). As RL agents become increasingly prevalent complex real-world applications, the notion of safety becomes increasingly important. Thus, safety related
research in RL has also seen a significant surge recent years (Zhang et al. 2020 Brown al.
2020
Mell et al. 2019 Cheng et al. Rahaman et al.).
1.1
SIDE EFFECTS IN REINFORCEMENT LEARNING ENVIRONMENTS
Our work focuses specifically on the problem of side effects, identified as a key topic in the area of safety in Al by Amodei et al. (2016). Here, an agent's 's actions to perform a task in its environment may cause undesired, and sometimes irreversible, changes in the environment. with measuring and investigating side effects is that it is challenging to define an appropriate side- effect metric, especially in a general fashion that can apply to many settings. The difficulty of quantifying side effects distinguishes this problem from safe exploration and traditional motion planning approaches that focus primarily on avoiding obstacles or a clearly defined failure state (Amodei et al.) 2016 Zhu et al.) 2020). As such, when learning task in an unknown environment with complex dynamics, it is challenging to formulate an appropriate environment framework to jointly encapsulate the primary task and side effect problem.
major issue

---

Under review as a conference paper at ICLR 2021
Previous work on formulating more precise definition of side effects includes work by Turner et al. (2019) on conservative utility preservation and by Krakovna et al. (2018) on relative reachability. These works investigated more abstract notions of measuring side effects based changes, reversible and irreversible, in the state space itself. While those works have made great progress advancing towards greater understanding of side effects, they have generally been limited to simple grid world environments where the RL problem can often be solved in a tabular way and value function estimations are often not prohibitively demanding. Our work focuses on expanding the concept of side effects to more complex environments, generated by the SifeLife suite (Wainwright and Eckersley 2020), which provides more complex environment dynamics and tasks that cannot be solved in a tabular fashion. Turner et al. (2020) recently extended their approach to environments in the SafeLife suite, suggesting that attainable utility preservation can be used as an alternative to the Safel .ife side metric described in Wainwright and Eckersley (2020) and Section 2 The primary differentiating feature of SARL is that it is metric agnostic, for both the reward and side effect measure, making it orthogonal and complimentary to the work by Turner al. (2020).
analysis of
In this paper, we make the following contributions which, to the best of our knowledge, are novel additions to the growing field of research in RL safety:
SARL: flexible, metric agnostic RL framework that can modulate the actions of a trained RL agent to trade off between task performance and safety objective. We utilize the distance between the action probability distributions two policies as a form of regularization during training.
A generalizeable notion of safety that allows us to train a safe agent independent of specific tasks an environment and port it across multiple complex tasks in that environment.
We provide description of the SafeLife suite in Section 2 a detailed description of our method in Section 3 our experiments and results for various environments Section and Section respectively, as well as discussion in Section
2 THE SAFELIFE ENVIRONMENT
The Sff suite Wainwright and Eckersley 2020) creates complex environments of systems of cellular automata based set of rules from Conway's Game of Life (Gardner 1970) that govern the interactions between, and the state (alive or dead) of, different cells:
any dead cell with exactly three living neighbors becomes alive; any live cell with less than two more than three neighbors dies (as if by under- or overpopulation); and
every other cell retains its prior state.
In addition to the basic rules, SafeLife enables the creation of complex, procedurally generated environments through special cells, such spawner, that create new cells and dynamically generated patterns. The agent can generally perform three tasks: navigation, prune and append which are illustrated in Figure taken from Wainwright and Eckersley (2020).
The flexibility of SafeLife enables the creation of still environments, where the cell patterns do not change over time without agent interference, and dynamic environments, where the cell patterns do change over time without agent interference. The dynamic environments create an additional layer of difficulty, as the agent now needs learn to distinguish between variations in the environment that are triggered by its own actions versus those that are caused by the dynamic rules independent of its actions. As described in Section 4 our experiments focus on the prune and append tasks in still and dynamic environments: prune-still, ptapssiillllprreee prune-dynaiic, append-still, append-dynanic.
2.1 SAFELIFE SIDE EFFECT METRIC
There are two separate side effect metrics that we use in this paper: training- time side effect metric that can easily be calculated every frame, and separate end-of -fd-os-episode side effect metric used to validate overall agent safety. These metrics are identical to the side effect metrics described in Wainwright and Eckersley (2020).
2

---

Under review as a conference paper at ICLR 2021
Figure 1: A simple level of the SafeLife environment containing an agent (D), spawner (#)) crates (0), and cells of life. The agent's goal to remove unwanted red cells (prune task) and to create new patterns of life in the blue squares (append task). Once the agent has satisfactorily completed its goals it can leave via the level exit (B). Note that all level boundaries wrap; they have toroidal topology.
The side effect metric used in training is found by comparing any given state to the starting state of that episode. Every cell in the grid that is different from the corresponding cell in the starting state (excluding differences due to agent movement or goal completion) is marked as having side effect of +1. The total side effect of a state the sum side effects of individual cells. The difference in side effects between subsequent states is then to train safe agents as described in Section
At the end of each episode, we perform a separate side effect calculation that more robust to stochastic and chaotic dynamics observed in SafeLife. We prepare counterfactual inaction baseline state at time teud, defined as the state that would have been achieved had the agent followed an inaction policy for tend time steps. We then evolve both the primary state and the inaction state for an additional n = 1000 time steps and store the average occupancy of green life cells at each point in the grid, producing two distributions. The total episodic side effect is then given the earth-mover distance between the two distributions using a modified Manhattan distance metric as described in Wainwright and Eckersley (2020). The normalized episodic side effect metric, shown in the figures in Section [] is defined as the total episodic side effect divided by the number of green cells present the start of the episode, showing the percentage of structures in each episode that are disrupted.
METHOD
3.1 TRAINING FOR REGULARIZED SAFE RL AGENT
Our method relies on regularizing the loss function of the RL agent with the distance of the task agent, Ag. from the virtual safe actor, Zμo shown in Figure
More formally, the general objective the task agent Ag can be expressed as:
FA) = La + 3 Laist (Pr,, Pr.)
where B is regularization hyperparameter, Pr,,, represents the probability of taking a given action given by Ag. and Px,, represents the probability distribution of taking given action according to Ze. As shown in Equation [ the actor loss Le is regularized by the distance between the actions suggested by the task agent and the virtual safe agent. The gradient of the objective in Equation I expressed as the expectation of rewards of task agent actions taken from a distribution of policies Pr, is then given by:
V (0)= = Volo-T [C(0] + 3 V Vollar CMPPy [dist (ae, C)]
(2)
where Va is independent from the virtual safe agent actions given that Zy is only dependent th. This formulation enables training Z independently from Ag, thereby abstracting the notion of safety away from the task. The gradient formulation underscores the importance for distance metric Caist that is differentiable to ensure that gradients update the task agent parameters from both terms of the augmented loss functions.

---

Under review as a conference paper at ICLR 2021
SARL Agent Training
Backpropagation
Task Action
Task Loss
Probabilities
Task Agent
Total Loss
Environment Rollout
Distribution Loss
Virtual Safe Agent
Safe Action Probabilities
Figure 2: A co-training framework for safety aware RL training. The task agent, Ag. determines the action taken in the environment and the resulting trajectories. The virtual safe agent, Zao receives the same state as As and makes a suggestion for safe action given the state. The distance between the action probabilities of the Ag and Zy is captured in the Distribution Loss, Cdist. Zy learns how to maximize the safety objective on its own set of environments in parallel to Ag.
3.2 DISTANCE METRICS FOR LOSs REGULARIZATION
The primary objective of the regularization term is to express a notion of distance between purely reward based action and a purely safety motivated action, thereby penalizing Ag for taking a purely reward motivated action. We model the regularization term the distance between probability distributions Fr, and T
Given this formulation, previous work (Nowozin et al. 2016 Arjovsky et al. 2017 Huszár 2015)
has provided number of choices for distance metrics in supervised learning problems with various advantages shortfalls. One common method of measuring the difference in probability distributions
the KI. Divergence, (xl(p)l)) = ((x) log p(a)) da. where p and q are probability distributions described by probability density functions.
g(x)
The KL Divergence, however, has some significant disadvantages the most significant one being that the KL Divergence is unbounded when probability density functions to express the underlying distribution cannot be easily described by the model manifold (Arjovsky et al. 2017). Furthermore, the KI. divergence is not symmetric given that D (p F D (qllp), and also does not satisfy the triangle inequality. One alternative to the KL Divergenc is the Jensen-Shannon distance Js (pll4) with m (p+44) which is symmetric, satisfies the triangle inequality and is bounded: 0 < DJs < log(2). These advantages make DJs good choice for the SARL algorithm, but as discussed extensively in Arrovsky et al. (2017), DJs also has notable disadvantaged the most important being that D,s is not guaranteed to always continuous and differentiable in low manifold settings.
_ xz (pllm) ( (gI(gll))
Another alternative to DJs the Wasserstein Distance. As discussed in Arjovsky et al. (017)) the Wasserstein Distance is generally better suited for calculating distances for low-dimension manifolds compared to DJs and other variants of the KL divergence. In its analytical form the Wasserstein Distance (p. = (inf (EF(g))) f||æ (x, vz) however, is intractable to compute in most cases leading many researchers to establish approximations of the metric. common way of optimal transport problem of moving probability mass from p to q. as shown in Cuturi (0013) and Pacchiano et al. (2019). The dual formulation based on behavior embedding maps of policy characteristics described in Pacchiano et al. (2019) is particularly applicable for the SARL algorithm, leading us to adapt it as an additional alternative to the Jensen-Shane -hasnnn Distance. In this formulation, policy characteristics are converted to distributions in a latent space of behavioral embeddings on which the Wasserstein Distance is then computed.
approximating the Wasserstein Distance is to re-formulate the calculation
4

---

Under review as a conference paper at ICLR 2021
For our experiments in Section 4 we apply both DJs and the dual formulation of the Wasserstein Distance described in Pacchiano et al. (019) to compute the distance between PT,, and Pmy.
3.3 SAFETY AWARE REINFORCEMENT LEARNING
The paragraphs above in Section describe the individual components of SARL. The experiments outlined in Section discuss SARL applied to Proximal Policy Optimization (PPO) (Schulman et al. 2017). Wainwright and Eckersley (2020) applied PPO for solving the different environments in the SafeLife suite, making SARL-PPO a natural extension. The loss formulation L3 used in
Algorithm T is the same the decribed in SShhulmmnn al. (0177):
PP E, (0) L. alve (0) c2S(Ta](8e))
As shown in more detail Algorithm Ag is trained using the regularized loss objective described in Equation while Zy is trained exclusively LPP0 using the frame-by- by-rame side effect information as the reward.
Algorithm SARL PPO
1: Initialize an actor As(s) and virtual safety agent (x) where 8 represents the state observation
2: Initialize distance metric Ldist
3: while training SARL-PPO do
4:
for each actor update Ap do
5: Run Ao to generate minibatch of transitions a with task rewards Ttask
6: Run Z (s) to generate minibatch of transitions Ç
7:
Compute 23 PPP and Pr,, using transitions in a
8: Compute P गे. using transitions in
9: Optimize Ag using LPPO + B Ldist (1 PPx)
10: end for
11:
for each virtual agent update Zy do
12:
Run Zo (8) to generate minibatch of transitions ( with side-effect metric -fide-offoct
13:
Compute C. using transitions in
14:
Optimize Zy using LPPO with effect as the reward
15:
end for
end while
The training algorithm agnostic to the side effect metric, leading itself plug-and-play approach where the virtual safe agent modulate the task agent for a variety of different environment specific side effect metrics without major modification to the overall structure of the method.
used in the environment,
In addition to training both Aa and Zy: from scratch as shown in Algorithm we also perform zero- previously trained Zg. from different environment investigate whether the concept of side effects can be abstracted out of the environmental dynamics and the intricacies of the task. this case, lines 11-15 from Algorithm| [ are not performed as no updates for Z are required, with Z only being used to modulate the behavior of Ag via the distance metric regularization.
shot generalization
3.4 TRACKING THE CHAMPION POLICY
The SafeLife suite includes complex set procedurally generated environments, which can lead to significant amount of variability throughout training and testing episodes. In order to account for this variability, we track the best policy throughout the training process for a fixed set test levels for the different metrics we care about, specifically episode length, performance ratio and side effects, as described in Algorithm
The champion policies operate on test levels, where no learning occurs, and track test-level metrics. This is particularly relevant to the side effect metric, described in Section 2] where track the episodic side effect even though training occurs with frame-by- ba-e-e---frmeee impact measure. We describe these metrics in greater detail in Section
5

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Champion Policy Tracking
1: Initialize Training and Champion Policy Co
2: for Every k Environment Steps do
3: Evaluate task agent A on fixed set of test levels
4: if Score A > Scorec, then
5: CØ A
6: end
7: end for
4 EXPERIMENTS
Our experiments contain the following algorithmic runs, where log the results for Ag:
The reward -ennatyy baseline method described in (Wainwright and Eckersley 2020) where the impact penalty of given action is subtracted from the reward the agent receives for that particular frame.
SARL agents where both the actor Ag and virtual safety agent Z are training from scratch on the same environment using the Jensen-Shannen Distance as well as the dual formulation the Wasserstein Distance described in (Pacchiano et al.) 2019)
SARL agents where Ag is trained while Z is taken zero-shot from previous training run on a different environment. The main purpose of this experiment to show that concept of side-effects in the SafeLife suite can be abstracted from the specific task (prune vs append) and the specific environment setting (still vs dynamic). The ability to extract a notion of side effects that does not rely environmental signal for every frame enables us to train the virtual safety agent Zu only once, usually on the simplest task, which can then be used to influence any agent on any subsequent task.
We conduct our experiments on four different tasks in the SafeLife suite: prune-still, append-still, prune- dynamic, append-dyname
As described in Section [ dynamic environments have natural variation independent of the actions of the agents, while all changes in still environments can be traced back to the actions of the agent. We evaluate our champion policies Cp for the task agent every 100,000 environment steps the episode length across set of 100 different testing environments whose configurations are not part of the configurations of environments used in the training process. The length of an episode is the number of steps the agent takes to complete episode, where shorter length indicates that the agent can solve the task better and more efficiently. our results in Section 5 we show the standard error of the champion measured in performance given by the ratio
Agent reward and the cumulative side effect measure described in Section 2 and Wainwright and
possible reward
Eckersley (2020).
The experimental results have a strong dependency on hyperparameters chosen, specifically the impact penalty fraction in the reward penalty baseline baseline and the regularization parameter B in SARL. Changing these parameters generally results in non-linear trade-offs between episode length, performance and side effects, meaning that policies with high performance often have high side effects and policies with low side effects often have low performance. In the cases of low side effects and low performance, the agent does not perform any significant actions that would either negatively (side effect) or positively (reward) disturb the environment. Our ideal goal is to have a policy that both performant on the task and has low side effects. As such, in Section we describe results of experiments that in our best judgement represent the best cases of such policies, and apply the same regularization hyperparameters across environments. The full set of our algorithmic and shown in Appendix [A] As discussed in more detail in Section 6 for future work we aim, and encourage others, to obtain Pareto optimal frontiers that describe the tff for the regularization hyperparameters more thoroughly.
regularization hyperparameters

---

Under review as a conference paper at ICLR 2021
SARL J5
Average Performance
0.9
Average Side Effects
0.6
Average Length
SARL WD
Baseline
600
Zera-Shot SARL Js
Zero-Sin Shot SARL WD
400
0.6
200
Training Samples (x10*)
Training Samples (x109)
Training Samples (x10
(a) Prune-Still Length
(b) Prume-Still Performance
(c) Prune-Still Side Effects
1000
dierage Side Effects
2,00
Average Performance
Average Length
980
1.75
900
1:25
940
0.2
1:00
15075
0.50
0.0 02 044
10 1.2 1.6
Training Samples (x10%)
Training Samples (x10%)
Training Samples (x10%)
(d) Append-Still Length
(c)
Append-Still Performance
(f) Append-Still Side Effects
Average Performance
0.85
Average Side Effects
1.1
Average Length
0.80
aoo
0.75
1.0
200
0.70
600
0.65
500
0.60
5.555
0.7
400
0.50
300
2.6
0.45
200
0.0 0.2 644
06
08 1.0 122 14 1.6
0. 02
04 05 08 10 12 14 15
0.0
0.5
10
1.5
Training Samples (x10*)
Training Samples (x10%)
Training Samples (x10*)
(g) Prume-Dynamic Length
(h) Prune-
Prune-Dynamic Dynamic Performance
(i) Prune-Dynamic Side Effects
Average Performance
85
Owerage Side Effects
2.00
Average Length
800
1.75
800
0.3
1.25
1.00
600
0.50
500
0.0
0.0 02 04 06 08
10 12 1.4
1.6
0.0
0.2
0.4
0.5
0.8 10 122 144 166
0.5
0.5
1.5
Training Samples (x10*)
Training Samples (x10*)
Training Samples (x10²)
Dopend-Dynamic Length
(k)
Append-Dynamic Performance
(1) Append-Dynamie Side Effects
Figure 3: Length Champion for SafeLife Suite of J. prune-still (a-c), 2. append-still (d-f), 3. prune- dynamic (g-i), 4.
(j-l) tasks evaluated for 100 testing environments every 100,000 steps Episode Length (left column) where shorter better, Performance Ratio (middle column) where higher is better, and Episodic Side Effect (right column) where lower is better
RESULTS
The results of the experiments shown in F demonstrate FegureBdemonstrate one task in the SARL framework can generalize zero-shot to other tasks and environment settings in the SafeLife suite, while maintaining competitive task and side effect scores compared to the baseline method. This allows us to abstract the notion of safety away from the environment specific side effect metric. and also increase the overall sample efficiency of the SARL method for subsequent training runs. The SARL methods that are trained from seratch also show competitive task and side effect scores compared to the baseline method.
that a virtual safety agent trained on
7

---

Under review as a conference paper at ICLR 2021
In the still environment we chose to generalize the virtual safety agents from prume-still and append- still to the other tasks using both distance metrics. The results show that zero-shot generalization of Zg matches the behavior of SARL trained from scratch, as well as matching outperforming the baseline method on episode length and performance.
Prune-Still Environment (Top Horizontal): The reward penalty baseline matches the episode length of all other methods while maintaining a slightly lower performance and side effects. All SARL methods, including both metrics and zero-shot SARL, generally perform equally well on length and side effects while the SARLd,s has a slightly better performance than SARLDwd
Append-Still Environment (2nd Horizontal): The reward penalty baseline generally matches the performance and side effects of the SARL methods, while slightly underperforming SARLD,s on episode length. SARLD,s generally performs better on episode length compared the other methods, both in training from scratch and the zero-shot experiments.
For the dynamic environment we chose to generalize the virtual safety agents trained on prune-stil and append-still to the dynamic environment tasks using both distance metrics. In the zero-shot experiments, we applied the version of Z that is furthest away from the given setting, meaning prune-still is generalized to appenl-dynamic and append-still is generalized to prane-dynumic. The results show that zero-shot generalization of Z. matches the behavior of SARL trained from scratch, as well matching or outperforming the baseline method some metrics.
Prune-Dynamic Environment (3rd Horizontal): In this environment, we observe that the baseline method cannot solve the task, as shown by the fact that the episode length does not decrease significantly. However, it incurs very little side-effect cost. This indicates that the baseline agent is acting safer by not doing much in the environment, but actually fails solve the primary task. All SARL agents outperform the baseline on episode length and performance ratio, indicating that SARL effectively learns the task.
Append-Dynan Environment (Bottom Horizontal): The reward penalty baseline generally matches the behavior of the zero-shot SARL methods on episode length, performance and side effects. The SARL method trained from scratch, both SARLD,s and SARLDwW outperform the baseline as well as the zero-shot experiments on episode length and slightly on performance
DISCUSSION
In this work, we explored the prospect of regularizing the loss function of an RL agent using distance metrics that encapsulate notion of safe behavior. We believe this work shows promising approach to train RL agents in environments where side effects are important. Our method allows safe policy to be trained once and transferred across tasks within the same environment. This may be especially important when the safety metric itself is difficult to calculate- for example, if it requires human feedback. As mentioned in Section | side effects are often difficult to define, especially when interwoven with the primary task. As such measuring and interpreting side effects is an ongoing area of research, which is why we designed SARL to be flexible to various side effect metries.
The idea of using suitable distance metrics to co-train multiple RL agents has a variety of future research directions. One such avenue is the development of new distance metrics, including different variations of the Wasserstein Distance, as well as ones that can exploit various channels of information that we did not consider in out work SParker-Holder et al., 2020). The ideal distance metric would capture both the information richness from the different channels and encode a notion of a safety objective which can then be transferred to the primary agent to influence its behavior.
There also exists a great opportunity to apply techniques from multi-objective optimization to side effect problems. The literature is rich with multi-objeccive optimization problems in supervised learning (Ma et al. (020) (Sener) 2018) and reinforcement learning (Yang et al., 2019) (Xu et al. 2020) that show promising approaches to adopt robust multi-objectivv framework the side effect problem. The greatest promise of a multi-objective framework is the possibility of obtaining Pareto fronts (Yang et al., 2019) deseribing the optimal trade-off between task performance and safety in a given environment, which would be immensely valuable to making decisions in real-world environments.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Manè. Concrete problems in ai safety, arXiv preprint arXiv: 1606. 06565, 2016.
M. Arjovsky, Chintala, and Bottou. Wasserstein GAN. 2017. URL http://arxiv..rgg abs/1701 07875
D. S. Brown, R. Coleman, R. Srinivasan, and S. Niekum. Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences. 2020.
R. Cheng, R. M. Murray, and J. W. Burdick. Safety-Critical Continuous Control Tasks.
M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems, pages 1-9, 2013. ISSN 10495258.
B. Eysenbach, A. Gupta, J. Ibarz, and Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint arXiv:1802.06600 2018.
M. Gardner. The fantastic combinations of jhon conway's new solitaire game life. Sc. Am., 223: 20-123, 1970.
Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.
Huszár. How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? pages 1-9, 2015. /tL/tttp//aaaxw...rr///////LLL1L
V. Krakovna. L. Orseau, R. Kumar, M. Martic, and Legg. Penalizing side effects using stepwise relative reachability. arXiv preprint arXiv:1806 01 186, 2018.
W. Liu, P. Zhuang, H. Liang. J. Peng, and Z. Huang. Distributed economic dispatch in microgrids based on cooperative reinforcement learning. IEEE transuctions on neural networks and learning systems, 2916:2212,, 2018.
P. Ma, T. Du, and W. Matusik. Efficient Continuous Pareto Exploration in Multi-Task Learning. 2020. URL
wtw.:///..........o.////o 16434
S. Mell, O. Brown, J. Goodwin, and S.-h. Son. Safe Predictors for Enforcing Input-Output Specifica- tions. pages 2-19
A. Mirhoseini, A. Goldie, M. Yazgan, Jiang, Songhori, S. Wang, Y.-J. Lee, E. Johnson, O. Pathak. S. Bae, et al. Chip placement with deep reinforcement learning. arXiv preprint arXiv:2004.10746, 2020.
Mnih, K. Kavukcuoglu, Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
S. Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pages 271-279, 2016.
A. Pacchiano, J. Parker-Holder, Tang, A. Choromanska, K. Choromanski, and M. I. Jordan. Learning to Score Behaviors for Guided Policy Optimization. 2019. URL orq/ abs/ 1906. 04349
Parker-Holdes A. Pacchiano, Choromanski, and S. Roberts. Effective diversity in population- based reinforcement learning, arXiv preprint arXiv:2002.000 2002.002..0 2020.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), volume 2017, 2017.
N. Rahaman, S. Wolf, A. Goyal, R. Remme, and Y. Bengio. Learning the Arrow of Time. (1892): 1-19.

---

Under review as a conference paper at ICLR 2021
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 17X0:1777.00347,,00000 2017.
O. Sener. Multi-Task Learning as Multi-Objective Optimization. (Nips), 2018.
D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Ku- maran, T. Graepel, al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 2017.
A. M. Turner, D. Maddield-Mennlle and P. Tadepalli.
Conservative agency. arXiv preprint
arXiv:1902.09725, 2019.
A. M. Turner, N. Ratzlaff, and P. Tadepalli. Avoiding side effects in complex environments. arXiv preprint arXiv:2006.065: 2020.
C. Wainwright and P. Eckersley. SafeLife 1.0: Exploring side effects in complex environments. CEUR Workshop Proceedings, 2560:117-1221 2020. ISSN 16130073.
J. Xu, Y. Tian, P. Ma, D. Rus, S. Sueda, and W. Matusik. Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control. 2020.
R. Yang, X. Sun, and K. Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems, 2019.
Zhang, B. Cheung. Finn, Levine, and D. Jayaraman. Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings. 2020.
H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine. The ingredients of real-world robotic reinforcement learning, 2020.
10

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
Hyperparameter
Value
0.97
Learning Rate
3e
Batch Size
64
Epochs per Training Batch
Environment Steps per Training Iteration
20
PPO Entropy Weight
0.01
PPO Entropy Clip
1.0
PPO Value Loss Coefficient
0.5
PPO Value Loss Clip
0.2
PPO Policy Loss Clip
0.2
Table 1: Hyperparameterss for PPO-SARL
Environment
Baseline
SARL
SARL WD
SARL Zero-Shot JS
SARL Zero-Shot WD
Prune-Still
0.3
0.01
0.01
0.005
0.005
Append-Still
0.3
0.01
0.01
0.005
0.005
Prune-Dynamic
0.3
0.01
0.01
0.005
0.005
Append-Dynamic
0.3
0.01
0.01
0.005
0.005
Table Hyperparameters for Side Effect Procedures Baseline: Impact Penalty; SARL: /
B SAFELIFE TASKS
In this paper, we tackle four different tasks in the SafeLife environment:
Prune-Still: In this task, the agent removes red cells placed different locations throughout the environment. Given the still nature of the environment, there are no variations occurring in the environment other than the ones caused by the agent's actions.
Append-Stilly Still::- In this task, the agent fills in patterns specified by a blue background placed at different locations throughout the environment. Given the still nature of the environment, there are no variations occurring in the environment other than the ones caused by the agent's actions.
Prune-Dyamic: In this task, the agent removes red cells placed at different locations throughout the environment. The dynamic nature of the environment, where cell patterns oscillate in certain areas, creates additional complexity in the types of cell patterns that the agent interacts with. These types of dynamic variation make completing the task more difficult for the agent, as the distinction between variation caused by the agent's actions and environmentally driven variation is often inherently linked.
this task, the agent fills in pattern specified by blue placed different locations throughout the environment. The dynamic nature of the environment, where cell patterns oscillate certain areas, creates additional complexity in the types of cell patterns that the agent interacts with. These types of dynamic variation make completing the task more difficult for the agent, as the distinction between variation caused by the agent's 's actions and environmentally driven variation is often inherently linked.
Append-Dynane
11

---

Under review as a conference paper at ICLR 2021
C LIMITATIONS AND FUTURE DIRECTIONS OF SARL
SARL relies on training two different agents and taking a probabilistic distance between their policies to regularize the loss of the primary task agent. As discussed in section [ finding good side effect metrics is a difficult task. Moreover, a side effect metric that is effective for the penalty method described in Wainwright and Eckersley (020), might not be ideal for SARL safety agent training. As results from this paper suggest, training the safety agent on that metric often encourages the safety agent to stand still for the entire episode, because the safest action is often no action at all. The limitations related to having rich metrics to train both agents also apply outside of Safel .ife, where designer would have to create appropriate functions for the task and safety objectives, and test their effectiveness and generalizability across various situations. Our work suggests that safety agents trained on the SafeLife side effect metric can generalize zero-shot within SARL to other environment and task settings in SafeLife. However, it is unclear whether policies trained in SafeLife could generalize to different environments outside of SafeLife, as generalization of RL policies remains a major area of research.
The above limitations create opportunities for various extensions on SARL.. Given that the safety agent of SARL is flexible enough to being trained with any "safety reward", we think that valuable extension for future work could be to explore how various agents trained on a diversity of metrics, such variations of the SafeLife metric and metries in the literature (Trrner et al., 2020), perform on their own and within the SARL framework. Safety agents could also be trained on variations of intrinsic reward metrics (Pathak et al. 2017, Eysenbach et al., 2018) that are modified for side effect scenario. Furthermore, the current version of SARL only applied distances between the policy outputs of Ag and Zz., but there are many more ways to characterize an agent, such as state- or reward-based trajectories, that might require different distance formulations. As an extension of SARL, one could research the effects of different methods of calculating distances of various policy characteristics to obtain desired behaviors.
D DESCRIPTION OF RESULT FIGURES
The figures in Figure show three different metrics that provide insight into how agent performs in SafeLife.
The left vertical pane shows the length of an episode. Since the agent is trying to complete a task and reach the level exit in as few steps as possible, a shorter episode length is that for prune-still, the casiest task, all training
Episode Length:
generally better. In the top left image we can methods perform equally on the episode length. On the other tasks shown in the left vertical pane, the various methods show different behaviors for episode length.
The middle vertical pane describes the reward fraction described in Section 4 which shows the fraction of all possible rewards the agent was able to achieve within an episode. Since the agent requires a minimum reward to be able to reach the level exit, while also being incentivized to reach the exit as soon as possible, this serves as surrogate metric see how quickly the agent learns the task. As expected, we can see that this metric is highly correlated with the episode length and that all methods generally perform similarly on this metric when they are able to solve the task. The only exception here is the baseline in prune-dynamic in the third horizontal, where the baseline is not able to learn the task effectively.
Performance:
The right vertical pane shows the side effect results based on the episodic metric described in Section We aim to minimize this metric while maintaining strong performance on the other two. In our results, we see that in prune-dynamic (3rd horizontal) the baseline agent achieves better score on side effects but also does not solve the task, which is not desirable. In append-stil (2nd horizontal) all agent show mostly similar behavior; in stine-still (top horizontal) the baseline agent does slightly better on side effect but also achieves a slightly worse reward fraction, and in append-dynamic (bottom horizontal) the SARL methods achieve slightly better side effect scores.
Side Effects:
12


Reviews:
[{"review_id": "o8gp3BF_PLC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a safety-aware reinforcement learning algorithm that learns to perform tasks with minimal side-effects. The key idea is that a safety policy is learned independent of the task reward. When learning the task, this safety policy is incorporated by minimizing the distance between the task agent and the safety agent. In this way, the paper claims that the safety agent can be generalized to different tasks. The method is tested on SafeLife Suite, and its performance can match task-specific safe learning baselines.\n\nSafe reinforcement learning is an extremely important research area, when we need to apply reinforcement learning to real-world applications, such as robotics, recommendation system, power grid, etc. This paper works in this direction and addresses the key challenges, including how to learn generalizable safety agents. While I think that the paper is promising, I have the following three major concerns:\n\n1) The \"side effect metric\" is not clear to me. The description in Section 2.1 is high-level and vague. More rigorous mathematical definition is preferred here. For a safe learning paper, it is extremely important to clearly define what safety means. Is the \"side effect metric\" the same as the \"safety metric\" in Line 12 of Algorithm 1? Reading from the text, it seems that the side effect metric is calculated per episode, while the safety metric is per step.  \n\n2) Section 3.4 seems to leak the testing set into training. One claim of this paper is that the learned safety agent is generalizable: Z(\\psi) can be taken zero-shot from previous trained environments. However, during training, by tracking the Champion policy , decisions are made based on the performance on the testing environments, by retaining the policy that performs the best in the testing environments. If my understanding is right, this makes any claim about generalization less convincing because the training directly optimizes the policy in the testing environments.\n\n3) Intuitively, I do not understand how Algorithm 1 could work. According to Algorithm 1, the training of the safety agent Z(\\psi) is totally independent of the task, whose only objective is to be safe. If it is the case, the learned safety agent would not move or take action at all. The action distribution P_Z_\\psi would be concentrated on the zero action. This would make optimizing A(\\theta) using the loss (eq. (1)) extremely difficult. This might explain why the paper observes that the hyperparameter \\beta is difficult to tune.\n\nHere are some minor suggestions about writing:\n1) A brief description of the 4 tasks is needed (prune-still, prune-dynamic, append-still, append-dynamic) to make this paper more self-contained. If the page-limit is a concern, this description could be added to the Appendix. Otherwise, it is difficult for readers to understand the difficulties and the usefulness of these tasks.\n\n2) \"Line 13-17 from Algorithm 1\": The pseudo-code ends at Line 16.\n\nFor the above reasons, I would not recommend acceptance at this time.", "extra_scores": {}, "extra_text": {}}, {"review_id": "kfE8pSO50sg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to address the issue of mitigating side effects in policy learning. The authors propose an algorithm SARL, which uses a safe policy to define a regularization term for penalizing the agent's actions deviating from the safe agent in policy learning. In the experiments, four variations for SARL are shown and compared with a baseline method based on reward penalty. The proposed algorithm is competitive across the experiments presented in the paper. \n\nI think side effects and safety in reinforcement learning is an important issue. However, this paper does a bad job in describing the problem it wishes to address and, therefore, it's unclear whether the proposed algorithm really achieves that goal. \n\n1. The main motivation of this paper is to mitigate the side effects in learning. However, the definition of side effects were never given. It's only until Algorithm 1 is presented where the paper mentions a safety metric that the safe agent aims to optimize (is this the same s appearing in A(s|theta) and Z(s|psi) in Sec 3.2?), which however is not defined. Therefore, I do not fully understand what the objective of this learning algorithm wants to achieve. From the paper's vague description, it seems like the goal is that the learner should have high performance in the original reward while not causing high side effects. This is a multi-objective MDP problem or at least can be framed as a constrained MDP. However, the proposed algorithm, based on simple regularization with a constant weight, can address neither of these two criteria. I am wondering if the authors consider to more explicitly outline the solution concept they wish to obtain. Current hand-wavy description makes me difficult to judge whether the proposed algorithm actually solves the problem they wish to solve.\n\n2. In Algorithm 1, since the safe agent Z is updated independently of the progress of the learner agent A, when there's only a single environment, there is no point of distinguishing the so-called \"zero-shot\" and the online version, as in high level this dependency allows us to pretrain the safety agent alone beforehand and get the same results. Or do the authors mean zero-shot in the sense that the safe agent is trained on a different set of environments and the online version means they're trained on the same environment? \n\n3. In the paper, the authors write multiple times that a difficulty in this problem setup is that the side effects are difficulty to define. But it seems that the proposed algorithm assumes some safety metric. How are the two related precisely? And what is that used in the experiments?\n\n4. What is S[\\pi_theta] in (3)?\n\nOverall, I think the paper is rather incomplete and therefore I do not recommend acceptance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QnY68Y3ogaa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update after authors' response**\nI want to thank the authors for their responses. My responses to the authors' comments are in the respective threads.\n---\n\n**Summary**\nThe paper addresses the timely and important problem of how to train RL agents such that they solve desired tasks while not engaging in undesired behavior that is not explicitly specified via the reward function. In particular, the paper focuses on training agents that learn to avoid unnecessary side effects, that is (irreversible) alterations to the environment which are not necessary to solve the task at hand. Experiments are performed on SafeLife, which provides a suite of tasks in an environment (potentially with rich intrinsic dynamics), along with a quantitative measure of the strength of undesired side effects. The main idea of the paper is to co-train an RL policy on this side effect measure with the aim of minimizing side effects. This policy is used for regularizing the reward-optimizing agent during training, such that the trained agent learns to bias its actions towards avoiding side effects when the task allows for multiple viable actions. The paper compares against a strong, previously reported baseline, both in static and dynamic SafeLife environments/tasks. Additionally, the generalization of the side-effect-avoiding policy is tested, by using it for training a reward-optimizing agent on task-versions that the side-effect-avoiding policy has not been trained on.\n\n---\n**Contributions, Novelty, Impact**\n\n1) Incorporation of the two (sometimes conflicting) objectives of maximizing reward and avoiding side-effects into a single training objective, where purely reward maximizing actions are regularized by action-distributions from a side-effect-minimizing policy. This is an interesting idea that turns trading off avoiding side effects against reward maximization into a learning problem. I think this is a promising way forward. What I’d like to see in the paper for even greater impact is a clear discussion of the requirements (the objective of avoiding side-effects must be specified as a trajectory-dependent, quantitative function, similar to a reward function), and the current limitations (unclear how to assess “how much” of the task-relevant state-space is well covered by the side-effect-avoiding policy, particularly in the zero-shot setting).\n\n2) Experimental evaluation of the proposed method on *the* state-of-the-art benchmark suite, and comparison against a strong, previously proposed baseline. The results are promising, though it’s hard to distill a very clear message in favor of the method from the results shown. I personally think that’s fine (and to some degree expected when discussing solutions that solve a particular trade-off in a different fashion), but I’d like to see even more of a multi-faceted evaluation and discussion in the paper.\n\n3) The idea of learning a side-effect-avoiding regularizer that generalizes well, e.g. to different tasks under the same environment dynamics. This is very interesting and a promising step towards tackling the side effects problem at scale. It is very nice to see the zero-shot results. To make the paper even stronger and more impactful it would be nice to evaluate the generalization of the trained side-effect-avoiding police in more detail.\n\n---\n**Score and reasons for score**\nI am (currently) in favor of accepting the paper, though I think that some additional work could improve the strength and potential impact of the work. The topic addressed is timely and very important, and the approach taken is interesting and sensible. Results look promising, and the paper does a great job at presenting the work. To further strengthen the paper it would be nice to discuss results in more detail and potentially perform additional experiments to highlight certain aspects that are “buried” in the current results. Additionally it would be good to say something more substantial about the generalization properties of the side-effect-avoiding policy. While the latter two issues are probably beyond what’s easily possible in the rebuttal phase, I want to strongly encourage the authors to add a short paragraph that clearly states the assumptions/requirements (the strongest assumption is perhaps the presence of a quantitative side effect measure which can be used directly as a reinforcement signal), and current limitations. I am looking forward to the other reviews and authors’ response, and will update my final verdict accordingly.\n\n---\n**Strengths**\n1) Empirically promising results on a timely and important problem, including the comparison against a strong baseline method.\n\n2) Evaluation of proposed method by: (i) multiple runs to assess statistical significance, (ii) ablation studies regarding the “distance” metric used by the method, (iii) control-experiments regarding the (zero-shot) generalization performance of the side-effect-avoiding policy.\n\n3) Well written paper, with good introduction to the problem and discussion of related literature (given the limited space of a conference-format publication).\n\n---\n**Weaknesses**\n1) The experimental results shown are interesting and promising, but it’s hard to distill a clear message from the results other than: “the proposed method seems to work on par with a previously proposed method but often makes the trade-offs (between high reward and low side-effects) differently, which makes comparison more difficult”. Drilling down on some of the findings and trying to control for more factors to get a clearer picture would strengthen the results.\n\n2) The generalization of the side-effect-avoiding policy is a very interesting aspect of the work, however the current analysis of how well that generalization behaves is a bit crude. It is unclear to which degree the previously trained side-effect-avoiding policy in the zero-shot regime covers the state-space encountered when solving a particular task. It is also unclear whether the side-effect-avoiding policy in the generalization setting “behaves mostly well overall” or whether it has some severe and potentially even systematic shortcomings (leading to undesired policies) in particular situations of the generalization regime. Addressing this in full generality is of course beyond the scope of this paper, but some more analysis into this issue would be very nice to see (e.g. comparing the zero-shot vs the trained side-effect-policies in isolation, and potentially drilling in on some of the differences encountered).\n\n---\n**Correctness**\nThe construction of the algorithm and training scheme presented in the paper seems correct to me.\n\n---\n**Clarity**\nThe paper is mostly well written, and the method is clearly described. Perhaps two things to improve: (i) the discussion of results could be expanded a bit more, there’s a lot going on in the plot and unfortunately there’s no intuitive message that one can easily take away visually. (ii) To facilitate the flow of the manuscript to readers unfamiliar with SafeLife it would be nice to include a short section describing the side-effect penalty.\n\n---\n**Improvements / major issues**\n1) The results currently shown are interesting but it’s hard to distill a clear message (which is understandable to some degree, as the paper also points out, because different solutions to a multi-objective optimization cannot be easily compared). It might be worthwhile though to expand the discussion (and perhaps even presentation) of the results a bit more. \n\n2) One of the most interesting aspects of the work is the potential to train a task-agnostic side-effects-avoiding policy that generalizes to a broad range of tasks. The paper demonstrates that this works by applying said policy in a zero-shot setting and analyzing the resulting policy. It would be nice to also do some more comparison of the side-effects-avoiding policies directly (e.g. what is the side effect score when directly comparing a zero-shot Z vs a Z trained on the current task/environment - are there any systematic deviations between the two, do certain biases get baked into the zero-shot Z that can be explained by the tasks/environment-variants it’s been trained on).\n\n3) A clear discussion of the requirements (the objective of avoiding side-effects must be specified as a trajectory-dependent, quantitative function, similar to a reward function), and the current limitations (unclear how to assess “how much” of the task-relevant state-space is well covered by the side-effect-avoiding policy, particularly in the zero-shot setting).\n\n4) Please clarify: why are there separate zero-shot agents shown in prune-still and append-still - shouldn’t they be the same SARL JS/WD since the zero-shot agents have been trained on these two tasks respectively?\n\n5) Please clarify and potentially discuss in the paper: perhaps the main requirement for the method is having a side-effect-strength signal s. This signal must be suitable for a reinforcement learning algorithm to train a side-effects-minimizing policy Z. But if such a signal is available, why not simply combine it with the task-specific reward function r to create a “safe reward function” to train a reward-optimizing agent that avoids side-effects? Would the solution obtained this way be qualitatively different (in some aspects) compared to the solution obtained by the proposed scheme? It’s fine to simply comment on this - the strongest version would include actual control experiments (but I understand that this might not be easily doable).\n\n6) Please comment and potentially discuss in the paper: What is the advantage of co-training Z with A (lines 11-15 of Algorithm 1)? Why not train Z first (e.g. would that improve training stability)?\n\n\n---\n**Minor comments**\n\nA) Please give a few details for the side effect metric that’s used by the experiment (fine to refer to the SafeLife paper for full details, but the rough idea should be in the paper to improve readability).\n\nB) How exactly is it ensured that Z sees the same parts of the state-space that A does (i.e. how is it ensured that Z “explores” similarly to A, which is solving some tasks)? I assume that the actions actually taken (which lead to a certain state on which A and Z are evaluated in line 5 and 6 in Algo 1) are driven by A?\n\nC) P4: “In this formulation, policy characteristics are converted to distributions in a latent space of behavioral embeddings on which the Wasserstein Distance is then computed.”. I have a hard time following this sentence, please consider unpacking it a bit.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1XHn-0-ReaC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to reduce the unwanted side effects of the actions of a reward-maximizing reinforcement-learning (RL) agent. The authors study a framework in which the environment issues a metric that measures the total side effects of the agent's actions at the end of each episode. The work's proposed solution trains an agent who focuses on the total reward and another agent who minimizes the total side effects. The authors then empirically investigate the effectiveness of combining the two agents via a distance measure between the policies that unifies them into one.\n\nI find the following items strong points in the submission:\n* The posed problem is relevant in the context of safety in AI.\n* The chosen testbed for the experiments matches the goals and premises of the posed problem.\n* The empirical results suggest that the proposed method is effective.\n* The discussion regarding the choice of the distance measure is thorough and makes sense.\n\nOn the other hand, I find the following issues as weaknesses in the submitted manuscript:\n* There are no theoretical developments to demonstrate whether the reported results generalize beyond the adopted environment settings and the value assigned to the parameter beta or not.\n* The paper dives right into introducing the loss function in Section 3 without establishing the required notation and preliminary materials. A brief summary of the task agent and the virtual safe agent descriptions is currently provided in the caption of Figure 2. In my opinion, Section 3 would read better if the authors append a preliminaries section wherein they establish the notations and the descriptions of the task agent and the virtual safe agent.\n* The loss function adopted in equation (3) provides little room for theoretical developments. The original paper that introduces the PPO algorithm (Schulman et al., 2017) offers multiple loss function choices. In my opinion, the combination of the Jenson-Shannon distance with the loss function that incorporates the KL divergences enables the authors to study their proposed algorithm beyond empirical results.\n\nI find the posed problem relevant in the context of safety in AI and the suggested method well-motivated and intuitive. I believe the submission is far from theoretically solving the posed problem; however, the methodology alongside the promising empirical results that the manuscript offers may be of interest.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the paper, reviewers' comments and discussions, and the responses, the meta-reviewer would like to suggest the authors to improve the paper and resubmit.
Decision: Reject