Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper aims to address the issue of mitigating side effects in policy learning. The authors propose an algorithm SARL, which uses a safe policy to define a regularization term for penalizing the agent's actions deviating from the safe agent in policy learning. In the experiments, four variations for SARL are shown and compared with a baseline method based on reward penalty. The proposed algorithm is competitive across the experiments presented in the paper. 

I think side effects and safety in reinforcement learning is an important issue. However, this paper does a bad job in describing the problem it wishes to address and, therefore, it's unclear whether the proposed algorithm really achieves that goal. 

1. The main motivation of this paper is to mitigate the side effects in learning. However, the definition of side effects were never given. It's only until Algorithm 1 is presented where the paper mentions a safety metric that the safe agent aims to optimize (is this the same s appearing in A(s|theta) and Z(s|psi) in Sec 3.2?), which however is not defined. Therefore, I do not fully understand what the objective of this learning algorithm wants to achieve. From the paper's vague description, it seems like the goal is that the learner should have high performance in the original reward while not causing high side effects. This is a multi-objective MDP problem or at least can be framed as a constrained MDP. However, the proposed algorithm, based on simple regularization with a constant weight, can address neither of these two criteria. I am wondering if the authors consider to more explicitly outline the solution concept they wish to obtain. Current hand-wavy description makes me difficult to judge whether the proposed algorithm actually solves the problem they wish to solve.

2. In Algorithm 1, since the safe agent Z is updated independently of the progress of the learner agent A, when there's only a single environment, there is no point of distinguishing the so-called "zero-shot" and the online version, as in high level this dependency allows us to pretrain the safety agent alone beforehand and get the same results. Or do the authors mean zero-shot in the sense that the safe agent is trained on a different set of environments and the online version means they're trained on the same environment? 

3. In the paper, the authors write multiple times that a difficulty in this problem setup is that the side effects are difficulty to define. But it seems that the proposed algorithm assumes some safety metric. How are the two related precisely? And what is that used in the experiments?

4. What is S[\pi_theta] in (3)?

Overall, I think the paper is rather incomplete and therefore I do not recommend acceptance.

review_title: Unclear problem formulation and not fully warranted algorithm

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Update after authors' response**
I want to thank the authors for their responses. My responses to the authors' comments are in the respective threads.
---

**Summary**
The paper addresses the timely and important problem of how to train RL agents such that they solve desired tasks while not engaging in undesired behavior that is not explicitly specified via the reward function. In particular, the paper focuses on training agents that learn to avoid unnecessary side effects, that is (irreversible) alterations to the environment which are not necessary to solve the task at hand. Experiments are performed on SafeLife, which provides a suite of tasks in an environment (potentially with rich intrinsic dynamics), along with a quantitative measure of the strength of undesired side effects. The main idea of the paper is to co-train an RL policy on this side effect measure with the aim of minimizing side effects. This policy is used for regularizing the reward-optimizing agent during training, such that the trained agent learns to bias its actions towards avoiding side effects when the task allows for multiple viable actions. The paper compares against a strong, previously reported baseline, both in static and dynamic SafeLife environments/tasks. Additionally, the generalization of the side-effect-avoiding policy is tested, by using it for training a reward-optimizing agent on task-versions that the side-effect-avoiding policy has not been trained on.

---
**Contributions, Novelty, Impact**

1) Incorporation of the two (sometimes conflicting) objectives of maximizing reward and avoiding side-effects into a single training objective, where purely reward maximizing actions are regularized by action-distributions from a side-effect-minimizing policy. This is an interesting idea that turns trading off avoiding side effects against reward maximization into a learning problem. I think this is a promising way forward. What I’d like to see in the paper for even greater impact is a clear discussion of the requirements (the objective of avoiding side-effects must be specified as a trajectory-dependent, quantitative function, similar to a reward function), and the current limitations (unclear how to assess “how much” of the task-relevant state-space is well covered by the side-effect-avoiding policy, particularly in the zero-shot setting).

2) Experimental evaluation of the proposed method on *the* state-of-the-art benchmark suite, and comparison against a strong, previously proposed baseline. The results are promising, though it’s hard to distill a very clear message in favor of the method from the results shown. I personally think that’s fine (and to some degree expected when discussing solutions that solve a particular trade-off in a different fashion), but I’d like to see even more of a multi-faceted evaluation and discussion in the paper.

3) The idea of learning a side-effect-avoiding regularizer that generalizes well, e.g. to different tasks under the same environment dynamics. This is very interesting and a promising step towards tackling the side effects problem at scale. It is very nice to see the zero-shot results. To make the paper even stronger and more impactful it would be nice to evaluate the generalization of the trained side-effect-avoiding police in more detail.

---
**Score and reasons for score**
I am (currently) in favor of accepting the paper, though I think that some additional work could improve the strength and potential impact of the work. The topic addressed is timely and very important, and the approach taken is interesting and sensible. Results look promising, and the paper does a great job at presenting the work. To further strengthen the paper it would be nice to discuss results in more detail and potentially perform additional experiments to highlight certain aspects that are “buried” in the current results. Additionally it would be good to say something more substantial about the generalization properties of the side-effect-avoiding policy. While the latter two issues are probably beyond what’s easily possible in the rebuttal phase, I want to strongly encourage the authors to add a short paragraph that clearly states the assumptions/requirements (the strongest assumption is perhaps the presence of a quantitative side effect measure which can be used directly as a reinforcement signal), and current limitations. I am looking forward to the other reviews and authors’ response, and will update my final verdict accordingly.

---
**Strengths**
1) Empirically promising results on a timely and important problem, including the comparison against a strong baseline method.

2) Evaluation of proposed method by: (i) multiple runs to assess statistical significance, (ii) ablation studies regarding the “distance” metric used by the method, (iii) control-experiments regarding the (zero-shot) generalization performance of the side-effect-avoiding policy.

3) Well written paper, with good introduction to the problem and discussion of related literature (given the limited space of a conference-format publication).

---
**Weaknesses**
1) The experimental results shown are interesting and promising, but it’s hard to distill a clear message from the results other than: “the proposed method seems to work on par with a previously proposed method but often makes the trade-offs (between high reward and low side-effects) differently, which makes comparison more difficult”. Drilling down on some of the findings and trying to control for more factors to get a clearer picture would strengthen the results.

2) The generalization of the side-effect-avoiding policy is a very interesting aspect of the work, however the current analysis of how well that generalization behaves is a bit crude. It is unclear to which degree the previously trained side-effect-avoiding policy in the zero-shot regime covers the state-space encountered when solving a particular task. It is also unclear whether the side-effect-avoiding policy in the generalization setting “behaves mostly well overall” or whether it has some severe and potentially even systematic shortcomings (leading to undesired policies) in particular situations of the generalization regime. Addressing this in full generality is of course beyond the scope of this paper, but some more analysis into this issue would be very nice to see (e.g. comparing the zero-shot vs the trained side-effect-policies in isolation, and potentially drilling in on some of the differences encountered).

---
**Correctness**
The construction of the algorithm and training scheme presented in the paper seems correct to me.

---
**Clarity**
The paper is mostly well written, and the method is clearly described. Perhaps two things to improve: (i) the discussion of results could be expanded a bit more, there’s a lot going on in the plot and unfortunately there’s no intuitive message that one can easily take away visually. (ii) To facilitate the flow of the manuscript to readers unfamiliar with SafeLife it would be nice to include a short section describing the side-effect penalty.

---
**Improvements / major issues**
1) The results currently shown are interesting but it’s hard to distill a clear message (which is understandable to some degree, as the paper also points out, because different solutions to a multi-objective optimization cannot be easily compared). It might be worthwhile though to expand the discussion (and perhaps even presentation) of the results a bit more. 

2) One of the most interesting aspects of the work is the potential to train a task-agnostic side-effects-avoiding policy that generalizes to a broad range of tasks. The paper demonstrates that this works by applying said policy in a zero-shot setting and analyzing the resulting policy. It would be nice to also do some more comparison of the side-effects-avoiding policies directly (e.g. what is the side effect score when directly comparing a zero-shot Z vs a Z trained on the current task/environment - are there any systematic deviations between the two, do certain biases get baked into the zero-shot Z that can be explained by the tasks/environment-variants it’s been trained on).

3) A clear discussion of the requirements (the objective of avoiding side-effects must be specified as a trajectory-dependent, quantitative function, similar to a reward function), and the current limitations (unclear how to assess “how much” of the task-relevant state-space is well covered by the side-effect-avoiding policy, particularly in the zero-shot setting).

4) Please clarify: why are there separate zero-shot agents shown in prune-still and append-still - shouldn’t they be the same SARL JS/WD since the zero-shot agents have been trained on these two tasks respectively?

5) Please clarify and potentially discuss in the paper: perhaps the main requirement for the method is having a side-effect-strength signal s. This signal must be suitable for a reinforcement learning algorithm to train a side-effects-minimizing policy Z. But if such a signal is available, why not simply combine it with the task-specific reward function r to create a “safe reward function” to train a reward-optimizing agent that avoids side-effects? Would the solution obtained this way be qualitatively different (in some aspects) compared to the solution obtained by the proposed scheme? It’s fine to simply comment on this - the strongest version would include actual control experiments (but I understand that this might not be easily doable).

6) Please comment and potentially discuss in the paper: What is the advantage of co-training Z with A (lines 11-15 of Algorithm 1)? Why not train Z first (e.g. would that improve training stability)?


---
**Minor comments**

A) Please give a few details for the side effect metric that’s used by the experiment (fine to refer to the SafeLife paper for full details, but the rough idea should be in the paper to improve readability).

B) How exactly is it ensured that Z sees the same parts of the state-space that A does (i.e. how is it ensured that Z “explores” similarly to A, which is solving some tasks)? I assume that the actions actually taken (which lead to a certain state on which A and Z are evaluated in line 5 and 6 in Algo 1) are driven by A?

C) P4: “In this formulation, policy characteristics are converted to distributions in a latent space of behavioral embeddings on which the Wasserstein Distance is then computed.”. I have a hard time following this sentence, please consider unpacking it a bit.
review_title: Timely and important topic, addressed with a simple but neat idea - results are a bit hard to put into perspective / generalize.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper aims to reduce the unwanted side effects of the actions of a reward-maximizing reinforcement-learning (RL) agent. The authors study a framework in which the environment issues a metric that measures the total side effects of the agent's actions at the end of each episode. The work's proposed solution trains an agent who focuses on the total reward and another agent who minimizes the total side effects. The authors then empirically investigate the effectiveness of combining the two agents via a distance measure between the policies that unifies them into one.

I find the following items strong points in the submission:
* The posed problem is relevant in the context of safety in AI.
* The chosen testbed for the experiments matches the goals and premises of the posed problem.
* The empirical results suggest that the proposed method is effective.
* The discussion regarding the choice of the distance measure is thorough and makes sense.

On the other hand, I find the following issues as weaknesses in the submitted manuscript:
* There are no theoretical developments to demonstrate whether the reported results generalize beyond the adopted environment settings and the value assigned to the parameter beta or not.
* The paper dives right into introducing the loss function in Section 3 without establishing the required notation and preliminary materials. A brief summary of the task agent and the virtual safe agent descriptions is currently provided in the caption of Figure 2. In my opinion, Section 3 would read better if the authors append a preliminaries section wherein they establish the notations and the descriptions of the task agent and the virtual safe agent.
* The loss function adopted in equation (3) provides little room for theoretical developments. The original paper that introduces the PPO algorithm (Schulman et al., 2017) offers multiple loss function choices. In my opinion, the combination of the Jenson-Shannon distance with the loss function that incorporates the KL divergences enables the authors to study their proposed algorithm beyond empirical results.

I find the posed problem relevant in the context of safety in AI and the suggested method well-motivated and intuitive. I believe the submission is far from theoretically solving the posed problem; however, the methodology alongside the promising empirical results that the manuscript offers may be of interest.
review_title:  Interesting work toward reducing the unwanted side effects of the actions of a reward-maximizing reinforcement-learning agent.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a safety-aware reinforcement learning algorithm that learns to perform tasks with minimal side-effects. The key idea is that a safety policy is learned independent of the task reward. When learning the task, this safety policy is incorporated by minimizing the distance between the task agent and the safety agent. In this way, the paper claims that the safety agent can be generalized to different tasks. The method is tested on SafeLife Suite, and its performance can match task-specific safe learning baselines.

Safe reinforcement learning is an extremely important research area, when we need to apply reinforcement learning to real-world applications, such as robotics, recommendation system, power grid, etc. This paper works in this direction and addresses the key challenges, including how to learn generalizable safety agents. While I think that the paper is promising, I have the following three major concerns:

1) The "side effect metric" is not clear to me. The description in Section 2.1 is high-level and vague. More rigorous mathematical definition is preferred here. For a safe learning paper, it is extremely important to clearly define what safety means. Is the "side effect metric" the same as the "safety metric" in Line 12 of Algorithm 1? Reading from the text, it seems that the side effect metric is calculated per episode, while the safety metric is per step.  

2) Section 3.4 seems to leak the testing set into training. One claim of this paper is that the learned safety agent is generalizable: Z(\psi) can be taken zero-shot from previous trained environments. However, during training, by tracking the Champion policy , decisions are made based on the performance on the testing environments, by retaining the policy that performs the best in the testing environments. If my understanding is right, this makes any claim about generalization less convincing because the training directly optimizes the policy in the testing environments.

3) Intuitively, I do not understand how Algorithm 1 could work. According to Algorithm 1, the training of the safety agent Z(\psi) is totally independent of the task, whose only objective is to be safe. If it is the case, the learned safety agent would not move or take action at all. The action distribution P_Z_\psi would be concentrated on the zero action. This would make optimizing A(\theta) using the loss (eq. (1)) extremely difficult. This might explain why the paper observes that the hyperparameter \beta is difficult to tune.

Here are some minor suggestions about writing:
1) A brief description of the 4 tasks is needed (prune-still, prune-dynamic, append-still, append-dynamic) to make this paper more self-contained. If the page-limit is a concern, this description could be added to the Appendix. Otherwise, it is difficult for readers to understand the difficulties and the usefulness of these tasks.

2) "Line 13-17 from Algorithm 1": The pseudo-code ends at Line 16.

For the above reasons, I would not recommend acceptance at this time.
review_title: Although an important direction and promising progress, I have concerns about the method and the results

Meta-review: Based on the paper, reviewers' comments and discussions, and the responses, the meta-reviewer would like to suggest the authors to improve the paper and resubmit.
Decision: Reject