Title: Avoiding Catastrophic States with Intrinsic Fear
Authors: Zachary C. Lipton
Kamyar Azizzadenesheli
Abhishek Kumar
Lihong Li
Jianfeng Gao
Li Deng
Venue: ICLR 2018
Keywords: reinforcement learning
safe exploration
dqn

Abstract: Many practical reinforcement learning problems contain catastrophic states that the optimal policy visits infrequently or never. Even on toy problems, deep reinforcement learners periodically revisit these states, once they are forgotten under a new policy. In this paper, we introduce intrinsic fear, a learned reward shaping that accelerates deep reinforcement learning and guards oscillating policies against periodic catastrophes. Our approach incorporates a second model trained via supervised learning to predict the probability of imminent catastrophe. This score acts as a penalty on the Q-learning objective. Our theoretical analysis demonstrates that the perturbed objective yields the same average return under strong assumptions and an $\epsilon$-close average return under weaker assumptions. Our analysis also shows robustness to classification errors. Equipped with intrinsic fear, our DQNs solve the toy environments and improve on the Atari games Seaquest, Asteroids, and Freeway.

TL;DR: Shape reward with intrinsic motivation to avoid catastrophic states and mitigate catastrophic forgetting.

Review discussion:
SUMMARY
The paper proposes an RL algorithm that combines the DQN algorithm with a fear model.  The fear model is trained in parallel to predict catastrophic states.  Its output is used to penalize the Q learning target.
COMMENTS
Not convinced about the fact that an agent forgets about catastrophic states. Because it does not experience it any more.  Shouldn’t the agent stop learning at some point in time?  Why does it need to keep collecting good data?  How about giving more weight to catastrophic data (e.g., replicating it)
Is the catastrophic scenario specific to DRL or RL in general with function approximation?
Why not specify catastrophic states with a large negative reward?
It seems that catastrophe states need to be experienced at least once.
Is that acceptable for the autonomous car hitting a pedestrian?
We are grateful to Reviewer3 for taking the time to review our paper but disagree with several of the assertions.
1. The reviewer states “Not convinced about the fact that an agent forgets about catastrophic states”. The susceptibility of neural networks to catastrophic forgetting (not to be confused with our safety-motivated notion of a catastrophe) is well-documented in the literature.
Whenever the policy is modified such that some states would never be encountered, they will eventually, as soon as they are flushed from the replay buffer, cease to influence the Q-network. If we continue to update the network as is necessary, especially in non-stationary environments (e.g. nearly all real-world settings) then nothing in the standard DQN formulation guards the agent from revisiting the catastrophic states.
In addition to being well-documented in the literature, we demonstrate this problem clearly in our paper using the simplest failure case. Even in AdventureSeeker, a 1-D environment with only two actions, the agent will eventually forget about the catastrophic states.
2.  Re: “Shouldn’t the agent stop learning at some point in time?”
a.
First, even when there is limited duration learning period, we may want an agent to make a minimal number of catastrophic errors while learning.
b.
Second, as stated above: in nonstationary environments, which describes most real-world environments, we want to learn continually.
Otherwise the policy will become stale and cease to perform owing to the shifting dynamics. In the case of driving, imagine new cars which appear on the road, or new street signs. In the case of a vacuum-cleaner, imagine that it is confronted with new household appliances that didn’t exist previously.
3. “Re: “Why not specify catastrophic states with a large negative reward?”
Unfortunately, even large negative rewards are eventually forgotten, leading the agent to revisit the catastrophic states.  Take AdventureSeeker as an example: no matter how negative the penalty is, the same catastrophic forgetting will eventually happen.
Moreover, this approach, unlike ours, has no notion of “danger zone” and therefore does not benefit from reward shaping. In our approach, the agent avoids even getting *close* to a catastrophe.
When this assumption is reasonable, this leads to significantly faster exploration.
4. Re:  “It seems that catastrophe states need to be experienced at least once. Is that acceptable for the autonomous car hitting a pedestrian?”
This is a good question that has implications for all of RL: If catastrophes can truly never be experienced even once, then is reinforcement learning off the table altogether?
However, in many settings, perhaps even car accidents, if enough cars are on the road and the probability of an accident is nonzero, then accidents will happen. Our work addresses how to learn from these mistakes rapidly and to guard against repeating the same mistakes in the future.
The paper addresses the problem of learners forgetting rare states and revisiting catastrophic danger states. The authors propose to train a predictive ‘fear model’ that penalizes states that lead to catastrophes. The proposed technique is validated both empirically and theoretically.
Experiments show a clear advantage during learning when compared with a vanilla DQN.
Nonetheless, there are some criticisms than can be made of both the method and the evaluations:
The fear radius threshold k_r seems to add yet another hyperparameter that needs tuning. Judging from the description of the experiments this parameter is important to the performance of the method and needs to be set experimentally. There seems to be no way of a priori determine a good distance as there is no way to know in advance when a catastrophe becomes unavoidable. No empirical results on the effect of the parameter are given.
The experimental results support the claim that this technique helps to avoid catastrophic states during initial learning.The paper however, also claims to address the longer term problem of revisiting these states once the learner forgets about them, since they are no longer part of the data generated by (close to) optimal policies.  This problem does not seem to be really solved by this method. Danger and safe state replay memories are kept, but are only used to train the catastrophe classifier.
While the catastrophe classifier can be seen as an additional external memory, it seems that the learner will still drift away from the optimal policy and then need to be reminded by the classifier through penalties.
As such the method wouldn’t prevent catastrophic forgetting, it would just prevent the worst consequences by penalizing the agent before it reaches a danger state. It would therefore  be interesting to see some long running experiments and analyse how often catastrophic states (or those close to them) are visited.
Overall, the current evaluations focus on performance and give little insight into the behaviour of the method. The paper also does not compare to any other techniques that attempt to deal with catastrophic forgetting and/or the changing state distribution ([1,2]).
In general the explanations in the paper often often use confusing and  imprecise language, even in formal derivations, e.g.  ‘if the fear model reaches arbitrarily high accuracy’ or ‘if the probability is negligible’.
It is wasn’t clear to me that the properties described in Theorem 1 actually hold. The motivation in the appendix is very informal and no clear derivation is provided. The authors seem to indicate that a minimal return can be guaranteed because the optimal policy spends a maximum of epsilon amount of time in the catastrophic states and the alternative policy simply avoids these states.
However, as the alternative policy is learnt on a different reward, it can have a very different state distribution, even for the non-catastrophics states.  It might attach all its weight to a very poor reward state in an effort to avoid the catastrophe penalty. It is therefore not clear to me that any claims can be made about its performance without additional assumptions.
It seems that one could construct a counterexample using a 3-state chain problem (no_reward,danger, goal) where the only way to get to the single goal state is to incur a small risk of visiting the danger state. Any optimal policy would therefore need to spend some time e in the danger state, on average. A policy that learns to avoid the danger state would then also be unable to reach the goal state and receive rewards. E.g pi* has stationary distribution (0,e,1-e) and return 0*0+e*Rmin + (1-e)*Rmax. By adding a sufficiently high penalty, policy pi~ can learn to avoid the catastrophic state with distribution (1,0,0) and then gets return 1*0+ 0*Rmin+0*Rmax= 0 < n*_M - e (Rmax - Rmin) = e*Rmin + (1-e)*Rmax - e (Rmax - Rmin).  This seems to contradict the theorem. It wasn’t clear what assumptions the authors make to exclude situations like this.
[1] T. de Bruin, J. Kober, K. Tuyls and R. Babuška, "Improved deep reinforcement learning for robotics through distribution-based experience retention," 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, 2016, pp. 3947-3952.
[2] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., ... & Hassabis, D. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 201611835.
Thanks for the thoughtful review of our paper.
1. We are glad that you noticed the issue in the proof of theorem 1. Per your feedback, we have corrected the proof and substantially revised the paper (see current revision). At a high level, the performance degradation, as described corrected theorem and proof are as follows:
If the optimal policy, \pi^*, of the original environment, without intrinsic fear, (M), visits the fear zone with probability at most \epsilon, then applying  pi^* on the environment with intrinsic fear (M,F), gives the return of eta^*-\epsilon\lambda(Rmax-Rmin) therefore, the optimal policy on (M,F), \tilde{\pi}, can not give a return less than \eta^*\epsilon-\lambda(Rmax-Rmin) on environment (M,F). If \tilde{\pi} visits the fear zone with probability \epsilon’, we can rewrite its return as:
(return from non intrinsic fears)-epsilon’(\lambda(Rmax-Rmin)
Therefore applying \tilde{\pi} on original environment (M) gives a return of at least \eta^*-\epsilon\lambda(Rmax-Rmin) +\epsilon’\lambda(Rmax-Rmin) which is lower bounded by \eta^*-\epsilon\lambda(Rmax-Rmin).
2.
Regarding the parameter k_r, as the reviewer mentioned, without any prior knowledge and posed safety constraint of the environment, this parameter needs to be chosen empirically, as with other hyper-parameters. We note however, that this is a kind of prior knowledge that might be reasonably to expect of an algorithm designer.
For example, a robot should perhaps never be too close to a cliff or a ledge.
Intuitively, small k_r’s better preserve the original policy, but for too small a k_r, the fear model might be ignored.
On the other hand, large k_r are better at preventing the agent from visiting the catastrophic states but run more risk of deviating substantially from the optimal policy. Prior knowledge of the environment can guide us to design a proper k_r, otherwise, k_r needs to be chosen experimentally.
3.
Regarding the (very) long term forgetting, the reviewer is correct that this paper doesn’t completely alleviate catastrophic forgetting an that we instead guard against the worst consequences. We have created a video to visualize the fear probability as a red overlay on the video game play and will continue to work on other ways to qualitatively understand how our algorithm is working.
4. We thank the reviewer for suggesting baselines to compare to.  They have some relevance but are designed for different purposes.
In particular,
[1] (IROS) uses a second experience replay buffer to store state transitions that covers the whole state space uniformly, in addition to a typical buffer used in standard DQN.  This approach aims mostly to reduce exploration, but can face the curse of dimensionality as it tries to cover the state space uniformly.
Moreover, the uniform covering idea is not efficient for avoiding catastrophic events that are rare, while our approach uses a fear classifier to target danger zones directly.
[2] (PNAS) takes a Bayesian approach to continual learning, trying to avoid catastrophic forgetting of solutions to earlier tasks that have not occured for a long time.
In contrast, our problem is to avoid running into catastrophic states in the same task.  It is not clear how a similar, Bayesian variant of DQN (such as BBQ) can be extended to address our safe exploration challenge.
I've slightly increased my score to reflect the improvements made by the authors. Theorem 1 seems to have been corrected. Unfortunately, the bound now indicates that the average reward is within lambda * epsilon * (R_max - R_min) of the optimal average reward (where lambda can be arbitrarily large). This does not provide much in the way of guarantees.
My final feeling about the paper is that it introduces a mostly heuristic method, which can provide some empirical benefit when properly tuned. It wasn't clear to me, however, that this fear model offers a generic method or that is capable of achieving the goals the authors mention.
The paper studies catastrophic forgetting, which is an important aspect of deep reinforcement learning (RL). The problem formulation is connected to safe RL, but the emphasis is on tasks where a DQN is able to learn to avoid catastrophic events as long as it avoids forgetting. The proposed method is novel, but perhaps the most interesting aspect of this paper is that they demonstrate that “DQNs  are susceptible to periodically repeating mistakes”. I believe this observation, though not entirely novel, will inspire many researchers to study catastrophic forgetting and propose improved strategies for handling these issues.
The paper is accurate, very well written (apart from a small number of grammatical mistakes) and contains appealing motivations to its key contributions.
In particular, I find the basic of idea of introducing a component that represents fear natural, promising and novel.
Still, many of the design choices appear quite arbitrary and can most likely be improved upon.
In fact, it is not difficult to design examples for which the proposed algorithm would be far from optimal.
Instead I view the proposed techniques mostly as useful inspiration for future papers to build on. As a source of inspiration, I believe that this paper will be of considerable importance and I think many people in our community will read it with great interest. The theoretical results regarding the properties of the proposed algorithm are also relevant, and points out some of its benefits, though I do not view the results as particularly strong.
To conclude, the submitted manuscript contains novel observations and results and is likely to draw additional attention to an important aspect of deep reinforcement learning. A potential weakness with the paper is that the proposed strategies appear to be simple to improve upon and that they have not convinced me that they would yield good performance on a wider set of problems.
We thank AnonReviewer1 for a clear and constructive review. We are encouraged that you recognize the importance of the problem addressed and the novelty of the methods. Per your suggestions, we have polished the paper, fixing several of the typos that had made it into the first draft. The reviewer’s point that many aspects of the algorithm can likely be improved upon in future work is well-taken. We hope that this is just one of the first among many papers to improve with respect to these fundamental problems.
We would like to thank the reviewers for taking the time to leave thoughtful reviews. Given this feedback, we have significantly improved the draft and hope the reviewers and area chair will take this into account when assessing the final scores.
For example, the harsh score from reviewer 2 owes largely to a mistake in one theorem that has since been fixed in the newest version. We are also grateful to the folks at the reproducibility project who noted the commendable clarity and reproducibility of our paper, algorithm and empirical findings. Please find specific rebuttals to each reviewer as replies to the respective reviews.

Meta-review:
role: PC
data: comment: This paper presents an interesting idea that is related to imitation learning, safe exploration,
and intrinsic motivation. However, in its current state the paper needs improvement in clarity. There are also some concerns about the number of hyperparameters involved. Finally, the experimental results are not completely convincing and should reflect existing baselines in one of the areas described above.


Decision: Reject