Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses an important and timely topic in a creative way. I consider it to have three flaws (and one good idea).

1) insufficient context of what is known and had been studied before (in shallow RL), for example within the field of “robust RL”. A good place to start might be with the work of Shie Mannor.

2) an ill-defined general problem setup. Does it make sense to do post-hoc labeling of certain actions as “catastrophic” if the agent is not informed about that metric during learning? Training a system to do one thing (maximize reward), but then evaluating it with a different metric is misleading. On the training metric, it could even be that the baseline outperforms the new algorithm? So I’d want to see plots for “average reward” in fig 3 as well. Also, what would the baseline learn if it was given large negative rewards for entering these otherwise invisible “danger states”?

3) a somewhat ad-hoc solution, that introduces new domain-specific hyperparameters (k_r, k_lambda and lambda) a second deep network and and two additional replay memories. In terms of results, I’m also unsure whether I can trust the results, given the long-standing track-record of cart-pole being fully solved by many methods: is DQN an outlier here? Or is the convnet not an appropriate function-approximator? Actually: which exact variant “state-of-the-art” variant of DQN are you using?

The good idea that I encourage the authors to pursue further is D_d, this set of rare but dangerous states, that should be kept around in some form. I see it as an ingredient for continual learning that most typical methods lack -- it is also one of the big differences between RL and supervised learning, where such states would generally be discarded as outliers.

Given my comment a couple of weeks ago, and the prompt response (“we implemented expected SARSA”), I would have expected that the paper had been revised with the new results by now? In any case, I’m open to discussing all these points and revising my opinion based on an updated version of the paper.

Minor comment: the bibliography is done sloppily, with missing years, conference venues and missing/misspelled author lists, e.g. “Sergey et al. Levine”. I also think it is good form to cite the actual conference publications instead of arXiv where applicable.
review_title: null

final_score: rating: null
review_content: I would like to echo R1 that Q-learning suffers as expected and past work has addressed such issues. I would like the authors to position their research contributions within the broader literature. 
review_title: baselines

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a heuristic for avoiding large negative rewards which have already been experienced by distilling such events into a "danger model". The paper is well written including some rather poetic language [*].

The heuristic is evaluated in two toy domains. I would think that in order to properly evaluate this one would use a well known benchmark e.g. Atari. Atari seems particularly apt since those games are full of catastrophes (i.e. sudden death).

[*] this reviewer's favourite quotes:
"Imagine a self-driving car that had to periodically hit a few pedestrians in order to remember that it’s undesirable."
"The child can learn to adjust its behaviour without actually having to stab someone."
"... the catastrophe lurking just past the optimal shave."
review_title: heuristic for avoiding large negative rewards

final_score: rating: 4: Ok but not good enough - rejection
review_content: - The topic of keeping around highly rewarding or dangerous states is important and has been studied extensively in the RL literature. After the pre-review comments, authors do mention that they compared against expected SARSA but I would really like to see these and other extensive baselines before accepting this paper. 

- There is also an increasing amount of literature of using reward replay buffers in deep RL agents (c.f. Jaderberg, Max, et al. "Reinforcement learning with unsupervised auxiliary tasks.", Blundell, Charles, et al. "Model-free episodic control." , Narasimhan et al. "Language understanding for text-based games using deep reinforcement learning"), which could perhaps reinforce the agent to avoid revisiting catastrophic states. 

- Overall, the approach presented is not very principled. For instance, why isn't catastrophe directly provided as a signal to the learner instead of a separate model? 
review_title: null

final_score: rating: null
review_content: Q-learning is learning off-policy about the greedy policy, so by construction it cannot learn about the consequences of the exploration epsilon in its behavior policy. There is well-established "cliff" style domains that study this effect when exploration can be catastrophic, and there are numerous well-established algorithmic solutions, namely Expected SARSA -- which can trivially replace the Q-learning update inside the DQN setup. I'd strongly encourage the authors to report those results as the (fair) baseline.
review_title: SARSE

Meta-review: This paper presents a few interesting ideas, namely the idea of keeping around a set of "danger states" and treating these states with some special consideration in reply to make sure that their impact is not neglected after collecting a lot of additional data.
 
 However, there are two main problems: 1) the actual implementation here seems fairly ad-hoc, and it's not at all clear to me that this particular algorithm (building a classifier with equal numbers of good and danger states, and then injecting an additional reward into the Q-learning task based upon this classifier), is the right way to go about this. The presentation is also difficult to follow, and the final results imply aren't that compelling (though this is improving after the revisions, but still has a way to go. We therefore encourage the authors to resubmit their work at a future conference venue.
 
 Pros:
 + Interesting idea of keeping around danger states and injecting them into training
 
 Cons:
 - Algorithm doesn't seem that well motivated
 - Presentation is a bit unclear, takes until page 6 to actually present the basic approach.
 - Experiments aren't that convincing (better after revisions, but still need work)
Decision: Reject