Task: Language Identification
Input (Spanish, English): This paper addresses an important and timely topic in a creative way. I consider it to have three flaws (and one good idea).  1) insufficient context of what is known and had been studied before (in shallow RL), for example within the field of “robust RL”. A good place to start might be with the work of Shie Mannor.  2) an ill-defined general problem setup. Does it make sense to do post-hoc labeling of certain actions as “catastrophic” if the agent is not informed about that metric during learning? Training a system to do one thing (maximize reward), but then evaluating it with a different metric is misleading. On the training metric, it could even be that the baseline outperforms the new algorithm? So I’d want to see plots for “average reward” in fig 3 as well. Also, what would the baseline learn if it was given large negative rewards for entering these otherwise invisible “danger states”?  3) a somewhat ad-hoc solution, that introduces new domain-specific hyperparameters (k_r, k_lambda and lambda) a second deep network and and two additional replay memories. In terms of results, I’m also unsure whether I can trust the results, given the long-standing track-record of cart-pole being fully solved by many methods: is DQN an outlier here? Or is the convnet not an appropriate function-approximator? Actually: which exact variant “state-of-the-art” variant of DQN are you using?  The good idea that I encourage the authors to pursue further is D_d, this set of rare but dangerous states, that should be kept around in some form. I see it as an ingredient for continual learning that most typical methods lack -- it is also one of the big differences between RL and supervised learning, where such states would generally be discarded as outliers.  Given my comment a couple of weeks ago, and the prompt response (“we implemented expected SARSA”), I would have expected that the paper had been revised with the new results by now? In any case, I’m open to discussing all these points and revising my opinion based on an updated version of the paper.  Minor comment: the bibliography is done sloppily, with missing years, conference venues and missing/misspelled author lists, e.g. “Sergey et al. Levine”. I also think it is good form to cite the actual conference publications instead of arXiv where applicable.
Output (English): en