Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed a framework to learn RL agents that are robust to lp-norm adversarial attack. Based on existing robustness verification bounds for neural networks, the paper proposed two strategies (constructing upper bound of the loss, and minimizing the overlap between output bounds) and integrate them with three popular RL methods (DQN, A3C and PPO). The paper also propose an efficient method to approximate the worst-case reward, which is generally NP-hard to compute. Extensive experiments demonstrate the efficiency and effectiveness of the proposed methods in most of the experiments.
main_review: The paper is an interesting application of the robustness verification bounds for neural networks to address the lp-norm adversarial attack problem in the context of reinforcement learning. I think the paper is well-structured and contains interesting discussions to this important problem. Below are some questions/detailed comments:
- At the end of Sec 2.1, the paper claims that "our framework is not limited to `p-norm perturbation and in fact can be easily extended to semantic perturbations (e.g. rotations, color/brightness changes, for vision-based deep RL agents (e.g. atari games) by leveraging the techniques proposed in [26]." I think if this can be achieved, then this may be a very important step towards robust RL against adversarial attack. From my perspective, certified defense for unrestricted (to lp-norm) adversarial attack is much more challenging if not impossible. And the bounds used for deriving the proposed method heavily relies on the lp-norm attack assumption. So please make sure this is not an overclaim and more detailed discussions on this will be appreciated.
- The motivation/rationale behind the design in Sec 3.1 - Sec 3.3 need to be elaborated in more details. Right now the paper directly presents the definitions of the adversarial loss without detailed explanation and I spent a considerable amount of time parsing the equations. 

For the method in Sec 3.1 (and Sec 3.2), the logic seems to be, whenever a sampled action is better than other action, we want to make sure it is still better than that action by a margin. Here are the questions: (1) This is only for one direction. What about the other direction: whenever an action has worse Q value than another action, we want to keep this consistent and make sure it is still worse than that action? (2) Why the margin is designed to be half of Q_diff? Is the algorithm sensitive to this choice?

For the method in Sec 3.3, I found it difficult to understand the loss function. But basically we want to get a conservative estimate of the action probability, based on whether it is an optimal action. More explanations needed here.

- From Sec 3.1 to Sec 3.2, it seems Approach #2 has better performance than Approach #1. Any insights on this? Do you think this is because Approach #2 has some computational advantages? And is it possible to apply Approach #2 to PPO, e.g. by replacing the sum over actions with an integral and using sample-based approximation?
limitations_and_societal_impact: The potential negative societal impact has been properly discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a class of defenses for adversarial norm-ball perturbations on observations in RL, compatible with a diverse range of algorithms such as DQL, A3C, and PPO.  They also provide a new evaluation criterial, greedy worst-case reward, to evaluate their approach.
main_review: This paper integrates successful work from supervised settings into defenses for RL in a way which, to the best of my knowledge, is novel and stands to elevate work in the subfield.  

The biggest concern is whether the results are replicable.  Mainly, (a) are the results statistically significant and (b) are to what extent was architectural tuning and hyper parameter search responsible for the differences.  For an example of (a), there are overlaps on some of the standard deviations of some of the results (like Fruitbot \epsilon=1/255 in Figure 2), and they are not highlighted, so it's hard to tell how many of the results have these sorts of overlaps.  Bar charts might make this easier to see, or a highlighting scheme which highlights the overlaps.  For (b) there should a discussion of what the procedure architectural tuning and hyper parameter search and why we should expect these results to be about the method itself.


minor comments: 
* on line 67 there is a claim that all existing work on adversarial attacks in RL deals with norm-ball perturbations.  This is not true, as there is a line of research around adversarial attacks in the form of policies of other agents in the same environment, for instance:

Gleave, Adam, et al. "Adversarial Policies: Attacking Deep Reinforcement Learning." International Conference on Learning Representations. 2019.

Sun, Yiwei, et al. "Node injection attacks on graphs via reinforcement learning." arXiv preprint arXiv:1909.06543 (2019).

Ilahi, Inaam, et al. "Challenges and countermeasures for adversarial attacks on deep reinforcement learning." arXiv preprint arXiv:2001.09684 (2020).

* on line 77, I don't know what is meant by "these works do not study how to train more robust deep RL agents", since it seems that is exactly the point of those papers
* on line 143 it's unclear to me what is meant by "strict", and why minimizing this upper bound would imply minimizing the true function.
* I couldn't find the definition of \bar{A_Q} mentioned on line 164
* In table the best methods should be bolded regardless of if they are that of the authors

------
** Update to review **

Thank you for your detailed response to my review.  Your response addressed the bulk of my concerns in points #1 and #2.  I also appreciate the detailed response to points #3-#8, and believe the proposed changes will improve the paper.

More specifically, the detailed comparison between architectures in point #2 made it easy for me to check with past work and compare 1 to 1 that there was more tuning in past architectures, and convinced me that any difference would not explain the results.  Moreover, in point #1 the promise to fix the presentation of the statistics and to fix the bolding to make it clear to readers which results are significant and which are not should greatly improve the clarity of the results.  Given these changes, I am increasing my score.



limitations_and_societal_impact: Beyond the reproducibility, the main limitation I see to this work is that the attack losses are not considered sequentially, and so it is possible that these losses can compound over time.  I do not think that it's necessary to solve this for the paper to be valuable, but I think it should be highlighted as a limitation and a place for future work, so readers do not get an impression that the defense is exhaustive.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper presents a well-thought and clear paper on adversarial deep RL that robustifies RL against adversarial attacks. Experimental evaluation is thorough on several benchmarks.
main_review: The presented motivation and writing is clear and concise. This work is clearly well thought through and well executed with several comprehensive experiments. The algorithm looks valid and the results are impressive. Overall, this is a solid paper with little to no flaws in my opinion (hence the short review). There are several novel contributions -- formulation of RADIAL, combination with several SOTA RL methods, new evaluation metrics, comprehensive experiments. Perhaps the only thing that could make the paper better is some real-world experiments, which is discussed in section 5.

 One minor concern is the similarity to RARL [21]. Discussion or comparison with it may further strengthen the paper. 


limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper introduces a new robust AI approach to strengthen the RL model because tiny perturbation on input to the policy network could greatly influence the action that the target agent takes.
main_review: The paper states RL imposes unique challenges for using existing robustness mechanisms to the domain of RL. These challenges include credit assignments and a lack of stationary training. The deep RL algorithm learns a policy. The policy is a deep neural network. Launch an adversarial attack in the context of classification or RL has no big difference. That is to perturb the input to the neural network. To the best of my knowledge, while applying existing certified AI methods to strengthen a neural network, the certified methods should not be influenced by the training process nor the loss function but the neural network only. While reading this paper, I tried to figure out how the so-called unique challenges could impact the effectiveness of certified learning. The authors spend quite a lot of space on introducing the background of RL and some space to introduce adversarial attacks. I would suggest the authors could provide me with a more concrete analysis (maybe an example) demontrating why an existing certified learning method for a classification task does not work for the RL context. Without a clear demonstration and convincing explanation, it is hard for me to change my score. 
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact.
review_title: null

Meta-review: I thank the authors for their submission and active participation in the discussions. The paper tackles an important problem as robustifying deep RL agents is crucial for making progress towards real-world applications. Reviewers appreciate its significance [Hgqp,naEG,hFQ2], novelty [naEG], and interesting discussion [Hgqp], as well as its thorough experiments [hFQ2]. During rebuttal and discussion, reviewer naEG was compelled by the author response and clarifications, as well as reviewer hFQ2's arguments. I am discounting 8ojS's clear stance against the paper as I believe the author response sufficiently addresses their concerns and they haven't indicate otherwise during the discussion. I also appreciate the author's openness about potential limitations of their work. Overall, I recommend acceptance and encourage the authors to further improve their paper based on the reviewer feedback.
Decision: Accept (Poster)