Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a computationally-efficient method to detect adversarial examples in reinforcement learning models. The detection method is based on the curvature of the loss landscape around the inputs, which is shown to have larger negative value for clean examples compared to adversarial ones. The experiments on Atari environment models show the effectiveness of the method.

main_review: I reviewed this paper for NeurIPS. The paper got rejected there mainly because of the issues with the experiments and the method. The authors just changed the title and resubmitted to ICLR. So I thought it's better to copy my NeurIPS review here. 
summary_of_the_review: The paper is well-written and backs up the experimental results with mathematical intuition and analysis. I, however, have the following questions and concerns.

- Adversarial examples in RL: Studying adversarial robustness in RL scenarios is interesting and important. However, most of the current research on adversarial examples is focused on classifier models and image domain. It also seems that the arguments and analysis presented in this paper could be directly applied to the classification problems as well. In fact the baseline of Roth et al. and all the attack methods used have been designed for image classifiers. Is there a particular reason that the paper focuses instead on RL? How will the proposed methods work e.g., on CIFAR10 and CNNs? The paper notes the real-time constraint in RL setting as an added constraint for detectors. But please note that, even without any constraints, designing a reliable detector is still an open problem for image classifiers.

- The paper posits that, as shown in figure 1, the cost has larger negative curvature at clean examples compared to adversarial examples. The detection method is, however, specified as $|\mathcal{L}-\mathcal{\bar{L}}|>\tau$. Shouldn’t it be $(\mathcal{L}-\mathcal{\bar{L}})>\tau$ (without the absolute value)? Also, the objective function of the adaptive attack is defined as: $\min (J(s_{adv}) + ||s-s_{adv}||^2 + \mathcal{L}(s_{adv}))$. Shouldn’t it be $\min (J(s_{adv}) + ||s-s_{adv}||^2 + |\mathcal{L}(s_{adv})-\bar{\mathcal{L}}|)$ (or maybe $\min (J(s_{adv}) + ||s-s_{adv}||^2 + |\mathcal{L}(s_{adv})-\bar{\mathcal{L}}|)$) to match it to the detection rule? With the current attack formulation, the curvature of the adversarial example could end up being lower than that of clean examples, which would be flagged as adversarial. Plotting L(s) (as in figure 1) for adaptive adversarial examples with different objectives could help visualize this.

- The paper does not report the attack success rates on the undefended models. This is an important sanity check to make sure that the attacks are implemented correctly and also helps with better evaluating the effect of the detector.

- The adaptive attack results are reported for the case that the decrease in the attack success is at most 10%. This doesn’t sound like a good metric. The paper should instead define the attack success rate (ASR) as the rate of examples fooling both the model and the detector, and then report the results for the hyperparameters for which the ASR is highest. The same thing holds for the results of non-adaptive attacks. For example, Table 1 reports the detection rate, but it is not clear what is the ASR and whether the attack hyperparameters are optimized to maximize it. For instance, decreasing the attack budget reduces both the detection rate and the misclassification rate. Thus, there could be a setting that even a lower attack budget results in a higher overall ASR (fooling both model and detector). Overall, it is hard to evaluate the proposed detector without having the best ASR results.

- Table 1 shows that the second-order detector works better than the first-order one. But why is the detection rate of Roth et al., so low? E.g., <5% TPR for CW attack. It seems that their detection method didn’t really need an adaptive attack to break it. Their paper [ICML ’19] and a few follow up works, however, have shown it achieves a pretty high detection rate for non-adaptive attacks on image classifiers.

- Section 5 mentions that the “feature matching” has been used as the first attempt of an adaptive attack. Is this the same attack used for breaking the Roth et al method? If yes, did the authors investigate using the feature matching for designing an adaptive attack specifically for the proposed SO-DATA detector? Please note that the feature matching attack should be designed with the exact same features that the detector uses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides a fast method to detect adversarially attacked states during reinforcement learning. The main improvement over previous (Roth) approach is to explicitly involve analysis of what's happening along the gradient direction. They relate second-order
behavior with common adversarial considerations to motivate the approach.  Results show a large improvement over a previous
random-sampling approach.  They also show the method resists even adversary-aware attacks fairly well.
main_review: The paper was well-written and it is good that they considered not just a detection method but also tried to attack it.

I found I had to reread the introduction to section 3.2 to really understand well.  It might be an idea to split this paragraph
in two.  At first introduce a "core concept", (so reader knows to definitely not skim this paragraph).
Then continue with consideration of Hessians, Taylor approximations and motivations for your fast method.

Another minor possibility would perhaps explicitly connect Eqn 14 to an actual strong attack (p==2 is Eqn (2)).

It seems the best adversary-aware attacks of SO-DATA still performed about as well as adversary-unaware
attack on the Roth approach.

Was the number of random perturbations for the Roth method given?
Was SO-DATA  (~ 2 J evals + gradient) also faster then Roth?

summary_of_the_review: The paper provided and adversarial detection method and a good attempt to break the method.
While strongly connected with methods in non-reinforcement settings, I'm unaware of a similar approach
for reinforcement learning. I enjoyed reading this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers the adversarial example detection in RL domain and it claims they are the first to do so.
main_review: - The local curvature of the neural network is considered in a lot of works such as DeepFool and it is not a novel idea. Applying it simply on RL domain sounds like a marginal contribution.
- The mathematical formulations in the paper sounds more decorative rather than adding value to the paper.
- The simulations and experiments are not comprehensive. The reviewer expects comparison with more baselines.
summary_of_the_review: Although they claim that they are the first to apply such an idea (detection of the adversarial example in RL domain), the contribution is marginal as it borrows a lot from the literature. Moreover, the performance evaluation is not comprehensive.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method on detecting RL adversarial attack. In particular, it uses the curvature of the cost function. The insight is that adversarial states have larger curvature than the normal states. To measure the curvature, it approximates the second-order hessian matrix using a first-order approximation. It further provides empirical evidence on the observation and some theoretical motivation. In the end, it evaluates the proposed method against two baseline methods on seven tasks under seven different kinds of attacks (including two adversary aware attacks) and show superior performance.
main_review: This paper addresses an important topic: detecting adversarial attacks on RL models. Given RL models have been increasingly widely used, it is important to make sure of their robustness. The proposed algorithm SO-DATA is a simple yet effective method for detecting adversarial attacks on the DRL models. I like the observation on the difference between the adversarial example and the normal examples in terms of the loss curvature this paper makes and how it leverages this observation to develop the method. Sec3.3 further provides a nice theoretical motivation for the use of negative curvature as the criteria for detecting the adversarial examples. The experiment results especially those on detection aware adversaries show the effectiveness of the proposed method.

Minor writing issue:
The font size of each figure’s caption, x label, y label and legend is too small.

Questions:
What is the intuition behind the superior performance of SO-DATA compared with Roth et al.?

“In our grid search over these hyperparameters we found that there is a trade-off between the attack success rate and the detection of the perturbations. In other words, if we optimize the perturbation to be undetectable the success rate of the perturbation” -- Can you give some experiment results regarding this trade-off?

summary_of_the_review: Pros:
- simple, theoretically well-motivated, and effective approach
- writing is easy to follow
- strong empirical performance improvement

Cons:
- Evaluations are only on Atari Games
review_title: null

Meta-review: This paper proposes a computationally-efficient method to detect adversarial examples in reinforcement learning models. The detection method is based on the curvature of the loss landscape around the inputs, which is shown to have larger negative value for clean examples compared to adversarial ones. The experiments on Atari environment models show the effectiveness of the method.

The paper is well-written and backs up the experimental results with mathematical intuition and analysis. 

However, the baseline of Roth et al. and all attack methods used have been designed for image classifiers. 

If the authors decide to focus on RL, the attack methods should be tailored to RL. The word  “worst-case” in the title is misleading, since the attacks used in the paper are not optimal for RL algorithms. This reduces the credibility of the claimed successful detection. 

If the authors decide to frame this work as introducing a new property of adversarial examples which can be applied to other tasks, the authors should test this method on other tasks such as benchmark image classification datasets (for example CIFAR10). 

With the current experiment section, it is unclear whether this method works in RL applications since the authors use attack methods designed for image classifiers rather than RL algorithms (Please refer to the following papers for some existing RL attack methods). It is also unclear whether this paper introduces a new property of adversarial examples that is general. 

- Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies
- Ezgi Korkmaz. Nesterov momentum adversarial perturbations in the deep reinforcement learning domain. 
- Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks.
- Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations.
- Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary.
Decision: Reject