Detecting Worst-case Corruptions via Loss Landscape Curvature in Deep Reinforcement Learning

Abstract: The non-robustness of neural network policies to adversarial examples poses a challenge for deep reinforcement learning. One natural approach to mitigate the impact of adversarial examples is to develop methods to detect when a given input is adversarial. In this work we introduce a novel approach for detecting adversarial examples that is computationally efficient, is agnostic to the method used to generate adversarial examples, and theoretically well-motivated. Our method is based on a measure of the local curvature of the neural network policy, which we show differs between adversarial and clean examples. We empirically demonstrate the effectiveness of our method in the Atari environment against a large set of state-of-the-art algorithms for generating adversarial examples. Furthermore, we exhibit the effectiveness of our detection algorithm with the presence of multiple strong detection-aware adversaries. 

Reviews and author responses:

 I agree with your summary, though I see this paper more as an adversarial example paper than an RL one, especially that, as you mentioned, the proposed method doesn't seem to be tied to RL. 

 Hi Reviewer x4K3,

Thank you for the reply! I understand your concerns and believe they are great research questions. In particular, the proposed method can be potentially tried in the image setting givens it seems not deeply tied to the DRL setting. 
In fact, if the paper can address these questions, I will also give a 10 to the paper. 

The question remains is if the paper's current contribution is enough as a preliminary work for adversarial DRL detection, which is an important topic but not well studied before. 

In summary, it seems we have:

The reasons to accept the paper include: 1.important, not well-studied topic. 2.interesting observation regarding the curvature of the loss landscape. 3.effective method with theoretical justification and experiment results in the RL setting.

The reason to reject it is mainly: 1.no exploration of the proposed method in the image setting and thus no connections made on adversarial RL and image settings.


 Hello Reviewer LjzU and thanks for your comment. 

By saying attacks and defenses are less explored in RL, I meant it is harder to evaluate the merit of the proposed method given that there are not many good benchmarks/baselines in RL. Sorry for the confusion. 

OTOH, for image data and CNNs, we know what is the SOTA for, say, CIFAR10. Also, over the past few years, we have gained an understanding of what might work and now better know how to evaluate defenses.

As far as I know, existing defenses that actually work are all some form of the adversarial training, i.e., training with adversarial examples in a brute-force manner. Other defenses that relied on some statistics or property of adversarial examples have been broken, e.g., Roth et al. (which this paper compares to as well). 

This paper proposes a new property: the curvature of the loss landscape has larger negative value around clean examples than adversarial ones. This is interesting. Is it a general property of adversarial examples? I can't tell after reading this paper. If this method gives nontrivial results for CIFAR10 (even if it's not SOTA), I'd give this paper a 10. 

But, given that no experimental results are provided for image data, let's assume it doesn't work for CIFAR10. Then, what are the implications? Can we have a defense method that works for RL applications but not for images/CNNs? Why? Without knowing the answer to these questions, I'm not sure what this paper contributes to the field. The paper just seems incomplete research-wise. 

There are many things to like in this paper. I hope the authors consider further improving the paper by clarifying these points. 


 I want to thank the authors for the reply with additional experimental results. I decide to keep my initial assessment and believe the work is good enough given its scope focuses on the RL setting where attacks/defenses are relatively less explored.

 Hi Reviewer x4K3,

As you also mentioned in your comments, attacks and defenses have been less explored in RL applications. I wonder if we can treat this work just as a work for RL application?

It will be great for the work to also explore if the current method works for CNN and image data. However, even without this suggested experiment, the current work seems to still contribute to the understanding of attack/defense in RL application (which is less explored).


 We respectfully disagree with the reviewer. The reason we were able to suggest a method to detect adversarial formulations lies in the fact that our aim is to understand the intrinsic difference between adversarial examples and clean examples. We realized that this difference can only be explained by a difference in the second order part of the Taylor approximation. Hence, this is described and explained in Section 3.1 and Section 3.2 in detail. Our method, as it is motivated theoretically in Section 3, obtains state-of-the-art detection rates in several different adversarial attacks against deep reinforcement learning policies. 

 We thank the reviewer for their valuable feedback and for acknowledging our contributions. We are very glad to hear the reviewer’s acknowledgement that our method is theoretically well-motivated and addresses an important topic.

**What is the intuition behind the superior performance of SO-DATA compared with Roth et al.?**

The intuition for the superior performance of SO-DATA compared to Roth et al. is connected to what precisely is being measured by each method. The measurement for Roth et al. is based on computing the change in the log-odds ratio between each pair of actions in a given state under random perturbations, and the original Roth et al. paper observes that this change in log-odds differs between natural and adversarial examples. 

As discussed in Section 3.1 and 3.2, this behavior cannot be explained if we assume the policy is locally well-approximated by its first-order Taylor approximation. Rather, differing curvature at clean and adversarial examples is necessary to explain the different values under random perturbations. We further hypothesize that this difference in curvature is fundamentally the result of the use of local optimization procedures to find adversarial examples. Thus, the SO-DATA method, which attempts to directly measure curvature, tends to outperform the Roth et al. method.

**Can you give some experiment results regarding this trade-off?**

Below we attached the Table for Carlini & Wagner with detection aware adversaries allowed to compromise on the policy performance degradation at most 10% and between 10% and 30%. As is expected, as the Carlini & Wagner formulation converges to an adversarial state that either does not change the optimal action in a given state or the change caused in the action is relatively less significant on the policy performance degradation in terms of the obtained expected cumulative discounted rewards, the TPR values also get smaller. This indicates that the detection method correctly classifies the adversarial examples.

Detection Method-Attack Method |RiverRaid |RoadRunner| Alien |Seaquest| Boxing |Pong |Robotank
-----------------------------------|--------|--------------|-----------|--------------|-----------|-------------|--------------
SO-DATA C&W Detection Unaware  | 0.910  |    0.988   | 0.945    |    0.723   |   0.856   |    0.850  |   0.713
SO-DATA C&W Detection Aware [0-10%] | 0.650  |   0.849   |    0.445  |     0.381 |    0.710   |   0.712   |   0.657
SO-DATA C&W Detection Aware [10-30%]| 0.543 |   0.772   |    0.341   |    0.292  |     0.598 |    0.629   |   0.549


 We thank the reviewer for their valuable review and for acknowledging our contributions. We are very glad that the reviewer enjoyed reading the paper.

**Was the number of random perturbations for the Roth method given? Was SO-DATA (~ 2 J evals + gradient) also faster then Roth?**

The number of random perturbations for the Roth et al. method was 256, which matches the choice in the original paper. The SO-DATA method was indeed faster as the 2 J evals + one gradient can be computed with just two forward passes and one backward pass through the network.


 We have gone back and forth about the scope of the paper a few times. The authors insist that they aimed for RL applications and the paper should be judged as such. The proposed method, however, seems generic enough to be applied to tasks other than RL. As a community, we have made some progress in understanding adversarial examples and establishing baselines on image data. So my question is: can the proposed method be applied to CNNs and image data? If yes, then how does it work for, say, CIFAR10? With the current experiment section, there's no way for readers to know whether this method just works in RL applications (where attacks and defenses are less-explored) or it introduces a new property of adversarial examples and can be applied to other tasks as well. So I think, at its current state, the paper looks incomplete and I'm not sure if it can contribute much to the field.  


 If the reviewer had in fact read the paper, the reviewer would have realized that there have indeed been changes in the paper. In the NeurIPS submission, after answering all of this reviewer’s questions, concerns and comments in detail, we received a simple response from the reviewer:

“Adversarial examples in RL: The related work on adversarial RL doesn't answer my question that ‘How will the proposed methods work e.g., on CIFAR10 and CNNs?’”

Motivated by this reply we added “in Deep Reinforcement Learning”  in the title to make sure that the reader understands that our paper is about deep reinforcement learning, and we added a whole section on why we work on deep reinforcement learning. Below once again we respond to the reviewer’s identical questions  in detail.

**"Adversarial examples in RL?"**

There is already a new section about adversarial examples in deep reinforcement learning and the motivation to work on deep reinforcement learning in the paper motivated by the reviewer’s previous response.

**Attack success rate (ASR)**

The “misclassification rate” (i.e. “attack success rate ”) is a metric defined  for image classification, which does not make sense in the reinforcement learning setup. In reinforcement learning, a policy $\pi(s|a)$ is used to determine which action $a$ to take in each state $s$. After taking action $a$ in state $s$ the agent receives a reward r(s,a). The goal is to learn a policy $\pi$ which maximizes the expected cumulative discounted rewards $E_{a \sim \pi(s|\cdot)} \sum_{t=0}^\infty [\gamma^t r(s_t,a_t)|s_0=s,a_0=a ]$ where $\gamma$ is the discount factor. Thus, in reinforcement learning the only source of ground truth is cumulative rewards obtained by the agent. For these reasons, we use the percentage decrease in cumulative rewards as our measurement for attack success. We then evaluate the detector by fixing a minimum level for the percentage decrease and searching for the parameters for the detection-aware adversary that minimize the TPR while still maintaining this minimum level.

While the corresponding analogy of “ASR” in reinforcement learning setup is the expected cumulative rewards obtained by the agent, we do vary the attack budget and record the trade-off between the TPR values and the rewards obtained by the agent (i.e. as the obtained rewards increase the TPR values will go down due to the adversarial examples not changing the optimal action $a^*$ in a given state$^1$). Hence, the results are consistent with Table 1 that SO-DATA is higher than Roth et al. (2019) and FO-DATA at the same relative rates.

[1] An increase in the obtained rewards indicates that the optimization procedure used to compute adversarial examples is unable to converge to a local optimum which changes the optimal action in a given state. This in turn means that the curvature used by our detection method will tend to be more similar to that at clean states, leading to a decrease in TPR.


 **Section 5 mentions that the “feature matching” has been used as the first attempt of an adaptive attack. Is this the same attack used for breaking the Roth et al method?**

The attack used in Section 5 is the same one used for breaking Roth et al. Motivated by the reviewer’s comments we did indeed further test “feature matching” based adversaries with features more tuned to our detection method at the time when this question was originally asked. 

Let $Q(s)$ represent the vector of logits for state $s$. Given a state $s_0$, we find a clean state $s’$ where the action taken by the agent is different than the one in state $s_0$. We then use PGD to find $s$ in an $\epsilon$ ball around $s_0$ minimizing

$||Q(s) - Q(s’)||^2 + \alpha ||\nabla J(s) - \nabla J(s’)||^2 + \beta ||Q(s’ + \epsilon \textrm{sign} \nabla J(s’)) - Q(s + \textrm{sign}\nabla J(s)||^2$.

These three terms correspond to the three sets of features used in our detection method: the logits of the state, the gradient at the state, and the logits at the state perturbed by $\epsilon$ times the sign of the gradient. The values $\alpha$ and $\beta$ are tunable parameters we use to optimize the trade-off between the three different types of features. We grid-searched over $\epsilon$ from $10^{-1}$ to $10^{-3}$, $\alpha$ between $0.1$ to $10$, and $\beta$ between $0.1$ to $10$ to find the setting that minimized TPR while maintaining the same  minimum level for the percentage decrease in the obtained rewards of the agent.

For only two of the games the TPR for SO-DATA with this new “gradient feature matching” adversary is a few percentage points lower than that of the feature matching adversary used to break Roth et. al. (i.e. a few percentage points lower than the SO-DATA results in Table 2). Thus our original Carlini-Wagner based adaptive adversary is more effective at reducing the TPR of our detection method (results for the original adaptive adversary are in Table 3). One important thing to note on this detection aware cost function is that as we make the perturbations less detectable the perturbations lose the ability to cause a drop in the obtained final rewards. Thus, less detectable adversarial perturbations are behaving like “clean” samples. Hence, these results support the claim made in Section 3.2 on lines 131-144.

**Shouldn’t it be $\mathcal{L} - \bar{\mathcal{L}}$ (without the absolute value)?**

Whenever $\mathcal{L}$ is multiple standard deviations from its mean $\bar{\mathcal{L}}$, the observed state is a clear outlier with respect to its local curvature. This is why we use the absolute value in the detection method. However, as suggested by the theoretical motivation, for both the standard and detection-aware adversaries $\mathcal{L}$ is larger than $\bar{\mathcal{L}}$ in more than a 0.99 fraction of states.

**The objective function of the adaptive attack...With the current attack formulation, the curvature of the adversarial example could end up being lower than that of clean examples, which would be flagged as adversarial.**

As mentioned above, in the detection aware setting $\mathcal{L}$ is larger than $\bar{\mathcal{L}}$ in at least a 0.99 fraction of states. So the concern that the curvature at adversarial examples produced by the detection aware adversary might “overshoot” and be lower than $\bar{\mathcal{L}}$ does not happen empirically. The reason for this seems to be that the original objective of finding an effective adversarial example via Carlini-Wagner, and the objective of minimizing $\mathcal{L}$ are in opposition.


**But why is the detection rate of Roth et al., so low? E.g., <5% TPR for CW attack.**

In the original Roth et al. (2019) paper the authors report PGD TPR results with FPR=0.01 and they report Carlini&Wagner TPR results with FPR=0.048 in the same table. Thus, we are not sure the exact TPR results of Carlini&Wagner with FPR=0.01 in image classification setup. However, it might be that Carlini&Wagner TPR results were quite low for FPR=0.01 and that is why the authors might have decided to report with FPR=0.048 instead. Additionally Roth et al. do not report the parameters used in the Carlini&Wagner attack, in particular the confidence parameter. In our paper we used a high value for the confidence parameter as this is known to defeat some defenses/detection methods Tramer et al. (2020).




This paper proposes a computationally-efficient method to detect adversarial examples in reinforcement learning models. The detection method is based on the curvature of the loss landscape around the inputs, which is shown to have larger negative value for clean examples compared to adversarial ones. The experiments on Atari environment models show the effectiveness of the method.
 I reviewed this paper for NeurIPS. The paper got rejected there mainly because of the issues with the experiments and the method. The authors just changed the title and resubmitted to ICLR. So I thought it's better to copy my NeurIPS review here.  The paper is well-written and backs up the experimental results with mathematical intuition and analysis. I, however, have the following questions and concerns.

- Adversarial examples in RL: Studying adversarial robustness in RL scenarios is interesting and important. However, most of the current research on adversarial examples is focused on classifier models and image domain. It also seems that the arguments and analysis presented in this paper could be directly applied to the classification problems as well. In fact the baseline of Roth et al. and all the attack methods used have been designed for image classifiers. Is there a particular reason that the paper focuses instead on RL? How will the proposed methods work e.g., on CIFAR10 and CNNs? The paper notes the real-time constraint in RL setting as an added constraint for detectors. But please note that, even without any constraints, designing a reliable detector is still an open problem for image classifiers.

- The paper posits that, as shown in figure 1, the cost has larger negative curvature at clean examples compared to adversarial examples. The detection method is, however, specified as $|\mathcal{L}-\mathcal{\bar{L}}|>\tau$. Shouldn’t it be $(\mathcal{L}-\mathcal{\bar{L}})>\tau$ (without the absolute value)? Also, the objective function of the adaptive attack is defined as: $\min (J(s_{adv}) + ||s-s_{adv}||^2 + \mathcal{L}(s_{adv}))$. Shouldn’t it be $\min (J(s_{adv}) + ||s-s_{adv}||^2 + |\mathcal{L}(s_{adv})-\bar{\mathcal{L}}|)$ (or maybe $\min (J(s_{adv}) + ||s-s_{adv}||^2 + |\mathcal{L}(s_{adv})-\bar{\mathcal{L}}|)$) to match it to the detection rule? With the current attack formulation, the curvature of the adversarial example could end up being lower than that of clean examples, which would be flagged as adversarial. Plotting L(s) (as in figure 1) for adaptive adversarial examples with different objectives could help visualize this.

- The paper does not report the attack success rates on the undefended models. This is an important sanity check to make sure that the attacks are implemented correctly and also helps with better evaluating the effect of the detector.

- The adaptive attack results are reported for the case that the decrease in the attack success is at most 10%. This doesn’t sound like a good metric. The paper should instead define the attack success rate (ASR) as the rate of examples fooling both the model and the detector, and then report the results for the hyperparameters for which the ASR is highest. The same thing holds for the results of non-adaptive attacks. For example, Table 1 reports the detection rate, but it is not clear what is the ASR and whether the attack hyperparameters are optimized to maximize it. For instance, decreasing the attack budget reduces both the detection rate and the misclassification rate. Thus, there could be a setting that even a lower attack budget results in a higher overall ASR (fooling both model and detector). Overall, it is hard to evaluate the proposed detector without having the best ASR results.

- Table 1 shows that the second-order detector works better than the first-order one. But why is the detection rate of Roth et al., so low? E.g., <5% TPR for CW attack. It seems that their detection method didn’t really need an adaptive attack to break it. Their paper [ICML ’19] and a few follow up works, however, have shown it achieves a pretty high detection rate for non-adaptive attacks on image classifiers.

- Section 5 mentions that the “feature matching” has been used as the first attempt of an adaptive attack. Is this the same attack used for breaking the Roth et al method? If yes, did the authors investigate using the feature matching for designing an adaptive attack specifically for the proposed SO-DATA detector? Please note that the feature matching attack should be designed with the exact same features that the detector uses.

This paper proposes a method on detecting RL adversarial attack. In particular, it uses the curvature of the cost function. The insight is that adversarial states have larger curvature than the normal states. To measure the curvature, it approximates the second-order hessian matrix using a first-order approximation. It further provides empirical evidence on the observation and some theoretical motivation. In the end, it evaluates the proposed method against two baseline methods on seven tasks under seven different kinds of attacks (including two adversary aware attacks) and show superior performance. This paper addresses an important topic: detecting adversarial attacks on RL models. Given RL models have been increasingly widely used, it is important to make sure of their robustness. The proposed algorithm SO-DATA is a simple yet effective method for detecting adversarial attacks on the DRL models. I like the observation on the difference between the adversarial example and the normal examples in terms of the loss curvature this paper makes and how it leverages this observation to develop the method. Sec3.3 further provides a nice theoretical motivation for the use of negative curvature as the criteria for detecting the adversarial examples. The experiment results especially those on detection aware adversaries show the effectiveness of the proposed method.

Minor writing issue:
The font size of each figure’s caption, x label, y label and legend is too small.

Questions:
What is the intuition behind the superior performance of SO-DATA compared with Roth et al.?

“In our grid search over these hyperparameters we found that there is a trade-off between the attack success rate and the detection of the perturbations. In other words, if we optimize the perturbation to be undetectable the success rate of the perturbation” -- Can you give some experiment results regarding this trade-off?
 Pros:
- simple, theoretically well-motivated, and effective approach
- writing is easy to follow
- strong empirical performance improvement

Cons:
- Evaluations are only on Atari Games



This paper provides a fast method to detect adversarially attacked states during reinforcement learning. The main improvement over previous (Roth) approach is to explicitly involve analysis of what's happening along the gradient direction. They relate second-order
behavior with common adversarial considerations to motivate the approach.  Results show a large improvement over a previous
random-sampling approach.  They also show the method resists even adversary-aware attacks fairly well. The paper was well-written and it is good that they considered not just a detection method but also tried to attack it.

I found I had to reread the introduction to section 3.2 to really understand well.  It might be an idea to split this paragraph
in two.  At first introduce a "core concept", (so reader knows to definitely not skim this paragraph).
Then continue with consideration of Hessians, Taylor approximations and motivations for your fast method.

Another minor possibility would perhaps explicitly connect Eqn 14 to an actual strong attack (p==2 is Eqn (2)).

It seems the best adversary-aware attacks of SO-DATA still performed about as well as adversary-unaware
attack on the Roth approach.

Was the number of random perturbations for the Roth method given?
Was SO-DATA  (~ 2 J evals + gradient) also faster then Roth?
 The paper provided and adversarial detection method and a good attempt to break the method.
While strongly connected with methods in non-reinforcement settings, I'm unaware of a similar approach
for reinforcement learning. I enjoyed reading this paper.


The paper considers the adversarial example detection in RL domain and it claims they are the first to do so. - The local curvature of the neural network is considered in a lot of works such as DeepFool and it is not a novel idea. Applying it simply on RL domain sounds like a marginal contribution.
- The mathematical formulations in the paper sounds more decorative rather than adding value to the paper.
- The simulations and experiments are not comprehensive. The reviewer expects comparison with more baselines. Although they claim that they are the first to apply such an idea (detection of the adversarial example in RL domain), the contribution is marginal as it borrows a lot from the literature. Moreover, the performance evaluation is not comprehensive. 

Meta-review: This paper proposes a computationally-efficient method to detect adversarial examples in reinforcement learning models. The detection method is based on the curvature of the loss landscape around the inputs, which is shown to have larger negative value for clean examples compared to adversarial ones. The experiments on Atari environment models show the effectiveness of the method.

The paper is well-written and backs up the experimental results with mathematical intuition and analysis. 

However, the baseline of Roth et al. and all attack methods used have been designed for image classifiers. 

If the authors decide to focus on RL, the attack methods should be tailored to RL. The word  “worst-case” in the title is misleading, since the attacks used in the paper are not optimal for RL algorithms. This reduces the credibility of the claimed successful detection. 

If the authors decide to frame this work as introducing a new property of adversarial examples which can be applied to other tasks, the authors should test this method on other tasks such as benchmark image classification datasets (for example CIFAR10). 

With the current experiment section, it is unclear whether this method works in RL applications since the authors use attack methods designed for image classifiers rather than RL algorithms (Please refer to the following papers for some existing RL attack methods). It is also unclear whether this paper introduces a new property of adversarial examples that is general. 

- Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies
- Ezgi Korkmaz. Nesterov momentum adversarial perturbations in the deep reinforcement learning domain. 
- Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks.
- Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations.
- Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary.

Decision: Reject