Detecting Worst-case Corruptions via Loss Landscape Curvature in Deep Reinforcement Learning
Authors: Ezgi Korkmaz
Jonah Brown-Cohen
Venue: ICLR 2022 Submitted (2022)

Abstract: The non-robustness of neural network policies to adversarial examples poses a challenge for deep reinforcement learning. One natural approach to mitigate the impact of adversarial examples is to develop methods to detect when a given input is adversarial. In this work we introduce a novel approach for detecting adversarial examples that is computationally efficient, is agnostic to the method used to generate adversarial examples, and theoretically well-motivated. Our method is based on a measure of the local curvature of the neural network policy, which we show differs between adversarial and clean examples. We empirically demonstrate the effectiveness of our method in the Atari environment against a large set of state-of-the-art algorithms for generating adversarial examples. Furthermore, we exhibit the effectiveness of our detection algorithm with the presence of multiple strong detection-aware adversaries. 

Under review as a conference paper at ICLR 2022
DETECTING WORST-CASE CORRUPTIONS VIA LOss LANDSCAPE CURVATURE IN DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The non-robustness of neural network policies to adversarial examples poses challenge for deep reinforcement learning. One natural approach to mitigate the impact of adversarial examples is adversarial. In this work we introduce novel approach for detecting adversarial examples that is computationally efficient, is agnostic the method used to gen- erate adversarial examples, and theoretically well-motivated. Our method is based on a measure the local curvature of the neural network policy, which we show differs between adversarial and clean examples. We empirically demonstrate the effectiveness of our method in the Atari environment against a large set of state-of- the-art algorithms for generating adversarial examples. Furthermore, we exhibit the effectiveness of our detection algorithm with the presence of multiple strong detection-awe adversaries.
develop methods detect when a given input
INTRODUCTION
Since Mnih al. (2015) showed that deep neural networks can be used to parameterize reinforce- ment learning policies, there has been substantial growth in new algorithms and applications for deep reinforcement learning. While this progress has resulted in a variety of new capabilities for reinforcement learning agents, it has at the same time introduced new challenges due to the non- robustness of DNNs to imperceptible adversarial perturbations originally discovered by (Szegedy et al. (2014). In particular, Huang et al. (2017); Kos & Song (2017) showed that the non-robustness of DNNs to adversarial perturbations extends to the deep reinforcement learning domain, where ap- plications such self-driving cars automatic financial trading cannot tolerate such a vulnerability.
In the setting of image classification there has been significant effort to make DNNs robust to ad- versarial perturbations Goodfellow et al. (2015); Madry et al. (2018). At the same time, there is line work focused on showing the inevitability adversarial examples and the intrinsic difficulty of learning robust classifiers Dohmatob (2019); Mahloujifar et al. (2019); Gourdeau al.) (019))
Given that it may not be possible to make DNNs completely robust to adversarial examples, a natu- ral objective is instead attempt detect the presence of adversarial examples Pang et al. (2018);
Yang et al. (2020); Ciinas et al. (2020). Additional work has shown that many adversarial defense and detection methods fail in the white-box setting where the adversary is aware of both the trained model and the method used to detect examples AAthalye et al. 2018);; Carlini & Wagner (017b))
In this paper we propose novel detection method for adversarial examples in deep reinforcement learning. This is the first method for detecting adversarial examples in this setting, where compu- tational efficiency is paramount since the method must be applied in real-time every state en- countered by the agent. Our approach relies on differences in the curvature of the neural policy in the neighborhood an adversarial example when compared to a natural example. At a high level our method is based on the intuition that while natural examples have neighborhoods determined by an optimization procedure intended to learn a policy that works well across all states, each ad- versarial example is the output of some local optimization in the neighborhood of one particular state. Our proposed method is computationally efficient, requiring only one gradient computation and two policy evaluations, requires training that depends the adversarial attack method, and

---

Under review as a conference paper at ICLR 2022
is theoretically well-founded. In summary, we focus on detection of adversarial examples and make the following contributions:
Our paper is the first to focus on detection of adversarial examples in the deep reinforcement
learning domain.
We propose novel method, Detection of Adversaries with Taylor Approximation (DATA), to detect adversarial examples based on the local curvature of the neural network policy. DATA is computationally efficient, independent of the method used to generate the adver- sarial example, and theoretically justified.
We conduct experiments in various games of the Arcade Learning Environment that demonstrate the effectiveness of DATA in detecting examples generated by several state- of-the-art adversarial attack methods.
Furthermore, we demonstrate the effectiveness of DATA in the white-box setting where the adversary is aware of the detection method used.
2 RELATED WORK AND BACKGROUND
Deep Reinforcement Learning: In this paper we focus on discrete action set Markov Decision Processes (MDPs) which are given by a continuous set of states S, a discrete set of actions A, transition probability function P : S x A. x S + R, and a reward function policy ग : - P(A) assigns a probability distribution on actions (-]s) to each state 8. In deep reinforcement learning the policy ı is parameterized by deep neural network.
S x A + R. A
Adversarial Examples: Goodfellow et al. (2013) introduced the fast gradient method (FGM) for producing adversarial examples for image classification. The method is based on taking the gradient of the training cost function (z, y) with respect to the input image, and bounding the perturbation by where is the input image and y is the output label. Later, iterative version of FGM called I-FGM was proposed by Kurakin et al. (2016). This is also often referred to as Projected Gradient Descent (PGD) in Madry al. (2018) where the I-FGM update
N+ F adv
clip, (1 + asign VV.. (= v))).
where addy x. Dong al. 2018 further modified I-FGM by introducing a momentum term in the update, yielding method called MI-FGSM. Korkmaz (2020) later proposed a Nesterov- momentum based approach for the deep reinforcement learning domain. The DeepFool method of Moosavi-Dezfooli et al. (2016) is an alternative approach to those based on FGSM. DeepFool per- forms iterative projection to the closest separating hyperplane between classes. Another alternative approach proposed by Carlini & Wagner (2017a) is based finding minimal perturbation that achieves a different target class label. The approach is based on minimizing the loss
min c (-s((d + || - =s| ER
where s is the clean input, Sadv is the adversarial example, and (s) a modified version the cost function used to train the network. [Chen et al. (2018) proposed variant of the Carlini & Wagner 2017a) formulation that adds an é--regularization term to produce sparser adversarial examples,
min e J(sady) + X, ady - $||, + Ax | adv s gal E2
Adversarial Deep Reinforvement Learning: The adversarial problem initially has been investi- gated by Huang et al. (2017) and Kos & Song (017) concurrently. In this work the authors show that perturbations computed via FGSM result in extreme performance loss on the learnt policy. Lin et al. (2017) and et al. (202)) focused on timing strategies in the adversarial formulation and uti- lized Carllii & Wagner (2017a) method to produce the perturbations. While there is a reasonable body of work focused on finding efficient and effective adversarial perturbations, a substantial body of work focused on building agents robust to these perturbations. Mandlekar et al. (2017) proposed to utilize FGSM perturbations during training time to obtain more robust agents. Pinto et al. (2017) modeled the adversarial interaction as a zero sum game and proposed a joint training strategy to increase robustness in continuous action space setting. Recently, Gleave et al. 2020)) considered an
2

---

Under review as a conference paper at ICLR 2022
adversary who is allowed to take natural actions in given environment instead of E. Ey-norm bounded perturbations and modeled the adversarial relationship as a zero sum Markov game. However, re- cent concerns have been raised about adversarial training methods by Korkmaz (2021) In this paper the authors show that the state-of-the-ar adversarial training techniques introduce a new set of non- robust features. Thus, with the rising concerns robustness of recent proposed adversarial training techniques our work aims to solve the adversarial problem from a different perspective by detecting adversarial perturbations.
Detection of Adversarial Examples: There has been long line of work on detection of adver- sarial examples for image classification. Metzen et al. (2017) proposed augmenting standard neural network image classifiers with a sub-network trained specifically to detect adversarial examples. However, this method has cost of separately training a binary classifier to be able to detect the adversarial examples. Pang et al. (2018) proposed modified neural network training strategy that is combined with a test-time thresholding method to distinguish adversarial from clean examples with additional training cost. Yang al. (2020) studied detection of adversarial examples via feature attribution, utilizing the Leave-One-Out (LOO) attribution method proposed by Li et al. (2016). The LOO method is based on erasing each pixel of the image and observing how the output of the neural network changes, and so requires a number of neural network classifier evaluations equal to the res- olution of the input image. Finally, Roth al. (2019) suggest statistical test based measuring the average change in the odds ratio between classes under random perturbations. Evaluation of the neural classifier at hundreds of randomly perturbed samples are required in order for this method to get sufficiently accurate estimates of the average change in order to detect adversarial examples. In contrast prior methods, our detection method does not require modifyyng the training of the neural network, does not require any training specific to the attack method used, and uses only two neural network function evaluations and one gradient computation.
DETECTION OF ADVERSARIES WITH TAYLOR APPROXIMATION (DATA)
In this section we give the high-level motivation for and formal description of our detection method. We begin by introducing necessary notation and definitions. We denote an original clean state by s and an adversarially perturbed state by gadly
Definition 3.1. The cost of state, [s, T),, is defined as the cross entropy loss between the policy T (alss) of the agent, and a target distribution on actions T()).
- -((
(4)
Definition 3.2. The argmax policy, x² <als), is defined as the distribution which puts all probability mass on the highest weight action of (a)s).
x <a(s)) 1(a arg max mag (alls))
(5)
We use the following notation for the gradient and Hessian with respect to states s:
VaJ TD)
V To)
3.1 FIRST-ORDER DETECTION OF ADVERSARIES WITH TAYLOR APPROXIMATION (FO-DATA)
As naive baseline we first describe detection method based on estimating how much the cost function ((, r) varies under small perturbations. Prior work on detection of adversarial examples Roth et al. 2019); Hu et al.] (2019) has shown that the behavior of DNN classifiers under small, random perturbations is different at clean versus adversarial examples. Therefore, natural baseline detection method is: given an input state sy sample small random perturbation 17 ~ (O el) and compute,
(6) The first-order detection method proceeds by first estimating the mean and the variance of K. over a clean run of the agent in the environment. Next a threshold is chosen so that a desired false positive rate (FPR) achieved (i.e. some desired fraction of the states in the clean run lie more
K( n) = Jsn + n, x" --1s=)) Js0, " (-1s0)).
3

---

Under review as a conference paper at ICLR 2022
than standard deviations from the mean). Finally, at test time state encountered by the agent is classified as adversarial it is at least standard deviations away from the mean. Otherwise the state is classified as clean. As first attempt, the first-order method can be naturally interpreted as a finite-difference approximation to the magnitude of the gradient at so- If we assume that the first-order Taylor approximation of J is accurate in ball of radius > centered at so, then
(50 + n, xt (-(+o) x J(50, x (-|s0)) + VaJ xt (+(30) 77.
Therefore,
K(so, 11 V,J x (-1so)) 1)-
Thus, for 77 ~ NN, el) the test statistic K(s0. 17) is approximately distributed as a Gaussian with mean 0 and variance V (-(ss)) Under this interpretation one would expect the test statistics for clean and adversarial states to have the same mean with potentially different standard deviations, possibly making it hard to distinguish clean from adversarial. However, this is not what we observe empirically, and in fact the first-order method does decent job of detecting adversarial examples. The method works because, sonably well separated from the mean of K[sadv 17) for adversarial examples gadv, The empirical performance of the first-order method thus indicates that the assumption of accuracy for the first- order Taylor approximation of J does not hold in practice. This leads naturally to the consideration of information on the second derivatives of J in order to detect adversarial examples.
fact, the mean of ((55, 1)) for clean examples 8 is rea-
3.2 SECOND-ORDER DETECTION OF ADVERSARIES WITH TAYLOR APPROXIMATION (SO-DATA)
The second-order detection method is based on measuring the local curvature of the cost function J(,,T, 7). The method exploits the fact that ((s 7) will have larger negative curvature a clean example as compared to an adversarial example. In particular, the high level theoretical motiva- tion for this approach that adversarial examples are the output of local optimization procedure which attempts to find nearby perturbed state بلdو with a low value for the cost J(s -)) for some T + x (.15). A direction of large negative curvature for J(s .7) indicates that a very small pertur- bation along this direction could dramatically decrease the cost function. Therefore, such points are likely to be unstable for local optimization procedures attempting to minimize the cost function in a small neighborhood. On the other hand, the curvature of (s, at clean state 3 is determined by the overall algorithm used to train the deep reinforcement learning agent. This algorithm optimizes the parameters of the neural network policy while considering all states visited during training, and thus is not likely to be heavily overfit to the state S. In particular, we expect larger negative curva- ture than at an adversarial example şādv We make the connection between negative curvature and instability for local optimization formal in Section Based on the above discussion. a natural choice of metric for distinguishing adversarial versus clean examples is the most negative eigenvalue of the Hessian Amin [V)J(so, (so, x (-1so)). While this is the most natural measurement of curvature, it requires computing the eigenvalues of matrix whose number of entries are quadratic in the input dimension. Since the input is very high-dimensional, and we would like to perform this computa- tion in real-time for every state visited by the agent, computing the value Amio is computationally prohibitive. Instead we approximate this value by measuring the curvature along direction which is correlated with the negative eigenvectors of the Hessian. Given this direction, the value that we measure is the accuracy of the first order Taylor approximation of the cost of the given state J(s, r). We denote the first order Taylor approximation at the state 80 in direction by
J [so: n)) = J (ss. x²( ((11o)) + V ' T (-180) 17.
The metric we will use to detect adversarial examples is the finite-difference approximation
L(su. 17) [ss + 17. मt" (-Iso)) J(so, 17)
(8)
To see formally that Equation (8) gives an approximation of the most negative eigenvector of the Hessian, will assume that the cost function J(s, r) is well approximated by its second-order Taylor approximation at the point 80 i.e.
[s50 + 17, " (-1so)) ss [so, T" (-1so)) + V xT" (-1s0)) - + 17 V.J म (-1s0))m
(9)
for a small enough perturbation n,. Substituting the above formula into Equation (8) yields
L(so. n) R 1) V x² ((1s0))m
(10)
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1: SO-DATA
Input: Mean C and variance (C) from clean run. Detection threshold > 0. Parameter e >> 0.
for states 9; visited by agent do
sign( [ x (-(ss))
7 V ,J π (+1s
J( mh) [S, x² (-1s;) V J x-' (+1s))
[(s,, mH) [ss Th, ==
(s,, 73))
if [C(ss, m7)) > o() then
Label state 8; as an adversarial example
end if
end for
The above quadratic form is minimized when r) lies in the same direction as the most negative eigenvector of the Hessian, in which case
[( 1)) = Amin (<(ss))) 1lm12
xt
(11)
We choose the sign of the gradient direction for measuring the accuracy of the first order Taylor approximation. To motivate this choice note that -v. is locally the direction of steepest decrease for the cost function. If the gradient direction additionally has negative curvature of large magnitude, then small perturbations along this direction will result in even more rapid decrease in the cost function value than predicted by the first-order gradient approximation. Note that this can long the negative curvature is large
be true even if the gradient itself has small magnitude, enough. Thus, by the discussion at the beginning of Scction 3.2 adversarial examples are likely to have relatively smaller magnitude negative curvature in the gradient direction than clean examples. Formally, for c > we set
7( sign (V VV.J ( (-1so)))
(12)
[s0 x x (-180))ll2
To calibrate the detection method we record the mean E E, [C(, (s))] and variance o² (C) Var, [C(s, n(s)) of our proposed test statistic over states from a clean run of the policy in the MDP. Then at test time we set threshold >> O, and for each state 8, visited by the agent test
[[ss ((::)) 2 > (C).
(13)
If the threshold of standard deviations is exceeded classify the state 8; as adversarial, and otherwise classify it as clean. Pseudo-code for the second order method is given in Algorithm
3.3 NEGATIVE CURVATURE AND INSTABILITY OF LOCAL OPTIMIZATION
In this section we formalize the connection between negative curvature and instability for local optimization procedures that motivated our definition of C(s, 17). Given a state 80 and target dis- tribution 7 * T. (- (-)s)), we assume the adversary is trying to find state gadv minimizing [s r) among all states close to so by some metric. Formally, let D (s) 2 0 be convex function of that should be thought of as measuring distance to 8p- One standard choice for the distance function is Dx (8) ||s so soll We model the adversary as minimizing the loss
((s) J(, + (s).
(14)
In particular, we make the following assumption:
Assumption 3.1. The adversarial state adv is local minimum of [s)).
Of course this assumption is violated in practice since different adversarial attack methods optimize objective functions other than f. and do not necessarily always converge to a local minimum. Nev- ertheless the assumption allows us to make formal qualitative predictions about the behavior of the second-order detection method that correspond well with empirical results across broad variety of adversarial attacks. We now state our main result lower bounding the curvature of J[gady, r).
Proposition 3.1. For C >> O assume that the maximum eigenvalue of the Hessian V?D., (s) is bounded by c. If 8* is local minimum of f(8) then Amin Js" -)) -c
5

---

Under review as a conference paper at ICLR 2022
2500
Figure 1: [() for our proposed method SO-DATA vs visited states with corresponding TPR values for the following attack methods: FGSM, MI-FGSM, Nesterov, DeepFool, Carlini&Wagner, Elastic Net Method. TPR values shown in the upper right box of the figure when FPR is equal to 0.01.
Proof. Let u be the eigenvector of 2 Jss r) corresponding to the minimum eigenvalue. At a local minimum 8 of (s) the Hessian ViJ(s must be positive semi-definite. Therefore,
0 S v V (2s((") e = Vs" r) + TV De0
S Amii J(s" T)) +
Rearranging the above inequality completes the proof.
In summary, the second order conditions for a local minimum of f imply a lower bound on the smallest eigenvalue of (²J(s", r). Thus, by Assumption 3.1 obtain lower bound on Amin( [ [ The assumption that the maximum eigenvalue of the Hessian D (8) is bounded by e is satisfied for example when De (s) I| so ll.. contrast, the local curvature of the cost function ((s,7) T) at a clean example is determined by an optimization procedure that updates the weights 0 of the neural network policy rather than the states s. If we write (o(s, 7) to make explicit the dependence on the weights, then the second order conditions for optimizing the original neural network apply the Hessian with respect to weights Vo.Jo(s, r) rather than the Hessian with respect to states (s, T). Additionally, first order optimality conditions can help to justify the choice of (s, T) as a good direction to check for negative curvature. Indeed by the first order conditions, at local optimum s" of f(s) we have
0 = V=](s*) = -
(15)
Therefore, V.J(s", .) -v. (s*). So assuming the adversary finds a local optimum, (s,(s, points in a direction that decreases the distance function D (s*). Thus sufficiently negative curvature in the direction of (.J(ss ()) implies not only that s not local minimum of f. but also that the distance function Da, (s) can be decreased by moving along this direction of negative curvature. To summarize, we have shown that second order optimality conditions arising from computing an adversarial example give rise to lower bounds on the smallest eigenvalue of the Hessian Amin (s. r)). The function C(s, 17) used to detect adversarial examples for SO-DATA is a finite difference approximation to
") TV 2 Amin (V)J(s,T) (8) |
Therefore the results of this section imply that L(s,n) should be larger at adversarial examples than clean examples.
4 EXPERIMENTS
In our experiments agents are trained with DDQN Wang et al. 2016) in the Atari environment Bellemare al. (2013) from O et al. (2016). For a baseline compare FO-DATA and SO-DATA with the detection method of Roth et al. (201). which is based on estimating the average change in the odds ratio between classes under noise. In Figure we plot the value of

---

Under review as a conference paper at ICLR 2022
PDC Deedtool
5.5
45-
62
2014
012
0.00
2015
2000
4.00
RDC RE-GSM
10
40110 0r00aii E:04
024 6.50
0.00 6.30
315
0.3
6.0
tx 8.0
Figure 2: ROC curves of FO-DATA, SO-DATA Roth et ( (011)) for the following attack meth- ods: FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini& Wagner, Elastic Net Method in RoadRunner. TPR values shown in the lower right box of the figure when FPR is equal to 0.01.
Table 1: True Positive Rates (TPR) for FGSM, MI-FGSM, Nesterov Momentum, Carlini& Wagner, Elastic-Net and DeepFool when False Positive Rate (FPR) is equal to 0.01. The proposed methods SO-DATA and FO-DATA
evaluated, and compared with Roth al. (209) in Riverraid, Road- Runner, Alien, Seaquest, Boxing, Pong, and Robotank games. More results for different FPR values are reported in the supplementary material.
Detection Method-Attack Method
RiverRaid
RoadRunner
Alien
Seaquest
Boxing
Pong
Robotank
SO-DATA FGSM
0.997
1.0
1.0
0.995
0.994
1.0
0.999
FO-DATA FGSM
0.990
0.843
0.803
0.931
0.793
0.622
0.413
Roth al. FGSM
0.681
0.767
0.885
0.403
0.264
0.424
0.911
SO-DATA M-IFGSM
0.998
1.0
1.0
0.985
0,910
1.0
0.985
FO-DATA M-IFGSM
0.952
0.863
0.991
0.981
0,827
0.622
0.470
Roth et al. M-IFGSM
0.775
0.554
0.929
0,581
0.499
0.679
0.777
SO-DATA Nesterov Momentum
0.995
0.989
0.996
0.952
0.865
1.0
0.954
FO-DATA Nesterov Momentum
0.990
0.714
0.997
0.979
0.746
0.633
0..74
Roth et al. Nesterov Momentum
0.785
0.646
0.925
0.671
0.517
0.687
0.753
SO-DATA Carlini& Wagner
0.910
0.988
0.945
0.723
0.856
0.850
0.713
FO-DATA Carlini& Wagner
0.695
0.594
0.642
0.516
0.785
0.494
0.119
Roth et al. Carlini& Wagner
0.036
0.118
0.018
0.004
0.016
0.028
0.032
SO-DATA Elastic Net
0.777
0.943
0.875
0.687
0.770
0.736
0.815
FO-DATA Elastic Net
0.685
0.454
0.561
0,502
0.743
0.361
0.212
Roth et al. Elastic Net
0.124
0.210
0.060
0.014
0.150
0.092
0.056
SO-DATA DeepFool
0.914
0.996
0.993
0.860
0.951
0.889
0.900
FO-DATA DeepFool
0.841
0.847
0.936
0.777
0.928
0.796
0.268
Roth al. DeepFool
0.397
0.447
0.611
0.234
0.381
0.367
0.607
C(s) over states for various games without an adversarial attack and under adversarial attack with the following methods: Carlini & Wagner, Elastic Net, Nesterov Momentum, DeepFool, MIFGSM and FGSM. We show in the legends of Figure | the true positive rate (TPR) values for the different attacks when false positive rate (FPR) is equal to 0.01. The value of C(s) for clean states is generally well-concentrated and negative. On the other hand, for states computed by the different adversarial attack methods L(s) is clearly larger, matching the predictions of Proposition 3.1 The fact that [(s) ) is consistently larger at adversarial examples across wide variety of adversarial perturbation methods indicates that Assumption 3.] qualitatively captures the behavior these methods. In particular the FGSM-based methods and DeepFool do not explicitly optimize objective function of the form f(s) = J(s, r) + DD. (s) as Assumption 3.1 However, by enforcing a constraint on the distance of the adversarial example from the original clean example, these methods implicitly solve an optimization problem of the form given in (14), and thus exhibit the qualitative behavior predicted by Proposition 3.1
7

---

Under review as a conference paper at ICLR 2022
10C PULM
KDt Doyphel
2016 1011
02
94
MDC Hine
Nedaay, Marerizn
16
05-
8:
8.4
1000h
15C 61B
10 AB3 0.0
06
03-46 (9 2011
048
0.11
Foe Phite Fostie Arter
178 50 Potter PAR
Figure 3: ROC curves of FO-DATA, SO-DATA and Roth et al. (2019) for the following attack meth- ods: FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini& Wagner, Elastic Net Method in Robotank. TPR values are reported in the lower right box of the figure when FPR is equal to 0.01.
In Table 1 we show TPR values for FO-DATA, SO-DATA, and the Roth et al. (019) method under the FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini& Wagner, Carneni&Wanner and Elastic-Net attacks when FPR is equal 0.01. For all of the attack methods in all of the environments SO-DATA is able to detect adversarial perturbations with large TPR. SO-DATA outperforms the other detection methods in all cases except for Nesterov Momentum in Alien and Seaquest where FO-DATA has TPR 0.997 and 0.980 while SO-DATA has 0.996 and 0.952. We also observe that while the pertur- bations computed by FGSM, MI-FGSM, Nesterov Momentum can generally be detected with large TPR values by all the detection methods, the perturbations computed by Carlini& Wagner and the Elastic-Net method are more difficult to detect. Despite the difficulty, SO-DATA achieves TPR val- ues ranging from 0.713 to 0.988 for Carlini& Wagner, and TPR values ranging from 0.687 to 0.943 for Elastic-Net when FPR is equal to 0.01. In Figure 2 and Figure 3 we show ROC curves for each detection method under the FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini& Wagner and Elastic-Net method for RoadRunner and Robotank respectively. In Robotank the Roth et al.) (2019) method outperforms FO-DATA and even approaches the TPR of SO-DATA for high FPR under FGSM, MI-FGSM, Nesterov Momentum and DeepFool. However for the Carlini& Warlin&&Wagnerr and Elastic-Net attacks, SO-DATA has a much higher TPR across wide range of FPR levels.
DETECTION AWARE ADVERSARY
Recently, Tramer et al. (020) introduced comprehensive methodology for tailoring the optimiza- tion procedure used to produce adversarial examples in order to overcome detection and defense methods. In particular, the high level idea is to keep the attack simple as possible while still accurately targeting the detection method. More specifically, the methodology is based on designing an attack based on gradient descent some loss function. Further, minimizing the loss function should correspond closely to subverting the full detection method while still being possible to op- timize. Critically, the authors highlight that while the choice of loss function to optimize can be difficult task, the use of "feature matching" [Gowal et al. (2019) can circumvent most of the current detection methods. We now describe how we applied the methodology from Tramer al. (2020) to design detection aware adversaries for SO-DATA. As first attempt, we tested the "feature match- ing" approach that was used to break the Roth et al. (2019) detection method in Tramer et al. (0020). This approach attempts to match the logits of the adversarial example to those of a clean example from a different class in order to evade detection. To optimize the loss for this method we used up to 1000 PGD iterations, and we searched step size varying from 0.01 to 10-6, We find that this method succeeds in reducing the TPR of the Roth et al. (2019) method to nearly zero. It is also able to slightly reduce the TPR of our SO-DATA method (see results in Table 2). However, will see next, a larger reduction in the TPR of SO-DATA can be achieved by optimizing a modified version of the loss from Carlini & Wagner (0077)) Our next attempt is based on modification of
8

---

Under review as a conference paper at ICLR 2022
Table 2: TPR for Feature Matching for SO-DATA and Roth et al. (2019) OAO FPR=0.01
Feature Matching
Riverraid
RoadRunner
Alien
Seaquest
Boxing
Robotank
SO-DATA
0.882
0.863 0.006
0.9016
0.955 0.0146
0.988 0.0106
0.8978
Roth et al. (2019)
0.0088
0.007
0.0158
the Carlini & Wagner 2017b) formulation to additionally minimize the cost function L(s) used in SO-DATA,
min c. ((
) + +||sndy s + X- L(sadv).
(16)
Recall that C(s) is consistently larger at adversarial examples than at clean examples. Thus the above optimization problem attempts to find adversarial examples with small values of L(s) as possible. Since the function [(s) involves taking the sign of the gradient, we use Backwards Pass Differentiable Approximation (BPDA) as introduced in Carlini & Wagner (2017b) to compute the gradients. However, we also tried designing an adversary with a fully differentiable cost function by using perturbation in the gradient direction (without the sign). We found that this fully dif- ferentiable adversary performed significantly worse than the one based on BPDA. We conducted exhaustive grid search over all the parameters in this optimization method: learning rate, iteration number, confidence parameter K, and objective function parameter A. In C&W we used up to 30000 iterations to find adversarial examples to bypass detection methods. We searched the confidence parameter from O to 50, the learning rate from 0.001 to 0.1, and À from 0.001 to 10. In our grid search over these hyperparameters we found that there is a trade-off between the attack success rate and the detection of the perturbations. In other words, if we optimize the perturbation to be unde- tectable the success rate of the perturbation (i.e. the rate at which the perturbation actually makes the agent choose non-optimal action) decreases. Therefore, when finalizing the hyperparameters for the SO-DATA detection- -warr adversary we restricted our search to a setting where the decrease in the success rate of the attack was at most 10%. Since FO-DATA is based on sampling a random
Table 3: TPR values of DATA in the presence of detection aware adversary when FPR=0.01.
Detection Method
RiverRaid
RoadRunner
Alien
Seaquest
Boxing
Pong
Robotank
SO-DATA C&W
0.650 0.346
0.849 0.348
0.445
0.381 0.193
0.710
0.712 0.325
0.657
FO-DATA - C&W
0.351
0.621
0.0973
perturbation, we use another approach introduced by Carlini & Wagner (2017b) to minimize the expectation of the original loss function when averaged over the randomness used in the detection method. In particular, we estimate the expectation by computing the empirical mean of the loss over 50 samples from the same noise source. As for the of SO-DATA we grid search over hyper- parameters to achieve as low a TPR as possible while losing at most 10% in the success rate of the attack. Table the TPR in the adversary-awand setting with the best hyperparameters found for each method. The fact that SO-DATA still performs quite well in the adversary-aware setting is an indication that there is fundamental trade-off between computing minimizing C(s). This trade-off makes sense in light of Proposition 3.1 which shows that searching for adversarial example small neighborhood will tend to increase L(s).
adversarial example and
6 CONCLUSION
In this paper we introduced DATA, the first method for detection of adversarial attacks in deep re- inforcement learning. Our method was theoretically motivated by the fact that local optimization objectives corresponding to the construction of adversarial examples lead naturally to lower bounds on the curvature of the cost function J J(s, We have further shown empirically that the curvature of (s, 7 is significantly larger at adversarial examples than at clean examples, leading to a highly effective method SO-DATA for detecting adversarial examples in deep reinforcement learning. We additionally demonstrate that SO-DATA remains effective in the diverrsay-wwar setting, and con- nect this fact to our original theoretical motivation. We believe that due to the strong empirical performance and solid theoretical motivation SO-DATA can be an important step towards producing robust deep reinforcement learning agents.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 274-283. PMLR, 2018.
Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael. Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. Journal of Artificial Intelligence Research. pp. 253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv:1606.01540 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57, 2017a
Nicholas Carlini and David A. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Bhavani M. Thuraisingham, Battista Biggio, David Mandell Freeman, Brad Miller, and Arunesh Sinha (eds.), Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec CCS 2017, Dallas, TX, USA, November 3, 2017, pp. 3-14. ACM, 2017b.
Pin-Yu Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. EAD: elastic-net attacks to deep neural networks via adversarial examples. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thiroo-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp- 10-17. AAAI Press, 2018.
Celia Cintas, Skyler Speakman, Victor Akinwande, William Ogallo, Komminist Weldemariam, Sri- hari Sridharan, and Edward McFowland. Detecting adversarial attacks via subset scanning of autoencoder activations and reconstruction error. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 876-882. ijcai.org. 2020.
Elvis Dohmatob. Generalized no free lunch theorem for adversarial robustness. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. (646-1654 PMLR, 09- Jun 2019.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018
Adam Gleave, Michael Dennis, Cody Wild, Kant Neel, Sergey Levine, and Stuart Russell. Ad- versarial policies: Attacking deep reinforcement learning. International Conference on Learning Representations ICLR, 2020.
Ian Goodfellow, Jonathan Shelens, and Christian Szegedy. Explaning and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Pascale Gourdeau, Varun Kanade, Marta Kwiatkowska, and James Worrell. On the hardness ro- bust classificatin.. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buc, Emily B. Fox, and Roman Gameet (eds.), Advances in Neural Information Processing Sys- tems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 7444-7453, 2019.
Sven Gowal, Jonathan Uesato, Chongli Qin, Po-Sen Huang, Timothy A. Mann, and Pushmeet Kohli. An alternative surrogate loss for pgd-based adversarial testing. 2019.
10

---

Under review as a conference paper at ICLR 2022
Shengyuan Hu, Tao Yu, Chuan Guo, Wei-Lun Chao, and Kilian Q. Weinberger. A new defense against adversarial images: Turning a weakness into a strength. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- fion Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp-
1633-1644, 2019.
Sandy Huang, Nicholas Papernot, Yan Goodfellow, lan Duan, and Pieter Abbeel. Adversar- ial attacks on neural network policies. Workshop Track of the 5th International Conference on Learning Representations, 2017.
Ezgi Korkmaz. Nesterov momentum adversarial perturbations the deep reinforcement learning domain. International Conference on Machine Learning, ICML 2020, Inductive Biases, Invari- ances and Generalization in Reinforcement Learning Workshop. 2020.
Ezgi Korkmaz. Investigating vulnerabilities of deep neural policies. Conference on Uncertainty in Artificial Intelligence (UAI), 2021.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. International Con-
ference on Learning Representations, 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint /60iv:6607 0.000000333, 2016.
Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation crasure. CoRR, abs/1612.08220, 2016. URL http:://aaww..o////a/a//////////
Ven-Chen Lin, Hong Zhang-Wor Yuan Hoan-Hong Liao, Meng-Li Shih, ing. Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. Proceedings of the Twenty-Sixit International Joint Conference on Artificial Intelligence, pp. 3756-3762, 2017
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Confer- ence Track Proceedings. OpenReview.net, 2018
Saeed Mahloujifar, Xiao Zhang, Mohammad Mahmoody, and David Evans. Empirically measur- ing concentration: Fundamental limits on intrinsic robustness. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- fion P Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp- 5210-5220 2019.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausible perturbations. In Proceedings the JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939, 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, are G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518: 529-533, 2015.
Seyed-Mohsen Moosavi-Dezfooe Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2582. IEEE Computer Society, 2016.
11

---

Under review as a conference paper at ICLR 2022
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial amples. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett Adda.n, in Neural Information Processing Systems 31: An- nual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 4584 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. International Conference on Learning Representations ICLR, 2017.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarial examples. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5498- 5507. PMLR, 2019.
Jianwen Sun, Tianwei Zhang, Lei Xiaofei, Xie Ma. Yan Zheng, Kangjie Chen, and Yang. Liu. Stealthy and efficient adversarial attacks against deep reinforcement learning, Association for the Advancement of Artificial Intelligence (AAAI). 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dimutru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. NeurIPS, 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanetot, and Nando. De Fre- itas. Dueling network architectures for deep reinforcement learning. Internation Conference on Machine Learning ICML., pp. 1995-20 2003, 2016.
Puyudi Yang. Jianbo Chen, Cho-Jui Hsich, Jane-Ling Wang, and Michael I. Jordan. ML-LOO: detecting adversarial examples with feature attribution. In The Thirty-Foure AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intellgence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 6639-666 1699-6677. AAAI Press, 2020.
12


Reviews:
[{"review_id": "l0bHK-eumdK", "reviewer": "Reviewer_Ccck", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper was well-written and it is good that they considered not just a detection method but also tried to attack it.\n\nI found I had to reread the introduction to section 3.2 to really understand well.  It might be an idea to split this paragraph\nin two.  At first introduce a \"core concept\", (so reader knows to definitely not skim this paragraph).\nThen continue with consideration of Hessians, Taylor approximations and motivations for your fast method.\n\nAnother minor possibility would perhaps explicitly connect Eqn 14 to an actual strong attack (p==2 is Eqn (2)).\n\nIt seems the best adversary-aware attacks of SO-DATA still performed about as well as adversary-unaware\nattack on the Roth approach.\n\nWas the number of random perturbations for the Roth method given?\nWas SO-DATA  (~ 2 J evals + gradient) also faster then Roth?\n", "summary_of_the_paper": "This paper provides a fast method to detect adversarially attacked states during reinforcement learning. The main improvement over previous (Roth) approach is to explicitly involve analysis of what's happening along the gradient direction. They relate second-order\nbehavior with common adversarial considerations to motivate the approach.  Results show a large improvement over a previous\nrandom-sampling approach.  They also show the method resists even adversary-aware attacks fairly well.", "summary_of_the_review": "The paper provided and adversarial detection method and a good attempt to break the method.\nWhile strongly connected with methods in non-reinforcement settings, I'm unaware of a similar approach\nfor reinforcement learning. I enjoyed reading this paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "X87QwdbngKt", "reviewer": "Reviewer_x4K3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I reviewed this paper for NeurIPS. The paper got rejected there mainly because of the issues with the experiments and the method. The authors just changed the title and resubmitted to ICLR. So I thought it's better to copy my NeurIPS review here. ", "summary_of_the_paper": "This paper proposes a computationally-efficient method to detect adversarial examples in reinforcement learning models. The detection method is based on the curvature of the loss landscape around the inputs, which is shown to have larger negative value for clean examples compared to adversarial ones. The experiments on Atari environment models show the effectiveness of the method.\n", "summary_of_the_review": "The paper is well-written and backs up the experimental results with mathematical intuition and analysis. I, however, have the following questions and concerns.\n\n- Adversarial examples in RL: Studying adversarial robustness in RL scenarios is interesting and important. However, most of the current research on adversarial examples is focused on classifier models and image domain. It also seems that the arguments and analysis presented in this paper could be directly applied to the classification problems as well. In fact the baseline of Roth et al. and all the attack methods used have been designed for image classifiers. Is there a particular reason that the paper focuses instead on RL? How will the proposed methods work e.g., on CIFAR10 and CNNs? The paper notes the real-time constraint in RL setting as an added constraint for detectors. But please note that, even without any constraints, designing a reliable detector is still an open problem for image classifiers.\n\n- The paper posits that, as shown in figure 1, the cost has larger negative curvature at clean examples compared to adversarial examples. The detection method is, however, specified as $|\\mathcal{L}-\\mathcal{\\bar{L}}|>\\tau$. Shouldn’t it be $(\\mathcal{L}-\\mathcal{\\bar{L}})>\\tau$ (without the absolute value)? Also, the objective function of the adaptive attack is defined as: $\\min (J(s_{adv}) + ||s-s_{adv}||^2 + \\mathcal{L}(s_{adv}))$. Shouldn’t it be $\\min (J(s_{adv}) + ||s-s_{adv}||^2 + |\\mathcal{L}(s_{adv})-\\bar{\\mathcal{L}}|)$ (or maybe $\\min (J(s_{adv}) + ||s-s_{adv}||^2 + |\\mathcal{L}(s_{adv})-\\bar{\\mathcal{L}}|)$) to match it to the detection rule? With the current attack formulation, the curvature of the adversarial example could end up being lower than that of clean examples, which would be flagged as adversarial. Plotting L(s) (as in figure 1) for adaptive adversarial examples with different objectives could help visualize this.\n\n- The paper does not report the attack success rates on the undefended models. This is an important sanity check to make sure that the attacks are implemented correctly and also helps with better evaluating the effect of the detector.\n\n- The adaptive attack results are reported for the case that the decrease in the attack success is at most 10%. This doesn’t sound like a good metric. The paper should instead define the attack success rate (ASR) as the rate of examples fooling both the model and the detector, and then report the results for the hyperparameters for which the ASR is highest. The same thing holds for the results of non-adaptive attacks. For example, Table 1 reports the detection rate, but it is not clear what is the ASR and whether the attack hyperparameters are optimized to maximize it. For instance, decreasing the attack budget reduces both the detection rate and the misclassification rate. Thus, there could be a setting that even a lower attack budget results in a higher overall ASR (fooling both model and detector). Overall, it is hard to evaluate the proposed detector without having the best ASR results.\n\n- Table 1 shows that the second-order detector works better than the first-order one. But why is the detection rate of Roth et al., so low? E.g., <5% TPR for CW attack. It seems that their detection method didn’t really need an adaptive attack to break it. Their paper [ICML ’19] and a few follow up works, however, have shown it achieves a pretty high detection rate for non-adaptive attacks on image classifiers.\n\n- Section 5 mentions that the “feature matching” has been used as the first attempt of an adaptive attack. Is this the same attack used for breaking the Roth et al method? If yes, did the authors investigate using the feature matching for designing an adaptive attack specifically for the proposed SO-DATA detector? Please note that the feature matching attack should be designed with the exact same features that the detector uses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "M2qa-tGXh1S", "reviewer": "Reviewer_LjzU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an important topic: detecting adversarial attacks on RL models. Given RL models have been increasingly widely used, it is important to make sure of their robustness. The proposed algorithm SO-DATA is a simple yet effective method for detecting adversarial attacks on the DRL models. I like the observation on the difference between the adversarial example and the normal examples in terms of the loss curvature this paper makes and how it leverages this observation to develop the method. Sec3.3 further provides a nice theoretical motivation for the use of negative curvature as the criteria for detecting the adversarial examples. The experiment results especially those on detection aware adversaries show the effectiveness of the proposed method.\n\nMinor writing issue:\nThe font size of each figure’s caption, x label, y label and legend is too small.\n\nQuestions:\nWhat is the intuition behind the superior performance of SO-DATA compared with Roth et al.?\n\n“In our grid search over these hyperparameters we found that there is a trade-off between the attack success rate and the detection of the perturbations. In other words, if we optimize the perturbation to be undetectable the success rate of the perturbation” -- Can you give some experiment results regarding this trade-off?\n", "summary_of_the_paper": "This paper proposes a method on detecting RL adversarial attack. In particular, it uses the curvature of the cost function. The insight is that adversarial states have larger curvature than the normal states. To measure the curvature, it approximates the second-order hessian matrix using a first-order approximation. It further provides empirical evidence on the observation and some theoretical motivation. In the end, it evaluates the proposed method against two baseline methods on seven tasks under seven different kinds of attacks (including two adversary aware attacks) and show superior performance.", "summary_of_the_review": "Pros:\n- simple, theoretically well-motivated, and effective approach\n- writing is easy to follow\n- strong empirical performance improvement\n\nCons:\n- Evaluations are only on Atari Games\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BpRrf85aI4c", "reviewer": "Reviewer_ZsRu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The local curvature of the neural network is considered in a lot of works such as DeepFool and it is not a novel idea. Applying it simply on RL domain sounds like a marginal contribution.\n- The mathematical formulations in the paper sounds more decorative rather than adding value to the paper.\n- The simulations and experiments are not comprehensive. The reviewer expects comparison with more baselines.", "summary_of_the_paper": "The paper considers the adversarial example detection in RL domain and it claims they are the first to do so.", "summary_of_the_review": "Although they claim that they are the first to apply such an idea (detection of the adversarial example in RL domain), the contribution is marginal as it borrows a lot from the literature. Moreover, the performance evaluation is not comprehensive. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a computationally-efficient method to detect adversarial examples in reinforcement learning models. The detection method is based on the curvature of the loss landscape around the inputs, which is shown to have larger negative value for clean examples compared to adversarial ones. The experiments on Atari environment models show the effectiveness of the method.

The paper is well-written and backs up the experimental results with mathematical intuition and analysis. 

However, the baseline of Roth et al. and all attack methods used have been designed for image classifiers. 

If the authors decide to focus on RL, the attack methods should be tailored to RL. The word  “worst-case” in the title is misleading, since the attacks used in the paper are not optimal for RL algorithms. This reduces the credibility of the claimed successful detection. 

If the authors decide to frame this work as introducing a new property of adversarial examples which can be applied to other tasks, the authors should test this method on other tasks such as benchmark image classification datasets (for example CIFAR10). 

With the current experiment section, it is unclear whether this method works in RL applications since the authors use attack methods designed for image classifiers rather than RL algorithms (Please refer to the following papers for some existing RL attack methods). It is also unclear whether this paper introduces a new property of adversarial examples that is general. 

- Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies
- Ezgi Korkmaz. Nesterov momentum adversarial perturbations in the deep reinforcement learning domain. 
- Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks.
- Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations.
- Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary.
Decision: Reject