Year: 2021

Abstract:
The non-robustness of neural network policies to adversarial examples poses a challenge for deep reinforcement learning. One natural approach to mitigate the impact of adversarial examples is to develop methods to detect when a given input is adversarial. In this work we introduce a novel approach for detecting adversarial examples that is computationally efficient, is agnostic to the method used to generate adversarial examples, and theoretically well-motivated. Our method is based on a measure of the local curvature of the neural network policy, which we show differs between adversarial and clean examples. We empirically demonstrate the effectiveness of our method in the Atari environment against a large set of state-ofthe-art algorithms for generating adversarial examples. Furthermore, we exhibit the effectiveness of our detection algorithm with the presence of multiple strong detection-aware adversaries.

text: The non-robustness of neural network policies to adversarial examples poses a challenge for deep reinforcement learning. One natural approach to mitigate the impact of adversarial examples is to develop methods to detect when a given input is adversarial. In this work we introduce a novel approach for detecting adversarial examples that is computationally efficient, is agnostic to the method used to generate adversarial examples, and theoretically well-motivated. Our method is based on a measure of the local curvature of the neural network policy, which we show differs between adversarial and clean examples. We empirically demonstrate the effectiveness of our method in the Atari environment against a large set of state-ofthe-art algorithms for generating adversarial examples. Furthermore, we exhibit the effectiveness of our detection algorithm with the presence of multiple strong detection-aware adversaries.

heading: 1 INTRODUCTION
text: Since Mnih et al. (2015) showed that deep neural networks can be used to parameterize reinforcement learning policies, there has been substantial growth in new algorithms and applications for deep reinforcement learning. While this progress has resulted in a variety of new capabilities for reinforcement learning agents, it has at the same time introduced new challenges due to the nonrobustness of DNNs to imperceptible adversarial perturbations originally discovered by Szegedy et al. (2014). In particular, Huang et al. (2017); Kos & Song (2017) showed that the non-robustness of DNNs to adversarial perturbations extends to the deep reinforcement learning domain, where applications such as self-driving cars or automatic financial trading cannot tolerate such a vulnerability.
In the setting of image classification there has been significant effort to make DNNs robust to adversarial perturbations Goodfellow et al. (2015); Madry et al. (2018). At the same time, there is a line of work focused on showing the inevitability of adversarial examples and the intrinsic difficulty of learning robust classifiers Dohmatob (2019); Mahloujifar et al. (2019); Gourdeau et al. (2019). Given that it may not be possible to make DNNs completely robust to adversarial examples, a natural objective is to instead attempt to detect the presence of adversarial examples Pang et al. (2018); Yang et al. (2020); Cintas et al. (2020). Additional work has shown that many adversarial defense and detection methods fail in the white-box setting where the adversary is aware of both the trained model and the method used to detect examples Athalye et al. (2018); Carlini & Wagner (2017b).
In this paper we propose a novel detection method for adversarial examples in deep reinforcement learning. This is the first method for detecting adversarial examples in this setting, where computational efficiency is paramount since the method must be applied in real-time to every state encountered by the agent. Our approach relies on differences in the curvature of the neural policy in the neighborhood of an adversarial example when compared to a natural example. At a high level our method is based on the intuition that while natural examples have neighborhoods determined by an optimization procedure intended to learn a policy that works well across all states, each adversarial example is the output of some local optimization in the neighborhood of one particular state. Our proposed method is computationally efficient, requiring only one gradient computation and two policy evaluations, requires no training that depends on the adversarial attack method, and
is theoretically well-founded. In summary, we focus on detection of adversarial examples and make the following contributions:
• Our paper is the first to focus on detection of adversarial examples in the deep reinforcement learning domain.
• We propose a novel method, Detection of Adversaries with Taylor Approximation (DATA), to detect adversarial examples based on the local curvature of the neural network policy. DATA is computationally efficient, independent of the method used to generate the adversarial example, and theoretically justified.
• We conduct experiments in various games of the Arcade Learning Environment that demonstrate the effectiveness of DATA in detecting examples generated by several stateof-the-art adversarial attack methods.
• Furthermore, we demonstrate the effectiveness of DATA in the white-box setting where the adversary is aware of the detection method used.

heading: 2 RELATED WORK AND BACKGROUND
text: Deep Reinforcement Learning: In this paper we focus on discrete action set Markov Decision Processes (MDPs) which are given by a continuous set of states S, a discrete set of actions A, a transition probability function P : S × A × S → R, and a reward function r : S × A → R. A policy π : S → P(A) assigns a probability distribution on actions π(·|s) to each state s. In deep reinforcement learning the policy π is parameterized by a deep neural network.
Adversarial Examples: Goodfellow et al. (2015) introduced the fast gradient method (FGM) for producing adversarial examples for image classification. The method is based on taking the gradient of the training cost function J(x, y) with respect to the input image, and bounding the perturbation by where x is the input image and y is the output label. Later, an iterative version of FGM called I-FGM was proposed by Kurakin et al. (2016). This is also often referred to as Projected Gradient Descent (PGD) as in Madry et al. (2018) where the I-FGM update is
xN+1adv = clip (x N adv + αsign(∇xJ(xNadv, y))). (1)
where x0adv = x. Dong et al. (2018) further modified I-FGM by introducing a momentum term in the update, yielding a method called MI-FGSM. Korkmaz (2020) later proposed a Nesterovmomentum based approach for the deep reinforcement learning domain. The DeepFool method of Moosavi-Dezfooli et al. (2016) is an alternative approach to those based on FGSM. DeepFool performs iterative projection to the closest separating hyperplane between classes. Another alternative approach proposed by Carlini & Wagner (2017a) is based on finding a minimal perturbation that achieves a different target class label. The approach is based on minimizing the loss
min sadv∈S
c · J(sadv) + ∥∥sadv − s∥∥
 (2)
where s is the clean input, sadv is the adversarial example, and J(s) is a modified version of the cost function used to train the network. Chen et al. (2018) proposed a variant of the Carlini & Wagner (2017a) formulation that adds an `1-regularization term to produce sparser adversarial examples,
min sadv∈S
c · J(sadv) + λ ∥∥sadv − s∥∥
 + λ ∥∥sadv − s∥∥ 
(3)
Adversarial Deep Reinforcement Learning: The adversarial problem initially has been investigated by Huang et al. (2017) and Kos & Song (2017) concurrently. In this work the authors show that perturbations computed via FGSM result in extreme performance loss on the learnt policy. Lin et al. (2017) and Sun et al. (2020) focused on timing strategies in the adversarial formulation and utilized the Carlini & Wagner (2017a) method to produce the perturbations. While there is a reasonable body of work focused on finding efficient and effective adversarial perturbations, a substantial body of work focused on building agents robust to these perturbations. Mandlekar et al. (2017) proposed to utilize FGSM perturbations during training time to obtain more robust agents. Pinto et al. (2017) modeled the adversarial interaction as a zero sum game and proposed a joint training strategy to increase robustness in continuous action space setting. Recently, Gleave et al. (2020) considered an
adversary who is allowed to take natural actions in a given environment instead of `p-norm bounded perturbations and modeled the adversarial relationship as a zero sum Markov game. However, recent concerns have been raised about adversarial training methods by Korkmaz (2021). In this paper the authors show that the state-of-the-art adversarial training techniques introduce a new set of nonrobust features. Thus, with the rising concerns on robustness of recent proposed adversarial training techniques our work aims to solve the adversarial problem from a different perspective by detecting adversarial perturbations.
Detection of Adversarial Examples: There has been a long line of work on detection of adversarial examples for image classification. Metzen et al. (2017) proposed augmenting standard neural network image classifiers with a sub-network trained specifically to detect adversarial examples. However, this method has a cost of separately training a binary classifier to be able to detect the adversarial examples. Pang et al. (2018) proposed a modified neural network training strategy that is combined with a test-time thresholding method to distinguish adversarial from clean examples with additional training cost. Yang et al. (2020) studied detection of adversarial examples via feature attribution, utilizing the Leave-One-Out (LOO) attribution method proposed by Li et al. (2016). The LOO method is based on erasing each pixel of the image and observing how the output of the neural network changes, and so requires a number of neural network classifier evaluations equal to the resolution of the input image. Finally, Roth et al. (2019) suggest a statistical test based on measuring the average change in the odds ratio between classes under random perturbations. Evaluation of the neural classifier at hundreds of randomly perturbed samples are required in order for this method to get sufficiently accurate estimates of the average change in order to detect adversarial examples. In contrast to prior methods, our detection method does not require modifying the training of the neural network, does not require any training specific to the attack method used, and uses only two neural network function evaluations and one gradient computation.

heading: 3 DETECTION OF ADVERSARIES WITH TAYLOR APPROXIMATION (DATA)
text: In this section we give the high-level motivation for and formal description of our detection method. We begin by introducing necessary notation and definitions. We denote an original clean state by s̄ and an adversarially perturbed state by sadv. Definition 3.1. The cost of a state, J(s, τ), is defined as the cross entropy loss between the policy π(a|s) of the agent, and a target distribution on actions τ(a).
J(s, τ) = − ∑ a τ(a) log(π(a|s)) (4)
Definition 3.2. The argmax policy, π∗(a|s), is defined as the distribution which puts all probability mass on the highest weight action of π(a|s).
π∗(a|s) = 1(a = arg max a′ π(a′|s)) (5)
We use the following notation for the gradient and Hessian with respect to states s:
∇sJ(s0, τ0) = ∇sJ(s, τ)|s=s0,τ=τ0 ∇2sJ(s0, τ0) = ∇2sJ(s, τ)|s=s0,τ=τ0

heading: 3.1 FIRST-ORDER DETECTION OF ADVERSARIES WITH TAYLOR APPROXIMATION
text: (FO-DATA)
As a naive baseline we first describe a detection method based on estimating how much the cost function J(s, τ) varies under small perturbations. Prior work on detection of adversarial examples Roth et al. (2019); Hu et al. (2019) has shown that the behavior of DNN classifiers under small, random perturbations is different at clean versus adversarial examples. Therefore, a natural baseline detection method is: given an input state s0 sample a small random perturbation η ∼ N (0, I) and compute, K(s0, η) = J(s0 + η, π∗(·|s0))− J(s0, π∗(·|s0)). (6) The first-order detection method proceeds by first estimating the mean and the variance of K over a clean run of the agent in the environment. Next a threshold t is chosen so that a desired false positive rate (FPR) is achieved (i.e. some desired fraction of the states in the clean run lie more
than t standard deviations from the mean). Finally, at test time a state encountered by the agent is classified as adversarial if it is at least t standard deviations away from the mean. Otherwise the state is classified as clean. As a first attempt, the first-order method can be naturally interpreted as a finite-difference approximation to the magnitude of the gradient at s0. If we assume that the first-order Taylor approximation of J is accurate in a ball of radius r > centered at s0, then
J(s0 + η, π ∗(·|s0)) ≈ J(s0, π∗(·|s0)) +∇sJ(s0, π∗(·|s0)) · η.
Therefore, K(s0, η) ≈ ∇sJ(s0, π∗(·|s0)) · η. (7)
Thus, for η ∼ N (0, I) the test statistic K(s0, η) is approximately distributed as a Gaussian with mean 0 and variance 2‖∇sJ(s0, π∗(·|s0))‖2. Under this interpretation one would expect the test statistics for clean and adversarial states to have the same mean with potentially different standard deviations, possibly making it hard to distinguish clean from adversarial. However, this is not what we observe empirically, and in fact the first-order method does a decent job of detecting adversarial examples. The method works because, in fact, the mean of K(s̄, η) for clean examples s̄ is reasonably well separated from the mean of K(sadv, η) for adversarial examples sadv. The empirical performance of the first-order method thus indicates that the assumption of accuracy for the firstorder Taylor approximation of J does not hold in practice. This leads naturally to the consideration of information on the second derivatives of J in order to detect adversarial examples.

heading: 3.2 SECOND-ORDER DETECTION OF ADVERSARIES WITH TAYLOR APPROXIMATION
text: (SO-DATA)
The second-order detection method is based on measuring the local curvature of the cost function J(s, τ). The method exploits the fact that J(s, τ) will have larger negative curvature at a clean example as compared to an adversarial example. In particular, the high level theoretical motivation for this approach is that adversarial examples are the output of a local optimization procedure which attempts to find a nearby perturbed state sadv with a low value for the cost J(sadv, τ) for some τ 6= π∗(·|s̄). A direction of large negative curvature for J(sadv, τ) indicates that a very small perturbation along this direction could dramatically decrease the cost function. Therefore, such points are likely to be unstable for local optimization procedures attempting to minimize the cost function in a small neighborhood. On the other hand, the curvature of J(s, τ) at a clean state s̄ is determined by the overall algorithm used to train the deep reinforcement learning agent. This algorithm optimizes the parameters of the neural network policy while considering all states visited during training, and thus is not likely to be heavily overfit to the state s̄. In particular, we expect larger negative curvature at s̄ than at an adversarial example sadv. We make the connection between negative curvature and instability for local optimization formal in Section 3.3. Based on the above discussion, a natural choice of metric for distinguishing adversarial versus clean examples is the most negative eigenvalue of the Hessian λmin ( ∇2sJ(s0, π∗(·|s0) ) . While this is the most natural measurement of curvature, it requires computing the eigenvalues of a matrix whose number of entries are quadratic in the input dimension. Since the input is very high-dimensional, and we would like to perform this computation in real-time for every state visited by the agent, computing the value λmin is computationally prohibitive. Instead we approximate this value by measuring the curvature along a direction which is correlated with the negative eigenvectors of the Hessian. Given this direction, the value that we measure is the accuracy of the first order Taylor approximation of the cost of the given state J(s, τ). We denote the first order Taylor approximation at the state s0 in direction η by
J̃(s0, η) = J(s0, π ∗(·|s0)) +∇sJ(s0, π∗(·|s0)) · η.
The metric we will use to detect adversarial examples is the finite-difference approximation
L(s0, η) = J(s0 + η, π∗(·|s0))− J̃(s0, η). (8) To see formally that Equation (8) gives an approximation of the most negative eigenvector of the Hessian, we will assume that the cost function J(s, τ) is well approximated by its second-order Taylor approximation at the point s0 i.e.
J(s0 + η, π ∗(·|s0)) ≈ J(s0, π∗(·|s0)) +∇sJ(s0, π∗(·|s0)) · η + η>∇2sJ(s0, π∗(·|s0))η (9)
for a small enough perturbation η. Substituting the above formula into Equation (8) yields
L(s0, η) ≈ η>∇2sJ(s0, π∗(·|s0))η (10)
Algorithm 1: SO-DATA Input: Mean L̄ and variance σ2(L) from clean run. Detection threshold t > 0. Parameter > 0. for states si visited by agent do
ηi = sign(∇sJ(si, π∗(·|si))) ‖∇sJ(si, π∗(·|si))‖2 J̃(si, ηi) = J(si, π ∗(·|si)) +∇sJ(si, π∗(·|si)) · ηi L(si, ηi) = J(si + ηi, π∗(·|si))− J̃(si, ηi) if |L(si, ηi)− L̄| > t · σ(L) then
Label state si as an adversarial example end if
end for
The above quadratic form is minimized when η lies in the same direction as the most negative eigenvector of the Hessian, in which case
L(s0, η) ≈ λmin ( ∇2sJ(s0, π∗(·|s0)) ) ‖η‖22 (11)
We choose the sign of the gradient direction for measuring the accuracy of the first order Taylor approximation. To motivate this choice note that −∇sJ(s, τ) is locally the direction of steepest decrease for the cost function. If the gradient direction additionally has negative curvature of large magnitude, then small perturbations along this direction will result in even more rapid decrease in the cost function value than predicted by the first-order gradient approximation. Note that this can be true even if the gradient itself has small magnitude, as long as the negative curvature is large enough. Thus, by the discussion at the beginning of Section 3.2, adversarial examples are likely to have relatively smaller magnitude negative curvature in the gradient direction than clean examples. Formally, for > 0 we set
η(s0) = sign (∇sJ(s0, π∗(·|s0))) ‖∇sJ(s0, π∗(·|s0))‖2 . (12)
To calibrate the detection method we record the mean L̄ = Es[L(s, η(s))] and variance σ2(L) = Vars[L(s, η(s))] of our proposed test statistic over states from a clean run of the policy in the MDP. Then at test time we set a threshold t > 0, and for each state si visited by the agent test if
|L(si, η(si))− L̄| > tσ(L). (13) If the threshold of t standard deviations is exceeded we classify the state si as adversarial, and otherwise classify it as clean. Pseudo-code for the second order method is given in Algorithm 1.

heading: 3.3 NEGATIVE CURVATURE AND INSTABILITY OF LOCAL OPTIMIZATION
text: In this section we formalize the connection between negative curvature and instability for local optimization procedures that motivated our definition of L(s, η). Given a state s0 and a target distribution τ 6= π∗(·|s0), we assume the adversary is trying to find a state sadv minimizing J(sadv, τ) among all states close to s0 by some metric. Formally, let Ds0(s) ≥ 0 be a convex function of s that should be thought of as measuring distance to s0. One standard choice for the distance function is Ds0(s) = ‖s− s0‖pp. We model the adversary as minimizing the loss
f(s) = J(s, τ) +Ds0(s). (14)
In particular, we make the following assumption: Assumption 3.1. The adversarial state sadv is a local minimum of f(s).
Of course this assumption is violated in practice since different adversarial attack methods optimize objective functions other than f , and do not necessarily always converge to a local minimum. Nevertheless the assumption allows us to make formal qualitative predictions about the behavior of the second-order detection method that correspond well with empirical results across a broad variety of adversarial attacks. We now state our main result lower bounding the curvature of J(sadv, τ). Proposition 3.1. For c > 0 assume that the maximum eigenvalue of the Hessian ∇2sDs0(s) is bounded by c. If s∗ is a local minimum of f(s) then λmin(∇2sJ(s∗, τ)) ≥ −c
Proof. Let v be the eigenvector of∇2sJ(s∗, τ) corresponding to the minimum eigenvalue. At a local minimum s∗ of f(s) the Hessian∇2sf(s∗) must be positive semi-definite. Therefore,
0 ≤ v>∇2sf(s∗)v = v>∇2sJ(s∗, τ)v + v>∇2sDs0(s∗)v ≤ λmin(∇2sJ(s∗, τ)) + c
Rearranging the above inequality completes the proof.
In summary, the second order conditions for a local minimum of f imply a lower bound on the smallest eigenvalue of ∇2sJ(s∗, τ). Thus, by Assumption 3.1, we obtain a lower bound on λmin(∇2sJ(sadv, τ)). The assumption that the maximum eigenvalue of the Hessian ∇2sDs0(s) is bounded by c is satisfied for example whenDs0(s) = c 2‖s−s0‖ 2 2. In contrast, the local curvature of the cost function J(s, τ) at a clean example is determined by an optimization procedure that updates the weights θ of the neural network policy rather than the states s. If we write Jθ(s, τ) to make explicit the dependence on the weights, then the second order conditions for optimizing the original neural network apply to the Hessian with respect to weights∇2θJθ(s, τ) rather than the Hessian with respect to states ∇2sJθ(s, τ). Additionally, first order optimality conditions can help to justify the choice of ∇sJ(s, τ) as a good direction to check for negative curvature. Indeed by the first order conditions, at a local optimum s∗ of f(s) we have
0 = ∇sf(s∗) = ∇sJ(s∗, τ) +∇sDs0(s∗). (15)
Therefore, ∇sJ(s∗, τ) = −∇sDs0(s∗). So assuming the adversary finds a local optimum, ∇sJ(s, τ) points in a direction that decreases the distance function Ds0(s∗). Thus sufficiently negative curvature in the direction of ∇sJ(s, τ) implies not only that s is not a local minimum of f , but also that the distance function Ds0(s) can be decreased by moving along this direction of negative curvature. To summarize, we have shown that second order optimality conditions arising from computing an adversarial example give rise to lower bounds on the smallest eigenvalue of the Hessian λmin ( ∇2sJ(s, τ) ) . The function L(s, η) used to detect adversarial examples for SO-DATA is a finite difference approximation to
η>∇2sJ(s, τ)η ≥ λmin ( ∇2sJ(s, τ) ) ‖η‖2 .
Therefore the results of this section imply that L(s, η) should be larger at adversarial examples than clean examples.

heading: 4 EXPERIMENTS
text: In our experiments agents are trained with DDQN Wang et al. (2016) in the Atari environment Bellemare et al. (2013) from OpenAI Brockman et al. (2016). For a baseline we compare FO-DATA and SO-DATA with the detection method of Roth et al. (2019), which is based on estimating the average change in the odds ratio between classes under noise. In Figure 1 we plot the value of
L(s) over states for various games without an adversarial attack and under adversarial attack with the following methods: Carlini & Wagner, Elastic Net, Nesterov Momentum, DeepFool, MIFGSM and FGSM. We show in the legends of Figure 1 the true positive rate (TPR) values for the different attacks when false positive rate (FPR) is equal to 0.01. The value of L(s) for clean states is generally well-concentrated and negative. On the other hand, for states computed by the different adversarial attack methods L(s) is clearly larger, matching the predictions of Proposition 3.1. The fact that L(s) is consistently larger at adversarial examples across a wide variety of adversarial perturbation methods indicates that Assumption 3.1 qualitatively captures the behavior of these methods. In particular the FGSM-based methods and DeepFool do not explicitly optimize an objective function of the form f(s) = J(s, τ) + Ds0(s) as in Assumption 3.1. However, by enforcing a constraint on the distance of the adversarial example from the original clean example, these methods implicitly solve an optimization problem of the form given in (14), and thus exhibit the qualitative behavior predicted by Proposition 3.1.
In Table 1 we show TPR values for FO-DATA, SO-DATA, and the Roth et al. (2019) method under the FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini&Wagner, and Elastic-Net attacks when FPR is equal to 0.01. For all of the attack methods in all of the environments SO-DATA is able to detect adversarial perturbations with large TPR. SO-DATA outperforms the other detection methods in all cases except for Nesterov Momentum in Alien and Seaquest where FO-DATA has TPR 0.997 and 0.980 while SO-DATA has 0.996 and 0.952. We also observe that while the perturbations computed by FGSM, MI-FGSM, Nesterov Momentum can generally be detected with large TPR values by all the detection methods, the perturbations computed by Carlini&Wagner and the Elastic-Net method are more difficult to detect. Despite the difficulty, SO-DATA achieves TPR values ranging from 0.713 to 0.988 for Carlini&Wagner, and TPR values ranging from 0.687 to 0.943 for Elastic-Net when FPR is equal to 0.01. In Figure 2 and Figure 3 we show ROC curves for each detection method under the FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini&Wagner and Elastic-Net method for RoadRunner and Robotank respectively. In Robotank the Roth et al. (2019) method outperforms FO-DATA and even approaches the TPR of SO-DATA for high FPR under FGSM, MI-FGSM, Nesterov Momentum and DeepFool. However for the Carlini&Wagner and Elastic-Net attacks, SO-DATA has a much higher TPR across a wide range of FPR levels.

heading: 5 DETECTION AWARE ADVERSARY
text: Recently, Tramer et al. (2020) introduced a comprehensive methodology for tailoring the optimization procedure used to produce adversarial examples in order to overcome detection and defense methods. In particular, the high level idea is to keep the attack as simple as possible while still accurately targeting the detection method. More specifically, the methodology is based on designing an attack based on gradient descent on some loss function. Further, minimizing the loss function should correspond closely to subverting the full detection method while still being possible to optimize. Critically, the authors highlight that while the choice of loss function to optimize can be a difficult task, the use of “feature matching” Gowal et al. (2019) can circumvent most of the current detection methods. We now describe how we applied the methodology from Tramer et al. (2020) to design detection aware adversaries for SO-DATA. As a first attempt, we tested the “feature matching” approach that was used to break the Roth et al. (2019) detection method in Tramer et al. (2020). This approach attempts to match the logits of the adversarial example to those of a clean example from a different class in order to evade detection. To optimize the loss for this method we used up to 1000 PGD iterations, and we searched step size varying from 0.01 to 10−6. We find that this method succeeds in reducing the TPR of the Roth et al. (2019) method to nearly zero. It is also able to slightly reduce the TPR of our SO-DATA method (see results in Table 2). However, as we will see next, a larger reduction in the TPR of SO-DATA can be achieved by optimizing a modified version of the loss from Carlini & Wagner (2017b). Our next attempt is based on a modification of
the Carlini & Wagner (2017b) formulation to additionally minimize the cost function L(s) used in SO-DATA,
min sadv∈S
c · J(sadv) + ∥∥sadv − s∥∥
 + λ · L(sadv). (16)
Recall that L(s) is consistently larger at adversarial examples than at clean examples. Thus the above optimization problem attempts to find adversarial examples with as small values of L(s) as possible. Since the function L(s) involves taking the sign of the gradient, we use Backwards Pass Differentiable Approximation (BPDA) as introduced in Carlini & Wagner (2017b) to compute the gradients. However, we also tried designing an adversary with a fully differentiable cost function by using a perturbation in the gradient direction (without the sign). We found that this fully differentiable adversary performed significantly worse than the one based on BPDA. We conducted exhaustive grid search over all the parameters in this optimization method: learning rate, iteration number, confidence parameter κ, and objective function parameter λ. In C&W we used up to 30000 iterations to find adversarial examples to bypass detection methods. We searched the confidence parameter from 0 to 50, the learning rate from 0.001 to 0.1, and λ from 0.001 to 10. In our grid search over these hyperparameters we found that there is a trade-off between the attack success rate and the detection of the perturbations. In other words, if we optimize the perturbation to be undetectable the success rate of the perturbation (i.e. the rate at which the perturbation actually makes the agent choose a non-optimal action) decreases. Therefore, when finalizing the hyperparameters for the SO-DATA detection-aware adversary we restricted our search to a setting where the decrease in the success rate of the attack was at most 10%. Since FO-DATA is based on sampling a random
perturbation, we use another approach introduced by Carlini & Wagner (2017b) to minimize the expectation of the original loss function when averaged over the randomness used in the detection method. In particular, we estimate the expectation by computing the empirical mean of the loss over 50 samples from the same noise source. As for the case of SO-DATA we grid search over hyperparameters to achieve as low a TPR as possible while losing at most 10% in the success rate of the attack. Table 3 shows the TPR in the adversary-aware setting with the best hyperparameters found for each method. The fact that SO-DATA still performs quite well in the adversary-aware setting is an indication that there is a fundamental trade-off between computing an adversarial example and minimizing L(s). This trade-off makes sense in light of Proposition 3.1, which shows that searching for an adversarial example in a small neighborhood will tend to increase L(s).

heading: 6 CONCLUSION
text: In this paper we introduced DATA, the first method for detection of adversarial attacks in deep reinforcement learning. Our method was theoretically motivated by the fact that local optimization objectives corresponding to the construction of adversarial examples lead naturally to lower bounds on the curvature of the cost function J(s, τ). We have further shown empirically that the curvature of J(s, τ) is significantly larger at adversarial examples than at clean examples, leading to a highly effective method SO-DATA for detecting adversarial examples in deep reinforcement learning. We additionally demonstrate that SO-DATA remains effective in the adversary-aware setting, and connect this fact to our original theoretical motivation. We believe that due to the strong empirical performance and solid theoretical motivation SO-DATA can be an important step towards producing robust deep reinforcement learning agents.

Review summaries:
This paper proposes a computationally-efficient method to detect adversarial examples in reinforcement learning models. The detection method is based on the curvature of the loss landscape around the inputs, which is shown to have larger negative value for clean examples compared to adversarial ones. The experiments on Atari environment models show the effectiveness of the method. 

This paper proposes a method on detecting RL adversarial attack. In particular, it uses the curvature of the cost function. The insight is that adversarial states have larger curvature than the normal states. To measure the curvature, it approximates the second-order hessian matrix using a first-order approximation. It further provides empirical evidence on the observation and some theoretical motivation. In the end, it evaluates the proposed method against two baseline methods on seven tasks under seven different kinds of attacks (including two adversary aware attacks) and show superior performance.

This paper provides a fast method to detect adversarially attacked states during reinforcement learning. The main improvement over previous (Roth) approach is to explicitly involve analysis of what's happening along the gradient direction. They relate second-order behavior with common adversarial considerations to motivate the approach.  Results show a large improvement over a previous random-sampling approach.  They also show the method resists even adversary-aware attacks fairly well.