Uncovering Directions of Instability via Quadratic Approximation of Deep Neural Loss in Reinforcement Learning
Authors: Ezgi Korkmaz
Jonah Brown-Cohen
Venue: Submitted to ICLR 2023 (2023)

Abstract: Learning in MDPs with highly complex state representations is currently possible due to multiple advancements in reinforcement learning algorithm design. However, this incline in complexity, and furthermore the increase in the dimensions of the observation came at the cost of non-robustness that can be taken advantage of (i.e. moving along worst-case directions in the observation space). To solve this policy instability problem we propose a novel method to ascertain the presence of these non-robust directions via quadratic approximation of the deep neural policy loss. Our method provides a theoretical basis for the fundamental cut-off between stable observations and non-robust observations.  Furthermore, our technique is computationally efficient, and does not depend on the methods used to produce the worst-case directions. We conduct extensive experiments in the Arcade Learning Environment with several different non-robust alteration techniques. Most significantly, we demonstrate the effectiveness of our approach even in the setting where alterations are explicitly optimized to circumvent our proposed method.

Under review as a conference paper at ICLR 2023
UNCOVERING DIRECTIONS OF INSTABILITY VIA QUADRATIC APPROXIMATION OF DEEP NEURAL Loss IN REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Learning MDPs with highly complex state representations is currently possible due to multiple advancements in reinforcement learning algorithm design. However, this incline complexity, and furthermore the increase the dimensions of the observation came at the cost of non-robustness that can be taken advantage of (i.e. moving along worst-case directions in the observation space). To solve this policy instability problem we propose novel method to ascertain the presence these non-robust directions via quadratic approximation of the deep neural policy loss. Our method provides theoretical basis for the fundamental cut- off between stable observations and non-robust observations. Furthermore, our technique is computationally efficient, and does not depend on the methods used produce the worst-case directions. We conduct extensive experiments in the Arcade Learning Environment with several different non-robust alteration techniques. Most significantly, we demonstrate the effectiveness of our approach even in the setting where alterations are explicitly optimized circumvent our proposed method.
INTRODUCTION
Since Mnih et al. (2015) showed that deep neural networks can be used to parameterize reinforcement learning policies, there has been substantial growth in new algorithms and applications for deep reinforcement learning. While this progress has resulted in variety of new capabilities for reinforce- ment learning agents, has at the same time introduced new challenges due to the non-robustness of deep neural networks to imperceptible adversarial perturbations originally discovered by [Szegedy (ct al. (2014). In particular, Huang et al. (2017): Kos & Song (2017) showed that the non-robustness of neural networks to adversarial perturbations extends to the deep reinforcement learning domain, where applications such as autonomous driving, automatic financial trading or healthcare decision making cannot tolerate such vulnerability.
There has been a significant amount of effort in trying to make deep neural networks robust to adversarial perturbations (Goodfellow al., 2015 Madry et al.) 2018 Pinto al., 2017). However, in this arms race it has been shown that deep reinforcement learning policies learn adversarial features independent from their worst-case (i.e. adversarial) training techniques (Korkmaz 2022). More
intriguingly, line of work has focused on showing the inevitability of adversarial examples and the intrinsic difficulty of learning robust models (Dohmatob) 2019 Mahloujilar al. 2019) Gourdeau et al.) 2019). Given that it may not be possible to make DNNs completely robust to adversarial examples, natural objective is to instead attempt detect the presence of adversarial manipulations.
In this paper we propose a novel identification method for adversarial directions in the deep neural policy manifold. Our study the first one that focuses on detection of non-robust directions in the deep reinforcement learning neural loss landscape. Our approach relies on differences in the curvature of the neural policy in the neighborhood of an adversarial direction when compared to baseline state observation. At high level our method is based on the intuition that while baseline states have neighborhoods determined by an optimization procedure intended to learn a policy that works well across all states, each non-robust direction is the output of some local optimization in the neighborhood of one particular state. Our proposed method is computationally efficient, requiring only gradient computation and two policy evaluations, requires no training that depends on the

---

Under review as a conference paper at ICLR 2023
method used to compute the adversarial direction, and is theoretically well-founded. Hence, our study focuses on identification of non-robust directions and makes the following contributions:
Our paper is the first to focus on identification of adversarial directions in the deep reinforce- ment learning policy manifold.
We propose a novel method, Identification of Non-Robust Directions (INRD), to detect ad- versarial state manipulations based on the local curvature of the neural network policy. INRD is independent of the method used generate the adversarial direction, computationally efficient, and theoretically justified.
We conduct experiments in various MDPs from the Arcade Learning Environment that demonstrate the effectiveness of INRD in identifying adversarial directions computed via several ofte-oo--the-r the-art art adversarial attack methods.
Most importantly, demonstrate that INRD remains effective against multiple methods for generating non-robust directions specifically designed to evade INRD.
2 RELATED WORK AND BACKGROUND
Deep Reinforcement Learning: In this paper we focus on discrete action set Markov Decision Processes (MDPs) which are given by a continuous set of states S, a discrete set of actions A, a transition probability function P: S x A x S RR, and reward function R. S x A x S - IR. A policy т + (AA assigns probability distribution on actions (-118) to each state s. The goal in reinforcement learning is to learn the state-action value function that maximizes expected cumulative discounted rewards R = -.,-),) E. (R,, at 8t-1) by taking action a in state s. The temporal difference learning is achieved by one step Q-learning which updates Q(3i, at) by
a(), + a R + max Q(s++1,a) - Q(+,, at)]-
Adversarial Examples: Goodfellow et al. 2015) introduced the fast gradient method (FGM) for producing adversarial examples for image classification. The method based on taking the gradient of the training cost function J (z, y) with respect to the input image, and bounding the perturbation by where is the input image and y is the output label. Later, an iterative version of FGM called I-FGM was proposed by Kurakin et al. (2016). This is also often referred to as Projected Gradient Descent (PGD) as in (Madry al. 2018) where the I-FGM update is
N+1 adv
(1 + asign V J (x
(1)
where and x. Dong et al. 2018 further modified I-FGM by introducing momentum term
in the update, yielding a method called MI-FGSM. Korkmaz (020) later proposed a Nesterov-
momentum based approach for the deep reinforcement learning domain. The DeepFool method
of (MMooavv--eefoolli et al. (2016) is an alternative approach to those based on FGSM. DeepFool performs iterative projection to the closest separating hyperplane between classes. Another alternative
approach proposed by Carlini & Wagner (2017)) is based on finding a minimal perturbation that achieves a different target class label. The approach is based on minimizing the loss
min c [(sddr) + || $|[ EB
(2)
where s is the clean input, Sadv is the adversarial example, and (ss is a modified version of the cost function used train the network. Chen et al. (2018) proposed a variant of the Carlini & Wagner (2017a) formulation that adds an L f-rreuularization term to produce sparser adversarial examples,
min c [ssadr) + >, sadv - -s|, + do s SAN
(3)
Our method focusing identifying non-robust directions in the deep neural policy manifold is the first method to investigate detection of adversarial manipulations in deep reinforcement learning. Our identification method does not require modifying the training of the neural network, does not require any training tailored to the adversarial method used, and uses only two neural network function evaluations and gradient computation.
2

---

Under review as a conference paper at ICLR 2023
Adversarial Deep Reinforcement Learning: The adversarial problem initially has been investigated by Huang et al. (2017) and Kos & Song (2017) concurrently. In this work the authors show that perturbations computed via FGSM result In extreme performance loss on the learnt policy. [inn et al. (2117) and Sun et al. 2020) focused on timing strategies in the adversarial formulation and utilized the Carlini& & Wagner (2017a) method to produce the perturbations. While there is reasonable body of work focused on finding efficient and effective adversarial perturbations, à substantial body of work focused on building agents robust these perturbations, Mandlekar al. (2017) proposed utilize FGSM perturbations during training time to obtain more robust agents. |iint et al. (017)) modeled the adversarial interaction as a zero sum game and proposed a joint training strategy to increase robustness in the continuous action space setting. Recently, Gleave et al. (2020) considered an adversary who allowed to take natural actions in a given environment instead of f--norm bounded perturbations and modeled the adversarial relationship as zero sum Markov game. However, recent concerns have been raised on the robustness of adversarial training methods by Korkmaz (2022). In this paper the authors show that the state-of-the-ar adversarial training techniques end up learning similar non-robust features. Thus, with the rising concerns on robustness of recent proposed adversarial training techniques our work aims to solve the adversarial problem from different perspective by detecting adversarial directions.
3 IDENTIFICATION OF NON-ROBUSE DIRECTIONS (INRD)
In this section we give the high-level motivation for and formal description of our identification method. We begin by introducing necessary notation and definitions. We denote an original clean state by 3 and an adversarially perturbed state by sath
Definition 3.1. The cost of a state, J(s, r), is defined as the cross entropy loss between the policy T (als) of the agent, and target distribution actions r(a).
J(,
Σ ((())oo((((lb))
(4)
Definition 3.2. The argmax policy, x² (als), is defined as the distribution which puts all probability mass on the highest weight action of (()ss).
x (als) L(a arg arg mag r(''
(5)
We use the following notation for the gradient and Hessian with respect states
V [so To) = V.J (s. 7)s V TD) = V.J (=
3.1 FIRST-ORDER IDENTIHICATION OF NON-ROBUST DIRECTIONS (FO-INRD)
As naive baseline we first describe an identification method based on estimating how much the cost function (s, 7) varies under small perturbations. Prior work Roth et al. (2019); Hu et (2019) has shown that the behavior of deep neural network classifiers under small, random perturbations different at clean versus adversarial examples. Therefore, natural baseline detection method is: given an input state so sample small random perturbation 1) ~ NN,,, el) and compute.
K(so, n) = (so + 77, T (-1so)) [soo . (-1so)).
(6)
The first-order identification method proceeds by first estimating the mean and the variance of IC over clean run of the agent in the environment. Next threshold is chosen so that desired false positive rate (FPR) is achieved (i.e. some desired fraction of the states in the clean run lie more than standard deviations from the mean). Finally, at test time state encountered by the agent is classified as adversarial if it is at least standard deviations away from the mean. Ötherwise the state classified as clean. As first attempt, the first-order method can be naturally interpreted as a finite-difference approximation to the magnitude of the gradient at sp. If we assume that the first-order Taylor approximation of is accurate in a ball of radius r > centered at so, then
[so + n, " (-|ss)) z J(00 R" (-180)) V.J (so, "t" (.180) -n-
3

---

Under review as a conference paper at ICLR 2023
Therefore,
K[(s 7) se VsJ( (-((s))) n
(7)
Thus, for 1 ,N(00, cT) the test statistic ((sg. 7) is approximately distributed as a Gaussian with Under this interpretation one would expect the test
mean O and variance (+) statisties for clean and adversarial states to have the same mean with potentially different standard deviations, possibly making it hard to distinguish clean from adversarial. However, this is not what we observe empirically, and in fact the first-order method does decent job of detecting adversarial examples. The method works because, in fact, the mean of (C(5. 17) for clean examples is reasonably well separated from the mean of ([ n)) for adversarial examples sadv The empirical performance of the first-order method thus indicates that the assumption of accuracy for the first-order Taylor approximation of J does not hold in practice. This leads naturally to the consideration information on the second derivatives (i.e. the local quadratic approximation) of J in order to identify non-robust directions.
3.2 SECOND-ORDER IDENTIFICATION OF NON-ROBUST DIRECTIONS (SO-INRD)
The second-order identification method is based on measuring the local curvature of the cost function Ts, The method exploits the fact that J(s, T) will have larger negative curvature at a clean example compared an adversarial example. In particular, the high level theoretical motivation for this approach is that adversarial examples are the output of a local optimization procedure which attempts to find a nearby perturbed state sadv with low value for the cost (sdd .7) for some T * x (-18). A direction of large negative curvature for [(sadv,7) indicates that very small perturbation along this direction could dramatically decrease the cost function. Therefore, such points are likely to be unstable for local optimization procedures attempting to minimize the cost function in a small neighborhood. On the other hand, the curvature of J [(,,7 r) at a clean state s is determined by the overall algorithm used to train the deep reinforcement learning agent. This algorithm optimizes the parameters of the neural network policy while considering all states visited during training, and thus is not likely to be heavily overfit to the state 5. In particular, we expect larger negative curvature at & than at an adversarial example gādv We make the connection between negative curvature and instability for local optimization formal in Section 3.3 Based the above discussion, a natural choice of metric for distinguishing adversarial versus clean examples is the most negative eigenvalue of the Hessian Amia (so. xt (-|so)). While this is the most natural measurement of curvature, it requires computing the eigenvalues of a matrix whose number of entries are quadratic in the input dimension. Since the input very high- nigh-dimensional, and we would like to perform this computation in real-time for every state visited by the agent, computing the value Amin is computationally prohibitive. Instead we approximate this value by measuring the curvature along a direction which is correlated with the negative eigenvectors of the Hessian. Given this direction, the value that we measure is the accuracy of the first order Taylor approximation of the cost of the given state J (s, T). We denote the first order Taylor approximation at the state so in direction 17 by
isso 1) = J [so, x (-1s0) + V.J T (-1so))
The metric we will use to detect adversarial examples is the finite-difference approximation
L( 17) [sso + 17, n (-1so) J Jss, 1))
(8)
To see formally that Equation ( gives an approximation of the most negative eigenvector of the Hessian, will assume that the cost function J(s, T) is well approximated by its quadratic Taylor approximation at the point Su i.e.
[(so + 77. x (-1so)) s (So, T" (+|so)) + v x" (-Isu)) 17 + 17 V म (+1sa))n
(9)
for a small enough perturbation 77. Substituting the above formula into Equation ( yields
L( 11) Rs 1) J x (-160))m
(10)
The above quadratic form is minimized when 77 lies in the same direction as the most negative eigenvector of the Hessian, in which case
C(so. R Amin (V? x (-1s0))) |lm
(11)
We choose the sign of the gradient direction for measuring the accuracy of the first order Taylor approximation. To motivate this choice note that - [s, 7) is locally the direction of steepest
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Second Order Identification of Non-Robust Directions (SO-INRD)
Input:
The clean run mean C and variance (C), identification threshold t >> 0.parameter e >> 0.
for states 8; visited by deep reinforcement learning policy do
sign( V [si( x (-[s;))
7 V [J x (-1x)))ll
J mh) [s,
x² (-|s;)) V
x (+ [s, m1)
[(s,, T): [ss Ti. =" ().
if [C(ss, m7)) - - then
Label state 8; as non-robust direction
end if end for
decrease for the cost function. If the gradient direction additionally has negative curvature of large magnitude, then small perturbations along this direction will result in even more rapid decrease in the cost function value than predicted by the first-order gradient approximation. Note that this can be true even if the gradient itself has small magnitude, as long as the negative curvature is large enough. Thus, by the discussion at the beginning of Section 3.2 adversarial examples are likely to have relatively smaller magnitude negative curvature in the gradient direction than clean examples. Formally, for
0 we set
(50)) sign V J x (-|so)))
(12)
V x+ (+
To calibrate the detection method we record the mean [ = E, (Css, [))))] and variance a² (L))
(ar,[C(s, nss)] of our proposed test statistic over states from a clean run of the policy in the MDP.
Then at test time we set a threshold > 0, and for each state s; visited by the agent test ((ss,)(s))
2 (C)
(13)
If the threshold of standard deviations is exceeded classify the state 8; as adversarial, and otherwise classify it as clean. Pseudo-code for the second order method is given in Algorithm
3.3 NEGATIVE CURVATURE AND INSTABILITY OF LOCAL OPTIMIZATION
In this section formalize the connection between negative curvature and instability for local optimization procedures that motivated our definition of [(s, I)). Given a state so and a target distribution # n" (-Isa), we assume the adversary trying find a state gadv minimizing ( .)) among all states close to so by some metric. Formally, let D., (s) 2 0 be a convex function of s that should be thought of measuring distance to $g- One standard choice for the distance function is Deo (s) |s so Ilp. We model the adversary as minimizing the loss
((s) = J(, + Dao
(14)
In particular, we make the following assumption: Assumption 3.1. The adversarial state gadv is local minimum of f(s).
Of course this assumption is violated in practice since different methods used to compute adversarial directions optimize objective functions other than f. and do not necessarily always converge to a local minimum. Nevertheless the assumption allows us to make formal qualitative predictions about the behavior of the second-order identification method that correspond well with empirical results across a broad variety of methods for generating adversarial directions. We now state our main result lower bounding the curvature of J (gadv,,7 ,T).
Proposition 3.2. For > O assume that the maximum eigenvalue of the Hessian ViDDo (8) is bounded by c. If s* is a local minimum of f(s) then Xmin ( (s"
Proof. Let v be the eigenvector of [sJ(s" . .) corresponding to the minimum eigenvalue. At local minimum s" of (s) the Hessian (2](s") must be positive semi-definite. Therefore,
0 S V2 = VV.s(s' T)E + V.D.o, (s")
S Amin
+c
5

---

Under review as a conference paper at ICLR 2023
150
Figure I: L(s) for our proposed method SO-INRD vs visited states with corresponding TPR values for the following attack methods: FGSM, MI-FGSM, Nesterov, DeepFool, Carlini& Wagner, Elastic Net Method. TPR values shown in the upper right box of the figure when FPR is equal to 0.01.
Rearranging the above inequality completes the proof.
The second order conditions for local minimum of f imply lower bound on the smallest eigenvalue of V²J[(",, r). Thus, by Assumption 3.1. we obtain a lower bound on Amin (V.J(s .T). The assumption that the maximum eigenvalue of the Hessian V (s) is bounded by is satisfied for example when Da, (a) = |s 8n l/-- In contrast, the local curvature of the cost function [(s, 7) at clean example is determined by optimization procedure that updates the weights @ of the neural network policy rather than the states If we write Jg(s, 7) to make explicit the dependence on the weights, then the second order conditions for optimizing the original neural network apply to the Hessian with respect to weights (o T) rather than the Hessian with respect to states Jo,s(s T). Additionally, first order optimality conditions can help to justify the choice of V (s. 7) as good direction to check for negative curvature. Indeed by the first order conditions, at local optimum s* of (s) we have
0 = -
+ VV (o0
(15)
So assuming the adversary finds a local optimum, V J 7) points in direction that decreases the distance function Dx, (s)). Thus sufficiently negative cur- vature in the direction of [s,(s, T) implies not only that is not a local minimum of f. but also that the distance function Dao (s) can be decreased by moving along this direction of negative curvature. To summarize, we have shown that second order optimality conditions arising from computing an adversarial example give rise to lower bounds on the smallest eigenvalue of the Hessian Amin JV= 7)). The function C(s, n)) used to identify adversarial directions for SO-INRD is finite difference approximation
Therefore,
17 V J T)n 2 Amia (V?J(s,T)) ||
Therefore the results of this section imply that L(s. should be larger at adversarial examples than clean examples.
4 EXPERIMENTS
In our experiments agents are trained with DDQN Wang et al.] (2016) in the Areade Learning Environment (ALE) Bellemare et al. (2013) from OpenAl Brockman et al. (2016). For a baseline we compare FO-INRD and SO-INRD with the detection method of OAO proposed by RRoth et al. (2019), which is based on estimating the average change in the odds ratio between classes under noise.
In Figure we plot the value of [(s) over states for various games without an adversarial attack and under adversarial attack with the following methods: Carlini & Wagner, Elastic Net, Nesterov Momentum, DeepFool, MIFGSM and FGSM. We show in the legends of Figure | the true positive rate (TPR) values for the different attacks when false positive rate (FPR) is equal to 0.01. The value of C(s) for clean states is generally colcennncetr and negative. On the other hand, for states computed by the different adversarial attack methods L(s) is clearly larger, matching the predictions

---

Under review as a conference paper at ICLR 2023
R00M
Deupfiz!
RDC Carlin
03
530
6.5
1.05
0.00
3315
9.00
0128
3025
8.300
630 0325
0150
8.55
(R (Willsss Pathes hant
RDC Medtonyy Moma maam
RCC Etartic-
10
01..00 180
canue aic00s 3014
3.35
3000
0001
0030
0.9
0.5
1.0
0.5
030
8.30 035
Figure 2: ROC curves of FO-INRD, SO-INRD and OAO method for the following attack meth- ods: FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini&Wagnee Elastic Net Method in RoadRunner. TPR values shown in the lower right box of the figure when FPR is equal to 0.01.
Table 1: True Positive Rates (TPR) for FGSM, MI-FGSM, Nesterov Momentum, Carlini& Wagner, Elastic-Net and DeepFool when False Positive Rate (FPR) is equal 0.01. The proposed methods SO- INRD and FO-INRD are evaluated, and compared with Roth et al. (OAO) Riverraid, RoadRunner, Alien, Seaquest, Boxing, Pong, and Robotank games. More results for different FPR values reported in the supplementary material.
Detection Method-Attack Method
RiverRaid
RoadRumner
Alien
Seaquest
Boxing
Pong
Robotank
SO-INRD FGSM
0.997
1.0
1.0
0.995
0.994
1.0
0,999
FO-INRD FGSM
0.990
0.843
0.803
0.931
0.793
0.622
0.413
OAO FGSM
0.681
0.767
0.885
0.403
0.264
0.424
0.911
SO-INRD M-IFGSM
0.998
1.0
1.0
0.985
0.910
1.0
0.985
FO-INRD M-IFGSM
0.952
0.863
0.991
0.981
0.827
0.622
0.470
OAO M-IFGSM
0).75
0.554
0.929
0.581
0.499
0.679
0.777
SO-INRD Nesterov Momentum
0.995
0.989
0.996
0.952
0.865
1.0
0.954
FO-INRD Nesterov Momentum
0.990
0.714
0.997
0.979
0.746
0.633
0.574
OAO Nesterov Momentum
0.785
0.646
0.925
0.671
0.517
0.687
0.753
SO-INRD Carlini& Wagner
0.910
0.988
0.945
0.723
0.856
0.850
0.713
FO-INRD Carlini& Wagner
0.695
0.594
0.642
0.516
0.785
0.494
0.119
OAO Carlini& Wagner
0.036
0.118
0.018
0.004
0,016
0.028
0.032
SO-INRD Elastic Net
0.777
0.943
0.875
0.687
0.770
0.736
0.815
FO-INRD Elastic Net
0.685
0.454
0.561
0.502
0.743
0.361
0.212
OAO Elastic Net
0.124
0.210
0.060
0.014
0.150
0.092
0.056
SO- SO-INRD DeepFool
0.914
0.996
0.993
0.860
0,951
0.889
0,900
FO-INRD DeepFool OAO DeepFool
0.841
0.847
0.936
0.777
0.928
0.796
0.268
0.397
0.447
0.611
0.234
0.381
0.367
0.607
of Proposition 3.2 The fact that L(s) is consistently larger at adversarial examples across wide variety of adversarial perturbation methods indicates that Assumption 3.1 qualitatively captures the behavior of these methods. In particular the FGSM-based methods and DeepFool do not explicitly optimize objective function of the form f() J(s,, r) + Deg (s) in Assumption 3.1 However, by enforcing a constraint on the distance of the adversarial example from the original clean example, these methods implicitly solve optimization problem of the form given in (14.. and thus exhibit the qualitative behavior predicted by Proposition 3.2
In Table I we show TPR values for FO-INRD, SO-INRD, and the OAO method under the FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini& Wagner, and Elastic-No attacks when FPR equal 0.01. For all of the attack methods in all of the environments SO-INRD is able detect

---

Under review as a conference paper at ICLR 2023
HA0 HUSM
Dosstu
VCC Curkiiiiturer
c3-
8/0 90-
Aile 200
1.30
8.3
2/4
1.3
825
625
030
025
0.8
C8
C3
C2
C3
C4
65
6.6
ROC Dante
1.01
03
3.04
5.55
8 03
03
40 0:00 -277
0210 0203
8:31
832
3.05 304
3.0
3:25
050
Figure 3: ROC curves of FO-INRD, SO-INRD and OAO method for the following attack meth- ods: FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini& Wagner, Elastic Net Method in Robotank. TPR values are reported in the lower right box of the figure when FPR is equal to 0.01.
Table 2: TPR for Feature Matching for SO-INRD and OAO method FPR=0.01
Feature Matching
Riverraid
RoadRunner
Alien
Seaquest
Boxing
Robotank
SO-INRD
0.882
0.863 0.006
0.9016
0.955
0.988
0.8978
OAO Method
0.0088
0.007
0.0146
0.0106
0.0158
adversarial perturbations with large TPR. SO-INRD outperforms the other detection methods in all cases except for Nesterov Momentum in Alien and Seaquest where FO-INRD has TPR 0.997 and 0.980 while SO-INRD has 0.996 and 0.952. We also observe that while the perturbations computed by FGSM, MI-FGSM, Nesterov Momentum can generally be detected with large TPR values by all the detection methods, the perturbations computed by Carlini& Wagner and the Elastic-Net method are more difficult to detect. Despite the difficulty, SO-INRD achieves TPR values ranging from 0.713 to 0.988 for Wagner, and TPR values ranging from 0.687 to 0.943 for Elastic-Net when FPR is equal to 0.01. In Figure 2 and Figure 3 we show ROC curves for each detection method under the FGSM, MI-FGSM, Nesterov Momentum, DeepFool, Carlini&Wagner and Elastic-Net method for RoadRunner and Robotank respectively. In Robotank the OAO method outperforms FO-INRD and even approaches the TPR of SO-INRD for high FPR under FGSM, MI-FGSM, Nesterov Momentum and Foop However for the Carlini& Wagner and Elastic-Net attacks, SO-INRD has a much higher TPR across wide range of FPR levels.
5 COMPUTING ADVERSARIAL DIRECTIONS SPECIFICALLY TO EVADE INRD
Recently, Tramer et al. 2020) introduced a comprehensive methodology for tailoring the optimization procedure used to produce adversarial examples in order to overcome detection and defense methods. In particular, the high level idea is to keep the attack as simple as possible while still accurately targeting the detection method. More specifically, the methodology is based on designing an attack based on gradient descent on some loss function. Further, minimizing the loss function should correspond closely to subverting the full detection method while still being possible to optimize. Critically, the authors highlight that while the choice of loss function to optimize can be difficult task, the use of "feature matching" Gooaa et al. (2019) can circumvent most of the current detection methods. We now describe how we applied the methodology discussed above to design detection aware adversaries for SO-INRD. As a first attempt, we tested the "feature matching" approach that was used to break the OAO detection method in Tramer et al. (2020). This approach attempts to match the logits of the adversarial example to those of clean example from a different class in order used up to 1000 PGD iterations, and we searched step size varying from 0.01 to 10 6 We find that this method succeeds in reducing the TPR of the OAO method to nearly zero. It is also able to slightly reduce the TPR of our SO-INRD method
to evade detection. To optimize the loss for this method
8

---

Under review as a conference paper at ICLR 2023
Table 3: TPR values of INRD in the presence of a identification aware adversary when FPR=0.01.
Detection Method
RiverRaid
RoadRunner
Alien
Seaquest
Boxing
Pong
Robotank
SO-INRD C&W 0.650 0.346
0.849 0.348
0.445
0,381
0.710
0.712
0.657
FO-INRD C&W W
0.351
0.193
0.621
0.325
0.0973
(see results in Table However, as we will see next, a larger reduction in the TPR of SO-INRD can be achieved by optimizing modified version of the loss from Carlini & Wagner 7b). Our next attempt is based on a modification of the Carlini Wagner (2017b) formulation to additionally minimize the cost function [(s) used in SO-INRD,
min c. [(sddv) + +||stdy =112 + x -(sudd)) E8
(16)
Recall that C(s) is consistently larger at adversarial examples than at clean examples. Thus the above optimization problem attempts to find adversarial examples with as small values of [(s) as possible. Since the function [(s) involves taking the sign of the gradient, we use Backwards Pass Differentiable Approximation (BPDA) as introduced in Carlini & Wagner (2017b) to compute the gradients. However, we also tried designing an adversary with fully differentiable cost function by using perturbation in the gradient direction (without the sign). We found that this fully differentiable adversary performed significantly worse than the one based on BPDA. We conducted exhaustive grid search over all the parameters in this optimization method: learning rate, iteration number, confidence parameter KG and objective function parameter A. In C&W we used up to 30000 iterations to find adversarial examples to bypass detection methods. We searched the confidence parameter from O to 50, the learning rate from 0.001 to 0.1, and from 0.001 to 10. In our grid search over these hyperparameters found that there is trade-off between the attack success rate and the detection of the perturbations. In other words, if we optimize the perturbation to be undetectable the success rate of the perturbation (i.e. the rate which the perturbation actually makes the agent choose non-optimal action) decreases. Therefore, when finalizing the hyperparameters for the SO-INRD detection- aware adversary we restricted our search to setting where the decrease the success rate of the attack was at most 10%.
Since FO-INRD is based on sampling a random perturbation, we use another approach introduced by Carlini& & Wagner (2177)) to minimize the expectation of the original loss function when averaged over the randomness used in the detection method. In particular, we estimate the expectation by computing the empirical mean of the loss over 50 samples from the same noise source. As for the case of SO-INRD we grid search over hyperparameters to achieve as low TPR as possible while losing at setting with
most 10% in the success rate of the attack. Table shows the TPR in the adversary the best hyperparameters found for each method. The fact that SO-INRD still performs quite well in the adversary-awy
setting is indication that there is a fundamental trade-off between computing an adversarial example and minimizing C(s). This trade-off makes sense in light of Proposition 3.2 which shows that searching for an adversarial example in small neighborhood will tend to increase (s)).
6 CONCLUSION
In this paper we introduce novel algorithm INRD, the first method for identification of adversarial directions in the deep neural policy manifold. Our method is theoretically motivated by the fact that local optimization objectives corresponding to the construction of adversarial directions lead naturally to lower bounds on the curvature of the cost function J(ss, r). We have further shown empirically that the curvature of (s, T) is significantly larger at adversarial states than at clean observations, leading to highly effective method SO-INRD for detecting adversarial directions in deep reinforcement learning. We additionally demonstrate that SO-INRD remains effective in the adversary and connect this fact to original theoretical motivation. We believe that due to the strong empirical performance and solid theoretical motivation SO-INRD can be an important step towards producing robust deep reinforcement learning policies.
setting,

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael. Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research., pp. 253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv: 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57, 2017a.
Nicholas Carlini and David A. Wagner. Adversarial examples are not casily detected: Bypassing ten detection methods. In Bhavani M. Thuraisingham, Battista Biggio, David Mandell Freeman, Brad Miller, and Arunesh Sinha (eds.), Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec @ CCS 2017, Dallas, TX, USA, November 2017, pp. 3-14. ACM, 2017b.
Pin- Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. EAD: elastic-net attacks to deep neural networks via adversarial examples. In Sheila A. Mcllraith and Kilian Q. Weinberger Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 10-17. AAAI Press, 2018.
(eds.), Proceedings of the Thirty-Second AAAI Conference
Elvis Dohmatob. Generalized no free lunch theorem for adversarial robustness. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1646-1654.
PMLR, 09-15 Jun 2019.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2.18.
Adam Gleave, Michael Dennis, Cody Wild, Kant Neel, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. International Conference on Learning Reprexentations ICLR, 2020.
Ian Goodfellow, Jonathan Shelens, and Christian Szegedy. Explaning and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Pascale Gourdeau, Varun Kanade, Marta Kwiatkowska, and James Worrell. On the hardness robust classification. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Ach--Buu Emily B. Fox, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp.
2019.
Sven Gowal, Jonathan Uesato, Chongli Qin, Po-Sen Huang. Timothy A. Mann, and Pushmeet Kohli. An alternative surrogate loss for pgd-based adversarial testing. 2019.
Shengyuan Hu, Tao Yu, Chuan Guo, Wei-Lun Chao, and Kilian Q. Weinberger. A new defense against adversarial images: Turning weakness into strength. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buce Buc, Emily B. Fox, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1633-1644, 2019.
Sandy Huang, Nicholas Papernot, Yan Goodfellow. Ian an Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. Workshop Track of the 5th International Conference on Learning Representations, 2017.
Ezgi Korkmaz. Nesterov momentum adversarial perturbations in the deep reinforcement learning domain. International Conference on Machine Learning, ICML 2020, Inductive Biases, Invariances and Generalization in Reinforcement Learning Workshop., 2020.
10

---

Under review as a conference paper at ICLR 2023
Ezgi Korkmaz. Deep reinforcement learning policies learn shared adversarial features across mdps. AAAI Conference on Artificial Intelligence, 2022.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. International Conference on Learning Representations, 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: (607.02200003 2016.
Yen-Chen Lin, Hong Zhang-War Meean Yuan-Hong Liao, Meng-Li Shih, ing-Yu Liu, and Min Sun. Tacties of adversarial attack on deep reinforcement learning agents. Proceedings of the Twenty-Sixit International Joint Conference on Artificial Intelligence, pp. 3756-3762, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, In 6th International Conference on Learning Representations, ICLR 2018, Vancouver. BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Saced Mahloujifar, Xiao Zhang. Mohammad Mahmoody, and David Evans. Empirically measuring Fundamental limits on intrinsic robustness. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5210-52211 2019.
concentration:
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of pyysically-plusibee perturbations. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, are G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518: 529-533, 2015.
Seyed- Moheen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Decpfool: A simple and accurate method fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2574 2582. IEEE Computer Society, 2016.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. International Conference on Learning Representations ICIR, 2017.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarial examples. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5498-5507. PMLR, 2019.
Jianwen Sun, Tianwei Zhang, Lei Xiaofei, Xie Ma. Yan Zheng, Kangjie Chen, and Yang. Liu. Stealthy and efficient adversarial attacks against deep reinforcement learning. Association for the Advancement of Artificial Intelligence (AAAI), 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dimutru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. NewrIPS, 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando. De Freitas. Dueling network architectures for deep reinforcement learning. Internation Conference on Machine Learning ICMI., pp. 1995-2003, 2016.


Reviews:
[{"review_id": "tpTqIPz7-Q", "reviewer": "Reviewer_9bNZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an algorithm for identification of adversarial directions in the deep neural policy manifold. The method is motivated from the insufficiency of first order approximation of the loss function; instead, the method relies on the second order information of the loss (curvature). Empirical results support the effectiveness of the proposed SO-INRD method.", "summary_of_the_review": "My major concern is what is the differences of studying adversarial detection in RL compared to that in classification problems. I am giving an initial negative rating, and happy to involve in the discussion with the authors. If my concern is lifted, I will raise my score.\n\n=========== Post author response ===========\n\nThank you for addressing my comments and questions!\n\nI still believe temporal dependency should be comprehensively discussed and tackled by the methodology that aims at MDPs, instead of treating it as a future direction. Otherwise, the method is not very different from those in standard classification problems. Given the vast literature in robust classification, the paper's contribution can be undermined.\n\nThe gap between theory and experiments are not explained convincingly in response. I am dubious about whether the experiments support the theory, as assumption is violated.", "strength_and_weaknesses": "=========== Strength ===========\n\nRather than directly working with the second order detection method, the paper starts with the first order method and indicates the inaccuracy of the first order Taylor approximation. This type of reasoning stimulates the interest of readers and provides background for later more involved discussions.\n\nApart from theoretical discussions, the paper provides abundant empirical results to showcase the effectiveness of the proposed method.\n\n=========== Weakness ===========\n\nThe connection to reinforcement learning is pretty weak. Suppose we consider the policy learning as a classification problem, where the state $s$ is viewed as input feature and the policy output is the classification odds (on a simplex). Then the paper indeed discusses detecting adversarial directions in general classification problems. I am a bit confused why reinforcement learning problems are the main target here, providing that the paper does not consider any temporal dependencies.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i-8yBJWJo4", "reviewer": "Reviewer_8ZPv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method of identifying adversarial directions for robustness in deep reinforcement learning. The authors show theoretically that there is a lower bound on the curvature of a cost function that is related to a local adversarial optimization. Empirically, they show that this curvature is greater at adversarial states. They develop a method based on this observation, which they show is more effective at identifying adversarial examples than some baselines.", "summary_of_the_review": "The authors method performs impressively in the scenario where it is tested, which is a proxy for the objective the authors really care about. The authors do not justify why their experimental setup is the correct one for this problem. While the method has theoretical justification, the evaluation should be primarily empirical.\n\nPost author response:\nI don't understand the authors' response that it is enough to detect adversarial states. The purpose has to be to achieve a high reward in the presence of an adversary. If it is just adversarial manipulation detection, it seems like the problem isn't specific to RL.\n", "strength_and_weaknesses": "Strengths:\n1. The paper suggested an interesting detection method for adversarial directions and conducted experiments under many different environments and attack methods.\n2. The paper addresses important issues for making unstable reinforcement learning robust.\n\nWeaknesses:\n1. The authors claim implicitly that better detection of adversarial states leads to better performance (e.g., in average return). But they do not explicitly test this.\n2. It is unclear why the particular test environments were selected, but they do not look randomly chosen.\n3. This \"adversarial defense\" seems to suffer especially badly from the adversary being aware of it—they can now broaden their search to try to find states that avoid the curvature condition.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Byi7NeMNZPL", "reviewer": "Reviewer_Soiv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors studied a method that can detect the presence of unstable states. They achieve that by measuring the gap between the perturbed cost and the first order approximation. I this gap is large (one the direction is chosen in a worst case fashion) then the second order curvature is large as well (in a worst case sense). The authors define such states as unstable. \n\nBesides of suggesting this method, the authors also conducted experiments on the ALE environment.", "summary_of_the_review": "The authors detect sensitive states by detecting gaps between first and second order approximation of the cumulative cost function. Assuming access to offline data, the authors can detect problematic states from which small perturbations can lead to different results.  ", "strength_and_weaknesses": "Strengths.\n-) The authors put forward a method to recover particularly sensitive states: states from which small perturbations can result in dramatic differences in the future. This is worthwhile from real-world perspective and can be a nice debugging tool for such applications.\n-) The paper is well written in my opinion (besides of minor comments). The intuition is very clear.\n\nWeaknesses.\n-) The algorithm and method are quite simple and straightforward. On the other hand, the experiments are also quite basic (in the sense that the authors didn't show that their method leads to a new capability that was not achieved prior to this work).\n-) The authors considered only detection. I believe it should be possible to built mitigations using a min-max approach (e.g., the one that was taken in the action robust framework https://arxiv.org/pdf/1901.09184.pdf). This would make this work more complete in my opinion.\n-) There's a need to estimate the cost function using model free approaches. It might be the case that this due to errors at this step the algorithm would not detect the non robust states.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8R6upIUC3T6", "reviewer": "Reviewer_FZra", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel approach to detect adversarial state manipulations in deep Reinforcement Learning (RL). It provides a theoretical analysis that motivates that second-order gradient information w.r.t. a cost function contains information that allows to identifiy non-robust directions (SO-INRD) and, thus, adversarial perturabtions. Furthermore, it proposes a method to approximate the measure of curvature (i.e., the Hessian eigenvalues) in a computationally feasible manner. The method is validated empirically on DDQN agents trained on the Arcade Learning Environment, adversarially attacked by various different methods, and compared against several baseline methods for detecting adversarial perturbations.\n", "summary_of_the_review": "The paper tackles an important problem, namely adversarial perturbations in RL, and proposes a theoretically well-founded and nicely motivated approach to detect these. The presentation is clear and seems to be theoretically and empirically sound. I propose to accept the submission, but me recommendation is with low confidence, so I might adjust my score after the discussion period.\n", "strength_and_weaknesses": "Unfortunately, I am unfamiliar with the reasearch area of adversarial perturbations and thus can only give high level feedback with low confidence. I think the paper is a strong submission because:\n\n- Detecting adversarial perturbations is important to make RL applicable to real-world scenarios.\n- The paper is very well structured and written.\n- The experimental section appears to be well fleshed out.\n- The method is very simple to implement.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject