Revealing Dominant Eigendirections via Spectral Non-Robustness Analysis in the Deep Reinforcement Learning Policy Manifold
Authors: Ezgi Korkmaz
Venue: Submitted to ICLR 2023 (2023)

Abstract: Deep neural policies have recently been installed in a diverse set of settings, from biotechnology to automated financial systems. However, the utilization of deep neural networks to approximate the state-action value function commences concerns on the decision boundary stability, in particular, with regard to the sensitivity of policy decision making to indiscernible, non-robust features due to highly non-convex and complex deep neural manifolds. These concerns constitute an obstruction to understanding the reasoning made by deep neural policies, and their foundational limitations. Thus, it is crucial to develop techniques that aim to understand the sensitivities in the learnt representations of neural network policies. To achieve this we introduce a method that identifies the dominant eigen-directions via spectral analysis of non-robust directions in the deep neural policy decision boundary across both time and space. Through experiments in the Arcade Learning Environment (ALE), we demonstrate the effectiveness of our spectral analysis algorithm for identifying correlated non-robust directions, and for measuring how sample shifts remold the set of sensitive directions in the neural policy landscape. Most importantly, we show that state-of-the-art adversarial training techniques yield learning of sparser high-sensitivity directions, with dramatically larger oscillations over time, when compared to standard training. We believe our results reveal the fundamental properties of the decision process made by the deep reinforcement learning policies, and can help in constructing safe, reliable and value-aligned deep neural policies.

Under review as a conference paper at ICLR 2023
REVEALING DOMINANT EIGENDIRECTIONS VIA SPEC- TRAL NON-ROBUSTNESS ANALYSIS IN THE DEEP RE- INFORCEMENT LEARNING POLICY MANIFOLD
Anonymous authors Paper under double-bling review
ABSTRACT
Deep neural policies have recently been installed in a diverse set of settings, from biotechnology automated financial systems. However, the utilization of deep neural networks to approximate the state-action value function commences con- cemss on the decision boundary stability, in particular, with regard to the sensitivity of policy decision making to indiscernible, non-robust features due to highly non- convex and complex deep neural manifolds. These concerns constitute an obstruc- tion to understanding the reasoning made by deep neural policies, and their foun- dational limitations. Thus, it is crucial to develop techniques that aim to under- stand the sensitivities in the learnt representations of neural network policies. To achieve this we introduce a method that identifies the dominant eigen-directions via spectral analysis of non-robust directions in the deep neural policy decision boundary across both time and space. Through experiments in the Arcade Learn- ing Environment (ALE), we demonstrate the effectiveness of our spectral analysis algorithm for identifying correlated non-robust directions, and for measuring how sample shifts remold the of sensitive directions in the neural policy landscape. Most importantly, we show that state-of-the-a adversarial training techniques yield learning of sparser high-sensitivity directions, with dramatically larger os- cillations over time, when compared to standard training. We believe our results reveal the fundamental properties of the decision process made by the deep rein- forcement learning policies, and can help in constructing safe, reliable and value- aligned deep neural policies.
INTRODUCTION
Reinforcement learning algorithms leveraging the power of deep neural networks have obtained state-of-the tott--of-the-at results initially in game- game-playing tasks tasks[Mnih et (2015) and subsequently in contin- control Lillicrap al., (2015). Since this initial success, there has been continuous stream of developments both of new algorithms Mnih et al.) (2016), Hasselt et al.) (2016). Wang et al.) (2016), and striking new performance records highly complex tasks Silver et al. (0117) While the field of deep reinforcement learning has developed rapidly, the understanding of the representations learned by deep neural network policies has lagged behind.
The lack of understanding of deep neural policies is of critical importance in the context of the sensitivities of policy decisions to imperceptible, non-robust features. Beginning with the work of Szegedy et al. (2014), Goodfellow et al.] (2055), deep neural networks have been shown to be vul- nerable to adversarial perturbations below the level of human perception. In response, line of work has focused on proposing training techniques to increase robustness by applying these perturbations to the input deep neural networks during training time (i.e. adversarial training) [Gooffeloo)) al. (2015) Madry et al. (2017). However, there are also concerns about adversarial training in- cluding decreased accuracy on clean data Bhagoji et al. (2019), prohibiting generalization (e.g. a performance gap between adversarial and standard training that increases as the size of the dataset grows Chen et al. (2020), and incorrect invariance to semantically meaningful changes Tramer et al.
2020

---

Under review as a conference paper at ICLR 2023
In paper focus understanding the learned representations and policy vulnerabilities and ask the following questions: (i) How do non-robust directions on the deep neural policy manifold interact with each other temporally and spatially? (ii) Do the non-robust features leartt by the deep reinforcement learning algorithms transform under adversarial attacks? (iii) How do these learnt correlated non-robust features change with distributional shift? (iv) Does adversarial training solve the problem of learning correlated non-robus features? To be able to answer these questions in our paper we focus on understanding the representations learned by deep reinforcement learning policies and make the following contributions:
We propose novel tracing algorithm analyze the spatially and temporally correlated vulnerabilities of deep reinforcement learning policies.
We conduct several experiments in the Arcade Learning Environment with policies trained in high-dimennional state representations.
We go over several benchmarked adversarial attack techniques and show how these attacks affect the vulnerable learned representations.
We inspect the effects of distributional shift on the correlated non-robust feature patterns learned by deep reinforcement learning policies.
Finally, we investigate the presence of non-robust features in adversarially trained deep neural policies, and show that the state-of-thed arrt adversarial training method leads to learn- ing sparser and spikier vulnerable representations.
2 BACKGROUND AND PRELIMINARIES
2.1 PRELIMINARIES
A Markov Decision Process (MDP) is defined by tuple (S, A, R) where S is set of states, A P,R) a set actions, P x A - [0. 1] is Markov transition kernel, R. S x reward function, and y is discount factor. A reinforcement learning agent interacts with an MDP
+ IR is
by observing the current state s E S and taking an action a E A. The agent then transitions to state s' with probability P(s, a, s') and receives reward R(s, a, s'). A policy action a in state with probability x(s, a). The main objective in reinforcement learning is to learn policy which maximizes the expected cumulative discounted rewards
A - [0. selects
R E (ne))))
Σn Dt, 8141)-
This maximization achieved by iterative Bellman update
Qss,, ax) ) = R(st, ax) + Σ ae
(I)
Hence, the optimal policy T5 a) can be achieved by executing the action a"(s) arg max, (ss a) i.c the action maximizing the state-action value function in state s.
2.2 ADVERSARIAL PERTURBATION TECHNIQUES AND FORMULATIONS
Following the initial study conducted by (Szegedy et al. (2014) Goodfellow et al. 2015)) proposed fast and efficient way to produce e-bounded adversarial perturbations in image classification based on linearization of J(r, y). the cost function used train the network, at data point with label Consequently, Kurakin et al. (2016) proposed the iterative form of this algorithm named iterative fast gradient sign method (I-FGSM).
.N1 adv
clip, (1 + asign V ( v))).
This algorithm further has been improved by the proposal of the utilization of the momentum term by Dong et al.) (2018). Following this Ezgi (2020) proposed a Nesterov momentum technique to compute e-bounded adversarial perturbations for deep reinforcement learning policies by computing
2

---

Under review as a conference paper at ICLR 2023
the gradient at the point suv + p vt,
Mati [ + # vz, a)
u++1 po
V Mate J (S + 1 ve, a)
(3)
.+1 adv
sadv + Or ||ee+ll2
U1+1
Another class of algorithms for computing adversarial perturbations focuses on different methods for computing the smallest possible perturbation which successfully changes the output of the target function. The DeepFool method of MMoosavi-Dezfooll et al. (2016) works by repeatedly computing projections to the closest separating hyperplane of a linearization of the deep neural network at the current point. Carlini & Wagner (2017) proposed targeted ad adversarial formulations in image classi- fication based on distance minimization between the original sample and the adversarial sample
min c (Zadv) 2aad x1|
(5)
Mat Ex
Another variant of this algorithm was proposed by Chen et al. 2018 based on f--regularization of the f2-norm bounded Carlini Wagner (2017) adversarial formulation.
min J (srad) + 0, Tady ar|1, + 0a 1 Fady 1r11
(6)
2.3 DEEP REINFORCEMENT LEARNING POLICIES AND ADVERSARIAL EFFECTS
Beginning with the work of Huang al. 2017 and Kos & Song (017). which introduced ad- versarial examples based on FGSM to deep reinforcement learning, there has been a long line of research on both adversarial attacks and robustness for deep neural policies. On the attack side, the work of Lin et al. (2017) utilized the Carlini&Wagner method to compute perturbations in only subset states, strategically chosen to maximize impact. Ezgi (020) showed Nesterov momentum produced adversarial perturbations that are faster to compute compared to Carlini & Wagner (2017) with similar or better impact on the policy performance. More intriguingly. the work of Korkmaz (022) showed that deep reinforcement learning policies learn similar adversarial directions across MDPs intrinsic to the training environment. On the defense side Pinto al. (2177) model the inter- action between adversary producing perturbations and the deep neural policy taking actions zero-sum game, and train the policy jointly with the adversary in order to improve robustness. More recently Huan et al. (2020) formalized the adversarial problem in deep reinforcement learning by introducing a modified MDP definition which they term State- Adversarial MDP (SA-MDP). Based this model the authors proposed theoretically principled certified robust adversarial training algorithm called SA-DQN, While recent studies raised some concerns on the drawbacks of adver- sarial training techniques Korkmaz (202I) in terms of limiting the generalization capabilities of the trained policy, these studies lack a method of explaining and understanding the main problems of the adversarial training techniques.
PRINCIPAL -ENSSIENSIY DIRECTION ANALYSIS
In our paper our goal is to seek answers for the following questions:
How can we inspect the correlated non-robust featturs across time and space in deep rein- forcement learning policies?
How does distributional shift affect the vulnerabilities of learned representations?
Do correlated non-robust features transform for policies under adversarial perturbations?
Does adversarial training prohibit learning correlated non- non-robust features?
To be able to answer these questions we propose novel policy imaging technique that traces corre- lated high-sensitivity directions in the deep neural policy landscape across time. In the remainder of this section explain in detail our proposed method.
Definition 3.1. Let IRd be a vector with ||o||2 1. Let Q be a state-action value function and for state let g = + ew. For 0, the vector is an e-high-sensitivity direction for Q in state
arg max Q($, arg max ( a)) Q(5, arg max Q(s,a)).
||

---

Under review as a conference paper at ICLR 2023
In words, is high-sensitivity direction if adding perturbation of Ey-norm epsilon along maxi- mizes the difference between the optimal state-action value in the new state and the value assigned in the new state to the previously optimal action. Eqn 7 can be approximated by using the softmax cross entropy loss. Hence, we define π (8, a) as the softmax policy of the state-action value function.
((Q(),))/T)
T (8, En'ea e(Q(=aa)/T)
The cross entropy loss between the softmax policy in state sy and the argmax policy r(s,a) mamarg m =(s,at) (a) at state is
(8,88) = T(( a) log(T((), log( (()))
aEA
Therefore by definition of the softmax policy we have
sg) = logg Σ
Q(sg. a' (s)) Q(sg. a" (s²)) Q(sg a" (s))
a'EA
where the final approximate equality becomes equality T O. Setting v Sy shows that maximizing the softmax cross entropy approximates the maximization Definition 7 Hence, the gradient J(s, (g)),== gives the direction of the largest increase in cross-entropy when moving from state s. Intuitively this is the direction along which the policy distribution π (s, a) will most rapidly diverge from the argmax policy. Hence, V., J(s, sy)ls, is a high-sensitivity direction in the neural policy landscape in state 8. Most importantly. the reason we trace high-sensitivity direc- tions uncover the non-robust features learnt by the deep neural policy. To capture the correlated non-robust features we must aggregate the information on high-sensitivis directions from a collec- tion of states visited while utilizing the policy æ in a given MDP. We thus define a single direction which captures the aggregate non-robust feature information from multiple states as follows:
Definition 3.2. Given a set of n states S = {s: the principal high- sensitvist direction is the vector Gs given by
G3
arg max || n
Σ (u, Vs, [ss, =,)) i=1
Proposition 3.1. Given set of n states S {8:) define the matrix C(S) by
(SS) Σ v. [ss, 8 ,,),, V., J(s, (((1,,88)las 8g)1,,
Then Gg the eigenvector corresponding to the largest eigenvalue of L(S).
Proof. Observe that by linearity of the inner product
a v (,,, 1
n
Tv
(s), (s, (g)s,
71
n1
n
5 (S,, 8g)l, Ve, J( sy)l
n
L(S)v.
Thus Gs = arg max (1ER<]]]v]l==) v. L(S).. Therefore, by the variational characterization of eigenvalues, Gs is the eigenvector corresponding to the largest eigenvalue of L(S).
Intuitively, Gs the direction that has the highest average correlation with the gradients the states in S. Also note that Gs has the same dimensions each state s, and so can easily be rendered in the same format as the states to visualize non-robust features. In Proposition 3.1 we show that Gs can be computed by solving eigenvalue problem. Proposition B.I the basis for Algorithm which computes Gg by first calculating L(S) by summing over states, and then outputs the maximum eigenvector.

---

Under review as a conference paper at ICLR 2023
Algorithm 1: RADEN: Robustness Analysis via Dominant Figendirections in Neural Manifold
Input: State-action value function Q(s, a), actions a E A, states E S, policy a(,, a) Output: Principal high-sensitivity direction (ii,))
for 8 = 80 to 8r do
T(s 1 a==ar max, (a)
a (sg.
(oftmax(Q(s,..1. as) J(s, 8g) = Laen T(s a) ((g
L + V [s, 88) --= V [s, >>,
end for Return: Eigenvector G corresponding to largest eigenvalue of C
Next we show how RADEN can be used to measure the effects of environment changes on the correlated non-robust features both visually and quantitatively. Let S' the set of states encountered when utilizing policy in an MDP M, and let S' be the set of states encountered when utilizing where the environment in M modified by applying some change each state s. In this setting, comparing the visualization of Gs and Gs can give qualitative picture of how the environmental change affects the learned correlated non-robust features. In order to give more quantitative metric for this change we define
Definition 3.3. For two sets of states S and S", the feature correlation quotient is given by
(SS)(S)
ASS S) 9 C(S)9s
Next show that the feature correlation quotient is bounded between zero and one. Proposition 3.2, For any two sets of states S and S
0 S MS', 5) < 1.
Proof. By Proposition 3.1
(LSS(S & max &' T (TS)(S) = 93 L(S)Ss ||v||==1
Thus the numerator of A(S",S) S) is always less than or equal to the denominator i.e. A(S", < 1. Furthermore, L(S) is positive semidefinite, it is sum of rank one projection matrices, and so S(S 2 0.
Therefore, the feature correlation quotient A(S", S) is a number between zero and one which intu- itively measures how correlated the non-robust features from S' are those from S. When mea- suring how an environmental change affects non-robust features, it is also important to take the stochastic nature of the MDP into account. In particular, the non-robust features observed with two different executions of the same policy may differ slightly due to the inherent randomness of the MDP. To account for this, we first collect baseline set of states with no modification Sp. We then collect set of states S₁ with no modification, and S with modification. By comparing ASS So) A (Si, 5o) we can see how much of the decrease in average correlation is caused by the stochastic nature of the MDP, and how much of the decrease is caused by the environmental change.
4 EXPERIMENTAL RESULTS
The deep reinforcement learning policies evaluated in our experiments are trained with the Dou- ble Deep Q-Network algorithm Hasselt et al. (2016) with the architecture proposed by Wang et al. (0116), and State-Adversand Schaul et al. (2016). The set of states S is collected over 10 episodes. The adversarial perturbation hyperparameters are: for the Carlini&Wagner formulation is 10, learning rate is 0.01, initial con- stant 10, for the elastic-net regularization formulation 3 is 0.0001, learning rate is 0.1, maximum iteration is 300, for Nesterov Momentum is 0.001, and decay factor is 0.1. These hyperparameters
Double Deep Q-Network (see Section 2.3) with experience replay

---

Under review as a conference paper at ICLR 2023
012
3i5
R0S 1317
Figure 1: RADEN results of untransformed states and states under adversarial perturbations com- puted via Carlini& Carneri&Wagner Nesterov Momentum, and elastic-net regularization for Pong. Columnl: Untransformed. Column2: C&W. Column3: Nesterov Momentum, Column4: Elastic-Net
5 55525
RADEN results of untransformed states and states under adversarial perturbations com- puted via Carlini& Wagner, Nesterov Momentum, and elastic-net regularization for BankHeist. Col- umnl: Untransformed. Column2: C&W. Column3: Nesterov Momentum. Column4: Elastic-Net
Figure 2:
Table 1: The feature correlation quotient A(S", S) in BankHeist, Freeway, RoadRunner, and Pong for the adversarial transformations: Carlini& Wagner, Wagner,&Wagner, Nesterov Momentum, DeepFool, Elastic-Net.
MDP
Untransformed
Carlini& Wagner
Nesterov
DeepFool
Net
Freeway
0,9917:10.0021
0.9499:10.02056
0.6869 0.786810.02162
0.72590=10.0922
BankHeist
0.8360#0.001 16
0.2837=:0.02316
0.3407:0.02412
0.1748=0.0442)
0.30917:: 0.0521
RoadRunner
0.7652:10.033 0338
0.1621::.02199
0.382610.03118
0.3353=0.03127
0.52506:: 0.0782
Pong
0.49344000391
0.0408-0000055
0.3444+0.01981
0.3277=0.02871
0.10529+ 0.0629
are consistent with prior work achieve the most effective adversarial perturbations (i.e. perturba-
tions causing the largest decrease on the discounted expected cumulative rewards obtained by the policy).
4.1 IMAGING NON-ROBUST FEATURE SHIFTS UNDER ADVERSARIAL PERTURBATIONS
In this section we investigate the effects of adversarial attacks on the learnt correlated non-robust fea- tures. Figure the RADEN results for the untransformed states and the adversarially attacked states in the Pong and BankHeist environments. In particular, these perturbations are computed via the Nesterov momentum, Carlini& Barlini&Wagner, and elastic-net regularization formulations (see Section 2.2 Figure demonstrates that different adversarial formulations surface different sets of corre- lated non-robust features. Depending the perturbation type, the correlated non-robust features can change quite noticeably. In fact, while the Carlini& Wagner formulation leaves a distinct signature on the vulnerable representation pattern, the non-robust features under Nesterov momentum appear most similar to those of the untransformed states. Our imaging technique can help to understand the vulnerabilities of deep reinforcement learning policies by allowing us visualize precisely how non-robust features change with different perturbations.
Table shows the feature correlation quotient A(S', results where S consists of untransformed states and S' consists of states modified by the Nesterov Momentum, Carnenr&Wagner, elastic-net regularization and DeepFool formulations respectively. Note that in all games the setting where S' consists of a set of untransformed states from an independent execution has the highest feature correlation quotient A(S', S). Therefore the additional decrease of A(S', S) when S' is modified by adversarial perturbations can be attributed to changes in non-robust features caused by the per- turbations. Observe also that the qualitative similarity between the visualizations in Figure 2 of the different transformed states is matched by their ranking under A(S', S) i.e. sorting from largest to smallest correlation quotient for BankHeist yields Nesterov momentum, elastic-net, and then Car- & Wagner. The fact that the feature correlation quotient has distinct results for untransformed states and for states under all the types of adversarial formulations indicates that RADEN can facil- itate detecting different types of adversarial perturbations.

---

Under review as a conference paper at ICLR 2023
Freeway Figure 3: RADEN results of the state-of-the afrtt adversarially trained deep reinforcement learning policies for BankHeist, Pong, Freeway and RoadRunner.
RoadRunner
BankHeist
Pong
164
750
18.0
2013
2003
1058
2002
250
25
16.5 1766
16.
1750
2010
204
303
080
120
1280
1955
2038
Figure 4: Standardized gradients | so [s, 12) for vanilla trained and state-of-the-an adversari- ally trained deep reinforcement learning policies.
4.2 EXISTENCE OF CORRELATED NON-ROBUST FEATURES IN ADVERSARIAL TRAINING
In this section we investigate the effects of adversarial training on the correlated non-robust features. In particular, the SA-DDQN algorithm adds the regularizer R.
R(00) Σ
max Q0(3,a) - Q0(c (s))
xED,(8) (a)
during training in the temporal difference loss. Figure 3 shows the RADEN results for the state- of-the-art adversarially trained deep reinforcement learning policies. The non-robust features of the adversarially trained deep reinforcement learning policies are much more tightly concentrated on small number of coordinates in the state observations, and these areas concentration have moved significantly from where they were under vanilla training. Thus, the visualization allows us to see that correlated, non-robust features persist in adversarially trained policies, albeit in different locations with sparser patterns than vanilla trained deep reinforcement learning policies.
In Figure 6 we plot the Fourier transform of Gs where S is collected from a vanilla trained and an adversarially trained policy in RoadRunner, BankHeist, Pong and Freeway. The Fourier transform reveals clear differences in the spatial frequencies occupied by 9s under vanilla and adversarial training. There is a consistent trend that the larger entries of the Fourier transform are more evenly and smoothly spread out for the adversarially trained policies. Thus, adversarial training leaves a consistent signature on the non-robust features detectable via the Fourier transform of Gs. There is also change in orientation: if the larger entries of the Fourier transform for the vanilla trained policy are more spread out along one axis, the adversarially trained Fourier transform is more spread along the other.
To complete our analysis of adversarial training we further include results on how non-robust fea- J(s;, 1g in each state 8; S for both adversarially trained and vanilla trained policies in RoadRunner, Pong, and Freeway. The results are plotted in Figure 4 In both RoadRunner and Freeway, the adversarially trained policy has relatively small gradients except for a few dramatic spikes corresponding to much higher sensitivity. This is in contrast to the vanilla trained policy in these games which tends to have a much smoother distribution which remains closer to the mean. These results indicate that adversarial training introduces higher jumps in sensitivity over states (i.e. extreme sensitivity for new set of states) when compared to vanilla training.
tures vary across time. To do so we record the form of the gradient || V v.,,
4.3 THE EFFECTS OF DISTRIBUTIONAL SHIFT ON THE NON-ROBUS, FEATURES LEARNT
To evaluate the effects distributional shift on the learnt policy we go over several environment modifications with RADEN. These transformations include blurring, perspective transform, rotation,

---

Under review as a conference paper at ICLR 2023
3
0.23
33
0.80
0003
-0.03
22075
22
Untransformed
Rotated
Blurred
C0.1
0000
0003
0.89
0095
000
33
-0.93
-6.05
-C333
54035RX 8.89 9. 828888888
Compression Artifacts
Perspective Transform
Brightness and Contrast
Figure 5: RADEN results of untransformed states and states under natural transformations with rotation, perspective transformation, blurring, compression artifacts, and brightness and contrast for Pong.
Table 2: The feature correlation quotient A(S", 8) in BankHeist, Freeway, RoadRunner, and Pong for the natural transformations: brightness and contrast, compression artifacts, rotation modification, perspective transform, blurred observations.
Distributional Shift
Freeway
BankHeist
RoadRunner
Pong
Untransformed States
0.9917=0.0023
0.836010.0116
0.7652=10.038
0,4934:100391
Brightness and Contrast
0.86756=0.0271
0.3095=0.0429
0.4369-0.0334
0.1678+0.0427
Compression Artifacts
0.90564=0.237
0.3881410.022
0.24358-00.004
0.49341=0.0191
Rotation Modification
0.0081
0.2951+:0.0062
0.3350+10.0550
0. 0.13648-00.0032
Perspective Transform
0.3010=0.0281
0.1723+000311
0.3308+400227
0,4278
Blurred Observations
0.2657=0.0148
0.0954+0.0127
0.2496+0.0012
0.0847+0.0083
brightness, contrast, and compression artifacts To keep these transformations small we bound by the LPIPS metric proposed by Zhang et al. (2018). Figure 5 shows the visualizations of Gs for states collected under the six environment modifications mentioned above. For the untransformed setting the visualization of Gs clearly emphasizes the center of the region where the agent's 's paddle moves up and down to hit the ball. The components of Gs take larger positive values at the center of this region and transition to negative values along the boundary. A similar emphasis can be found for the case of compression artifacts, but with the signs reversed (i.e. the center of the region negative and the boundary is positive). The other transformations exhibit larger changes in the regions emphasized the visualization with perspective transform, blurring, rotation, and brightness and contrast causing the emphasized region move different locations.
Table contains the values of A(S', SS) where S is collected from an untransformed run and S' is collected from each of the six different transformations. In every game the largest value of A(S',S) occurs when S' comes from an independent untransformed run, indicating that the additional de- crease observed for S' from transformed runs caused by the respective environmental transfor- mations. It is notable that in Pong the second highest value for A(S', occurs for 5" collected with compression artifacts, as this corresponds precisely to the qualitative similarity between the regions emphasized in the visualization of Gs for untransformed and compression artifacts. Hence, the results for A(5", S)) help us to quantitatively understand the effects of the environmental trans- formations, while agrecing well with the qualitative results of the RADEN visualizations. While the RADEN visualizations give us semantically meaningful information on the non-robust features learnt by the deep neural policy, also provides detailed understanding of how these non-robust features change under environmental modifications.
In particular, blurring refers to median blur, brightness and contrast to linear transformation, and compres- sion artifacts to diminution high frequency components due JPEG conversion.
8

---

Under review as a conference paper at ICLR 2023
RoadRunner
BankHeist
Pong
Freeway
RoadRunner
BankHeist
Pong
Freeway
Figure 6: Fourier spectrum the RADEN results of the state-of-the-art adversarially trained and vanilla trained deep neural policies. Rowl: Adversarially trained. Row2: Vanilla trained.
The fact that RADEN can provide fine- ane-grained vulnerability analysis of deep reinforcement learn- ing policies under adversarial attacks, with distributional shift and with adversarial training can help with diagnosis of deep reinforcement learning policy vulnerabilities in the development phase. Conducting ablation studies with RADEN in reinforcement learning algorithm design can prevent building policies with embedded non-robust features, and our algorithm can be utilized to visualize and identify the effects of several design choices (e.g. algorithm, neural network architecture) on the non-robust features learnt by the policy from the MDP. In particular, given a visualization of the vulnerability pattern for trained policy, one can try modify the training environment in way that will make the policy invariant to the non-robust features revealed by the visualization. Such modification could include changing the state representation in a way that does not change the se- mantics of the MDP, but does change the non-robust features in question. Furthermore, the effect of modifications training algorithms can also be directly visualized, as exemplified by our results for adversarial training. Thus our method gives straightforward way to diagnose debug any proposed methods in terms their effects on high-sensitivity directions.
CONCLUSION
In our paper we aim to seek answers for the following questions: (i) How do non-robust features learned by deep reinforcement learning policies correlate temporally and spatially? (ii) Do the cor- related non-robust features remold under adversarial attacks? (iii) Does adversarial training result in policies that still end up learning non-robust features? (iv) How does distributional shift affect the learnt correlated non-robust features? To be able to answer these questions we analyze high- sensitivity directions in the deep neural policy landscape and we propose novel tracing technique to detect correlated non-robust features learned by deep reinforcement learning policies. We show that deep reinforcement learning policies do end up learning correlated non-robust features, and that adversarial attacks lead to surfacing new set of non-robust features or highlighting the exist- ing ones. Most importantly, we show that the state-of-the adversarial training techniques also end up learning temporally and spatially correlated non-robust features. Finally, we show that dis- tributional shifts introduce different sets of correlated non-robust features compared to adversarial attacks. Hence, our analysis not only allows to effectively visualize correlated non-robust fea- tures, but also allows for precise understanding of changes the learnt non-robust features caused by different training algorithms and different methods for altering states. Thus, believe that our analysis can be useful both diagnosing the vulnerabilities of deep neural policies, and in designing algorithms to improve robustness.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Lower bounds on adversarial robustness from optimal transport. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 7496-7508, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39 39-57, 2017.
Lin Chen, Yifei Min, Mingrui Zhang, and Amin Karbasi. More data can expand the generalization gap between adversarially robust and standard models. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1670-16800 PMLR, 2020.
Pin-Yr Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. EAD: elastic-net attacks to deep neural networks via adversarial examples. In Sheila A. Mcllraith and Kilian Q Weinberger (eds.), Proceedings of the Thirty-Secome AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 10-17, AAAI Press, 2018.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 189-9193, 2018
Korkmaz Ezgi. Nesterov momentum adversarial perturbations in the deep reinforcement learning domain. International Conference on Machine Learning. ICML 2020, Inductive Biases, Invari- ances and Generalization in Reinforcement Learning Workshop. 2020.
Ian Goodfellow, Jonathan Shelens, and Christian Szegedy. Explaning and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Association for the Advancement of Artificial Intelligence (AAAI), 2016.
Zhang Huan, Chen Hongge, Xiao Chaowei, Bo Li, Mingyan Boning, Duane Liu, and ChoJui Hsiesh. Robust deep reinforcement learning against adversarial perturbations on state observatons. Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Sandy Huang, Nicholas Papernot, Yan Goodfellow, Ian Duan, and Pieter Abbeel. Adversar- ial attacks on neural network policies. Workshop Track of the 5th International Conference on Learning Representations, 2017.
Ezgi Korkmaz. Adversarial training blocks generalization neural policies. International Conference Learning Representation (ICLR) Robust and Reliable Machine Learning in the Real World Workshop, 2021.
Ezgi Korkmaz. Deep reinforcement learning policies learn shared adversarial features across mdps. AAAI Conference on Artificial Intelligence, 2022.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. International Con- ference on Learning Representations, 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 1rXiv:1607.02533 2016.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan. Wierstra. Continuous control with deep reinforcement learning. arX- ivpreprint / 2015.
10

---

Under review as a conference paper at ICLR 2023
Yen-Chen Lin, Hong Zhang- Wei, Yuan-Hong Liao, Meng-Li ing- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. Proceedings of the Twenty-Sinn International Joint Conference Artificial Intelligence, pp. 3756-3762, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 770..66883. 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, are Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518: 529-533, 2015.
Volodymyr Mnih, Adria Badia Puigdomenech, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray. Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928-1937, 2016.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2574-2582. IEEE Computer Society, 2016.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. International Conference on Learning Representations ICLR, 2017.
Tom Schaul, John Quan, Ioannis Antonogloua, and David Silver. Prioritized experience replay. International Conference on Learning Representations (ICLR), 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Toannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, atthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, van den George Driessche, Thore Graepel, and Demis Hassabis. Mastering the game go without human knowledge. Nature, 500:354-3555 2017.
Christian Szegedy, Wojciech Zaremha, Ilya Sutskever, Joan Bruna, Dimutru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Florian Tramèr, Jens Behrmann, Nicholas Carlini, Nicolas Papernot, and Jörn-Henrik Jacobsen. Fundamental tradeoffs between invariance and sensitivity to adversarial perturbations. In Pro- ceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9561-9571. PMLR, 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanetot, and Nando. De Fre- itas. Dueling network architectures for deep reinforcement learning. Internation Conference on Machine Learning ICML, pp. 1995-2003, 2016.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver. Wang. The unreasonable effectiveness of deep features perceptual metric. Conference on Computer Vision and Pattern Recognition (CVPR), 2018


Reviews:
[{"review_id": "t4AkEWN2We", "reviewer": "Reviewer_B6vV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method to analyze the vulnerabilities of deep reinforcement learning policies. Experiments are conducted to illustrate the non-robust features, and show how adversarial attack techniques and robust training affect these features. ", "summary_of_the_review": "The authors propose a new method to study the non-robustness of the DRL policy. I suggest the authors add more explanations to their method and results. Instead of directly giving the definition of the notions $G_S$ and $\\Lambda$, maybe it's better to address why these notions are important for understanding the DRL policy, and how they can be used to guide robust training. ", "strength_and_weaknesses": "Some questions or suggestions about the paper: \n\n1. The authors should address more why Definition 3.1 is an important notion that we should care about. And the formulation of $J(s, s_g)$ also seems to be arbitrary. It doesn’t make sense to let one policy to be the softmax policy and the other be the argmax policy. I guess the author is trying to show that in this setting, the gradient of $J$ is equivalent to Definition 3.1, but the equation (after the sentence “Therefore by definition of the softmax policy we have”) is clearly problematic, missing the factor $T$ somewhere. Even after correcting the equation, the formulation of $J$ still doesn’t make sense, one can just let both policies be softmax and still get a similar relation. Furthermore, wouldn’t it be much simpler to directly define $v$ to be the gradient of some metric function $J$?\n\n2. The authors further define the notion $G_S$ given a set of states. However, the meaning of $G_S$ is also unclear. I would expect each state has its own sensitive direction, which may be of some interest for studying the instability of the policy under each state. But I don’t understand why the seemingly “averaged” vector $G_S$ still conveys the information that worth studying. \n\n3. Propositions 3.1 and 3.2 are straightforward facts, and could be stated more concisely. And the boundedness condition of $\\Lambda$ seems to be somehow unimportant. \n\n4. The authors should give a more formal definition on what is a non-robust feature. According to my understanding, it refers to the high-sensitivity direction $G_S$, though I don’t know why it should be called a feature. $G_S$ is clearly a vector in the raw state space, whose dimension could be prohibitively large for the downstream computation. \n\n5. The figures are hard to read and understand. While Figure 1 and Figure 2 are both RADEN results, in the text only Figure 2 is referred to. In Figure 1, the legend of the color is inconsistent across each subfigure. And the author should explain the meaning of these figures, does the highlighted areas imply the coordinates that are more vulnerable?\n\n6. The authors claim the non-robust features still exist in adversarial training. They should given more explanations how they get this conclusion. The direction $G_S$ always exist, because any symmetric matrix always has the eigenvector with largest eigenvalue. Maybe it’s better to explain which shapes of $G_S$ indicate there exist non-robust features and which do not?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qw2xigGBhZI", "reviewer": "Reviewer_iSgv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method to determine the correlated high-sensitivity directions  in the deep neural policy manifold across space and time in the context of reinforcement learning. In this direction, they first define the concept of a high-sensitivity direction for the Q-function at a given state, and show that it is equal to the gradient of some cross-entropy loss at that state. The high-sensitivity direction will contain non-robust features, along which the Q-function changes the most. To capture the correlated non-robust features, the authors then propose to aggregate the information on high-sensitivity directions from a collection of states visited under some given policy. In particular, they suggest to use the direction with the highest average correlation with the gradients of all visited states, and show how this direction can be computed with a spectral algorithm. In addition, they introduce a feature correlation quotient, which captures how correlated the non-robust features from one set of states S' are to a different set of states S.\n\nBased on these contributions, the authors then empirically investigate various questions. First, the effect of adversarial attacks on the learned correlated non-robust features, where they show that these can change quite significantly under adversarial perturbations. Second, the effects of adversarial training on the correlated non-robust features, where they show that they are more tightly concentrated in a small number of coordinates but in different locations that under vanilla training. Third, the effects of distributional shift, where they show that different transformations will change the location of the regions of correlated non-robust features.", "summary_of_the_review": "Overall, I believe that this is a work with interesting ideas and experiments. However, I am not convinced that is ready for publication in its current form; I think that was is mainly missing is some empirical \"proof\" that the proposed framework can really improve RL via a better fine-grained vulnerability analysis, e.g., through concrete examples. The authors claim repeatedly that that this is the case, but, given the paper is not so theoretical, I would have expected a more through empirical evaluation where the authors clearly demonstrate the benefit of the proposed methodology.", "strength_and_weaknesses": "Strengths\n- The problem of determining the non-robust features in the neural policy manifold is very important with several potential applications.\n- The definition of high-sensitivity directions is well-motivated and intuitive. The connection to the softmax cross entropy is also interesting.\n- The authors complement the average high-sensitivity direction with the feature correlation quotient, and investigate how the two notions are related in their empirical analysis.\n- The authors provide an empirical study of how correlated non-robust features also arise under adversarial attacks, adversarial training or distribution shift, and how they differ from those in vanilla training.\n\nWeaknesses\n- I am not clear how the paper shows that the direction with the highest average correlation can be so useful in practice. I am convinced about the usefulness of high-sensitivity directions at a given state. But I feel that the authors have not explained with convincing arguments the necessity and usefulness of the average high-sensitivity direction and the corresponding feature correlation quotient. For instance, assume we do RL. At a given time step, we would naturally be interested in the non-robust features and try to mitigate their impact; note that these may vary from one timestep to another. The average quantity certainly deserves merit, but I believe the authors should have motivated it further.\n- I feel that a large part of the experimental analysis is about investigating how the proposed qualitative and quantitative metrics perform under vanilla training, as well as under adversarial attacks, adversarial training, or distributional shift. The findings are for sure interesting. But the question that was left unanswered: how can we leverage all these findings to improve RL? The authors devote one paragraph at the end of Section 4 to discuss how their framework can in principle provide fine-grained vulnerability analysis, but it would have made more sense in my opinion to show in practice rather than in theory how the introduced concepts can improve RL through improved vulnerability analysis (e.g., in a given MDP). I am concerned that much of this discussion has a very theoretical character.\n- Some claims are not fully explained. As an example, the authors claim that there is a change in orientation in the Fourier transform: if the larger entries of the Fourier transform for the vanilla trained policy are more spread out along one axis, the adversarially trained Fourier transform is more spread along the other. This is true in the figure, but I am not sure that an adequate explanation was given.\n- I did not find the contributions to be particularly novel - I liked the definition of high-sensitivity directions but I have some concerns with the treatment for average high-sensitivity directions. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Cj-YE31gOSF", "reviewer": "Reviewer_dkrX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work analyzes high-sensitivity directions in the deep neural policy landscape and proposes a novel tracing technique to detect correlated non-robust features learned by deep reinforcement learning policies. It firstly defines $v$ as a high-sensitivity direction from state $s$ and $\\hat s$ and then approximates $v$ by using the softmax cross entropy loss between the softmax policy and the argmax policy. It also proposes the RADEN algorithm to measure the effects of environmental changes on the correlated non-robust features visually and quantitatively by the Eigenvector G corresponding to the largest eigenvalue of L for the given state set S. Experiments on the Arcade Learning Environment are done to evaluate the idea.\n", "summary_of_the_review": "The main body of the paper is far away from the title and claimed idea.", "strength_and_weaknesses": "Strengths And Weaknesses: \nPros: This paper studies a significant problem to understand the representations learned by deep neural network policies. The method proposed, as far as I know, is the first one to introduce a tracing technique to detect correlated non-robust features. The overall idea is interesting and innovative. \nCons: \n\t(1) I think this paper may need lots of improvement in its writing. It's informal to just use Eqn to refer to the equation, such as Eqn(5) rather than Equation (5) or Eq. (5). There are a lot of sentences in the text that confuse me.\n\t(2) The title of the paper mentions the use of spectral analysis, but the methods section does not clearly explain how to use spectral analysis.\n\t(3) The experimental section does not have an introduction to the experimental setup.\nQuestions:\n\t(4) The title of the paper mentions policy manifold, but there is no definition of policy manifold in the context. Please give a clear explanation of the policy manifold.\n\t(5) Please give a clear Explanation of the relationship between the learned representations of neural network policies and policy manifold.\n\t(6) What is the distributional shift in section 3 question 2?\n\t(7 )Give an explanation of ε and ω in Definition 3.1.\n\t(8) Give a clear explanation of the experimental graph, including the meaning of the coordinate axes, etc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9bhJxDVfGmF", "reviewer": "Reviewer_gVJV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors have considered the problem of understanding the sensitivity of deep reinforcement learning representation landscape. The authors have proposed the tracing technique to detect correlated non-robust features. The authors have defined \"high sensitive\"  direction of Q function in Definition 3.1. This helps to define the high sensitivity direction in Definition 3.2, and eventually connected it to the eigenvector of the highest average correlation with the gradients of the states. The authors have provided experimental results as well in the paper. ", "summary_of_the_review": "The paper is difficult to follow. The ideas is interesting but the paper requires a lot of work before publication.  ", "strength_and_weaknesses": "*Strengths*\n- The authors are considering an interesting problem of understanding of the representations learned by DNN policies in RL. This is of sufficient interest to the community. \n\n*Weakness*\n\n- What is non-robust direction? It is used on page 2 without clear definition/explanation. \n- The motivation is not clear to me from the introduction. The authors started talking about the adversarial training issues in the standard supervised learning and then directly mentioned the challenges in deep reinforcement learning. A little more context and motivation is needed.\n- There is no sufficient-related work to contrast the contributions with respect to the literature. \n- Sec 2.2. and 2.3 seems redundant. \n- In definition 3.1, is v and w are the same? \n- The constraint in (7) is the equality constraint, can you comment if the contraint is convex, or if unique solution to problem in (7) exists? \n- What is T in the definition of the policy after (7) ?\n- \"Eqn 7 can be approximated by using the softmax cross entropy loss.\"--why? \n- How to calculate the  gradient of J with respect to state s? Would that require access to the true transition model P ?\n- What is the meaning of non-robust and robust features with respec to the definition on 3.3? \n- I am not sure what Figure 3 is describing? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "5JuJlhBqlmP", "reviewer": "Reviewer_eHD9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies how to visualize high sensitivity directions in reinforcement learning. Given a set of states, authors proposed a RADEN algorithm to find the leading eigenvector of certain covariance matrix relevant to the perturbation analysis. The obtained leading eigenvector indicates the directions of vulnerability. Plenty of experimental results are presented to support the effectiveness of the proposed algorithm as well as extend to broader context of interest in reinforcement learning, e.g., distributional shift.", "summary_of_the_review": "I think the manuscript in its current form needs substantial revision to meet the acceptance standard. My major concern is the unclear implications of the proposed method.", "strength_and_weaknesses": "================= Strength =================\n\nExperimental study is comprehensive and provides many graphical illustrations.\n\nThe focus on the robustness of policies in reinforcement learning is of importance to many applications.\n\n================= Weakness =================\n\nAlthough RADEN helps to visualize high sensitivity directions, it is unclear how to incorporate such information for better design policy learning algorithms. In other word, practical implication of RADEN seems to be limited.\n\nIf I were not missing anything, spatial and temporal patterns of vulnerabilities in learned policies are very vaguely discussed, especially the temporal part. I don't think collecting states along a trajectory is an equivalence to temporal dependencies, as the vulnerabilities found by RADEN do not have temporal or even spatial information. The contributions are a bit overclaimed.\n\nThere lacks a coherent motivation of why studying the proposed four questions. The transition from the second paragraph to the third paragraph in Introduction is rather abrupt.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject