Daylight: Assessing Generalization Skills of Deep Reinforcement Learning Agents
Authors: Ezgi Korkmaz
Venue:  (2021)
Keywords: deep reinforcement learning
generalization

Abstract: Deep reinforcement learning algorithms have recently achieved significant success in learning high-performing policies from purely visual observations. The ability to perform end-to-end learning from raw high dimensional input alone has led to deep reinforcement learning algorithms being deployed in a variety of fields. Thus, understanding and improving the ability of deep reinforcement learning agents to generalize to unseen data distributions is of critical importance. Much recent work has focused on assessing the generalization of deep reinforcement learning agents by introducing specifically crafted adversarial perturbations to their inputs. In this paper, we propose another approach that we call daylight: a framework to assess the generalization skills of trained deep reinforcement learning agents. Rather than focusing on worst-case analysis of distribution shift, our approach is based on black-box perturbations that correspond to semantically meaningful changes to the environment or the agent's visual observation system ranging from brightness to compression artifacts. We demonstrate that even the smallest changes in the environment cause the performance of the agents to degrade significantly in various games from the Atari environment despite having orders of magnitude lower perceptual similarity distance compared to state-of-the-art adversarial attacks. We show that our framework captures a diverse set of bands in the Fourier spectrum, giving a better overall understanding of the agent's generalization capabilities. We believe our work can be crucial towards building resilient and generalizable deep reinforcement learning agents.

Under review as a conference paper at ICLR 2021
DAYLIGHT: ASSESSING GENERALIZATION SKILLS OF DEEP REINFORCEMENT LEARNING AGENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep reinforcement learning algorithms have recently achieved significant success in learning high-pe perfforrnnn policies from purely visual observations. The ability to perform end-to-end learning from raw high dimensional input alone has led to deep reinforcement learning algorithms being deployed in a variety of fields. Thus, understanding and improving the ability of deep reinforcement learning agents to generalize to unseen data distributions is of critical importance. Much recent work has focused on assessing the generalization of deep reinforcement learning agents by introducing specifically crafted adversarial perturbations to their inputs. In this paper, propose another approach that call daylight: framework to assess the generalization skills of trained deep reinforcement learning agents. Rather than focusing worst-case analysis of distribution shift, approach is based on black-box perturbations that correspond to semantically meaningful changes the environment the agent's visual observation system ranging from brightness to compression artifacts. We demonstrate that even the smallest changes in the environment cause the performance of the agents to degrade significantly in various games from the Atari environment despite having orders of magnitude lower perceptual similarity distance compared to state-of-the-ar adversarial attacks. We show that our framework captures diverse set of bands in the Fourier spectrum, giving a better overall understanding of the agent's 's generalization capabilities. We believe our work can be crucial towards building resilient and generalizable deep reinforcement learning agents.
INTRODUCTION
Following the initial work of Mnih et al. (2015), the use of DNNs as function approximators in reinforcement learning has led to a dramatic increase in the capabilities of RL agents Schulman et al. Lillicrpp et al. (2013). In particular, these developments allow for the direct learning of strong policies from raw, high- high-dimensional inputs such visual observations. With the successes of these new methods come new challenges regarding the robustness and generalization capabilities of deep RL agents.
One line of research has focused on the high sensitivity of deep neural networks to imperceptible, adversarial perturbations to visual inputs, first in the setting of image classification Szegedy al. (2014); Goodfellow et al. (2015) and more recently for deep reinforcement learning Huang et al. (2017 Kos & Song (2017) Since one of the main reasons for the success and popularity of deep RL is the ability to learn directly from visual observations alone, this non-robustness to small adversarial perturbations is serious concern Chokshi (2020): Vlasic Boudette (2016); Kunkle (2018). However, existing adversarial formulations for deep reinforcement learning require high computational effort to produce the perturbations, knowledge the network used train the agent, knowledge of the environment, real-time access to and manipulation of the agent's 's state observations.
In this paper, we propose a more realistic scenario where we do not have access to any the above, and the adversary essentially consists of realistic changes in the natural environment in the agent's observation system. For instance, if our deep reinforcement learning agent is operating a self-driving could plausibly expect changes in daylight levels, shifts in angle due to terrain, fog the camera lens, or compression artifacts from the camera processor. We believe that our proposed framework is semantically more meaningful than arbitrary f--norm bounded pixel perturbations.

---

Under review as a conference paper at ICLR 2021
Prior work on image classification Dodge & Karam (0066) showed that image quality distortions can reduce the accuracy of DNN classfiers. Moreover, recent work by Ford et al. (2019) showed that while adversarial training for image classifiers reduced their vulnerability towards perturbations corresponding to high frequency in the Fourier domain, it actually made the models more vulnerable to low frequency perturbations including fog and contrast changes. Therefore, it is important to investigate model robustness and generalization throughout different bands in the frequency domain. We believe that being able to accurately assess the generalization capabilities of deep reinforcement learning agents is an initial step towards building robust and reliable agents. For these reasons, in this work we investigate the robustmess of trained deep reinforcement learning agents and make the following contributions:
We propose realistic threat model called daylight and a generalization framework for deep reinforcement learning agents that aims to assess the robustness of the agents to basic environmental and observational changes.
We rum multiple experiments in the Atari environment various games to demonstrate the degradation in performance of deep reinforcement learning agents.
We compare our threat model with the sate-of-theeart adversarial method based on l, no-norm changes, and we show that our daylight framework results in competitive, and almost always larger impact, with lower pereeptual similarity distance.
We evaluate the daylight framework in the time domain and show that several works based on the timing perspective of adversarial formulations can be revisited within our daylight framework.
Finally, we investigate the frequency domain of our framework and state-of-the-art targeted attacks. We show that our framework captures different bands of the frequency spectrum, thus yielding better estimate of the model robustness.
2 RELATED WORK
2.1 PRELEMINARIES
In this paper we consider Markov Decision Processes (MDPs) given by a tuple (S, P.r, v, su). The reinforcement learning agent interacts with the MDP by observing states S, and then taking actions a E A. The probability of transitioning to state s' when the agent takes action a in state is determined by the transition probability P: S A S R. The reward received by the agent when taking action a in state s is given by the reward function r S x A + R. The goal of the agent is learn a policy wy : S x A + which takes an action in state s that maximizes the cumulative discounted reward D ("((st,rr) Here so is the initial state of the agent, and 7 is the discount factor. For deep Q networks (DQN) the optimal policy is determined by learning the state-action value function Q(s, a). For a state s we use F(s) to denote the 2D discrete Fourier transform.
2.2 CRAFTING ADVERSARIAL PERTURBATIONS
Szegedy et al. 2014)) proposed to minimize the distance between the original image and adversarially produced image create adversarial perturbations. The authors used box-con nox-constrained L-BFGS to solve this optimization problem. [Goodfellow et al. (2015) introduced the fast gradient method (FGM)
Tadv x + 6 V. v)l))
(1)
for crafting adversarial examples in image classification by taking the gradient of the cost function J(z, y) used to train the neural network in the direction of the input, where is the input, is the output label, and [xx y) is the cost function for image classification. Carlini & Wagner (0117) introduced targeted attacks in the image classification domain based on distance minimization between the adversarial image and the original image while targeting a particular label. In the deep reinforcement learning domain the Carlini & Wagner (207)) formulation is
2

---

Under review as a conference paper at ICLR 2021
min Bab
||sdy s|1p
subject to
a" a'(s) # a' (Sadr);
where s the unperturbed input, Sadv is the adversarially perturbed input, a" (s) is the action taken in the unperturbed state, and a" ("(aadv) is the action taken in the adversarial state. This formulation attempts minimize the distance to the original state, constrained to states leading to sub-optimal actions as determined by the Q-network. In contrast to adversarial attacks, in our proposed threat model we will not need any information on the cost function used to train the network, Q-network of the trained agent or the visited states themselves.
2.3 ADVERSARIAL APPROACH IN DEEP REINFORCEMENT LEARNING
The first adversarial attacks on deep reinforcement learning introduced by Huang et al. (2017) and Kos & Song (2017 7) adapted FGSM from image classification the deep RL setting. Subsequently, Mandlekar et al. (2017 77) used FGSM perturbations for adversarial training of deep RL agents. Piito al. (2017); Gleave al. (2020) focused on modeling the interaction between adversary and the agent, while Lin et al. (01 20177: Sun et al. (2020) focused on strategically timing when (i.e. in which state) to attack an agent using perturbations computed with the Carlini Wagner (2017) adversarial formulation.
2.4 PERCEPTUAL SIMILARITY
Zhang et al. 2018) found that internal activations of networks trained for high-level tasks corre- spond to human perceptual judgements across different network architectures landola al. (2016), Krizhevsky et al. (2012). Simonyan & Zisserman (2015) without calibration. Furthermore, the authors propose method measure the perceptual distance between images with the Learned Perceptual Image Patch Similarity (LPIPS) metric. We compare the distance between adversarial states Sadv and the original states s with the LPIPS metric. We refer to the LPIPS metric as Psimilrrit throughout the paper. P Pssmilarity (s, Sadv) returns the distance between s and Sadv based on network activations. /Zhang et al. (2018) show that Psimilartt results in a reliable approximation of human perception.
2.5 IMPACT
We define the normalized impact of an adversary on the agent as,
I Scoremax Scoready
(2)
Scoremax Scoremin
Scoremax is the score the end of the episode achieved by the agent who takes the action that maximizes its Q(s, a) function in every state visited, and Scoremin is the score at the end of the episode achieved by the agent who takes the action that minimizes its Q(s, a) function in every state visited. Score is the score at the end of the episode achieved by the agent who takes the action that maximizes Q (Sadv; a) under the influence of the adversary in every state visited. We report the results in normalized scale, because we observed the agent can collect stochastic rewards even though it chooses the action that minimizes its Q8, a) function in every state visited.
DAYLIGHT: A GENERALIZATION TESTING FRAMEWORK
In our generalization framework we propose a baseline test the robustness of trained deep reinforcement learning agents with respect to several realistic failures of the agent's observations. This is in contrast to prior work that focused on the presence of a strong adversary with prior access to training details of the agent's 's neural network, real time access agent's perception system, and highly computationally demanding adversarial formulations used to compute simultancous perturbations. In model we consider the environment itself as adversary and examine several environmental

---

Under review as a conference paper at ICLR 2021
changes such as: changes in the brightness of the environment, blurring of the observation, rotation of the observation, perspective transformation, shifting, and compression artifacts. These changes from model can be easily linked to naturally occurring changes in the environment. For instance, for a self driving car a brightness change can be linked to the time of day, or the appearance of reflective objects shadows. Rotation, perspective transformation, and shifting be linked to driving on a road with varied terrain. Blurring can be linked to rainy day, foggy weather or a fogged up camera lens utilized by the agent. In the remainder of this section compare the impact values and perceptual similarities of the daylight framework with the state-of-the-ard targeted adversarial attack proposed by Carlini Wagner (2017).
3.1 BRIGHTNESS AND CONTRAST
The first component of our framework focusing on testing the trained agents in different brightness and contrast levels using linear brightness and contrast transformation,
Sddv (i.j) =(i,j) or 8,
where (i. is the in pixel of state and a and 3 are the linear brightness parameters. In Table we show the impacts and perceptual similarity distances with corresponding a, f values. In all of the games except BankHeist brightness and contrast change results in higher impact. The perceptual similarity distance brightness and contrast is lower in every game. Figure we show the corresponding states and Sady for all of the daylight framework.
Table 1: Impacts of Carlini & Wagner 2017 and brightness & contrast (B&C) with corresponding perceptual similarity P Psimilarity, and the aa, 3
Games
C&W Impact
B&C Impact
C&W P Pamilarity
B&C Pamilarity
[a,5]
BankHeist
0.986:1.0099
0.971 0.030
0.171
0.127
[1.2,40]
JamesBond
0.51610.231
0.806=0.007
0.326
0,015
[0.9,20]
Pong
0,995
0.996210.009
0.608
0.245
[1.7.40]
RiverRaid
0.9498+0.009
0.973+0.016
0.255
0.168
[2.4,-275]
TimePilot
0.774+0.1.9
0.906+0.239
0.279
0.070
[2.4,-260]
3.2
BLURRING
The second component in our daylight framework blurring. Median bluring is a nonlinear noise removal technique that replaces the original pixel value with the median pixel value of its neighbouring pixels. A kernel size k means that the median is computed over pixel. In Table 2 we show the impact values and perceptual similarity distance with corresponding kernel size. Only in BankHeist do we observe that the impacts and perceptual similarity distance are comparable. For the rest of the games impact is higher and perceptual similarity distance is lower for blurring.
neighborhood of the original
Table Impacts of Carlini & Wagner (017) and blurring with corresponding perceptual similarity Psimilarity, and kernel size.
Games
C&W w Impact
Blurring Impact
C&W Pimilarit
Blurring Psimilaity
Kernel Size
BankHeist
0.986=t0.009
0,983+00.000
0.171
0.168
JamesBond
0.516
0.634:10.200
0.326
0.048
Pong
0.995=10.014
1.0:10.000
0.245
0.098
RiverRaid
0.9498-10.030
0,96810.015
0.255
0.134
TimePilot
0.774+0.1199
0.805+0.150
0.279
0.124
4

---

Under review as a conference paper at ICLR 2021
3.3 ROTATION
The next component in our daylight framework is rotation. In Table we show impact values and perceptual similarity distance with corresponding rotation angle. In all of the games except Pong rotation results in higher impact and orders of magnitude lower perceptual similarity distance. In Pong the impact is comparable and the perceptual similarity distance is lower by factor of 2.
Table 3: Impacts of Carlini & Wagner 2017) and rotation with corresponding perceptual similarity P Psimilariy. and the rotation angle (RD denotes rotation degree).
Games
C&W Impact
Rotation Impact
C&W Pumilarity
Rotation P Paimilariy
RD
BankHeist
0.986100.00 .009
1.0.0.004
0.171
0.064
1,4
JamesBond
0.516+0.231
0.725=0.189
0.326
0.029
1.6
Pong
0.995+0.014
0,99::0.115
0.245
0.112
RiverRaid
0.0090
0.005
0.255
0.059
TimePilot
0.77440.159
0.910:10.158
0.279
0.068
3.4 SHIFTING
The next component our daylight framework is shifting. Shifting an image moves the elements of the image matrix along any dimension by any number of elements. In this subsection we will shift the inputs in the x or y direction with as few pixels shifted as possible. We use [4g [ to denote distance shifted, where t, is in the direction of a and is in the direction of y. In Table 4 we show the impact values and perceptual similarity distances for both (Callii & Wagner (2017) and shifting. For all of the games shifting yields higher impact and lower perceptual similarity distance.
Table 4: Impacts of Carlini & Wagner (017) and shifting with corresponding perceptual similarity Piimilarity and the shifting (..
Games
C&W Impact
Shifting Impact
C&W P iimilarity
Shifting P Psimilarity
14.4]
BankHeist
0.986= 009
0, 0.989=0.005
0.171
0.060
[1.1]
JamesBond
0.516:10.231
0.989-10.140
0.326
0.048
[0.1]
Pong
0.995=000114
1.0=0.00
0.608
0.217
[2.1]
RiverRaid
0.9498=00003
0.906
0.255
0.095
[1,2]
TimePilot
0.774-:0.199
0.851-00.19
0.279
0.120
[2,2]
3.5 COMPRESSION ARTIFACTS
In this section we look at jpeg compression artifacts caused by the discrete cosine transform (DCT) resulting in the loss of high frequency components (ringing and blocking). In Table 3 we show the impact values and perceptual similarities of Carlini & Wagner (2017) and compression artifacts (CA).
Table 5: Impacts of Carlini & Wagner 2017 and compression artifacts (CA) with corresponding
perceptual similarity P Psumilarity
Games
C&W Impact
CA Impact
C&W P. Paimilarity
CA Psimilarity
BankHeist
0.986-10.009
0.984+1.013
0.171
0.067
JamesBond
0.516+00231
1.0+0.128
0.326
0.035
Pong
0,99510.014
0.962-10.032
0.608
0.029
RiverRaid
0.9498+00.030
0.8218+0.051
0.255
0.057
TimePilot
0.774+-0.159
0.790++0.271
0.279
0.067
Only in Pong and Riverraid do we observe a lower impact than Carlini & Wagner 2017 while
the perceptual similarity distance was orders of magnitude smaller for compression artifacts. In BankHeist the impact is comparable, and the rest of the games compression artifacts result in higher impact and lower perceptual similarity distance compared to Carlini & Wagner (0017).
5

---

Under review as a conference paper at ICLR 2021
3.6 PERSPECTIVE TRANSFORMATION
The final component of our dayligh/ framework is perspective transformation. Given four points in the plane defining convex quadrangle, there is unique perspective transformation mapping the corners of the square to these four points (see Equation [ and Equation (6) We define the norm of perspective transformation as the maximum distance that one of the corners of the square moves under this mapping. In Table we show impact values and perceptual similarity distance with respect to perspective norms. For all the games we observe perspective transformation yields higher impact and lower perceptual similarity distance.
Table 6: Impacts of Carlini & Wagner 2017) and perspective transformation (PT) with corresponding perceptual similarity P Sumiilarity and the perspective norm.
Games
C&W Impact
PT Impact
C&W Pimillrity
P Piimilarit
Perspective Norm
BankHeist
0.009
1.0++0.003
0.171
0.022
JamesBond
0.516=0.231
0.978=000887
0.326
0.007
Pong
0.995=00014
0.99610.009
0,608
0.029
RiverRaid
0.9498+0.030
0.99+0.0066
0.255
0.046
TimePilot
0.774+0.119
0.852+0.198
0.279
0.029
FOURIER DOMAIN
Ford al. (2019) showed DNN models are robust to certain bands of perturbations in the frequency domain. Furthermore, they showed that adversarial training shifts the vulnerability from high frequency noise towards low frequency noise. Moreover, Yin et al. (2019) claim that framework that aims to measure robustness and generalization needs to firmly encapsulate different directions of the spectrum in the frequency domain. In this section we show that the daylight framework indeed captures broader set of directions in the frequency domain.
Vgri
US
Ug
Engerte
Spemoo
18"
10'
Quern TIngund
Seamen
10
32'
15'
16
25
3
Figure 1: Riverraid power spectrum change with various perturbations: Carlini & Wagner, compres- sion artifacts, brightness and contrast, perspective transformation, shifting, rotation.
In Figure we show the Fourier spectrum of the original state and the perturbed states Sadv from the daylight tramework and Carlini Wagner (2017). In Figure we show the power spectral density of the original state compared to several perturbations from proposed daylight framework and Carlini & Wagner (2017). Figure [ we observe that while Carlini & Wagner (2017) increases the higher frequencies, compression artifacts decrease the magnitude of the higher frequency band.

---

Under review as a conference paper at ICLR 2021
Figure 2: Rows: F(s) for BankHeist, F(s) Riverraid. Columns: original state, Carlini & Wagner, brightness and contrast, blurring, rotation, shifting, perspective tansformation, compression artifacts.
On the other hand, brightness and contrast decreases the magnitude of the low frequency band, and shifting increases the midband. Blurring decreases the midband and high frequencies together, and perspective transformation decreases the low frequecies and high frequencies while increasing the midband. We believe capturing the susceptibilities towards perturbations in different bands of the frequency domain a key step towards building robust agents.
5 TIMING PERSPECTIVE
In the previous sections we tested our agents generalization capabilities in modified environments with our initial thread model. This modification to the environment was applied every state that the agent visited for both Carlini & Wagner (2017) and our daylight framework. In this section we will examine the effect of both adversarial models when the perturbations are applied to only a small fraction of states. For this purpose we introduce the adversarial states Sadv in randomly sampled states where the observation Sadv is observed by the agent with probability p, and the original states is observed observed by the agent with probability where the agent observed Sadv instead of the original state s, and we use n, to denote the total number of states visited by the agent in the given episode. We use eadv to denote the fraction n / of adversarial perturbations per episode.
p. We use Ta to denote the number of states
Table 7: Impact comparison with the fraction of adversarial observation probability
RiverRaid
TimePilol
Banklleist
Pong
JamesBond
C&W Impact
0.359
0.148
0.249
0.077
0.021
Shifting Impact
0.513
0.374
0.326
0.114
0.165
Perspective fromatoonttion Impact
0.391
0.315
0.338
0.108
0.121
Blurring Impact
0.501
0.155
0.304
0.12
0,319
Brightness Contrast Impact
0.517
0.188
0.313
0.098
0.154
Rolation Impact
0.192
0.079
0.044
Compression Artifacts Impact
0.184
0.262
0.267
0.017
0.198
Sady observation probability p
0.1
0.02
0.02
0.06
0.08
In Table 7 we show the attack impacts of Carlini & Wagner (2017) and daylight framework with
corresponding adversarial observation probability averaged over 10 random episodes. See Appendix for more details. Even for low p values our proposed daylight framework obtains higher impact. Thus, to capture a broader view on the robustness of the agent, the prior work on the timing perspective by (Sun al.) (2020); Lin et al. (2017) based on worst-case distributional shift, can be revisited with our daylight framework.
EXPERIMENTS
In experiments trained agents with DDQN Wang al. (2016) in the OpenAl Gym Brockman et al.] (2016) Atari environment Bellemare et al. (2013). We test trained agents from several Atari environments averaged over IO episodes. In Figure show the original states
7

---

Under review as a conference paper at ICLR 2021
0D4303
14
3
Figure Original frame and environmental modifications. Columns: original frame, shifting. rotation, perspective transformation, blurring, compression artifacts. brightness and contrast. Rows: Bankheist, Timepilot, Pong, JamesBond, Riverraid.
and the environmental modifications. Interestingly, we found that a majority of the games have high robustness against rotation. On the other hand, shifting and perspective transformation can reach a higher impact level than the state-of-the-un targeted attack while not being recognizable by human perception. observed that in some games, such as Pong and Riverraid, brightness and contrast requires radical changes cause the agent to fail, while for others the change required is imperceptible. Another thing we observed is that for games like Pong, which is relatively trivial compared to other games in the Atari environment, the threshold values for the environmental modification are higher. When the complexity the game increases the environment modification thresholds decrease drastically. We think that this issue could become more important deep reinforcement learning agents are deployed in more complex and realistic scenarios.
CONCLUSION
In this paper we studied realistic threat model based basic environmental changes and proposed a framework called daylight to asses the generalization capabilities deep reinforcement learning agents. We compared daylight framework with the of-t-of-the-art adversarial attacks in the Atari environment. We demonstrated that our framework achieves higher impact on agent performance with lower perceptual similarity distance without having access agents training details, real time access
8

---

Under review as a conference paper at ICLR 2021
to agents memory and perception system, and computationally demanding adversarial formulations to compute simultaneous perturbations. We investigated perturbations in the time domain and showed that the studies based on imperceptible perturbations can be revisited within the daylight framework. Finally, we show that each component of our framework contains distinct bands in the frequency domain, resulting in better estimate of the generalization capabilities of trained agents. We believe our framework can be instrumental towards generalization and robustification of deep reinforcement learning algorithms.
REFERENCES
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael. Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research., pp. 253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv:1606.01540 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57, 2017.
Niraj Chokshi. Tesla autopilot system found probably at fault in 2018 crash. The New York Times, 2020.
Samuel F. Dodge and Lina J. Karam. Understanding how image quality affects deep neural networks. In Eighth International Conference Quality of Multimedia Experience, QoMEX 2016, Lisbon, Portugal, June 6-8, 2016, pp- 1-6. IEFE, 2016.
Nicholas Ford, Justin Gilmer, Nicholas Carlini, and Ekin Cubuk. Adversarial examples natural consequence of test error in noise. International Conference on Machine Learning (ICML), 2019.
Adam Gleave, Michael Dennis, Cody Wild, Kant Neel, Sergey Levine, and Stuart Russell, Adver- sarial policies: Attacking deep reinforcement learning. International Conference on Learning Representations ICLR, 2020.
Ian Goodfellow, Jonathan Shelens, and Christian Szegedy. Explaning and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Sandy Huang, Nicholas Papernot, Yan Goodfellow. Ian an Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. Workshop Track of the 5th International Conference on Learning Representations, 2017.
Forrest N. landola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprini 1602.0760 2016.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. International Conference on Learning Representations, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 2012.
Fredrick Kunkle. Fatal crash with self-driving car was a first like bridget driscoll's was 121 years ago with one of the first cars. Washington Post, 2018.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan. Wierstra. Continuous control with deep reinforcement learning, arXivpreprint 1rXiv.150..02971. 2015.
Yen-Chen Lin, Hong Zhang- Wei, Yuan-Hong Liao, Meng-Li Li Shih, ing Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. Proceedings of the Twenty-Sixth International Joint Conference Artificial Intelligence, pp. 3756-372,, 2017.

---

Under review as a conference paper at ICLR 2021
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of poysically-plausiae perturbations. In Proceedings of the IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3932-3939 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, arc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland. Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518: 529-533, 2015.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. International Conference on Learning Representations ICLR, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg, Klimov. Proximal policy optimization algorithms. 1oXiv:1707006344 [cs.LG)., 2017.
Karen Simonyan and Andrew. Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, ICLR, 2015.
Jianwen Sun, Tianwei Zhang, Lei Xiaofei, Xie Ma, Yan Zheng, Kangjie Chen, and Yang. Liu. Stealthy and efficient adversarial attacks against deep reinforcement learning. Association for the Advancement of Artificial Intelligence (AAAT), 2020.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dimutru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Bill Vlasic and Neal E. Boudette. Self-driving tesla was involved in fatal crash, u.s. says. The New York Times, 2016.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando. De Freitas. Dueling network architectures for deep reinforcement learning. Internation Conference on Machine Learning ICMI.,, pp. 1995-2003, 2016.
Dong Yin, Raphael Gonjito Lopes, Jonathan Shlens, Ekin D. Cubuk, and Justin. Gilmer. fourier perspective on model robustness in computer vision. Conference on Neural Information Processing Systems (NeurIPS), 2019.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver. Wang. The unreasonable effectiveness of deep features perceptual metric. Conference on Computer Vision and Pattern Recognition 2018..
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A. RAW SCORE RESULTS
In this section we provide the raw scores of the agents under the observation modifications from both the sfate-of-the-art adversarial attack and the Daylight framework with components: Brightness & Contrast, Blurring, Rotation, Shifting, Compression Artifacts, and Perspective Transform.
Table 8: Raw Scores of Carlini & Wagner (2017) (C&W) W) and brightness & contrast (B&C) with corresponding perceptual similarity P similarity and the (a: 8].
Games
C&W W Raw Scores
B&C Raw Scores
C&W Pamilarity
B&C P Peimilarity
[a. 3]
BankHeist 285.0
15.0
17.0
0.171
0.127
[1.2,40]
JamesBond
45.0
0.326
0.015
[0.9,20]
Pong
-20.8
-21.0
0.608
0.245
[1.7.40]
RiverRaid
1168.0
744.0
0.255
0.168
[2.4,-275]
TimePilot
4090.0
3180.0
0.279
0.070
[2.4,-260]
Table 9: Raw Scores of Carlini & Wagner (2017) (C&W) and blurring with corresponding perceptual similarity Pamilarity and kernel size.
Games
C&W Raw Scores
Blurring Raw Scores
C&W P Pamilarity
Blurring Pamilarity
Kernel Size
BankHeist
15.0
18.0
0.171
0.168
5
JamesBond
285.0
190.0
0.326
0.048
3
Pong -20,8
-21.0
0,245
0.098
3
RiverRaid
1168.0
820.0
0.255
0.134
5
TimePilot
4090.0
3880.0
0.279
0.124
5
Table 10: Raw Scores of Carlini & Wagner 2017) (C&W) and rotation with corresponding perceptual similarity P Psamilarity. and the rotation angle (RD denotes rotation degree).
Games
C&W Raw Scores
Rotation Raw Scores
C&W W Paimilaity
Rotation Psimilarity
RD
BankHeist
15.0
2.0
0.171
0.064
1.4
JamesBond
285.0
190.0
0.326
0.029
1.6
Pong
-20.8
-20.6
0.245
0.112
RiverRaid
1168.0
873.0
0.255
0.059
1.8
TimePilot
4000.0
3150.0
0.279
0.068
Table 11: Raw Scores of Carlini Wagner 2017 similarity P. Pamillaitty and the shifting [t,, t
(C&W) and shifting with corresponding perceptual
Games
C&W Raw Scores
Shifting Raw Scores
C&W Pa Paimiturity
Shifting P Paimitarity
[t.,
BankHHeist
15.0
13.0
0.171
0.060
[1.1]
JamesBond
285.0
70.0
0,326
0.048
[0,1]
Pong
-20.8
-21.0
0.608
0.217
[2.1]
RiverRaid
1168.0
988.0
0.255
0.095
[1,2]
TimePilot
4000.0
3560.0
0.279
0.120
[2.2]
11

---

Under review as a conference paper at ICLR 2021
Table 12: Raw Scores of Carlini & Wagner (2117)
(C&W) and compression artifacts (CA) with
corresponding perceptual similarity similarity.
Games
C&W Raw Scores
CA Raw Scores
C&W P Psimilarity
CA Psimilarity
BankHeist
15.0
17.0
0.171
0.067
JamesBond
250.0
60.0
0,326
0.035
Pong
-20.8
-19.4
0,608
0.029
RiverRaid 1168.0
2589.0
0.255
0.057
TimePilot 4090.0
3980.0
0.279
0.067
Table 13: Raw Scores Carlini & Wagner (2017) (C&W) and perspective transformation (PT) with corresponding perceptual similarity P Psimilarity. and the perspective norm.
Games
C&W Raw Scores
Raw Scores
C&W P Paiilarity
PT Piiiilaiity
Perspective Norm
BankHeist 15.0
1.0
0.171
0.022
JamesBond
285.0
75.0
0.326
0.007
Pong
-20.8
-20.9
0.608
0.029
3
RiverRaid
1168.0
486.0
0.255
0.046
2
TimePilot 4000.0
3550.0
0.279
0.029
3
A.2 IMPACTS WITH FIXED SCORES
For the scope of the paper we used the Impact definition in Equation when we compare our proposed Daylight framework to the state-of-the-art targeted attacks. For more generalized comparison between different algorithms and different games one can use Scorectean as the score of the agent without fixed any modification to agent's 's observations system at the end of the episode and Score as the fixed minimum score for given game. Thus, we define the generalized impact as,
Scorcelean Scoreadv
Igenrall
(4)
Fixed Scorcelcan Score
From Table through Table 19 we Score fixed 0, and TimePilot O.
for Bankheist O, JamesBond Pong -21, Riverraid
Table 14: Generalized Impacts of Carlini & Wagner (2017) (C&W) and brightness & contrast (B&C)
with corresponding perceptual similarity P Psimilarity and the (a, (3)
Games
C&W Zgeneal
B&C Igonenl
C&W P Pamilarity
B&C Paamilarity
[a, 3]
BankHeist
0.982
0.966
0.171
0.127
[1.2,40]
JamesBond
0.451
0.913
0.326
0.015
[0.9,20]
Pong 0.995
1.0
0.608
0.245
[1.7,40]
RiverRaid
0,928
0.951
0.255
0.168
[2.4,-275]
TimePilot
0.567
0.663
0.279
0.070
[2.4,-260]
Table 15: Generalized Impacts Carlini & Wagner (0117) (C&W) and blurring with corresponding
perceptual similarity Psimilarity, and kernel size.
Games
C&W Igeneral
Blurring Igeneral
C&W Psimilurty
Blurring Psimilarity
Kernel Size
BankHeist
0,982
0.979
0.171
0.168
JamesBond
0.451
0.635
0,326
0.048
Pong
0.995
1.0
0.245
0.098
RiverRaid
0.928
0.946
0.255
0.134
TimePilot
0.567
0.589
0.279
0.124
12

---

Under review as a conference paper at ICLR 2021
Table 16: Generalized Impacts of Carlini & Wagner (017)) (C&W) and rotation with corresponding perceptual similarity P Piiiilarity, and the rotation angle (RD denotes rotation degree).
Games
C&W Igeneral
Rotation Zgeneral
C&W P Psimilarity
Rotation P Primilarit
RD
BankHeist
0,982
0.997
0.171
0.064
1.4
JamesBond
0.451
0.635
0.326
0.029
1.6
Pong
0.995
0.99
0.245
0.112
RiverRaid
0,928
0.942
0.255
0.059
1.8
TimePilot 0.567
0.581
0.279
0.068
Table 17: Generalized Impacts of Carlini & Wagner (017) (C&W) and shifting with corresponding perceptual similarity Pimilariity and the shifting [t;, tyl.
Games
C&W Egeneral
Shifting Igeneral
C&W P Psimilarity
Shifting P Psimilarity
[+,,,]
BankHeist
0.982
0.985
0.171
0.060
[1.1]
JamesBond
0.451
0.865
0.326
0.048
[0.1]
Pong 0.995
1.0
0.608
0.217
[2.1]
RiverRaid
0.928
0.935
0.255
0.095
[1.2]
TimePilot
0.567
0.623
0.279
0.120
[2,2]
Table 18: Generalized Impacts of Carlini & Wagner (017) with corresponding perceptual similarity Psimilarity.
(C&W) W) and compression artifacts (CA)
Games
C&W Igemeral
CA Igenural
C&W Psimilarity
CA P Paimilarity
BankHeist
0.982
0.980
0.171
0.067
JamesBond
0.451
0.884
0.326
0.035
Pong
0.995
0.962
0.608
0,029
RiverRaid
0.923
0.803
0.255
0.057
TimePilot 0.567
0.578
0,279
0.067
Table 19: Generalized Impacts of Carlini & Wagner (017)) (C&W) and perspective transformation
(PT) with corresponding perceptual similarity P Piiiiluriyy and the perspective norm.
Games
C&W Igeneral
Igeneral
C&W Psimilarity
PT Psimilarity
Perspective Norm
BankHeist
0.982
0.998
0.171
0.022
JamesBond
0.451
0.865
0.326
0.007
Pong
0.995
0.996
0.608
0.029
RiverRaid 0,928
0.968
0.255
0.046
2
TimePilot
0.567
0.624
0.279
0.029
3
A.3 POLICY GRADIENT METHOOSS UNDER DAYLIGHTT FRAMEWORK
In this section we investigate policy gradient methods. In particular, Table 20 shows the raw scores, and generalized impacts Igeneral of the agent trained with A3C under the Daylight framework with following observation modifications: brightness & constrast, blurring, rotation, shifting, compression artifacts and perspective transform. In Table 20 exact same hyperparameters have been used as stated in Table 1 through Table 6 for the Daylight framework. Note that Daylight hyperparameters refers for brightnes and contrast to [ax, 3]. for blurring to the kernel size, for rotation to rotation degree, for shifting to [f;, Lj]. and for perspective transformation to perspective norm. Shifting and compression artifacts have nearly maximal impact on the performance the agent trained with A3C, while the other perturbations all have impact at least 0.9. Note that did not change the Daylight hyperparameters for direct comparison between the A3C agent and the DDQN agent. Therefore, although impact is slightly lower for brightness & contrast for A3C than for DDQN, it is possible
13

---

Under review as a conference paper at ICLR 2021
that choosing different values of and | while minimizing the perceptual similarity P similarit still result in a higher impact for an agent trained with A3C.
Table 20: Raw Scores and generalized impacts of the agent trained with A3C algorithm in Pong environment and evaluated with Daylight frame work: brightness &contrast, blurring, rotation, shifting, compression artifacts (CA) and perspective transform (PT).
Pong
Bright&Contrast
Blurring
Rotation
Shifting
CA
PT
Raw Scores
-17
-20.35
-9999
-20.71
-20.89
-19.11
Generalized Impacts ggeneral
0.904
0.984
0.974
0.993
0.997
0.954
Daylight hyperparameters
[1.7,40]
3
3
[2,1]
3
A.4 TIMING PERSPECTIVE
Note that the fraction Cadv can differ slightly from p due to random fluctuations, therefore we also report these values in Table Note that eadv varies between games. This was done because each game has a different minimum threshold for Eadv to achieve stable impact across episodes.
Table 21: Impact comparison with the fraction of adversarial observations per episode Eadv-
RiverRaid TimePilot
BankHeist
Pong
JamesBond
C&W Impact
0.359
0.148
0.249
0.077
0.021
Shifting Impact
0.513
0.374
0.326
0.114
0.165
Perspective Transformation Impact
0.391
0,315
0.338
0.108
0.121
Blurring Impact
0.501
0.155
0.304
0.12
0.319
Brightness Contrast Impact
0.517
0.188
0.313
0.098
0.154
Rotation Impact
0.417
0.192
0.260
0.079
0.044
Compression Artifacts Impact
0.184
0.262
0.267
0.017
0.198
C&W Galy
0.096
0.020
0.021
0.062
Shifting Cale
0.100
0.019
0.020
0.062
0,084
Perspective Transform eacr
0.098
0,020
0.020
0.060
0,082
Blurring
0.099
0,019
0.020
0.061
0,083
Brightness Cule
0.101
0,020
0.018
0.062
0.082
Rotation fudv
0.099
0.021
0.020
0.061
0.083
Compression Artifacts Cad.
0.020
0.056
Sul, observation probability p
0.02
0.02
0.06
0.08
A.5 FORMULAS FOR PERSPECTIVE TRANSFORMATION
S
ts
M
STEA Sj
(5)
Mi18; + M128; + Mis
Ma18; + M228; + M23
Sad (i,j) = M318; + M328, + Ma9
(6)
M3 + Ma28 M
14


Reviews:
[{"review_id": "tp4lp0_axag", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strong points:\n1. The authors propose a new framework for constructing adversarial data samples for deep reinforcement learning training. It contains six different concrete methods. Some of them can generate data samples fairly different from the original data samples.\n2. The authors do experiments with the proposed technique to check the generalizability on 10 atari games, and show that the generated data samples can fail the agent training to a different extent.\n\nWeak points:\n1. The biggest problem is that: although the authors have demonstrated that the generated data samples are fairly different from the original data samples, how to guarantee that there is still enough meaningful information contained in these training data samples? Definitely, we can look at the generated figures and it seems that they are still recognizable. However, we probably need some quantitative measures to show that these data samples can still be used to train good agents. To the extreme, we can enumerate some random samples, which will be extremely different from the original samples, and they will also fail the training of the rl agents.\n2. More quantitative experimental results are needed. It is more convincing to see how much score loss can the proposed framework cause, compared with clean data samples.\n3. One central part of the evaluation should aim to answer: whether this framework can differentiate the generalizability of algorithms better than other frameworks. This should be shown in experiments. For instance, say we have three algorithms, A, B and C, and we believe the groundtruth is their generalizability is A >> B ~ C.  Then, under this framework, we should see the score align with this conclusion better than under other frameworks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "h_YnaFs8lb-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "################################################\n\nSummary:\nPrevious work on crafting attacks for deep reinforcement learning has relied on computing adversarial examples using knowledge of the environment, policy, and optimizer. Using Atari games and DDQN, this paper shows that simple image distortions, such as brightness changes, blurring, and rotations, often has greater impact on the agent's performance and is perceptually more similar to the original images. \n\n################################################\n\nPros:\n1. The proposed attacks are simple, intuitively meaningful, and computationally cheap. They are black box, not requiring information about the environment and policy, which is a more realistic setting.\n2. The proposed attacks seem to be better than Carlini & Wagner for most games. I was a little surprised how much, especially for JamesBond.\n\nCons:\n1. The results are shown for only one algorithm, DDQN, only on Atari. It would be good to provide results for newer algorithms like Ape-X or other environments like DMControl.\n2. Deep RL algorithms usually have large variance. It would be good to provide standard errors of the results, to accurately compare Daylight to Carlini & Wagner.\n3. The writing has some weaknesses, please see below for details.\n\n################################################\n\nOverall: I would lean toward accepting this paper. Technically it is not very sophisticated and the proposed attacks have been considered for image classification [1], but I believe that the results have strong practical implications. That is, simple image distortions are surprisingly impactful for attacking deep RL agents.\n\n[1] Samuel Dodge and Lina Karam. \"Understanding How Image Quality Affects Deep Neural Networks\". ArXiv 1604.04004.\n\n################################################\n\nSuggestions for writing:\n1. I think it would be better if the experimental setup preceded the results in Section 3. In addition, more details should be given, such as: How were the specific games and algorithm chosen? \n2. I think the paper [1] is relevant for related work, as it shows similar distortions are good attacks for image classification.\n3. The notation D(s) at the bottom of page 2 is undefined.\n4. The differences in Figure 2 are not very apparent. It may be helpful to add bounding boxes.\n\nFurther comments and questions:\n1. What happens if we combine several of these attacks? Would that lead to even greater impact?\n2. Is there any intuition for why certain games are more robust to certain image distortions than others? For example, why are the results for compression artifacts more mixed?\n\n################################################\n\nUpdate after reading other reviews and author responses:\n\nI am happy to keep my score and remain positive about this paper; the authors have answered my questions and partially addressed my main concerns in the revised paper. Like Reviewer 3, I would hope to see complete results for A3C in the final paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ip35lr5b-9y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work focuses on assessing the generalization of deep reinforcement learning by applying semantically meaningful perturbations to the RL agent’s observation system, e.g., brightness, blurring, contrats, shifting etc. The authors demonstrate strong degradation in the performance of RL agents in various Atari games, despite the semantic perturbations having significantly lower perceptual distance as opposed to planned adversarial attacks.\n\n\nReview:\n\nThe paper is well-written and easy to understand. Related work has been adequately referenced and the paper is of significant interest to the ICLR community.\n\n\nStrengths:\n\n1. The evaluation is quite comprehensive and covers multiple Atari environments.\n\n2. The chosen perturbations are semantically meaningful and cover a broad range of frequencies in the fourier domain.\n\n2. The degradation impact from semantically meaningful perturbations has also been compared to that from adversarial attacks, while at the same time characterising their perceptual similarity to original observations.\n\n\nWeaknesses:\n\n1. My major concern is that the metric defined to capture Impact (eq 2 in section 2.5) is somewhat unclear. Normally, one defines a score and then normalizes it with constants \"max\" and \"min\", for instance, like (max - score)/(max - min). In such a case, \"max\" and \"min\" are generally constant upper and lower bounds on score. However, while at the surface eq 2 seems to be doing the same, it might actually be doing something different. This is because of how Score_{max} and Score_{min} are defined. Firstly, Score_{max} and Score_{min} are not constants but differ for an agent depending on its Q-function. Secondly, the Score_{min} term seems somewhat misleading. The authors state that \"Score_{min} is the score at the end of the episode achieved by the agent who takes the action that minimizes its Q(s, a) function in every state visited\". Since, Q(s,a) is the value of action a in state s given that the agent maximizes the value from the next state onwards, just choosing the minimizing action in the current state may not be akin to choosing the actual minimum return from an episode. Hence, this functional form used to capture Impact may be doing more than just a score normalization. Can the authors please comment on why they chose the definition of Impact as such and if this could be having potential impacts on the results shown throughout the paper?\n\n2. Additionally, since the authors have conditioned the definition of Impact on Q-values, it seems like the framework is only restricted to evaluation of agents which learn a Q-function. Direct policy-based agents may not be amenable to the Daylight framework. Is it possible to extend the framework to handle RL agents, which do not learn a Q-function?\n\nI am happy to reconsider my score if the above two major weaknesses can be addressed.\n\nMinor comments:\n\n1. In most experiments from section 3, BankHeist stands out as an exception in the sense that Carlini & Wagner (2017) seems to be competitive on it as opposed to the proposed perturbations. It would be interesting to analyze and understand what sets this game apart.\n\n2. The text and labels on the axes in figure 1 are unreadable. Please rectify the font size.\n\n3. I might have misunderstood this, but why does figure 1 have only one spatial frequency axis? Shouldn't it also have two spatial frequency axes like in figure 2?\n\n4. Minor punctuation error in the abstract: \"state-of-the-art adversarial attacks, .\"\n\n------ Post-rebuttal update ------\n\nThe reviewers have addressed my concerns for the most part and I am happy to update my score to recommend acceptance. I also hope to see the A3C experiments in the final draft of the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes several simple alternatives to generate adversarial examples for deep reinforcement learning algorithms based on image distortions such as lighting change, blurring and rotation, and show the performance of DRL agents degenerate as a result. Most reviewers appreciate the simplicity and computational efficiency of the proposed attacks. The results revealed by the work is however rather unsurprising, given similar attacks have been evaluated for DNNs. The authors did not offer much more insight on the presented results beyond that, for example, robustness of different DRL algorithms with regards to these attacks as mentioned by reviewer 2, sensitivities of the parameters for each attack proposed, effectiveness of different attacks on different environment and possible combination of attacks. 
Decision: Reject