Exploring Transferability of Perturbations in Deep Reinforcement Learning
Authors: Ezgi Korkmaz
Venue:  (2021)
Keywords: transferability
deep reinforcement learning
generalization
adversarial

Abstract: The use of Deep Neural Networks (DNNs) as function approximators has led to striking progress for reinforcement learning algorithms and applications. At the same time, deep reinforcement learning agents have inherited the vulnerability of DNNs to imperceptible adversarial perturbations to their inputs. Prior work on adversarial perturbations for deep reinforcement learning has generally relied on calculating an adversarial perturbation customized to each state visited by the agent. In this paper we propose a more realistic threat model in which the adversary computes the perturbation only once based on a single state. Furthermore, we show that to cause a deep reinforcement learning agent to fail it is enough to have only one adversarial offset vector in a black-box setting. We conduct experiments in various games from the Atari environment, and use our single-state adversaries to demonstrate the transferability of perturbations both between states of one MDP, and between entirely different MDPs. We believe our adversary framework reveals fundamental properties of the environments used in deep reinforcement learning training, and is a tangible step towards building robust and reliable deep reinforcement learning agents.

Under review as a conference paper at ICLR 2021
EXPLORING TRANSFERABILITY OF PERTURBATIONS IN DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The use of Deep Neural Networks (DNN) as function approximators has led to striking progress for reinforcement learning algorithms and applications. At the same time, deep reinforcement learning agents have inherited the vulnerability of DNNs imperceptible adversarial perturbations to their inputs. Prior work on adversarial perturbations for deep reinforcement learning has generally relied on calculating an adversarial perturbation customized to each state visited by the agent. In this paper propose more realistic threat model in which the adversary computes the perturbation only once based on a single state. Furthermore, we show that to cause deep reinforcement learning agent to fail it is enough to have only one adversarial offset vector a black-box setting. We conduct experiments various environments from the Atari baselines, and to demonstrate the transferability of perturbations both between states one MDP, and between entirely different MDPs. We believe our adversary framework reveals fundamental properties of the environments used in deep reinforcement learning training, and is tangible step towards building robust and reliable deep reinforcement learning agents.
single-state adversaries
INTRODUCTION
Building on the success of DNNs for image classification, deep reinforcement learning has seen remarkable advances in various complex environments Mnihh al. ( |Schulman et al. (2017); Lillicrap et al. 2015). Along with these successes come new challenges stemming from the lack of robustness of DNNs to small adversarial perturbations to inputs. This lack of robustness is especially critical for deep reinforcement learning, where the actions taken by the agent can have serious real-life
consequences Leviin Carrie (2018)).
The initial work adversarial examples in deep learning by SZzgedy et al. 2014) showed that imperceptible perturbations added to images could be used to cause a DNN image classifier to misclassify, and hypothesized that this Follow-up work by Goodfellow et al. (2015) proposed a more computationally efficient fast gradient sign method (FGSM) for computing adversarial examples, while simultaneously explaining the presence adversarial examples as a result of DNN classifiers learning approximately linear functions. The authors additionally argue that the presence of common high-dimensional linear decision boundary is an explanation for the transferability of adversarial examples between models trained on different minibatches of the same training set or with different architecture. The transferability of perturbations themselves, in contrast to transferability of complete adversarial examples, was investigated by (Moosavi-Deztorss Dezfooli al. (017). The authors showed how to compute one universal perturbation from a minibatch of images that could be used to fool a classifier by adding it to each image in the validation set. The authors argue the existence of universal perturbations explained by correlations between the local decision boundaries near many different points in the dataset.
result of the DNN's complexity and nonlinearities.
More recently, a new body of research on adversarial perturbations for deep reinforcement learning agents has developed, demonstrating that it is possible to make a segh-performing agent fail by adding perturbations to the agent's 's perception of the current state Lin et al. (2017); Sun et al. (2020); Huang et al. (2017); Pattanaik et al. (2018) 1); Mandlekar et al. (2017). A major drawback of this prior work is the threat model, in which the attacker computes perturbations and adds them to the agent's 's state in real-time before the state is perceived by the agent. Such threat model is not practically

---

Under review as a conference paper at ICLR 2021
feasible considering the heavy duty optimization methods used by Lin et al. (017): : Sun et al. (2020).
To address this we propose a new threat model where the attacker computes just one adversarial
offset vector based single state. Not only is this more practical threat model from security perspective, but the success of attacks in this model can be used to give insight into the transferabilityy of adversarial examples and the properties of functions learned by trained deep reinforcement learning agents,
We believe the adversarial perspective is an initial step towards understanding the loss landscape and geometry of the decision boundaries of the algorithms in use, assessing the generalization capabilities of trained agents, and building robust and reliable deep reinforcement learning agents. For these reasons in our paper we focus on the adversarial perspective and make the following contributions:
We introduce framework of six unique adversary types encapsulating our new threat model where the attacker is restricted to computing an adversarial perturbation based on a single state.
We use our framework to investigate the transferability of adversarial perturbations, both between states of the same MDP and between MDPs.
We show with experiments in the Atari baselines that single state attacks have significant impact on agent performance.
In particular, we show that by using only one perturbation computed based on a random state of random episode from completely different environment, it is possible to have a dramatic impact on the deep reinforcement learning agent's 's performance. This transfer- ability of adversarial perturbations between environments implies that there is a correlation between the features learned by agents trained in different environments. While these results demonstrate that deep reinforcement learning agents are indeed learning representations that generalize across environments, they also demonstrate a critical vulnerability from the security point of view that can be exploited without any knowledge of the training environment or neural architecture.
2 RELATED WORK AND BACKGORUND
2.1 DEEP REINFORCEMENT LEARNING
In this paper we examine discrete action space MDPs that are represented by a tuple: M (S, A, P. r, 7, so) where S a of continous states, A is the transition probability, r S x A + I is the reward function, y is the discount factor, and sg is the initial state distribution. The agent interacts with the environment by observing s S and taking actions E A. The goal is to learn a policy Tg : S x A + R which takes an action in state s that maximizes the cumulative discounted reward ΣT-1 7"r(st, az). For an MDP M and policy I(s, a) we call a sequence of state, action, reward tuples, (8i, (s,,,,;;), ), that occurs when utilizing ((s, a) in M an episode. We PM. denote the probability distribution over the episodes generated by the randomness in M and the policy т.
of discrete actions, IP S x A x R
2.2 ADVERSARIAL PERTURBATION METHODS
In this work we create the adversarial perturbations via two different methods. The first is the derliii Wagner (2017) formulation. In the deep reinforcement learning setup this formulation is,
min c (-((su + || - >|12 E8
(I)
where s is the unperturbed input, Sadv is the adversarially perturbed input, and J(s) is the augmented cost function used to train the network. The second method we use to produce the adversarial examples is the elastic net regularization (EAD) Chen et al. (2018) adversarial formulation,
CheD)
min c. (Satv) + X, || s|l, + A 1|sadv =||
(2)
2

---

Under review as a conference paper at ICLR 2021
3 EXPLORING TRANSFERABILITY
In this work we introduce the notion of single-state transferability in deep reinforcement learning, and investigate its effects in state-of-the-ar deep reinforcement learning environment baselines. We explore single-state transferability in our proposed adversary framework, which consists of computing an adversarial perturbation from random state, and then adding that particular adversarial perturbation to several other states, both within the same MDP and across different MDPs. We refer to the transferability of adversarial perturbations between states in the same MDP state transferability. Similarly, we refer to transferability of perturbations between states in different MDPs as environment transferability. To be able to fully explore the different types of transferability define framework with several adversary variants in Section 3.2
3.1 THREAT MODEL
In our work we propose threat model that could plausibly be deployed in physical deep reinforce- ment learning setup. Physical adversaries for image classification have been extensively studied. Kurakin et al. (2017) showed that it is possible fool DNN by printing out an adversarial picture and then presenting the print out as an input to the classifier. Evtimov et al. (2018) demonstrated that placing stickers on an object can also make DNN image classifiers fail. Athalye al. (2017) showed that one can physically build an adversarial object with 3D printer that can fool DNN classifier. Recently, [Li B. al, (2019) proposed a realistic physical perturbation, a camera sticker, fool DNN image classifiers.
Previous work on adversary models in deep reinforcement learning has focused on computing the adversarial examples customized to each state Huang al. (2017) 77: Mandlekar et al. (2017); Pattanaik itt al. (2018); Lin et al. (2017); Pinto et al. (2017); Sun et al. (2020). In particular, the adversary model of Lin et al. (2017) Sun al. (2020) requires real-time access to the agents perception system, while using the highly computationally demanding Carlini & Wagner (017) adversarial formulation to compute perturbations. Such model, where the adversarial perturbation for each input state is computed and added in real time, is firstly not computationally efficient, and secondly not a realistic adversarial scenario in many settings of interest. In our threat model we propose to only have one offset vector to add to every clean image observed by the deep reinforcement learning agent. We believe this scenario be more physically viable, since this single adversarial perturbation could be a change to the lens of the camera of the deep reinforcement learning agent.
3.2 ADVERSARIAL FRAMEWORK
In the following section we define six unique adversary models in our framework evaluate our proposed threat model. Given a policy (s,, a) we define r7(s, π (s. a)) to be perturbation computed based only on the state s and the policy =ss, a) in state In all the following definitions we fix bound is and set
K
<(n)
(3)
|
Definition 1: Individual state adversary, Aindividual is the adversary that calculates the perturbation custom each state.
= E(n) 17(5,, (s,
Definition 2: Initial state adversary, iiniiill, is the adversary that calculates the perturbation based on the initial state of the agent in the given episode, and adds the initial state perturbation to the rest of the states visited.
(5) Definition 3: Episode independent initial state adversary, Ainitial, is the adversary that calculates the perturbation based on the initial state of the agent in a random episode e, and adds the initial state perturbation to all states visited in episode e'. Given an episode pM, the perturbed state is,
s" = <(n)) (755,, T as).a))
3

---

Under review as a conference paper at ICLR 2021
s'= (()())
(6)
where M the MDP and so(e) is the initial state of episode e.
Definition 4: Episode independent random state adversary, A is the adversary that calculates the perturbation based on a random state of the agent in a random episode e, and adds the random state perturbation all the states visited in episode e'. Given an episode pM, the perturbed state is,
s" = + E(n) 77( (random π (Srandom (rrandomee),o)
where M the MDP and Srandom (e) a random state of episode e.
Definition S: Environment independent initial state adversary. Ajiltiall is the adversary that calculates the perturbation based on the initial state of an agent in a random episode of the MDP M, and adds the initial state perturbation to all of the states visited in episode of the MDP M'. Given episode e ~ par, the perturbed state is,
= s E(T) m7sso(e M), a (M)
(8)
where so(e, M) the initial state of episode e of MDP M.
Definition 6: Environment independent random state adversary, Arandom, is the adversary that calculates the perturbation based on random state of the agent in a random episode of the MDP M, and adds the random state perturbation all the states visited in episode of the MDP M'. Given an episode ~ pas, the perturbed state
s" = s + E(n) (7 (e, M), π (Srandom (e, Af), a))
(9)
where Syandom (e, M) is a random state of episode of MDP M.
EXPERIMENTS
In our experiments agents are trained with DDQN Wang et al. (2016) in the Atari environment Bellemare et al. (2013) from Orockaan Brockman al. (2016). We average over 10 episodes in our experiments. We define the impact of the adversary on the agent by normalizing the performance drop follows
Scoremax Scoready
Impact-
(10)
Scoremax
Scoremin
Here Scoremax is the score of the baseline trained agent following the learned policy without the presence of adversary. Score is the score of the agent under the influence of the adversary, and Scoremin is the score the agent receives when choosing the action minimizing its Q(s, a) function in every state. All scores are recorded at the end of an episode. We chose this normalization because we observed that the agent can still receive non-zero reward even when choosing the worst action in each state.
In our framework the adversaries Ainitial Alnitial and Arandom all explore different aspects of state transferability.. Arandom the most natural test for transferability between states of one MDP, while Ainitial and Almitial represent more restricted adversaries that only have access to the initial state. Similarly, A Arundom is the most natural test for environment transferability, while Ainitial is a more restricted version where the adversary does not have access to arbitrary states in the MDP M. Therefore, higher impact in A4 mmndom and grandom indicates higher state and environment transferability respectively.
Table and Table 2 show impact values of six different adversary models from our adversarial framework utilizing the Carlini & Wagner (017) formulation and EAD formualtion respectively
4

---

Under review as a conference paper at ICLR 2021
for various environments from the Atari baselines. These results show striking level of both state transferability and environment transferabilitty. One can observe that while the EAD formulation has significant state transferability for all environments, the Carlini & Wagner (2017) formulation only has significant state transferability for CrazyClimber and Pong. Additionally ly, the EAD formulation also has higher environment transferability in all environments except Pong and Roadrunner.
Table I: Impacts of Carlini & Wagner (217)) for the six different adversary definitions within the proposed adversarisal framework.
Environments individual
A initial
Ainitial
A
graddom
inittial M
Aandom M
BankHeist
0.694
0.232
0.069
0.118
0.086
0.022
RoadRunner
0.876
0.674
0.755
0.649
0.064
0.919
JamesBond
0.038
0.048
0.086
0.019
0.942
0.304
CrazyClimber
0.646
0.637
0.601
0.956
0.014
0.036
TimePilot
0.734
0.334
0.316
0.228
0.068
0.017
Pong 0.998
1.0
1.0
1.0
0,007
0.966
Table Impacts of EAD for the six different adversary definitions within the proposed adversarisal framework.
Environments
A Ainiividual
A initial
Ainitial
Faadom
Annitial
Arandom
BankHeist
0.646
0.144
0.009
0.764
0.182
0.496
RoadRunner
0.821
0.964
0.612
0.816
0.034
0.816
JamesBond
0.098
0.096
0.038
0.612
0.865
0.910
CrazyClimber
0.750
0.002
0,028
0.980
0.925
0,983
TimoPilot 0.776
0.394
0.380
0,517
0.202
0.312
Pong
0.995
0.997
1.0
1.0
0.983
0.946
extremely surprising to notice that the impact of Aíndividuul in JamesBond is distinctily lower than Arandom, The reason for this in JamesBond is that Arrandom consistently shifts all actions towards action 12 while Aiddividual causes the agent choose different suboptimal actions in every state. See section 4.1 for more details. We observe that this consistent shift towards one particular action results much more successful attack in certain environments. In JamesBond, there are certain obstacles that the agent must jump over in order avoid death, and consistently taking action 12 prevents the agent from jumping far enough. In CrazyClimber, the consistent shift towards one action results in the agent getting stuck in one state where choosing any other action would likely free it. See our github page for videos of the agent's behavior for these cases.
Table Gaussian noise impact for the same f--norm bound used in Table and Table 2
BankHeist
RoadRunner
JamesBond
CrazyClimber
TimePilol
Pong
Impacts
0.078
0.027
0.038
0.054
0.031
0.045
In all the experiments we set the f2-norm bound k to a level so that Gaussian noise with & - has insignificant impact. In Table we show the Gaussian noise impacts on the environments of interest with the same fo-norm bound A used the EAD and Carlini & Wagner (2017) formulations. It is important note that unbounded Carlini & Wagner (2017) produces perturbations large enough in l2 norm that Gaussian noise the same magnitude has similar impact. Therefore, we bound the perturbation level where Gaussian noise is not effective.
5

---

Under review as a conference paper at ICLR 2021
4.1 EMPIRICAL EVALUATION OF ACTIONS
In this subsection we will define the functions on actions taken by the agent that we will use to understand the effects of the adversarial perturbations. In the following subsections will use these functions to explore the decision boundaries and representations learned by the deep reinforcement learning agent.
In the OpenAl Atari baselines the discrete action set is numbered from 0 to A|. For this purpose for each episode we collected statistics on the probability P(a) of taking action in the following scenarios:
Pisse(a) the fraction states in which action a E A is taken by the agent in an episode with no adversarial perturbation added.
Empirical Action Statistics
Empirical
0.7
0.40
0.8
0.35
0.5
0.25
0.4
0.15
0.30
0.0
0.00
Pong
BankHeist
Empirical Stattstic
Empirical Action Statistics
5.8
0.25
0.6
0.20
0.4
0.7
0.05
0.0
0.00
CrazyClimber
RoadRunner
Empirical Action Statistics
Empirical Action Statistics
0.30
6.3
0.20
0.15
6.2
0.05
0.00
CrazyClimber
JamesBond
Figure 1: Empirical action statistics of Arandom with EAD formulation for Pauntros (a). Powee (a) and (a)

---

Under review as a conference paper at ICLR 2021
(a) the fraction of states in which action A is taken by the agent in episode with the adversarial perturbation added.
(a, b) the fraction of states in which action a E A would have been taken by the agent if there were perturbation, but action b A was taken due to the added perturbation.
E. Padv (a, b) the fraction of states in which action E A would have been taken by the agent there were no perturbation, in an episode with the adversarial perturbation added.
Pocatro (a) =
The difference between (a) and Pndv(a) measures how effectively the adversary changes the action taken by the agent in each state. We refer to this as action manipulation. As a baseline an adversary must conduct action manipulation at least one state. The difference between Poontrol (a) and (a) measures how effectively the adversary changes the distribution of states visited by the agent. We refer to this as state manipulation. Unlike action manipulation that significant state manipulation is necessary to achieve high impact on the agent's performance.
not readily apparent
we observe that Roadrunner, JamesBond, and TimePilot Pountrol (a) and Pasee (a) are similar. Thus, for these enviromments state manipulation is relatively small. However, significant action manipulation occurring. Pong, BankHeist, and CrazyClimber the adversary achieves both significant state manipulation and action manipulation.
In Figure I
4.2 ACTION MANIPULATION
In Figure we show the heatmaps corresponding to (a (a,a), b). In these heatmaps we show which particular control action has been shifted to a particular adversarial action. In some environments there is a dominant action shift towards one particular adversarial action from one control action as in CrazyClimber or in TimePilot. In some games there are one or more clear bright columns where all the control actions are shifted towards the same particular adversarial action, for instance in JamesBond. Recall that all experiments were conducted using single adversarial offset vector added to every visited state. Thus, the presence of a bright column indicates that, for many different input states, this vector points across decision boundary that separates particular action from
Payla.b)
Pay
Pas ll
0.125
0.2
2.00
0.0
0.000
10
24135611
Pong
BankHeist
TimePilot
Pavle
Pacile
0.233
63
0.288
CrazyClimber
RoadRunner
JamesBond
Figure 2: Heat map of Pass (a, b) for Arandmm with EAD formulation.
7

---

Under review as a conference paper at ICLR 2021
Game Transferability
1.0
BankHelst
0.76
0.12
0.48
0.5
-0.081
0.2
0.8
TimePilot
0.52
0.23
0.27
0.087
JamesBond
-0.07
0.14
-0.058
0.25
Impact
RoadRunner
0.082
0.48
0.56
0.82
0.88
0.82
0.4
Pang
0.026
0.95
0.002
0.082
1
0.06
0.2
CrazyClimber
0.31
0.19
0.052
0.7
0.98
0.98
0.0
BankHelst
TimePilat
JamesBond
RoadRunner
Pong
CrazyClimber
Figure Environment transferability for AM Anadomm with EAD formulation. Each row shows the environment where the perturbation is added, and each column shows the environment from which the perturbation is computed.
all other actions. A plausible explanation for these bright columns that each one corresponds to diverse set of states for which the neighboring decision boundary geometry is highly correlated.
4.3 ENVIRONMENT TRANSFERABILITY
In this section we investigate transferability between environments. In these experiments we utilize Arandom where the perturbation computed from random state of a random episode one envi- ronment, and then added to a completely different environment. In Figure 3 we show the impacts for the A adversary in six different cnvironments. Each row shows the environment where the perturbation is added, and each column shows the environment from which the perturbation is com- puted. Note that the diagonal the environment transferability matrix represents state transferability, while the off diagonal elements represent environment transferabilty. One observe intriguing properties of the Atari baselines where certain environments are highly transferable. We show that the perturbations computed from RoadRunner highly effective in other environments and the perturbations computed from other environments are effective in RoadRunner. Additionally, in half of the environments higher or similar impact achieved when transferring a perturbation from different environment rather than using one computed in the environment itself. This high level of environment transferability is sign deep reinforcement learning agents are learning representations that have correlated features across different environments.
CONCLUSION
In this paper introduce framework based six different adversaries within our proposed transferability threat model. Using our framework we investigate the transferability properties of the environments and algorithms used for deep reinforcement learning training. We show that using a perturbation computed from random state of random episode of one environment can cause significant detrimental effects in completely different environment. While this transferability brings several complications from the security point of view, where an attacker can manipulate an agent in black-box setting without having access to the training environment and neural architecture, we also argue that it is clear sign that deep reinforcement learning agents indeed learn representations that generalize across different environments. In particular, environment transferability of perturbations indicates that agents have promising underlying generalization skills, We believe our results can give us a perspective on the generalization capabilities of deep reinforcement learning agents, and can be instrumental in building reliable and robust deep reinforcement learning agents.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. http://arxioor///d//7777777,7 2017.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael. Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research. pp. 253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv:1606.015400, 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEFE Symposium Security and Privacy (SP), pp. 39-57, 2017.
Pin- Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. EAD: elastic-net attacks to deep neural networks via adversarial examples. pp. 10-17, 2018.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https
// /gittub... cct///opeeeaaaaaaeeeeees 2017.
I. Evtimov, K. Eykholt, E. Fernandes, T. Kohno, B. Li, A. Prakash, A. Rahmati, and D. Song. Robust physicalworld attacks on deep learning models. Computer Vision and Pattern Recognition (CVPR), 2018.
Ian Goodfellow, Jonathan Shelens, and Christian Szegedy. Explaning and harnessing adversarial examples. International Conference Learning Representations, 2015.
Sandy Huang, Nicholas Papernot, Yan Goodfellow. lan an Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. Workshop Track of the 5th International Conference on Learning Representations, 2017.
Alexey Kurakin, Ian Goodfellow, and Samy. Bengio. Adversarial examples in the physical world. International Conference Machine Learning, 2017.
Sam Levin and Julia. Carrie. Self-driving uber kills arizona woman in first fatal crash involving pedestrian. The Guardian, 2018
Juncheng Li B., Frank Schmidt R., and J. Zico. Kolter. Adversarial camera stickers: A physical camera-based attack on deep learning systems. International Conference on Machine Learning (ICML), 2019.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan. Wierstra. Continuous control with deep reinforcement learning. arXivpreprint arXiv:1509.02977 02977999999..... 2015.
Yen-Chen Lin, Hong Zhang- Wei, Yuan-Hong Liao, Meng-Li Li Shih, ing- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. Proceedings of the Twenty- International Joint Conference on Artificial Intelligence, pp. 3756-3762, 2017.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausinee perturbations. In Proceedings the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939, 2017.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016.

---

Under review as a conference paper at ICLR 2021
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, are G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518: 529-533. 2015.
Moosavi-Dezfooli, Alhussein Seyed-Mohsen, Fawzi, Omar Fawzi, and Pascal. Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7765-1773, 2017.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, and Bommannan Gautham. Robust deep reinforce- ment learning with adversarial attacks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgen Systems, pp. 2040-2042, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. International Conference Learning Representations ICLR, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg. Klimov. Proximal policy optimization algorithms. 1TXiv:1707.0634772 [cs.LG]., 2017.
Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang. Liu. Stealthy and efficient adversarial attacks against deep reinforcement learning. Association for the Advancement of Artifical intelligence (AAAI), 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dimutru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Ziyu Wang. Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando. De Freitas. Dueling network architectures for deep reinforcement learning. Internation Conference on Machine Learning ICMI.,, pp. 1995-2003, 2016.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver. Wang. The unreasonable effectiveness of deep features as a perceptual metric. Conference on Computer Vision and Pattern Recognition (CVPR),
A APPENDIX
A.1 INVESTIGATION OF POLICY GRADIENT METHODS
In this section we provide an investigation into policy gradient algorithms. In Table 4 we demonstrate the impact results of the adversary definitions within the proposed adversarial framework with the EAD formulation. Note that the Arandom _niiiiall A and A adversaries in this example are computed by using one single perturbation from random state of agent trained with DDQN and then applying this perturbation to the agent trained with A3C. It is quite striking to observe that compared Gaussian noise the Amando adversary has great success on degrading the agent's performance. Recall that the A Arrandom adversary is computing single adversarial perturbation from random state of a completely different MDP trained with a different algorithm.
Table 4: Impacts of the adversary definitions within the proposed adversarial framework applied to A3C trained agent's observation system where the perturbation is computed with the EAD formulation for agent trained with DDQN.
Environment
A Gaussian
initial
Aradoo
iniitill
tandom
Pong
0.0789
0.783
0.751
0.724
0.632
10

---

Under review as a conference paper at ICLR 2021
A.2 PERCEPTUAL SIMILARITIES
In this section we provide an analysis of the perceptual similarities between states in the same MDP and states from different MDPs. In particular the perceptual similarity distance P measures the difference between two inputs based on internal activations of a neural network independent from the network architecture Zhang al. (2018). Figure demonstrates Fignneeeeosssssssees distance between states of different MDPs and states in the same MDP. Note that the perceptual similarity between states is not correlated with environment transferability as shown in Figure 3 In particular, the transferability from Pong to CrazyClimber is high as can be seen in Figure 3) while the perceptual similarity distance between states from the two MDPs is also high (i.e the states are perceptually dissimilar). Thus the perturbations transfer between these two MDPs despite the perceptual dissimilarity between the states of the two different MDPs.
the perceptual similarity
Perceptual Similarity Distances
BankHelst
0.214
0.622
0.601
0,656
0.613
0.502
0.7
Perceptual Similarity Distance (P)
TimePilat
0.622
0.341
0.542
0.609
0.534
0.685
0.6
0.5
JamesBond
0.601
0.609
0.179
0.686
0.647
0.584
0.4
RoadRunner
0.656
0.601
0.686
0.145
0.647
0.691
Pong
0.613
0.534
0.647
0.647
0.224
0.77
0.2
0.1
CrazyClimber
0.502
0.685
0.584
0.691
0.77
0.01
BankHeist
TimePilot
JamesBond
RoadRunner
Pang
CrazyClimber
Figure 4: Perceptual similarity distances between states inside the same MDP and between different MDPs.
A.3 ARCHITECTURAL DIFFERENCES AND HYPERPARAMETER
In this section we provide transferability results between different architectures for Araniom and Arandom adversaries. In general we found no difference between DDQN architectures. In the experiments this section the perturbation computed in the DDQN Prior Duel architecture, and then added to the agent's 's observation system trained with DDQN, DDQN Prior, DDQN Duel and DDQN Prior Duel.
Table Impacts of the adversary definitions within the proposed adversarisal framework where the perturbation is computed in the DDQN Prior Duel architecture and added to the agent's 's observa- tion system trained with DDQN, DDQN Prior, DDQN Duel and DDQN Prior Duel for the Pong environment.
Architectures
Gaussian
initial
Aradom
A
Ainitial
A
trandom
DDQN
0.002
1.0
1.0
1.0
0.995
DDQN Prior
0.075
1.0
1.0
1.0
0.985
DDQN Duel
0.036
1.0
1.0
1.0
0.966
DDQN Prior-Duel
0.041
1.0
1.0
1.0
0.981
11

---

Under review as a conference paper at ICLR 2021
DDQN hyperparameters as follows: learning rate for Adam optimizer is 5 10 buffer size is 50000, final value of random action probability 0.02, batch size 32, discount factor 1.0. See more details implementation and hyperparameters in Dhariwal al. (2017).
A.4 ENVIRONMENTS
In Figure 5 we show the screenshots of the MDPs considered in this paper. We observe that some environments from the Atari baselines have extremely high transferability. For the extreme example. the impact of the EAD formulation in Pong is 0.974 for Atandom Moreover, it is 0.906 for Note that this means the perturbation is computed based on a random state of random episode from random environment M, and still causes the agent to perform extremely poorly.
25700
1330
TimePilot
JamesBond
BankHeist
RoadRunner
CrazyClimber
Pong
Figure 5: Different environments from Atari Baselines
12


Reviews:
[{"review_id": "heTbsQJLYph", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\nThis paper aims at discovering transferability of perturbations across different environments in RL. The authors propose some different types of advasaries and tested those adversaries on 6 atari games.\n\nReview Summary:\nThe idea of analyzing perturbations' transferability is interesting, but it is hard to say that the approaches are satisfactory. The authors did not propose any novel algorithms or modifications based on existing algorithms, and provide only preliminary experiment results. As such, I suggest a clear rejection.\n\nDetailed Comments:\nSection 3.1, paragraph 2. \"Such a model... of the deep reinforcement learning agent\". I am hardly convinced by this statement and led by this statement, the proposed approach in this work. When we consider transferability in machine learning, we assume that there are something common to learn between the two domains. My first thought when I read the title is that it would be a transfer learning method for adversarial attacks. But the authors seem to prefer a universal offset on the raw pixel input. In this case, the only insights we might possibly learn is the input similarity between different Atari games, which is hardly a contribution to any field. Moreover, since adversarial attack is already used in this work, I don't see the reason why none of the referred methods for adversarial attacks in RL are tested.\n\nBesides, all experiments are done only on DDQN, hence the claims are hardly validated. What I expect is either a new algorithm that is more resilient or a comparison on robustness between different RL algorithms. It is also not explained how the environments are chosen.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "faV4pSOhfU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "— idea:\nA framework composed of 6 different adversaries is proposed using which it is claimed that the transferability properties among Atari environments can be studied.\n\n— comments:\n\nThe paper is well-written and easy to follow. However, I have major concerns about the novelty of the material discussed in the paper. The only novelty is about using which states as adversarially perturbed input. It mostly looks like an incremental study. \n\t\nIn the experiments section, authors have used a pre-trained DDQN agent which originally does not show significant generalizability compared to methods like A3C. It would be interesting to see the impact of the proposed framework on algorithms with better generalization capabilities. Thus, I believe the experiment section lacks this very important part.\n\nIn section 4.2, it is discussed which actions are switched to which ones due to adversary. For example, in the case of Pong, all of the actions are changed to 0, 2, or 3, which are “no op”, “right”, and “left”, respectively. I speculate that a better generalized policy (like A3C) instead of DDQN could still perform well since it has all the actions required to get the maximum return. Thus, the impact would not be this high (according to tables 1 and 2). In addition, in Pong, agent needs to press the “fire” button (action 1) to start each round of the game. Without loss of generality, this could be overwritten in an engineering matter (like what has been done in DeepMind Atari wrapper) so the agent still be able to perform without being concerned about the “fire” button. There is a lot to investigate and talk about in this paper.\n\n— minor issues:\nIn figure 1, there are two CrazyClimber plots. One of them needs to be changed.\n\nI believe the paper in its current form requires substantial changes. Due to the aforementioned reasons, I don't feel comfortable supporting this version of the paper. I think this paper could benefit from another round of editing pass. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ULPv5ro1NLa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors studied how perturbations on states would affect the performance of deep reinforcement learning. They defined different types of perturbations, like different perturbations for each state, or apply the perturbations on the initial state on every state. The authors tested these perturbations in some existing environments.\n- The conclusions of this submission are unclear and questionable. The authors showed many results in their submission, but all the conclusions are plausible. The results look pretty random, and I do not believe we can draw conclusions from them. For example, in table 2, the impacts of A_M^random are large, even comparable with A^individual. The A_M^random is more like a fixed random noise, so we cannot draw conclusions about transferability from it. It would be helpful to add two new baselines, one is fixed random noise, another is iid random noise, to demonstrate these adversaries are different from random noise.\n- Equations 1 and 2 do not make sense. For the equations wrote by the authors, J(s) is a constant, and we should always let s_adv equal to s.\n\n--Post Rebuttal--\nThank the authors for the response. I agree with other reviewers that the task is interesting and the submission has great potential, but might need another round of editing. The results show \"a single perturbation from a random state of a completely different MDP\" is not normally distributed. However, I agree with R4 that applying a single perturbation from a random state of another MDP does not make sense unless the two MDP and the two states have some similarities.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-erFiYVdN33", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary of Contributions\n\nThe paper explores adversarial perturbations in deep RL, providing a new thread model where the perturbation is computed based on a single state. The paper explores the impact of various types of perturbations between states in the same environment (state transferability), and between states in different environments (environment transferability).\n\n### Review\n\nI think the paper does a good job at capturing the extent/prevalence of the issue of adversarial perturbations. I liked the breadth in the types of perturbations considered, and how they map to scnarios that could happen in practice (e.g., restricted adversaries, etc.).\n\nOne thing that I think could improve the paper is teasing apart properties of environments or learning algorithms that are telling of the transferrability of the perturbations. For example, beyond suggesting that the offset pushes things beyond the decision boundary, checking things like how action gaps change may be insightful. Some of the action manipulation may further be exacerbated by things like using ε-greedy behavior policies which immediately snaps to highest-valued actions, in contrast with things like Boltzmann policies over action-values or policy gradient methods, which are smooth with respect to changes in them. Do the authors have any insight as to how the trends might change should a smooth policy be used?\n\nWhile the focus is on deep RL, perhaps a simple, motivating example (e.g., a little gridworld or Markov chain) could make a stronger/clearer case as to what exactly is happening, and suggest what situations one might expect it to be a more prevalent issue.\n\nBeyond this, I have the following questions/concerns:\n\n1) Is there a reason for the choice of DDQN, over say, regular DQN? While one algorithm may perform a bit better, if the emphasis of the paper is to measure the extent of adversarial perturbations, it seems like it may paint a clearer/more convincing picture if a simpler algorithm is used, with fewer moving parts to attribute performance differences to. Along these lines, I think it would be more convincing to carry out the same experiments with another deep RL algorithm (perhaps a policy gradient one, to be representative of both value-based and policy-based methods) to see if comparable levels of and trends in transferrability are observed there. Can the authors comment on whether \n\n2) How were hyper-parameters chosen for the DDQN agent, and can the authors comment on whether such choices can have reasonably strong interplay with the transferability of perturbations? For example, would larger/smaller learning rates, or deeper vs wider networks be more or less resilient to such perturbations? I think in any case such details need to be included in the paper, to be clearer that the transferability quantified is in the scope of a particular instantiation of a specific algorithm.\n\nOverall, I'm erring toward acceptance in that it outlines an interesting framework to study adversarial attacks in deep RL, and provides some early empirical results and intuitions around them. I do think the paper falls a little short in that there wasn't a representative sample of deep RL methods, as well as not commenting on design choices made and how/whether they might interplay with the transferability measured. I'm willing to adjust my score should my concerns be addressed.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work studies the transferability of perturbations/adversarial attacks on DRL agents. As a way to mitigate the cost of generating individual perturbation for each state in each episode, the authors proposed several variants to use same perturbation across different states across different episodes. While reviewers recognize the potential of the direction, they are not comfortable accepting the paper at its current state. The experiment results in its current form does not provide enough support to the claim. In particular, it is unclear how much the shared perturbation changes the original perception in comparison to the individual comparison, and how should the impact number differences be interpreted. Reviewers brought up concerns regarding all the experiments being evaluated on a DDQN agent, and not enough clarities has been provided on the different design choices. If perceptual similarity is not the indicator of environment transferability, do the authors have intuition on what does? 
Decision: Reject