Characterizing Attacks on Deep Reinforcement Learning

1 INTRODUCTION . In recent years , deep neural networks ( DNNs ) have become pervasive and led a trend of fast adoption in various commercial systems performing image recognition ( Krizhevsky et al. , 2012 ) , speech recognition ( Hannun et al. , 2014 ) , and natural language processing ( Sutskever et al. , 2014 ) . DNNs have also encouraged increased success in the field of deep reinforcement learning ( DRL ) , where the goal is to train an agent to interact with the environments for maximizing an expected return . DRL systems have been evaluated on games ( Ghory , 2004 ; Mnih et al. , 2013 ; 2016 ) , autonomous navigation ( Dai et al. , 2005 ) , and robotics control ( Levine et al. , 2016 ) , etc . To take advantage of this , industries are integrating DRL into production systems ( RE WORK , 2017 ) . However , it is well-known that DNNs are vulnerable to adversarial perturbations ( Goodfellow et al. , 2015 ; Li & Vorobeychik , 2014 ; 2015 ) . DRL systems that use DNNs to perform perception and policy making also have similar vulnerabilities . For example , one of the main weaknesses of RL models in adversarial environments is their heavy dependence on the input observations . Since RL models are trained to solve sequential decision-making problems , an attacker can perturb multiple observations . In fact , the distribution of training and testing data could be different due to random noise and adversarial manipulation ( Laskov & Lippmann , 2010 ) . Therefore , the learned policy can be vulnerable in adversarial environments . In this paper , we first present an extensive study of adversarial attacks on DRL systems . Second , we propose and evaluate 10 adversarial attacks in order to explore points in the taxonomy that have not previously been examined in the literature . We organize adversarial attacks on DRL into a taxonomy based on details of the victim model and other properties of the attacker . First , we categorize these attacks based on what component of the system the attacker is capable of perturbing . The organization of this categorization resembles the components of a Markov decision process ( MDP ) : we recognize attacks that perturb an agent ’ s observations , actions , or the system ’ s environment dynamics . We summarize these categories in Figure 1 . Second , the attacker ’ s knowledge . We categorize these attacks based on what knowledge the attacker needs to perform the attack . Broadly , this breaks attacks down into the already recognized white-box attacks , where the attacker has full knowledge of the target DRL system , and black-box attacks , where the attacker has less or no knowledge . We will discuss this taxonomy further in Section 3 . On the other hand , existing attacks that perturb the observation operate independently on each frame , which are too computational intensive to run in real-time . We propose two novel strategies for quickly creating adversarial perturbations to use in real-time attacks . The first strategy , N-attack , trains a neural network to generate a perturbation , reducing the computation to a single forward pass over this network . The second strategy exploits the property that , in RL environments , the states are not independent , and later states depends on previous state and action . Therefore , we propose online sequential attacks , which , in contrast to attacks that operate independently on each frame , generate a perturbation using information from a few frames and then apply the generated perturbation to later frames . We include our experiments with these strategies as part of our exploration of attacks that perturb the observation . We describe our attacks in detail in Section 4 , and we present our evaluation in Section 5 . To summarize , our contributions are : ( 1 ) We systematically organize adversarial attacks on DRL systems into a taxonomy and devise and evaluate 10 new attacks on several DRL environments ; ( 2 ) We propose two practical strategies , N-attack and online sequential attack , for performing real-time attacks on DRL systems ; ( 3 ) We propose the attack that adversarially perturbs a DRL system ’ s environment dynamics , and the attack that perturbs a DRL system ’ s actions ; ( 4 ) We apply finite difference method to estimate gradient and perform black-box attack , and we propose an adaptive sampling based method to improve the gradient estimation efficiency ; ( 5 ) We propose to select the optimal frames within consecutive frames to attack and achieve optimal attack efficiency . We also provide corresponding mathematical analysis . 2 RELATED WORK . Adversarial attacks on machine learning models . Our attacks draw some of their techniques from previously proposed attacks . ( Goodfellow et al. , 2015 ) describes the fast gradient sign method ( FGSM ) of generating adversarial perturbations in a white-box setting . ( Carlini & Wagner , 2017 ) describes additional methods based on optimization , which result in smaller perturbations . ( MoosaviDezfooli et al. , 2017 ) demonstrates a way to generate a “ universal ” perturbation that is effective on a set of multiple inputs . ( Evtimov et al. , 2018 ) shows that adversarial examples can be robust to natural lighting conditions and viewing angles . In our black-box attacks , we apply techniques that have been proposed for adapting white-box methods to black-box scenarios ( Papernot et al. , 2017 ; Chen et al. , 2017 ) . Recently , ( Huang et al. , 2017 ) demonstrates an attack that uses FGSM to perturb observation frames in a DRL setting . They experiment with a white-box attack and a black-box attack based on transferability . In this work , we propose novel black-box attacks , including the attacks that do not rely on transferability . In addition , we propose several ways to reduce the computational complexity of attacks . ( Lin et al. , 2017 ) designs an algorithm to achieve targeted attack for DRL models . However , their work only considers targeted attack and requires training a generative model to predict future states , which is already a computational intensive task . ( Behzadan & Munir , 2017 ) proposes a black-box attack method that trains another DQN network to minimize the expected return while still using FGSM as the attack method . Robust RL via adversarial training . Safety in various robotics and autonomous driving applications has drawn lots of attention for training robust models . Knowing how RL models can be attacked is beneficial for training robust RL agent . ( Pinto et al. , 2017 ) proposes to train a RL agent to provide adversarial attack during training so that the agent can be robust against dynamics variations . However , since they manually selected the perturbations on environment dynamics , the attack provided in their work may not be able to generalize to broader RL systems . Additionally , their method relies on an accurate modeling of the environment dynamics , which may not be available for real world tasks such as robotics and autonomous driving systems . 3 TAXONOMY OF ATTACKS IN DRL . Existing work on attacking DRL systems with adversarial perturbations focuses on perturbing an agent ’ s observations . This is the most appealing place to start , with seminal results already suggesting that recognition systems are vulnerable to adversarial examples ( Goodfellow et al. , 2015 ; MoosaviDezfooli et al. , 2017 ) . It naturally follows that we should ask whether perturbations introduced in other places in a RL system can cause the agent to misbehave , and in what scenarios , taking into account ( i ) can the attacker perform the attack with limited knowledge about the agent , ( ii ) can the attacker perform the attack in real time , and ( iii ) can the attacker introduce the perturbation physically . To systematically explore this question , we propose a taxonomy of possible adversarial attacks . Attack components . In the first layer of our proposed taxonomy , we divide attacks based on what components in an MDP the attacker chooses to perturb : the agent ’ s observations , actions , and environment dynamics . We will discuss some of the scenarios where attacks on these components can be practical . For attacks applied on the observation space , the pixel values of images can be changed by installing some virus into the software that is used to process captured photos from the sensor or in the simulator that is rendering the environment . In case images are transmitted between robots and computers , some communications can be altered by an attacker wirelessly ( Lonzetta et al. , 2018 ) . Some physical observation based attacks have been analyzed in autonomous driving ( Evtimov et al. , 2018 ) . For attacks applied on the action space , the action outputs can be modified by installing some hardware virus in the actuator executing the action . This can be realistic in some robotic control tasks where the control center sends some control signals to the actuator , a vulnerability in the implementation , for example , vulnerability in the bluetooth signal transmission , may allow an attacker to modify those signals ( Lonzetta et al. , 2018 ) . For attacks applied on the environment dynamics , in the autonomous driving case we can change the material surface characteristic of the road such that the policy trained in one environment will fail in the perturbed environment ; in the robotic control case , the robot ’ s mass distribution can be changed such that the robot may lose balance when executing its original policy because the robot hasn ’ t been trained in this case . Attacker ’ s knowledge . In the second layer of our proposed taxonomy , we categorize attacks based on what information the attacker needs to perform the attack . This divides attacks into white-box attacks and black-box attacks . We make a further categorization based on the attacker ’ s knowledge about the policy network ’ s architecture , weight parameters and whether the attacker can query the network . In white-box attacks , the agent has access to the architecture and weight parameters of the policy network and of course can query the network . In black-box attacks , the attackers don ’ t have access to weight parameters of the policy network and may or may not have access to the policy network ’ s architecture . The attacker may or may not have access to query the policy network . Further categorization . We consider these additional properties of attacks . Real-time : while some attacks require more computation than can be performed in real-time , some are fast enough to run . Still other attacks perform some precomputation and then are able to generate perturbations quickly for each step . We identify this pragmatic property as part of our taxonomy . Physical : for RL tasks that take place in the real world , this property concerns the feasibility of physically applying the perturbation on the environment . Temporal dependency : we distinguish between attacks that generate a perturbation in each frame independently from other frames and online sequential attacks that use information about from previous frames to generate perturbations on later frames . 4 ADVERSARIAL ATTACKS ON REINFORCEMENT LEARNING POLICIES . In order to study the unexplored parts of our proposed taxonomy from Section 3 , in this section we develop several concrete attacks . 4.1 ATTACKS ON STATE OBSERVATIONS . We now describe attacks that perturb an agent ’ s state observations . In this category of attacks , the attacker changes the input state observation s to s̃ = s + h ( s ; w ) , where the attacker generates perturbation h ( s ; w ) from the original observation s and some learned parameters w. In order to ensure that perturbations are small , we require that ||h ( s ; w ) ||∞ ≤ , which we can enforce by choosing h to be of the form tanh ( · ) . We present both white-box attacks and black-box attacks .

Summary: The authors design a new taxonomy of attacks on deep RL agents - they developed three classes of attacks - attacks the modify the observation given to the agent, attacks that modify the action used by the agent and attacks that change the dynamics of the environment. In settings that have been studied previously, the authors show that they can find attacks more effectively than previous approaches can. They also study learning based and online attack generation approaches, that can be effectively used to quickly find adversarial attacks on the agent. The authors validate their approaches experimentally on Mujoco tasks and the TORCS driving simulator.