Learning Key Steps to Attack Deep Reinforcement Learning Agents

1 INTRODUCTION . Reinforcement learning ( RL ) is a framework for sequential decision problems , where an agent interacts with an unknown environment and tries to maximize the total reward it receives . With the rapid development of deep learning , RL agents parametrized by neural networks , usually referred to as deep RL agents , are able to learn complex policies from raw inputs ( Mnih et al. , 2015 ) . Recently , deep RL agents have shown great success across various domains , such as achieving superhuman performance on games ( Mnih et al. , 2015 ; Silver et al. , 2016 ; 2018 ) , completing complex robotic tasks ( Levine et al. , 2016 ) , optimizing patient treatments ( Escandell-Montero et al. , 2014 ; Raghu et al. , 2017 ) , and developing autonomous driving skills ( Pan et al. , 2017 ; Isele et al. , 2018 ) . However , if we wish to deploy these RL agents into security-critical applications , we must take their reliability into consideration . It is discovered that neural network classifiers may make errors on deliberately crafted inputs , known as adversarial examples ( Biggio et al. , 2013 ; Szegedy et al. , 2013 ; Behzadan & Munir , 2017a ) . Deep RL is no exception : Many works have demonstrated that deep RL agents are also vulnerable when attacked by adversarial examples ( Behzadan & Munir , 2017a ; Huang et al. , 2017 ) , raising serious concerns on the reliability of these agents for security-critical applications . For example , there may be severe consequences if an RL-agent-driven autonomous vehicle is compromised by adversarial examples fed from a malicious attacker . Given the sequential nature of RL , it is not necessary to attack at every time step to degrade the agent ’ s performance significantly ( Lin et al. , 2017 ; Kos & Song , 2017 ) . The reason is that not all decisions made by the agent are equally important . Some decisions may be critical to the agent , such as those for long-term planning or immediate reward gathering ; some other decisions may not have much effect on the environment nor the rewards , and thus attacking those steps would not affects the agent ’ s performance . The critical steps will be named key steps in this work , and attacking these steps decreases the cumulative reward that the agent could gather . Figure 1 illustrates a key step and a non-key step in Atari Pong , where the main difference between them is whether the ball has already passed through the agent . If the attacker is aware of this difference , the attacker can concentrate on attacking the key steps . By attacking fewer steps , the attacker reduces the costs of generating , injecting , and hiding the adversarial examples . Therefore , an intelligent attacker should prefer attacking key steps over non-key steps . A natural question arises : How does an attacker identify the key steps to attack ? As an initial attempt to address this question , Lin et al . ( 2017 ) formulate the problem from an optimization perspective , and propose a heuristic that attacks the agent when it strongly prefers one action . Concurrently , Kos & Song ( 2017 ) propose another heuristic that attacks when the action appears rewarding to the agent . However , since these heuristics are designed based on human knowledge , it is unclear whether there exist more effective key steps . In this work , we study the possibilities of finding more effective key steps , and of finding them automatically without human knowledge . We confirm both possibilities by introducing an RL framework where the attacker learns the key steps from scratch through interacting with the agent . In particular , we study the key-step identification problem proposed by Lin et al . ( 2017 ) . We first make a crucial observation that this constrained optimization problem can be converted into another form that matches the objective of RL . Then , we design a corresponding RL environment to train the attacker . Our contributions are summarized as follows : • We formulate the key-step identification problem in an unconstrained form , and introduce an RL framework that solves it directly without incorporating human knowledge . The proposed RL framework is independent of how adversarial examples are generated , and thus could be combined with white-box or black-box attacks . • We justify our learning framework by giving a toy example where those human-designed heuristics fail to identify the key steps . • To test the effectiveness of our framework , we conduct experiments on four benchmark Atari games . The results suggest that the attacker trained by our framework learns effective key steps and has the potential to outperform prior methods . 2 BACKGROUND AND RELATED WORK . 2.1 REINFORCEMENT LEARNING BACKGROUND . We begin with an introduction to the Markov Decision Process ( MDP ) ( Sutton & Barto , 2018 ) . For a set S , we use ∆ ( S ) to denote the set of probability distributions over S. Consider an agent interacting with an environment : At each step t , the agent picks an action at based on a current state st , and then the environment returns a reward rt and a next state st+1 . We formalize this interaction as an MDP M , which is a 5-tuple ( S , A , P , R , γ ) , where S is a finite set of states , A is a finite set of actions , P : S × A → ∆ ( S ) defines the transition dynamics from st to st+1 , R : S × A → ∆ ( R ) defines the reward distribution , and γ ∈ [ 0 , 1 ] is a discount factor . For convenience , we use r : ( st , at ) 7→ Ert∼R ( st , at ) [ rt ] to denote the mean reward function . We specify an agent by its ( stochastic ) policy π : S→ ∆ ( A ) , and define its Q-value function as Qπ ( st , at ) = E [ ∞∑ k=0 γkr ( st+k , at+k ) ] , where si ∼ P ( si−1 , ai−1 ) and ai ∼ π ( si ) for all i > t. The agent ’ s goal is to find a policy that maximizes the expected cumulative reward , or expected return , defined by Qπ ( s1 , π ( s1 ) ) , starting from an initial state s1 . For finite-horizontal MDPs , we use T to denote the maximum number of steps , and define an episode to be a sample of ( s1 , a1 , r1 , . . . , sT , aT , rT , sT+1 ) . To learn the optimal policy , deep Q-network ( DQN , Mnih et al . 2015 ) first approximates the Q-value function by a neural network that takes a state as input and outputs the Q-values for all actions . The policy is then induced by picking the action with the largest Q-value at each state . Given a transition step ( st , at , rt , st+1 ) , the network fθ is trained by minimizing the square loss Lθ = ( ( rt + γmax a fθ ( st+1 ) a ) − fθ ( st ) at ) 2 , where we use fθ ( s ) a to denote the a-th entry of the Q-values fθ ( s ) . 2.2 ADVERSARIAL ATTACK BACKGROUND . In the context of adversarial attacks , an attacker tries to craft adversarial examples that the target model would misclassify . In gradient-based adversarial attacks , the attacker generates adversarial examples by adding small perturbations to data examples , such that the perturbations are imperceptible to human eyes . Take the fast gradient sign method ( FGSM ) by Goodfellow et al . ( 2014 ) for example . Given a target model fθ parametrized by θ and an image–label pair ( x , y ) , the FGSM computes the perturbation as η = sign ( ∇xJ ( θ , x , y ) ) , where is a scaling factor that controls the norm of the perturbation , and J ( θ , x , y ) is an objective function that depends on the attacker ’ s goal . In untargetted attacks , the attacker aims to minimize the target model ’ s classification accuracy . Thus , the objective J ( θ , x , y ) is substituted with DKL ( e ( y ) ‖fθ ( x ) ) , the Kullback-Leibler divergence between the label ’ s one-hot encoding e ( y ) and the predicted probabilities fθ ( x ) . However , this type of white-box attacks requires full knowledge of the network parameter θ , which may be hidden in practical tasks . In contrast , black-box attacks assume no access to θ and study other conditions , such as the ability to query model outputs ( Chen et al. , 2017 ) . One black-box attack of particular interest is the substitute model approach ( Papernot et al. , 2016a ; 2017 ) , where the attacker computes perturbation using a substitute model that is trained to perform the same task as the target model . This approach is based on the finding that adversarial examples are transferable ; that is , if an adversarial example fools a model , it usually fools another models trained to perform the same task , regardless of the network architecture ( Szegedy et al. , 2013 ) . Consequently , once the attacker gains access to the training set , black-box attacks reduce to white-box attacks with a lower attack success rate . 2.3 ADVERSARIAL ATTACK ON DEEP RL AGENTS . As opposed to traditional adversarial attacks , attacking an RL agent is a unique and challenging task . When we attack a classifier , we typically aim to minimize the classification accuracy or to maximize the probability that the classifier predicts some given class . When we attack an RL agent , however , we usually do not care about the individual actions that the agent picks . In this paper , we choose to minimize the agent ’ s cumulative reward ( Huang et al. , 2017 ; Lin et al. , 2017 ) as the attacker ’ s goal . Other possible goals include luring the agent into a designated state ( Lin et al. , 2017 ) , or misguiding the agent to optimize an adversarial reward ( Tretschk et al. , 2018 ) . Previous works have investigated the effects of white-box and black-box attacks against RL agents on the Atari benchmark . Given full knowledge of the target agent ’ s network parameters , an attacker can craft adversarial examples to fool the target agent into picking wrong actions ( Behzadan & Munir , 2017a ; Huang et al. , 2017 ; Kos & Song , 2017 ; Mandlekar et al. , 2017 ; Pattanaik et al. , 2018 ; Hussenot et al. , 2019 ) using the FGSM . Other white-box methods such as the Jacobian saliency map algorithm ( Papernot et al. , 2016b ) and the attack proposed by Carlini & Wagner ( 2017 ) are also effective against RL agents ( Behzadan & Munir , 2017a ; Lin et al. , 2017 ) . Even if the attacker can not access the target agent ’ s network parameters , adversarial examples can still be crafted with a substitute agent that is trained in the same environment as the target agent ( Behzadan & Munir , 2017a ; Huang et al. , 2017 ) , similar to the substitute model approach . Another line of studies tries to improve the RL agent ’ s robustness with adversarial attacks . Several works improve the agent ’ s robustness to visual perturbations through adversarial training ( Kos & Song , 2017 ; Mandlekar et al. , 2017 ; Behzadan & Munir , 2017b ; Pattanaik et al. , 2018 ) , which is a defense technique that adds adversarial examples into training sets ( Goodfellow et al. , 2014 ) . On the other hand , Pinto et al . ( 2017 ) propose to improve the agent ’ s robustess to environment changes by training the agent and an adversary in an alternating procedure , where the attacks are operated on the environment dynamics . Tessler et al . ( 2019 ) study the agent ’ s robustness to action-space perturbations in continuous control domains , and achieve better performance even in the absence of the adversary .

Summary: This paper proposes to learn the ‘key-steps’ at which to to apply an adversarial attack on a reinforcement learning agent. The framing of this problem is a Lagrangian relaxation of a constrained minimization problem which takes the form of an RL problem itself, where the attacking agent’s reward is the negative reward of the target agents plus a penalty (lambda, hyper-parameter) for choosing to attack. The attacking agent’s action space is binary, attack or no attack.