Optimal Attacks on Reinforcement Learning Policies

1 INTRODUCTION . Advances in Deep Reinforcement Learning ( RL ) have made it possible to train end-to-end policies achieving superhuman performance on a large variety of tasks , such as playing Atari games Mnih et al . ( 2013 ; 2016 ) , playing Go Silver et al . ( 2016 ; 2017 ) , as well as controlling systems with continuous state and action spaces Lillicrap et al . ( 2016 ) ; Schulman et al . ( 2015 ) ; Levine et al . ( 2016 ) . Recently , some of these policies have been shown to be vulnerable to adversarial attacks at test time see e.g . Huang et al . ( 2017 ) ; Pattanaik et al . ( 2018 ) . Even if these attacks only introduce small perturbations to the successive inputs of the policies , they can significantly impact the average collected reward by the agent . So far , attacks on RL policies have been designed using heuristic techniques , based on gradient methods , essentially the Fast Gradient Sign Method ( FGSM ) . As such , they do not explicitly aim at minimizing the reward collected by the agent . In contrast , in this paper , we investigate the problem of casting optimal attacks with well-defined objectives , for example minimizing the average reward collected by the agent . We believe that casting optimal attacks is crucial when assessing the robustness of RL policies , since ideally , the agent should learn and apply policies that resist any possible attack ( of course with limited and reasonable amplitude ) . For a given policy learnt by the agent , we show that the problem of devising an optimal attack can be formulated as a Markov Decision Process ( MDP ) , defined through the system dynamics , the agent ’ s reward function and policy . We are mainly interested in black-box attacks : to devise them , the adversary only observes the variables from which she builds her reward . For example , if the objective is to lead the system to a certain set of ( bad ) states , the adversary may just observe the states as the system evolves . If the goal is to minimize the cumulative reward collected by the agent , the adversary may just observe the system states and the instantaneous rewards collected by the agent . In black-box attacks , the aforementioned MDP is unknown , and optimal attacks can be trained using RL algorithms . The action selected by the adversary in this MDP corresponds to a perturbed state to be fed to the agent ’ s policy . As a consequence , the action space is typically very large . To deal with this issue , our optimal attack is trained using DDPG Lillicrap et al . ( 2016 ) , a Deep RL algorithm initially tailored to continuous action space . We train and evaluate optimal attacks for some OpenAI Gym Brockman et al . ( 2016 ) environments with discrete action spaces ( discrete MountainCar , Cartpole , and Pong ) , and continuous state-action spaces ( continuous MountainCar and continuous LunarLander ) . As expected , optimal attacks outperform existing attacks . We discuss how the methods used to train RL policies impact their resilience to attacks . We show that the damages caused by attacks are upper bounded by a quantity directly related to the smoothness of the policy under attack1 . Hence , training methods such as DDPG leading to smooth policies should be more robust . Finally , we remark that when under attack , the agent faces an environment with uncertain state feedback , and her sequential decision problem can be modelled as a Partially Observable MDP ( POMDP ) . This suggests that training methods specifically tailored to POMDP ( e.g . DRQN Hausknecht & Stone ( 2015 ) ) result in more resistant policies . These observations are confirmed by our experiments . 2 RELATED WORK . Adversarial attacks on RL policies have received some attention over the last couple of years . As for now , attacks concern mainly the system at test time , where the agent has trained a policy using some Deep RL algorithm ( except for Behzadan & Munir ( 2017 ) , which considers an attack during training ) . The design of these attacks Huang et al . ( 2017 ) ; Pattanaik et al . ( 2018 ) ; Lin et al . ( 2017 ) are inspired by the Fast Gradient Sign Method ( FGSM ) Goodfellow et al . ( 2015 ) , which was originally used to fool Deep Learning classifiers . Fast Gradient Methods ( FGMs ) are gradient methods that changes the input with the aim of optimizing some objective function . This function , and its gradient , are sampled using observations : in case of an agent trained using Deep RL , this function can be hence related to a distribution of actions given a state ( the actor-network outputs a distribution over actions ) , or the Q-value function ( the critic outputs approximate Q-values ) . In Huang et al . ( 2017 ) , FGM is used to minimize the probability of the main agent selecting the optimal action according to the agent . On the other hand , in Pattanaik et al . ( 2018 ) , the authors develop a gradient method that maximizes the probability of taking the action with minimal Q-value . The aforementioned attacks are based on the unperturbed policy of the agent , or its Q-value function , and do not consider optimizing the attack over all possible trajectories . Therefore , they can be considered a first-order approximation of an optimal attack . If the adversary wishes to minimize the average cumulative reward of the agent , she needs to work on the perturbed agent policy ( to assess the impact of an attack , we need to compute the rewards after the attack ) . Proactive and reactive defence methods have been proposed to make RL policies robust to gradient attacks . Proactive methods are essentially similar to those in supervised learning , i.e. , they are based on injecting adversarial inputs during training Kos & Song ( 2017 ) ; Pattanaik et al . ( 2018 ) . A few reactive methods have also been proposed Lin et al . ( 2017 ) . There , the idea is to train a separate network predicting the next state . This network is then used to replace the state input by the predicted state if this input is believed to be too corrupted . But as we show in the paper , adversarial examples introduce partial observability , and the problem is not Markov anymore . It is known Singh et al . ( 1994 ) that deterministic memory-less policies are inadequate to deal with partial observability , and some kind of memory is needed to find a robust policy . In this work , we make use of recurrent neural networks to show that we can use techniques from the literature of Partially Observable MDPs to robustify RL policies . It is finally worth mentioning that there has been some work studying the problem of robust RL with a game-theoretical perspective Morimoto & Doya ( 2005 ) ; Pinto et al . ( 2017 ) . However , these papers 1A policy is smooth when the action it selects smoothly varies with the state . Refer to proposition 5.1 for details . consider a very different setting where the adversary has a direct impact on the system ( not indirectly through the agent as in our work ) . In a sense , they are more related to RL for multi-agent systems . 3 PRELIMINARIES . This section provides a brief technical background on Markov Decision Process and Deep Reinforcement Learning . The notions and methods introduced here are used extensively in the remainder of the paper . Markov Decision Process . A Markov Decision Process ( MDP ) defines a discrete-time controlled Markov chain . It is defined by a tupleM = 〈S , ( As , s ∈ S ) , P , p0 , q〉 . S is the finite state space , As is the finite set of actions available in state s , P represents the system dynamics where P ( ·|s , a ) is the distribution of the next state given that the current state is s and the selected action is a , p0 is the distribution of the initial state , and q characterizes rewards where q ( ·|s , a ) is the distribution of the collected reward in state is s when action a is selected . We denote by r ( s , a ) the expectation of this reward . We assume that the average reward is bounded : for all ( s , a ) , |r ( s , a ) | ≤ R. A policy π for M maps the state and to a distribution of the selected action . For a ∈ As , π ( a|s ) denotes the probability of choosing action a in state s. The value function V π of a policy π defines for every s the average cumulative discounted reward under π when the system starts in state s : V π ( s ) = E [ ∑ t≥0 γ tr ( sπt , a π t ) |s0 = s ] where sπt and aπt are the state and the selected action at time t under policy π , and γ ∈ ( 0 , 1 ) is the discount factor . The Q-value function Qπ of policy π maps a ( state , action ) pair ( s , a ) to the average cumulative discounted reward obtained starting in state s when the first selected action is a and subsequent actions are chosen according to π : Qπ ( s , a ) = r ( s , a ) + γEs′∼P ( ·|s , a ) [ V π ( s′ ) ] . For a given MDPM , the objective is to devise a policy with maximal value in every state . Deep Reinforcement Learning To train policies with maximal rewards for MPDs with large state or action spaces , Deep Reinforcement Learning consists in parametrizing the set of policies or some particular functions of interest such as the value function or the Q-value function of a given policy by a deep neural network . In this paper , we consider various Deep RL techniques to train either the policy of the agent or the attack . These include : DQN Mnih et al . ( 2013 ) : the network aims at approximating the Q-function of the optimal policy . DDPG Lillicrap et al . ( 2016 ) : an actor-critic method developed to deal with continuous state-action spaces . To this aim , the actor returns a single action rather than a distribution over actions . DRQN Hausknecht & Stone ( 2015 ) : introduces memory in DQN by replacing the ( post-convolutional ) layer by a recurrent LSTM . The use of this recurrent structure allows us to deal with partial observability , and DRQN works better with POMDPs . 4 OPTIMAL ADVERSARIAL ATTACKS . To attack a given policy π trained by the main agent , an adversary can slightly modify the sequence of inputs of this policy . The changes are imposed to be small , according to some distance , so that the attack remains difficult to detect . The adversary aims at optimizing inputs with a precise objective in mind , for example , to cause as much damage as possible to the agent . We denote byM = 〈S , ( As , s ∈ S ) , P , p0 , q〉 the MDP solved by the agent , i.e. , the agent policy π has been trained forM .

Summary: The authors of this paper propose a novel adversarial attack for deep reinforcement learning. Different from the classical attacks, e.g., FGSM, they explicitly minimize the reward collect by the agent in a form of Markov decision process. Experiment results demonstrate that the proposed approach can damage the well-performed policy with a much bigger performance drop than gradient-based attacks.