Optimal Attacks on Reinforcement Learning Policies

1 INTRODUCTION . Advances in Deep Reinforcement Learning ( RL ) have made it possible to train end-to-end policies achieving superhuman performance on a large variety of tasks , such as playing Atari games Mnih et al . ( 2013 ; 2016 ) , playing Go Silver et al . ( 2016 ; 2017 ) , as well as controlling systems with continuous state and action spaces Lillicrap et al . ( 2016 ) ; Schulman et al . ( 2015 ) ; Levine et al . ( 2016 ) . Recently , some of these policies have been shown to be vulnerable to adversarial attacks at test time see e.g . Huang et al . ( 2017 ) ; Pattanaik et al . ( 2018 ) . Even if these attacks only introduce small perturbations to the successive inputs of the policies , they can significantly impact the average collected reward by the agent . So far , attacks on RL policies have been designed using heuristic techniques , based on gradient methods , essentially the Fast Gradient Sign Method ( FGSM ) . As such , they do not explicitly aim at minimizing the reward collected by the agent . In contrast , in this paper , we investigate the problem of casting optimal attacks with well-defined objectives , for example minimizing the average reward collected by the agent . We believe that casting optimal attacks is crucial when assessing the robustness of RL policies , since ideally , the agent should learn and apply policies that resist any possible attack ( of course with limited and reasonable amplitude ) . For a given policy learnt by the agent , we show that the problem of devising an optimal attack can be formulated as a Markov Decision Process ( MDP ) , defined through the system dynamics , the agent ’ s reward function and policy . We are mainly interested in black-box attacks : to devise them , the adversary only observes the variables from which she builds her reward . For example , if the objective is to lead the system to a certain set of ( bad ) states , the adversary may just observe the states as the system evolves . If the goal is to minimize the cumulative reward collected by the agent , the adversary may just observe the system states and the instantaneous rewards collected by the agent . In black-box attacks , the aforementioned MDP is unknown , and optimal attacks can be trained using RL algorithms . The action selected by the adversary in this MDP corresponds to a perturbed state to be fed to the agent ’ s policy . As a consequence , the action space is typically very large . To deal with this issue , our optimal attack is trained using DDPG Lillicrap et al . ( 2016 ) , a Deep RL algorithm initially tailored to continuous action space . We train and evaluate optimal attacks for some OpenAI Gym Brockman et al . ( 2016 ) environments with discrete action spaces ( discrete MountainCar , Cartpole , and Pong ) , and continuous state-action spaces ( continuous MountainCar and continuous LunarLander ) . As expected , optimal attacks outperform existing attacks . We discuss how the methods used to train RL policies impact their resilience to attacks . We show that the damages caused by attacks are upper bounded by a quantity directly related to the smoothness of the policy under attack1 . Hence , training methods such as DDPG leading to smooth policies should be more robust . Finally , we remark that when under attack , the agent faces an environment with uncertain state feedback , and her sequential decision problem can be modelled as a Partially Observable MDP ( POMDP ) . This suggests that training methods specifically tailored to POMDP ( e.g . DRQN Hausknecht & Stone ( 2015 ) ) result in more resistant policies . These observations are confirmed by our experiments . 2 RELATED WORK . Adversarial attacks on RL policies have received some attention over the last couple of years . As for now , attacks concern mainly the system at test time , where the agent has trained a policy using some Deep RL algorithm ( except for Behzadan & Munir ( 2017 ) , which considers an attack during training ) . The design of these attacks Huang et al . ( 2017 ) ; Pattanaik et al . ( 2018 ) ; Lin et al . ( 2017 ) are inspired by the Fast Gradient Sign Method ( FGSM ) Goodfellow et al . ( 2015 ) , which was originally used to fool Deep Learning classifiers . Fast Gradient Methods ( FGMs ) are gradient methods that changes the input with the aim of optimizing some objective function . This function , and its gradient , are sampled using observations : in case of an agent trained using Deep RL , this function can be hence related to a distribution of actions given a state ( the actor-network outputs a distribution over actions ) , or the Q-value function ( the critic outputs approximate Q-values ) . In Huang et al . ( 2017 ) , FGM is used to minimize the probability of the main agent selecting the optimal action according to the agent . On the other hand , in Pattanaik et al . ( 2018 ) , the authors develop a gradient method that maximizes the probability of taking the action with minimal Q-value . The aforementioned attacks are based on the unperturbed policy of the agent , or its Q-value function , and do not consider optimizing the attack over all possible trajectories . Therefore , they can be considered a first-order approximation of an optimal attack . If the adversary wishes to minimize the average cumulative reward of the agent , she needs to work on the perturbed agent policy ( to assess the impact of an attack , we need to compute the rewards after the attack ) . Proactive and reactive defence methods have been proposed to make RL policies robust to gradient attacks . Proactive methods are essentially similar to those in supervised learning , i.e. , they are based on injecting adversarial inputs during training Kos & Song ( 2017 ) ; Pattanaik et al . ( 2018 ) . A few reactive methods have also been proposed Lin et al . ( 2017 ) . There , the idea is to train a separate network predicting the next state . This network is then used to replace the state input by the predicted state if this input is believed to be too corrupted . But as we show in the paper , adversarial examples introduce partial observability , and the problem is not Markov anymore . It is known Singh et al . ( 1994 ) that deterministic memory-less policies are inadequate to deal with partial observability , and some kind of memory is needed to find a robust policy . In this work , we make use of recurrent neural networks to show that we can use techniques from the literature of Partially Observable MDPs to robustify RL policies . It is finally worth mentioning that there has been some work studying the problem of robust RL with a game-theoretical perspective Morimoto & Doya ( 2005 ) ; Pinto et al . ( 2017 ) . However , these papers 1A policy is smooth when the action it selects smoothly varies with the state . Refer to proposition 5.1 for details . consider a very different setting where the adversary has a direct impact on the system ( not indirectly through the agent as in our work ) . In a sense , they are more related to RL for multi-agent systems . 3 PRELIMINARIES . This section provides a brief technical background on Markov Decision Process and Deep Reinforcement Learning . The notions and methods introduced here are used extensively in the remainder of the paper . Markov Decision Process . A Markov Decision Process ( MDP ) defines a discrete-time controlled Markov chain . It is defined by a tupleM = 〈S , ( As , s ∈ S ) , P , p0 , q〉 . S is the finite state space , As is the finite set of actions available in state s , P represents the system dynamics where P ( ·|s , a ) is the distribution of the next state given that the current state is s and the selected action is a , p0 is the distribution of the initial state , and q characterizes rewards where q ( ·|s , a ) is the distribution of the collected reward in state is s when action a is selected . We denote by r ( s , a ) the expectation of this reward . We assume that the average reward is bounded : for all ( s , a ) , |r ( s , a ) | ≤ R. A policy π for M maps the state and to a distribution of the selected action . For a ∈ As , π ( a|s ) denotes the probability of choosing action a in state s. The value function V π of a policy π defines for every s the average cumulative discounted reward under π when the system starts in state s : V π ( s ) = E [ ∑ t≥0 γ tr ( sπt , a π t ) |s0 = s ] where sπt and aπt are the state and the selected action at time t under policy π , and γ ∈ ( 0 , 1 ) is the discount factor . The Q-value function Qπ of policy π maps a ( state , action ) pair ( s , a ) to the average cumulative discounted reward obtained starting in state s when the first selected action is a and subsequent actions are chosen according to π : Qπ ( s , a ) = r ( s , a ) + γEs′∼P ( ·|s , a ) [ V π ( s′ ) ] . For a given MDPM , the objective is to devise a policy with maximal value in every state . Deep Reinforcement Learning To train policies with maximal rewards for MPDs with large state or action spaces , Deep Reinforcement Learning consists in parametrizing the set of policies or some particular functions of interest such as the value function or the Q-value function of a given policy by a deep neural network . In this paper , we consider various Deep RL techniques to train either the policy of the agent or the attack . These include : DQN Mnih et al . ( 2013 ) : the network aims at approximating the Q-function of the optimal policy . DDPG Lillicrap et al . ( 2016 ) : an actor-critic method developed to deal with continuous state-action spaces . To this aim , the actor returns a single action rather than a distribution over actions . DRQN Hausknecht & Stone ( 2015 ) : introduces memory in DQN by replacing the ( post-convolutional ) layer by a recurrent LSTM . The use of this recurrent structure allows us to deal with partial observability , and DRQN works better with POMDPs . 4 OPTIMAL ADVERSARIAL ATTACKS . To attack a given policy π trained by the main agent , an adversary can slightly modify the sequence of inputs of this policy . The changes are imposed to be small , according to some distance , so that the attack remains difficult to detect . The adversary aims at optimizing inputs with a precise objective in mind , for example , to cause as much damage as possible to the agent . We denote byM = 〈S , ( As , s ∈ S ) , P , p0 , q〉 the MDP solved by the agent , i.e. , the agent policy π has been trained forM .

Summary: The paper investigates adversarial attacks on learned (fixed) policies. In particular, they devise optimal attacks in the sense that e.g. the agent’s objective should be minimised by the attacker. It is assumed that the attacker can manipulate the observed state of the agent at each time step during testing in a restricted way such that perturbations are small in the state or feature space. The paper derives attacker value functions when the attacker’s goal ist to minimise the original agent’s average cumulative reward and shows how to maximise them using gradient methods. In order to show when attacks are more likely to be successful, a theorem is presented and proved that shows that the resilience of a policy against attacks depends on its smoothness. Experiments show that a) attacks that are trained for optimality w.r.t. to minimising the average reward of the agent outperform a baseline method that only optimises a related criterion. b) Agent policies that are trained with DRQN, yielding a recurrent controller, outperform non recursive ones.