Toward Evaluating Robustness of Deep Reinforcement Learning with Continuous Control

1 INTRODUCTION . Deep reinforcement learning ( RL ) has revolutionized the fields of AI and machine learning over the last decade . The introduction of deep learning has achieved unprecedented success in solving many problems that were intractable in the field of RL , such as playing Atari games from pixels and performing robotic control tasks ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ; Tassa et al. , 2018 ) . Unfortunately , similar to the case of deep neural network classifiers with adversarial examples , recent studies show that deep RL agents are also vulnerable to adversarial attacks . A commonly-used threat model allows the adversary to manipulate the agent ’ s observations at every time step , where the goal of the adversary is to decrease the agent ’ s total accumulated reward . As a pioneering work in this field , ( Huang et al. , 2017 ) show that by leveraging the FGSM attack on each time frame , an agent ’ s average reward can be significantly decreased with small input adversarial perturbations in five Atari games . ( Lin et al. , 2017 ) further improve the efficiency of the attack in ( Huang et al. , 2017 ) by leveraging heuristics of detecting a good time to attack and luring agents to bad states with sample-based Monte-Carlo planning on a trained generative video prediction model . Since the agents have discrete actions in Atari games ( Huang et al. , 2017 ; Lin et al. , 2017 ) , the problem of attacking Atari agents often reduces to the problem of finding adversarial examples on image classifiers , also pointed out in ( Huang et al. , 2017 ) , where the adversaries intend to craft the input perturbations that would drive agent ’ s new action to deviate from its nominal action . However , for agents with continuous actions , the above strategies can not be directly applied . Recently , ( Uesato et al. , 2018 ) studied the problem of adversarial testing for continuous control domains in a similar but slightly different setting . Their goal was to efficiently and effectively find catastrophic failure given a trained agent and to predict its failure probability . The key to success in ( Uesato et al. , 2018 ) is the availability of agent training history . However , such information may not always be accessible to the users , analysts , and adversaries . Besides , although it may not be surprising that adversarial attacks exist for the deep RL agents as adversarial attacks have been shown to be possible for neural network models in various supervised ∗Work done during summer internship at DeepMind , UK . ♣ Equal contributions . learning tasks . However , the vulnerability of RL agents can not be easily discovered by existing baselines which are model-free and build upon random searches and heuristics – this is also verified by our extensive experiments on various domains ( e.g . walker , humanoid , cartpole , and fish ) , where the agents still achieve close to their original best rewards even with baseline attacks at every time step . Hence it is important and necessary to have a systematic methodology to design non-trivial adversarial attacks , which can efficiently and effectively discover the vulnerabilities of deep RL agents – this is indeed the motivation of this work . This paper takes a first step toward this direction by proposing the first sample-efficient model-based adversarial attack . Specifically , we study the robustness of deep RL agents in a more challenging setting where the agent has continuous actions and its training history is not available . We consider the threat models where the adversary is allowed to manipulate an agent ’ s observations or actions with small perturbations , and we propose a two-step algorithmic framework to find efficient adversarial attacks based on learned dynamics models . Experimental results show that our proposed modelbased attack can successfully degrade agent performance and is also more effective and efficient than model-free attacks baselines . The contributions of this paper are the following : • To the best of our knowledge , we propose the first model-based attack on deep RL agents with continuous actions . Our proposed attack algorithm is a general two-step algorithm and can be directly applied to the two commonly-used threat models ( observation manipulation and action manipulation ) . • We study the efficiency and effectiveness of our proposed model-based attack with modelfree attack baselines based on random searches and heuristics . We show that our modelbased attack can degrade agent performance in numerous MuJoCo domains by up to 4× in terms of total reward and up to 4.6× in terms of distance to unsafe states ( smaller means stronger attacks ) compared to the model-free baselines . • Our proposed model-based attack also outperform all the baselines by a large margin in a weaker adversary setting where the adversary can not attack at every time step . In addition , ablation study on the effect of planning length in our proposed technique suggests that our method can still be effective even when the learned dynamics model is not very accurate . 2 BACKGROUND . Adversarial attacks in reinforcement learning . Compared to the rich literature of adversarial examples in image classifications ( Szegedy et al. , 2013 ) and other applications ( including natural language processing ( Jia & Liang , 2017 ) , speech ( Carlini & Wagner , 2018 ) , etc ) , there is relatively little prior work studying adversarial examples in deep RL . One of the first several works in this field are ( Huang et al. , 2017 ) and ( Lin et al. , 2017 ) , where both works focus on deep RL agent in Atari games with pixels-based inputs and discrete actions . In addition , both works assume the agent to be attacked has accurate policy and the problem of finding adversarial perturbation of visual input reduces to the same problem of finding adversarial examples on image classifiers . Hence , ( Huang et al. , 2017 ) applied FGSM ( Goodfellow et al. , 2015 ) to find adversarial perturbations and ( Lin et al. , 2017 ) further improved the efficiency of the attack by heuristics of observing a good timing to attack – when there is a large gap in agents action preference between most-likely and leastlikely action . In a similar direction , ( Uesato et al. , 2018 ) study the problem of adversarial testing by leveraging rejection sampling and the agent training histories . With the availability of training histories , ( Uesato et al. , 2018 ) successfully uncover bad initial states with much fewer samples compared to conventional Monte-Carlo sampling techniques . Recent work by ( Gleave et al. , 2019 ) consider an alternative setting where the agent is attacked by another agent ( known as adversarial policy ) , which is different from the two threat models considered in this paper . Finally , besides adversarial attacks in deep RL , a recent work ( Wang et al. , 2019 ) study verification of deep RL agent under attacks , which is beyond the scope of this paper . Learning dynamics models . Model-based RL methods first acquire a predictive model of the environment dynamics , and then use that model to make decisions ( Atkeson & Santamaria , 1997 ) . These model-based methods tend to be more sample efficient than their model-free counterparts , and the learned dynamics models can be useful across different tasks . Various works have focused on the most effective ways to learn and utilize dynamics models for planning in RL ( Kurutach et al. , 2018 ; Chua et al. , 2018 ; Chiappa et al. , 2017 ; Fu et al. , 2016 ) . 3 PROPOSED FRAMEWORK . In this section , we first describe the problem setup and the two threat models considered in this paper . Next , we present an algorithmic framework to rigorously design adversarial attacks on deep RL agents with continuous actions . 3.1 PROBLEM SETUP AND FORMULATION . Let si ∈ RN and ai ∈ RM be the observation vector and action vector at time step i , and let π : RN → RM be the deterministic policy ( agent ) . Let f : RN × RM → RN be the dynamics model of the system ( environment ) which takes current state-action pair ( si , ai ) as inputs and outputs the next state si+1 . We are now in the role of an adversary , and as an adversary , our goal is to drive the agent to the ( un-safe ) target states starget within the budget constraints . We can formulate this goal into two optimization problems , as we will illustrate shortly below . Within this formalism , we can consider two threat models : Threat model ( i ) : Observation manipulation . For the threat model of observation manipulation , an adversary is allowed to manipulate the observation si that the agent perceived within an budget : ‖∆si‖∞ ≤ , Ls ≤ si + ∆si ≤ Us , ( 1 ) where ∆si ∈ RN is the crafted perturbation and Us ∈ RN , Ls ∈ RN are the observation limits . Threat model ( ii ) : Action manipulation . For the threat model of action manipulation , an adversary can craft ∆ai ∈ RM such that ‖∆ai‖∞ ≤ , La ≤ ai + ∆ai ≤ Ua , ( 2 ) where Ua ∈ RM , La ∈ RM are the limits of agent ’ s actions . Our formulations . Given an initial state s0 and a pre-trained policy π , our ( adversary ) objective is to minimize the total distance of each state si to the pre-defined target state starget up to the unrolled ( planning ) steps T . This can be written as the following optimization problems in Equations 3 and 4 for the Threat model ( i ) and ( ii ) respectively : min ∆si T∑ i=1 d ( si , starget ) s.t . ai = π ( si + ∆si ) , si+1 = f ( si , ai ) , Constraint ( 1 ) , i ∈ ZT , ( 3 ) min ∆ai T∑ i=1 d ( si , starget ) s.t . ai = π ( si ) , si+1 = f ( si , ai + ∆ai ) , Constraint ( 2 ) , i ∈ ZT . ( 4 ) A common choice of d ( x , y ) is the squared ` 2 distance ‖x − y‖22 and f is the learned dynamics model of the system , and T is the unrolled ( planning ) length using the dynamics models . 3.2 OUR ALGORITHM . In this section , we propose a two-step algorithm to solve Equations 3 and 4 . The core of our proposal consists of two important steps : learn a dynamics model f of the environment and deploy optimization technique to solve Equations 3 and 4 . We first discuss the details of each factor , and then present the full algorithm by the end of this section . Step 1 : learn a good dynamics model f . Ideally , if f is the exact ( perfect ) dynamics model of the environment and assuming we have an optimization oracle to solve Equations 3 and 4 , then the solutions are indeed the optimal adversarial perturbations that give the minimal total loss with -budget constraints . Thus , learning a good dynamics model can conceptually help on developing a strong attack . Depending on the environment , different forms of f can be applied . For example , if the environment of concerned is close to a linear system , then we could let f ( s , a ) = As+Ba , where A and B are unknown matrices to be learned from the sample trajectories ( si , ai , si+1 ) pairs . For a more complex environment , we could decide if we still want to use a simple linear model ( the next state prediction may be far deviate from the true next state and thus the learned dynamical model is less useful ) or instead switch to a non-linear model , e.g . neural networks , which usually has better prediction power but may require more training samples . For either case , the model parameters A , B or neural network parameters can be learned via standard supervised learning with the sample trajectories pairs ( si , ai , si+1 ) . Step 2 : solve Equations 3 and 4 . Once we learned a dynamical model f , the next immediate task is to solve Equation 3 and 4 to compute the adversarial perturbations of observations/actions . When the planning ( unrolled ) length T > 1 , Equation 3 usually can not be directly solved by off-theshelf convex optimization toolbox since the deel RL policy π is usually a non-linear and non-convex neural network . Fortunately , we can incorporate the two equality constraints of Equation 3 into the objective and with the remaining -budget constraint ( Equation 1 ) , Equation 3 can be solved via projected gradient descent ( PGD ) 1 . Similarly , Equation 4 can be solved via PGD to get ∆ai . We note that , similar to the n-step model predictive control , our algorithm could use a much larger planning ( unrolled ) length T when solving Equations 3 and 4 and then only apply the first n ( ≤ T ) adversarial perturbations on the agent over n time steps . Besides , with the PGD framework , f is not limited to feed-forward neural networks . Our proposed attack is summarized in Algorithm 2 for Step 1 , and Algorithm 3 for Step 2 . Algorithm 1 Collect trajectories 1 : Input : pre-trained policy π , MaxSampleSize ns , environment env 2 : Output : a set of trajectory pairs S 3 : k ← 0 , S ← φ 4 : s0 ← env.reset ( ) 5 : while k < ns do 6 : ak ← π ( sk ) 7 : sk+1 ← env.step ( ak ) 8 : S ∪ { ( sk , ak , sk+1 ) } 9 : k ← k + 1 10 : end while 11 : Return S

Summary:   This paper looks at a new framework for adversarial attacks on deep reinforcement learning agents under continuous action spaces. They propose a model based approach which adds noise to either the observation or actions of the agent to push the agent to predefined target states. They then report results against several model-free/unlearned baselines on MuJoCo tasks using a policy learned through D4PG.