Bounded Myopic Adversaries for Deep Reinforcement Learning Agents

Adversarial attacks against deep neural networks have been widely studied . Adversarial examples for deep reinforcement learning ( DeepRL ) have significant security implications , due to the deployment of these algorithms in many application domains . In this work we formalize an optimal myopic adversary for deep reinforcement learning agents . Our adversary attempts to find a bounded perturbation of the state which minimizes the value of the action taken by the agent . We show with experiments in various games in the Atari environment that our attack formulation achieves significantly larger impact as compared to the current state-of-the-art . Furthermore , this enables us to lower the bounds by several orders of magnitude on the perturbation needed to efficiently achieve significant impacts on DeepRL agents . 1 INTRODUCTION . Deep Neural Networks ( DNN ) have become a powerful tool and currently DNNs are widely used in speech recognition ( Hannun et al. , 2014 ) , computer vision ( Krizhevsky et al. , 2012 ) , natural language processing ( Sutskever et al. , 2014 ) , and self learning systems as deep reinforcement learning agents ( Mnih et al . ( 2015 ) , Mnih et al . ( 2016 ) , Schulman et al . ( 2015 ) , Lillicrap et al . ( 2015 ) ) . Along with the overwhelming success of DNNs in various domains there has also been a line of research investigating their weaknesses . Szegedy et al . ( 2014 ) observed that adding imperceptible perturbations to images can lead a DNN to misclassify the input image . The authors argue that the existence of these so called adversarial examples is a form of overfitting . In particular , they hypothesize that a very complicated neural network behaves well on the training set , but nonetheless , performs poorly on the testing set enabling exploitation by the attacker . However , they discovered different DNN models were misclassifying the same adversarial examples and assigning them the same class instead of making random mistakes . This led Goodfellow et al . ( 2015 ) to propose that the DNN models were actually learning approximately linear functions resulting in underfitting the data . Recent work by Mnih et al . ( 2015 ) introduced the use of DNNs as function approximators in reinforcement learning , improving the state of the art in this area . Because these deep reinforcement learning agents utilize DNNs , they are also susceptible to this type of adversarial examples . Currently , deep reinforcement learning has been applied to many areas such as network system control ( Jay et al . ( 2019 ) , Chu et al . ( 2020 ) , Chinchali et al . ( 2018 ) ) , financial trading Noonan ( 2017 ) , blockchain protocol security Hou et al . ( 2019 ) , grid operation and security ( Duan et al . ( 2019 ) , Huang et al . ( 2019 ) ) , cloud computing Chen et al . ( 2018 ) , robotics ( Gu et al . ( 2017 ) , Kalashnikov et al . ( 2018 ) ) , autonomous driving Dosovitsky et al . ( 2017 ) , and medical treatment and diagnosis ( Tseng et al . ( 2017 ) , Popova et al . ( 2018 ) , Thananjeyan et al . ( 2017 ) , Daochang & Jiang ( 2018 ) , Ghesu et al . ( 2017 ) ) . A more particular scenario where adversarial perturbations might be of significant interest is a financial trading market where the DeepRL agent is trained on observations consisting of the order book . In such a setting it is possible to compromise the whole trading system with an extremely small subset of adversaries . In particular , the ` 1-norm bounded perturbations dicussed in our paper have sparse solutions , and thus can be used as a basis for an attack in such a scenario . Moreover , the magnitude of the ` 1-norm bounded perturbations produced by our attack is orders of magnitude smaller than previous approaches , and thus our proposed perturbations result in a stealth attack more likely to evade automatic anomaly detection schemes . Considering the wide spectrum of deep reinforcement learning algorithm deployment it is crucial to investigate the resilience of these algorithms before they are used in real world application domains . Moreover , adversarial formulations are a first step to understand these algorithms and build generalizable , reliable and robust deep reinforcement learning agents . Therefore , in this paper we study adversarial attack formulations for deep reinforcement learning agents and make the following contributions : • We define the optimal myopic adversary , whose aim is to minimize the value of the action taken by the agent in each state , and formulate the optimization problem that this adversary seeks to solve . • We introduce a differentiable approximation for the optimal myopic adversarial formulation . • We compare the impact results of our attack formulation to previous formulations in different games in the Atari environment . • We show that the new formulation finds a better direction for the adversarial perturbation and increases the attack impact for bounded perturbations . ( Conversely , our formulation decreases the magnitude of the pertubation required to efficiently achieve a significant impact . ) 2 RELATED WORK AND BACKGROUND . 2.1 ADVERSARIAL REINFORCEMENT LEARNING . Adversarial reinforcement learning is an active line of research directed towards discovering the weaknesses of deep reinforcement learning algorithms . Gleave et al . ( 2020 ) model the interaction between the agent and the adversary as a two player Markov game and solve the reinforcement learning problem for the adversary via Proximal Policy Optimization introduced by Schulman et al . ( 2017 ) . They fix the victim agent ’ s policy and only allow the adversary to take natural actions to disrupt the agent instead of using ` p-norm bound pixel perturbations . Pinto et al . ( 2017 ) model the adversary and the victim as a two player zero-sum discounted Markov game and train the victim in the presence of the adversary to make the victim more robust . Mandlekar et al . ( 2017 ) use a gradient based perturbation to make the agent more robust as compared to random perturbations . Huang et al . ( 2017 ) and Kos & Song ( 2017 ) use the fast gradient sign method ( FGSM ) to show deep reinforcement learning agents are vulnarable to adversarial perturbations . Pattanaik et al . ( 2018 ) use a gradient based formulation to increase the robustness of deep reinforcement learning agents . 2.2 ADVERSARIAL ATTACK METHODS . Goodfellow et al . ( 2015 ) introduced the fast gradient method ( FGM ) x∗ = x+ · ∇xJ ( x , y ) ||∇xJ ( x , y ) ||p , ( 1 ) for crafting adversarial examples for image classification by taking the gradient of the cost function J ( x , y ) used to train the neural network in the direction of the input . Here x is the input and y is the output label for image classification . As mentioned in the previous section FGM was first adapted to the deep reinforcement learning setting by Huang et al . ( 2017 ) . Subsequently , Pattanaik et al . ( 2018 ) introduced a variant of FGM , in which a few random samples are taken in the gradient direction , and the best is chosen . However , the main difference between the approach of Huang et al . ( 2017 ) and Pattanaik et al . ( 2018 ) is in the choice of the cost function J used to determine the gradient direction . In the next section we will outline the different cost functions used in these two different formulations . 2.3 ADVERSARIAL ATTACK FORMULATIONS . In a bounded attack formulation for deep reinforcement learning , the aim is to try to find a perturbed state sadv in a ball D , p ( s ) = { sadv ∣∣ ‖sadv − s‖p ≤ } , ( 2 ) that minimizes the expected cumulative reward of the agent . It is important to note that the agent will always try to take the best action depending only on its perception of the state , and independent from the unperturbed state . Therefore , in the perturbed state the agent will still choose the action , a∗ ( sadv ) = argmaxaQ ( sadv , a ) , which maximizes the state action value function in state sadv . It has been an active line of research to find the right direction for the adversarial perturbation in the deep reinforcement learning domain . The first attack of this form was formulated by Huang et al . ( 2017 ) and concurrently by Kos & Song ( 2017 ) by trying to minimize the probability of the best possible action in the given state , a∗ ( s ) = argmax a Q ( s , a ) min sadv∈D , p ( s ) π ( sadv , a ∗ ( s ) ) . ( 3 ) Note that π ( s , a ) is the softmax policy of the agent given by πT ( s , a ) = e Q ( s , a ( s ) ) T∑ ak e Q ( s , ak ) T , ( 4 ) where T is called the temperature constant . When the temperature constant is not relevant to the discussion we will drop the subscript and use the notation π ( s , a ) . It is important to note that π ( s , a ) is not the actual policy used by the agent . Indeed the DRL agent deterministically chooses the action a∗ maximizing Q ( s , a ) ( or equivalently π ( s , a ) ) . The softmax operation is only introduced in order to calculate the adversarial perturbation direction . Lin et al . ( 2017 ) formulated another attack with unbounded perturbation based on the Carlini & Wagner ( 2017 ) attack where the goal is to minimize the perturbation subject to choosing any action other than the best . min sadv∈D , p ( s ) ‖sadv − s‖p subject to a∗ ( s ) 6= a∗ ( sadv ) . Pattanaik et al . ( 2018 ) formulated yet another attack which aims to maximize the probability of the worst possible action in the given state , aw ( s ) = argmin a Q ( s , a ) max sadv∈D , p ( s ) π ( sadv , aw ( s ) ) , ( 5 ) and further showed that their attack formulation ( 5 ) is more effective than ( 3 ) . Pattanaik et al . ( 2018 ) also introduce the notion of targeted attacks to the reinforcement learning domain . In their paper they take the cross entropy loss between the optimal policy in the given state and their adversarial probability distribution and try to increase the probability of aw . However , just trying to increase the probability of aw in the softmax policy i.e . π ( sadv , aw ) is not sufficient to target aw in the actual policy followed by the agent . In fact the agent can end up in a state where , π ( sadv , aw ) > π ( s , aw ) aw 6= argmax a π ( sadv , a ) . ( 6 ) Although π ( sadv , aw ) has increased , the action aw will not be taken . However , it might be still possible to find a perturbed state s′adv for which , π ( sadv , aw ) > π ( s ′ adv , aw ) aw = argmax a π ( s′adv , a ) . ( 7 ) Therefore , maximizing the probability of taking the worst possible action in the given state is not actually the right formulation to find the correct direction for adversarial perturbation .

Summary: The work introduces a new method for observation-perturbation adversarial attacks against deep neural network policies. The idealized version of the method is an optimal attack assuming (a) the agent's Q-values are calibrated; (b) the attacker can only act once, at the current time step. While the attack is in fact conducted at every time step, so is not "myopic", this assumption greatly simplifies the problem. Moreover, the work approximates the (deterministic, hard-max) policy with a low-temperature softmax policy to enable gradient-based methods to work. Evaluating in a range of Atari games, the method has a significantly larger "impact" (normalized return) than previous methods, across $\ell_1$, $\ell_2$ and $\ell_{\infty}$ norms. Moreover, the results suggest the attack succeeds because of increasing the probability of low-ranked actions, consistent with the attack's derivation.