Distilled Agent DQN for Provable Adversarial Robustness

1 INTRODUCTION . To ensure Reinforcement Learning ( RL ) behaves reliably in the real world , it is important to consider settings where an adversary aims to actively hinder the learning process of the agent . While prior work has explored robustness in the setting of discrete RL ( Morimoto & Doya , 2005 ; Boyan & Moore , 1995 ) , most recent progress in the field has focused on dealing with continuous states by using neural networks ( Sutton et al. , 2000 ; Mnih et al. , 2013 ; Peters & Schaal , 2008 ) . In this work we present a new approach for training RL systems to be more ( provably ) robust . The key idea is to decouple the DQN network architecture into a policy ( student ) network S and a Q-network in a way which allows us to robustly train the policy network and use it for exploration while at the same time preserving the standard way in which the Q network is trained . We then show how to naturally incorporate state of the art defenses , developed in the context of deep supervised , learning to the reinforcement learning setting by training the student network in two ways : ( i ) via adversarial training with methods such as FGSM where we generate adversarial states that decrease the chance the optimal action is selected , and ( ii ) via provably robust training with symbolic methods which guarantee the network will select the right action in a given state despite any possible perturbation ( within a range ) of that state . Contributions Our main contributions are : • The first deep RL algorithm , DadQN , designed to be adversarially defended with state-ofthe-art adversarial training as well as provably robust training into our RL algorithm . To our best knowledge , this is the first time , on-line robustness certification has been achieved with a deep RL agent . • An attack , illustrated in Figure 1 , that hinders learning in modern deep RL algorithms . • An evaluation demonstrating that DadQN can defend against adversarial attacks when defenses against DQNs catastrophically fail , while being of comparable performance when adversarial attacks are not present . 2 RELATED WORK . At roughly the same time as deep reinforcement learning methods were first being developed Schulman et al . ( 2015 ) ; Mnih et al . ( 2013 ) it was noticed by Goodfellow et al . ( 2014 ) that in the 1 tag sale by PetalArt , money by DesignContest , crying face , microscope , shopping cart and graduation cap by Twitter , Inc. , are licensed under CC BY 4.0. setting of supervised learning , classifiers were susceptible to adversarial attacks by introducing imperceptible perturbations to the inputs . At the same time , it was determined that training using these adversarially crafted images could serve as a way to defend against them . Later , Carlini & Wagner ( 2016 ) determined that many defensive techniques were insufficient . A particularly successful defense is that of Madry et al . ( 2018 ) . This method uses adversarial examples generated by the PGD attack as training examples ( instead of the original examples ) . While it is experimentally effective , it provides no guarantees about the resulting network ’ s robustness . Carlini & Wagner ( 2017 ) introduced further method for evaluating the robustness of neural networks . Katz et al . ( 2017 ) provided the first system which could ( for very small networks ) determine local-robustness , that is , at test time the network is proven to be either -robust for an individual image . Gehr et al . ( 2018 ) used abstract interpretation to scale the local robustness analysis by sacrificing completeness . Raghunathan et al . ( 2018 ) introduced a technique to train networks to be certifiably robust ( though again limited to small networks ) . Multiple recent techniques have been developed since to train increasingly larger networks to be certifiably robust including Wong & Kolter ( 2018 ) ; Mirman et al . ( 2018 ) ; Dvijotham et al . ( 2018 ) ; Wong et al . ( 2018 ) . These techniques represent different points in the spectrum of classification accuracy , certifiable robustness , experimental robustness , and training speed . While Pattanaik et al . ( 2017 ) used adversarial examples to train an agent and improve its experimental robustness , Behzadan & Munir ( 2017b ) noticed that often doing this prevents learning . Behzadan & Munir ( 2017a ) further demonstrated that this effect could be intentionally amplified to prevent learning intentionally on the part of an adversary . Gu et al . ( 2018 ) demonstrated a technique for training the A3C algorithm to be adversarially robust by training a parallel adversarial actor-critic pair for each protagonist actor-critic pair . The analysis of their technique is limited to noisy environment adversaries , and not intentional agent-aware adversaries such as FGSM . Similarly , Ferdowsi et al . ( 2018 ) created an adaptive adversary to improve robustness . Pinto et al . ( 2017 ) first used adversarial training to improve robustness of deep reinforcement learning algorithms , specifically TRPO . Mandlekar et al . ( 2017 ) demonstrated techniques to construct physically plausible perturbations adversarially to improve deep RL . Gu et al . ( 2018 ) then demonstrated how adversarial training could also be used to improve A3C . Papernot et al . ( 2015 ) first proposed that distilling a neural network by training the same architecture with the probability vector outputs of the original network might act as a defense against adversarial examples . While Carlini & Wagner ( 2016 ) found that defensive distillation is insufficient to protect against adversarial examples for supervised learning , our algorithm uses distillation in combination with more effective defenses to protect reinforcement learning . Chen et al . ( 2017 ) use a pre-defined rule-based teacher policy to assist a deep RL system . They introduce a new DQN algorithm which predicts when to consult the teacher and how to learn from the teacher ’ s experiences . In our setting there is no known external teacher . Lin et al . ( 2017a ) use distillation to improve reinforcement learning itself by introducing the collaborative asynchronous advantage actor-critic algorithm ( cA3C ) . Their method allows for knowledge transfer between agents in cases where multiple agents are playing simultaneously in different environments with potentially different tasks . To do this , they use a “ deep alignment network ” which learns to transfer the outputs from a teacher network to a student network . Unlike our system , their system only improves agents learning from different tasks , and does not provide a method to defend the learned policy . Shashua & Mannor ( 2017 ) train a DQN , that avoids the worst case outcome and performs well in the presence of a non-deterministic environments by using an extended Kalman-Filter . However , their Kalman-update requires multiplication of matrices of size quadratic in the total number of weights and thus does not scale to neural networks that play Atari-Games with the screen input . On the other hand , we optimize a policy to generalize to non-game effecting perturbations in a way which we show is scalable to very large neural networks . We furthermore demonstrate how to do this in a way that increases provability . 3 BACKGROUND . The goal of reinforcement learning is to determine a behavior policy2 π ( a | s ) for a given game G : State × Action → State × R , that maximizes the expected discounted sum of future rewards Qπ ( s ) = E [ ∑∞ t=1 γ trt|s1 = s , a1 = a , π ] , where γ ∈ [ 0 , 1 ] is a discount factor and si+1 , ri = G ( si , ai ) . The objective of Q-Learning ( Watkins & Dayan , 1992 ) is to learn the function Q∗ ( s ) a = maxπ Qπ ( s ) a and construct a policy by greedily picking its optimal action . 3.1 DEEP Q LEARNING . In Deep Q-Learning ( Mnih et al. , 2015 ; 2013 ) the goal is to progressively approximate Q∗ by weights θi for a neural network Q ( s ; θi ) a at iteration i . Samples are generated and stored in an experience replay bufferDi = { e1 , . . . , ei } where ei = ( si , ai , ri , si+1 ) is generated using a sufficiently random agent based on the approximation Q ( s ; θi ) . Integral to the DQN algorithm is the use of a network with lagging weights θ−i = θkb ik c for some k. The loss at iteration i is described by Li = E ( s , a , r , s′ ) ∼U ( Di ) [ ( r + γmax a′ Q ( s′ ; θ−i ) a′ −Q ( s ; θi ) a ) 2 ] A few iterations of stochastic gradient descent are used to minimize Li and produce θi+1 , although often multiple exploration steps are used first . 3.2 ADVERSARIAL ATTACK . Unlike traditional reinforcement learning , the presence of adversarial attacks introduces the complication that the game has some level of knowledge about the agent playing the game and can act in specific ways knowing its behavior . The environment that our algorithm is designed to address is given by our definition of an agent aware game . Agent Aware Game In agent aware reinforcement learning , the game G not only has access to the agent ’ s choices , but also has access to the learning system and the agent itself . Thus , an adversarial agent ’ s aim is to play the game in such a way as to poison learning , but also potentially play differently when it is aware if the RL system is training or testing . We consider only manipulative agent aware games , defined to be games G : State× Action× Agent→ State× R which are composed of a core game H : State× Action→ State× R and a manipulative adversary M : State×Agent→ State such that G ( si , ai , π ) = ( M ( s−i+1 , π ) , ri ) where ( s − i+1 , ri ) = H ( si , ai ) , and such that for any π where si+1 = M ( s−i+1 , π ) it is true that ∀a.H ( si+1 , a ) = H ( s − i+1 , a ) . In other words , manipulation does not effect how the core game works or is played . The attacks we consider are L∞ -attacks ( not to be confused with the used for greedy exploration ) . Specifically M ( s , π ) ∈ B ( s ) = { s′ | ||s− s′||∞ ≤ } , that is , B ( s ) is an sized L∞ ball . We consider both , test time attacks , and attacks that change their behavior when the agent is training . 3.3 TESTING ATTACKS . Huang et al . ( 2017 ) attack a network policy trained by reinforcement learning , at testing time and find that traditional untargeted attacks such as FGSM are sufficient to significantly reduce the performance 2We write π ( s ) to mean vector of scores for each action in the deterministic greedy policy where argmaxa π ( s ) a is the action taken . When written this way , it is assumed that π is a neural network . of the policy produced by DQN , A3C and TRPO . Lin et al . ( 2017b ) used targeted attacks to “ enchant ” the agent into performing specific actions . We evaluate only with untargeted FGSM attacks .

Summary: The goal of this paper is to train deep RL agents that perform well both in the presence and absence of adversarial attacks at training and test time. To achieve this, this paper proposes using policy distillation. The approach, Distilled Agent DQN (DaDQN), consists of: (1) a "teacher" neural network trained in the same way as DQN, and (2) a "student" network trained with supervised learning to match the teacher’s outputs. Adversarial defenses are only applied to the student network, so as to not impact the learning of Q-values by the teacher network. At test time, the student network is deployed.