Detecting Worst-case Corruptions via Loss Landscape Curvature in Deep Reinforcement Learning

The non-robustness of neural network policies to adversarial examples poses a challenge for deep reinforcement learning . One natural approach to mitigate the impact of adversarial examples is to develop methods to detect when a given input is adversarial . In this work we introduce a novel approach for detecting adversarial examples that is computationally efficient , is agnostic to the method used to generate adversarial examples , and theoretically well-motivated . Our method is based on a measure of the local curvature of the neural network policy , which we show differs between adversarial and clean examples . We empirically demonstrate the effectiveness of our method in the Atari environment against a large set of state-ofthe-art algorithms for generating adversarial examples . Furthermore , we exhibit the effectiveness of our detection algorithm with the presence of multiple strong detection-aware adversaries . 1 INTRODUCTION . Since Mnih et al . ( 2015 ) showed that deep neural networks can be used to parameterize reinforcement learning policies , there has been substantial growth in new algorithms and applications for deep reinforcement learning . While this progress has resulted in a variety of new capabilities for reinforcement learning agents , it has at the same time introduced new challenges due to the nonrobustness of DNNs to imperceptible adversarial perturbations originally discovered by Szegedy et al . ( 2014 ) . In particular , Huang et al . ( 2017 ) ; Kos & Song ( 2017 ) showed that the non-robustness of DNNs to adversarial perturbations extends to the deep reinforcement learning domain , where applications such as self-driving cars or automatic financial trading can not tolerate such a vulnerability . In the setting of image classification there has been significant effort to make DNNs robust to adversarial perturbations Goodfellow et al . ( 2015 ) ; Madry et al . ( 2018 ) . At the same time , there is a line of work focused on showing the inevitability of adversarial examples and the intrinsic difficulty of learning robust classifiers Dohmatob ( 2019 ) ; Mahloujifar et al . ( 2019 ) ; Gourdeau et al . ( 2019 ) . Given that it may not be possible to make DNNs completely robust to adversarial examples , a natural objective is to instead attempt to detect the presence of adversarial examples Pang et al . ( 2018 ) ; Yang et al . ( 2020 ) ; Cintas et al . ( 2020 ) . Additional work has shown that many adversarial defense and detection methods fail in the white-box setting where the adversary is aware of both the trained model and the method used to detect examples Athalye et al . ( 2018 ) ; Carlini & Wagner ( 2017b ) . In this paper we propose a novel detection method for adversarial examples in deep reinforcement learning . This is the first method for detecting adversarial examples in this setting , where computational efficiency is paramount since the method must be applied in real-time to every state encountered by the agent . Our approach relies on differences in the curvature of the neural policy in the neighborhood of an adversarial example when compared to a natural example . At a high level our method is based on the intuition that while natural examples have neighborhoods determined by an optimization procedure intended to learn a policy that works well across all states , each adversarial example is the output of some local optimization in the neighborhood of one particular state . Our proposed method is computationally efficient , requiring only one gradient computation and two policy evaluations , requires no training that depends on the adversarial attack method , and is theoretically well-founded . In summary , we focus on detection of adversarial examples and make the following contributions : • Our paper is the first to focus on detection of adversarial examples in the deep reinforcement learning domain . • We propose a novel method , Detection of Adversaries with Taylor Approximation ( DATA ) , to detect adversarial examples based on the local curvature of the neural network policy . DATA is computationally efficient , independent of the method used to generate the adversarial example , and theoretically justified . • We conduct experiments in various games of the Arcade Learning Environment that demonstrate the effectiveness of DATA in detecting examples generated by several stateof-the-art adversarial attack methods . • Furthermore , we demonstrate the effectiveness of DATA in the white-box setting where the adversary is aware of the detection method used . 2 RELATED WORK AND BACKGROUND . Deep Reinforcement Learning : In this paper we focus on discrete action set Markov Decision Processes ( MDPs ) which are given by a continuous set of states S , a discrete set of actions A , a transition probability function P : S × A × S → R , and a reward function r : S × A → R. A policy π : S → P ( A ) assigns a probability distribution on actions π ( ·|s ) to each state s. In deep reinforcement learning the policy π is parameterized by a deep neural network . Adversarial Examples : Goodfellow et al . ( 2015 ) introduced the fast gradient method ( FGM ) for producing adversarial examples for image classification . The method is based on taking the gradient of the training cost function J ( x , y ) with respect to the input image , and bounding the perturbation by where x is the input image and y is the output label . Later , an iterative version of FGM called I-FGM was proposed by Kurakin et al . ( 2016 ) . This is also often referred to as Projected Gradient Descent ( PGD ) as in Madry et al . ( 2018 ) where the I-FGM update is xN+1adv = clip ( x N adv + αsign ( ∇xJ ( xNadv , y ) ) ) . ( 1 ) where x0adv = x. Dong et al . ( 2018 ) further modified I-FGM by introducing a momentum term in the update , yielding a method called MI-FGSM . Korkmaz ( 2020 ) later proposed a Nesterovmomentum based approach for the deep reinforcement learning domain . The DeepFool method of Moosavi-Dezfooli et al . ( 2016 ) is an alternative approach to those based on FGSM . DeepFool performs iterative projection to the closest separating hyperplane between classes . Another alternative approach proposed by Carlini & Wagner ( 2017a ) is based on finding a minimal perturbation that achieves a different target class label . The approach is based on minimizing the loss min sadv∈S c · J ( sadv ) + ∥∥sadv − s∥∥ ( 2 ) where s is the clean input , sadv is the adversarial example , and J ( s ) is a modified version of the cost function used to train the network . Chen et al . ( 2018 ) proposed a variant of the Carlini & Wagner ( 2017a ) formulation that adds an ` 1-regularization term to produce sparser adversarial examples , min sadv∈S c · J ( sadv ) + λ ∥∥sadv − s∥∥ + λ ∥∥sadv − s∥∥ ( 3 ) Adversarial Deep Reinforcement Learning : The adversarial problem initially has been investigated by Huang et al . ( 2017 ) and Kos & Song ( 2017 ) concurrently . In this work the authors show that perturbations computed via FGSM result in extreme performance loss on the learnt policy . Lin et al . ( 2017 ) and Sun et al . ( 2020 ) focused on timing strategies in the adversarial formulation and utilized the Carlini & Wagner ( 2017a ) method to produce the perturbations . While there is a reasonable body of work focused on finding efficient and effective adversarial perturbations , a substantial body of work focused on building agents robust to these perturbations . Mandlekar et al . ( 2017 ) proposed to utilize FGSM perturbations during training time to obtain more robust agents . Pinto et al . ( 2017 ) modeled the adversarial interaction as a zero sum game and proposed a joint training strategy to increase robustness in continuous action space setting . Recently , Gleave et al . ( 2020 ) considered an adversary who is allowed to take natural actions in a given environment instead of ` p-norm bounded perturbations and modeled the adversarial relationship as a zero sum Markov game . However , recent concerns have been raised about adversarial training methods by Korkmaz ( 2021 ) . In this paper the authors show that the state-of-the-art adversarial training techniques introduce a new set of nonrobust features . Thus , with the rising concerns on robustness of recent proposed adversarial training techniques our work aims to solve the adversarial problem from a different perspective by detecting adversarial perturbations . Detection of Adversarial Examples : There has been a long line of work on detection of adversarial examples for image classification . Metzen et al . ( 2017 ) proposed augmenting standard neural network image classifiers with a sub-network trained specifically to detect adversarial examples . However , this method has a cost of separately training a binary classifier to be able to detect the adversarial examples . Pang et al . ( 2018 ) proposed a modified neural network training strategy that is combined with a test-time thresholding method to distinguish adversarial from clean examples with additional training cost . Yang et al . ( 2020 ) studied detection of adversarial examples via feature attribution , utilizing the Leave-One-Out ( LOO ) attribution method proposed by Li et al . ( 2016 ) . The LOO method is based on erasing each pixel of the image and observing how the output of the neural network changes , and so requires a number of neural network classifier evaluations equal to the resolution of the input image . Finally , Roth et al . ( 2019 ) suggest a statistical test based on measuring the average change in the odds ratio between classes under random perturbations . Evaluation of the neural classifier at hundreds of randomly perturbed samples are required in order for this method to get sufficiently accurate estimates of the average change in order to detect adversarial examples . In contrast to prior methods , our detection method does not require modifying the training of the neural network , does not require any training specific to the attack method used , and uses only two neural network function evaluations and one gradient computation . 3 DETECTION OF ADVERSARIES WITH TAYLOR APPROXIMATION ( DATA ) . In this section we give the high-level motivation for and formal description of our detection method . We begin by introducing necessary notation and definitions . We denote an original clean state by s̄ and an adversarially perturbed state by sadv . Definition 3.1 . The cost of a state , J ( s , τ ) , is defined as the cross entropy loss between the policy π ( a|s ) of the agent , and a target distribution on actions τ ( a ) . J ( s , τ ) = − ∑ a τ ( a ) log ( π ( a|s ) ) ( 4 ) Definition 3.2 . The argmax policy , π∗ ( a|s ) , is defined as the distribution which puts all probability mass on the highest weight action of π ( a|s ) . π∗ ( a|s ) = 1 ( a = arg max a′ π ( a′|s ) ) ( 5 ) We use the following notation for the gradient and Hessian with respect to states s : ∇sJ ( s0 , τ0 ) = ∇sJ ( s , τ ) |s=s0 , τ=τ0 ∇2sJ ( s0 , τ0 ) = ∇2sJ ( s , τ ) |s=s0 , τ=τ0

Summary: This paper provides a fast method to detect adversarially attacked states during reinforcement learning. The main improvement over previous (Roth) approach is to explicitly involve analysis of what's happening along the gradient direction. They relate second-order behavior with common adversarial considerations to motivate the approach.  Results show a large improvement over a previous random-sampling approach.  They also show the method resists even adversary-aware attacks fairly well.