Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL

1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has achieved incredible success in many applications . However , recent works ( Huang et al. , 2017 ; Pattanaik et al. , 2018 ) reveal that a well-trained RL agent may be vulnerable to test-time evasion attacks , making it risky to deploy RL models in high-stakes applications . As in most related works , we consider a state adversary which adds imperceptible noise to the observations of an agent such that its cumulative reward is reduced during test time . In order to understand the vulnerability of an RL agent and to improve its certified robustness , it is important to evaluate the worst-case performance of the agent under any adversarial attacks with certain constraints . In other words , it is crucial to find the strongest/optimal adversary that can minimize the cumulative reward gained by the agent with fixed constraints , as motivated in a recent paper by Zhang et al . ( 2021 ) . Therefore , we focus on the following question : Given an arbitrary attack radius ( budget ) for each step of the deployment , what is the worstcase performance of an agent under the strongest adversary ? Finding the strongest adversary in RL is challenging . Many existing attacks ( Huang et al. , 2017 ; Pattanaik et al. , 2018 ) are based on heuristics , crafting adversarial states at every step independently , although steps are interrelated in contrast to image classification tasks . These heuristic methods can often effectively reduce the agent ’ s reward , but are not guaranteed to achieve the strongest attack under a given budget . This type of attack is “ myopic ” since it does not plan for the future . Figure 1 shows an intuitive example , where myopic adversaries only prevent the agent from selecting the best action in the current step , but the strongest adversary can strategically “ lead ” the agent to a trap , which is the worst event for the agent . Achieving computational efficiency arises as another challenge in practice , even if the strongest adversary can be found in theory . A recent work ( Zhang et al. , 2020a ) points out that learning the optimal state adversary is equivalent to learning an optimal policy in a new Markov Decision Process ( MDP ) . A follow-up work ( Zhang et al. , 2021 ) shows that the learned adversary significantly outperforms prior adversaries in MuJoCo games . However , the state space and the action space of the new MDP are both as large as the state space in the original environment , which can be high-dimensional in practice . For example , video games and autonomous driving systems use images as observations . In these tasks , learning the state adversary directly becomes computationally intractable . To overcome the above two challenges , we propose a novel attack method called Policy Adversarial Actor Director ( PA-AD ) , where we design a “ director ” and an “ actor ” that collaboratively finds the optimal state perturbations . In PA-AD , a director learns an MDP named Policy Adversary MDP ( PAMDP ) , and an actor is embedded in the dynamics of PAMDP . At each step , the director proposes a perturbing direction in the policy space , and the actor crafts a perturbation in the state space to lead the victim policy towards the proposed direction . Through a trail-and-error process , the director can find the optimal way to cooperate with the actor and attack the victim policy . Theoretical analysis shows that the optimal policy in PAMDP induces an optimal state adversary . The size of PAMDP is generally smaller than the adversarial MDP defined by Zhang et al . ( 2021 ) and thus is easier to be learned efficiently using off-the-shelf RL algorithms . With our proposed director-actor collaborative mechanism , PA-AD outperforms state-of-the-art attacking methods on various types of environments , and improves the robustness of many DRL agents by adversarial training . Summary of Contributions . ( 1 ) We establish a theoretical understanding of the optimality of evasion attacks from the perspective of policy perturbations , allowing a more efficient implementation of optimal attacks . ( 2 ) We introduce a Policy Adversary MDP ( PAMDP ) model , whose optimal policy induces the optimal state adversary under any attacking budget . ( 3 ) We propose a novel attack method , PA-AD , which efficiently searches for the optimal adversary in the PAMDP . PA-AD is a general method that works on stochastic and deterministic victim policies , vectorized and pixel state spaces , as well as discrete and continuous action spaces . ( 4 ) Empirical study shows that PA-AD universally outperforms previous attacking methods in various environments , including Atari games and MuJoCo tasks . PA-AD achieves impressive attacking performance in many environments using very small attack budgets , ( 5 ) Combining our strong attack PA-AD with adversarial training , we significantly improve the robustness of RL agents , and achieve the state-of-the-art robustness in many tasks . 2 PRELIMINARIES AND NOTATIONS . The Victim RL Agent In RL , an agent interacts with an environment modeled by a Markov Decision Process ( MDP ) denoted as a tuple M = 〈S , A , P , R , γ〉 , where S is a state space with cardinality |S| , A is an action space with cardinality |A| , P : S × A → ∆ ( S ) is the transition function 1 , R : S × A → R is the reward function , and γ ∈ ( 0 , 1 ) is the discount factor . In this paper , we consider a setting where the state space is much larger than the action space , which arises in a wide variety of environments . For notation simplicity , our theoretical analysis focuses on a finite MDP , but our algorithm applies to continuous state spaces and continuous action spaces , as verified in experiments . The agent takes actions according to its policy , π : S → ∆ ( A ) . We suppose the victim uses a fixed policy π with a function approximator ( e.g . a neural network ) during test time . We denote the space of all policies as Π , which is a Cartesian product of |S| simplices . The value of a policy π ∈ Π for state s ∈ S is defined as V π ( s ) = Eπ , P [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] . Evasion Attacker Evasion attacks are test-time attacks that aim to reduce the expected total reward gained by the agent/victim . As in most literature ( Huang et al. , 2017 ; Pattanaik et al. , 2018 ; Zhang et al. , 2020a ) , we assume the attacker knows the victim policy π ( white-box attack ) . However , the attacker does not know the environment dynamics , nor does it have the ability to change the environment directly . The attacker can observe the interactions between the victim agent and the environment , including states , actions and rewards . We focus on a typical state adversary ( Huang et al. , 2017 ; Zhang et al. , 2020a ) , which perturbs the state observations returned by the environment before the agent observes them . Note that the underlying states in the environment are not changed . 1∆ ( X ) denotes the the space of probability distributions over X . Formally , we model a state adversary by a function h which perturbs state s ∈ S into s̃ : = h ( s ) , so that the input to the agent ’ s policy is s̃ instead of s. In practice , the adversarial perturbation is usually under certain constraints . In this paper , we consider the common ` p threat model ( Goodfellow et al. , 2015 ) : s̃ should be in B ( s ) , where B ( s ) denotes an ` p norm ball centered at s with radius ≥ 0 , a constant called the budget of the adversary for every step . With the budget constraint , we define the admissible state adversary and the admissible adversary set as below . Definition 1 ( Set of Admissible State AdversariesH ) . A state adversary h is said to be admissible if ∀s ∈ S , we have h ( s ) ∈ B ( s ) . The set of all admissible state adversaries is denoted by H . Then the goal of the attacker is to find an adversary h∗ in H that maximally reduces the cumulative reward of the agent . In this work , we propose a novel method to learn the optimal state adversary through the identification of an optimal policy perturbation defined and motivated in the next section . 3 UNDERSTANDING OPTIMAL ADVERSARY VIA POLICY PERTURBATIONS . In this section , we first motivate our idea of interpreting evasion attacks as perturbations of policies , then discuss how to efficiently find the optimal state adversary via the optimal policy perturbation . Evasion Attacks Are Perturbations of Policies Although existing literature usually considers state-attacks and actionattacks separately , we point out that evasion attacks , either applied to states or actions , are essentially equivalent to perturbing the agent ’ s policy π into another policy πh in the policy space Π . For instance , as shown in Figure 2 , if the adversary h alters state s into state s̃ , the victim selects an action ã based on π ( ·|s̃ ) . This is equivalent to directly perturbing π ( ·|s ) to πh ( ·|s ) : = π ( ·|s̃ ) . ( See Appendix A for more detailed analysis including action adversaries . ) In this paper , we aim to find the optimal state adversary through the identification of the “ optimal policy perturbation ” , which has the following merits . ( 1 ) πh ( ·|s ) usually lies in a lower dimensional space than h ( s ) for an arbitrary state s ∈ S. For example , in Atari games , the action space is discrete and small ( e.g . |A| = 18 ) , while a state is a high-dimensional image . Then the state perturbation h ( s ) is an image , while πh ( ·|s ) is a vector of size |A| . ( 2 ) It is easier to characterize the optimality of a policy perturbation than a state perturbation . How a state perturbation changes the value of a victim policy depends on both the victim policy network and the environment dynamics . In contrast , how a policy perturbation changes the victim value only depends on the environment . Our Theorem 4 in Section 3 and Theorem 12 in Appendix B both provide insights about how V π changes as π changes continuously . ( 3 ) Policy perturbation captures the essence of evasion attacks , and unifies state and action attacks . Although this paper focuses on state-space adversaries , the learned “ optimal policy perturbation ” can also be used to conduct action-space attacks against the same victim . Characterizing the Optimal Policy Adversary As depicted in Figure 3 , the policy perturbation serves as a bridge connecting the perturbations in the state space and the value space . Our goal is to find the optimal state adversary by identifying the optimal “ policy adversary ” . We first define an Admissible Adversarial Policy Set ( Adv-policy-set ) BH ( π ) ⊂ Π as the set of policies perturbed from π by all admissible state adversaries h ∈ H . In other words , when a state adversary perturbs states within an ` p norm ball B ( · ) , the victim policy is perturbed within BH ( π ) . Definition 2 ( Admissible Adversarial Policy Set ( Adv-policy-set ) BH ( π ) ) . For an MDP M , a fixed victim policy π , we define the admissible adversarial policy set ( Adv-policy-set ) w.r.t . π , denoted by BH ( π ) , as the set of policies that are perturbed from π by all admissible adversaries , i.e. , BH ( π ) : = { πh ∈ Π : ∃h ∈ H s.t ∀s , πh ( ·|s ) = π ( ·|h ( s ) ) } . ( 1 ) Remarks ( 1 ) BH ( π ) is a subset of the policy space Π and it surrounds the victim π , as shown in Figure 3 ( middle ) . In the same MDP , BH ( π ) varies for different victim π or different attack budget . ( 2 ) In Appendix B , we characterize the topological properties of BH ( π ) . We show that for a continuous function π ( e.g. , neural network ) , BH ( π ) is connected and compact , and the value functions generated by all policies in the Adv-policy-set BH ( π ) form a polytope ( Figure 3 ( right ) ) , following the polytope theorem by Dadashi et al . ( 2019 ) .

Summary: The paper proposes a method for computing the strongest adversarial perturbation on state observations of an RL agent from a specified set of perturbations. It relates perturbations on states to perturbations on policies. It poses a specific problem to this end, develops a solution, and establishes its optimality.