Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL

1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has achieved incredible success in many applications . However , recent works ( Huang et al. , 2017 ; Pattanaik et al. , 2018 ) reveal that a well-trained RL agent may be vulnerable to test-time evasion attacks , making it risky to deploy RL models in high-stakes applications . As in most related works , we consider a state adversary which adds imperceptible noise to the observations of an agent such that its cumulative reward is reduced during test time . In order to understand the vulnerability of an RL agent and to improve its certified robustness , it is important to evaluate the worst-case performance of the agent under any adversarial attacks with certain constraints . In other words , it is crucial to find the strongest/optimal adversary that can minimize the cumulative reward gained by the agent with fixed constraints , as motivated in a recent paper by Zhang et al . ( 2021 ) . Therefore , we focus on the following question : Given an arbitrary attack radius ( budget ) for each step of the deployment , what is the worstcase performance of an agent under the strongest adversary ? Finding the strongest adversary in RL is challenging . Many existing attacks ( Huang et al. , 2017 ; Pattanaik et al. , 2018 ) are based on heuristics , crafting adversarial states at every step independently , although steps are interrelated in contrast to image classification tasks . These heuristic methods can often effectively reduce the agent ’ s reward , but are not guaranteed to achieve the strongest attack under a given budget . This type of attack is “ myopic ” since it does not plan for the future . Figure 1 shows an intuitive example , where myopic adversaries only prevent the agent from selecting the best action in the current step , but the strongest adversary can strategically “ lead ” the agent to a trap , which is the worst event for the agent . Achieving computational efficiency arises as another challenge in practice , even if the strongest adversary can be found in theory . A recent work ( Zhang et al. , 2020a ) points out that learning the optimal state adversary is equivalent to learning an optimal policy in a new Markov Decision Process ( MDP ) . A follow-up work ( Zhang et al. , 2021 ) shows that the learned adversary significantly outperforms prior adversaries in MuJoCo games . However , the state space and the action space of the new MDP are both as large as the state space in the original environment , which can be high-dimensional in practice . For example , video games and autonomous driving systems use images as observations . In these tasks , learning the state adversary directly becomes computationally intractable . To overcome the above two challenges , we propose a novel attack method called Policy Adversarial Actor Director ( PA-AD ) , where we design a “ director ” and an “ actor ” that collaboratively finds the optimal state perturbations . In PA-AD , a director learns an MDP named Policy Adversary MDP ( PAMDP ) , and an actor is embedded in the dynamics of PAMDP . At each step , the director proposes a perturbing direction in the policy space , and the actor crafts a perturbation in the state space to lead the victim policy towards the proposed direction . Through a trail-and-error process , the director can find the optimal way to cooperate with the actor and attack the victim policy . Theoretical analysis shows that the optimal policy in PAMDP induces an optimal state adversary . The size of PAMDP is generally smaller than the adversarial MDP defined by Zhang et al . ( 2021 ) and thus is easier to be learned efficiently using off-the-shelf RL algorithms . With our proposed director-actor collaborative mechanism , PA-AD outperforms state-of-the-art attacking methods on various types of environments , and improves the robustness of many DRL agents by adversarial training . Summary of Contributions . ( 1 ) We establish a theoretical understanding of the optimality of evasion attacks from the perspective of policy perturbations , allowing a more efficient implementation of optimal attacks . ( 2 ) We introduce a Policy Adversary MDP ( PAMDP ) model , whose optimal policy induces the optimal state adversary under any attacking budget . ( 3 ) We propose a novel attack method , PA-AD , which efficiently searches for the optimal adversary in the PAMDP . PA-AD is a general method that works on stochastic and deterministic victim policies , vectorized and pixel state spaces , as well as discrete and continuous action spaces . ( 4 ) Empirical study shows that PA-AD universally outperforms previous attacking methods in various environments , including Atari games and MuJoCo tasks . PA-AD achieves impressive attacking performance in many environments using very small attack budgets , ( 5 ) Combining our strong attack PA-AD with adversarial training , we significantly improve the robustness of RL agents , and achieve the state-of-the-art robustness in many tasks . 2 PRELIMINARIES AND NOTATIONS . The Victim RL Agent In RL , an agent interacts with an environment modeled by a Markov Decision Process ( MDP ) denoted as a tuple M = 〈S , A , P , R , γ〉 , where S is a state space with cardinality |S| , A is an action space with cardinality |A| , P : S × A → ∆ ( S ) is the transition function 1 , R : S × A → R is the reward function , and γ ∈ ( 0 , 1 ) is the discount factor . In this paper , we consider a setting where the state space is much larger than the action space , which arises in a wide variety of environments . For notation simplicity , our theoretical analysis focuses on a finite MDP , but our algorithm applies to continuous state spaces and continuous action spaces , as verified in experiments . The agent takes actions according to its policy , π : S → ∆ ( A ) . We suppose the victim uses a fixed policy π with a function approximator ( e.g . a neural network ) during test time . We denote the space of all policies as Π , which is a Cartesian product of |S| simplices . The value of a policy π ∈ Π for state s ∈ S is defined as V π ( s ) = Eπ , P [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] . Evasion Attacker Evasion attacks are test-time attacks that aim to reduce the expected total reward gained by the agent/victim . As in most literature ( Huang et al. , 2017 ; Pattanaik et al. , 2018 ; Zhang et al. , 2020a ) , we assume the attacker knows the victim policy π ( white-box attack ) . However , the attacker does not know the environment dynamics , nor does it have the ability to change the environment directly . The attacker can observe the interactions between the victim agent and the environment , including states , actions and rewards . We focus on a typical state adversary ( Huang et al. , 2017 ; Zhang et al. , 2020a ) , which perturbs the state observations returned by the environment before the agent observes them . Note that the underlying states in the environment are not changed . 1∆ ( X ) denotes the the space of probability distributions over X . Formally , we model a state adversary by a function h which perturbs state s ∈ S into s̃ : = h ( s ) , so that the input to the agent ’ s policy is s̃ instead of s. In practice , the adversarial perturbation is usually under certain constraints . In this paper , we consider the common ` p threat model ( Goodfellow et al. , 2015 ) : s̃ should be in B ( s ) , where B ( s ) denotes an ` p norm ball centered at s with radius ≥ 0 , a constant called the budget of the adversary for every step . With the budget constraint , we define the admissible state adversary and the admissible adversary set as below . Definition 1 ( Set of Admissible State AdversariesH ) . A state adversary h is said to be admissible if ∀s ∈ S , we have h ( s ) ∈ B ( s ) . The set of all admissible state adversaries is denoted by H . Then the goal of the attacker is to find an adversary h∗ in H that maximally reduces the cumulative reward of the agent . In this work , we propose a novel method to learn the optimal state adversary through the identification of an optimal policy perturbation defined and motivated in the next section . 3 UNDERSTANDING OPTIMAL ADVERSARY VIA POLICY PERTURBATIONS . In this section , we first motivate our idea of interpreting evasion attacks as perturbations of policies , then discuss how to efficiently find the optimal state adversary via the optimal policy perturbation . Evasion Attacks Are Perturbations of Policies Although existing literature usually considers state-attacks and actionattacks separately , we point out that evasion attacks , either applied to states or actions , are essentially equivalent to perturbing the agent ’ s policy π into another policy πh in the policy space Π . For instance , as shown in Figure 2 , if the adversary h alters state s into state s̃ , the victim selects an action ã based on π ( ·|s̃ ) . This is equivalent to directly perturbing π ( ·|s ) to πh ( ·|s ) : = π ( ·|s̃ ) . ( See Appendix A for more detailed analysis including action adversaries . ) In this paper , we aim to find the optimal state adversary through the identification of the “ optimal policy perturbation ” , which has the following merits . ( 1 ) πh ( ·|s ) usually lies in a lower dimensional space than h ( s ) for an arbitrary state s ∈ S. For example , in Atari games , the action space is discrete and small ( e.g . |A| = 18 ) , while a state is a high-dimensional image . Then the state perturbation h ( s ) is an image , while πh ( ·|s ) is a vector of size |A| . ( 2 ) It is easier to characterize the optimality of a policy perturbation than a state perturbation . How a state perturbation changes the value of a victim policy depends on both the victim policy network and the environment dynamics . In contrast , how a policy perturbation changes the victim value only depends on the environment . Our Theorem 4 in Section 3 and Theorem 12 in Appendix B both provide insights about how V π changes as π changes continuously . ( 3 ) Policy perturbation captures the essence of evasion attacks , and unifies state and action attacks . Although this paper focuses on state-space adversaries , the learned “ optimal policy perturbation ” can also be used to conduct action-space attacks against the same victim . Characterizing the Optimal Policy Adversary As depicted in Figure 3 , the policy perturbation serves as a bridge connecting the perturbations in the state space and the value space . Our goal is to find the optimal state adversary by identifying the optimal “ policy adversary ” . We first define an Admissible Adversarial Policy Set ( Adv-policy-set ) BH ( π ) ⊂ Π as the set of policies perturbed from π by all admissible state adversaries h ∈ H . In other words , when a state adversary perturbs states within an ` p norm ball B ( · ) , the victim policy is perturbed within BH ( π ) . Definition 2 ( Admissible Adversarial Policy Set ( Adv-policy-set ) BH ( π ) ) . For an MDP M , a fixed victim policy π , we define the admissible adversarial policy set ( Adv-policy-set ) w.r.t . π , denoted by BH ( π ) , as the set of policies that are perturbed from π by all admissible adversaries , i.e. , BH ( π ) : = { πh ∈ Π : ∃h ∈ H s.t ∀s , πh ( ·|s ) = π ( ·|h ( s ) ) } . ( 1 ) Remarks ( 1 ) BH ( π ) is a subset of the policy space Π and it surrounds the victim π , as shown in Figure 3 ( middle ) . In the same MDP , BH ( π ) varies for different victim π or different attack budget . ( 2 ) In Appendix B , we characterize the topological properties of BH ( π ) . We show that for a continuous function π ( e.g. , neural network ) , BH ( π ) is connected and compact , and the value functions generated by all policies in the Adv-policy-set BH ( π ) form a polytope ( Figure 3 ( right ) ) , following the polytope theorem by Dadashi et al . ( 2019 ) .

Summary: The paper proposes a method to craft stronger and more efficient attacks on state observation of the RL agent.  Here, the 'strongest' attacks refer to the attacks that minimize the reward the most. An 'efficient' attacker can craft such attacks using the least computational resources. In the work of (Zhang et al., 2021), the attacking strategy is described by a mapping from the state space of the underlying MDP problem to the same space. The agent observes the falsified states rather than the actual ones and takes actions based on the observed states. This misinformation eventually tricks the agent into taking a different action. In this paper, the authors argue that instead of finding the mapping from the state space to the same space, the attacker can first find the mapping from the state space to the action space that generates the least accumulated rewards to the agent. The 'strongest' attacks on the state observation can then be crafted based on the mapping found. Since in many deep RL applications, the state space is much larger than the action space. Hence, finding the mapping from the state space to the action space is more efficient than finding the one from the state space to the state space.