Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL
Authors: Yanchao Sun
Ruijie Zheng
Yongyuan Liang
Furong Huang
Venue: ICLR 2022 Poster (2022)
Keywords: adversarial RL
robustness of RL
evasion attack
optimal attack
observation perturbation

Abstract: Evaluating the worst-case performance of a reinforcement learning (RL) agent under the strongest/optimal adversarial perturbations on state observations (within some constraints) is crucial for understanding the robustness of RL agents. However, finding the optimal adversary is challenging, in terms of both whether we can find the optimal attack and how efficiently we can find it. Existing works on adversarial RL either use heuristics-based methods that may not find the strongest adversary, or directly train an RL-based adversary by treating the agent as a part of the environment, which can find the optimal adversary but may become intractable in a large state space. 
This paper introduces a novel attacking method to find the optimal attacks through collaboration between a designed function named "actor" and an RL-based learner named "director'". The actor crafts state perturbations for a given policy perturbation direction, and the director learns to propose the best policy perturbation directions. Our proposed algorithm, PA-AD, is theoretically optimal and significantly more efficient than prior RL-based works in environments with large state spaces. Empirical results show that our proposed PA-AD universally outperforms state-of-the-art attacking methods in various Atari and MuJoCo environments. By applying PA-AD to adversarial training, we achieve state-of-the-art empirical robustness in multiple tasks under strong adversaries.

Published as a conference paper at ICLR 2022
WHO Is THE STRONGEST ENEMY? TOWARDS OPTI- MAL AND EFFICIENT EVASION ATTACKS IN DEEP RL
Yanchao Sun'
Ruijie Zheng²
Yongyuan Liang
Furong Huang
1,44 University of Maryland, College Park 3 Sun Yat-sen University
124 (ycs, rzheng12, furonhh umd. edu
$1iangyy58ema eel
ABSTRACT
Evaluating the worst-case performance of reinforcement learning (RL) agent un- der the strongest/optimal adversarial perturbations on state observations (within some constraints) is crucial for understanding the robustness of RL agents. How- ever, finding the optimal adversary is challenging, in terms of both whether we find the optimal attack and how efficiently adversarial RI. either use heuristics-based methods that may not find the strongest adversary, or directly train an RL-based adversary by treating the agent as a part of the environment, which can find the optimal adversary but may become intractable in a large state space. This paper introduces novel attacking method to find the optimal attacks through collaboration between a designed function named "actor" and an RL-based learner named "director". The actor crafts state perturbations for a given policy perturbation direction, and the director learns to propose the best policy perturbation directions. Our proposed algorithm, PA-AD, theoretically optimal and significantly more efficient than prior RL-based works in environ- ments with large state spaces. Empirical results show that our proposed PA-AD universally outperforms state-of-the-as attacking methods in various Atari and MuJoCo environments. By applying PA-AD to adversarial training, we achieve state-of-the-ane empirical robustness multiple tasks under strong adversaries.
find it. Existing works
INTRODUCTION
Deep Reinforcement Learning (DRL) has achieved incredible success in many applications. How- recent works (Huang et al., 2017; Pattanaik et al., 2018) reveal that a well-trained RL agent may be vulncrable to test-time evasion attacks, making it risky to deploy RL models in high-stakes applications. As in most related works, we consider state adversary which adds imperceptible noise to the observations of an agent such that its cumulative reward is reduced during test time.
In order understand the vulnerability of RL agent and to improve its certified robustness, is important to evaluate the worst-case performance of the agent under any adversarial attacks with certain constraints. In other words, it is crucial to find the strongest/optimal adversary that can minimize the cumulative reward gained by the agent with fixed constraints, as motivated in a recent paper by Zhang al. (2021). Therefore, we focus on the following question:
Given an arbitrary attack radius (budget) e for each step of the deployment, what is the worst- case performance of an agent under the strongest adversary?
Finding the strongest adversary in RL is challenging. Many existing attacks (Huang et al., 2017; Pattanaik al., 2018) are based on heuristics, crafting adversarial states at every step in- dependently, although steps are interrelated in contrast to im- age classification tasks. These heuristic methods can often ef- fectively reduce the agent's reward, but are not guaranteed to achieve the strongest attack under given budget. This type of attack is "myopic" since does not plan for the future. Fig- ure shows an intuitive example, where myopic adversaries only prevent the agent from selecting the best action in the current step, but the strongest adversary can strategically "lead" the agent to trap, which is the worst event for the agent.
myopic
strongest
Victim x
myopic
An example that a myopic adversary not the strongest.
Figure 1:

---

Published as a conference paper at ICLR 2022
Achieving computational efficiency arises another challenge in practice, even if the strongest adversary can be found in theory. A recent work (Zhang et al., 2020a) points out that learning the optimal state adversary is equivalent to learning an optimal policy in new Markov Decision Process (MDP). A follow-up work (Zhang et al., 2021) shows that the learned adversary significantly outper- forms prior adversaries in MuloCo games. However, the state space and the action space of the new MDP are both as large as the state space in the original environment, which can be high-dimensional in practice. For example, video games and autonomous driving systems use images as observations. In these tasks, learning the state adversary directly becomes computationally intractable.
To overcome the above two challenges, we propose a novel attack method ealled Policy Adver- sarial Actor Director (PA-AD), where we design a "director" and an "actor" that collaboratively finds the optimal state perturbations. In PA-AD, a director learns an MDP named Policy Adversary MDP (PAMDP), and an actor is embedded in the dynamics of PAMDP. At each step, the director proposes a perturbing direction in the policy space, and the actor crafts a perturbation in the state space to lead the victim policy towards the proposed direction. Through a trail-and-error process, the director can find the optimal way to cooperate with the actor and attack the victim policy. Theoret- ical analysis shows that the optimal policy in PAMDP induces optimal state adversary. The size of PAMDP is generally smaller than the adversarial MDP defined by Zhang et al. (2021) and thus is easier to be learned efficiently using off-the-shelf RL algorithms. With our proposed director-actor collaborative mechanism, PA-AD outperforms state-of-the-art attacking methods on various types of environments, and improves the robustness of many DRL agents by adversarial training.
Summary of Contributions
(1) We establish theoretical understanding of the optimality of evasion attacks from the perspective of policy perturbations, allowing a more efficient implementation of optimal attacks.
(2) We introduce a Policy Adversary MDP (PAMDP) model, whose optimal policy induces the op- timal state adversary under any attacking budget
(3) We propose a novel attack method, PA-AD, which efficiently searches for the optimal adversary in the PAMDP. PA-AD is a general method that works on stochastic and deterministic victim poli- cies, vectorized and pixel state spaces, as well as discrete and continuous action spaces.
(4) Empirical study shows that PA-AD universally outperforms previous attacking methods in vari- ous environments, including Atari games and MuJoCo tasks. PA-AD achieves impressive attacking performance in many environments using very small attack budgets,
(5) Combining our strong attack PA-AD with adversarial training, we significantly improve the ro- bustness of RL agents, and achieve the state-of-the-ari robusiness in many tasks.
PRELIMINARIES AND NOTATIONS
The Victim RL Agent In RL, an agent interacts with an environment modeled by a Markov Decision Process (MDP) denoted as a tuple M = (S, AA P. R, 7), where S is a state space with cardinality S), A is an action space with cardinality [A]. P S A + A(S) is the transition function R S x A - IR is the reward function, and E (0, 1) is the discount factor. In this paper, we consider a setting where the state space is much larger than the action space, which arises in wide variety of environments. For notation simplicity, our theoretical analysis focuses on finite MDP, but our algorithm applies to continuous state spaces and continuous action spaces, as verified in experiments. The agent takes actions according its policy, S + (AA). We suppose the victim uses a fixed policy π with a function approximator (e.g. a neural network) during test time. We denote the space of all policies as II, which is Cartesian product of |S] simplices. The value of a policy π E IT for state 8 E S is defined as V (8) = Ex,riEioo 7d Ri(s, a+)]s0 (6 s).
Evasion Attacker Evasion attacks are test-time attacks that aim to reduce the expected total reward gained by the agent/victim. As in most literature (Huang et al., 2017; Pattanaik et al., 2018; Zhang et 2020a), we assume the attacker knows the victim policy ι (white-box attack). However, the attacker does not know the environment dynamics, nor does it have the ability to change the environment directly. The attacker can observe the interactions between the victim agent and the environment, including states, actions and rewards. We focus on typical state adversary (Huang et al., 2017; Zhang et al., 2020a), which perturbs the state observations returned by the environment before the agent observes them. Note that the underlying states in the environment are not changed.
A (X) denotes the the space of probability distributions over X.
2

---

Published as a conference paper at ICLR 2022
Formally, model state adversary by function h which perturbs state S into 3 h(s), so that the input to the agent's 's policy is instead of s. In practice, the adversarial perturbation is usually under certain constraints. In this paper, we consider the common lp threat model (Goodfellow et al., 2015): & should be in (s),,, where (s) denotes an lp norm ball centered at with radius é 2 0, a constant called the budget of the adversary for every step. With the budget constraint, we define the admissible state adversary and the admissible adversary set as below.
:=
Definition (Set of Admissible State Adversaries H.). A state adversary h is said to be admissible Vs S, we have h((s) E (s). The set of all admissible state adversaries is denoted by He
Then the goal of the attacker is to find an adversary h" in H. that maximally reduces the cumulative reward of the agent. In this work, propose novel method to learn the optimal state adversary through the identification of an optimal policy
policy perturbation defined and motivated in the next section.
3 UNDERSTANDING OPTIMAL ADVERSARY VIA POLICY PERTURBATIONS
In this section, we first motivate our idea of interpreting evasion attacks as perturbations of policies, then discuss how to efficiently find the optimal state adversary via the optimal policy perturbation.
Evasion Attacks Are Perturbations of Policies Although existing literature usually considers state-attacks and action- attacks separately, we point out that evasion attacks, either ap- plied to states or actions, are essentially equivalent to perturb- ing the agent's policy æ into another policy Ty in the policy space II. For instance, as shown in Figure 2, if the adversary alters state s into state 3, the victim selects an action based on (-13). This is equivalent to directly perturbing rT-|s) to (n(-1s) (-13). (See Appendix A for more detailed analysis including action adversaries.)
TT
Figure 2: Equivalence between eva- sion attacks and policy perturbations.
In this paper, aim to find the optimal state adversary through the identification of the "optimal policy perturbation", which has the following merits. (1) TT. (-Is) usually lies in a lower dimensional space than h(s) for an arbitrary state E S. For example, in Atari games, the action space is discrete and small (e.g. |A| = 18), while a state is a high-dimensional image. Then the state perturbation h(s)) is an image, while (-Is) is vector of size |A] (2) is easier to characterize the optimality of a policy perturbation than a state perturbation. How state perturbation changes the value of a victim policy depends on both the victim policy network and the environment dynamics. In contrast, how policy perturbation changes the victim value only depends on the environment. Our Theorem 4 in Section and Theorem 12 in Appendix both provide insights about how V changes as changes continuously. (3) Policy perturbation captures the essence of evasion attacks, and unifies state and action attacks. Although this paper focuses on state-space adversaries, the learned "optimal policy perturbation" can also be used to conduct action-space attacks against the same victim.
Characterizing the Optimal Policy Adversary As depicted in Figure 3, the policy perturbation serves as a bridge connecting the perturbations in the state space and the value space. Our goal is to find the optimal state adversary by identifying the optimal "policy adversary". We first define an Admissible Adversarial Policy Set (Adv-policy-set) R. (TT) C II as the set of policies perturbed from æ by all admissible state adversaries h E H.- In other words, when a state adversary perturbs states within an lp norm ball (.).), the victim policy is perturbed within 1 (TT).
Definition 2 (Admissible Adversarial Policy Set (Adv-policy-set) B (T)). For an MDP M, a fixed victim policy we define the admissible adversarial policy set (Adv-policy-set) w.r.t n, de- noted by B (x), as the set of policies that are perturbed from n by all admissible adversaries, i.e,
B" (T)) (Th E IT: 3h E H. s.I Vs, mT((1)) ((11(()))
Remarks (1) 8.th (T)) is a subset of the policy space II and it surrounds the victim म, as shown in Figure 3(middle). In the same MDP, B (T) varies for different victim æ or different attack budget e. (2) In Appendix B, we characterize the topological properties of B²((T) (r). We show that for continuous function π (e.g.. neural network), B.d (T)) is connected and compact, and the value functions generated by all policies in the adv-policy-set 1 (T) form a polytope (Figure 3(rghht), following the polytope theorem by Dadashi al. (2019).
3

---

Published as a conference paper at ICLR 2022
End-to-end SA-RL:
An RL problem in large MDP
V
Victim Tr
T 1
Environment
VT
h(s)
(Palicy network)
V
V
S
(,(s)
II
8" (N))
PA-AD (ours):
An optimization problem
An RL problem in a small MDP
The Actor
The Director
Figure 3: A state adversary h perturbs into h(s) E Be(s) in the state space; hence, the victim's 's policy is perturbed into Th within the Adv-policy-set 88 (x): as result, the expected total reward the victim can gain becomes TThh instead of V A prior work SA-RL (Zhang et al., 2021) directly uses an RL agent to learn the best state adversary h", which works for MDPs with small state spaces, but suffers from high complexity in larger MDPs. In contrast, we find the optimal state adversary h" efficiently through identifying the optimal policy adversary my- Our proposed attack method called PA-AD contains an RL-based "director" which learns to propose policy perturbation Tr in the policy space, and a non-RL "actor", which targets at the proposed m. and computes adversarial states in the state space. Through this collaboration, the director can learn the optimal policy adversary गт- using RL methods, such that the actor executes h" as justified in Theorem 7.
Given that the Adv-policy-set 8.td (T) contains all the possible policies the victim may execute under admissible state perturbations, we can characterize the optimality of state adversary through the lens of policy perturbations. Recall that the attacker's goal is to find a state adversary ht" E He that minimizes the victim's 's expected total reward. From the perspective of policy perturbation, the attacker's s goal is to perturb the victim's 's policy to another policy тn- E B (i)) with the lowest value. Therefore, we can define the optimal state adversary and the optimal policy adversary as below.
Definition 3 (Optimal State Adversary h" and Optimal Policy Adversary πh-). For an MDP M, a fixed policy n, and an admissible adversary set H. with attacking budget e, (1) optimal state adversary lt' satisfies he E argmin H. VT* (s), Vs E 8, which leads to (2) an optimal policy adversary The satisfies Ts- E aarminn EB" V (s), Vs E.
Recall that ah is the perturbed policy caused by adversary h, i.e., m = (())))), Vs E.
Definition 3 implies an equivalent relationship between the optimal state adversary and the optimal policy adversary: an optimal state adversary leads to optimal policy adversary, and any state adversary that leads to an optimal policy adversary is optimal. Theorem 19 in Appendix D.1 shows that there always exists policy adversary is an RL problem. (A similar result have been shown by Zhang et al. (2020a) for the optimal state adversary, while we focus on the policy perturbation.)
optimal policy adversary for fixed victim æ, and learning the optimal
Due to the equivalence, if one finds an optimal policy adversary Th, then the optimal state adversary can be found by executing targeted attacks with target policy nh- However, directly finding the optimal policy adversary in the Adv-policy-set _ (T) is challenging since 13d (x) ) is generated by all admissible state adversaries in H. and hard to compute. To address this challenge, we first get insights from theoretical characterizations of the Adv-policy-se B. (Tr). Theorem 4 below shows that the "outermost boundary" of 1 (T) always contains an optimal policy adversary. Intuitively, a policy n' is in the outermost boundary of B (T)) if and only if no policy in /3 (TT) is farer away from t than T' in the direction n' म. Therefore, if an adversary can perturb a policy along a direction, it should push the policy far away as possible in this direction under the budget constraints. Then, the adversary is guaranteed to find an optimal policy adversary after trying all the perturbing directions. In contrast, such guarantee does not exist for state adversaries, justifying the benefits of considering policy adversaries. Our proposed algorithm in Section 4 applies this idea to find the optimal attack: an RL-based director searches for the optimal perturbing direction, and an actor is responsible for pushing the policy to the outermost boundary of B (π) with a given direction.
Theorem 4. For an MDP M, a fixed policy n, and an admissible adversary set Ho define the outermost boundary of the admissible adversarial policy set B (T)) w.rl n
0. (w) = (n' (8 Vs S,0 > 0, AA E 18l (x) s.a. ((-1s) = <+1ss) ((1ss <({s)))- (2) Then there exists a policy ă e 8 (TT) such that is the optimal policy adversary w.E.. n.
Theorem 4 is proven in Appendix B.3, and we visualize the outermost boundary in Appendix B.5.
4

---

Published as a conference paper at ICLR 2022
4 PA-AD: OPTIMAL AND EFFICIENT EVASION ATTACK
In this section, we first formally define the optimality of an attack algorithm and discuss some exist- ing attack methods. Then, based on the theoretical insights in Section 3, we introduce our algorithm, Policy Adversarial Actor Director (PA-AD) that has an optimal formulation and is efficient to use.
Although many attack methods for RL agents have been proposed (Huang et al., 2017; Pattanaik et al., 2018; Zhang et al., 2020a), it is not yet well-understood how to characterize the strength and the optimality of an attack method. Therefore, we propose to formulate the optimality of an attack algorithm, which answers the question "whether the attack objective finds the strongest adversary".
Definition 5 (Optimal Formulation of Attacking Algorithm). An attacking algorithm Algo is said to have an optimal formulation iff for any MDP M. policy x and admissible adversary set He under attacking budget e, the set of optimal solutions to its objective, A is subset of the optimal adversaries against n, i.e., A Ç H == (h*) Ih* E argminiEHH V = (s), Vs E S).
Many heuristic-based attacks, although are empirically effective and efficient, do not meet the re- quirements of optimal formulation. In Appendix D.3, we categorize existing heuristic attack meth- ods into four types, and theoretically prove that there exist scenarios where these heuristic methods may not find the strongest adversary. A recent paper (Zhang et al., 2021) proposes to learn the opti- mal state adversary using RL methods, which we will refer to as SA-RL in our paper for simplicity. SA-RL can be viewed an "end-to-end"' RL attacker, as it directly learns the optimal state ad adversary such that the value of the victim policy is minimized. The formulation of SA-RL satisfies Defini- tion 5 and thus is optimal. However, SA-RL learns MDP whose state space and action space are both the same as the original state space. If the original state space is high-dimensional (e.g. images), learning good policy in the adversary's ' MDP may become computationally intractable, as empirically shown in Section 6.
Can address the optimal attacking problem in an efficient manner? SA-RL treats the victim and the environment together as a black box and directly learns a state adversary. But if the victim policy known to the attacker (e.g. in adversarial training), we can exploit the victim model and simplify the attacking problem while maintaining the optimality. Therefore, we propose novel algorithm, Policy Adversarial Actor Director (PA-AD), that has optimal formulation and is generally more ef- ficient than SA-RL. PA-AD decouples the whole attacking process into two simpler components: policy perturbation and state perturbation, solved by a "director" and an "actor" through collabora- tion. The director learns the optimal policy perturbing direction with RL methods, while the actor crafts adversarial states at every step such that the victim policy is perturbed towards the given di- rection. Compared to the black-box SA-RL, PA-AD is a white-box attack, but works for a broader range of environments more efficiently. Note that PA-AD can be used to conduct black-box attack based on the transferability of adversarial attacks (Huang et al., 2017), although it is out of the scope of this paper. Appendix F.2 provides a comprehensive comparison between PA-AD and SA-RL in terms of complexity, optimality, assumptions and applicable scenarios.
A Heuristic Attacker: Efficient But Non-optimal
Our Method: Policy Adversarial Actor Director:
Optimal And Efficient
State Adversary
State
Environmant
a
Rowad
Envirenment
Director 1
(8)
State
Action
Vicim Policy
X (
End-to-end RL Attacker (SA-RL): Optimal But Inefficient
Palay Perturting Directin &
State
Reward Stata
Enviranment
Action
Actor g
Victim Palkc
Adversary
(1)
actor's similor to (targetec) evasion sofved by optimtzation methods (2 (PGSM, PGD, etc).
supervised leaming.
Victin Policy
The dircctrrs task minimize the reward gained from the environment. Can be sohvdd by Rt. methoele (PP0) DQN, enc).
An overview of PA-AD compared with heuristic atlacker and end-to-end RL attacker. Heuristic attacks are efficient, but may not find the optimal adversary as they do not learn from the environment dynamics. An end-to-end RL. attacker directly learns a policy to generate state perturbations, but is inefficient in large-state- space environments. In contrast, our PA-AD solves the attack problem with a combination of an RL-based director and non- RL actor, that PA-AD achieves both optimality and efficiency.
Figure 4:
5

---

Published as a conference paper at ICLR 2022
Formally. for given victim policy म, our proposed PA-AD algorithm solves Policy Adversary MDP (PAMDP) defined in Definition An actor denoted by g is embedded in the dynamics of the PAMDP, and director searches for optimal policy v" in the PAMDP.
Definition 6 (Policy Adversary MDP (PAMDP) M). Given an MDP M (S.A, R. fixed stochastic victim policy n, an attack budget e 2 O, we define a Policy Adversarial MDP M = (S.A, R 7), where the action space is Ā= {d El-1,1]|A1 Els di = 0}, and Vs, E
S, Va E Å.
a()lss,))=> = E.ea m(alg(ā, s)) s,a), A(s,a) = -
- Ded (l))
where is the actor function defined as
g(ō, s) = argmax;en.(s) (=) T())l subject to (Λ(3) -
- (G)
If the victim policy is deterministic, i.., TD = argmax, (alss) (subscript stands for determin- istic), the action space of PAMDP is Ap: = A. and the actor function 9D is
gD(ā, argmax;eB. (e) (π(a()) maXuEAAaa
(GD)
Detailed definition of the deterministic-vicciim version of PAMDP is in Appendix C.I.
key to PA-AD the director-actor collaboration mechanism. The input to director policy v the current state in the original environment, while its output ã is signal to the actor denoting "which direction to perturb the victim policy into". A is designed contain all "perturbing directions" in the policy space. That is, Vā A, there exists a constant 0g such that v0 0o T[-ss) + 0 +0 belongs to the simplex (A). The actor takes in the state & and director's direction â and then computes state perturbation within the attack budget. Therefore, the director and the actor together induce a state adversary: 1(()) == g(v(s), s). Vs E S. The definition of PAMDP is slightly different for a stochastic victim policy and a deterministic victim policy, as described below.
[[a
For a stochastic victim π, the director's action ă A designed to be unit vector lying in the policy simplex, denoting the perturbing direction in the policy space. The actor, once receiving the perturbing direction ā, will "push" the policy as far as possible by perturbing s to g(ā, 8) E (s), characterized by the optimization problem (G). In this way, the policy perturbation resulted by the director and the actor is always in the outermost boundary of B (т) w.r.t. the victim n, where the optimal policy perturbation can be found according Theorem 4.
For deterministic victim T the director's action â E AD can be viewed original action space, and the actor conducts targeted attacks to let the victim execute a, by forcing the logit corresponding to the target action to be larger than the logits of other actions.
target action in the
In both the stochastic-victim and deterministic-vietin case, PA-AD has an optimal formulation as stated in Theorem (proven in Appendix D.2).
Theorem 7 (Optimality of PA-AD). For any MDP M, any fixed victim policy and any altack budget O. optimal policy v" in M induces optimal state adversary against in M. That is, the formulation of PA-AD is optimal, Le., PADDDD C H
Algo Algorithm 1: Policy Adversarial Actor Director (PA-AD)
Input: Initialization of director's policy v; victim policy æ; budget e; start state 80
for f. = 0, 1,2.... do
Director samples a policy perturbing direction â ~ v(([s:)
Actor perturbs 5, 5, gD (ā,, st) if Victim is deterministic, otherwise to 5; = g(āes st)
Victim takes action a; ~ x-112t) proceeds to 81+1, receives
Director saves (st, a. -T', 11+1) ) to its buffer
Director updates its policy using any RL algorithm
Efficiency of PA-AD As commonly known, the sample complexity and computational cost of learning an MDP usually grow with the cardinalities of its state space and action space. Both SA- RL and PA-AD have state space S, the state space of the original MDP, But the action space of SA-RL is also S, while our PA-AD has action space RIAl for stochastic victim policies, or AA for deterministic victim policies. In most DRL applications, the state space (e.g., images) is much larger than the action space, then PA-AD is generally more efficient than SA-RL as it learns smaller MDP.

---

Published as a conference paper at ICLR 2022
The attacking procedure is illustrated in Algorithm 1. At step t, the director observes state st, and proposes policy perturbation Qr then the actor searches for a state perturbation to meet the policy perturbation. Afterwards, the victim acts with the perturbed state $t, then the director updates its policy based on the opposite value of the victim's reward. Note that the actor solves a constrained optimization problem, (GD) (G). Problem (Gp) is similar
targeted attack in supervised learning, while the stochastic version (G) can be approximately solved with a Lagrangian relaxation. In Appendix C.2, we provide our implementation details for solving the actor's optimization, which empirically achieves -fte-of-thh-arr attack performance as verified in Section 6.
Extending to Continuous Action Space Our PA-AD can be extended to environments with con- tinuous action spaces. where the actor minimizes the distance between the policy action and the target action, i.e., argmin (s) (s() all. More details and formal definitions of the variant of PA-AD in continuous action space are provided in Appendix C.3. In Section 6, we show experimen- tal results in MuJoCo tasks, which have continuous action spaces.
5 RELATED WORK
Heuristic- Evasion Attacks on States There are many works considering evasion attacks on the state observations in RL. Huang et al. (2017) first propose to use FGSM (Goodfellow et al., 2015) to craft adversarial states such that the probability that the agent selects the "best" action is minimized. The same objective is also used in a recent work by Korkmaz (2020), which adopts Nesterov momentum-basee based optimization method to further improve the attack performance. Pat- tanaik al. (2018) propose to lead the agent to select the "worst" action based on the victim's Q function and use gradient descent to craft state perturbations. Zhang et al. (2020a) define the con- cept of aave-aaversarial MDP (SAMDP) and propose two attack methods: Robust SARSA and Maximal Action Difference. The above heuristic-based methods are shown to be effective in many environments, although might not find the optimal adversaries, as proven in Appendix D.3.
RL-based Evasion Attacks on States As discussed in Section 4, SA-RL (Zhang et al., 2021) uses an end-to-end RL formulation to learn the optimal state adversary, which achieves state-of- the-art attacking performance in MuJoCo tasks. For a pixel state space, an end-to-end RL attacker may not work as shown by our experiment in Atari games (Section 6). Russo & Proutiere (2021) propose to use feature extraction to convert the pixel state space to small state space and then learn an end-to-end RL attacker. But such feature extractions require expert knowledge and can be hard to obtain in many real-world applications. In contrast, our PA-AD works for both pixel and vector state spaces and does not require expert knowledge.
Other Works Related to Adversarial RL There are many other papers studying adversarial RL from different perspectives, including limited-stepp attacking (Lin et al., 2017; Kos & Song, 2017), multi-agent scenarios (Gleave et al., 2020), limited access to data (Inkawhich et al., 2020), and etc. Adversarial action attacks (Xiao et al., 2019; Tan et al., 2020; Tessler et al., 2019; Lee et al., mainly consider state adversaries,
2021) are developed separately from state attacks; although our PA-AD can be extended to action attacks as formulated in Appendix A. Poisoning (Behzadan & Munir, 2017; Huang & Zhu, 2019; Sun et al., 2021; Zhang et al., 2020b; Rakhsha et al., 2020) is another type of adversarial attacks that manipulates the training data, different from evasion attacks that deprave a well-trained policy. Training a robust agent is the focus of many recent works (Pinto et al., 2017: Fischer et al., 2019; Lütjens et al., 2020; Oikarinen et al., 2020; Zhang et al., 2020a; 2021). Although our main goal is to find a strong attacker, we also show by experiments that our proposed attack method significantly improves the robustness of RI. agents by adversarial training.
6 EXPERIMENTS
In this section, show that PA-AD produces stronger evasion attacks than sfate-of-the-arr attack algorithms on various OpenAl Gym environments, including Atari and MuJoCo tasks. Also, our experiment justifies that PA-AD can evaluate and improve the robustness of RL agents.
Baselines and Performance Metric We compare our proposed attack algorithm with existing evasion attack methods, including MinBest (Huang et al., 2017) which minimizes the probability that the agent chooses the "best" action, MinBest +Momentum (Korkmaz, 2020) which uses Nesterov momentum improve the performance of MinBest, MinQ (Pattanaik et al., 2018) which leads
7

---

Published as a conference paper at ICLR 2022
Environment
Natural Reward
Random
MinBest
MinBest
MinQ
MaxDin
SA-RL
PA-AD
Momentum
(ours)
Boxing
9
0.001
96 =
53
52
58:7
95 =
94::
19 19:17 1I
Pong
21 + D
0.0002
21 +
10
+14
144:3 3
15 4
20 +: 1
21 d
RoadRunner
46278
4447
0.0005
44725
6614 17012
6243 15823 + 625227655 + 123331 30074 + 6544 43615 + 7183
0 0
DQN
Freeway
84
0.0003
34 1:1
12 1:1
12 22::1 1
15 1::2 2
22 4: 3
34 1: 1
9 9
Seaquest
10650 2716 0.0005
4: 2062
18820 4: 1947
2337 962
6468 2493
5718 1884 8152 3113 2304 1:
Alien
1623 + 252 0.00075
1650
381
819 4 486
775 648
938
446
869 + 279
1693 + 439
256 +: 210
Tutankham
227 + 29
0.00075
221 + 65
30 + 13
26 + 16
88 74
130 + 48
202 + 65
0
Breakout
356 79
0.0005
355 79
86 = 104
74 95
N/A
304 111
353 +
44 d 62
Seaquest
1752 + 70
0.005
1752 +: 73
356 +: 153
179 +: 83
N/A
46 52
1752 71
4 13
A2C
Pong
20
0.0005
20 +
-4 1 8
-11 7
NIA
18 3
20 +: 1
- -13 6
Alien
1615 601
0.001
1629
592
1062 610
910 565
N/A
1/182 = 633
1661 + 625
507 278
Tutankhum
258 # 53
0.001
260
54
139
26
134 + 28
N/A
196 +
260
71 47
RoadRunner 34307 6355
0.002
35851 6675 9198 + 3814 5410 +: 3058
N/A
31 8D6 + 7125 36550 + 2778 + 3468
Table I: Average episode rewards standard deviation of vanilla DQN and A2C agents under different evasion attack methods in Atari environments. Results are averaged over 1000 episodes. Note that RS works for continuous action spaces, thus is not included. MinQ is not applicable to A2C which does not have Q network. In each row, we bold the strongest (hest) attack performance over all attacking methods.
the agent to select actions with the lowest action values based on the agent's Q network, Robust SARSA (RS) (Zhang et al., 2020a) which performs the MinQ attack with a learned stable Q network, MaxDiff (Zhang et al., 2020a) which maximizes the KL-ddivegencce between the original victim policy and the perturbed policy, as well as SA-RL (Zhang et al., 2021) which directly learns the state adversary with RL methods. We consider state attacks with for norm as in most literature (Zhang et al., 2020a; 2021). Appendix 1 provides hyperparameter settings and implementation details.
PA-AD Finds the Strongest Adversaries in Atari Games We first evaluate the performance of PA-AD against well-trained DQN (Mnih et al., 2015) and A2C (Mnih et al., 2016) victim agents
on Atari games with pixel state spaces. The observed pixel values are normalized to the range of [0, 1]. SA-RL and PA-AD adversaries are learned using the ACKTR algorithm (Wu et al., 2017) with the same number of steps. (Appendix E.1 shows hyperparameter settings.) Table I presents the experiment results, where PA-AD significantly outperforms all baselines against both DQN and A2C victims. In contrast, SA-RL does not converge to good adversary in the tested Atari games with the same number of training steps as PA-AD, implying the importance of sample efficiency. Surprisingly, using relatively small attack budget e, PA-AD leads the agent to the lowest possible reward many environments such as Pong, RoadRunner and Tutankham, whereas other attackers may require larger attack budget to achieve the same attack strength. Therefore, we point out that vanilla RL agents are extremely vulnerable to carefully learned adversarial attacks. Even if an RL agent works well under naive attacks, carefully learned adversary can let agent totally fail with the same attack budget, which stresses the importance of evaluating and improving the robustness of RL agents using the strongest adversaries. Our further investigation in Appendix F.3 shows that RL models can be generally more vulnerable than supervised classifiers, due to the different loss and architecture designs. In Appendix 1.2. we show more experiments with various selections of the budget e, where one can see PA-AD reduces the average reward more than all baselines over varying e's in various environments.
PA-AD Finds the Strongest Adversaries MuJoCo Tasks We further evaluate PA-AD on Mu- JoCo games, where both state spaces and action spaces are continuous. We use the same setting with Zhang et al. (2021), where both the victim and the adversary are trained with PPO (Schulman et al., 2017). During test time, the victim executes a deterministic policy, and we use the deterministic
Environment
State
Natural Reward
Random
MaxDiff'
RS
SA-RL
PA-AD (ours)
Dimension
Hopper
3167 542
0.075 2101
793
1410 +: 655
794 : 238
636 : 9
160 + 136
Walker
4472 635
0.05
3007 + 1200
2869
1271
1336 654
1096 + 516
804 4 130
HallCheetah
7117 + 98
0.15
5486 + 1378
1836 + 866
489 + 758
-660
218
-356 + 307
Ant
1I1
5687 + 758
0.15
5261 # 1005
1759 + 828
268 # 227
-872 436
- 2580 + 872
Table 2: Average episode rewards standard deviation of vanilla PPO agent under different evasion attack methods in MuJoCo environments. Results are averaged over 50 episodes. Note that MinBest and MinQ do not fit this setting, since MinBest works for discrete action spaces, and MinQ requires the agent's Q network.
8

---

Published as a conference paper at ICLR 2022
version of PA-AD with a continuous action space, as discussed in Section 4 and Appendix C.3. We use the same attack budget e as in Zhang et al. (2021) for all MuJoCo environments. Results in Ta- ble 2 show that PA-AD reduces the reward much more than heuristic methods, and also outperforms SA-RL in most cases. In Ant, our PA-AD achieves much stronger attacks than SA-RI, since PA-AD) is more efficient than SA-RL when the state space is large. Admittedly, PA-AD requires additional knowledge of the victim model, while SA-RL works in a black-box setting. Therefore, SA-RL is more applicable to black-box scenarios with a relatively small state space, whereas PA-AD is more applicable when the attacker has access to the victim (e.g. in adversarial training as shown in Ta- ble 3). Appendix E.2.3 provides more empirical comparison between SA-RL and PA-AD, which shows that PA-AD converges faster, takes less running time, and is less sensitive to hyperparamelers than SA-RI by a proper exploitation of the victim model.
Environment
Model
Natural Reward
Random
MaxDiff
SA-RI
PA-AD (ours)
Average reward acress attacks
Hopper
SA-PPO
3705 2 2710 a 501
2652 a 835
1130 42
1076 11766.77 791
856 1: 21
1684.8
(state caste-dim: 11)
ATLA-PPO
3291 + 000 3165 + 676
2814 + 726
2244 + 618
1772
802
1282 360
2246-4
0.075
PA-ATLA-PPO (ours) 3449 1 237
3325 4: 230
3145 a: 546
3002 d: 129
1529 1: 284
2521 1: 325
2704.4
Walker
SA-PPO
1187 61
4867 4 39
3668
4: 1789
3808
=: 138
2908 #: 1136
1042 + 153
3258.6
(state-dim: 17)
ATL.A-PPO
3842 475
3927
368
3836
400
32339
804
3603
707
1224
770
3177.8
0.05
PA-ATIA-PPO (ours) 4178 + 529
4129 + 78
4024
572
3068 + 307
3450
478
2248 1: 131
356344
Halfcheetah
SA-PPO
3632 1:20
3819 4: 18
3624 =: 23
3283 1: 20
3028 à: 23
2512 1: 16
3213.2
(state dim: 17)
ATLA-PPO
6167 + 852
6164
+ 603
5790 + 174
4506 =: 603
5008 718
2576 + 15448
4878.8
0.155
PA-ATLA- (ours) 342
6215
346
5981
5226
114
4572
3840
673
Ant (state-imm 111) 0.15
SA-PPO
4986
0682
522
3112
1755
-129
923
2855.0
API.A-PPO
5359 + 153 5366 + 104 5240 + 170
41:36 + 140
3765 + 101
220 d: 838
3745.4
PA-ATLA-PPO (ours) 5460 + 106 5496 158 5328
196
41:24
291
3004 188
2986 1 864
4325-6
Table 3: Average episode rewards standard deviation of robustly trained PPO agents under different attack methods. Results are averaged over 50 episodes. In each row corresponding to a robust agent, we bold the strongest attack. The gray cells are the most robust agents with the highest average rewards across attacks. Our PA-AD achieves the strongest attack against robust models, and our PA-ATLA-PPOO achieves the most robust performance under multiple attacks. The attack budget e's are the same as in Zhang et al. (2021).
Training and Evaluating Robust Agents A natural application of PA-AD is to evaluate the bustness of a known model, or to improve the robustness of an agent via adversarial training, where the attacker has white-box access the victim. Inspired by ATLA (Zhang et al., 2021) which al- ternately trains an agent and an SA-RL attacker, we propose PA-ATLA, which alternately trains an agent and PA-AD attacker. In Table 3, we evaluate the performance of PA-ATLA for a PPO agent (namely PA-ATLA-PPOO in MuJoCo tasks, compared with state-of-thee art robust training methods, SA-PPO (Zhang et al., 2020a) and ATLA-PPO (Zhang et al., 2021) 2. From the table, we make the following observations. (1) Our PA-AD) atlacker can significuntly reduce the reward of previ- ous "robust" agents. Take the Ant environment as an example, although SA-PPO and ATLA--PO agents gain 2k+ and 3k+ rewards respectively under SA-RL, the previously strongest attack, our PA- AD still reduces their rewards to about -1.3 and 200+ with the same attack budget. Therefore, we emphasize the importance of understanding the worst-case performance of RL agents, even robustly- trained agents. (2) Our PA-ATLA-PPO robust agents gain noticeably higher average rewards across attacks than other robust agents, especially under the strongest PA-AD attack. Under the SA-RL attack, PA-ATLA-PPO achieves comparable performance with ATLA-PPO, although ATLA-PPO agents are trained to be robust against SA-RL. Due the efficiency of PA-AD, PA-ATLA-PPO requires fewer training steps than ATLA-PPO, as justified in Appendix E.2.4. The results of attack- ing and training robust models in Atari games are in Appendix E.2.5 and E.2.6, where PA-ATLA improves the robustness of Atari agents against strong attacks with c as large as 3/255.
7 CONCLUSION
In this paper, we propose an attack algorithm called PA-AD for RL problems, which achieves opti- mal attacks in theory and significantly outperforms prior attack methods in experiments. PA-AD can be used to evaluate and improve the robustness of RL agents before deployment. A potential future direction is to use our formulation for robustifying agents under both state and action attacks.
We use
Reg, the most robust method reported by Zhang et al. (2021).

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work is supported by National Science Foundation IIS-1850220 CRII Award 030742-00001 and DOD-DARPA-Defemte Advanced Research Projects Agency Guarantecing Al Robustness against Deception (GARD), and Adobe, Capital One and JP Morgan faculty fellowships.
ETHICS STATEMENT
Despite the rapid advancement of interactive Al and ML systems using RL agents, the learning agent could fail catastrophically in the presence of adversarial attacks, exposing a serious vulnerability in current RL systems such as autonomous driving systems, market-making systems, and security monitoring systems. Therefore, there is an urgent need to understand the vulnerability of RL. model, otherwise, it may be risky to deploy a trained agent in real-life applications, where the observations of a sensor usually contain unavoidable noise.
Although the study of a strong attack method may be maliciously exploited to attack some RL systems, it is more important for the owners and users of RL systems to get aware of the vulnerability of their RL agents under the strongest possible adversary. As the old saying goes, "if you know yourself and your enemy, you'll never lose a battle". In this work, we propose an optimal and efficient algorithm for evasion attacks in Deep RL (DRL), which can significantly influence the performance of a well-trained DRL agent, by adding small perturbations to the state observations of the agent. Our proposed method can automatically measure the vulnerability of an RL agent, and discover the "flaw" in a model that might be maliciously attacked. We also show in experiments that our attack method can be applied to improve the robustness of an RL agent via robust training- Since our proposed attack method achieves sfate-of-the-ar performance, the RL agent trained under our proposed attacker could be able "defend" against any other adversarial attacks with the same constraints. Therefore, our work has the potential to help combat the threat to high-stakes systems.
A limitation of PA-AD is that it requires the "attacker" to know the victim's policy, i.e., PA-AD is white-box attack. If the attacker does not have full access to the victim, PA-AD can still be used based on the transferability of adversarial attacks (Huang et al., 2017), although the optimal- ity guarantee does not hold in this case. However, this limitation only restricts the ability of the malicious attackers. In contrast, PA-AD should be used when one wants to evaluate the worst-case performance of one's own RL agent, or to improve the robustness of an agent under any attacks, since PA-AD produces strong attacks efficiently. In these cases, PA-AD does have white-box access to the agent. Therefore, PA-AD is more beneficial to defenders than attackers.
REPRODUCIBILITY STATEMENT
For theoretical results, we provide all detailed technical proofs and lemmas in Appendix. In Ap- pendix A, analyze the equivalence between evasion attacks and policy perturbations. In Ap- pendix B, we theoretically prove some topological properties of the proposed Adv-policy-set, and derive Theorem 4 that the outermost boundary of B. (T)) always contains optimal policy per- turbation. In Appendix D, we systematically characterize the optimality of many existing attack methods. We theoretically show (1) the existence of an optimal adversary, (2) the optimality of our proposed PA-AD, and (3) the optimality of many heuristic attacks, following our Definition 5 in Section 4.
For experimental results, the detailed algorithm description in various types of environments is provided in Appendix C. In Appendix F, we illustrate the implementation details, environment set- tings, hyperparameter settings of our experiments. Additional experimental results show the per- formance of our algorithm from multiple aspects, including hyperparameter sensitivity, learning efficiency, etc. In addition, in Appendix F,, we provide some detailed discussion on the algorithm design, as well as a comprehensive comparison between our method and prior works.
The source code and running instructions for both Atari and MuJoCo experiments are in our sup- plementary materials. We also provide trained victim and attacker models so that one can directly test their performance using a test script provide.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining Pattern Recogni- tion, pp. 262-275. Springer, 2017.
Robert Dadashi, Adrien Ali Taiga, Nicolas Le Roux, Dale Schuurmans, and Mare G. Bellemare. The value function polytope in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhut- dinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1486-1495, Long Beach, California, USA, 09-15 Jun 2019, PMLR.
Mare Fischer. Matthew Mirman, Steven Stalder, and Martin Vechev. Online robustness training for deep reinforcement learning. arXiv preprint arXiv:1911.0053 2019.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations, 2020.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Intermational Conference on Learning Representations, 2015.
David Silver Hado Van Hasselt, Arthur Guez. Deep reinforcement learning with double q-larnning. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Ashley Hill, Antonin Rafin,, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alee Rad- ford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: https://github. com/hi hill-ahlla a st www.abbleeblllleeee ines, 2018.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks neural network policies. arXiv preprint 2702:v:7722.222220 2017.
Yunhan Huang and Quanyan Zhu. Deceptive reinforcement learning under adversarial manipula- tions on cost signals. In International Conference on Decision and Game Theory for Security, pp- 217-237. Springer, 2019.
Matthew Inkawhich, Yiran Chen, and Hai Li. Snooping attacks on deep reinforcement learning- In Proceedings of the 19th International Conference on Autonomous Agents and MulriAgent Sys- tems, AAMAS '20, pp. 557-565, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781 9781450375188 844.
Ezgi Korkmaz. Nesterov momentum adversarial perturbations in the deep reinforcement learn- ing domain. In ICML 2020 Inductive Biases. Invariances and Generalization Reinforcement Learning Workshop, 2020.
Jernej Kos and Dawn Song. Delving into adversarial attacks deep policies. 17Xiv:170S.06452. 2017.
arXiv preprint
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://
github. com htt///kkkttkko//yorrrtttt alc a2c-ppo- aeto-acktr--oo gair, 2018.
Xian Yeow Lee, Yasaman Esfandiari, Kai Liang Tan, and Soumik Sarkar. Query- targeted action-space adversarial policies on deep reinforcement learning agents. In Proceedings of the ACM/IÉEE 12th International Conference on Cyber- Systems, ICCPS '21 pp. 87-97, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 9781450383530.
Yen-Chen Lin, Zhang- Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yt Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. In Proceedings of the 26th International Joint Conference Artificial Intelligence, IJCA 17, pp. 3756-3762, AAAI Press, 2017. ISBN 9780999241103.
Bjöm Lütjens, Michael Everett, and Jonathan How. Certified adversarial robustness for deep reinforcement learning. In Conference Robot Learning, pp. 3238-133. PMLR, 2020.
11

---

Published as a conference paper at ICLR 2022
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7500):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1932 2016.
Tuomas Oikarinen, Tsui-Wei Weng, and Luca Daniel. Robust deep reinforcement learning through adversarial loss. arXiv preprint arXiv:2008.00 01976, 2020.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '18, pp. 2040-2044 Rich- land, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Koarning-Volunee 70, pp. 2817 2817-2826. JMLR. org, 2017.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779.
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training- timeingg-tmeeeeee adversarial attacks against reinforcement learning. In International Conference on Machine Learning, pp. 7974-7984, 2020.
Alessio Russo and Alexandre Proutiere. Optimal attacks reinforcement learning policies. In American Control Conference (ACC)., 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 107.00777 2017.
Yanchao Sun, Da Huo, and Furong Huang. Vulnerability-awar poisoning mechanism for online rl with unknown dynamics. In International Conference Learning Representations, 2021.
Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, Soumik Sarkar, et al. Robustifying reinforce- ment learning agents via action space adversarial training. In 2020 American control conference (ACC), pp. 3959-3964. IEEE, 2020.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control, In International Conference on Machine Learning, pp. 6215-6224. PMLR, 2019.
Ioannis Antonoglou Tom Schaul, John Quan and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016.
Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factonn approximation. Advances in neural information processing systems, pp. 5279-5288, 2017.
Chaowei Xiao, Xinlei Pan, Warren He, Jian Peng. Mingjie Sun, Jinfeng Yi. Mingyan Liu, Bo Li. and Dawn Song. Characterizing attacks on deep reinforcement learning. arXiv preprint arXiv:1907.000 2019.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforeement learning against adversarial perturbations state observations. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21024-21037.
Curran Associates, Inc., 2020a.
12

---

Published as a conference paper at ICLR 2022
Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsich. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations, 2021.
Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning, 2020b.
13

---

Published as a conference paper at ICLR 2022
Appendix: Who Is the Strongest Enemy? Towards Optimal and Effi- cient Evasion Attacks in Deep RL
A RELATIONSHIP BETWEEN EVASION ATTACKS AND POLICY PERTURBATIONS.
As mentioned in Section 2, all evasion attacks can be regarded as perturbations in the policy space. To be more specific, we consider the following cases, where we assume the victim uses policy n.
Case (attack on states): define the state adversary as function such that Vs E S
h(s) E (s) = fs' S ||s - =|l S c).
(For simplicity, we consider the attacks within c-radius norm ball.)
In this case, for 8 E S, the victim samples action from n (-1s) = x x-11(1)) = =x()), which equivalent to the victim executing perturbed policy II.
Case 2 (attack on actions for deterministic T): define the action adversary as function ((
8 x A
A, and Vs E S, A
A(A) (als) = à (a = fa' E A: Ila' - all S ct.
In this case, there exists a policy (A(A) such that TACO)) (s) (a)ss a, which is equivalent to the victim exccuting policy T((A) E II.
Case (attack on actions for stochastic T): define the action adversary as function ((A); : A, and Vs 8,a A
(a) 8) = à such that (1|=(-1s) - Pr(-1s)] S <ch
where Pr(āls) denotes the probability that the action is perturbed into ā. In this case. there exists a policy TT(()) such that (s) Pr(+ss), which is equivalent to the victim executing policy *((A) E IT.
Most existing evasion RL works (Huang et al., 2017; Pattanaík et al., 2018; Zhang et al., 2020a; 2021) 1) focus on state attacks, while there are also some works (Tessler et al., 2019; Tan et al., 2020) studying action attacks. For example, Tessler al. (Tessler et al., 2019) consider Case and Case above and train agent that is robust to action perturbations.
These prior works study either state attacks or action attacks, considering them in two different scenarios. However, the ultimate goal of robust RL is train RL agent that is robust to any threat models. Otherwise, an agent that is robust against state attacks may still be ruined by an action attacker. We take step further to this ultimate goal by proposing framework, policy attack, that unifies observation attacks and action attacks.
Although the focus of this paper is on state attacks, would like point out that proposed method can also deal with action attacks (the director proposes policy perturbation direction, and an actor perturbs the action accordingly). It is also exciting direction explore hybrid attacks (multiple actors conducting states perturhations and action perturbations altogether, directed by a single director.) Our policy perturbation framework can also be easily incorporated in robust training procedures, as an agent that is robust to policy perturbations is simultaneously robust to both state attacks and action attacks.
B TOPOLOGICAL PROPERTIES OF THE ADMISSIBLE ADVERSARIAL POLICY SET
As discussed in Section 3, finding the optimal state adversary in the admissible adversary set He can be converted to a problem of finding the optimal policy adversary in the Adv-policy-set B (T)). In this section, we characterize the topological properties of B (T). and identify how the value function changes the policy changes within 13d (x).
In Section B. we show that under the settings we consider, T (TT) is a connected and compact subset of IT. Then, Section B.2, we define some additional concepts and formulate the notations.
14

---

Published as a conference paper at ICLR 2022
In Section B.3, prove Theorem 4 in Section that the outermost boundary of 3 (T) always contains an optimal policy perturbation. In Section B.4, we prove that the value functions of policies in R (17) (or more generally, any connected and compact subset of II) form polytope. Section 6.6 shows an example of the polytope result with 2-state MDP, and Section B.5 shows examples of the outermost boundary defined in Theorem 4.
B.1 THE SHAPE OF ADV-POLICY-SET B (TT)
It is important to note that _ (Ir) is generally connected and compact as stated in the following lemma.
Lemma (B (T) is connected and compact), Given an MDP M. a policy that is continuous mapping, and admissible adversary set H. == { h(s) E (s(s), Vs S} (where >> is constant), the admissible adversarial policy set B (π) is connected and compact subset of
Proof of Lemma 8. For arbitrary state S, an admissible adversary h He perturbs it within an E norm ball (8), which connected and compact. Since π is a continuous mapping, we know (s) ) is compact and connected.
Therefore, = (T) as a Cartesian product of a finite number of compact and connected sets, is com- pact and connected.
B.2 ADDITIONAL NOTATIONS AND DEFINITIONS FOR PROOFS
We first formally define some concepts and notations.
For stationary and stochastic policy π A(A), we can define the state-to-state transition function as
pd (*(('ss Σ
aEA
and the state reward function as
Rd (*) = Σ (()
aEA
Then the value of n, denoted as Va, can computed via the Bellman equation
V Rd +YP"
(1-
We further use II, to denote the projection of II into the simplex of the i-th state, i.e. the space of action distributions at state s;.
Let fo II + gl be mapping that maps policies to their corresponding value functions. Let == (11) be the space of all value functions.
Dadashi et al. (Dadashi et al., 2019) show that the image of fo applied to the space of policies, i.e., (II), form a (possibly non-convex) polytope as defined below.
Definition (PPosiibly non-convex) polytope). A is called convex polytope iff there are i E N points 3 I2,... , E R" such that A Conw(x,, ,ıx). Furthermore, (possibly non- convex) polytope defined as a finite union of convex polytopes.
And more general concept (possibly non-convex) polyhedron, which might not be bounded. Definition 10 (Posssibly non-convex) polyhedron). A is called convex polyhedron iff is the intersection of k. N half-spaces B1, B2, Br, iee. = n B, Farthermore, (possibly non-convex) polyhedron is defined as a finite union of convex polyhedra.
$1 be the set of policies that agree with on states 81. 8g- Dadashi et al. (Dadashi et al., 2019) also prove that the values of policies that agree all but state i.e., fo (Y form a line segment, which can be bracketed by two policies that are deterministic on s. Our Lemma 14 extends this line segment result to our setting where policies are restricted a subset of policies.
In addition, let y*
15

---

Published as a conference paper at ICLR 2022
B.3 PROOF OF THEOREM 4: BOUNDARY CONTAINS OPTIMAL POLICY PERTURBATIONS
Lemma 4 in Dadashi et al. (2019) shows that policies agreeing on all but one state have certain monotone relations. We restate this result in Lemma 11 below.
Lemma 11 (Monotone Policy Interpolation). For any Tu, E Y S) that agree with on all
states except for S, define function I: [0, 1] V as
I(a) = fo(ami + (1 a)o)).
Then we have
(1) > 1(1) or (11 > 1(0) (>= stands for element-wise greater than or equal to):
(2) [f( = ((1), then laa) = 1(0), Va E [0,
(3) ((0) + 1(1), then there strictly monotonic rational function p [0, - R, such that
((a) = p(a)((1)
(()))
More intuitively, Lemma suggests that the value of x" == am1 (1 ao changes (strictly)
monotonically with o, unless the values of To, Th and Ta all equal. With this result, we proceed to prove Theorem 4.
Proof of Theorem 4. We will prove the theorem by contradiction.
Suppose there is policy î E B. (r) such that स & B (T) and f.) = V* < V VI B. (T), i.e, there no optimal policy adversary on the outermost boundary of B (T).
Then according to the definition of 0,, (T), there exists at least one state E S such that we can find another policy x' B (T)) agreeing with on all states except for s, where ' (s) satisfies
*(-1s) = an (-1s) + (1 a1)= (-1s)
for scalar E (0, 1).
Then by Lemma 11, either of the following happens:
(1) f+(m)) f.(#) fx(x))
(2) fm() f.((t) = fo()');
Note that (+) > (T) is impossible because we have assumed â has the lowest value over all policies in _ (T) including
If (1) is true, then assumption.
better policy adversary than in B (r), which contradicts with the
(2) is true, then n' another optimal policy adversary. By recursively applying the above process to n finally find an optimal policy adversary on the outermost boundary of B. (r), which also contradicts with our assumption.
In summary, there is always optimal policy adversary lying the outermost boundary of B (T).
B.4 PROOF OF THEOREM 12: VALUES OF POLICIES IN ADMISSIBLE ADVERSARIAL POLICY SET FORM A POLYTOPE
We first present a theorem that describes the "shape" of the value functions generated by all admis- sible adversaries (admissible adversarial policies).
Theorem 12 (Policy Perturbation Polytope). For finite MDP M, consider policy and an Adv-policy-set B (T). The space of values (a subspace of (R of all policies in B (T), denoted by (T), is (possibly non-convex) polytope.
In the remaining of this section, we prove more general version of Theorem as below. Theorem 13 (Policy Subset Polytope). For finite MDP M. consider connected and compact subset of II, denoted as T. The space of values (a subspace of (R of all policies in T. denoted by yT, is (possibly non-convex) polytope.
16

---

Published as a conference paper at ICLR 2022
According Lemma 8, B (Tr) is connected and compact subset of II, thus Theorem 12 is special case of Theorem 13.
Additional Notations To prove Theorem 13, we further define variant of Y which is the set of policies that are in T and agree with æ on states $1, i.e,
as T 7
Tel on == { T 7 = (s:), vi 1, ,,,,,,k
Note that different from T.A (*). T is no longer restricted under an admissible adversary set and can be any connected and compact subset of IT.
The following lemma shows that the values of policies in T that agree on all but one state form line segment.
Lemma 14. For a policy π E T and an arbitrary state E S, there are two policies in 0, 7x SV(e)
namely ug nig such that V' E T5yss
fo(m)) fo(T') (o(T)))
(3)
where & denotes element-wiss less than equal (if a + b, then a, S for all index i). Moreover, the image of Ju restricted to (stss) is a line segment.
Proof of Lemma 14. Lemma 5 in Dadashi al. (2019) has shown that fo is infinitely differentiable on 11, hence we know fo fT(T)((a)) is compact and connected. According to Lemma 4 in Dadashi al. (2019), for any two policies TI, सं E (at) either fu(m)) & fo(22), (u(T2) < (m() (there exists total order). The same property applies T5, since (5((s) is subset of (5
Therefore, there exists ve and n. that achieve the minimum and maximum over all policies in (.). Next we show n. and T5 can be found the outermost boundary of (5
Assume ... & O77S and for all से E (o) < (. Then we can find another policy x' Tx such that T Lemma 11, (T(')) > f.(=)), contradicting with the assumption. Therefore, one should be able to find policy the outermost boundary of (T5)(s) whose value dominates all other policies. And similarly, we can also find पミ on 0 ((())
+ (1 - a)xx' for some scalar (0, 1). Then according to
Furthermore, (1))) is subset of fo(Y) since ( is a subset of V Si(a)- Given that (o)))))))) is line segment, and (e)) connected, we can conclude that fo (T is also line segment.
Next, the following lemma shows that कद and मद and their linear combinations can generate values that cover the set fo (1))
Lemma 15. For a policy T, an arbitrary state s E S, and क मh defined in Lemma 14, the following three sets are equivalent:
(1) (T
(2) fo (closure(T) (e) where closure(-) is the convex closure of a set;
(3) (folan + a) Ja [0, 1]}:
(4) (afolT)) + (1 (F Jax E 1]}
Proof of Lemma 15. We show the equivalence by showing (1) S (3) (1) as below.
(2) C (1): For any I1, 22 E T (()) without loss of generality, suppose [o(T)) & fo(m2). According Lemma 11, for any
[0, 1], fo( & aTi + (1 a)Tz & {T))))) Therefore, any convex combinations of policies in T ( has value that is in the range of (T, (()))) So the values of
policies in the convex closure of ( do not exceed (T) i.e. (2) (1).
17

---

Published as a conference paper at ICLR 2022
(3) C (2): Based the definition, ang + (1 a)sg closure(T (o))) (3) (2).
(4) S (3): According Lemma 11, there exists a strictly monotonic rational function p [0, 1] + R, such that
I(a) = fo + (11 - ((a). + (1 - ((a))
Therefore, due to intermediate value theorem, for E [0, I]. p(a) takes values from 0 to So (4) (3).
(1) S (4): Lemma 14 shows that fo (T Therefore, for any x' TStee its value is convex combination of fo(x)) and fo(T)))
line segment bracketed by fe(xT) and fo(T,))
Next, show that the relative boundary of the value space constrained T is covered by policies that dominate or are dominated in at least one state. The relative interior of set B is defined as the set of points in A that have relative neighborhood in A n B, denoted relint A. The relative boundary of set A in B, denoted as A, defined as the set of points in that are not in the relative interior of A. i.e., dn A relint When there is ambiguity, we omit the subscript of to simplify notations.
In addition, introduce another notation F 81.--,8 V span
( where CT stands for the i-th column of the matrix -/->P")= Note that F st..._s is the same with H si.--e in Dadashi et al. Dadashi al. (2019), and we change admissible adversary set H. defined in our paper.
F order distinguish from the
Lemma 16. For a policy и E T, k S (S), and a set of policies Te, as that agree with a on $1. 8k (perturb v only at sk+1, (18)), define V == fo(TT se( Define two sets of policies क, (-1x))
x = [n' E ".......x <1(ss) 1000=(((-000 (.1s)) and X,- = { E T "Ti.....s +-(s8) We have that the relative boundary V in FT s1, sr is included in the value functions spanned by policies Ty..... (X U xxz))
for least one 8 (81, ..... , i.e.
[S]
ov c U fo
nx UX,,))
j=k+1
Proof of Lemma 16. We first prove the following claim:
"i."". .az" Vf E {li + 1, ... [S]}. An' E closure(Ty, si ) (X U Claim 1: For a policy To E T
X such that fu(mr') (TT). then (T)) has relative neighborhood in V n Fx si
First, based on Lemma 14 and Lemma 15, that (+t (To) through the following steps:
construct policy स closure(TT) sn) such
Algorithm 2: Constructing
Set * mo
for j = + 1.... |S] do
Find -.. 3, (
Find n² a +1 -1- a,)T,, such that fo(T)) fo(TT)1)
Return a = N [s|
Denote the concatenation of aj's 's as a vector := [02+1. ... d.s]..
According to the assumption that Vj (Ir + 1,... [S]]. 7m' E closure "T. se n UX;;)
ff((m)) (T(),, we have &j & {0, (0,1), vj = k + 1.... .SS.. Then, define a function Φ (0, 1) + Ve such that
such that (((') =
((a) (o(Ta) where (.(3g)))) = .ant, + (1 a)m (.(3))))) = +((18,)
,|S}
otherwise
18

---

Published as a conference paper at ICLR 2022
Then we have that
1. é is continuously differentiable.
2. ((ā) - (f)...
3. do; is non-zero at (because of Lemma 11 (3)).
4. doy is along the i-the column of (I -Pa)-1 (see Lemma 3 in Dadashi et al. Dadashi
et al. (2019)).
Therefore, by the inverse theorem function, there is neighborhood of Φ(a) (T) in the image space.
Now we have proved Claim 1. As result, for any policy E TT . if fo mo)) is in the relative
.6. boundary of V in F "......s"" then 3j E {k + 1, |8]], n' closure (Te n (X+ U (x7)) such that (T fT(T)). Based on Lemma 15, we can also find x" T an (X UX such that Jo (TT") = (mo). So Lemma holds.
Now, we are finally ready prove Theorem 13.
Proof of Theorem 13. We will show that V(81," ,8 S S, the value V = So(Ta .nr) is a polytope.
We prove the above claim by induction on the cardinality of the number of states k. In the base case where k: (S], y (f)(T) is polytope.
Suppose the claim holds for k + 1, then we show also holds for k. i.e., for a policy E II, the value of T,,,-....
.% C ".,,..... .se C II for polytope.
According to Lemma 16, we have
(8)
ov c U fo (xxttxx))))) j=k+1
U v (FFUF)) j=k+1
where 0v denotes the relative boundary of V in FT FT.... .n Ft and F are two affine hyperplanes of F FT. $5, standing for the value space of policies that agree with x, and Tzy in state s, respec- tively.
Then we can get
1. V fo a closed T Tg. is compact and is continuous.
2. av c U == (Fit U F.))), finite number of affine hyperplanes in F Fa,....
3. Vt i FtJ (or Det n F.,)) a polyhedron by induction assumption.
Hence, based Proposition by Dadashi et al. Dadashi et al. (2019), get D is polyhedron. Since V" C V is bounded, we can further conclude that y a polytope.
Therefore, for arbitrary connected and compact set of policies T C II, let iT T be an arbitrary policy in T. then (T) fn((T)) is a polytope.
B.5 EXAMPLES OF THE OUTERMOST BOUNDARY
See Figure 5 for examples of the outermost boundary for different B (Tr'ss
19

---

Published as a conference paper at ICLR 2022
(s)
my (5)
[(a)
(x)))
(s]
my (s)
Ta (5)
Figure 5: Two examples the outermost boundary with |AA| 3 actions at one single state 8. The large triangle denotes the distributions over the action space at state s, i.c., II; Wa, T2 and Na are three policies that deterministically choose d1, d2 and respectively.
is the victim policy, the dark green area is the B.d (=), 8 (=) IIg The red solid curve depicts the outermost boundary of B (T)). Note that policy in the outermost boundary of B (I)) iff it is in the outermost boundary of 8 (T), for all E S.
(M
V152)
yn
V(sa)
Figure Value space an example MDP. The values whole policy space 11 form a polytope (blue) as suggested by Dadashi et al. (2019). The values of all perturbed policies with H, also form polytope (green) suggested by Theorem 12.
B.6 AN EXAMPLE OF THE POLICY PERTURBATION POLYTOPE
example given by Figure 6, where we define an MDP with states and 3 actions. train an DQN agent with one-hot encodings of the states, and then randomly perturb the states within an loo ball with e = 0.8. By sampling 5M random policies, and 100K random perturbations, we visualize the value space of approximately the whole policy space II and the admissible adversarial policy set B (r), both of which are polytopes (boundaries are flat). A learning agent searches for the optimal policy x² whose value is the upper right vertex of the larger blue polytope, while the attacker attempts to find an optimal adversary h², which perturbs a given clean policy to the worst perturbed policy тhe whose value the lower left vertex of the smaller green polytope, This also justifies the fact that learning an optimal adversary is as difficult as learning an optimal policy in an RL problem.
The example MDP Mexi
A 3, 0.8
01,01,
P
[0..9 (0.2,,0 [0.7, 0.3,, [0.05. 0.95], [0.25, 0.75], [0.3, 0.7]
The base/clean policy क:
(a [8) 0.215, m(a2(8)) 0.429, (a3(81)) 0.356 π (8) = 0.271, (T2)(22) 0.592, (a) = 0.137
20

---

Published as a conference paper at ICLR 2022
C EXTENTIONS AND ADDITIONAL DETAILS OF OUR ALGORITHM
C.I ATTACKING A DETERMINISTIC VICTIM POLICY
For deterministic victim ap argmax, 7 (als), we define Deterministic Policy Adversary MDP (D-PAMDP) as below, where a subscript D is added to all components to distinguish them from their stochastic counterparts. In D-PAMDP, the director proposes target action än E A(=: An), and the actor tries its best to let the victim output this target action.
Definition 17 (Deterministic Policy Adversary MDP (D-PAMDP)). Given an MDP M (S, A, P. R),)), a fixed and deterministic victim policy TD, we define a Deterministic Policy Ad- versarial MDP MD = AS,AA,, RD,") 7), where the action space is ĀD Ad. and Vs, E S, E A.
RD(s,ā) -R
The uctor function g is defined as
9D (ā, 8) = argmax (4) (T(a) max
(GD)
The optimal policy of D-PAMDP is an optimal adversary against Tp as proved in Appendix D.2.2
C.2 IMPLEMENTATION DETAILS OF PA-AD
To address the actor function g (or gD) defined in (G) and (G,), we let the actor maximize objectives JD and / within the (-) ball around the original state, for deterministic victim and a stochastic victim, respectively. Below we explicitly define JD and J.
Actor Objective for Deterministic Victim For the deterministic variant of PA-AD, the actor func- tion (GD) is simple and can be directly solved identify the optimal adversary. Concretely, we define the following objective
(D π (ā max.aA.a7a (a)3).
(JD)
which can be realized with the multi-class classification hinge loss. In practice, a relaxed cross- entropy objective can also be used maximize T ((ā
Actor Objective for Stochastic Vietim Different from the deterministic-vietin case, the actor function for stochastic victim defined in (G) requires solving more complex optimization prob- lem with non-convex constraint set, which in practice can be relaxed to (J) (a Lagrangian relax- ation) to efficiently get an approximation of the optimal adversary.
(J) where CosineSim in the second refers to the cosine similarity function; the first term measures how far away the policy is perturbed from the victim policy; À a hyper-paramen parameter controlling the trade- off between the two terms. Experimental results show that our PA-AD is not sensitive the value of x. In our reported results in Section 6, we set À as 1. Appendix E.2.2 shows the evaluation of our algorithm using varying X's.
argmax&eB, (s) J (s; 8)= == (1)) - (-1ss)|| + À x CosineSim (T(13) π (-1s), @)
The procedure of learning the optimal adversary is depicted in Algorithm 3, where we simply use the Fast Gradient Sign Method (FGSM) (Goodfellow al., 2015) approximately solve the actor's objective, although more advanced solvers such as Projected Gradient Decent (PGD) can be applied to further improve the performance. Experiment results in Section 6 verify that the above FGSM- based implementation achieves state-of-the-an attack performance.
What is the Influence of the Relaxation in (J)? First, it is important that the relaxation only needed for a stochastic victim. For a deterministic victim, which is often the case in practice, the actor solves the original unrelaxed objective.
Second, as we will discuss in the next paragraph, the optimality of both SA-RL and PA-AD is re- garding the formulation. That is, SA-RL and PA-AD formulate the optimal attack problem as an MDP whose optimal policy is the optimal adversary. However, in large-scale task, deep RL algo- rithms themselves usually do not converge to the globally optimal policy and exploration becomes the main challenge. Thus, when the adversary's MDP is large, the suboptimality caused by the RL solver due to exploration difficulties could be much more severe than the suboptimality caused by the relaxation of the formulation. The comparison between SA-RL and PA-AD in our experiments
21

---

Published as a conference paper at ICLR 2022
Algorithm 3: Policy Adversarial Actor Director (PA-AD) with FGSM
Input: Initialization of director's policy v; victim policy æ; budget e; start state 8o for i = 1.2...... .......d. do
Director samples a policy perturbing direction 0g ~ v(-1s)) Victim is deterministic then
# for a deterministic victim, Jp is defined in Equation (Jp) Actor computes the gradient of its objective VsJD(s ag else
# for a stochastic victim, J is defined in Equation (J) Actor computes the gradient of its objective VgJ(ss 8; āz 8t)
Actor sets 8t sign(6)
11
Victim takes action 4₁ ~ (-)Bt), proceeds to 81+1, receives Director saves (st, at, -Yt, Se-1 ) to its buffer
12
13
Director updates its policy using any RL algorithm
can justify that the size of the adversary MDP has a larger impact than the relaxation of the problem on the final solution found by the attackers. Third, in Appendix F.I. we empirically show that with the relaxed objective, PA-AD can still find the optimal attacker in 3 example environments.
Optimality in Formulation v.S. Approximated Optimality in Practice PA-AD has an optimal formulation, the optimal solution its objective (the optimal policy in PAMDP) is always an op- timal adversary (Theorem 7). Similarly, the previous attack method SA-RL has an optimal solution since the optimal policy the adversary's MDP is also an optimal adversary. However, in practice where the environments are in large scale and the number of samples is finite, the optimal policy not guaranteed be found by either PA-AD and SA-RL with deep RL algorithms. Therefore, for practical consideration, goal is to search for good solution or approximate the optimal solution using optimization techniques (e.g. actor-critic learning, one-step FGSM attack, Lagrangian relax- ation for the stochastic-victin attack). In experiments (Section 6). show that implementation universally finds stronger attackers than prior methods, which verifies the effectiveness of both our theoretical framework and practical implementation.
C.3 VARIANTS FOR ENVIRONMENTS WITH CONTINUOUS ACTION SPACES
Although the analysis the main paper focuses on an MDP whose action space is discrete, our algorithm also extends to continuous action space as justified in our experiments.
C.3.1 FOR A DETERMINISTIC VICTIM
In this case, we can still use the formulation D-PAMDP, but a slightly different actor function (Q s) = argmin ( ( - āl].
(GCD)
C.3.2 FOR A STOCHASTIC VICTIM
Different from stochastic victim in discrete action space whose actions are sampled from cate- gorical distribution, a stochastic victim in continuous action space usually follows a parametrized probability distribution with certain family of distributions, usually Gaussian distributions. In this case, the formulation of PAMDP in Definition is impractical. However, since the mean of still use the formulation in (GCD), while replacing л (8) with the mean of the output distribution. Then, the director and the actor can collaboratively let the victim output Gaussian distribution whose mean is the target action. If higher accuracy is needed, we can use another variant of PAMDP, named Continuous Policy Adversary MDP (C-PAMDP) that can also control the variance of the Gaussian distribution.
Gaussian distribution has the largest probability to be selected,
Definition 18 (Continuous Policy Adversary MDP (C-PAMDP)). Given an MDP M AS,A, P, R, where A is continuous, a fixed and stochastic victim policy n, we define a Con- tinuous Policy Adversarial MDP Mc (S, Åc, Pc, Rc, where the action space is Ān= A.
22

---

Published as a conference paper at ICLR 2022
and s' E S, E. A. ā( [ xlalg(ā, s)) R(s,a)= = n(alg(ā, 8) T()lo(,,s))B(ss a)da.
The actor function g defined as g(ō, (s (LLTTT of) (()))
(Gc)
where a is hypee--prammetr, and N denotes a multivariate Gaussian distribution.
In short, Equation (Gc) encourages the victim to output a distribution that is similar to the target distribution. The hyperparameter controls the standard deviation of the target distribution. One can set a to be small in order to let the victim execute the target action ă with higher probabilities.
D CHARACTERIZE OPTIMALITY OF EVASION ATTACKS
In this section, we provide a detailed characterization for the optimality of evasion attacks from the perspective of policy perturbation, following Definition 5 in Section 4. Section D.I establishes the existence of the optimal policy adversary which is defined in Section 3. Section D.2 then provides a proof for Theorem that the formulation of PA-AD optimal. We also analyze the optimality of heuristic attacks in Section D.3.
D.I EXISTENCE OF AN OPTIMAL POLICY ADVERSARY
Theorem 19 (Existence of An Optimal Policy Adversary). Given an MDP M (S,A,P,R..) and a fixed stationary policy on M, let H. be a non-empty set of admissible state adversaries and B (ar) be the corresponding Adv-policy-set, then there exists an optimal policy adversary The H (r) such that Th- E argmin,,, EB (x) VM (s), E.
Proof. We prove Theorem 19 by constructing a new MDP corresponding to the original MDP M and the victim π.
Definition 20 (Policy Perturbation MDP). For given MDP M, fixed stochastic victim pol- icy n, and an admissible state adversary set H. define a policy perturbation MDP as Mp (S, Ap, Pr, Rp, 7h. where Ap ((A), and Vs S,ap E Ap,
Rp(s,ap) = Exes - ((((s)))R)) if H. s.t. ap(-ls) x(-1(ss)
(4)
-
otherwise
Pp(s'(s, = Σ ((p(ss))((8((s.c)
(5)
aEA
Then we can prove Theorem 19 by proving the following lemma.
Lemma 21. The optimal policy in Mr is an optimal policy adversary for n in M.
Let Np denote the set of deterministic policies in Mp. According to the traditional MDP the- ory (Puterman, 1994), there exists a deterministic policy that is optimal in Mp. Note that H. is non-empty, so there exists at least one policy in MP with value 2 oo, and then the optimal policy should have value > co.. Denote this optimal and deterministic policy v.. Np. Then we write the Bellman equation of vp, i.e.,
V/ (s)
max Rr(s,up(5)) + Σ Pr(ssla (o(6)) (s') oPENP
e'ES
Σ up(a)s)R(ssaa)
Σ ΣΣ [(p(
max upENp
(6)
s'EScEA
max PPENN aEA
up(als)
-R(s,a) + Σ
P(8
s'ES
23

---

Published as a conference paper at ICLR 2022
Note that vs(ss is distribution action space, v/((ass) is the probability of given by distribution v" (s).
Multiply both sides of Equation (6) by -,, and we obtain
(s) (s)
min Σ
(a(()a [((,,a) +7 -
VENP aEA
3'ES
In the original MDP M, an optimal policy adversary (if exists) mh- for should satisfy
v (s)
min Ty EB (m)
Σ
(als) R(s,a)+ +7 P(s' a))
(8)
REA
s'E8
By comparing Equation (7) and Equation (8) we get the conclusion that v/ is an optimal policy adversary for in M.
D.2 PROOF OF THEOREM 7: OPTIMALITY OF OUR PA-AD
In this section, we provide theoretical proof of the optimality of our proposed evasion RL algorithm PA-AD.
D.2.1
OPTIMALITY OF PA-AD FOR A STOCHASTIC VICTIM
We first build connection between the PAMDP M defined in Definition 6 (Section 4) and the policy perturbation MDP defined in Definition 20 (Appendix D.1).
A deterministic policy v the PAMDP M can induce policy úp in Mp in the following way: (p(s) = (- x--lg(((s) s)), Vs E S. More importantly, the values of and Dp in M and Mp are equal because of the formulations of the two MDPs, i.e., Vo Vpp where V and Vp denote the value functions in M and Vp respectively.
Proposition 22 below builds the connection of the optimality between the policies in these two MDPs.
Proposition 22. An optimal policy in M induces an optimal policy in My.
Proof of Proposition 22. Let v² be an deterministic optimal policy in M, and induces policy in Mp, namely Pp.
Let us assume Üp is not an optimal policy in Mp, hence there exists a policy vi in Mp s.t. Ve (s) vp-(s (s) for least one S. And based on Theorem 4, we are able find such a Vi whose corresponding policy perturbation on the outermost boundary of ((T)), i.e., v" E 0, (T).
Then we can construct policy v' in M such that v'(s) = v)((s) ((s), (s,s), Vs E S. And based on in (()()) for all S. According to the definition of O. if two policy perturbations perturb in the same direction and are both on the outermost boundary, then they are equal. Thus, we can conclude that π (gg(v(s), (s). vs(s), Vs S. Then we obtain O (s) (x)
Equation (G), T (L(gg((ss)
v (s), Vs
Now we have conditions:
(1) Ve" (8) (*"(s), Vs E S;
(2) VI (s) > vpr (s) for at least one E S;
(3) 3V such that Pu (s) = Vs)
From (1), (2) and (3), we can conclude that P (s) V (s) for at least one E S, which conflicts
with the assumption that v² is optimal in M. Therefore, Proposition 22 proven.
24

---

Published as a conference paper at ICLR 2022
Proposition 22 and Lemma 21 together justifies that the optimal policy of M. namely v*, induces an optimal policy adversary for æ in the original M. Then, if the director learns the optimal policy in M. then it collaborates with the actor and generates the optimal state adversary ht by h" (s))) = S.
s), Vs g(v"(s),
D.2.2 OPTIMALITY OF OUR PA-AD FOR DETERMINISTIC VICTIM
In this section, we show that the optimal policy in D-PAMDP (the deterministic variant PAMDP defined in Appendix C.1) also induces optimal policy adversary in the original environment.
Let nD be a deterministic policy reduced from a stochastic policy n, i.c.,
(D()) ;= a
Vs E S.
Note that in this case, the Adv-policy-set B (T) is not connected as contains only deterministic policies. Therefore, re-formulate the policy perturbation MDP introduced in Appendix D.I with a deterministic victim as below:
Definition 23 (Deterministic Policy Perturbation MDP). For a given MDP M, a fixed determin- istic victim policy æ, and admissible adversary set H. define deterministic policy perturbation MDP as MpP (S, ApP, PDP, RDP, 7),, where App = A, and Vs E S,aDP E App.
Apr(s,0Dr) = - R(ss, abp) if E H. s.t. aDP(s) = TD(h(s))
(9)
-00
otherwise
PDp(s (s,('((8,aD)) aps
(10)
MDP can be viewed special case of MM where only deterministic policies have 2 -0 values. Therefore Theorem 19 and Lemma 21 also hold for deterministic victims.
Next will show that optimal policy in MD induces optimal policy in Mpr.
Proposition 24. An optimal policy in MD induces an optimal policy in MDP.
Proof of Proposition 24, We will prove Proposition 24 by contradiction. Let be an optimal policy in Mp. and it induces a policy in MpP, namely Dpp.
Let us assume PDP is not an optimal policy MDP, hence there exists deterministic policy vDp in MDP s.t. V (s) V for at least one S. Without loss of generality, suppose Ve (so) > V.or ()p)
Next construct another policy 1/ in Mp by setting v'(s) = (s),s), Vs S. Given that VDp is deterministic, / is also deterministic policy. So we use (s) and (s) to denote the action selected by "hp and v respectively state
For an arbitrary state 8;, let 0: := "Dr(s)). Since "Dr is the optimal policy in MpP, we get that there exists state adversary h E H. such that TD(h(s;)) state 8; E (s.) such that argmax.e, (5.) = a, Then, the solution to the actor's optimization problem (GD) given direction a, and state 8,, denoted as s*, satisfies
a; or cquivalently, there exists
3* argmax, (.) (T(()')) - argmax (asa
(11)
and get T 3*) ar > क (al 1 a T
(12)
Given that argmax (a,15, a, we obtain argmax, (a,(s*) Tn(gn(a;,s;) a; Since this relation holds for an arbitrary state 8, we can get # (o)(( #p(gn(v'
ai and hence
v' (s), E
(13)
25

---

Published as a conference paper at ICLR 2022
Also, we have Vs S
(s) (s) Ro(s, - Σ ((()))
(14)
S'ES
V (8) Ror(s, +
Σ
Ppr(s'|s, (bb (bp ((6)
(15)
S'ES
Therefore, Vg'(s) (s) VD' (s), Vs
Then we have
V (so) - v- (so) = VD (so) < VDP (so) V/ (so)
(16)
which gives Vg < Vg' (so). so there a contradiction.
Combining the results of Proposition 24 and Lemma 21 for deterministic victim, the optimal policy in D-PAMDP gives an optimal adversary for the victim.
D.3 OPTIMALITY OF HEURISTIC-BASED ATTACKS
There are many existing methods of finding adversarial state perturbations for a fixed RL policy, most of which solving some optimization problems defined by heuristics. Although these meth- ods are empirically shown to be effective in many environments, it is not clear how strong these adversaries in general. In this section, we carefully summarize and categorize existing heuristic attack methods into 4 types, and then characterize their optimality in theory.
D.3.1 TYPE MINIMIZE THE BEST (MINBEST)
common idea of evasion attacks in supervised learning is to reduce the probability that the learner selects the "correct answer" Goodfellow et al. (2015). Prior works Huang al. (2017); Kos & Song (2017); Korkmaz (2020) apply similar idea to craft adversarial attacks in RL, where the objective minimize the probability of selecting the "best" action, i.e.,
h MinBest argmin (a (a+1s), Vs
(I)
where a the "best" action to select at state s. Huang et al.Huang et al. (2017) define at as argmax Q= a) for DQN, a (al ())s) for TRPO and A3C with stochastic Since the agent's policy π is usually well-trained in the original MDP, at can be viewed as (approximately) the action taken by optimal deterministic policy (s).
Lemma 25 (Optimality of MinBest). Denote the set of optimal solutions to objective (I) as Miiieeer There exist an MDP M and an agent policy n, such that H MinBesst does not contain an optimal adversary h² i.e. MinBett H 0.
Proof of Lemma 25. We prove this lemma by constructing the following MDP such that for any victim policy, there exists a reward configuration in which MinBest attacker is not optimal.
#
PL
Figure simple MDP where MinBest Attacker cannot find the optimal adversary for a given victim policy.
Here, let rı = (2, a, = r(s5|s2, 02), r3 = r(s (2)). Assuming all the other rewards zero, transition dynamics deterministic, and states 83, 84, 85 the terminal states. For the
26

---

Published as a conference paper at ICLR 2022
sake of simplicity, also assume that the discount factor here 7 1.
Now given a policy и such that x ({a]|s) = 31 and T(a1(82) = 32 (31, E [0. 1D. we could find
r1, 22, rg such that the following constraints hold:
ri > 12 x (51, 01) > (s), 02)
(17)
6e1 + (1 - B2)r2 > + Q2 01) > Q" ($2, 02)
(18)
r3 > (32 62)r + (1- 32 + 22 Now we consider the Adv-policy-set
- r3 > Q" (81. c2 - -12)
(19)
(x) = n' E II [+|s( - T-1s))l < c, || (+122) - x(112))| < {
Under these three linear constraints, the policy given by MinBest attacker satisfies that Ti (41 [s1) = 31 é1, and Ta (a1)82) = B2 62. On the other hand, we can find an-
other admissible policy adversary The (ail81) = + e,, and Th- (a1 32 62. Now we show that V (81) < Vi (81), and thus MinBest attacker is not optimal.
Va (s1) (31 - e1) (B2 - e2)ri + 81 + 22
+ (1 31 + E1)r3
(20)
(31 c))(82 - -2) + (81 (1) 82 + + (1 B1 + cirrs
(21)
VTA* (s1) (31 + ()) (B2 - c2) + 82 (2) (1
31 - (1)
(22)
= =(1 + c1) e2)r + (8) + 82 +2) + +(1 31 =1)r's
(23)
Therefore,
v (s1) / Mintie (51) 2e, (52 02)12 + 22 (1 Bg + 62) 2E153
(24)
20 (32 c2)r (1 32 + c2)r2 r3
(25)
< 0 Because of the constraint (19)
(26)
D.3.2 TYPE II MAXIMIZE THE WORST (MAXWORST)
Pattanaik et al. (Pattanaik et al., 2018) point out that only preventing the agent from selecting the best action does not necessarily result in low total reward. Instead, Pattanaik et al. (Pattanaik et al., 2018) propose another objective function which maximizes the probability of selecting the worst action, i.e.,
A Max E argmaxnE Th (a [s),
(II)
where at refers to the "worst" action at state Pattanaik et al.ttaata et al., 2018) define the "worst" action as the actions with the lowest Q value, which could be ambiguous, since the Q function is policy-dependent. If a worst policy π E argmin, (s),,, Vs E S is available, one can use a argminQ" (s,a), a). However, in practice, the attacker usually only has access to the agent's 's current policy n, so it can also choose a = argminQ** (s, a). Note that these two selections are different, as the agent's 's policy æ is usually far away from the worst policy.
Lemma 26 (Optimality of MaxWorst). Denote the set of optimal solutions to objective (II) as H MarWorst which include both versions of Max attacker formulations we discussed above. Then there exist an MDP M and an agent policy n, such that H MacWors contains a non-optimal adversary h², ie.. H MusWorst & H
Proof of Lemma 26.
Case I: Using current policy to compute the target action
We prove this lemma by constructing the MDP in Figure 8 such that for any victim policy, there exists a reward configuration which Max Worst attacker is not optimal.
Here, let r1 r(s111s: [$1, a1),72 r(s12) $1 02), ra r(s21 (1,,a)),7 74 = r(222 $2, az). Assuming all the other rewards are zero, transition dynamics are deterministic, and states $11, 812, $21, 822 are the terminal states. For the sake of simplicity, we also assume that the discount factor here 7 1. Now given a policy æ such that (aj (8)) = So. (ail81) = 31, and π (a2)82) = 32 (Bo. 31. B2 E [o, 1)), consider the Adv-policy-set
B (T)) = {xt n' E II || (-181) (- < cp, || x (-181)) < (-12)))))))))))))()))))) - - (.16)] < 62,
27

---

Published as a conference paper at ICLR 2022
81L
F1
812
P21
F3
62
821
Figure 8: A simple MDP where the first version of Max Worst Attacker cannot find the optimal adversary for given victim policy.
We could find r1, r3, r4 such that the following linear constraints hold:
Biri + (1 - Bi)rz >B273 + (1 (1- 32rr4 <> Q" ai) > ("(00002)
(27)
ri >r2 + (7, > Q= a2)
(28)
13 >r 2= (* 01) > (= (2)
(29)
(81- + (1 31 + e1)r2 ((32-62))
(1 32 e2)r4
(30)
Now, given these constraints, the perturbed policy given by Maxsorst attaker satisfies
Tosaoooe (0) = Bo 60. Ta (11 = 31 -1-11, El, and TyMeier (a1]52) = B2 E2. How-
ever, consider another perturbed policy Th- in Adv-policy-set such that Ta. (aI[50) = 30 + ED.
Th (a1 [s]) = B1 c1, and Th- (a1 32 c2- We will prove that V (s1) < Manae (s1), and thus Max Worst attacker is not optimal.
On the one hand,
VTA 1 MacWooss (81) =(30 (0)0-c0)(0 [(8 e1)r (1- 31 )1)2 + +11 - Bo + t00 ((222-))r3+
02)r3 (1- B2 E2)r1
(31)
-180 (0) - -i)r) + co) telrr2
+ 3o + (0) - 62)r (1 3 + c0 - B2 + E2)r
(32)
On the other hand,
VTh (s1) =Bo + 60) [18] - c1) + +1 - 31 + + -11 - c0) (82 -2)r3 + (1 Bg + e2)ra
(33)
=(Bo + (31 - Ei)ri + (80 + (1 - 31 +61)r2
+ - 3o ----) -00) -2)r3 + +(1 - 30 -0) 83 + 62)
(34)
Therefore,
VTa (81) V MacHin (81) =20 (B1 - .)m + 2c - + (1)2
- 200 (2) 26 - f2 + 2
(35)
< Because of the constraint (30)
(36)
Case II: Using worst policy to compute the target action
81
T1
Q:
$p
82
Q3
83
T33
Figure 9: A simple MDP where the second version of Worst Attacker cannot find the optimal adversary for given victim policy.
28

---

Published as a conference paper at ICLR 2022
Same before, construct MDP where MaxWorst contains non-optimal adversary. Let "1 = r(51 Su, 01), 22 r($2|80, a2), T3 = r(s3 so, az). Assuming all the other rewards are zero, transition dynamics are deterministic, and states $1, 82, 83 are the terminal states. For the sake of simplicity, we also assume that the discount factor here 7 = 1.
Let pi be the given policy such that [
B1 and क (a2 (so) = B2. Now without loss of gener-
ality, we assume 12 > 13 (*). Then the worst policy x satisfies that I ra3/s))) = =. Consider
the Adv-policy-set 1.d (T) = श E II
(=(00=))) x--150)]la < e). Then H MMa
π
n' (0)) = (1 - B1 (2) + +
Now consider two policies T nAa E H MaxWorrs where TA (a1ls0) = B1.. nn (a2)s0) = 32 Tha (aj 80) = B1 6, TT (a2 (8)) $2. Then VTAa (so) (s0) <(r1 - -r2) > Therefore,
The E H Maxoossi but it's not optimal.
D.3.3 TYPE III MINIMIZE Q VALUE (MINQ).
Another idea of attacking Pattanaik et al. (2018); Zhang et al. (2020a) to craft perturbations such that the agent selects actions with minimized Q values at every step, i.e.,
h MinQ E argmin
Σ AEA
(III)
where Q is the approximated Q function of the agent's original policy. For example, Pattanaik et al.Pattanaik et al. (2018) directly use the agent's Q network (of policy л), while the Robust SARSA (RS) attack proposed by Zhang et al.Zhang et al. (2020a) learns more stable Q network for the agent's policy æ. Note that in practice, this type of attack is usually applied to deterministic agents (e.g., DQN, DDPG, etc), then the objective becomes argmin H. Q² (s, Th(s) (s)). Vs E S Pattanaik et al. (2018); Zhang al. (2020a); Oikarinen al. (2020). this case, the MinQ attack is equivalent to the Max Worst attack with the current policy the target.
Lemma 27 (Optimality of MinQ). Denote the set of optimal solutions to objective (III) as MinQ which include both versions of MinQ attacker formulations as we discussed above. Then there exist an MDP M and an agent policy æ, such that Mii contains non-optimal adversary h", i.e., Mine e H
Proof of Lemma 26.
Case I: For deterministic victim In the deterministic case 1 MinQ E argmin (s, TA (+)) argmaX Ta (argmin, (s, (s, ES ES In this case, the objective is equivalent to objective (II), thus Lemma 27 holds.
(III))
Case II: For stochastic victim In this case, we consider the MDP in Figure 8 and condition (27) to (30). Then the MinQ objective
gives TyMio (a (s0) GB0- 60. Tymee [c]/s/s = 31 Ej, and TyMin (a. (22) 32 -2.
According to the proof of the first case of Lemma 26, MAMin Machio is not an optimal adversary.
Thus Lemma 27 holds.
D.3.4 TVPEIV MAXIMIZE DIFFERENCE (MAXDIFF).
The MAD attack proposed by Zhang et al. (Zhang et al., 2020a) is to maximize the distance between the perturbed policy mh and the clean policy क, i.e..
(IV) where TV denotes the total variance distance between two distributions. In practical imple- mentations, the TV distance can be replaced by the KL-divergence, as Drv [Mx (-]s)]][([ < (DKL (= [|]ss)](((
f Maxxiif E argmax Drv (-]) Vs S
This type of attack is inspired by the fact that if two policies select ac- tions with similar action distributions on all the states, then the value of the two policies is also small (see Theorem in Zhang al. (2020a)).
29

---

Published as a conference paper at ICLR 2022
Lemma 28 (Optimality of MaxDiff). Denote the set of optimal solutions to objective (IV) as Ma There exist an MDP M and an agent policy n, such that MMusDiff contains a non-optimal adversary ht, i.e., H Ma of H
Proof of Lemma 28. The proof follows from the proof of lemma 25. In the MDP we constructed, n' = 31 {1, Tiatnten (01 $2) = 32 62 is one of the policies that has the maximum KL divergence from the victim policy within Adv-policy-set. However, as we proved in 25, this is not the optimally perturbed policy. Therefore, MaxDiff attacker may not be optimal.
E ADDITIONAL EXPERIMENT DETAILS AND RESULTS
In this section, provide details of our experimental settings and present additional experimental results. Section E.1 describes our implementation details and hyperparameter settings for Atari and MuloCo experiments. Section E.2 provide additional experimental results, including experiments with varying budgets (c) in Section E. more comparison between SA-RL and PA-AD in terms of convergence rate and sensitivity to hyperparameter settings as in Section E.2.3, robust training in MuloCo games with fewer training steps in Section E.2.4, attacking performance on robust models in Atari games in Section E.2.5, as well as robust training results in Atari games Section E.2.6.
E. IMPLEMENTATION DETAILS
E.1.1 ATART EXPERIMENTS
In this section we report the configurations and hyperparameters we use for DQN, A2C and ACKTR in Atari environments. We use GeForce RTX 2080 7i GPUs for all the experiments.
DQN Victim We compare PA-AD algorithm with other attacking algorithms on 7 Atari games. For DQN, we take the softmax of the Q values Q(s,-) as the victim policy T(-ss) in prior works (Huang et al., 2017). For these environments, we use the wrappers provided by stable- baselines (Hill et al., 2018), where we clip the environment rewards to be -1 and during training and stack the last 4 frames as the input observation to the DQN agent. For the victim agent, we im- plement Double Q learning (Hado Van Hasselt, 2016) and prioritized experience replay (Tom Schaul & Silver, 2016). The clean DQN agents are trained for 6 million frames, with a learning rate 0.00001 and the same network architecture and hyperparameters as the ones used Mnih et al. (2015). In addition, we use a replay buffer of size 5 x 10 Prioritized replay buffer sampling is used with = 0.6 and 3 increases from 0.4 to 1 linearly during training. During evaluation, we execute the agent's 's policy without epsilon greedy exploration for 1000 episodes.
A2C Victim For the A2C victim agent, we also use the same preprocessing techniques and con- volutional layers the one used in Mnih et al. (2015). Besides, values and policy network share the same CNN layers and fully-comnected layer with 512 hidden units. The output layer is a cat- egorical distribution over the discrete action space. We use 0.0007 as the initial learning rate and apply linear learning rate decay, and we train the victim A2C agent for 10 million frames. During evaluation, the A2C victim executes a stochastic policy (for every state, the action sampled from the categorical distribution generated by the policy network). Our implementation of A2C is mostly based on an open- open-source source implementation by Kostrikov Kostrikov (2018).
ACKTR Adversary To train the director of PA-AD and the adversary in SA-RL, we use ACKTR (Wu et al., 2017) with the same network architecture as A2C. We train the adversaries of PA-AD and SA-RL for the same number of steps for fair comparison. For the DQN victim, we use learning rate 0,0001 and train the adversaries for 5 million frames. For the A2C victim, we use learning rate 0.0007 and train the adversaries for 10 million frames. Our implementation of ACKTR is mostly based on an open-source implementation by Kostrikov Kostrikov (2018).
Heuristic Attackers For the MinBest attacker, we following the algorithm proposed by Huang et al. (2017) which uses FGSM to compute adversarial state perturbations. The MinBest Mo-
30

---

Published as a conference paper at ICLR 2022
mentum attacker is implemented according to the algorithm proposed by Korkmaz (2020), and set the number of iterations to be 10, the decaying factor μ to be 0.5 (we tested 0.01, 0.1, 0.5, 0.9 and found 0.5 is relatively better while the difference is minor). Our implementation of the MinQ attacker follows the gradient-based attack by Pattanaik et al. (2018), and we also set the number of iterations be 10. For the MaxIDiff attacker, we refer to Algorithm 3 in Zhang al. (2020a) with the number of iterations equal to 10. In addition, we implement random attacker which perturbs state s to s + esign(u), where μ is sampled from standard multivariate Gaussian distribution with the same dimension as S.
E. MuJoCo EXPERIMENTS
For four OpenAl Gym MuJoCo continuous control environments, we use PPO with the original fully connected (MLP) structure the policy network to train the victim policy. For robustness evaluations, the victim and adversary are both trained using PPO with independent value and policy optimizers. We complete all the experiments MuJoCo using 32GB Tesla V100.
PPO Victim
We directly use the well-trained victim model provided by Zhang et al. (2020a).
PPO Adversary Our PA-AD adversary is trained by PPO and we use a grid search of part of adversary hyperparameters (including learning rates of the adversary policy network and policy network, the entropy regularization parameter and the ratio clip for PPO) to train the adversary as powerful as possible. The reported optimal attack result is from the strongest adversary among all 50 trained adversaries.
For Robust Sarsa (RS) attack, we use the implementation and the optimal RS hyperparameters from Zhang et al. (2020a) to train the robust value function to attack the victim. The reported RS attack performance is the best one over the 30 trained robust value functions.
Other Attackers
For MaxDiff attack, the maximal action difference attacker is implemented referring to Zhang et al. (2020a).
For SA-RL attacker, following Zhang et al. (2021), the hyperparameters is the same the optimal hyperparameters vanilla PPO from a grid search. And the training steps are set for different environments. For the strength of SA-PPO regularization K, we choose from 10 to 1 and report the worst-case reward.
Robust Training For ATLA Zhang et al. (2021), the hyperparameters for both victim policy and adversary remain the same as those vanilla PPO training. To ensure sufficient exploration, we run small-scale grid search for the entropy bonus coefficient for agent and adversary. The experiment results show that a larger entropy bonus coefficient allows the agent to learn a better policy for the continual-improving adversary. In robust training experiments, the MuJoCo environments to guarantee policy convergence. We train million steps in Hopper, Walker, and HalfCheetah environments and 10 million steps for Ant. For reproducibility, the final results we reported are the experimental performance of the agent with medium robustness from agents training with the same hyperparameter set.
larger training steps in all
E.2 ADDITIONAL EXPERIMENT RESULTS
E.2.1 ATTACKING PERFORMANCE WITH VARIOUS BUDGETS
In Table I, we report the performance of our PA-AD attacker under chosen epsilon across different environments. To see how PA-AD algorithm performs across different values of c's, here we select three Atari environments each for DQN and A2C victim agents and plot the performance of PA-AD under various e's compared with the baseline attackers in Figure 10. We can see from the figures that our PA-AD universally outperforms baseline attackers concerning various e's.
In Table 2. provide the evaluation results of PA-AD under commonly unused epsilon in four MuJoCo experiments (Zhang et al. (2020a; 2021)) to show that PA-AD attacker also has the best attacking performance compared with other attackers under different c's in Figure 11.
31

---

Published as a conference paper at ICLR 2022
Moneatin
AMrrgy Douess
MasDal
Arcraee Rexura
MasDdal
0-
MinBet
Miniker Momoomm
MinQ
20
Moataff
B4-AD
-00
0.2
1.2
0.6
0.7
(a) DQN Boxing
(b) DQN Pong
(c) DQN RoadRunner
20
MieBles:
100
MidBest
Avonge Reine
Machir
IA-AD
Milllest Momentar
300
PATAD
MidBeet Mencetorr
Mashii
1,900
PA-AI3
Minlea Momemun
MicDilf
1,000
Average
200
-10
500
20
0-
0.0
10
100
-10
(d) A2C Pong
(e) A2C Breakout
(1) A2C Seaquest
Figure 10: Comparison of different attack methods against DQN and A2C victims in Atari w.r.t different
budget e's.
6,000
ManDiff Robao Saroo
5,000
ManDel Robust Sussa SA.RI
MasDar
Robra Susa
- SA RL
4,000
S/V RL
Arcige feaars
PA-AD
4,000
PA-AD AD
AvengeBeeee Renurn
AD
3,900
8,000
2080
0-
095300755 1000 1250, 1500 17S 2000 2350.250 -1000
0.02 nb0.
od8
0.10
0.12
6.14
012 od1
0/66
0.10
0:13
0.1
(a) PPO Hopper
(b) PPO Walker2d
(c) PPO Ant
Figure 11: Comparison of different attack methods against PPO victims in MuJoCo w.r.t. different budget e's.
32

---

Published as a conference paper at ICLR 2022
E.2.2 HYPERPARAMETER TEST
In our Actor-Director Framework, solving an optimal actor is a constraint optimization problem. Thus, in our algorithm, we instead use Lagrangian relaxation for the actor's ' constraint optimization. In this section, we report the effects of different choices of the relaxation hyperparameter À on the final performance of our algorithm. Although we set À by default to be 1 and keep it fixed throughout all of the other experiments, here we find that in fact, difference choice of À has only minor impact the performance of the attacker. This result demonstrates that PA-AD algorithm is robust to different choices of relaxation hyperparameters.
Table 4: Performance of PA-AD across difference choices of the relaxation hyperparameter À
Pong
Boxing
Ant
Walker
Nature Reward
21 + 0
96 + 4
Nature Reward
5687 + 758
4472 + 635
X = 0.2
-19 +
16 + 12
À = 0.2
-2274 + 632
897 = 157
À 0.4
-88 +
17 + 12
À 0.4
-2239 + 716
923 = 132
À = 0.6
-20 2
19 + 15
À = 0.6
-2456 + 853
954 105
À = 0.8
-19 11
14 + 12
== 0.8
-2597 + 662
872 162
À 1.0
-19 +
15 + 12
λ 1.0
-2580 + 872
804 130
À 2.0
-20 +
21 + 15
À 2.0
-2378 1 794
795 = 124
== 5.0
-0 =
19 14
== 5.0
-2425 + 765
814 = 140
(a) Atari
(b) Mujoco
E.2.3
EMPIRICAL COMPARISON BETWEEN PA-AD AND SA-RL
In this section, we provide more empirical comparison between PA-AD and SA-RL. Note that PA- AD and SA-RL are different in terms of their applicable scenarios: SA-RL is a black-box attack methods, while PA-AD is a white-box attack method. When the victim model is known, we can see that by proper exploitation of the victim model, PA-AD demonstrates better attack performance, higher sample and computational efficiency, as well as higher scalability. Appendix F.2 shows de- tailed theoretical comparison between SA-RL and PA-AD.
PA-AD has better convergence property than SA-RL. In Figure 12, we plot the learning curves of SA-RL and PA-AD in the CartPole environment and the Ant environment. Compared with SA- RL attacker, PA-AD has a higher attacking strength in the beginning and converges much faster. In Figure 12b, can see that PA-AD has a "warm-start" (the initial reward of the victim is already significantly reduced) compared with SA-RL attacker which starts from scratch. This is because PA-AD always tries to maximize the distance between the perturbed policy and the original victim policy in every step according to the actor function (G). So in the beginning of learning, PA-AD works similarly to the Maxf attacker, while SA-RI. works similarly to a random attacker. We also note that although PA-AD algorithm is proposed particularly for environments that have state spaces much larger than action spaces, in CartPole where the state dimensions is fewer than the number of actions, PA-AD still works better than SA-RL because of the distance maximization.
SARRL
nowwad
Clean Reward
SA RL
PA-AD
-80
Steps
Ewisiin
(a) Learning curve of SA-RL and PA-AD attacker against A2C victim in CartPole.
(b) Learning curve of SA-RL and PA-AD attacker against PPO victim in Ant.
Figure 12: Comparison of convergence rate between SA-RU and PA-AD in Ant and Cartpole. Results averaged over 10 random seeds.
33

---

Published as a conference paper at ICLR 2022
PA-AD is more computationally efficient than SA-RL. Our experiments in Section 6 show that PA-AD converges to better adversary than SA-RL given the same number training steps, which verifies the sample efficiency of PA-AD. Another aspect of efficiency is based on the computational resources, including running time and required memory. For RL algorithms, the computation cost comes from the interaction with the environment (the same for SA-RL and PA-AD) and the pol- icy/value update. If the state space S is higher-dimensional than the action space A, then SA-RL requires larger policy network than PA-AD since SA-RL has higher-dimensional output, and thus SA-RL has more network parameters than PA-AD, which require more memory cost and more computation operations. On the other hand, PA-AD requires solve an additional optimization problem defined by the actor objective (G) (G In our implementation, we use FGSM which only requires one-step gradient computation and thus efficient. But more advanced optimiza- tion algorithms (e.g. PGD) are used, more computations may be needed. In summary, if S is much larger than A, PA-AD is more computational efficient than SA-RL. if A is much larger than S, SA- RL is more efficient than PA-AD; if the sizes of S and A are similar, PA-AD may be slightly more expensive than SA-RL, depending on the optimization methods selected for the actor.
To verify the above analysis, compare computational training time for training SA-RL and PA- AD attackers, which shows that PA-AD is more computationally efficient. Especially on the envi- ronment with high-dimensional states like Ant, PA-AD takes significantly less training time than SA-RL (and finds a better adversary than SA-RL), which quantifies the efficiency our algorithm empirical experiments.
Method SA-RL
Hopper
Walker2d
HalfCheetah
Ant
1.80
1.92
1.76
4.88
PA-AD
1.43
1.46
1.40
3.76
Table Average training time (in hours) of SA-RI. and PA-AD MuloCo environments, using GeForce RTX 2080 Ti GPUs. For Hopper, Walker2d and HalfChectah, SA-RL and PA-AD are both trained for million steps; for Ant, SA-RL and PA-AD are both trained for 5 million steps
PA-AD is less sensitive to hyperparameters settings than SA-RL. In addition to better final attacking results and convergence property, we also observe that PA-AD much less sensitive to hyerparameter settings compared to SA-RL. On the Walker environment, 216 different configurations of hyperparameters, including actor learning rate, critic learning rate, entropy regularization coefficient, and clipping threshold in PPO. Here for comparison we plot two histograms of the agent's 's final attacked results across different hyperparameter configurations.
grid search over
200
280
2010
208
1300
1800
3001
2000
(a) SA-RL Attacker
(b) PA-AD Attacker
Figure 13: Histograms of victim rewards under different hyperparameter settings of SA-RL and PA-AD on Walker.
The perturbation radius is set to be 0.05, for which the mean reward reported by Zhang al. al. (2020a) is 1086. However, RL achieves an attacking reward within the range 1000-2000, while in most hyperparameter settings, the mean attacked return lies in the range 4000-4500. In contrast, about 10% hyperparameter settings
see from this histogram, only out of the 216 configurations of SA-
34

---

Published as a conference paper at ICLR 2022
of PA-AD algorithm able to reduce the reward to 500-1000, and another 10% settings could reduce the reward to 1000-2000. Therefore, the performance of PA-AD attacker is generally better and more robust across different hyperparameter configurations than SA-RL.
E.2.4 ROBUST TRAINING EFFICIENCY ON MuJoCo BY PA-ATLA
In the ATLA process proposed by Zhang et al. (2021), one alternately trains an agent and an adver- sary. As a result, the agent policy may learn to adapt to the specific type of attacker encounters during training. In Table 3. we present the performance of our robust training method PA-ATLA- PPO compared with ATLA-PPO under different types of attacks during testing. ATLA-PPO uses SA-RL to train the adversary, while PA-ATLA-PPO uses PA-AD to train the advensary during al- ternating training. As a result, we can see that ATLA-PPO models perform better under the SA-RL attack, and PA-ATLA-PPO performs better under the PA-AD attack. However, the advantage of ATLA-PPO over PA-ATI.A-PP -A-PPO against SA-RL attack is much smaller than the advantage of PA- ATLA-PPO over ATLA-PPO against PA-AD attack. In addition, our PA-ATLA-PPO models signif- icantly outperform ATLA-PPO models against other heuristic attack methods, and achieve higher average rewards across all attack methods. Therefore, PA-ATLA-PPO is generally more robust than ATLA-PPO.
Furthermore, the efficiency of training an adversary could be the bottleneck in the ATLA Zhang et al. (2021) process for practical usage. Appendix E.2.3 suggests that our PA-AD generally con- verges faster than SA-RI., Therefore, when the computation resources are limited, PA-ATIA--PPO can train robust agents faster than ATLAA-PO. We conduct experiments on continuous control envi- ronments to empirically show the efficiency comparison between PA-ATLA-PPO and ATLA--PO. In Table 6, we show the robustness performance of two ATLA methods with 2 million training steps for Hopper, Walker and Halfcheetah and million steps for Ant (Compared with results in Table 3, have reduced training steps by half or more). It can be seen that our PA-ATLA-PPO models still significantly outperform the original ATLA-PPO models under different types of attacks. More im- portantly, our PA-ATLA-PPO achieves higher robustness under SA-RL attacks in Walker and Ant, suggesting the efficiency and effectiveness of our method.
Environment
stepúmillion
Model
Natural Reward
RS Zhang et (2020s)
SA-RL
PA-AD (ours)
Average reward
Zhang al (2021)
Hopper
0.075
2
APLA-PPO
1763 + 818
1940 + 174
1172 = 344
477 +: 30
900.3
PA-ATLA.PPO
2164 + 121
1720 + 400
1119 = 123
1024 + 188
1287.7
Walker
0.05
ATLA-PPO
3180 842
2405 529
1032
516 4:
1697.0
PA-ATLA-PPO
3206 115
2749 1: 106
2832 = 198
1072 + 247
2051.0
Halfcheetah
0.15
2
ATLA-PPO
4871 1: 112
3781 1: 645
3193 372
856 :: 118
2710.0
PA-ATLA-PPO
5257 1 94
4012 1. 290
$329 183
1670 + 1419
3003.7
Ant
0.15
5
ATLA-PPO
3267 51
3062 + 140
56
100
1750.7
PA-ATLA-PPO
3981 + 71
3964 + 254
2685 + 41
2403 =:: 82
2817.3
Table Average episode rewards standard deviation of robust models with fewer training steps under different evasion attack methods. Results are averaged over 50 episodes. We bold the strongest attack in each row. The gray cells are the most robust agents with the highest average rewards across all attacks.
E.2.5 ATTACKING ROBUSTLY TRAINED AGENTS ON ATART
In this section, we show the attack performance of our proposed algorithm PA-AD against DRL agents that are trained to be robust by prior works (Zhang et al., 2020a; Oikarinen et al., 2020) in Atari games.
Zhang et al. (2020a) propose SA-DQN, which minimizes the action change under possible state perturbations within Ep norm ball, i..., to minimize the extra loss
RDQN Σ max max max Qo(3, a) - Qo (8,
(37)
BE)(s) aga"
where 0 refers to the Q network parameters, a as) argmax (alss) and c is small constant. Zhang et al. (2020a) solve the above optimization problem by a convex relaxation of the Q network, which achieves 100% action certification (i.e. the rate that action changes with a constrained state perturbation) in Pong and Freeway, over 98% certification in BankHeist and over 47% certification in RoadRunner under attack budget 1/255.
35

---

Published as a conference paper at ICLR 2022
Enviromment
Natural Reward
Random
Huang et al. (2017)
MinBest
Minilest+ Momentum
Pattanaik et al. (2018) Zhang al. (2020l)
Min
MaxDiff
PA-AD (ours)
Korkman (20200)
ReadRunner 46440 5707
45009 7125
6901
45856
42700 8456
456466
38052
SA-DON
HankHeist
1237 = 11
1206 1: 12
1285 15
1233 :: 17
1237 = 14
1230 1: 13
1237 1: 14
RoadRunner 30102 18727
41084
6351
41824 7838
42530 8525
40872
42014 8307
38214
RADIAL -DQN
235
25766 + 6120
9808 +: 4345
95306 it 81.1
30860 s: 6001
18004 = 6451
1306 1: 0354
BankHeiss
1060
1007
105
988 102
1021 96
1042= 112
908
1011 + 130
801 + 114
460 + +10
842 +12+t00 33
1023 = 110
397 : 172
RoadRunner 30854 7281
7297
31296 7095
31132
30838
52038
30550
RADIAL -A3C
30600 1. 7006
30198 1, 6075
29906 1. 5088
20068 6340
31170 L. 7453
29768 1: 5892
BankHeist
847
31
847 33
648 +:31
848
848
31
848 + 31
644 + 168
822 +
842 88
834 = 30 620 d: 1668
Table 7: Average episode rewards + standard deviation of SA-DQN, RADIAL-DON, RADIAL-A3C robust agents under different evasion attack methods in Atari environments RoadRunner and BankHeist. All attack methods use 30-step PGD to compute adversarial state perturbations. Results are averaged over 50 episodes. In each row, bold the strongest attack, except for the rows where none of the attacker reduces the reward significantly (which suggests that the corresponding agent is relatively robust).)
Oikarinen et al. (2020) propose another robust training method named RADIAL-RL. By adding a adversarial loss the classical loss of the RL agents, and solving the adversarial loss with interval bound propagation, the proposed RADIAL-DQN and RADIAL-A3C achieve high rewards in Pong, Freeway, BankHeist and RoadRunner under attack budget = 1 1/255 and = 3/255.
Implementation of the Robust Agents and Environments. We directly use the trained SA-DQN agents provided by Zhang et al. (2020a), as well as RADIAL-DQN and RADIAL-A3C agents pro- vided by Oikarinen et al. (2020). During test time, the agents take actions deterministically. to reproduce the results in these papers, et al. (2020a) and Oikarinen et al. (2020), respectively. But note that the environment configura- tions of SA-DQN and RADIAL-RL are simpler versions of the traditional Atari configurations use (described in Appendix E.1.I). Both SA-DQN and RADIAL-RL use a single frame instead of the stacking as 4 frames. Moreover, SA-DQN restricts the number of actions as 6 (4 for Pong) in each environment, although the original environments have 18 actions (6 for Pong). The above simplifications in environments can make robust training casier since the dimensionality of the input space is much smaller, and the number of possible outputs is restricted.
order use the same environment configurations in Zhang
Attack Methods In experiments, we find that the robust agents are much harder to attack than vanilla agents in Atari games, as claimed by the robust training papers (Zhang et al., 2020a: Oikari- nen et al., 2020). A reason is that Atari games have discrete action spaces, and leading an agent to make different decision at a state with limited perturbation could be difficult. Therefore, we use 30-step Projected Gradient Descent for all attack methods (with step size e/10), including MinBest (Huang et al., 2017) and our PA-AD which use FGSM for attacking vanilla models. Note that the PGD attacks used by Zhang et al. (2020a) and Oikarinen et al. (2020) in their experiments are the same as the MinBest-PGD attack we use. For our PA-AD, we use PPO to train the adversary since PPO is relatively stable. The learning rate is set to be 5e 4, and the clip threshold is 0.1. Note that SA-DQN, RADIAL-DQN and RADIAL-A3C agents all take deterministic actions, so we use the deterministic formulation of PA-AD described in Appendix C. In our implementation, we simply use CrossEntropy loss for the actor as in Equation (38).
gā s) argmingeenna (s) CrossEntropy (T('))))TT)))))))) ())).)).
(38)
Experiment Results In Table 7, we reproduce the results reported by Zhang et al. (2020a) and Oikarinen et al. (2020), and demonstrate the average rewards gained by these robust agents under different attacks in RoadRunner and BankHeist. Note that SA-DQN is claimed to be robust to attacks with budget e = 1/255, and RADIAL-DON and RADIAL-A3C are claimed to be relatively robust against up to c 3/255 attacks. (fxe is used in both papers.) So use the same c's for these agents in our experiments.
It can be seen that compared with vanilla agents in Table 1, SA-DQN, RADIAL-DON and RADIAL- A3C are more robust due to the robust training processes. However, in some environments, PA-AD can still decrease the rewards of the agent significantly. For example, in RoadRunner with
36

---

Published as a conference paper at ICLR 2022
3/255, RADIAL-DON gets 1k+ reward against our PA-AD attack, although RADIAL-DON under other attacks can get 10k+ reward as reported by Oikarinen et al. (2020). In contrast, we find that RADIAL-A3C is relatively robust, although the natural rewards gained by RADIAL-A3C are not as high as RADIAL-DDQ and SA-DQN. Also, as SA-DQN achieves over 98% action certification in BankHeist, none of the attackers is able noticeably reduce its reward with 1/255.
Therefore, our PA-AD can approximately evaluate the worst-case performance of an RL agent under attacks with fixed constraints, i.e., PA-AD can serve as a "detector" for the robustness of RL agents. For agents that perform well under other attacks, PA-AD may still find flaws in the models and decrease their rewards; for agents that achieve high performance under PA-AD attack, they are very likely to be robust against other attack methods.
E.2.6 IMPROVING ROBUSTNESS ON ATARI BY PA-ATLA
Note that different from SA-DQN (Zhang et al., 2020a) and RADIAL-RI (Oikarinen et al., 2020) discussed in Appendix E.2.5, use the traditional Atari configurations (Mnih et al., 2015) without any simplification (e.g. disabling frame stacking, or restricting action numbers). We aim to im- prove the robustness of the agents in original Atari environments, in real-world applications, the environments could be complex and unchangeable.
Baselines We propose PA-ATLA-A2C by combining our PA-AD and the ATLA framework pro- posed by Zhang et al. (2021). We implement baselines including vanilla A2C, adversarially trained A2C (with MinBest (Huang al., 2017) and MaxDiff (Zhang et al., 2020a) adversaries attacking 50 frames). SA-A2C (Zhang et al., 2020a) is implemented using SGLD and convex relaxations in Atari environments.
In Table 6, naive adversarial training methods have unreliable performance under most strong attacks and SA-A2C is ineffective under PA-AD strongest attack. To provide evaluation using different e, we provide the attack rewards of all robust models with different attack budgets Under all at- tacks with different e value, PA-ATLA-A2C models outperform all other robust models and achieve consistently better average rewards across attacks. We can observe that our PA-ATLA-A2C training method can considerably enhance the robustness in Atari environments.
MinBest
Mashiff Zhang et (2122a)
Avcrage
(2817)
Zhang (2021)
vamillia
A3C
1228
3/255
1/255
1064 a: 129
12.23
77
697 = 158
972 99
1085 107
913 164
926 s 124
1132 30
284 +: 110
436 1:
777.2
971.6
Inds: MinBled Humig el al. (2017))
A2C
968 94
1/255
1/255
952 60
874
927 ::30
813
996 - 11
819) 27
9:00 + 105
196
704 4: 19
521 72
774-2
887.8
(adv: MaxDeT Zhang (20284))
ABC
743
3/255
1/255
712
766
700 89
691 115
762 79
740
629 45
SA-A2CZhing et (222)
152
1/255
1051 +
900 89
1070 42
1007 + 18
636 1:
985.8
3/255
985 47
796 52
923 52
972 : 126
644 :: 163
962.0
PA-ATLA-AAC
1076
36
1/255
1055 =: 204
957 0: 78
1060 94
1046 +: 1443
882 #: 106
907.6
(ours)
1/255
10:00 78
80
+ 132
9136
Table 8: Average episode rewards + standard deviation over 50 episodes of A2C, A2C with adv. training. SA-A2C and PA-ATLA-A2C robust models under different evasion attack methods in Atari environment BankHeist. In each row, we bold the strongest attack. The gray cells are the most robust agents with the highest average rewards across all attacks.
F ADDITIONAL DISCUSSION OF OUR ALGORITHM
F.1 OPTIMALITY OF OUR RELAXED OBJECTIVE FOR STOCHASTIC VICTIMS
Proof of Concept: Optimality Evaluation in A Small MDP
We implemented and tested heuristic attacks and our PA-AD in the 2-state MDP example used in Appendix B.6, and visualize the results in Figure 14. For simplicity, assume the adversaries can perturb only perturb the policy directly based on their objective functions. As shown in Figure 14a, all possible ((s1)* I's form disk in the policy simplex, and executing above methods, as well as PA-AD, leads to 4 different policies on this disk. All these computed policy perturbations are on the boundary of the policy perturbation ball, justifying our Theorem 4.
81 within a E2 norm ball of radius 0.2. And we let all adversaries perturb
37

---

Published as a conference paper at ICLR 2022
Counday
336
LIE poiny Guallia
Narbest
540
M921
1921
(a)
(b)
(c)
Figure 14: Comparison of the optimality of different adversaries. (a) The policy perturbation generated for si by attack methods. (b) The values corresponding policy perturbations. (c) zoomed version (b), where the values of all possible policy perturbations are rendered. Our method finds the policy perturbation
that achieves the lowest reward among all perturbations.
As our theoretical results suggest, the resulted value vectors lie on line segment shown in Fig- ure 14b and a zoomed in version Figure 14c. where one can see that MinBest, Max Worst and MAD all fail to find the optimal adversary (the policy with lowest value). On the contrary, our PA-AD finds the optimal adversary that achieves the lowest reward over all policy perturbations.
For Continuous MDP: Optimality Evaluation in CartPole and MountainCar
We provided comparison between SA-RL and PA-AD in the CartPole enviromment in Figure 15, where we can see the SA-RL and PA-AD converge to the same result (the learned SA-RL adversary and PA-AD adversary have the same attacking performance).
Reward
Figure 15: Learning curve SA-RL and PA-AD attacker against an A2C victim in CartPole.
CartPole has 4-dimensional state space, and contains discrete actions. Therefore since SA-RI. has an optimal formulation, we expect SA-RL to converge to the optimal adversary in small MDP like CartPole. Then the result in Figure suggests that our PA-AD algorithm, although with relaxation in the actor optimization, also converges to the optimal adversary with even a faster rate than SA-RL (the reason is explained Appendix E.2.3).
In addition to CartPole, also experiments in MountainCar with 2- 2-dimensional state space against DQN victim. The SA-RL attacker reduces the victim reward to -128, and our PA-AD attacker reduces the victim reward to 199.45 within the same number of training steps. Note that the lowest reward MountainCar -200, so our PA-AD indeed converges to a near-optimal adversary. while SA-RL fails converge to near-optimal adversary. This is because MountainCar is environment with relatively spare rewards. The actor in PA-AD utilizes our Theorem 4 and only focuses on perturbations in the outermost boundary, which greatly reduces the exploration burden in solving an RL problem. In contrast, SA-RL directly uses RL algorithms to learn the perturbation, and thus has difficulties in converging to the optimal solution.
F.2 MORE COMPARISON BETWEEN AA-RL AND PA-AD
We provide a more detailed comparison between SA-RL and PA-AD from the following multiple aspects to claim our contribution.
38

---

Published as a conference paper at ICLR 2022
1. Size of the Adversary MDP
Suppose the original MDP has size S) |A] for its state space and action space, respectively. Both PA-AD and SA-RL construct an adversary's MDP and search for the optimal policy in it. But the adversary's ' MDPs for PA-AD and SA-RL have different sizes.
PA-AD: state space is of size [S], action space is of size R141-1 for a stochastic victim, or |A| for deterministic victim. SA-RL: state space is of size |S], action space is of size |S].
2. Learning Complexity and Efficiency
When the state space is larger than the action space, which is very common in RL environments, PA- AD solves smaller MDP than SA-RL and thus more efficient. In environments with pixel-based states, SA-RL becomes computationally intractable, while PA-AD still works. It is also important note that the actor's argmax problem in PA-AD further accelerates the convergence, it rules out the perturbations that do not push the victim policy to its outermost boundary. Our experiment and analysis in Appendix E.2.3 verify the efficiency advantage of our PA-AD compared with SA-RL, even in environments with small state spaces.
Optimality
PA-AD: (1) the formulation is optimal for a deterministic victim policy; (2) for a stochastic victim policy, the original formulation is optimal, but practical implementations, a relaxation is used which may not have optimality guarantees.
SA-RL: the formulation is optimal.
Note that both SA-RL and PA-AD require training an RL attacker, but the RL optimization process may not converge to the optimal solution, especially in deep RL domains. Therefore, SA-RL and PA-AD are both approximating the optimal adversary in practical implementations.
4. Knowledge of the Victim
PA-AD: needs to know the victim policy (white-box), Note that in a black-box setting, PA-AD can still be used based on the transferability of adversarial attacks in RL agents, as verified by Huang et al. (2017). But the optimality guarantee of PA-AD does not hold in the black-box setting.
SA-RL: does not need to know the victim policy (black-box).
should be noted that the white-box setting is realistic and helps in robust training:
(1) The white-box assumption is common in existing heuristic methods.
(2) It always white-box process to evaluate and improve the robustness of a given agent, for which PA-AD is the SOTA method. As discussed in our Ethics Statement, the ultimate goal of find- ing the strongest attacker is to better understand and improve the robustness of RL agents. During the robust training process, the victim is the main actor one wants to train, so it is a white-box set- ting. The prior robust training art ATLA (Zhang et al., 2021) uses the black-box attacker SA-RL, despite the fact that it has white-box access to the victim actor. Since SA-RL does not utilize the knowledge of the victim policy. it usually has to deal with a more complex MDP and face converg- ing difficulties. In contrast, if one replaces SA-RL with our PA-AD, PA-AD can make good use of the victim policy and find a stronger attacker with the same training steps as SA-RI,, as verified in Section 6 and Appendix F.2.6.
5. Applicable Scenarios
SA-RL is a good choice if (1) the action space is much larger than the state space in the original MDP, or the state space is small and discrete: (2) the attacker wants to conduct black-box attacks. PA-AD is good choice if (1) the state space is much larger than the state space in the original
MDP; (2) the victim policy is known to the attacker; (3) the goal is to improve the robustness of one's own agent via adversarial training.
In summary. as we discussed in Section 4, there is a trade-off between efficiency and optimality in evasion attacks in RL. SA-RL has an optimal RL formulation, but empirical results show that SA-RL usually do not converge to the optimal adversary in a continuous state space, even in a low- dimensional state space (e.g. see Appendix F. for an experiment in MountainCar). Therefore, the difficulty of solving an adversary's MDP is the bottleneck for finding the optimal adversary. Our PA-AD, although may sacrifice the thcoretical optimality in some cases, greatly reduces the size and the exploration burden of the attacker's RL problem (can also be regarded as trading some estimation bias off for lower variance). Empirical evaluation shows our PA-AD significantly outperforms SA- RL in wide range of environments.
39

---

Published as a conference paper at ICLR 2022
Though PA-AD requires to have access to the victim policy, PA-AD solves smaller-sized RL problem than SA-RL by utilizing the victim's 's policy and can be applied on evaluating/improiing the robustness of RL policy. It is possible to let PA-AD work in black-box setting based on the transferability of adversarial attacks. For example, in a black-box setting, the attacker can train a proxy agent in the same environment, and use PA-AD to compute state perturbation for the proxy agent, then apply the state perturbation to attack the real victim agent. This is out of the scope of this paper, and will be part of our future work.
F.3 VULNERABILITY OF RL AGENTS
It is commonly known that neural networks are vulnerable adversarial attacks (Goodfellow al,, 2015). Therefore, it is natural that deep RL policies, which are modeled by neural networks, are also vulnerable to adversarial attacks (Huang et al., 2017). However, there are few works discussing the difference between deep supervised classifiers and DRL policies in terms of their vulnerabilities. In this section, we take step further and investigate the vulnerability of DRL agents, through comparison with standard adversarial attacks on supervised classifiers. Our main conclusion is that commonly used deep RL policies be instrinsically much more vulnerable small-radius adversarial attacks. The reasons are explained below.
1. Optimization process
Due to the different loss functions that RL and supervised learning agents are trained on, the size of robustness radius of an RL policy is much smaller than that of a vision-based classifier.
On the one hand, computer vision-based image classifiers are trained with cross-entropy loss. There- fore, the classifier is encouraged to make the output logit of the correct label to be larger than the logits of other labels to maximize the log probability of choosing the correct label. On the other hand, RL agents, in particular DQN agents, trained to minimize the Bellman Error instead. Thus the agent is not encouraged to maximize the absolute difference between the values of different ac- tions. Therefore, if we assume the two networks are lipschitz continuous and their lipschitz constants do not differ too much, it is clear that a supervised learning agent has a much larger perturbation radius than an RL agent.
To prove our claim empirically, we carried out simple experiment, we compare the success rate of target attacks of well-trained DQN agent Pong with image classifier trained the CIFAR-10 dataset with similar network architecture. For a fair comparison, we use the same image preprocessing technique, which is divide the pixel values by 255 and no further normalization is applied. On both the image-classifier and DQN model, we randomly sample a target label other than the model predicted label and run the same 100-step projected gradient descent (PGD) attack to minimize the cross-entropy loss between the model output and the predicted label. We observe that for a perturbation radius of 0.005 (1 norm), the success rate of a targeted attack for the image classifier is only 15%, whereas the success rate of targeted attack for the DQN model is 100%. This verifies our claim that a common RL policy is much more vulnerable to small-radius adversarial attacks than image classifiers.
2. Network Complexity
In addition, we also want to point out that the restricted network complexity of those commonly used deep RL policies could play an important role here. Based on the claim by Madry et (2018), a neural network with greater capacity could have much better robustness, even when trained with only clean examples. But for the neural network architectures commonly used in RL applications, the capacity of the networks is very limited compared to SOTA computer vision applications. For example, the commonly used DQN architecture proposed in Mnih et al. (2015) only has convolu- tional layers and 2 fully connected layers. But in vision tasks, a more advanced and deeper structure natural that the perturbation radius need for
(e.g. ResNet has 100 layers) is used. Therefore, attacking an RL agent is much smaller than the common radius studied in the supervised evasion attack and adversarial learning literature.
40


Reviews:
[{"review_id": "zmfGNMpG31", "reviewer": "Reviewer_73ny", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper's novelty lies in the fact that the attacks are crafted based on the mapping from the state space to the action space, which is more efficient than the previously proposed methods. And the technique is provably efficient and generates relatively stronger attacks than its alternatives. Studying adversarial attacks on RL is an important subject, and the paper is original and novel. \n \nHowever, some of the claims made in the paper are not accurate.\n \nFirst, on page 1, \"many existing attacks (Huang et al., Zhang et al., 2020) are based on heuristics...\". To the best of my knowledge, the attacks studied in (Zhang et al., 2020) are not based on heuristics, which are actually the results of the SA-MDP.\n \nSecond, on page 2, \"summary of contributions,\" the authors say they introduce a \"compact\" policy adversarial MDP. Can the authors explain \"compact\" in the context? Does it mean the policy space is compact?\n \nThird, in definition 6, when the authors define the action space, \\hat{\\mathcal{A}}, the sum of d_i should be one if I understand it correctly.\n \nFourth,in fig 4, since the victim observes \\tilde{s}, should the policy in the figures be written as \\pi(\\tilde{s})?\n \nHere are some other comments:\n \n1. In definition 2, there are assumptions on the MDP and \\pi that one needs to make sure the admissible adversarial policy set is connected and compact. Would the authors mind listing them in the main body of the paper? This is because not every MDP problem and \\pi satisfy these assumptions. People who want to apply your method better get notified of these conditions.\n2. The authors mention several times that their approach is more efficient because the action space is much smaller than the state space in most deep RL applications. However, the authors did not demonstrate and quantify such efficiency in the experiments, especially not in the tables presented. In figure 12, the authors did show that PA-AD converges faster. But it didn't show that the PA-AD approach is more computationally tractable.\n2. The authors list an incomplete list of \"related works to adversarial RL.\" Adversarial (deep?) RL has attracted much attention in recent years.  People studied adversarial RL for both robustness and security purposes. If the authors only want to talk about works that are developed for robustness purposes. The papers listed in the manuscript are almost complete. Otherwise, below are some works that I deem relevant. The authors can consider including them when discussing related works:\n      * Zhang, Xuezhou, et al. \"Adaptive reward-poisoning attacks against reinforcement learning.\" International Conference on Machine Learning. PMLR, 2020.\n     * Rakhsha, Amin, et al. \"Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning.\" International Conference on Machine Learning. PMLR, 2020.\n     * Huang, Yunhan, and Quanyan Zhu. \"Deceptive reinforcement learning under adversarial manipulations on cost signals.\" International Conference on Decision and Game Theory for Security. Springer, Cham, 2019.\n    *  Russo, Alessio, and Alexandre Proutiere. \"Towards optimal attacks on reinforcement learning policies.\" 2021 American Control Conference (ACC). IEEE, 2021.\n\n\n    \n\n", "summary_of_the_paper": "The paper proposes a method to craft stronger and more efficient attacks on state observation of the RL agent.  Here, the 'strongest' attacks refer to the attacks that minimize the reward the most. An 'efficient' attacker can craft such attacks using the least computational resources. In the work of (Zhang et al., 2021), the attacking strategy is described by a mapping from the state space of the underlying MDP problem to the same space. The agent observes the falsified states rather than the actual ones and takes actions based on the observed states. This misinformation eventually tricks the agent into taking a different action. In this paper, the authors argue that instead of finding the mapping from the state space to the same space, the attacker can first find the mapping from the state space to the action space that generates the least accumulated rewards to the agent. The 'strongest' attacks on the state observation can then be crafted based on the mapping found. Since in many deep RL applications, the state space is much larger than the action space. Hence, finding the mapping from the state space to the action space is more efficient than finding the one from the state space to the state space.", "summary_of_the_review": "In general, the paper is well-structured, and the problem studied here leads to a robust RL agent, which fuels the applicability of RL in real-world situations. However, some claims in the paper are not accurate. But these claims do not affect the main results. Also, there is still room to improve the writing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b4pj643Qc9_", "reviewer": "Reviewer_pYuF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I enjoyed reading the paper, and in my opinion, this paper contains interesting results and provides solid contributions to the line of work on adversarial attacks. The paper both formally justifies its approach and provides concrete experimental evidence that the approaches improves over the state-of-the-art methods. Admittedly, compared to prior work, some of the modeling assumptions might be restrictive - more detailed remarks about the paper (i.e., its strengths and weaknesses) are listed below. \n\n- *Novelty*: While the problem of designing optimal evasion attacks in deep RL has been studied by prior work, the key idea that the paper introduces seems rather novel. To my knowledge, this type of attack method has not been considered in the recent literature. \n\n- *Methodology*: The paper provides justifications grounded in theory. Utilizing the results from e.g. Dadashi et al. 2019, it shows that the novel framework admits an optimal solution, despite the fact that the proposed method searches over policies rather than over the state space. While this fact seems intuitive, I still find this result interesting and non-trivial. \n\n- *Modeling assumptions*: One drawback of the proposed approach is that it relies on the assumption that the attacker has access to the victim's policy. This assumptions is somewhat restrictive, corresponding to a white-box attack, and it seems to be critical for the formal analysis. Some of the SOTA method do not seem to require this assumption (SA-RL).\n\n- *Experiments*: The paper conducts extensive experiments, and empirical results show that the proposed approach outperforms the SOTA attack methods under the studied setting, and significantly so. \n\n- *Clarity*: The paper is clearly written and it provides intuitions behind the most important technical aspects. Minor comments: The paper contains a few typos, the most important one might be in Definition 6: why is the sum $\\sum_i d_i$ equal to  $0$? Is this supposed to be $\\sum_i d_i = 1$? Moreover, some of the references could be updated, e.g., Zhang et al. 2021 is referenced as an arXiv paper, but this paper seems to have appeared at ICLR2021.\n", "summary_of_the_paper": "The paper studies evasion attacks in deep reinforcement learning (RL). More specifically, the paper considers a novel approach to performing evasion attacks based on two-component design --- director and actor modules, where the latter perturbs a given state based on the policy direction that the former specifies. Effectively, the search for an optimal attack is performed in the policy space, and since the policy space is typically more compact than the state space, the search is more efficient. The paper formally justifies its design choices, and experimentally validates the efficacy of the propose approach, showing that it yields significant improvements compared to the state-of-the-art methods. \n", "summary_of_the_review": "This paper provides interesting contributions to the line of work on adversarial attacks. The paper contributes a novel attack method and shows that it outperforms the SOTA baselines. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Gkbv-5wHE_9", "reviewer": "Reviewer_mGyi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the paper very well-written and easy to follow. The idea is novel and the proposed method gives new insights about the evasion attacks. My only issue is that many significant details are left to the appendix while mentioning them in a short sentence could give the reader a better understanding of the limitations. For example\n\n- The algorithm is assuming access to the learner's policy. I don't mind this assumption since white-box attacks have their own purposes and scenarios, but the authors could be more direct about this. Especially considering the amount of comparison done with SA-RL which is a black-box attack. Seems like the advantages of PA-AD stem from this additional knowledge. I still find the method to utilize this knowledge is a respectable contribution.\n\n- The details of solving optimization problem (G) could be more explained in the main text. It is a challenging problem to solving without relaxation and \"can be implemented in various ways\" is to vague.\n\nEven though the above points leave room for improvement, I think the paper is a solid contribution.", "summary_of_the_paper": "The paper introduces a novel algorithm to find the optimal evasion attack against RL in scenarios where the agent's policy is known. The key idea is to only learn the aspect of problem depending on the environment and is unknown through a clever decomposition. The algorithm shows promising empirical results.", "summary_of_the_review": "The paper has some limitation on the knowledge assumptions and need for relaxation, but the contribution is enough in my opinion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BwadR9AJ_F0", "reviewer": "Reviewer_RTbW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The definition of admissible perturbations relies on that the attacks would be hard to be perceived. How can one determine a reasonable epsilon?\n\nThe attacker not knowing the victim policy yet not knowing the environment dynamics is relatively contrived.\n\nThere is an optimality argument in the paper for the problem posed in the paper. Yet, it is not clear whether this problem as posed is a realistic or even a relevant one. For example, the set B_epsilon^H, which is used in the problem statement itself, is hard obtain. \n\nAdditionally, the importance of the theoretical result even in this contrived setting is unclear because there is no guarantee that the algorithm devised in the paper will be able to compute that optimal. \n\nThe claims made in the paper are likely unjustifiably strong. For example, how can a few empirical examples be used to conclude that a method \"universally\" outperforms another one? What is even the universe?\n\n----------------\n\nComments after the author response:\n\nThe fact that there are other papers (even if they are popular papers) is not a justification of the importance of a problem. In an adversarial setting, if your adversary knows the policy, which itself can be a complicated object, you are implementing, then the problem you are studying is the least of your issue. You are your strongest enemy. Maybe consider changing the policy, first. \n\nYou are going to try to justify that this is the worst-case analysis and you are empowering the adversary. Then, why does the adversary not know anything about the environment? So, it is really not a worst-case analysis either. Also, isn't it odd that the adversary does not understand *anything* about the environment but understands a policy used in that environment?", "summary_of_the_paper": "The paper proposes a method for computing the strongest adversarial perturbation on state observations of an RL agent from a specified set of perturbations. It relates perturbations on states to perturbations on policies. It poses a specific problem to this end, develops a solution, and establishes its optimality. ", "summary_of_the_review": "The paper is possibly on an interesting topic yet the problem studied in the paper is contrived and the results are of limited interest. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall the paper makes good contributions to the area of robust deep reinforcement learning. The presentation needs to be improved to avoid any confusions. Please take all the reviews into account and revise the paper accordingly.
Decision: Accept (Poster)