Robust Reinforcement Learning on State Observations with Learned Optimal Adversary
Authors: Huan Zhang
Hongge Chen
Duane S Boning
Cho-Jui Hsieh
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
robustness
adversarial attacks
adversarial defense

Abstract: We study the robustness of reinforcement learning (RL) with adversarially perturbed state observations, which aligns with the setting of many adversarial attacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found, which is guaranteed to obtain the worst case agent reward. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones. To enhance the robustness of an agent, we propose a framework of alternating training with learned adversaries (ATLA), which trains an adversary online together with the agent using policy gradient following the optimal adversarial attack framework. Additionally, inspired by the analysis of state-adversarial Markov decision process (SA-MDP), we show that past states and actions (history) can be useful for learning a robust agent, and we empirically find a LSTM based policy can be more robust under adversaries. Empirical evaluations on a few continuous control environments show that ATLA achieves state-of-the-art performance under strong adversaries. Our code is available at https://github.com/huanzhang12/ATLA_robust_RL.

Published as a conference paper at ICLR 2021
ROBUST REINFORCEMENT LEARNING ON STATE OB- SERVATIONS WITH LEARNED OPTIMAL ADVERSARY
Huan Zhang" Hongge Chen "2 Duane Boning²
Cho-Jui Hsieh'
'Department of Computer Science, UCLA "Department of EECS, MIT huan@huan- -zhang. com, chenhg@mi t. edu, boning@mtl mi ..edu chohsi eh@cs. ucla. aolla.dd. edu
"Huan Zhang and Hongge Chen contributed equally.
ABSTRACT
We study the robustness of reinforcement learning (RL) with adversarially per- turbed state observations, which aligns with the setting of many adversarial at- tacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise. With fixed agent pol- icy, we demonstrate that an optimal adversary to perturb state observations can be found, which is guaranteed to obtain the worst case agent reward. For DRL set- tings, this leads to a novel empirical adversarial attack to RL agents via learned adversary that is much stronger than previous ones. To enhance the robustness of an agent, we propose a framework of alternating training with learned adver- saries (ATLA), which trains an adversary online together with the agent using policy gradient following the optimal adversarial attack framework. Additionally, inspired by the analysis of state-ad adtesarallaal Markov decision process (SA-MDP), we show that past states and actions (history) can be useful for learning a robust agent, and we empirically find a LSTM based policy can be more robust under ad- versaries, Empirical evaluations on a few continuous control environments show that ATLA achieves state-of-the-art performance under strong adversaries. Our code is available at httph::///// // //g thub. com/ hont/huanzhangg 12/ 2/ATLA LECeLAArobust_t__
INTRODUCTION
Modern deep reinforcement learning agents (Mnih et al. 2015 Levine et al. 2015 Lillierap et al,
2015 Silver et al. (2016] Fujimoto et al., 2018) typically use neuron networks as function approxima-
tors. Since the discovery of adversarial examples in image classification tasks (Szegedy ct al. 2013).
the vulnerabilities in DRL agents were first demonstrated in (Huang et al. 2017 Lin et al. 2017
Kos & Song 2017) and further developed under more environments and different attack scenar- ios (Behzadan & Munir 2017a| Pattanaik al., [0188] Xiao et al.) (219)). These attacks commonly add imperceptible noises into the observations of states, e.g., the observed environment slightly dif- fers from true environment. This raises concerns for using RL in safety-crucial applications such as autonomous driving (Sallab et al., [2017] Voyage 2019); additionally, the discrepancy between ground-truth states and agent observations also contributes to the "reality gap" - an agent working well in simulated environments may fail in real environments due to noises in observations (Jakobi al. 1995 Muratore et al. 2019), as real-worke sensing contains unavoidable noise (Brooks 1992).
We classify the weakness of DRL agent on the perturbations of state observations into two classes: the vulnerability in function approximators, which typically originates from the highly non-linear and blackbox nature of neural networks; and intrinsic weakness of policy: even perfect features for states are extracted, an agent can still make mistakes due to an intrinsic weakness in its policy.
For example, in the deep Q networks (DQNs) for Atari games, a large convolutional neural network (CNN) is used for extracting features from input frames. To act correctly, the network must extract crucial features: e.g., for the game of Pong, the position and velocity of the ball, which can observed
by visualizing convolutional layers (Hausknecht & Stone 2015] Guo et al. 2014). Many attacks to the DQN setting add imperceptible noises (Huang et al. 2017: Lin al. 2017| Kos & Song 2017
Behzadan & Munir 20)7a) that exploit the vulnerability of deep neural networks so that they extract
wrong features, we have seen in adversarial examples of image classification tasks. On the other

---

Published as a conference paper at ICLR 2021
0.00
0.73
0.81
11753
0.88
42L
.520
-0.0
-00
-00
0.50
0.58
00
0.01
0.73
4.74
6.28
4.0
0.0
0.0
0.50
0.:
0.0
110
0.8
0.81
5.25
-00
-00
0.58
0.73
0.9
Ralino
0.81
4.25
0.
0.
0.06
0.78
0.11
3.34
4.74
4.0
-00
400
(a) Path in unperturbed environ- ment (found by policy iteration). Agent's reward = +1. Black ar- rows and numbers show actions and value function of the agent.
(b) Path under the optimal adver- sary. Agent's reward = -oo. Red arrows and numbers show actions and value function of the optimal adversary (Section B.T).
(c) A robust POMDP policy solved by SARSOP (Kurniawati ct al. 2008) under the same ad- versary. This policy is history de- pendent (Section 3.2
Figure I: We show agent in gridworld environment trained with function approximators, and its optimal policy is intrinsically not robust to perturbations of state observations. The red square and blue circle are the starting point and target (reward +1) of the agent, respectively. The green triangles are traps, with reward- - once encountered. The adversary is allowed to perturb the observation to adjacent states along four directions: up. down, left, and right. Adversary earns +1 at traps and -1 at the target. We set y = 0.9 for both agent and adversary. This example shows that the vulnerability of a RL agent does not only come from the errors in function approximators such as DNNs.
hand, the fragile function approximation is not the only source of the weakness of a RL agent- in a finite-state Markov decision process (MDP), we can use tabular policy and value functions so there is no function approximation error. The agent can still be vulnerable to small perturbations observations, c.g., perturbing the observation of a state to one of its four neighbors in a gridworld- like environment can prevent an agent from reaching its goal (Figure D]. To improve the robustness of RI, we need to take measures from both aspects - a more robust function approximator, and a policy aware of perturbations in observations.
Techniques developed in enhancing the robustness of neural network (NN) classifiers can be ap- plied to address the vulnerability in function approximators. Especially, for environments like Atari games with images as input and discrete actions as outputs, the policy network my behaves similarly to a classifier in test time. Thus. Fischer al. (2019): Mirman al. (20I8a) utilized existing cer- tified adversarial defense (Mirman et al.) 2018b Wong & Kolter] 2018 Gowal et al., 2018 Zhang al. 2020a) approaches in supervised learning to enhance the robustness of DQN agents. Another successful approach (Zhang et al.) 2020b) for both Atari and high-dimensional continuous control environment regularizes the smoothness of the learned policy such that max)e (s), no (8)) small for some divergence D and ((s) is a neighborhood around s. This maximization can be solved using a gradient based method or convex relaxations of NNs (Salman et al., 2019) Zhang al. 2018 Xu et al., 2020), and then minimized by optimizing 0. Such an adversarial minimax regularization is in the same spirit as the ones used in some adversarial training approaches for (semi- ssupervised learning, e.g., TRADES (Zhang et al. 2019) and VAT (Miyato et al. 2015). However, regularizing the function approximators does not explicitly improve the intrinsic policy robustness.
In this paper, we propose an orthogonal approach, alternating training with learned adversaries (ATLA), to enhance the robustness of DRL agents. We focus on dealing with the intrinsic weakness of the policy by learning an adversary online with the agent during training time, rather than directly regularizing function approximators. Our main contributions can be summarized as:
We follow the framework of state-adversarial Markov decision process (SA-MDP) and show how to learn an optimal adversary for perturbing observations. We demonstrate practical attacks under this formulation and obtain learned adversaries that are significantly stronger than previous ones.
We propose the alternating training with learned adversaries (ATLA) framework to improve the robustness of DRL agents. The difference between our approach and previous adversarial training approaches is that we use a stronger adversary, which is learned online together with the agent.
SA-MDP also shows that history can be important for learning robust agent. We thus propose to use LSTM based policy in the ATLA framework and find that is more robust than policies parameterized as regular feedforward NNs.
Our analysis
2

---

Published as a conference paper at ICLR 2021
We evaluate approach empirically four continuous control environments. We outperform explicit regularization based methods in a few environments, and our approach can also be directly combined with explicit regularizations on function approximators to achieve ofte-off-hhean art results.
2 RELATED WORK
State-adversarial Markov decision process (SA-MDP) (Zhng et al. 2020b) characterizes the de- cision making problem under adversarial attacks on state observations. Most importantly, the true state in the environment is not perturbed by the adversary under this setting; for example, perturbing pixels in an Atari environment (Huang et al. 2017 Kos & Song 2017: Lin et al. 2017: Behzadan & Munir 2017a Inkawhich et al. (2119) does not change the true location of an object in the game simulator. SA-MDP can charactenize agent performance under natural or adversarial noise from sen- car naturally noisy, but the ground truth location of the car is not affected by the noise. Importantly, this setting is different from robust Markov decision process (RMDP) (Nilim & El Ghaoui 2004; [yengar 2005), where the worst case transition probabilities of the environment are considered. "Robust reinforcement learning" in some works (Mankowitz et al.) 2018; 2019) refer to this different definition of robustness in RMDP, and should not be confused with our setting of robustness against perturbations on state observations.
measurements. For example, GPS sensor readings
Several works proposed methods to learn an adversary online together with an agent. RARL (Pinto et al. 2017) proposed to train an agent and an adversary under the two-player Markov game (Littman 1994) setting. The adversary can change the environment states through actions directly applied to environment. The goal of RARL is to improve the robustness against environ- ment parameter changes, such as mass, length or friction. Gleave et al. (2019) discussed the learning of an adversary using reinforcement learning to attack a victim agent, by taking adversarial actions that changes the environment and consequentially change the observation of the victim agent. Both Pinto et al. (2017): : Gleave et al. (2019) conduct their attack under the two-player Markov game framework, rather than considering perturbations on state observations. Besides, [i et al.] (2019) consider similar Markov game setting in multi-agent RL environments. The difference between these works and ours can be clearly seen in the setting where the adversary is fixed under the framework of (Pinto al. 2017 Gleave et al. 2019), the learning of agent is still MDP, but in our setting, it becomes a harder POMDP problem (Section 3.2)
Training DRL agents with perturbed state observations from adversaries have been investigated in a few works, sometimes referred to as adversarial training. Kos & Song (2017); Behzadan & Munir 20176) used gradient based adversarial attacks to DQN agents and put adversarial frames into replay buffer. This approach is not very successful because for Atari environments the main source of weakness is likely to come from the function approximator, so an adversarial regularization framework such as (Zhang et al., 2020b Qu et al.) (020) which directly controls the smoothness of the Q function is more effective. For lower dimensional continuous control tasks such as the Pattanaik et al. (2018) conducted FGSM and multi- MuJoCo environments, Mandlekar et al. 2
step gradient based attacks during training time; however, their main focus was on the robustness against environment parameter changes and only limited evaluation
the adversarial attack setting was conducted with relatively weak adversaries. ZZhang et al. (2020b) systematically tested this approach under newly proposed strong attacks, and found that it cannot reliably improve robustness. These early adversarial training approaches typically use gradients from a critic function. They are usually relatively weak, and not sufficient to lead robust policy under stronger attacks.
The robustness of RL has also been investigated from other perspectives. For example, Tessler et al. (019)) study MDPs under action perturbations: Tan al. (2020) use adversarial training on action space to enhance agent robustness under action perturbations. Besides, policy teaching (Zhang & Zhang et al. 2009 Ma et al. 2019) and policy poisoning (Rakhsha et al. 2020)|uang
Parkes 2008:
& Zhu 2019 manipulate the reward or cost signal during agent training time to induce desired
agent policy. Essentially, policy teaching is a training time "attack" with perturbed rewards from the environments (which can be analogous to data poisoning attacks in supervised learning settings), while our goal is to obtain a robust agent against test time adversarial attacks. All these settings differ from the setting of perturbing state observations discussed in our paper.
3 METHODOLOGG
In this section, we first discuss the case where the agent policy is fixed, and then the case where the adversary is fixed in SA-MDPs. This allows us to propose an alternating training framework to improve robustness of RL agents under perturbations on state observations.
3

---

Published as a conference paper at ICLR 2021
Notations and Background We use S and A represent the state space and the action space, respectively; P(S) defines the set of all possible probability measures on S. We define a Markov decision process (MDP) as (S, A. R. p.?). where R S x A x S + R and p : S x A P(S) are two mappings represent the reward and transition probability. The transition prohability at
time step can be written ps
Pr(se+1 = s'1s;
8,0, = =a). Reward function
is defined as the expected reward R[ss,,,5) == B/rr|s = 8, at = 0,41+1 = s'). E [o. I] the discounting factor. We denote stationary policy as π 8 + P(A) which is inde- st)
pendent of history. We denote history hn at time as fsu, ag. ... 8t-1, St) and H. as the set of all histories. history-dependent policy is defined as π H + P(A). A par- tially observable Markov decision process (Astrom) 1965) (POMDP) can be defined as 7-tuple (S, A, O, Q p. 7) where O set of observations and Ω is a set of conditional observation probabilities p(ols). Unlike MDPs, POMDPs typically require history-dependen optimal policies.
See 3.1: Fix agenn find optinal adversury
See 3.2: Fix adversory, optinal agent policy
To study the decision problem under adversaries state observations, we use state-adversarial Markov decision process (SA-MDP) framework (Zhang et al. 2020b). In SA-MDP, an adversary v/ S P(S) is introduced to perturb the input state of an agent; however, the en- vironment state is unchanged (Figure 2). Formally, an SA-MDP is a 6-tuple (S, B, R. p. 7) where B is map- ping from a state E S to set of states B(s) E S. The agent sees the perturbed state ~ r(-[s) and takes the action T[alS) accordingly. B limits the power of adver- sary: supp (v(-[s)) E B(s). The goal of SA-MDP is to solve an optimal policy x² under its optimal adversary v² (TT";; an optimal adversary is defined as v²(T) such that achieves the lowest possible expected discounted return (or value) on all states. Zhhag et al. (2020b) did not give an explicit algorithm to solve SA-MDP and found that stationary optimal policy need not exist.
Adversary
Agent
Environment
Figure 2: SA-MDP introduces an adver- sary on state observations in a MDP.
3.1 FINDING THE OPTIMAL ADVERSARY UNDER A FIXED POLICY
In this section, we discuss how to find an optimal adversary v for a given policy m. An optimal adversary leads to the worst case performance under bounded perturbation set B, and is absolute lower bound of the expected cumulative reward an agent can receive. It is similar to the concept of "minimal adversarial example" in supervised learning tasks. We first show how solve the optimal adversary in MDP setting and then apply it to the DRL, settings.
A technical lemma (Lemma 1) from Zhang al. (020b) shows that, from the adversary's point of view, a fixed and stationary agent policy π and the environment dynamics can be essentially merged into an MDP with redefined dynamics and reward functions:
and a fixed and sta- tionary policy π (-I-),, there exists an MDP M (S, Å, R. p. 7) such that the optimal policy of M is the optimal adversary u for SA-MDP given the fixed n, where A = S, and
Lemma 1 (Zhang et al. (2020b)) Given an SA-MDP M (S.A,R,B.P))
R(s, s') = E[j s7 Easa (s.
for s, s' S and à E B(s)) A, for s, S and & B(s).
Eeea apaaa)p(s (s.a)
where C a large negative constant, and
p()'ls,a)= = Σ ((ala)p(<(1s,)) for s, E E and à E Á.
aEA
The intuition behind Lemma 1 is that the adversary's goal is to reduce the reward earned by the agent. Thus, when a reward T, is received by the agent at time step t, the adversary receives negative reward of f. = To prevent the agent from taking actions outside of set B(s), a large negative reward C' is assigned to these actions such that the optimal adversary cannot take them. For actions within B(s), we calculate R(s, à, s') by its definition, R(ss, à') == EF ss à, s'] which yields the term in Lemma] [ The proof can be found in Appendix B of Zhang et al. (2020b).
After constructing the MDP M, it is possible to solve an optimal agent y of M, which will be the optimal adversary on SA-MDP M given policy æ. For MDPs, under mild regularity assumptions an
4

---

Published as a conference paper at ICLR 2021
Moving right
Not moving
Moving left
Moving right
Not maving
Moving left
(a) No attack (reward 5851)
(b) RS Attack (reward 284)
(c) Our Attack (reward 11400 (reward 7094)
(d) No attack
(c) RS Attack (reward 85)
(f) Our Attack (reward -743)
Figure 3: Our "Optimal" Attack and Robust Sarsa attack (a previous strong attack proposed in Zhang et al. (2020b)) on Ant and HalfCheetah environments. Previous strong attacks make the agent fail and receive small positive reward (less than 1/10 of the reward without attack). Our attack strong enough to trick the agent into moving to the opposite direction, receiving a large negulive reward.
optimal policy always exists (Puterman) 2014). our case, the optimal policy on M corresponds to optimal adversary in SA-MDP, which is the worst case perturbation for policy л. As an illustration, in Figure [ we show a GridWorld environment. The red square is the starting point. The blue circle and green triangles are the target and traps, respectively. When the agent hits the target, it earns reward +1 and the game stops and earns reward -1 when it encounters trap. We set y 0.9 for both agent and adversary. The adversary is allowed perturb the observation to adjacent cells along four directions: up, down, left, and right. When there is no adversary, after running policy iteration, the agent can easily reach target and earn a reward of +1, in Figure Ta However, if we train the adversary based on Lemma ! and apply it to the agent, we are able to make the agent repeatedly encounter a trap. This leads 0O reward for the agent and +oo reward for the adversary as shown in Figure Tb
We now extend this Lemma | to the DRL setting. Since the learning of adversary equivalent to solving an MDP, we parameterize the adversary as neural network function and use any popular DRL algorithm to learn an "optimal" adversary. Here we quote the word "optimal" as we use function approximator to learn the agent so it's no longer optimal, but we emphasize that it follows the SA-MDP framework of solving an optimal adversary. No existing adversarial attacks follow such theoretically guided framework. We show to produce E (s) directly, since (s)) is usually a small set nearby s (e.g., {s'll|s- s S c), our adversary learns a perturbation vector A, and project A to B(s).
algorithm in Algorithm [ Instead of learning
The first advantage of attacking a policy in this way that strong as we allow to optimize the adversary in online loop of interactions with the agent policy and environment, and keep improving the adversary with goal of receiving as less reward as possible. is strong because it follows the theoretical framework of finding optimal adversary, rather than using any heuristic to generate a perturbation. Empirically, in the cases demonstrated in Figure 3 previous strong attacks Robust Robust Sarsa attack) can successfully fail an agent and make it stop moving and receive a small
positive reward; our learned attack can trick the agent into moving toward the opposite direction of the goal and receive a large negative reward. We also find that this attack can further reduce the reward of robustly trained agents, like SA-PPO (Zhang et al.) 2020b).
The second advantage of this attack that it requires no gradient access to the policy itself; in fact, it treats the agent part of the environment and only needs to run it in blackbox. Previous attacks (e.g., Lin et al. (2077); Pattanaik et al. (2088); Xiao et al. (2019)) are mostly gradient based approach and need access the values gradients access to gradients, the overall learning process is still just a MDP and we can apply any popular modern DRL methods to learn the adversary.
policy or value function. Even without
3.2 FINDING THE OPTIMAL POLICY UNDER FIXED ADVERSARY
We now investigate SA-MDP when we fix the adversary and find an optimal policy. In Lemma we show that this case SA-MDP becomes a POMDP:
Lemma (Optimal policy under fixed adversary) Given an SA-MDP M = -S,,A,B.RPP.1 and fixed and stationary adversary v(8ss), there exists a POMDP M -8, O.P.
5

---

Published as a conference paper at ICLR 2021
Algorithm Learning an "optimal" adversary for perturbations on state observations
Input: Policy :((ls) under attack, number of iterations Niters batch size B, perturbation set (s))
1: initialize adversary us (-1s) parameterized by a neural network with parameters Ø,
2: for 1 to Nater do
3: 'D < Adv Adv_Traj(us T, B)
# #colecttio of samples (for simplicity we ignore episodes here)
4: Φ PolieyOptimize((D.c
5: end for
Function Adv Traj( n. B):
6: 8 +- 80
Initial state
7: D <-
8: for
to B do
9:
Á sample from (-(ss)
10:
å +- (s) + A)
# projection will be clipping for for norm set B(s)
a f sample from (-|6))
12:
obtain current step reward Tr, next state s' from environment given action # state, action, reward and next state for the adversary
13:
14:
8+
15:
end for
16:
return D
such that the optimal policy of M is the optimal policy x for SA-MDP given the fixed v. where
Ω = Ut''s supp(u(-1s))}
O = v((ss)
(1)
#ES
where Ω is the set of observations, and O defines the conditional observational probabilities (in our case conditioned only on and does not depend on actions). To prove Lemma [ we construct the a POMDP with the observations defined on the support of all v(-(s), E S and the observation process exactly the process of generating an adversarially perturbed state This POMDP functionally identical to the original SA-MDP when is fixed. This lemma unveils the connection between POMDP and SA-MDP: SA-MDP can be seen as a version of "robust" POMDP where the policy needs to be robust under set of observational processes (adversaries). SA-MDP is different from robust POMDP (RPOMDP) (Osogami 2015 Rasouli Saghafian, 2018 which optimizes for the worst
environment transitions.
As proof of concept, we use a modern POMDP solver, SARSOP (Kurniawati et al. 2008 to solve the GridWorld environment in Figure 1 to find policy that defeat the adversary. The POMDP solver produces a finite state controller (FSC) with states (FSC an efficient representation of history dependent policies). This FSC policy can almost eliminate the impact of the adversary and receive close to perfect reward, as shown in Figure Ic
Unfortunately, unlike MDPs, is challenging solve an optimal policy for POMDPs; state-of-the- art solvers (Bai et al., 2014 Sunberg & Kochenderfer 2017) can only work on relatively simple environments which are much smaller than those used in modern DRL. Thus, we do not aim to solve the optimal policy. We follow (Wierstra et al.) (007) to use recurrent policy gradient theorem on POMDPs and use LSTM as function approximators for the value and policy networks. We denote h; = (So. ag. $1. aj ... &t} containing all history of states (perturbed states in our setting) and actions. The policy æ parameterized by 0 takes an action a, given all observed history hts and h is typically encoded by recurrent neural network (e.g., LSTM). The recurrent policy gradient theorem (Wierstra et al. 2007) shows that
N
T
V
Vo log no (a(aa (a"(
1
where is the number of sampled episodes, T is episode length (for notation similarity, we assume each episode has the same length), and he is the history of states for episode n up to time t, and r the reward received for episode n at time t. We can then extend Eq. modern DRL algorithms such proximal policy optimization (PPO), similarly done in (Azizzadenesheli al. 2018), by

---

Published as a conference paper at ICLR 2021
using the following loss function:
((0) 1
min To (h (T (aū h(g)
Ang clip( To (o (n; Tiau (0]))))
(11+)))An )]
(3)
where A&p is baseline advantage function for episode n time step L, which is based on a LSTM value function. e is the clipping threshold in PPO. The loss can be optimized via a gradient based optimizer and Ooid is the old policy parameter before optimization iterations start. Although LSTM or recurrent policy network has been used in the DRL setting in few other works (Hausknecht & Stone 2013 Azizzadeneshhlii al.) 2018). our focus is improve agent robustness rather than learning a policy purely for POMDPs. In our empirical evaluation, we will compare feedforward and LSTT policies under our ATLA framework.
3.3 ALTERNATING TRAINING WITH LEARNED ADVERSARIES (ATLA)
we can solve an optimal adversary given any fixed policy. In our ATLA framework, we train such an adversary online with the agent: we first keep the agent and optimize the adversary: the adversary is also parameterized as neural network. Then we keep the adversary and optimize the agent. Both adversary and agent can be updated using policy gradient algorithm such as PPO. We show our full algorithm in Algorithm 2
As we have discussed in Section 3.1
Algorithm 2 Alternating Training with Learned Adversaries (ATLA)
Input: Environment &, number of iterations Naters and batch size B.
1: Initialize the agent's actor network Ta) with parameters 0.
2: Initialize the adversary's actor network vS]]s) with parameters
3: for iori= to Nater do N₂ do
4: for j =
5: Run Te with fixed vs to collect a set of trajectories D., : {( (i kij kJ skj (
,
6: 0 + PoliicyOptimizer(DD..
7:
end for
8:
for j = to N₂ do D. f- Adv Traj(ve, no. B) <- PolicyOptimizer(D...
9:
# Adv Traj defined Algorithm
10:
end for
12:
end for
Our algorithm is designed to use a strong and learned adversary that tries find intrinsic weakness of the policy, and to obtain good reward the policy must learn to defeat such an adversary. In other words, it attempts to solve the SA-MDP problem directly rather than relying on explicit regu- larization on the function approximator like the approach in (Zhang et al. (2200b). In our empirical evaluation, we show that such regularization can be unhelpful in some environments and harmful for performance when evaluating the agent without attacks.
The difference between our approach and previous adversarial training approaches such as (Pat- tanaik et al. 2018) is that we use stronger adversary, learned online with the agent. Our empirical evaluation finds that using such a learned "optimal" adversary in training time allows the agent to learn robust policy generalized to different types of strong adversarial attacks during test time. Additionally, it is important to distinguish between the original state and the perturbed state s. We find that using instead of é to train the advantage function and policy of the agent leads worse performance, as it does not follow the theoretical framework of SA-MDP.
4 EXPERIMENTS
"Optimal" attack on DRL agents" In section 3.1 we show that it is possible cast the optimal adversary finding problem as an MDP problem. In practice, the environment dynamics are unknown but model-free RL methods can be used to approximately find this optimal adversary. In this section, we use PPO to train an adversary on four OpenAl Gym MuloCo continuous control environments.
"Code for the optimal attack and ATL. available https: http://////tttttiocoo///aaaaaaaa

---

Published as a conference paper at ICLR 2021
Table 1: Average episode rewards + standard deviation over 50 episodes on PPO and SA-PPO agents. We report natural rewards (no attacks) and rewards under six adversarial attacks, including a simple random noise attack, the critic based attack in Pattanaik et al. (2018),MA) and RS attacks in Zhang et al. (2020b), Snooping attack proposed in Inkawwhich et al. (201)) and the optimal attack proposed in this paper. each row we bold the best (lowest) attack reward over all five attacks. "Optimal" attack is better than other attacks in all environments, sometimes by large margin.
Env.
perturb ation budger e
Method
Natural Reward
Critic
Random
MAD
Attack Reward
Snooping
"Optimal"
Hopper
0.075
SA-PDO
PPO
31674-521
37051 2
1464 +-223
37891 15
271011 801
2101+793
2652.1 835
1410++ 655
2234+1 100
2509
1130 1:12
794+-23
1076.1 791
636+ 9
Walker2d
0.05
PPO
4472
4487
1 635
3424 = 1296
30
3007 1: 1200
39
2869 4: 1271
36681 1789
392811661
27861962
1336 654
38081
2908.
1086::16
PPO
5687 1 758
19311 1022
32611 1005
17591 828
36681517
268 1227
.82 436
Ant
0.15
SA-PPO
42424 184
4805 128
4986 1:452
4662 1:522
407947668
3412 11755
2511
1117
HalfChhetth
0.15
PFU
7117
37611119
5186 1 1378
18361 866
10371811
1891175 758
-00. 219
SA-PPO
3589.1
18
3624.1 23
3616.121
1
Table presents results on attacking vanilla PPO and robustly trained SA-PPO (Zhang et al. 2020b)
agents, As comparison, we also report the attack reward of five other baseline attacks: critic attack is based on (Pattanaik et al. 2018); random attack adds uniform random noise to state observations; MAD (maximal action difference) attack (Zhang et al. 22200b maximizes the differences in action under perturbed states; RS (robust sarsa) attack is based on training robust action-value functions and is the strongest attack proposed in (Zhang al. 2020b). Additionally, we include the black- box Snooping attack (Inkawhich et al., 2019). For all attacks we consider B(x) as a foe norm ball around s with radius E, set similarly as in (Zhang et al., 2020b). During testing, we run the agents without attacks as well as under attacks for 50 episodes and report the mean and standard deviation of episode rewards. In Table T our "optimal" attack achieves noticeably lower rewards than all the other five attacks. We illustrate few examples of attacks in Figure 3 For RS and "optimal" attacks, we report the best (lowest) attack reward obtained from different hyper-parameters
Evaluation of ATLA In this experiment, we study the effectiveness of proposed ATLA method. Specifically, we use PPO as our policy optimizer. For policy networks, we have two different structures: the original fully connected (MLP) structure, and an LSTM structure which takes historical observations. The LSTMs are trained using backpropagation through time for up to 100 steps. In Table 2 we include the following methods for comparisons:
PPO (vanilla) and PPO (LSTM): PPO with feedforward NN LSTM as the policy network. SA-PPO (Zhang et al. 2020b): the state-of-the-an approach for improving the robustness of DRI. in continuous control environments, using a smooth policy regularization on feedforward NNs
solved by convex relaxations.
Adversarial training using critic attack (Pattanaik et al. 2018): : a previous work using critic based attack to generate adversarial observations in training time, and train a feedforward NN based agent with this relatively weak adversary.
ATLA-PPO (MLP) and ATLA-PPO (LSTM): Our proposed method trained with a feedforward
NN (MLP) or LSTM the policy network. The agent and adversary are trained using PPO with independent value and policy networks. For simplicity, we set Nr = N, = 1 in all settings.
ATLA-PPO (LSTM) +SA reg: Based on ATLA-PPO (LSTM), but with an extra adversarial smoothness constraint similar to those in SA-PPO. We use a 2-step stochastic gradient Langevin dynamics (SGLD) to solve the minimax loss, as convex relaxations of LSTMs are expensive.
For each agent, we report its "natural reward" (episode reward without attacks) and best attack reward in Table 2 To comprehensively evaluate the robustness of agents, the best attack reward is the lowest episode reward achieved by all six types attacks in Table [] including our new "optimal" attack (these attacks include hundreds of independent adversaries for attacking a single agent, see Appendix for more details). For reproducibility, for each setup we train 21 agents, attack all of them and report the one with median robustness. We include detailed hyperparameters in A.5
In Table 2 we can see that vanilla PPO with MLP or LSTM are not robust. For feedforward (MLP) agent policies, critic based adversarial training (Pattanaik et al. 2018) is not very effective under our suite of strong adversaries and is sometimes only slightly better than vanilla PPO. ATLA-PPO (MLP) outperforms SA-PPO on Hopper and Walker2d and is also competitive on HalfCheetah; for high dimensional environments like Ant, the robust function approximator regularization in SA-PPO is more effective. For LSTM agent policies, compared to vanilla PPO (LSTM) agents,
8

---

Published as a conference paper at ICLR 2021
Table Average episode rewards = standard deviation over 50 episodes on ATLA agents and base- lines. We report natural rewards (no attacks) and the best (lowest) attack rewards among six types of adversarial attacks, including simple random noise attack, the critic based attack in (Pattanaik) 2018), MAD and RS attacks in Zhang al. (2020b). Snooping attack proposed in Inkawhich ct al. 2019), J, and the optimal attack proposed this paper. For each environment, we bold most robust agent. Since both RS attack and our "optimal" attack are parameterized attacks, the "best attack" column represents the worst case agent performance under hundreds of adversaries. See
et al.
Appendix A.I for more details.
State
Eae perturb- ation budget
Natural Reward
Best
Dimeasion
Method
Attack
PPO (vanilla)
31671:542
6361
SA. PPO Zhany et et. 2020b
.3005
1076 791
Pattunak et 2018
2755:1582
XILX PPO (MLP)
2559 1. 958
976 40
Hopper
11
0,075
PPO (LSIM)
639.3
784
ATLA PPO (LSTM)
3487.1 452
1224 191
(LSTM) Reg
600
802
PPO (vanilla)
4472 635
1086=516
SA-PPO Zhang et 20206
4487
2908-+ 1136
Partanank et
2018
4058= 1410
733+ 1012
Walker2d
17
0.05
APLA--POFMMERR
3138
2213+
PPO (LSTM)
2783= 1121
1259 937
ATLA-PPO (LMM
3920-4 129
3219 1132
ATLA-PPO (LSTM) +SA Reg
3842
3239= 894
PPO (vanilla)
5687 1 758
-872 1 436
SA PPO Zhany al. 20206
4292.1 384
2511 1117
Patianan et at 2018
34692 1139
-672: 100
111
0.15
AILA PPUTMMP)
48941 123
33.1327
PPO (LSTM)
5696 1 165
-513 = 104
ATLA-PPO (LSTM)
56121 130
716.1. 256
ATLA-PPO (LSTM) +SA Reg
53591153
3765 101
PRO (vanila))
7117=
-600=
SA.PPO Zhang et al. 2020b
3632- 20
3028 4-23
Partanaik
5241# 1162
447+ 192
HalfCheetah
17
0.15
APL-A-PP (MTE)
5417+
2170+ 2097
PPO (LSTM)
5009= 98
-886+
ATI (1 STM)
5766 + 109
2485+ 1488
ATLA-PPO (LSTM) +SA Reg
6157+ 852
4806 603
ATLA-PPO (LSTM) can significantly improve agent robust- ness; a LSTM agent trained without a robust training pro- cedure like ATLA cannot improve robustness. We find that
Roward Under "Optimal Attack
1400
LSTM agents tend to be more robust than their MLP coun-
1000
terparts, validating our findings Section 3.2 ATLA-PPO (LSTM) is better than SA-PPO on Hopper and Walker2d. In all settings, especially for high dimensional environments like Ant, ATLA approach that also includes State-Ad Adversaaaaaa regularization (ATLA-PPO +SA Reg) outperforms all other baselines, this combination improves both the intrinsic bustness of policy and the robustness of function approximator.
800
600
400
PA ATLA PPOOLSTMI Rog
200
0
0.636.1
D.3
SA-PPO Regulartaation Constant x
Figure The performance under the strongest attack for SA-PPO Hopper with different regulariza- tion K. Even increase regular- ization, it cannot outperform our ATLA agents.
robust function approximator can be insufficient some environments, SA-PPO method has its limitations using an increasingly larger regularization parameter x (which controls how robust the function approximator needs to be), we still cannot reach the same performance as our ATLA agent (Figure 4p. Additionally, when large regularization is used, agent performance becomes much worse. In Figure under the largest = 1.0, the natural reward (1436 + 96) is much lower than other agents reported in Table 2
For
CONCLUSION
In this paper, we first propose the optimal adversarial attack on state observations of RL agents, which significantly stronger than many existing adversarial attacks. We then show the alternat- ing training with learned adversaries (ATLA) framework to train agent together with a learned optimal adversary to effectively improve agent robustness under attacks. We also show that a his- tory dependent policy parameterized by LSTM be helpful for robustness. Our approach is orthogonal to existing regularization based techniques, and can be combined with atate-adversarial regularization achieve state-of-the-art robustness under strong adversarial attacks.

---

Published as a conference paper at ICLR 2021
REFERENCES
Karl Astrom. Optimal control of markov processes with incomplete state information. Journal of mathematical analysis and applications, 10(1):174-205, 1965.
Kamyar Azizzadenesheli, Manish Kumar Bera, and Animashree Anandkumar. Trust region policy optimization for pomdps. arXiv preprint arXhv:1810.07900 2018.
Haoyu Bai, David Hsu, and Wee Sun Lee. Integrated perception and planning in the continuous space: A pomdp approach. The International Journal of Robotics Research, ((9))1288-1300 2014.
Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining in Pattern Recogni- fion, pp. 262-275. Springer, 2017a.
Vahid Behzadan and Arslan Munir. Whatever does not kill deep reinforcement learning, makes it stronger. arXiv preprint arXiv: 17Xiv:1712.09344, 2017b.
Rodney A Brooks. Artificial life and real robots. In Proceedings of the First European Conference on artificial life, pp. 3-10, 1992.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudoiph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on PPO and TRPO. arXiv preprint arXiv:2005. 12729, 2020.
Marc Fischer, Matthew Mirman, and Martin Vechev. Online robustness training for deep reinforce- ment learning. arXiv preprint
2019.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint 1802.08477, 2018.
Adam Gleave, Michael Dennis, Neel Kant, Cody Wild, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. arXiv preprint
2019.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Ue- sato, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv: 2018.
Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard L Lewis, and Xiaoshi Wang. Deep learning for real-time atari game play using offline monte-carlo tree search planning. In Advances in neural information processing systems, pp. 3338-3346. 2014.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv: 150..00677..7. 2015.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv: 2017.
Yunhan Huang and Quanyan Zhu. Deceptive reinforcement learning under adversarial manipula- tions on cost signals. In International Conference on Decision and Game Theory for Security, pp- 217-237. Springer, 2019.
Matthew Inkawhich, Yiran Chen, and Hai Li. Snooping attacks on deep reinforcement learning. arXiv preprint arXiv: 1832, 2019.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
Nick Jakobi, Phil Husbands, and Inman Harvey. Noise and the reality gap: The use of simulation in evolutionary roboties. In European Conference on Artificial Life, pp. 704-720. Springer, 1995.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint 170.001705.06455. 2017.
10

---

Published as a conference paper at ICLR 2021
Hanna Kurniawati, David Hsu, and Wee Sun Lee. Sarsop: Efficient point-based pomdp planning by approximating optimally reachable belief spaces. In Robotics: Science and systems, volume 2008. Zurich, Switzerland., 2008. 2008.
Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4213-4220, 2019.
Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
Yen-Chen Lin, Zhang- Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Yu Liu, and Min Sun. Tac- tics of adversarial attack on deep reinforcement learning agents. arXiv preprint 170.:::7703067.. 2017.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine Learning Proceedings 1994, pp. 157-163. Elsevier, 1994.
Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning in batch reinforcement
learning and control. In Advances in Neural Information Processing Systems, pp. 14570-14580, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physicall-plaasible perturbations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939. IEEE, 2017.
Daniel Mankowitz, Timothy A Mann, Pierre-Luc Bacon, Doina Precup, and Shie Mannor. Learn- ing robust options. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Daniel Mankowitz, Nir Levine, Rae Jeong. Abbas Abdolmaleki, Jost Tobias Springenberg, Tim- othy Mann, Todd Hester, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv:1966.07516, 2019.
Matthew Mirman, Marc Fischer, and Martin Vechev. Distilled agent DQN for provable adversarial robustness, 2018a. URL
Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for prov- ably robust neural networks. In International Conference on Machine Learning, pp. 3575-3583, 2018b.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. arXiv preprint arXiv:1507.00677 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-553 2015.
Fabio Muratore, Michael Gienger, and Jan Peters. Assessing transferability from simulation real- ity for reinforcement learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2019.
Arnab Nilim and Laurent El Ghaoui. Robustness in Markov decision problems with uncertain tran- sition matrices. In Advances in Neural Information Processing Systems, pp. 839-846, 2004.
Takayuki Osogami. Robust partially observable Markov decision process. International Confer- on Machine Learning, pp. 106-115, 2015.

---

Published as a conference paper at ICLR 2021
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2042. International Foun- dation for Autonomous Agents and Multiagent Systems, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 2817-2826. JMLR. org, 2017.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Xinghua Qu, Yew-Soon Ong, Abhishek Gupta, and Zhu Sun. Defending adversarial attacks without adversarial attacks in deep reinforcement learning. arXiv preprint arXiv:2008.06/99 2020.
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning. arXiv preprint arXiv:2003. /00,:29990..999,, 2020.
Mohammad Rasouli and Soroush Saghafian. Robust partially observable markov decision processes. No. RWP18-027, 2018
Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep reinforcement learning framework for autonomous driving. Electronic Imaging, 2017(19):70-76, 2017.
Hadi Salman, Greg Yang, Huan Zhang. Cho-Jui Hsich, and Pengchuan Zhang. A convex relaxation barrier tight robustness verification of neural networks. In Advances in Neural Information Processing Systems 32, pp. 9832-9842. Curran Associates, Inc., 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Joannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Zachary Sunberg and Mykel Kochenderfer. Online algorithms for pomdps with continuous state, action, and observation spaces. arXiv preprint 1xXiv::709.066
2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2013.
Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, Soumik Sarkar, et al. Robustifying reinforce- ment learning agents via action space adversarial training. In 2020 American Control Conference (ACC), pp. 3959-3964. IEEF, 2020.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. arXiv preprint arXiv: /90.00940,., 2019.
Voyage. Introducing voyage deepdrive unlocking the potential of deep reinforcement learning. 2019.
Daan Wierstra, Alexander Foerster, Jan Peters, and Juergen Schmidhuber. Solving deep memory pomdps with recurrent policy gradients. In International Conference Artificial Neural Net- works, pp. 697-706. Springer, 2007.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 5283-5292, 2018.
Chaowei Xiao, Xinlei Pan, Warren He, Jian Peng, Mingjie Sun, Jinfeng Yi, Bo Li, and Dawn Song. Characterizing attacks on deep reinforcement learning. arXiv preprint arXiv: 907.0947700 2019.
Kaidi Xu, Zhouxing Shi, Huan Zhang, Minlie Huang, Kai- Wei Chang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis general computational graphs. arXiv preprint 2rxi::2002. /2920, 2020.
Haoqi Zhang and David C Parkes. Value-based policy teaching with active indirect elicitation. In AAAI, volume 8, pp. 208-214, 2008.
12

---

Published as a conference paper at ICLR 2021
Haoqi Zhang, David C Parkes, and Yiling Chen. Policy teaching through reward function learning. In Proceedings of the 10th ACM conference on Electronic commerce, pp. 295-304, 2009.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv: 100p:099000555 2019.
Huan Zhang, Tsui-Wa Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. In NIPS, 2018.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Duane Boning, and Cho-Jui Hsich. Towards stable and efficient training of verifiably robust neural networks. ICLR, 2020a.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on observations. In Advances in Neural Information Processing Systems, 2020b.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 FULL RESULTS OF ALL ENVIRONMENTS UNDER DIFFERENT TYPES OF ATTACKS
In T we only include the best attack rewards (lowest rewards over all attacks). In Table we list the rewards under each specific attack. Note that, Robust Sarsa (RS) attack and our "optimal" policy attack both have hyperparameters. For RS attack we use the same set of 30 different settings of hyperparameters as in (Zhang et al. 2020b) to train robust value function to attack the network. The reported RS attack result for each agent is the strongest one over the 30 trained value functions. For Snooping based attack, we use the "imitator" attack proxy it was the strongest reported in (Inkawhich et al. 2019), and we attack every step of the agent. The imitator a MLP or LSTM network according to agent policy network. We use the same loss KI. divergence function as in the MAD attacks for this Snooping attack. We first collect state-action pairs for 100 episodes to train the "imitators", whose network structures are the same the corresponding agents. In the test time, we first run MAD attack on it the "imitator" and then input the generated perturbed observation to the agent in a transfer attack fashion. For our "opptmall" policy attack, the hyperparameters are PPO training parameters for the adversary (including the learning rate of the adversary policy network, learning rate of the adversary value network, the entropy regularization parameter and the ratio clip e for PPO). We use a grid search of these hyperparameters to train an adversary that is as strong as possible, resulting in 100 200 adversaries produced for each agent. The reported optimal attack rewards is the lowest reward among all trained adversaries. Under this comprehensive adversarial evaluation, each agent tested using hundreds of adversaries and the strongest adversary determines the true robustness of an agent.
Table 3: Average episode rewards + standard deviation over 50 episodes on five baselines and SA- PPO (Zhang et al. (0220b). We report natural episode rewards (no attacks) and episode rewards under six adversarial attacks, including simple random noise attack, the critic based attack in (Pattanaik) al. 2018), MAD and RS attacks in Zhang et al. (2020b), Snooping attack proposed in Inkawhich et al. 2019), and the optimal attack proposed in this paper. In each row we bold the best (lowest) attack reward over all five attacks. The row for the most robust method highlighted.
Method
Roxani
Rueskee
Rewart
"Opinall"
Attack
2.35
210
1100 035
110
635
SA.EOO Zhane Lanae Jrose
3705 2
37891: 15
2710:: 901
2652: 815
1130 2:42
107K2 791
1075
Paltehite o p01n
2755
2681 555
2283:: 502
13052:357
1340::436
12191 174
291:2 7
201
Hopper
XILX PRLA-PreMin Mar
2589 918
3407- 516
2153- 483
16794-576
1790--667
23394 570
976-
577
(LSTM:
452
9130=1992
1224:
1224
ATLAUPOO (LSTMI TMU Rey
3291 600
2003
824
2163 1165:2550 575
25146 725
2857 724
224488 615
1772: 802
1772
PCU KCUreatilan
4412 635
5424 2: 1255
3037 = 1.00
3359 1241
1330
3: 104
165622566
1085:
SA.PP3 Zhaus ol Ro0o6
44872 61
64752:30 30
48671220
3608-2 1789
5915-21661
3908-1 138
2508 1136
2005
Phitehric 2015
1410
1410
35451
1954
2568.
1915 1150
753.
73%
Wilkerdd
0.08
PS0 (LSTM)
1121
1042
2471
228661 1156
12594 937
15234 509
1259
ATLA-PPO (LSTM)
3920 120
39152
274
3770 1
3063-4
3716-190 196
3319 1832
3463 1016
3219
ATL ATL-AGOOO SESPPOIITTTTTTTT -SA Ray
1502
475
3854 132
3027 356
1136 402
1222-100
2230
2914
25551
707
3230
1911 10.1
32015
1151 315
201 2221
1873
430
-372
$A,PRO Phass st Hoooe
4292 36
4805
4996
2452
2:522
3412 +1755
1117
2513
Aai
0.15
125
4423 104
1591
365
527
51
(LSMM
5095 = 165
5515
134
5475
691
3600 363
1000 2 382
.513 194
-513
ATLA (LSTM)
5612 130
5196
2 134
5300
2 704
3043
21.7
44552577
1006
2 525
716: 256
716
ATT
155
3709 185
3109
$3.01
5155 1911
S13N1 129
37851
1065
PoUTo milla
3961=119
5480 1375
100
450
SA.EEO
Pullebit of p016
S2111 1162
5440 676
2013: 1694
1723. 1348
5465-1726
1502 1157
447 192
447
Haltheechh
0.15
54176 09
51364-34 34
5758 4 36
4023 7144
4167
23705: 3007
3705- 50
2170
PPO (LSMM
50091-00 08
4394 112
5391- 196
47641 106
4088--748
7645 2006
-866
-436
ATLA-PO ILSTM)
4507
4906:112
13382
2485:
A.2 AGENT PERFORMANCE DURING TRAINING
In Table we only report the agent performance at the end of training. In this subsection, we evaluate our agent performance during 20%, 40% 60% and 80% of total training epochs using Robust Sarsa (RS) attacks. The results are presented in Table The overall trend is that agents are getting stronger over time ("RS attack reward" is increasing), achieving better robustness in later checkpoints.
A.3 NETWORK STRUCTURE
For fully connected networks, we use the same network as in (Zhang et al. 2020b), which contains 2 hidden layers with 64 hidden neurons each layer, for both policy and value networks, for both the
14

---

Published as a conference paper at ICLR 2021
Table 4: Natural and RS attack rewards of ATLA-PPO (LSTM)+ SA Reg checkpoints during train- ing. We report Average rewards standard deviation over 50 episodes.
Environment
Reward
20%
40%
60%
80%
100%
Natural Reward
3440:111
1161 +:85
3013=584
15691111
=20
Hopper
RS Attack Reward
716++82
631-551
001
3181+663
2244+-618
Natural Reward
989+2554
3506- 174
2203 988
3803+722
3842--475
Walker2d
RS Attack Reward
882:1269
1744=347
73911531
2550:10200
3239-1944
Natural Reward
2634++122
4532-106
5007+143
5127+542
5393++139
Ant
RS Attack Reward
2161171
1903-13
3040=244
3040:1241
41361114
Natural Reward
140
5367=138
5955=177
59561181
63001261
HalfCheetah
RS Attack Reward
3986+:564
3986=564
4911=922
4571
483
single layer LSTM with 64 hidden newrons, along with an input embedding layer projecting state dimension to 64 and an output layer projecting 64 to output dimension. For LSTT agents, when conducting the "optimal" attack, also LSTM network for the adversary to ensure the adversary is powerful enough.
agent and the adversary. For LSTM agents,
A.4 HYPERPARAMETER FOR THE LEARNING-BASHR "OPTIMAL" ATTACK
Our "optimal" attacks uses policy gradient methods to learn the optimal adversary during agent testing, and each learning process involves the selection of hyperparameters. Specifically, the hy- perparameters include the learning rates of the adversary's 's policy and value networks, the entropy coefficient, and the annealing of the learning rate. To reduce search space, for ATLA agents, the learning rates of the testing phase adversary's policy and value networks 0.3X to 3X of the learning rates of adversary's 's policy and value networks used in training. For other agents trained without an adversary, the learning rates of the testing phase adversary's policy and value networks are chosen ranging from 0.3X to 3X of the learning rates of the agent's pol- icy and value networks. We tested both linearly annealed learning rate and non-annealing learning rate. The adversary's 's entropy coefficient is chosen form 0 and 0.003, The final results reported in all tables are the best (lowest) reward achieved by the "optimal" attacks among all hyperparameter configurations. Typically this includes around 100 to 200 different adversaries trained with different hyperparameters. This guarantees the strength of this attack and allows comprehensive evaluation of the robustness of all agents.
chosen ranging from
A.S HYPERPARAMETERS FOR ATLA PERFORMANCE EVALUATION
Hyperparametersstee for PPO (vanilla) For the Walker2d and Hopper environment, set of hyperparameters as in (Zhang et al. 2020b): the hyperparameters were originally from (En- gstrom et al.) 2020) and found using a grid search experiment. We found that this set of hyperparam- eters work well. For HalfCheetah and Ant environment, we use grid search of hyperparameters, including the learning rate of the policy network, learning rate of the value network and the entropy bonus coefficient. For Hopper, Walker2d and HalfCheetah environments, we train for 2 million steps (2 million environment interactions). For Ant, we train for 10 million steps. Training for longer may slightly improve agent performance under no attacks, but has no impact for performance under strong adversarial attacks.
the same
Hyperparameters for PPO (LSTM) For PPO (LSTM), we conduct a smaller scale hyperparam- eter search. We search hyperparameter values that are close to the optimal ones found for the PPO vanilla agent. We train these LSTM agents for the same steps as those vanilla PPO.
Hyperparameters for SA-PPO except SA-PPO's extra for the strength of SA-PPO regularization. For k, we choose from 1 x 10-6 to 1. We train agents with each K 21 times and choose the k value whose median agent has the highest worst-case reward under all attacks.
We use the same value for hyperparameters as in vanilla PPO
Hyperparameters for ATLA-PPO For ATLA-PPO, have hyperparameters for both agent and adversary. We keep all agent hyperparameters the same as those in vanilla MLP/LSTM agents, except for the entropy bonus coefficient. We find that sometimes need larger entropy bonus
15

---

Published as a conference paper at ICLR 2021
Table 5: Hyperparameters for all environments and settings. For vanilla environments, use the hyperparameters from Zhang et al. (2020b) and Engstrom et al. (020) if they are available for that environment (Hopper and Walker2d), Other environments' hyperparameter for the vanilla PPO model is found by grid search. For SA-PPO and ATLA-PPO (MLP), the same set of hyperpa- rameters as in the vanilla models are used, except that for SA-PPO we tune the parameter and for ATLA-PPO (MLP) we tune the entropy bonus coefficients as well as learning rates for the adversary. For LSTM models, we first tune the vanilla LSTM PPO models and find the best learning rates, keep using them in all LSTM based models.
policy
val
entropy coel..
policy
val abV. entropy coell.
PPOIvanila)
36-1
2.5c-1
SA-PPO
3e-4
2.5c-4
0.03
Hopper
ATLA-PPO (MLP)
3e-4
2.5e-4
0.01
0.001
0.0001
0.001
PPO (LSTM)
le-3
30-
0.0
ATLA-PPO (LSTM)
1e-3
0.01
ATLA-PPO (LSTM)+ SA Rog
lo-3
3-4
0.01
0.3
0.003
0.01
0.003
PPO(vanilla)
4p-4
3044
SA-PPO
4e4
3e.4
Walker2d
ATLA-PPO (MLP)
4--
3o-4
0,0003
0.0001
0,0001
0.002
ATLA-PPO (LSTM)
le-3
3e-2
0.001
0,0003
0.03
ATL A-PO (LSTM)+ SA Reg
le-3
3e-2
0.001
0.3
0.003
0.03
0.001
PPO(vamilla)
Se-5
1e.5
SA-PPO
S6-5
le:5
3e-3
ATLA-PPO
Ant
PPO (LSTM)
3-44
3e-4
ATLA--PO (1 (TSTM)
3e44
3o.4
0.0003
0.0003
0.0001
0.0003
ATLA- (LSTM)+ SA Reg
3--4
3e.4
0.003
0.1
0,0003
3e
3e-05
PPOtvanilaa
36-1
16-4
SA-PPO
3-44
1e-4
0.1
HaleCheetah
ATLA-PPO (MLP)
3e-4
le-4
0,0003
0.001
0.0003
0.003
1o-3
ATLA-PPO (LSTM)
le:3
3e-4
0,0003
0.903
0.001
ATLA-PPO (LSTM+ SA Reg
16-3
30-4
0.03
0.003
0.003
0.0003
efficient in ATLA to allow sufficient exploration of the agent, as learning with an adversary harder than learning in tttack-free environments. For the adversary, we run a small-scale hyperparameter search on the learning rate of adversary policy and value networks, and the entropy bonus coefficient for the adversary. To reduce the number of hyperparameters for searching, we use values close to those the agent. We set N, = N₂ 1 in all experiments and not tune this hyperparameter. For ATLA, train 5 million steps for Hopper, Walker and HalfCheetah and 10 million steps for Ant. We find that similar to the observations (Madry et al. 2018). training with an adversary typically requires more steps converge, however all environments the training does reliably converge.
Agent selection For each setup, we repeat the experiments using the same set of hyperparameters for 21 times due to the high performance variance in RL. We then attack all the agents using random, critic, MAD and RS attacks. We use the lowest reward among all attacks as a metric to rank those agents. Then, we select the agent with median robustness our final agent. This final agemt is then attacked using the "optimal" attack to further reduce its reward. The numbers we report in Table 2 are not from the best runs, but the runs with median robustness. This is done to improve reproducibility as RL training process can have high variance.
16


Reviews:
[{"review_id": "zjW8gy6u_Rj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is very well written and the considered problem of training an adversary along with the agent is very interesting. Within the proposed concept, the parameterized adversary can be trained by viewing the agent as a part of the environment, so it avoids to access the parameters of the agent policy. From the perspective of the agent, with an unknown adversary, the MDP becomes a POMDP with uncertainty hidden in the adversary, and hence the fact of using LSTM policy is much better for the agent is reasonable. The entire problem is wisely formulated.  The experimental settings are well designed and results support the positiveness of the proposed framework.\n\nI only have one comment that the SA-MDP can also be understood from another perspective. That is, SA-MDP is actually an asymmetric competitive multi-agent problem, and the alternative training of agent and adversary can be viewed as an instance of self-play. Also, the optimality of SA-MDP for either the agent or the adversary can be explained through multi-agent RL or game theory. It would be interesting if the authors could take a look into such a direction.", "extra_scores": {}, "extra_text": {}}, {"review_id": "vEFm4Yx8QQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes to improve the robustness of a reinforcement learning agent by alternatively training an agent and an adversary who perturbs the state observations. The learning of an “optimal” adversary for a fixed policy is based on the theory of SA-MDP in prior work. The learning of an optimal policy under a fixed adversary is done by solving a POMDP problem. Experimental results show that the proposed alternating training with learned adversaries (ATLA) framework can improve the performance and robustness of PPO.\n\nStrengths:\n1. The paper is well-organized and easy to follow.\n2. The authors distinguish between the vulnerability of function approximations and the intrinsic weakness of policy, which is interesting and can be useful for the community to investigate the vulnerability of deep RL.\n3. The experiment results show that both the learned adversary and the trained agent perform well, in terms of attacking and learning respectively. In addition, the proposed LSTM-based policy is shown to be more robust than regular feedforward NNs.\n\nWeakness:\n1. The novelty of this paper is a little limited. (1) The idea of alternative training the agent and the adversary is similar to RARL[1], and Algorithm 2 (ATLA) is similar to Algorithm 1 in [1]. Although RARL focuses on the case where adversary directly changes the environment and ATLA focuses on the observation perturbation attacks, the whole ideas are still similar. (2) The main method of learning an adversary is based on the theoretical work of SA-MDP[2]. \n2. The authors claim that the proposed adversary is strong since it follows the theoretical framework of SA-MDP. However, in Lemma 1, the adversary reward in SA-MDP, \\hat{R}, is defined as the weighted average of -R, while in Algorithm 1, the adversary reward is given by -R itself. It is not clear why such a relaxation still follows the theoretical framework. More details illustrating the approximation and some analysis about the optimality will be appreciated.\n3. In the experiment section, the authors only compare the proposed algorithm with [2] in terms of “optimal” attack and robust training. However, there are a lot of works that attack the observations of a fixed policy [3,4,5]. And more importantly, [6] also proposes to train a robust agent under adversarial attacks. It will be more convincing if the authors empirically or theoretically compare with some of these potential baselines.\n4.The computational complexity / sample complexity of the proposed ATLA might be problematic, as for each iteration of learning, the adversary needs to solve a new MDP, which makes the proposed robust training less practical to use. \n\nMinor comments:\n- This paper sometimes uses the phrase \"training time attack\" to refer adversarial attacks, which is misleading, e.g. the second contribution, the third paragraph of related work. Training-time attack usually refers to poisoning attack, which changes the training dataset and alters the learned policy, different from the scenario in this paper where an adversary wants to fool a fixed policy.\n\n\nRefs:\n[1] Pinto, Lerrel, et al. \"Robust adversarial reinforcement learning.\" arXiv preprint arXiv:1703.02702 (2017).\n[2] Zhang, Huan, et al. \"Robust Deep Reinforcement Learning against Adversarial Perturbations on Observations.\" arXiv preprint arXiv:2003.08938 (2020).\n[3] Huang, Sandy, et al. \"Adversarial attacks on neural network policies.\" arXiv preprint arXiv:1702.02284 (2017).\n[4] Lin, Yen-Chen, et al. \"Tactics of adversarial attack on deep reinforcement learning agents.\" arXiv preprint arXiv:1703.06748 (2017).\n[5] Inkawhich, Matthew, Yiran Chen, and Hai Li. \"Snooping Attacks on Deep Reinforcement Learning.\" arXiv preprint arXiv:1905.11832 (2019).\n[6] Pattanaik, Anay, et al. \"Robust deep reinforcement learning with adversarial attacks.\" arXiv preprint arXiv:1712.03632 (2017).", "extra_scores": {}, "extra_text": {}}, {"review_id": "oaVa5eMhyic", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:  The paper studies adversarial attacks in RL, focusing both on the design of optimal attack strategies on RL agents, as well as robust training RL procedures for mitigating attacks. Building on the results of (Zhang et al., 2020), the paper proposes a new learning framework (ATLA), that simultaneously trains a (strong) adversary and a (robust) deep RL agent. The paper showcases the importance of the new framework through extensive experimental evaluation. \n\nReasons for score: Overall, I find the paper to be an interesting read and its contribution relevant to the line of work on adversarial attacks in RL. The contributions of the paper seem non-trivial, and include a framework for designing optimal attacks and training procedures that can optimize for robustness. As shown by the experiments, the proposed solution leads to significant increase in performance compared to state of the art baselines. These results complement those of (Zhang et al., 2020). Nonetheless, the presentation of the paper could be improved in terms of clarity. Some parts of the paper could be reorganized and explained in more detail. Suggestion for improvements and questions are outlined below. \n\nClarity: The paper is overall enjoyable to read, but some parts are not clearly/precisely written. There are quite a few typos, some of which might be important for understanding the content. I did not follow equation (2), which seems to contain typos. Could you explain in more detail the loss function defined with this equation? Furthermore, notation in the paragraph before section 3.1 is partly confusing, in particular, $\\nu$ seems to be a deterministic function, but then for observation $\\hat s$ we have $\\hat s \\sim \\nu(s)$ indicating that $\\nu(s)$ might be a distribution from which we sample $\\hat s \\sim \\nu(s)$. I would also suggest reorganizing the content related to Figure 1, which is introduced in section 1, but only explained in detail in section 3.1. \n\nRelated Work: The related work is generally covered well, but it could be expanded by providing some connection to the line of work that studies policy teaching and policy poisoning attacks in RL. Could you explain how the setting of this paper compares to those studied in this line of work (e.g., Parkes et al. 'Policy Teaching Through Reward Function Learning', Ma et al. 'Policy Poisoning in Batch Reinforcement Learning and Control', Rakhsha et al., 'Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning', etc.)?\n\nExperiments: I'm wondering to what extent are the results for different methods in Table 2 comparable. Namely, the methods seem to be based on different architectures, so it is not immediately clear what conclusions should be drawn from these results. Surprisingly, the discussion on page 8 does not seem to compare the results of ATLA-PPO (MLP) and SA-PPO. Could you elaborate more on these results and make relative comparison? Furthermore, in the following sentence: 'For reproducibility, for each setup we train at least 5 (up to 21) agents, attack all of them and report the one with median robustness.' - why is there discrepancy between different setups in the number of trained agents? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "f7IPgfJ8BQ8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors presents how to learn optimal adversary following the state-adversarial Markov decision process (SA-MDP), and also proposes alternating training with learned attacks (ATLA) framework that trains the optimal adversary online together with the agent to improve the robustness of the DRL agent. Experiment result shows that ATLA outperforms the explicit regularization based methods. \n\nOverall I think the paper is well-written and clearly illustrated the methodology. The experimental results are mostly comprehensive. The contribution is clear. I still have a few questions and concerns below:\n\n1) It is a natural and interesting idea to use alternating training to optimize both the adversary and the agent online. In terms of finding the (near) optimal adversary following the theory of SA-MDP, the authors argue that because of using \"function approximator to learn the agent so it’s no longer optimal\". However, it seems unclear how much the approximation affects the optimality. For tabular case instead of DRL, is it possible to really find the optimal adversary, and if so, how? How far away the learned strong adversary from the real optimal one? Indeed from experiments the learned adversary can better attack than other baselines, but it would still be important to understand the advantage and room to improve in a principled way. For finding the policy, it is understandable that because of the difficulty of solving POMDP, this paper does not solve the optimal policy. \n\n2) ATLA-PPO + SA Reg further improves performance over ATLA-PPO in experiments. This seems to suggest that the advantage of ATLA-PPO and SA Reg are complementary. Does the regularization on the function approximator provides additional robustness or it covers the error of function approximator using LSTM? Is it possible to understand this in experiments?\n\n3) Are the adversary and the agent both getting stronger over time? The paper only showed final results and did not show the running time result. Hypothetically because of the alternating training, the adversary and the agent should both be improved and it would be interesting to verify this in experiments. \n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a new technique to train an adversarial MDP to perturb the observations provided by the environment.  This adversarial MDP is then used to train an RL agent to be more robust.  Since the adversarial agent essentially defines an observation distribution for the environment, the RL agent needs to optimize a POMDP.  This is nice work that was unanimously praised by the reviewers.  It produces stronger adversaries and more robust RL agents than previous work.  This represents an important contribution to the state of the art of robust RL.  
Decision: Accept (Poster)