Robust Reinforcement Learning using Adversarial Populations
Authors: Eugene Vinitsky
Yuqing du
Kanaad V Parvate
Kathy Jang
Pieter Abbeel
Alexandre Bayen
Venue:  (2021)
Keywords: Robust Control
Reinforcement Learning
Multiagent Systems

Abstract: Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness, failing catastrophically when the underlying system dynamics are perturbed. The Robust RL formulation tackles this by adding worst-case adversarial noise to the dynamics and constructing the noise distribution as the solution to a zero-sum minimax game. However, existing work on learning solutions to the Robust RL formulation has primarily focused on training a single RL agent against a single adversary. In this work, we demonstrate that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary; the resulting policy is highly exploitable by new adversaries. We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training. We empirically validate across a variety of benchmarks that the use of an adversarial population results in a less exploitable, more robust policy.  Finally, we demonstrate that this approach provides comparable robustness and generalization as domain randomization on these benchmarks while avoiding a ubiquitous domain randomization failure mode.

Under review as a conference paper at ICLR 2021
ROBUST REINFORCEMENT LEARNING USING ADVERSARIAL POPULATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement Learning (RL) is effective tool for controller design but struggle with issues of robustness, failing catastrophically when the underlying system dynamics are perturbed. The Robust RL formulation tackles this by adding worst-case adversarial noise to the dynamics and constructing the noise distribution as the solution zero-sum minimax game. However, existing work on learning solutions to the Robust RL formulation has primarily focused on training single RI. agent against single adversary. In this work, we demonstrate that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary; the resulting policy highly exploitable by new adversaries. We propose population-based augmentation to the Robust RL formulation in which we randomly initialize population of adversaries and sample from the population uniformly during training. We empirically validate across robotics benchmarks that the use of an adversarial population results in a less exploitable, more robust policy. Finally, demonstrate that this approach provides comparable robustness and generalization as domain randomization on these benchmarks while avoiding a ubiquitous domain randomization failure mode.
INTRODUCTION
Developing controllers that work effectively across wide range of potential deployment environments one of the core challenges in engineering. The complexity of the physical world means that the models used to design controllers are often inaccurate. Optimization based control design approaches, such as reinforcement learning (RL), have no notion model inaccuracy and can lead to controllers that fail catastrophically under mismatch. In this work, we aim to demonstrate effective method for training reinforcement learning policies that are robust to model inaccuracy by designing controllers that are effective in the presence of worst-case adversarial noise in the dynamies.
An easily automated approach to inducing robustness is to formulate the problem as a zero-sum game and learn an adversary that perturbs the transition dynamics (Tsssler et al. 2019 Kamalaruban eet al. 2020 Pinto et al., 2017). If a global Nash equilibrium of this problem is found, then that equilibrium provides lower bound on the performance of the policy under some bounded set of perturbations. Besides the benefit of removing user design once the perturbation mechanism is specified, this approach maximally conservative, which useful for safety critical applications.
However, the literature on learning an adversary predominantly uses a single, stochastic adversary. This raises a puzzling question: the zero-sum game does not necessarily have any pure Nash equilibria (see Appendix C in Tessler et al. (2019) but the existing robust RL literature mostly appears to attempt solve for pure Nash equilibria. That is, the most general form of the minimax problem searches over distributions of adversary and agent policies, however, this problem is approximated in the literature by a search for a single agent-adversary pair. We contend that this reduction to single adversary approach can sometimes fail to result in improved robustness under standard parametrizations of the adversary policy.
The following example provides some intuition for why using single adversary can decrease robustness. Consider robot trying to learn to walk east-wards while an adversary outputs a force representing wind coming from the north or the south. For fixed, deterministic adversary the agent knows that the wind will come from either south or north and can simply apply a counteracting force at each state. Once the adversary is removed, the robot will still apply the compensatory forces and

---

Under review as a conference paper at ICLR 2021
possibly become unstable. Stochastic Gaussian policies (ubiquitous in continuous control) offer little improvement: they cannot represent multi-moda perturbations. Under these standard policy parametrizations, we cannot use an adversary endow the agent with prior that strong wind could persistently blow either north or south. This leaves the agent exploitable to this class of perturbations.
The use of single adversary in the robustness literature is in contrast to the multi-player game literature. In multi-player games, large sets of adversaries are used to ensure that an agent cannot easily be exploited (Vinyals al. 2019; Czarnecki et al. 2020 Brown & Sandholm, 2019). Drawing inspiration from this literature, we introduce RAP (Robustness via Adversary Populations); randomly initialized population of adversaries that we sample from at each rollout and train alongside the agent. Returning to our example of a robot perturbed by wind, if the robot learns to cancel the north wind effectively, then that opens niche for an adversary to exploit by applying forces in another direction. With population, we can endow the robot with the prior that a strong wind could come from either direction and that must walk carefully to avoid being toppled over.
Our contributions are as follows:
Using a set of continuous robotics control tasks, we provide evidence that single adversary does not have a consistent positive impact on the robustness of an RL policy while the use of an adversary population provides improved robustness across all considered examples.
We investigate the source of the robustness and show that the single adversary policy is exploitable by new adversaries whereas policies trained with RAP are robust to new adversaries.
We demonstrate that adversary populations provide comparable robustness to domain randomization while avoiding potential failure modes of domain randomization.
RELATED WORK
This work builds upon robust control Zhou & Doyle 1998). a branch of control theory focused on finding optimal controllers under worst-case perturbations of the system dynamics. The Robust Markov Decision Process (R-MDP) formulation extends this worst-case model uncertainty to uncer- tainty sets on the transition dynamics of an MDP and demonstrates that computationally tractable solutions exist for small, tabular MDPs (Nilim El Ghaoui 2005 Lim et al., (013). For larger or continuous MDPs, one successful approach has been to use function approximation to compute approximate solutions to the R-MDP problem (Tamar et al., 2014).
One prominent variant of the R-MDP literature is to interpret the perturbations as an adversary and attempt to learn the distribution of the perturbation under a minimax objective. Two variants of this idea that tie in closely to our work are Robust Adversarial Reinforcement Learning (RARL)(Pint) jtt al. 2017) and Noisy Robust Markov Decision Processes MDRMMDP) (Tessler et al. 2019) which differ in how they parametrize the adversaries: RARL picks out specific robot joints that the adversary acts on while NR-MDP adds the adversary action to the agent action. Both of these works attempt to find an equilibrium of the minimax objective using single adversary; in contrast our work uses large set of adversaries and shows improved robustness relative to a single adversary.
strong alternative to the minimax objective, domain randomization, asks designer to explicitly define a distribution over enviromments that the agent should be robust to. For example, (Peng et al.) 2018 varies simulator parameters train a robot robustly push puck a target location in the real world; (Antonova et al., 2017) adds noise to friction and actions to transfer an object pivoting policy directly from simulation to a Baxter robot. Additionally, domain randomization has been successfully used to build accurate object detectors solely from simulated data (Tobin et al.] 2017) and to zero-shot transfer quadcopter flight policy from simulation (Sadeghi & Levine 2016).
The use of population based training is standard technique in multi-agent settings. Alphastar, the grandmaster-level Starcraft bot, uses a population of "exploiter" agents that fine-tune against the bot to prevent it from developing exploitable strategies (Vinyals et al.) 2019). (Czarnecki et al. 2020) establishes a set of sufficient geometric conditions on games under which the use of multiple adversaries will ensure gradual improvement in the strength of the agent policy. They empirically demonstrate that learning in games can often fail to converge without populations. Finally, Active Domain Randomization (Mehta al. 2019) is very close approach to ours, they use population
2

---

Under review as a conference paper at ICLR 2021
of adversaries to select domain randomization parameters whereas population of adversaries to directly perturb the agent actions. However, they explicitly induce diversity using a repulsive term and use discriminator generate the reward.
BACKGROUND
In this work we use the framework of multi-agent, finite-horizon, discounted, Markov Decision Process (MDP) (Puterman 1990) defined by tuple (Aggont Andversary TT,, Here Aagent is the set of actions for the agent, Andversary is the set of actions for the adversary, S is a set states, Augem x Andversury S + R is
T Augent X Andversary S A(S) is transition function, a reward function and 7 is a discount factor. S is shared between the adversaries as they share state-space with the agent. The goal for given MDP is find policy me parametrized by
that maximizes the expected cumulative discounted reward J E oo 7'r(se, at) [e) The conditional in this expression is short-hand to indicate that the actions in the MDP are sampled via a (St, d2-1)- We denote the agent policy parametrized by weights 0 mo and the policy of adversary as क. Actions sampled from the adversary policy ãø. will be written as a. We use & to denote the parametrization of the system dynamics (e.g. different values of friction, mass, wind, etc.) Se((), az).
and the system dynamics for a given state and action as 814-1
3.1 BASELINES
Here we outline prior work and the approaches that will be compared with RAP. Our baselines consist of single adversary and domain randomization.
3.1.1 SINGLE MINIMAX ADVERSARY
Our adversary formulation uses the Noisy Action Robust MDP (Tessler et al. 2019) in which the adversary adds its actions onto the agent actions. The objective
T
max E
r am)) Ts
min E
rr((si, (oo T
where a is a hyperparameter controlling the adversary strength. This is a game in which the adversary and agent play simultaneously. We an important restriction inherent this adversarial model. Since the adversary is only able attack the agent through the actions, there is restricted class of dynamical systems that can represent; this set of dynamical systems may not necessarily align with the set of dynamical systems that the agent may be tested in. This is restriction caused by the choice of adversarial perturbation and could alleviated by using different adversarial parametrizations c.g. perturbing the transition function directly.
3.1.2 DYNAMICS RANDOMIZATION
Domain randomization is the setting in which the user specifies set of environments which the agent should robust to. This allows the user to directly encode knowledge about the likely deviations between training and testing domains. For example, the user may believe that friction is hard to measure precisely and wants to ensure that their agent is robust to variations in friction; they then specify that the agent will be trained with wide range of possible friction values. We use & to denote some vector that parametrizes the set of training environments (c.g. friction, masses, system dynamics, etc.). We denote the domain which & is drawn from = and P (E) denote
3

---

Under review as a conference paper at ICLR 2021
some probability distribution over &. The domain randomization objective is
max P()) 0
Σ
rr((s,, at)
(2)
8t+1 ae)
~ TT()))
Here the goal is to find an agent that performs well on average across the distribution of training environment. Most commonly, and this work, the parameters & are sampled uniformly over E.
4 RAP: ROBUSTNESS VIA ADVERSARY POPULATIONS
RAP extends the minimax objective with population based approach. Instead a single adversary, at each rollout will sample uniformly from population of adversaries. By using population, the agent is forced to be robust to a wide variety of potential perturbations rather than a single perturbation. If the agent begins to overfit to any one adversary, this opens up a potential niche for another adversary to exploit. For problems with only one failure mode, we expect the adversaries to all come out identical to the minimax adversary, but as the number of failure modes increases the adversaries should begin to diversify to exploit the agent. To induce this diversity, we will rely on randomness in the gradient estimates and randomness in the initializations of the adversary networks rather than any explicit term that induces diversity.
Denoting के as the i-th adversary and UL, n)) as the discrete uniform distribution defined 1 through the objective becomes
Σ (rr(ss ar, act) mo, To.
min de
Σ r ae, aat) (To, #o; I,...,
8111 f(s,, + +aū,)
For single adversary, this is equivalent to the minimax adversary described in Sec. 3.1.1 This is game in which the adversary and agent play simultaneously.
We will optimize this objective by converting the problem into the equivalent zero-sum game, At the start of each rollout, we will sample an adversary index from the uniform distribution and collect a trajectory using the agent and the selected adversary. For notational simplicity, we assume trajectory is of length T and that adversary i will participate in J; total trajectories while, since the agent participates in every rollout, the agent will receive J total trajectories. We denote the j-th collected trajectory for the agent Tj (So, Ap, To, 81) 7 (so, ag. -T0, 81) x X (8M, aMM -TM, 8M).
(SM, TM, and
the associated trajectory for adversary Note that the adversary reward is simply the negative of the agent reward. We will use Proximal Policy Optimization (Schulman al.] 2017) (PPO) to update our policies. We caution that have overloaded notation slightly here and for adversary i, Tj=1;,, refers only to the trajectories which the adversary selected: adversaries will only be updated using trajectories where they were active.
4

---

Under review as a conference paper at ICLR 2021
At the end of training iteration, we update all our policies using gradient descent. The algorithm is summarized below:
Algorithm I: Robustness via Adversary Populations
Initialize 8, Φ1 ... on using Xavier initialization (Glorot & Bengio 2010
while converged do for rolloul j=l...J do
sample adversary ~ n).
run policies Ta, T.. in environment until termination;
collect trajectories Tj. rj
end
update 0, Φ1 on using PPO (Schulman et al. 2017) and trajectories Tj for 0 and rj for each Øi
end
EXPERIMENTS
In this section present experiments continuous control tasks from the OpenAl Gym Suite (Brockman et al. 2016) Todorov et al. 2012). We compare with the existing literature and evaluate the efficacy of a population of learned adversaries across wide range of state and action space sizes. We investigate the following hypotheses:
H1. Agents are more likely to overfit to a single adversary than population of adversaries, leaving them less robust on in-distribution tasks.
H2. Agents trained against population adversaries will generalize better, leading to improved performance dutto--dist tasks.
In-distribution tasks refer to the agent playing against perturbations that are in the training distribution: adversaries that add their actions onto the agent. However, the particular form of the adversary and their restricted perturbation magnitude means that there are many dynamical systems that they cannot represent (for example, significant variations of joint mass and friction). These tasks are denoted as out-of-distribution tasks. All of the tasks in the test set described in Sec. 5.1 are likely out-of-distribution tasks.
5.1 EXPERIMENTAL SETUP AND HYPERPARAMETER SELECTION
While we provide exact details of the hyperparameters in the Appendix, adversarial settings require additional complexity in hyperparameter selection. In the standard RL procedure, optimal hyperpa- rameters selected the basis of maximum expected cumulative reward. However, if an agent playing against an adversary achieves a large cumulative reward, it is possible that the agent was simply playing against weak adversary. Conversely, low score does necessarily indicate strong ad adversary nor robustness: it could simply mean that we trained weak agent.
To address this, we adopt version of the train validate-tett split from supervised learning. We use the mean policy performance on suite of validation tasks to select the hyperparameters, then we train the policy across ten seeds and report the resultant mean and standard deviation over twenty trajectories. Finally, we evaluate the seeds on holdout test set of eight additional model-min miimmatch tasks. These tasks vary significantly in difficulty; for visual clarity we report the average across tasks in this paper and report the full breakdown across tasks in the Appendix.
We experiment with the Hopper, Ant, and Half Cheetah continuous control environments used in the original RARL paper Pinto et al. (2017): these are shown in Fig. [ To generate the validation model mismatch, we pre-define ranges of mass and friction coefficients as follows: for Hopper, mass [0.7, 1.3] and friction E [0.7, 1.3]: Half Cheetah and Ant, mass E [0.5, 1.5] and friction E [0.1, 0.9). We scale the friction of every Mujoco geom and the mass of the torso with the same (respective) coefficients. We compare the robustness of agents trained via RAP against: 1) agents trained against a single adversary in a zero-sum game, 2) oracle agents trained using domain randomization, and 3) an agent trained only using PPO and perturbation mechanism. To train the domain randomization
5

---

Under review as a conference paper at ICLR 2021
Figure I: From left to right, the Hopper, Half-Cheetah, and Ant environments we use to test our algorithm.
oracle, at each rollout we uniformly sample friction and mass coefficient from the validation set ranges. We then scale the friction of all geoms and the mass of the torso by their respective coefficients; this constitutes directly training on the validation set. To generate the test set of model mismatch, we take both the highest and lowest friction coefficients from the validation range and apply them different combinations of individual geoms. For the exact selected combinations, please refer to the Appendix.
As further validation of the benefits of RAP, we include an additional set of experiments on continuous control task, a gridworld maze search task, and a Bernoulli Bandit task in Appendix Sec. F Finally, we note that both our agent and adversary networks are two layer-neural networks with 64 hidden units in each layer and a tanh nonlinearity.
6 RESULTS
H1. In-Distribution Tasks: Analysis of Overfitting
globally minimax optimal adversary should be unexploitable and perform equally well against any adversary of equal strength. We investigate the optimality of our policy by asking whether the minimax agent is robust to swaps of adversaries from different training runs, i.e. different seeds. Fig. 2 shows the result of these swaps for the one adversary and three adversary case. The diagonal corresponds to playing against the adversaries the agent was trained with while every other square corresponds playing against adversaries from a different seed. To simplify presentation, in the three adversary case, each square is the average performance against all the adversaries from that seed. We observe that the agent trained against three adversaries (top row right) is robust under swaps while the single adversary case is not (top row left). The agent trained against single adversary is highly exploitable, as can be seen by its extremely sub-par performance against an adversary from any other seed. Since the adversaries off-diagonal are feasible adversaries, this suggests that we have found a poor local optimum of the objective.
In contrast, the three adversary case is generally robust regardless of which adversary it plays against, suggesting that the use of additional adversaries has made the agent more robust. One possible hypothesis for why this could be occurring is that the adversaries in the "3 adversary" case somehow weaker than the adversaries in the "I adversary" case. The middle row of the figure shows that it is not the case that the improved performance of the agent playing against the three adversaries is due to some weakness of the adversaries. If anything, the adversaries from the three adversary case are stronger as the agent trained against 1 adversary does extremely poorly playing against the three adversaries (left) whereas the agent trained against three adversaries still performs well when playing against the adversaries from the dingleaddersee runs. Finally, the bottom row investigates how an agent trained with domain randomization fairs against adversaries from either training regimes. In neither case the domain randomization agent robust on these tasks.
H2.
Out-of- Distributitiiini Tasks: Robustness and Generalization of Population Training
Here we present the results from the validation and holdout test sets described in Section [.T We compare the performance of training with adversary populations of size three and five against vanilla PPO. the domain randomization oracle, and the single minimax adversary. We refer to domain randomization as oracle as it is trained directly the test distribution.
Fig shows the average reward (the average of ten seeds across the validation or test sets respectively) for each environment. Table gives the corresponding numerical values and the percent change of each policy from the baseline. Standard deviations are omitted on the test set due to wide variation in task difficulty; the individual tests that we aggregate here reported in the Appendix with

---

Under review as a conference paper at ICLR 2021
RARL Agent v. RARL Adv
RAP Agent v. RAP Adv
3500
3500
3000
3000
agent seed Indlex
2
agent seed index
2900
0 6
1500
1500
1000
1000
500
adversary seed Index
adversary seed index
RARL Agent V. RAP Adv
RAP Agent v. RARL Adv
3500
2540
3000
3000
agent seed Index
2-
Index
a
2500
2500
*
2000
8
-1500
agent
1500
1000
2000
500
2
8
adversary seed index
seed index
DR Agent vs. RARL Adv
DR Agent vs. RAP Adv
3503
3000
3000
agert seed index
2500
2500
1500
1500
1000
1000
500
500
adversary
Index
adversary seed Index
Figure 2: Top row: Average cumulative reward under swaps for one adversary training (left) and three-adversary training (right). Each square corresponds to 20 trials. In the three adversary each square is the average performance against the adversaries from that seed. Middle row: (Left) Playing the agent trained against adversary against the adversaries from the three adversary case. (Right) Playing the agent trained against 3 adversaries against the adversaries from the one adversary case. Bottom row: (Left) Playing the DR agent against the adversaries from the three adversary case. (Right) Playing the DR agent against the adversaries from the one adversary case.
appropriate error bars. In all environments we achieve a higher reward across both the validation and holdout test set using RAP of size three and/or five when compared the single minimax adversary
case. These results from testing on new environments with altered dynamics supports hypothesis
H2. that training with a population of adversaries leads more robust policies than training with single adversary in out-of-distin out-of-distribution tasks. Furthermore, while the performance is only comparable with the domain randomization oracle, the adversarial approach does not require prior engineering of appropriate randomizations. Furthermore, despite domain randomization being trained directly on these diitt-i-iittibttion tasks, domain randomization can have serious failure modes domain randomization due its formulation. A detailed analysis of this can be found in Appendix E
For more detailed comparison of robustness across the validation set, Fig. shows heatmaps the performance across all the mass, friction coefficient combinations, Here we highlight the heatmaps for Hopper and Half Cheetah for vanilla PPO, domain randomization oracle, single adversary, and best adversary population size. Additional heatmaps for other adversary population sizes and the Ant environment can be found in the Appendix. Note that Fig. is an example of a case where a single adversary has negligible effect or slightly reduces the performance of the resultant policy the
7

---

Under review as a conference paper at ICLR 2021
Hopper, Validation Set Reward
Cheetah, Validation Set Reward
Ant. Validation Reward
Avg. Reward
6000
Avg. Rewann
2000
2000
1500
3000
2000
300
Adv
DR
Adv
AdV
5 Adv
Adv
DR
Adv
Adv
Adv
Adv
DR
1 Adv
3 Adv
Adv
1
Hopper, Test Set Reward
Cheetah, Test Set Reward
Ant. Test Set Reward
3000
Reward
2500
Reward
Avg. Reward
2000
1000
5000
1500
2000
2000
0 Adv
DR
Adv
Adv
Adv
0 Adv
DR
Adv
Adv
Adv
Adv
DR
1 Adv
3 Adv
Adv
Figure 3: Average reward for Ant, Hopper, and Cheetah environments across ten seeds and across the validation set (top row) and across the holdout test set (bottom row). We compare vanilla PPO, the domain randomization oracle, and the minimax adversary against RAP of size three and five. Bars represent the mean and the each test-set sample. The std. deviation for the test set are not reported here for visual clarity due to the large variation in holdout test difficulty.
represent the std. deviation. Both are computed over 20 rollouts for
Validation
Test
Ant Mean Rew. % Change
Adv
DR
Adv
Adv
Adv
Adv
Adv
$ Adv
6336
6743
6349
6432
6438
2908
3613
3206
3272
3203
6.4
0.2
1.5
1.6
24.3
10.2
12.5
10.2
Validation Adv
Test
Hopper
0 Adv
DR
3 Adv
Adv
0 Adv 472
DR
Adv
3 Adv
S Adv
Mean Rew. % Change
1182
2662
1094
2039
2021
1636
913
1598
1565
125
-7.4
72.6
71
246
93.4
238
231
Validation
Test
Cheetah
Adv
DR
Adv
Adv
Adv
0 Adv
DR
Adv
Adv
Adv
Mean Rew.
5659
3864
5593
5912
6323
5592
3656
5664
6046
6406
% Change
-32
-1.2
4.5
11.7
-35
1.3
8.1
14.6
Table 1: Average reward and % change from vanilla PPO (0 Adv) for Ant, Hopper, and Cheetah environments across ten seeds and across the validation (left) or holdout test set (right). Across all environments, we see consistently higher robustness using RAP than the minimax adversary. Most robust adversarial approach is bolded as domain randomization is an oracle and outside the class of perturbations that our adversaries construct, and best result overall is italicized.
validation set. This supports our hypothesis that a single adversary can actually lower the robustness of an agent.
CONCLUSIONS AND FUTURE WORK
In this work we demonstrate that the use of a single adversary approximate the solution minimax problem does not consistently lead to improved robustness. We propose a solution through the use of multiple adversaries (RAP), and demonstrate that this provides robustness across a variety of robotics benchmarks. We also compare RAP with domain randomization and demonstrate that while DR can lead to more robust policy, it requires careful parametrization of the domain we sample from to ensure robustness. RAP does not require this tuning, allowing for in domains where appropriate tuning requires extensive prior knowledge or expertise.
There several open questions stemming from this work. While empirically demonstrate the effects of RAP, we do not have a compelling theoretical understanding of why multiple adversaries helping. Perhaps RAP helps approximate mixed Nash equilibrium discussed in Sec. Tor
8

---

Under review as a conference paper at ICLR 2021
Adv
Adv
Adv
3950
3000
#000
4000
2000
334
2000
2520
2340
2500
2000
2000
2300
2500
2000
1900
roon
Lan
1003
1930
120
1030
1003
990
900
500
Finction coef
Friction coef
Friction coef
Frictinn coef
0 Adv
DR
Adv
5 Adv
2000
7020
7500
1X333
9930
5000
5200
200
5000
5000
4000
3000
2000
Friction coof
Friction CooT
Friction coof
Frict coef
Figure 4: Average reward acTOSS ten seeds on each validation set parametrization - friction coefficient on the x-axis and mass coefficient on the y-axis. DR refers domain randomization and Adv an agent trained against X adversaries. Top row is Hopper and bottom row is Half Cheetah.
perhaps population based training increases the likelihood that of the adversaries is strong? Would the benefits RAP disappear if single adversary had the ability to represent mixed Nash?
There extensions of this work that would like to pursue. We have looked at the robustness of our approach in simulated settings; future work will examine whether this robustness transfers to real-world settings. Additionally, adversary identification; perhaps memory leads to a system-identiiiatioo procedure that improves transfer performance. Our adversaries also be viewed as forming task distribution, allowing them to be used in continual learning approaches like MAML (Nagabandi et al. 2018) where domain randomization is frequently used to construct task distributions.
agents currently memory-less and therefore cannot perform

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rika Antonova, Silvia Cruciani, Christian Smith, and Danica Kragic. Reinforcement learning for pivoting task. arXiv preprint arXiv:1703.00472. 2017.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456): 885-890, 2019.
Wojeiech Marian Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafiei, David Balduzzi, and Max Jaderberg. Real world games look like spinning tops. arXiv preprint arXiv:2004.09440 094668 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
Parameswaran Kamalaruban, Yu-Ting Huang. Ya-Ping Hsich, Paul Rolland. Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics. arXiv preprint arXiv:2002.066633 2020.
Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph E Gonzalez, Michael I Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. arXiv preprint arXiv: arXiv:1712.09381 2017.
Shiau Hong Lim, Huan Xu, and Shie Mannor. Reinforcement learning in robust markov decision
processes. In Advances in Neural Information Processing Systems, pp. 701-709, 2013.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher Pal, and Liam Paull. Active domain randomization. arXiv preprint 1904 047622, 2019.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based arXiv preprint arXiv:1812.07671, 2018.
Arnab Nilim and Laurent El Ghaoui Robust control markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-799, 2005.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1-8 8. IEEE, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings of the 34th International Conference on Machine Eetrning-Volumee 70, pp. 2817-2826. JMLR. org, 2017.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Aravind Rajeswaran, Sarvject Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv: 1610.0/233. 2016.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. arXiv preprini 1611::611 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust mdps using function approximation. In International Conference Machine Learning, pp. 181-189, 2014.
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm_ control: Software and tasks for continuous control. arXiv preprint arXiv:2006. 0200006/29880 /2983, 2020.
10

---

Under review as a conference paper at ICLR 2021
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control. arXiv preprint arXiv:1901.091%: 2019.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-355 2019.
Kemin Zhou and John Comstock Doyle. Essentials of robust control, volume 104. Prentice hall Upper Saddle River, NJ, 1998.
A FULL DESCRIPTION OF THE CONTINUOUS CONTROL MDPS
use the Mujoco ant, cheetah, and hopper environments as test of the efficacy of our strategy versus the 0 adversary. adversary, and domain randomization baselines. We use the Noisy Action Robust MDP formulation Tessler et al. (2019) for our adversary parametrization, the normal system dynamics are
sk+1 = Sh. + f(sks og At
the system dynamics under the adversary are
Sk+1 Sy + f(sk. an + a adv )
At
where adv is the adversary action at time k.
The notion here is that the adversary action is passed through the dynamics function and represents standard to clip actions within some boundary but for the above
some additional set of dynamics. reason, clip the agent and adversary actions separately. Otherwise, agent would be able limit the effect of the adversary by always taking actions at the bounds its clipping range. The agent is clipped between 1- in the Hopper environment and the adversary is clipped between [-225, .2].
The MDP through which we train the agent policy is characterized by the following states, actions, and rewards:
St agent = [or, az] where is an observation returned by the environment, and is the action
taken by the agent.
We use the standard rewards provided by the OpenAl Gym Mujoco environments at hhtttpp //github com openai / gym /t ree tree/master wws.maatt/r/eeeee wttpsstterr///////////////////// functions, please refer the code ANONYMIZED..
For the exact
agent
E [Amin= Omax]
The MDP for adversary is the following:
st
agent
The adversary sees the same states as the agent.
The adversary reward is the negative of the agent reward.
adv
E
a
adv
min
adv
max
For our domain randomization Hopper baseline, we use the following randomization: at each rollout, scale the friction of all joints by single value uniformly sampled from [0.7, 1.3]. We also randomly scale the mass of the 'torso" link by a single value sampled from [0.7. 1.3]. For Half-Cheetal and Ant the range for friction is 10.1, 0.9] and for mass the range is [0.5, 1.5].
11

---

Under review as a conference paper at ICLR 2021
Hopper, Validation Set Reward
Hopper, Test Set Reward
3000
Avg. Reward
Reward
2500
2000
2000
1500
1000
1000
500
500
Adv
1 Adv
2 Adv
3 Adv
5 Adv
7 Adv
9 Adv
11 Adv
0 Adv
Adv
2 Adv
3 Adv
5 Adv
7 Adv
9 Adv
11 Adv
Figure 5: Average reward for Hopper across varying adversary number.
B INCREASING ADVERSARY POOL SIZE
We investigate whether RAP is robust to adversary number as this would be useful property to minimize hyperparameter search. Here we hypothesize that while having more adversaries can represent wider range of dynamics learn to be robust to, we expect there to be diminishing returns due to the decreased batch size that each adversary receives (total number of environment steps held constant across all training variations). We expect decreasing batch size lead to worse agent policies since the batch will contain under-trained adversary policies. We cap the number of adversaries at eleven our machines adversary value and Fig. shows the results for Hopper. Agent robustness on the test set increases monotonically up to three adversaries and roughly begins decrease after that point. This suggests that a trade-off between adversary number and performance exists although we do not definitively show that diminishing batch sizes is the source of this trade-off. However, we observe in Fig. 6 that both three and five adversaries perform well across all studied Mujoco domains.
of memory at this value. We run ten seeds for every
Ant. Validation Set Reward
Hopper, Validation Set Reward
Cheetah, Validation Set Reward
Avg. Reward
6200
Avg. Reward
4200
1300
1000
2000
300
Adv
DR
1 Adv
3 Adv
5 Adv
Adv
DR
Adv
Adv
Adv
AdV
DR
Adv
Adv
Adv
Ant, Test Set Reward
Hopper, Test Set Reward
Cheetah, Test Set Reward
Avg. Reward
0205
Reward
2500
Avg. Reward
2000
4200
#000
1203
2000
2000
Adv
DR
1 Adv
3 Adv
5 Adv
0 Adv
DR
Adv
Adv
Adv
0 Adv
DR
Adv
Adv
5 Adv
Figure 6: Average reward for Ant, Hopper, and Cheetah environments across ten seeds and across the
validation set (top row) and across the holdout test set (bottom row). We compare vanilla PPO, the domain randomization oracle, and the minimax adversary against RAP of size three and five. Bars represent the mean and the arms represent the std. deviation. Both computed over 20 rollouts for each test-set sample. The std. deviation for the test set are not reported here for visual clarity due to the large variation in holdout test difficulty.
C
HOLDOUT TESTS
In this section we describe in detail all of the holdout tests used.
12

---

Under review as a conference paper at ICLR 2021
torso
thigh
floor
leg
1oot
Figure 7: Labelled Body Segments of Hopper
Table 2: Hopper Holdout Test Descriptions
Test
Body with Friction Coeff 1.3
Body with Friction Coeff 0.7
A
Torso, Leg
Floor, Thigh, Foot
Floor, Thigh
Torso, Leg, Foot
C
Foot, Leg
Floor, Torso, Thigh
D
Torso, Thigh, Floor
Foot, Leg
E
Torso, Foot
Floor, Thigh, Leg
F
Floor, Thigh, Leg
Torso, Foot
G
Floor, Foot
Torso, Thigh, Leg
H
Thigh, Leg
Floor, Torso, Foot
C.1 HOPPER
The Mujoco geom properties that we modified are attached to particular body and determine its appearance and collision properties. For the Mujoco holdout transfer we pick subset of the hopper 'geom' elements and scale the contact friction values by maximum friction coefficient, 1.3. Likewise, for the rest of the 'geom' elements, we scale the contact friction by the minimum value of 0.7. The body geoms and their names are visible in Fig. 7
The exact combinations and the corresponding test name are indicated in Table T Hopper.
C.2 CHEETAH
The Mujoco geom properties that we modified are attached to particular body and determine its appearance and collision properties. For the Mujoco holdout transfer tests we pick subset of the
head
bthigh
torso
Kingh
Hooot
Ishi
bshin
floor
bfoot
Figure 8: Labelled Body Segments of Cheetah
13

---

Under review as a conference paper at ICLR 2021
Table 3: Cheetah Holdout Test Deseriptions. Joints in the table receive the maximum friction coefficient of 0.9, Joints not indicated have friction coefficient 0.1
Test
Geom with Friction Coeff 0.9
A
Torso, Head, Fthigh Floor, Head, Fshin Bthigh, Bshin, Bfoot Floor, Torso, Head Floor, Bshin, Ffoot Bthigh, Bfoot, Ffoot Bthigh, Fthigh, Fshin Head, Fshin, Ffoot
B
C
D
G
H
fron
firomt
left
carso
right
back-
right
Neeefttt
aux
leg
Figure 9: Labelled Body Segments of Ant
cheetah 'geom' elements and scale the contact friction values by maximum friction coefficient, 0.9. Likewise, for the rest of the 'geom' elements, we scale the contact friction by the minimum value of 0.1. The body geoms and their names visible in Fig. g
The exact combinations and the corresponding test name are indicated in Table for Hopper.
C.3 ANT
We will use torso to indicate the head piece, leg to refer to one of the four legs that contact the ground, and "aux' to indicate the geom that connects the leg to the torso. Since the ant is symmetric we adopt a convention that two of the legs are front-left and front-right and two legs are back-left and back-right. Fig. 9ig.9depiict depicts the convention. For the Mujoco holdout transfer tests pick subset of the ant 'geom' elements and scale the contact friction values by maximum friction coefficient, 0.9. Likewise, for the rest of the 'geom' elements, scale the contact friction by the minimum value of 0.1.
Table 4: Ant Holdout Test Descriptions. Joints in the table receive the maximum friction coefficient of 0.9. Joints not indicated have friction coefficient 0.1
Test
Geom with Friction Coeff 0.9
A
Front-Leg-Left, Aux-Front-Left, Aux-Back-Left Torso, Aux-Front-Left. Back-Leg-Right
B
C
Front-Leg-Right, Aux-Front-Right. Back-Leg-Left
D
Torso, Front-Leg-Left Aux- Aux-Front-Left
E
Front-Leg-Left, Aux-Front-Righs
Back-Right
F
Front-l.eg-Right, Back-Lee-LLeft, Lux-Bac-Riigh
G
Front-Leg-Leet,, Aux-Back-Left. Back-Leg-RRige
H
Aux-Front-Lefi, Back-Leg- Right, Aux-Back-RR Right
14

---

Under review as a conference paper at ICLR 2021
Test Name
Adv
Adv
Adv
Five Adv
Domain Rand
Test A
410 = 140
1170 570
2210 630
2090 920
1610 310
Test
430 + 150
+40
2240 730
2200 880
1610 290
Test
560 + 120
490 150
610 250
580 120
1660 + 260
Test D
420 + 150
1140 560
2220 + 680
2130 + 890
1612+3600
Test
550 + 120
500 + 150
600 240
590 120
1680 + 280
Test F
420 = 150
1200 + 620
2080 750
2160 + 890
1650 = 360
Test
560 + 130
500 140
600 230
600 + 140
1710 370
Test G
420 + 150
1160 + 590
2210 + 680
2160 + 920
1560 + 340
Table 5: Results on holdout tests for each of the tested approaches for Hopper. Bolded values have the highest mean
Test Name
0 Adv
I Adv
Adv
Five Adv
Domain Rand
Test A
4400 + 2160
5110 730
4960 + 1280
5560
2800 + 1540
Test
6020 880
5980 = 290
6440 + 1620
6
3340 + 600
Test C
5880 + 1030
5730 640
6740
6410 790
4280 + 240
Test
5990 940
5960 = 260
6430 + 1610
6880
3360 570
Test E
5570 = 570
5670 = 290
5800 = 1316
6530
3720 = 540
Test
5870 = 750
5800 = 350
6500 = 1100
6770+
3810 = 330
Test H
5310 + 1060
5270 + 700
5610 + 720
5660 + 980
4560 + 560
Test
5710 + 650
5790 + 300
5890 + 1240
6560 1240
3380 + 720
Table Results on holdout tests for each of the tested approaches for Half Cheetah. Bolded values have the highest mean
The exact combinations and the corresponding name indicated in Table 4 for Hopper.
D RESULTS
Here recompute the values of all the results and display them with appropriate standard deviations in tabular form.
There space for the validation results so they reproduced here.
Test Name
Adv
Add
3 Adv
Five Adv
Domain Rand
Test
590 + 650
730+630
+ 630
600 440
560 580
900 580
Test B
5240 + 280
5530 200
1770
5710 180
6150 180
Test
750 + 820
1090 - 660
1160
1040
1370 800
Test D
5220 + 300
5560 = 220
5770 =
5660 190
6120 180
Test
5270 + 290
5570 + 210
5770 + 100
5660 + 220
6140 + 150
Test F
780 + 860
1160 + 570
1120 580
1140
1390 + 750
Test
130 + 290
420 + 300
210 220
160 + 270
700 + 560
Test
5290 280
5560 220
5770 + 100
5700 + 190
6150 160
Table 7: Results holdout tests for each of the tested approaches for Ant. Bolded values have the
highest mean
15

---

Under review as a conference paper at ICLR 2021
0 Adv
7000
7000
5000
6000
Mass coef
3000
0.0
1:0
3000
14-
118
120
2000
120
379039 wer Mab box sairentnniree
2000
Friction coef
Friction coef
Adv
Adv
5 Adv
7000
7000
6000
6000
6000
0.00
A49
380
5000
148
5000
190
5000
4000
4000
4000
110
190
140
2000
2000
#1s0terder oniin fer
2000
Friction coef
Friction coef
Friction coef
Figure 10: Ant Heatmap: Average reward across 10 seeds on each validation set (mass, friction) parametrization.
E CHALLENGES OF DOMAIN RANDOMIZATION
In our experiments, we find that naive parametrization of domain randomization can result a brittle policy, even when evaluated on the same distribution it was trained on.
Effect of Domain Randomization Parametrization
From Fig. 6 we that in the Ant and Hopper domains, the DR oracle achieves the highest transfer reward in the validation set as expected since the DR oracle trained directly on the validation set. Interestingly, found that the domain randomization policy performed much worse on the Half Cheetah environment, despite having access to the mass and friction coefficients during training. Looking the performance for each mass and friction combination in Fig. [1] we found that the DR agent was able to perform much better at the low friction coefficients and learned to prioritize those values the cost of significantly worse performance on average. This highlights potential issue with domain randomization: while training across wide variety of dynamics parameters can increase robustness, naive parametrizations can cause the policy to exploit subsets the randomized domain and lead brittle policy. This is problem inherent to the expectation across domains that used in domain randomization; if some subset of randomizations have sufficiently high reward the agent will prioritize performance those the expense of robustness.
We hypothesize that this is due to the DR objective Eq. 2 optimizing in expectation over the sampling range. To test this, created separate range of 'good' friction parameters [0.5, 1.5) and compared the robustness a DR policy trained with 'good' range against a DR policy trained with "bad" range [0.1, 0.9) in Fig. [1 Here we see that 'good' parametrization leads the expected result where domain randomization is the most robust. We observe that domain randomization underperforms adversarial training the validation despite the validation literally constituting the training set for domain randomization. This suggests that underlying optimization difficulties caused by significant variations in reward scaling are partially to blame for the poor performance of domain randomization. Notably, the adversary-based methods are not susceptible to the same parametrization issues.
Alternative DR policy architecture
As discussed above and also identified in Rajeswaran et al. (016)) the expectation across random- izations that is used in domain randomization causes prioritize policy that performs well in a high-reward subset of the randomization domains. This is harmless when domain randomization used for randomizations of state, such color, where all the randomization environments have the same expected reward, but has more pernicious effects in dynamics randomizations. Consider set of N randomization environments, N 1 of which have reward Row and one of which has
16

---

Under review as a conference paper at ICLR 2021
Cheetah, Valid. Set Reward Bad Param.
Cheetah, Test Set Reward Bed Param.
Avg. Reward
6002
Avg. Reward
6000
6000
#000
2000
2000
AdV
Adv
Adv
5 Adv
Adv
D8
Adl
Adv
Cheetah, Valid. Set Reward Good Param.
Cheetah. Test Set Reward- Good Param.
2000
Avg. Rewaard
6003
1000
4000
AdV
Dm
AdV
Adv
Adv
Adi
Dn
AdV
Adv
Adi
Figure 11: Average reward for Half Cheetah environment across ten seeds. Top row shows the average reward when trained with a 'bad' friction parametrization which lead to DR not learning robust agent policy, and bottom row shows the average reward when trained with 'good' friction parametrization.
has reward Rnigh where Rnigh >> Rlow If the agent cannot identify which of the randomization environments it is in, the intuitively optimal solution is pick the policy that optimizes the high reward environment. One possible way out of the quandary is to use an agent that has some memory, such as an LSTM-based policy, thus giving the possibility of identifying which environment the agent is in and deploying the appropriate response. However, if Rnigh is sufficiently large and there is some reduction in reward associated with performing the iestem-identification necessary to identify the randomization, then the agent will not perform the system identification and will prioritize achieving Rnigh- As an illustration of this challenge, Fig. T2 compares the results of domain randomization on the half-cheetah environment with and without memory. In the memory case, we use 64 unit LSTM. As can be seen, there an improvement the ability of the domain randomized policy to perform well on the full range of low-friction/ / high mass values, but the improved performance does not extend to higher friction values. In fact, the performance contrast is enhanced even further as the policy does good deal worse the high friction values than the case without memory.
DR
with Memory
7000
3.00
7000
3.60
4.20
4.80
5.40
5000
4030
6.60
174
3000
3000
2000
0.30 0.26 0.42 0.58 0.74 0.90 Friction coef
Friction coef
Figure 12: Left: heatmap of the performance of the half-cheetah domain randomized policy across the friction and mass value grid. Right: Left: heatmap of the performance of the half-cheetah domain randomized policy across the friction and mass value grid where the agent policy is an LSTMM
F ADDITIONAL EXPERIMENTS
Here outline few more experiments we that demonstrate the value of additional adversaries. We run the following tasks:
F.I DEEPMIND CONTROL CATCH
This task uses the same Markov Decision Process described in Sec. A The challenge TTssaa et al. 2020), pictured in Fig. 13. is to get the ball fall inside the cup. As in the other continuous control
17

---

Under review as a conference paper at ICLR 2021
U
Figure 13: The DeepMind Control catch task. The cup moves around and attempts get the ball to fall inside.
tasks, we apply the adversary to the actions of the agents (which is controlling the cup). We then test variations of the mass of both the ball and the cup. The heatmaps for this task presented in Fig. 14 where the adversary case provides a slight improvement in the robustness region relative to the adversary case.
0.05
adv
1000
0.05
adv
1000
0.45
0.45
000
1,63
1.63
2.02
202
242
2.42
201
400
400
3.00
3.00
200
4.03
4.00
0.10
0.48
0.55
1.24
1.82 2.00
0.10 D.A5
1.24
2.50
Ball Maa Couft
dall Mass Coeft
adv
1000
3.ddv
1000
0.05
0.05
0.45
0.45
0.84
1,89
2.02
2.02
2.42
2.42
2.81
400
2.61
400
Sell
200
4.00
4.00
0.10 0.48 0.55 1224 162 2.00
0.10 0.48 0.86 1.24 1.62 2.30 eull Mass Cooff
Bill Maz Coeff
Figure 14: (Top left) adversary, (top right) I adversary, (bottom left) 3 adversary, (bottom right) adversaries for variations of cup and ball mass.
F.2 MUL.TI-ARMED BERNOULLI BANDITS
As an illustrative example, we examine multi-armed stochastic bandit, a problem widely studied in reinforcement learning literature. Generally, successful strategies for multi-arm bandit problems involve successfully balancing the exploration across arms and exploiting the 'best' arm. A "robust" strategy should have vanishing regret as the time horizon goes to infinity. We construct 10-armed bandit where each arm is parametrized by value p where p is the probability of that returning
18

---

Under review as a conference paper at ICLR 2021
1. The goal of the agent is to minimize total cumulative regret R, over a horizon of steps:
R max p4
at
where a, corresponds to picking particular arm. At each step, the agent is given an observation buffer of stacked frames consisting of all previous (action, reward) pairs padded with zeros to keep I; at time-step zero receives an observation of and outputs the probability for each arm. At the termination of the horizon the adversary receives the negative of the cumulative agent reward. For our domain randomization baseline we use uniform sampling of the p value for each arm. We chose horizon length of T 100 steps. The MDP of the agent is characterized as follows:
the length fixed. The adversary has a horizon
St- [o 1 Tt, Ge, 1t-1, as-1, 4g
= ((a)) - max; (4g
agent
E ....9
At each step, the agent is given an observation buffer of stacked frames consisting of all previous (action, reward) pairs. The buffer matching the horizon length is padded with zeros. For each training step, the agent receives a reward the negative expected regret. We set up the adversary problem as an MDP with horizon of 1.
8t- = [0.0]
ΣE= 22
an E [0, 1j
During adversarial training, we sample a random adversary at the beginning of each rollout, and allow it to pick 10g p values that then shuffled randomly and then assigned to each arm (this is to prevent the agent from deterministically knowing which arm has which value). The adversary is always given observation of vector of zeros and is rewarded once at the end of the rollout. We also construct a hold-out test of two bandit examples which we colloquially refer to as "evenly spread" and "one good arm." In "evenly spread", the arms, going from I to 10 have evenly spaced probabilities in steps of 0.1 0.1, 0.2, 0.3. 0.8, 0.9. In "one good arm" 9 arms have probability 0.1 and one arm has probability 0.9. As our policy for the agent, we use Gated Recurrent Unit network with hidden size 256.
An interesting feature of the bandit task that makes clear that the single adversary approach corresponds to training on a single, adversarially constructed bandit instance. Surprisingly, as indicated in Fig. [5] this does not perform terribly on our two holdout tasks. However, there is a clear improvement both tasks in the four adversary case. All adversarial approaches outperform an Upper Confidence bosed expert (shown in red). Interestingly, domain randomization, which had superficially good reward at training time, completely fails on the "one good arm" holdout task. This suggests another possible failure mode of domain randomization where in high dimensions uniform sampling may just fail yield interesting training tasks. Finally, we note that since the upper confidence approach only tries to minimize regret asymptotically, our outperforming it may simply be due to our relatively short horizon; simply provide it baseline.
G COST AND HYPERPARAMETERS
Here reproduce the hyperparameters used in each experiment and compute the expected run- time and cost of each experiment. Numbers indicated in O were each used for one run. Otherwise the parameter was kept fixed at the indicated value.
G.1 HYPERPARAMETERS
For Mujoco the hyperparameters are:
Learning rate:
19

---

Under review as a conference paper at ICLR 2021
10-armed Bandit Transfer Performance
0
-10
-30
RARL
Adversaries
10 Adversaries
40
Domain Randomization
Expert UCB1
Evenly Spread
One good
Figure 15: Two transfer tests for the bandit task. On both tasks the 4 adversary case has improved performance relative to RARL while domain randomization performs terribly all tasks. Bars indicate one std. deviation of the performance over 100 trials.
{.0003, .0055) for half cheetah
{.0005, .00005) for hopper and ant
Generalized Advantage Estimation À
{0.9, 0.95, 1.0} for half cheetah {0.5, 1.0} for hopper and ant
Discount factor 0.995
Training batch size: 100000
SGD minibatch size: 640
Number of SGD steps per iteration:
Number of iterations: 700
We set the seed to 0 for all hyperparameter runs.
The maximum horizon is 1000 steps.
For the validation across seeds used 10 seeds ranging from 0 to 9. All other hyperparameters the default values in RLlib Liang et al. (2017) 0.8.0
G.2
COST
For all of our experiments we used AWS EC2 exlaare instances which come with virtual CPUs. For the Mujoco experiments, we use 2 nodes and 11 CPUs per hyper-parameter, leading to one full hyper-parameter sweep fitting onto the 72 CPUs. We run the following set of experiments and ablations, each of which takes 8 hours.
0 adversaries
adversary
3 adversaries
adversaries
Domain randomization
for total of 5 experiments for each of Hopper, Cheetah, Ant. For the best hyperparameters and each experiment listed above we run a seed search with 6 CPUs used per-seed, a process which takes about 12 hours. This leads total of
2 12 * 5 600 node hours and 36 600 22000
20

---

Under review as a conference paper at ICLR 2021
3500
3000
2500
2000
1500
1000
adv
adv
adv
adv
5000
10000
20000
25000
Figure 16:
Wall-clock time vs. reward for varying numbers adversaries. Despite varying adversary numbers, the wall-clock time of 3, 5, and 7 adversary runs are all the same.
CPU hours. At cost of 0.3 dollars per node per hour for EC2 spot instances, this gives 180 dollars to fully reproduce our results for this experiment. the chosen hyperparameters are used and only the seeds are sweep, this 100 dollars.
G.3 RUN TIME AND SAMPLE COMPLEXITY
Here we briefly analyze the expected run-time of our algorithms. While there is an additional cost for adding single adversary equal to the sum of the cost of computing gradients train time and actions run-time for additional agent, there is additional cost for adding additional adversaries. Since we divide the total set of samples per iteration amongst the adversaries, we compute approximately do in the single adversary
the same number of gradients and actions in the many-adversary case case. In Fig. [6] of reward vs. wall-clock time supports this argument: the 0 adversary case runs the fastest but all the different adversary numbers complete 700 iterations of training in approximately the same amount of time. Additionally, Fig. 17 demonstrates that there is some variation in sample complexity but the trend is not consistent across adversary number.
G.4 CODE
Our code is available ANONYMIZED. For our reinforcement learning code-base used RLlib Liang et al. (2017) version 0.8.0 and did not make any custom modifications to the library.
H PURE NASH EQUILIBRIA DO NOT NECESSARILY EXIST
While there are canonical examples of games in which pure Nash equilibria do exist such as nock-paper-scissee
aware for sequential games with continuous actions. Tessler ct al. (2019) contains an example a simple, horizon MDP where duality is not satisfied. The pure minimax solution does not equal the value of the pure maximin solution and greater value can be achieved by randomizing one the policies showing that there is no pure equilibrium.
21

---

Under review as a conference paper at ICLR 2021
3500
3000
2500
1500
adv
500
adv
adv
100
200
300
400
500
600
700
Inerations
Figure 17: Iterations vs. reward for varying numbers of adversaries. Despite varying adversary numbers, the wall-clock time of 5, and 7 adversary runs all the same.
22


Reviews:
[{"review_id": "OCTwp4ylrkl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends the existing work on robust adversarial RL by training multiple adversarial agents from a population. Solid experimental results are presented to show that the proposed method improves the single adversary setting and domain randomization. \n\nThe experimental results in this paper seem solid to me. My biggest concern for this paper is that the conceptual novelty seems quite incremental. I mean, on the conceptual level, it is a common sense from robust control that multiple uncertainty sources can be treated together (e.g. the robust control theory handles the structured uncertainty in such a manner). One can augment all the adversary networks as a big network and then the problem formulation is the same as before. If we think each phi_i is a block in this big \"single adversary network\", then what the authors have done can be thought as doing  block coordinate descent. From this perspective, there is not too much conceptual novelty, and the main contribution of this paper is doing some more detailed study showing how to combine augmentation and adversarial RL. I am not sure whether such a contribution itself is enough for ICLR or not.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DhQXct3DIFZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm to improve the robustness of reinforcement learning. The algorithm , RAP, combines ideas from domain randomization and adversarial training. Specifically, during learning, it trains an ensemble of adversary to attack the learner, with the hope that the learner can be robust to various situations. The experimental results show the proposed algorithm indeed outperform the respective baselines here (single-adversary training and domain randomization) in its ability to generalize the other test domains. \n\nI think overall the proposed algorithm presents a simple and nice way to join the strengths of the adversary training and domain randomization. Indeed, we can view single-adversary training and domain randomization as special cases of RAP, which either uses a single adversary or not perform any update on the adversaries. For this, I would imagine RAP can be quite effective in practice. \n\n* Issue about motivation and explaination\n\nHowever, despite this nice design, I think the main motivation and the explanation why the proposed algorithm works are not completely correct. In introduction, the authors motivate the use of multiple adversary from that pure Nash equilibrium does not always exist in zero sum two player games. However, adversary training is not necessarily about solving the Nash equilibrium (which aims to find solutions such that minimax = maximin) but rather solving a maximin \"only\", whose solution is always well defined. \n\nThe motivating robot example is also quite misleading. In that case, the failure is due to the learner policy is not even solving maximin problem. Note in maximin, the adversary is chosen after knowing the learner's policy. This robot example is rather saying that minimax, where the learner optimizes after the adversary is chosen, is insufficient to generate robust behavior. However, this is not what adversarial training is about. \n\nI think, it's probably because of this misunderstanding, the authors motivate the issue of the single-adversary training as the agent would overfit to the single adversary. Again this is due to incorrectly interpreting maximin in (1) as minimax. \n\nNonetheless, I do believe the proposed algorithm is effective in practice, but for a different reason from what the authors explain. I think the RAP does improve upon single-adversary training. Because it uses multiple randomly initialized adversarial policies, it may have a higher chance to overcome the non-convexity issue in the min part of maxmin and  therefore has a higher chance to find the maximin solution. In other words, the failure of the single-adversary implementation is most due to optimization difficulty not that the solution concept is incorrect. And the proposed algorithm is more of an optimization heuristic to better approximate the maximin problem (which actually is quite commonly used in the optimization literature). In fact, when given the learner policy, I believe by further taking the min among the multiple adversaries here and uses that for the learner update (i.e. the leaner would not use trajectories from all but the worst one), the robustness of the algorithm might further improve. \n\n* Experiments \n\nGiven this work is lacking theoretical insights, I expect more experiments to be done to verify the proposed algorithm. The current paper only test RAP on mujoco environments. I think for understanding how well this algorithm works, the authors should test the algorithm on a more tasks with diverse characteristics (e.g. tabular, video games, tabular, traffic,) rather just the continuous robotics control domain. \n\nIn figure 2, I think a fairer comparison should let agents of both sides face the same adversary. Nonetheless, I agree that the performance plots later on are sufficient to show that RAP is better. \n\nLastly I like the in-depth discussion about the failure of domain randomization in halfcheetah, as it's also my main question when reading the previous part of the paper. However, I'm wondering if the bad performance is due to that the learner's policy is not expressive enough. I cannot find the description of the exact architecture used in the paper, but can you try training with a more expressive policy and see if domain randomization still perform worse than the direct training without the adversary?\n\n\n\n\n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5wc1ZUqVmM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes to improve robustness in reinforcement learning via a population of diverse adversaries, where previous works mainly focus on the use a single adversary to mitigate the problem that the trained policy could be highly exploitable by the adversary. Specifically, at each iteration, it randomly selects an adversary from the population for rollouts, and it is trained by PPO. Experiments are conducted on 3 MuJoCo environments in comparison with vanilla PPO, domain randomization. \n\nStrong points: Using a population of adversaries to improve robustness in RL is interesting. The idea is simple, and the writing is clear.\n\nConcerns:\nMy major concern is in the experimental evaluation.\na. Results are shown using final performance. I am curious about the learning curve – how does the method compare against other baselines in terms of sample efficiency? A side-effect using a population is that RAP needs to update n adversaries at each training iteration compared with using a single adversary, and will incur more computation overhead. Could authors fairly compare with other baselines in terms of this and show the learning curve? \n\nb. How *MUCH LONGER* does it take to run RAP compared with other baselines? How much more memory does it take to use n adversaries compared with a single adversary?\n\nc. Could authors compare with a naive extension of the single adversary case in which the single adversary sample n actions? Is the baseline comparable with RAP using n adversaries? \n\nd. I am confused why RAP is built upon an on-policy algorithm. A number of works using population-based methods are built upon off-policy algorithms as agents in the population can share the samples and could be beneficial. Could authors build the method upon off-policy algorithms to further improve the applicability of RAP?\n\ne. For Figure 3, the performance gain over using a single adversary is not significant on HalfCheetah and Ant, and the results is not convincing enough to support the claim. \n\nAs the paper uses the population-based methods, it is also worth discussing its relation with Khadka et al. 2018, etc.", "extra_scores": {}, "extra_text": {}}, {"review_id": "13vwhyz4HtM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe authors present a scheme that can be used to train agents to be robust against a population of adversarial policies, in which adversaries can perturb actions via an additive perturbation.  Motivated by the observation that agents trained against a single policy may overfit to that policy and hence will lack robustness to new/unseen policies, the authors seek to show that their method generalizes well to unseen policies at test time.  Their experiments consider several simulated environments, in which they show generally good performance against several baselines.\n\n#### Strengths\n\n- I find the argument that agents will overfit to a single policy convincing.  While the motivating example WRT different forces acting on an agent may not be a scenario that robustness against adaptive perturbations can handle, the general claim seems to hold water.  That is, it seems plausible that an agent might receive high reward in a zero sum min-max game subject to a single adversary simply because the adversary is not strong enough to limit the cumulative reward that agent receives.\n\n- The formulation for RAP is presented very clearly.  It is quite helpful to first consider the single minmax adversary and domain randomization formulations, both of which seemed to have played roles in the development of RAP.  Indeed, this seems a very natural way of formulating the problem.  More generally, the paper is quite well written.\n\n- The experiments, and in particular Fig 2, clearly demonstrate the utility of this approach.  One can see a notable difference when an agent is trained with respect to three adversarial policies vs. when it is only trained with a single adversarial policy.\n\n#### Weaknesses\n\n- The utility of this approach is less clear when one considers Fig. 3.  It seems that domain randomization often outperforms RAP.  In my opinion, this weakens part of the claim made by the authors.  However, it does seem true that given that DR is designed to perform well on new domains _in expectation_, it may be preferable to use RAP when one suffers worst case dynamics changes (e.g. Fig. 5).  \n\n- Further study should be done as to how many adversaries one needs to provide meaningful levels of robustness.  It seems that in different experiments, 1, 3, and 5 adversaries were considered.  How should one decided how many adversaries to use?  \n\n- I felt the paper was a bit unclear about what states are available to train adversarial and regular policies.  That is, it is unclear whether in the trajectories $\\tau_j$ and $\\tau_j^i$, the agents/adversaries observe their own actions, the actions of their counterpart (e.g. the agent observing the adversaries action), or both agent and adversary observing the perturbed action $a + \\alpha \\bar{a}$.  The latter case would lead to a lack of observability for both agent and adversary.  Perhaps the authors can clarify this in the rebuttal.\n\n- One weakness is that only one scenario (e.g. walking within these three environments) was considered.  It seems that the claims of the paper could be strengthened if more environments/tasks were considered.\n\n#### Further questions/clarifications\n\n- The reward function seems to be denoted as $\\mathcal{R}$, $R$, and $r$ in various places.\n\n- The bolding in Table 1 is a bit confusing.  I would be fairer to bold the most robust approach overall, rather than the most robust approach of the methods you propose.\n\n- In the notation of Section 4, why are the rollouts of length $M$ rather than $T$, as indicated in the formulation?\n\n#### Final Thoughts\n\nOverall, I thought this was a solid and interesting paper.  The motivation is compelling, the formulation is relatively clean, and the experiments generally back up the claims that are made by the authors.  There are a few weaknesses, as I enumerated above, but even still I think that this is a valuable contribution.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies reinforcement learning in the presence of (adversarial) perturbations in the underlying system dynamics. The main (novel) observation is that  agents trained against a single policy may overfit  to that policy and hence will lack robustness to new/unseen policies. The paper proposes a population-based augmentation to the Robust RL formulation in which a population of adversaries are randomly initialized and samples from during training. The authors seek to show that their method generalizes well to unseen policies at test time.

Most reviewers agree that the paper provides a range of solid experimental results (with in-distribution and out-of-distribution tasks) showing robustness and generalization of their methods on several robotics benchmarks while avoiding a ubiquitous domain randomization failure mode. However, all the reviewers (and myself) agree that some of the conceptual claims of the paper may not be precise. For example, some of the reviewers disagree with the authors on finding the (mixed) Nash equilibria. Such general claims are hard to validate (may not even be true) and need theoretical justification. Hence, it is not conceptually clear why using multiple adversaries would not suffer from the same limitations as in the single adversary case.  Also, in the discussion phase, the reviewers agreed that the results/claims of the paper (i.e. overfitting to a single adversary and the need for multiple adversaries) are very interesting, but at the same time need to be confirmed by more extensive experiments.  

Indeed, if the above are addressed, the paper would make a strong contribution to the area of RL. 


Decision: Reject