ERMAS: Learning Policies Robust to Reality Gaps in Multi-Agent Simulations
Authors: Eric Zhao
Alexander R Trott
Caiming Xiong
Stephan Zheng
Venue:  (2021)
Keywords: Robustness
Multi-Agent Learning
Sim2Real
Reinforcement Learning

Abstract: Policies for real-world multi-agent problems, such as optimal taxation, can be learned in multi-agent simulations with AI agents that emulate humans. However, simulations can suffer from reality gaps as humans often act suboptimally or optimize for different objectives (i.e., bounded rationality). We introduce $\epsilon$-Robust Multi-Agent Simulation (ERMAS), a robust optimization framework to learn AI policies that are robust to such multi-agent reality gaps. The objective of ERMAS theoretically guarantees robustness to the $\epsilon$-Nash equilibria of other agents – that is, robustness to behavioral deviations with a regret of at most $\epsilon$. ERMAS efficiently solves a first-order approximation of the robustness objective using meta-learning methods. We show that ERMAS yields robust policies for repeated bimatrix games and optimal adaptive taxation in economic simulations, even when baseline notions of robustness are uninformative or intractable. In particular, we show ERMAS can learn tax policies that are robust to changes in agent risk aversion, improving policy objectives (social welfare) by up to 15% in complex spatiotemporal simulations using the AI Economist (Zheng et al., 2020).

Under review as a conference paper at ICLR 2021
ERMAS: LEARNING POLICIES ROBUST TO REALITY GAPS IN MULTI-AGENT SIMULATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Policies for real-world multi-agent problems, such as optimal taxation, can be learned in multi-agent simulations with Al agents that emulate humans. How- ever, simulations can suffer from reality gaps as humans often act suboptimally or optimize for different objectives (i.e., bounded rationality). We introduce e- Robust Multi-Agent Simulation (ERMAS), robust optimization framework learn AI policies that are robust to such multi-agent reality gaps. The objective of ERMAS theoretically guarantees robustness to the c-Nash equilibria of other agents that is, robustness to behavioral deviations with a regret of at most c. ERMAS efficiently solves first-order approximation of the robustness objective using meta-learning methods. We show that ERMAS yields robust policies for repeated bimatrix games and optimal adaptive taxation in economic simulations, even when baseline notions of robustness are uninformative or intractable. In particular, we show ERMAS can learn tax policies that are robust changes in agent risk aversion, improving policy objectives (social welfare) by up to 15% in complex spatiotemporal simulations using the Al Economist (Zheng et al., 2020).
INTRODUCTION
Reinforcement learning (RL) offers a tool to optimize policy decisions affecting complex, multi- agent systems; for example, to improve traffic flow or economic productivity. In practice, the need for efficient policy evaluation necessitates training on simulations of multi-agent systems (MAS). Agents in these systems can be emulated with fixed behavioral rules, or by optimizing for a reward function using RI. (Zheng al., 2020). For instance, the impact of economic policy decisions are of- ten estimated with baseet models (Holland & Miller, 1991; Bonabeau, 2002). This commonly introduces a reality gap as the reward function and resulting behavior simulated agents might differ from those of real people (Simon & Schaeffer, 1990). This becomes especially problematic as the complexity of the simulation grows, for example, when increasing the number of agents, or adding agent affordances (Kirman, 1992; Howitt, 2012). As result, policies learned in imperfect simulations need to be robust against reality gaps in order to be effective in the real world.
We introduce e-Robust Multi-Agent Simulation (ERMAS), robust optimization framework for training robust policies, termed planmers, plammerrs that interact with real-world multi-agent systems. ERMAS trains robust planners by simulating multi-agent systems with RL and sampling worst-case behaviors from the worst-case agents. This form of multi-agent robustness poses a very challenging multi- level (e.g., max-min-min) optimization problem. Existing techniques which could be applied to ERMAS's multi-agent robustmess objective, e.g., naive adversarial robustness (Pinto et al., 2017) and domain randomization (Tobin al., 2017; Peng et al., 2018). are intractable as they would require an expensive search through large space of agent reward functions. Alternative frameworks improve robustness, e.g., to changes in environment dynamics, observation or action spaces (Pinto et al., 2017: Li et al., 2019; Tessler et al., 2019), but do not address reality gaps due to reward function mismatches, as they use inappropriate metrics on the space adversarial perturbations.
To solve this problem, ERMAS has three key features: 1) It formulates multi-agent robustness ob- jective equivalent to finding the worst case c-Nash equilibria. 2) It optimizes a tractable dual problem to the equivalent objective. 3) It approximates the dual problem using local solution concepts and first-order weta-learning techniques (Nichol et al., 2018; Finn et al., 2017). ERMAS ultimately yields policies that robust to other agents' behavioral deviations, up to regret of e.

---

Under review as a conference paper at ICLR 2021
We show that ERMAS learns robust policies in repeated bimatrix games by finding the worst-case reality gaps, corresponding to highly adversarial agents, which in turn leads to more robust planners. We further consider challenging, large-scale spatiotemporal economy that features social plan- ner that learns to adjust agent rewards. In both settings, we show policies trained by ERMAS are more robust by testing them in perturhed environments with agents that have optimized for reward functions unused during ERMAS training. This generalization error emulates the challenge faced in transferring policies the real world. In particular, we show ERMAS can find Al Economist policies that achieve higher social welfare across a broad range of agent risk aversion objectives, In all, we demonstrate ERMAS is effective even in settings where baselines fail or become intractable.
Contributions To summarize, our contributions are:
We derive multi-agent adversarial robustness problem using e-Nash equilibria, which poses a challenging nested optimization problem.
We describe how ERMAS efficiently solves the nested problem using dualization, trust- regions, and first-order meta-learning techniques.
We empirically validate ERMAS by training robust policies in two multi-agent problems: sequential bimatrix games and economic simulations. In particular, ERMAS scales to com- plex spatiotemporal multi-agent simulations.
ROBUSTNESS AND REALITY GAPS IN MULTI-AGENT ENVIRONMENTS
We seek to learn a policy Tp for an agent, termed the planner, that interacts with an environment featuring N other agents. The planner's objective depends both its policy and the behavior of other agents in response to that policy; this is multi-agent RL problem which the planner and agents co-adapt. In practice, evaluating (and optimizing) Tp requires use of simulation with agents that emulate those in the environment of interest (i.e. the real world), which might contain agents whose reward function differs from those used in the simulation. Our goal is to train planner policies that are robust to such reality gaps.
Formally, we build partially-observall 2018), defined by the tuple M == (S, T. o, o,), where S and A are the state and action spaces, respectively, and are agent indices. Since the MG played by the agents depends on the choice of planner policy, we denote the MG given by πp [Tp]]- MGs proceed in episodes that last H + steps (possibly infinite), covering H transitions. At each time E [0, H.. the world state is denoted 8,- Each agent = 1...., N receives an observation 0g executes an action di, and receives a reward ri,t. The environment transitions to the next state 8t-1, according to the transition distribution T (814-1 [8t, a.)." Each agent observes Ui,ts a part of the state 8t. Agent policies m, are parameterized by 0, while the planner policy Tp is parameterized by 0p.
multi-agent Markov Games (MGs) (Sutton & Barto,
The Nash equilibria of M(Tp) are agent policies where any unilateral deviation is suboptimal:
ANE(T,) == { Vi E [I, 7. II J. J, -_ mp) S J1 -. Tp).:
where JL(T, mp) = Ex,,, E/E 1f y(r(())
denotes the objective of agent i. Hence, a rational agent
would not unilaterally deviate from π E ANE(T,).
To evaluate fixed planner policy Tp. we simply sample outcomes using policies π ANE(Tp)- Also optimizing Tp introduces form of rwo-level learning. Under appropriate conditions, this can be solved with simultaneous gradient descent (Zheng et al., 2020; Fiez et al., 2019).
Robustness Objective As noted before, we wish to learn planner policies Tp that are robust to re- ality gaps arising from changes in agent reward functions, e.g., when agents are boundedly rational." We develop a robustness objective for the planner by formalizing such reality gaps perturbations
'Bold-faced quantities denote vectors or sets, c.g., (ai...., ax), the action profile for N agents. "This type of reality gap occurs when the simulated environment's 's reward function r fails to rationalize the actual behavior of the agents in the real environment, i.c., when agents in the real world act suboptimally with respect the simulation's 's reward function.
2

---

Under review as a conference paper at ICLR 2021
to agent objectives, where the uncertainty set E (S, A) I is the space of possible perturbations and represents uncertainty about the objectives of other agents. We extend ANE E) to condition the Nash equilibria perturbations &:
ATE E) = { E [ [1, f. E II: J$ (Mt, - Tp) S (T
(2)
J ( T-1 == JT T- TT) + B (T.(
(3)
where т; a trajectory (sequence of state-action pairs). Following Morimoto & Doya (2001), a ro- bust planner optimizes its reward, subject to agents playing perturbed Nash equilibrium ANE(TTT))))) {) that maximally penalizes the planner:
arg max min EEE
min TE ()
Jp(TT, Tp).
P
Note that agent policies π E ANF p.p {) describes agents that optimize their own reward function, and assume adversary chooses &.
Bounded Uncertainty Set There are two challenges with Equation 4. First, if the adversary can arbitrarily choose E. the worst case is uninformative." Second, depending on the complexity of II, the uncertainty set E may be high-dimensional and intractable to search. We address these issues by upper-bounding the size of the uncertainty set, T norm of & E, by the term Thus upper-bounds the difference between the reward functions of agents in the training and testing environments, c.g., between simulation and the real world. This bounded uncertainty set is:
5= sup (T, mp)l < e, for all E h,p
(5)
This uncertainty set is equivalent to the e-equilibria of M[mp]:
ANE pTp c) = { Vi [1, #. E IT: J. n-i, mp) S (Ti + +
(6)
e is a tunable hyperparameter this is the case with most robust RL (Pinto et al., 2017: Li et al., 2019) but good starting value is the anticipated error in reward objective estimates (application- specific). Using the robustness objective becomes the following constrained optimization prob- lem:
arg max Jr p.min (Tps c), Tp
where Jm ( (Tp, c):
min TEANE(Tp..)
Jp(
P Planeer-OPT
agent-Adv-Seaze
Using ANE(TT (Tpp <) replaces the problem of intractably searching through E with searching through ANE(Tp: c), and thus merges the two nested min operations in Equation 4. Conceptually, this trans- fers the worst-case search problem to the agents: Agent equilibrium; Planner- OPT optimizes the planner given adversarial agents. Note that the con- straint set in Eq 7 is non-empty for 2 O; the constraints (Eq 6) simply upper-bound the regret of agents. By definition, for non-empty bounded II, there exists an optimal policy with zero regret.
agents find an adversarial
ERMAS: ROBUST POLICIES IN MULTI-AGENT SIMULATIONS
We now introduce ERMAS, an efficient optimization framework to solve the robustness problem in Equation 7. ERMAS proceeds in three steps. First, it dualizes Equation 7 following constrained RL. Second, it defines a trust-region for the uncertainty set ANE(Tp, e), approximating the dual problem. Finally, it uses first-order meta-learning with trust-regions solve the approximate dual problem. See Appendix A.1 for the detailed Algorithm description.
Dualizing Agent -dont-Adv-Search The agent search problem in Equation can be formulated similar to a constrained RL problem (Paternain et al., 2019), where the primary objective of the agents is to minimize the planner's ss reward and the secondary objective to maximize their own reward. While conventional constrained RI enforces constant lower bound in the constraint, e.g.,
For instance, by setting & such that J -Jp-
3

---

Under review as a conference paper at ICLR 2021
Agent Payoff Matrix
Agent Payoff Matrix
Planner Payoff Matrix
Agent Position
Agent Position
Agent 1 action:
Agent action:
(a) Repeated Bimatrix Game
(b) Spatiotemporal Economic Simulation
Figure 1: ERMAS trains planners that are robust to reality gaps in these testbed multi-agent simula- tions. (Left) In the repeated bimatrix game, pair of agents navigates a 2D landscape. Both agents and the planner receive rewards based on visited coordinates. Brighter squares indicate higher pay- off. (Right) In the spatiotemporal economic simulation, 4 heterogenous agents perform labor, trade resources, earn income, and pay taxes according to the schedules set by the planner.
mp) 2 C. we enforce dynamic one: Vi 1...... JI 7p)) 2 Ji(m); --in p where Tे is the optimal unilateral deviation for agent i: := arg max;, En ( T-i; Tp).
Letting denote Lagrange multipliers, we can dualize Agent-Ae Adv--Sarrc-,S,, i.e., Equation 4, as:
min
Jp(
A
J(( Rp) Gs)
(8)
p.min (p.).)
This is identical to the dualization of constrained reinforcement learning, whose duality gap em- pirically negligible and provably zero under weak assumptions (Paternain et al., 2019). We now abuse notation to denote := (0......6 fr, 0pl and Jp(0) == Jp(T,, ((,,T,T xp) where 9 are the parameters of и. To solve Equation 8, the agents apply gradients:
........yy
Vo, Jf p.min (TD.e) = (,)) V ())) -.())
where (() is the parameters of the optimal unilateral deviation मं for agent i.e. the parameters that minimize local regret, which depends on the current policy parameters 0,, A, is updated as:
V Jt p.min (Tp: +) J+ --;, - Ji( (T, -4-
(10)
Equation 8 still poses challenge through the (, T-.. Tp) terms, which correspond unknown agent regret. We now detail the efficient approximation of the value and derivative of agent regret using local and meta-leaming approximations, respectively.
Trust Regions using Local c-equilibria Estimating regret requires knowledge of the optimal uni- lateral deviation for agent i. We can simplify this problem by proposing a refinement of e-equilibria inspired by the notion of local Nash equilibria in differentiable games (Ratliff al., 2014).
Definition 3.1. A strategy such that n; E W; and for each i E (1.....N) we have that JA(n(, - & J₁(T)) + é for all T/ W, \ (T;), where é' csup,,ew, KL(T;llm))
local e-Nash equilibrium if there exists open sets W; N
By instead performing Agent -ieen--Av-Searrde on the local e-Nash equilibria, we can limit the set of unilateral deviations to consider to small trust region, (T):
ANE(T mT 7) = { Vi E [1,N], #1 E (T, ( mp) ) J, T + +c, (T) = {nr E II| KL(=I(=)) < m))
(11)
(12)
4

---

Under review as a conference paper at ICLR 2021
Impact of Reward Slack
ERMAS vs Agent Regret
Results Using Fixed A
8000
Nash Equilbbrium
MARL
Agent Regret
ERMAS:
Planner Reward
ERMAS: rm-10
Planner Reward
A=1
4000
A=2
0.5
A=
2000
1-
20
0.3
-2000
More Agent Regret
0.2-
-2000
Bxoected Outcome
6590 7000 7500 6000 8500 5000 9530
-10
5
10 25
30
0000
7000
8000
5000
Combined Agent Reward
Reward Slack
Combined Agent Reward
in constrained repeated bimatrix games. Each point in the above scatter plots describes the average outcome the end of training for the agents (x-coordinate) and the planner (y-coordinate). Error bars indicate standard deviation. (Left) The bi- matrix reward structure encodes social dilemma featuring Nash equilibrium, which is where vanilla MARL converges. Agents trained with ERMAS deviate from this equilibrium in order reduce planner reward. e governs the extent of the allowable de- viation. (Middle) As increases, the average per-timestep regret experienced by the agents also increases. Each average is taken over the final 12 episodes after rewards have converged. (Right) Using fixed values of À (rather than allowing it to update, as in the full algorithm) distorts perfor- mance and prevents agents from reaching the same e-equilibria discovered with learned X.
Figure 2: Validating ERMAS Agent-Adv-Seed
wagent-teward/hight plannerr---er
where 17 0 defines the size of the trust region. For small 17, algorithms such as TRPO (Schul- man et al., 2017) can be used to efficiently approximate optimal local deviations of Tú affording reasonable approximations of Jj [m mp) Note that our usage of trust region algorithms is not for optimization purposes. ERMAS requires the use of trust region optimization to ensure that the equilibria considered by ERMAS are limited a local neighborhood of the policy space (Eq 11).
First-Order Meta Learning Approximation The full gradient in Equation 9 is also complicated by the need to estimate the derivative local regret Vo, (J,00(0),0 maximizes the performance of the agent's policy and is simply found with policy gradient. The first term is less straightforward: it minimizes the performance of the best agent policy in the current trust region. We note that this first term corresponds to meta-learning gradient. We follow REPTILE (Nichol et al., 2018) to obtain a first-order wwweerppppxxxiee approximation of a M-step meta-learning gradient:
J((O). The second term
M
i-1
Ve, Ve,J,((((),00))) = M
9:.
g. = Vo
0, +
2
9j.
1-..00.. 0g
(13)
where gi denotes the ith policy gradient in the direction of J;. In practice, we scale this meta-learning term with the hyperparameter /3 as B < incorporates helpful inductive bias where maximizing agent reward leads local maxima. We can alternatively apply this gradient update periodically, to both mimic B < 1 and reduce computation overhead. First-order meta-learning approximations are known to be empirically effective, and are necessary for ERMAS to efficiently solve Eq 8.
ERMAS By solving the dual problem (Eq. 8), ERMAS yields robustness e-equilibria and, cquivalently, uncertainty in agent objectives. ERMAS solves the dual problem by combining trust regions and meta-learning techniques to estimate and differentiate agent regret. Algorithm and (Appendix A.I) describe an efficient implementation of this procedure for nested policy learning.
4
EXPERIMENTAL VALIDATION IN MULTI-AGENT SIMULATIONS
4.1 ERMAS SOLVES THE INNER LOOP: AGENT-ADV-SEAN
We analyze the agent behaviors learned by Agent-Adv- in experiments depicted in Figure 2. These experiments apply ERMAS to the classic repeated bimatrix game, which is well-studied in the game theory literature as its Nash equilibria can be solved efficiently. At each timestep,
Constrained Repeated Bimatrix Game
player (Agent and column player
5

---

Under review as a conference paper at ICLR 2021
Beta
Beta 0.2
Beta
6003
Planner Reward
Agent 1 Reward
Agent 2 Reward
4500
3800
4000
1000
3000
1500
-1003
2400
2000
-200
3200
Training Episodes
Figure 3: Planner and agent rewards in the repeated bimatrix game over training time. Each line represents an average over 10 seeds, with error bars indicating standard error. The lines correspond to runs of ERMAS and are colored according 3, the weight of the meta-learning term of Equation 13. (Left) Planner performance over training episodes. (Middle) Agent performance over training episodes. (Right) Agent performance over training episodes.
(Agent 2) simultaneously choose from finite number of actions. Each pair of actions (t,j) cor- responds to a pair of payoffs for the agents ri (i..)).r(i..)) We select the payoff matrices ri and r2, illustrated in Figure la, so that only one Nash equilibrium exists and that the equilibrium consti- tutes a mragedy-of-thhe-ommee
where agents selfishly optimizing their own reward leads less reward overall. To extend this repeated bimatrix game into sequential decision making, we further constrain the game so that, at any timestep, agents can only choose an action adjacent to their action in the previous timestep. We also introduce a passive planner that observes the game and receives a payoff rp (1. The planner does not take any actions and its payoff is constructed such that its reward is high when the agents are the Nash equilibrium. In effect, this toy setting allows us to verify that ERMAS samples realistic worst-case behaviors-that is, that Agents and learn to equilibrium in order to reduce the planner's reward but also without significantly increasing their own regret.
deviate from their tragedy-of-commn
Discovering e-Equilibria Figure (left) visualizes how the equilibria reached by Al agents bal- ance the reward of the planner (y-axis) and agents (x-axis). Conventional multi-agent RL discovers the Nash equilibrium, which visualized in the top left. At this equilibrium, the agents do not cooperate and the planner receives high reward. For small values of e, ERMAS also discovers the Nash equilibrium. Because acts as a constraint on agent regret, larger values of enable ERMAS to deviate farther from the Nash equilibrium, discovering e-equilibria the bottom-right that result in lower planner rewards. Deviations from a Nash equilibrium are associated with higher regret, meaning that regret should increase with e. Figure 2 (middle) clearly demonstrates that ERMAS im- parts this trend. As described by Equation 7, this display of adversarial behavior is key to learning more robust planner.
4.2 ALL THE COMPONENTS OF ERMAS ARE NECESSARY
Dynamic or Frozen Lagrange multipliers A, The Lagrange multipliers À balance the dual ob- jectives of seeking stable agent equílibria and minimizing the planner's reward. Recall that smaller values of X mean agent objectives are more antagonistic. The X are updated using local estimates of agent regret, as described in Equation We can validate that these updates are necessary by analyz- ing the equilibria learned by ERMAS when À are not updated. Fixing value of À reduces Equation to learning with shared rewards. Figure 2 (right) visualizes the equilibria discovered with frozen A in the same format as the Figure 2 (left) plot. This visualization shows that freezing À affects the equilibria discovered by ERMAS; the bottom right quadrant which contains the e-equilibria discov- ered with dynamic À are not reached for any values of X. This validates that certain e-equilibria are only reachable with dynamic and hence the updates in are necessary for proper behavior.*
However, we recommend temporarily freezing at the top of Algorithm 2 "warm up" agent policies.

---

Under review as a conference paper at ICLR 2021
Abolute Change vs: MARL
8000
ERMAS
3000
Planner Reward
MARL
(Q-0.5)
DR
4000
Rik Robustress
0
2000
1000
-2000
3000
-4000
0.5 0.6
0.7
Test Env Reward Perturbation Coefficient,
Test Env Reward Perturbation Coefficient,
Figure 4: Robust planner learning in the repeated bimatrix game. Average planner performance as function of the reward perturbation coefficient Q in the test environment. Each point represents an average over seeds, with error bars indicating standard error. Vertical dashed lines denote the training environment Q. (Left) Average planner performance convergence. (Right) Performance relative to the vanilla MARL baseline, which is trained without a robustness objective.
First-order Approximation Using the first order approximation of Equation 13 might slow learn- ing, since we step "away" from the direction that the standard policy gradient suggests. In contrast, find that the ERMAS update can significantly speed up convergence. Figure depicts the learn- ing curve of variants of ERMAS with varying weights, 3, on the meta-learning term. For instance, when 0, the meta-learning term is completely dropped. We find that lines corresponding to larger values of 3 reach improved agent rewards half an order of magnitude faster than lower values of B. However, larger values of 3 fail reach the desirable region of adversarial e-equilibria. In both cases, very large (3 > n) and very small (3 0) fail to fully reach and remain at the adversarial region. However, an appropriate range for 3 gives rapid convergence and good asymptotic behavior.
4.3 SOLVING PLANNER-OP1 USING
We now show that ERMAS yields planners Tp which are more robust to uncertainty in the agent's reward function. Specifically, we empirically validate that ERMAS can find strong solutions to the nested optimization problems Agent-A.ee -aennttAv---aarrraa and P P aanner-OPT,, in two simulations: repeated bimatrix games and economic simulations (see Figure 1).
Evaluation Workflow To measure planner robustness, we evaluate trained planners in test envi- ronments with reality gap (i.e. reward parameterizations that differ from the training environment), containing test agents that are optimized for the test environment's reward functions and that have been seen by the planner. We proceed as follows: 1) Train the agents and planner with ERMAS. 2) Fix the planner and transfer it to a test environment, then train new agents from scratch in the test environment with reward functions unseen by the planner. 3) Report the value of the planner objective after the new agents have converged.
Augmented Repeated Bimatrix Games We now extend the bimatrix game nested reinforce- ment learning problem by allowing the planner to take actions and influence its own experienced re- ward. The planner now selects an integer a 2 O that modifies its new reward function: arp -2 This function is chosen for its quadratic form: scales the reward but adds a cost a² which disincen- tives large values of a. Agents are modified to receive a new reward function r5 r; Qr, where an unknown scalar which may differ between training and testing environments. If Q = 0, agents act identical to the experiments in Figure 2. For environments with larger values of Q, e.g., Q = 0.5, agents have an incentive to be adversarial.
r, is their original reward function and Q <
Figure 4 shows the test performance of ERMAS against vanilla multi-agent RL policies learned with Quain = 0.5 or with Quain O. Naturally, MARL trained with Quain = 0.5 underperforms MARL trained with Qmiin
when evaluated in environments where Ques 0.3. However, MARL trained ch successfully finds the
with Quain = 0.5 outperforms when Quet > 0.3. -ddo-ttAdvSeeee worst-case perturbation, and yields a planner policy least good as MARL Quain 0.5. This shows that ERMAS can produce policies which are robust to uncertainty in agents' objectives and is tractable adversarial robustness algorithm.
7

---

Under review as a conference paper at ICLR 2021
Existing single-agent and multi-agent robust reinforcement learning algorithms optimize for differ- ent robustness objectives than ERMAS. However, some general techniques for addressing Sim2Real can be extended provide alternative to ERMAS's ' adversarial approach. Our first baseline extends the technique of domain randomization (DR) by applying random perturbations to agent rewards. For DR, agents receive ry r; + o, where
U[-1,1] (recall r; E [0, 1). These pertur- bations are randomized periodically between episodes. Even in a simple 4-by-4 bimatrix game, there are 2304- 16 states 16 states 9 actions possible reward values to perturb; 2304-dimensional space is too sparse to cover uniformly. Furthermore, in contrast to DR of visual inputs or dynamics, there is natural latency the effect of DR of agent rewards: randomization only has an effect agents learn to adapt to their perturbed rewards. Our second baseline, risk robustness (RR), applies a concavity to planner rewards, e.g., logrp. This encourages robustness to environment stochas- ticity, including the randomness of agent policies. As illustrated in Figure 4, neither the domain randomization nor risk robustness baseline yields meaningful robustness even in this simple setting.
4.4 ERMAS FOR LARGE-SCALE TWO-LEVEL LEARNING: AI ECONOMIST
We now address important use-case, tax policy design, to demonstrate ERMAS at scale. We train a robust AI Economist: planner for optimal taxation that acts in a spatiotemporal economic simulation, see Zheng et al. (2020) for detailed description. This setting is very challenging given the scale of the simulation and agent affordances. In particular, adversarial and randomization baselines do not scale to this setting. In this simulation, agents optimize their expected utility r where the utility ri, is function of labor 4,, and post-tax endowments Zi,ti
Ex,
=1-7 -
Fie = Zz,1 -(z,,)) F.. Σ 2.,.
1 > O,
(14)
where 77 sets the degree of risk aversion (higher 7) means higher risk aversion). Agents carn income Zie and pay taxes 7 (2i,)), which are set by the planner. The planner optimizes social welfare swf:
N swf eq(x) prod(r), eq(x) = N gini(())) prod(x) =
Σ Tb
(15)
a combination of equality (Gini, 1912) and productivity.
Figure 5 shows the performance of ERMAS tax policies along with baseline tax policies (Saez, US Federal) and an Al Economist (MARL). All models are trained on agents with 1j = 0.23. We replicate Zheng et al. (2020): MARL outperforms the Saez tax. However, MARL fails to outperform Saez when testing with 7) < 0.22 and ry > 0.28. contrast, ERMAS yields consistent gains across the depicted range of 1/. This shows that ERMAS's tax policy can improve swf even if agent risk aversion significantly increases or decreases (c.g., due to economic cycles or exogenous shocks).
In fact, ERMAS outperforms the baseline AI Economist for the original setting of 17 0.23. This suggests the robustness and performance do not necessarily pose a zero-sum game: ERMAS can find equilibria with high performance and strong generalization. It has been observed empirically in various studies that single-agent robust reinforcement learning similarly yields performance im- provements even in the absence of environment perturbations (Pinto et al., 2017).
RELATED WORK
Sim2real: Reality Gaps The Sim2Real problem considers how reality gaps, i.e., mismatches between a simulation and reality, can (negatively) impact RL policies. Robust performance across reality gaps has been studied when applying deep RL to robotic control with visual inputs (James et al., 2019; Christiano et al., 2016; Rusu et al., 2016: Tzeng et al., 2015). To close reality gaps, domain adaptation aims to transfer simulated distribution (of the world) to the real world (Higgins et al., 2017; Kim et al., 2019). In comparison, the study of reality gaps in multi-agent RL is relatively nascent (Suh et al., 2019; Nachum et al., 2019).
Multi-Agent Robustness Morimoto & Doya (2001) proposed robust RL for the worst-case ad- versarial objective (i.e., max-min optimization problem), inspired by Hoo control, and provided
8

---

Under review as a conference paper at ICLR 2021
Planner Reward
500
ERMMAS US Federal
% Change us. MARL
20
Sacc
10
200
100
0.21
0.22
0.23
0.24
0.25
0.26
0.27
0.28
0.29
0.21
0.22
0.23
0.24
0.25
0.26
0.27
0.28
0.29
Test Env Risk Aversion, 0
Test Env Risk Aversion, n
Figure 5: Learning robust optimal taxation in economic simulation. Results plotted following the same conventions in Figure 4. Due to the intractability of other robustness algorithms in this setting, we instead compare against different planner choices. "US Federal" uses fixed tax scheme adapted from the 2018 US Federal income tax rates. "Saez" uses an adaptive, theoretical formula to estimate optimal tax rates.
analytical solutions in the linear setting. The robust RL problem be solved using adversarial player that chooses perturbations. Rather than the worst-case, Pinto et al. (2017) optimized for the a-quantile of reward, using conditional value-at-risk (CVAR) interpretation of adversarial robust- ness. We consider perturbations to agent rewards. Other works have studied perturbing transition matrices, observation spaces, and action probabilities, which are relevant to robotics and control ap- plications (Pinto et al., 2017; Tessler et al., 2019; Hou et al., 2020; Li et al., 2019). We also address a more challenging multi-agent setting than previous works. First, agents do not necessarily optimize for the same (robustness) objective. We therefore consider a one-sided robustness problem where only the planner is expected works that assume all agents act robustly (Li et al., 2019). Second, we address challenging nested reinforcement learning setting of interest to important real- weal-wordd applications such as economies mechanism design (Zheng et al., 2020). Although we formulate ERMAS as using simultaneous gra- dient descent, ERMAS can be easily extended to use nested optimization solvers such as competitive gradient descent (Schäfer& & Anandkumar, 2020).
robustly. This setting is significantly more difficult than previous
See Appendix A.2 for additional related work.
FUTURE WORK
Future work could extend ERMAS to reality gaps due to differrnnt causes, such changes in transi- tion dynamics, agent affordances, number of agents, etc, which might require more refined approxi- mations the planner's 's robustness objective. Furthermore, it would be interesting apply ERMAS to problems that feature more general solution concepts, e.g., Bayesian Nash equilibria, correlated equilibria, Walrassian equilibria, and others.
REFERENCES
Dirk Bergemann and Stephen Morris. Robust mechanism design. Econometrica, 136):1771-1813,
2005. ISSN 00129682, 14680262. URL http: / /www. jstor. wttp//ttaae//39811.5.
Eric Bonabeau. Agent-based modeling: Methods and techniques for simulating human systems. Proceedings of the National Academy of Sciences, 99(uppl 3):7280-7287, May 2002. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas 082080899. 1...177/pnas.08208889999 URL https: : www.ppaaoooooo Publisher: National Academy of Sciences Section: Collo-
content/ tttonn/////eeee h099/supl..0 3/tppl33/78000 3/7280. quium Paper.
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojeiech Zaremba. Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model. October 2016. URL https: //arxiv///xiii////aa/ wwtp://gggaabbbbbbe.ee.s abs//61s..11

---

Under review as a conference paper at ICLR 2021
Paul Dütting, Zhe Feng, Harikrishna Narasimhan, David Parkes, and Sai Srivatsa Ravindranath. Optimal auctions through deep learning. In International Conference on Machine Learning, pp- 1706-1715, 2019.
Tanner Fiez, Benjamin Chasnov, and Lillian J. Ratliff. Convergence of Learning Dynamics in Stack- elberg Games. arXiv. jun 2019. URL http: /ttp:://axxiiii wwwpxii.oog/aa.....eo 01217.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adapta- tion of Deep Networks. urXiv:1705 703.03400 [cs], July 2017. URL
arXiv: 1703.03400.
C. Gini Variabilità e mutabilità. 1912.
Irina Higgins, Arka Pal, Andrei A Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. arXiv preprint arXiv: 717770077.8555 2017.
John H. Holland and John H. Miller. Artificial Adaptive Agents in Economic Theory. American Eco- ht /ttts////////pppeeeeeeeeee repec aeaaecrev/ 3a81 _a81__ay____ 3a1 991 3ai 3a2 3ap 3a365- 71 htm. Publisher: Amer-
nomic Review, (22):365-,,, 1991.
ican Economic Association
Linfang Hou, Liang Pang, Xin Hong, Yanyan Lan, Zhiming Ma, and Dawei Yin. Robust Rein- forcement Learning with Wasserstein Constraint. 17Xiv:2006.0094 [cs, stat), June 2020. URL http:// /tttp://aaxvv.../// org/ abs/ 00945.
arXiv: 2006.00945.
Peter Howitt. What have central bankers learned from modern macroeconomic theory? Jour- nal of Mucroeconomiiss 34(1):11-22, March 2012. ISSN 0164-0704. doi: 10.1016/j. waacrr.2011.08.0000 URL http://wwwwciicceireeeee.e. wwt.///ciiin///artellll..c
pp11/0014400411000650
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data- efficient robotic grasping via randomized-to-caoonii adaptation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12627-12637, 2019.
Kun Ho Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Cross Domain Im- itation Learning. arXiv:1910.00105 [cs, stat], September 2019. URL htttp://arxivvort abs/ /1910. 00105,
arXiv: 1910.00105.
Alan P. Kirman. Whom What Does the Representative Individual Represent? Journal of Eco- nomic Perspectives, 6(2):1 June 1992. ISSN 0895-3309. doi: 10.1257/jep.6...111 URL https: http://wwwwww.. ttwp:/wwwwwwbbbbb///////// wttp://ttg/eteteeeeeeoor.oe
1257/ j.p. 6.2.1....11 117.
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang. and Stuart Russell. Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient. Proceedings of the AAAI Conference on Artificial Intelligence, 33:4213 July 2019. doi: 10. 10.1609/aaai.v335
33014213.
Jun Morimoto and Kenji Doya. Robust Reinforcement Learning. In T. K. Leen, T. G. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Sys- tems 13, pp. 1061-1067. MIT Press, 2001. URL http: httt:://papprs.ntte nips. ct/ccc///aaa cc/paper/ 1:41-robust reinforcement r pdf..
Roger Myerson. Mechanism Design, pp. 1-13. Palgrave Macmillan UK, London, 2016. ISBN 978-1-349-95121-5. doi: 10. 113577778-1-44-9522221 10.1057/978-1-23
URL https: http:://ddd /ttp::://///rr/rr//
Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. Multi-Agent Manipula- tion via Locomotion using Hierarchical Sim2Real. arXiv: 05224 [cs], October 2019. URL http::/arx: wrw.orrgg/.......ooo 908. 05224. arXiv: 1908.05224.
10

---

Under review as a conference paper at ICLR 2021
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms. /
arXiv: 02999 [cs]. October 2018. URL arXiv: 1803.02999.
Santiago Paternain, Luiz F. O. Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained Reinforcement Learning Has Zero Duality Gap. arXiv:1910.13392 [cs, math, stat], October 2019. URL http://arxix/ org/ abs/ 19100 13393. arXiv: 1910.13392
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-Real Trans- fer of Robotic Control with Dynamics Randomization. 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 3803-3810, May 2018. doi: 10.1109/CCR.22088.5544 8460558... arXiv: 1710.06537.
URL http:p:///axii..oo
abs 1 06537.
Wolfgang Pesendorfer. Behavioral economics comes of age: A review essay on advances in behav-
ioral economics. Journal of Economic Literature, 44(3)712-721, 2006.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust Adversarial Rein- forcement Learning. vrXiv:/703.. 02702 [cs], March 2017. URL hhtt::///aaaiii wwtpxiiiorrg/ssss/oooooooo arXiv: 1703.02702.
Lillian J. Ratliff, Samuel A. Burden, and S. Shankar Sastry. On the Characterization Local
Nash Equilibria in Continuous Games. arXiv:1411.2166 [math], November 2014. URL http:
//arxiv. wwtp/oog////baaaa/ee /11b..////11... .2 68. arXiv: 1411.2168.
Andrei A. Rusu, Mel Vecerik, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-Real Robot Learning from Pixels with Progressive Nets. October 2016. URL https: //a www.iiiorrgg....o..o.ooooooooooe abs/ / 0428448886 6v2.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust Region Policy Optimization. arXiv:1502,00 05477 [cs], April 2017. URL 1502. 05477. arXiv: 1502.05477.
Florian Schäfer and Anima Anandkumar. Competitive Gradient Descent. vrXiv:1905.12100 [cs, math June 2020. URL http: htttp://arxiv.oot abs 905 12103. arXiv: 1905.12103.
Herbert A. Simon. From substantive to procedural rationality. In T. J. Kastelein, S. K. Kuipers, W. A. Nijenhuis, and G. R. Wagenaar (eds.), 25 Years of Economic Theory: Retrospect and prospect, pp. 65-86. Springer US, Boston, MA, 1976. ISBN 978-1-4613-4367-1 doi: 1001007/ 778-1-4613-4367-1 URL https: //ooi. wac.eooi..rrr/// /ttp://.........0000/111116661-1
Herbert A. Simon and Jonathan Schaeffer. The Game of Chess. Technical report, CARNEGIE- MELLON UNIV PITTSBURGH PA ARTIFICIAL INTELLIGENCE AND PSYCHOL- OGY PROJECT, December 1990. URI. https://apps: dtic. mit. httt://tattttiss/////// ADA225613. Section: Technical Reports.
Young-Ho Suh, Sung-Pil Woo, Hyunhak Kim, and Dong-Hwan Park. A sim2real framework en- abling decentralized agents to execute MADDPG tasks. In Proceedings of the Workshop on Distributed Infrastructures for Deep Learning. DIDL 19, pp. 1-6, Davis, CA, USA, December 2019. Association for Computing Machinery. ISBN 978-1-4503-7037-- doi: 10. 10.1145/3366622. 3368146 URL https: meri..rg/100 httgg//...11//366680 3368146.
Richard Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action Robust Reinforcement Learning and Applications in Continuous Control. arXiv: 19. [cs, stat), May 2019. URL http://
www.oor.//abba.o..oooo..ooo... 1901 09184. arXiv: 1901.09184
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main Randomization for Transferring Deep Neural Networks from Simulation to the Real World. arXiv:1703.0690 077077000000777710 [cs]. March 2017. URL http: : arriiv. org/ abs awwaass17 /1703 06907. arXiv: 1703.06907.
11

---

Under review as a conference paper at ICLR 2021
Eric Tzeng, Coline Devin, Judy Hoffmann Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, and Trevor Darrell. Adapting Deep Visuomotor Representations with Weak Pairwise Constraints. November 2015. URL https: /wwpivawivvvv/aaa/////....//.av http://ss/aa.........seeeoe http:/v..rggbb//////// 07111v5.
Stephan Zheng, Alexander Trott, Sunil Srinivasa, Nikhil Naik, Melvin Gruesbeck, David C. Parkes, and Richard Socher. The Al Economist: Improving Equality and Productivity with Al-Driven Tax Policies. arXiv:2004.1332. [es, econ, q-fin, stat), April 2020. URL http: //axxvv. org/abs/2004 13332.
arXiv: 2004.13333 13332.
12


Reviews:
[{"review_id": "ahC5kuaM5Ro", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper tackles robust RL under the multi-agent setting. They formulate the multi-agent adversarial robustness problem as a nested optimization problem and propose a practical algorithm (ERMAS) to solve it. Theoretical proof and empirical study on two environments are provided to demonstrate the effectiveness of the proposed framework.\n\n##########################################################################\n\npros: \n\n+ The paper is well written with clear and interesting motivation. Bridging the gap between simulation and reality for multi-agent environments is an important topic. Although I'm not an expert in this direction, it feels to me the proposed framework is technically and theoretically sound with comprehensive empirical evaluations.\n\n##########################################################################\n \ncons: \n\n- It would be better to include more baselines from the MARL field, especially the ones you cited in related work that also considers robust MARL. Only a vanilla MARL baseline cannot actually tell how much you improve empirically from SOTA.\n- It would be better to include literature from the nested optimization field in the related work. Since you formulate the robust MARL as a nested optimization problem, have you considered or compared ERMAS with any other existing nested optimization solvers?\n\n\n##########################################################################\n\nPost Rebuttal\n\nThe paper has been updated to include additional reviews about nested optimization. I would like to keep my original score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "W40J-v6V6BD", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting method for being able to act and plan robustly in a multiagent simulation and be robust to the reality gap between training time and testing time for agents in a marl setting. the method does show improvements in terms of being able to train the policy for this use case and being more robust to some out of distribution configuration of the environment however these improvements appear to be rather limited. in addition, the organization and writing for the paper is very technical and could be improved with additional background information on the uses of metrics and environments as well as better flow between the content in the paper to understand the importance of the different aspects of the method. These improvements could help the reader understand the novelty and important aspects of the method that are difficult to measure. At the moment it comes across as a mix of different methods combined to be able to support this more robust method without a very clear story about the primary problem the paper is trying to solve or the more significant technical aspect of the method that provides this novel solution.\n\nThe paper appears to be very technical. The impact of this paper on a more general audience is going to be limited due to the number of details left out of the paper that would help motivate the reasoning for using certain metrics for learning why you would want bounded regret and Nash equilibria. For example, in figure 2 all of the plots have different quantities on the x and y-axis that are only briefly described and not very well motivated why these are good metrics for analyzing this type of algorithm. The writing really needs to include more motivation to clearly describe why these environments and metrics are the right kind of analysis to perform in order to indicate the efficacy of the method.\n\nFor example they bi Matrix game used at the beginning of the experiment section has no explanation. How does this game work how can we understand whether or not the method is improving in this game without understanding the difficulties and or nuanced importance of analyzing this environment.\n\nIn the augmented repeated bi Matrix game scenario that is used in test environment that is a modified version of the bi matrix game in order to show the robustness of the method there, is again, not a clear description about why these environment changes would be significant. Overall since little is understood about this game one can only assume that these particular changes to the environment could be selectively biased in order to favor the method that's proposed in the paper and overall the analysis of this section is unclear.\n\nFigure four does not appear to be referenced in the paper anywhere I'm going to assume that it should be referenced inside of the augmented repeated by metrics game section and if that is the case the marl baseline is not very well explained for this figure before it is used in figure 4.\n\nAdditional Comments:\n- The list of contributions at the end of the introduction seems to be fairly repetitive from the content in the last paragraph in the introduction.\n- The part the connects the work to TRPO is not discussed carefully paper. Is the method just using TRPO for multi-agent RL?\n- In figure 3 we will assume that the ordering in the text is actually left center-right.\n\n\n----- Post Discussion ----\nUpdates to the paper have helped make technical parts of the paper more clear. The paper has also been edited to improve the motivation and experimental explanation.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Pzos_xIApLT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to learn _robust_ policies in multiagent environments: in particular, the policies should continue to work even if other agents in the environment deviate slightly in their behavior. Unlike previous work which allows for perturbations in other agents’ policies, this work allows for perturbations in the reward functions that those agents optimize, allowing for more relevant robustness.\n\nOverall I liked the work -- robustness to what the agents optimize seems quite important and (to my limited knowledge) novel. The experiments seem reasonable and establish that the method (ERMAS) works well. I note a few qualms below.\n\n----\n\nOne worry about the theory is that it is not well-defined because it allows for arbitrary exogenous perturbations, as in this line:\n\n> The robustness objective in Equation 4 considers general agent perturbations, e.g., agents may exhibit an exogenous aversion to complex policy choices.\n\nWith this, it is no longer necessarily the case that a Nash equilibrium exists: Nash equilibria are only guaranteed to exist when the utility functions are of the _game outcomes_; they need not exist when the utility functions can also apply to the _chosen policies_.\n\nFor example, consider rock-paper-scissors between Alice and Bob. Wins get +1 utility, losses get -1, and ties get 0. Now suppose we add the perturbation “-100 if you play a stochastic policy” (normally not allowed, but allowed in this paper’s formalism as an “exogenous preference”). It is easy to see that there is no Nash equilibrium where either player plays a stochastic policy, as they could then switch to a deterministic policy. But we know there is no Nash equilibrium for rock-paper-scissors under deterministic policies.\n\nSince Nash equilibriums are no longer guaranteed to exist, the objective in (4) may be undefined in some settings.\n\nThis could be fixed by restricting perturbations to only affect utilities of states, though this would significantly decrease the expressive power of perturbations.\n\n----\n\nI was confused by the fact that ERMAS outperformed MARL even when there is no change in agent policies. The authors note this fact as well:\n\n> In fact, ERMAS outperforms the baseline AI Economist for the original setting of η = 0.23. This\nsuggests the robustness and performance do not necessarily pose a zero-sum game: ERMAS can find equilibria with high performance and strong generalization.\n\nI am not sure I share the authors’ enthusiasm. It is straightforward to show that the optimal policy in the MARL setting does at least as well as the optimal policy in the robust setting. The fact that ERMAS actually outperforms MARL is surprising and suggests that ERMAS optimizes better than MARL for some reason. This could happen in one of two ways:\n\n1. Something about ERMAS leads to better learning, in a way that can’t apply to MARL (e.g. perhaps by having adversaries learning progresses faster since any flaws are found more quickly)\n2. Something about ERMAS leads to better learning, in a way that could apply to MARL (e.g. better hyperparameter tuning).\n\nIf it’s the second case (especially hyperparameter tuning), then it calls into question whether any of the improvements in the experiments come from the design of ERMAS, rather than coming from something unrelated like better hyperparameter tuning.\n\n----\n\nQuality: Decent. I liked the algorithm derivation (though I did not carefully check the math), and the evaluation does show the benefits of the approach, though it would have been nice to test the approach on more environments (there is just one toy environment and one more complex environment). In particular, as far as I can tell the algorithm can be applied to any multiagent RL (MARL) problem; there are several benchmark suites on which this could be tested.\n\nClarity: I found the paper to be quite clear.\n\nOriginality: I believe this is a novel formulation, though I am not very familiar with the robust RL literature.\n\nSignificance: Clearly relevant and important.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Odgj86dEWyG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " \n##########################################################################\n\nSummary:\nThis paper purposes a robust optimization framework for training robust policies, such that policies learned in imperfect\nsimulations can be robust against reality gaps in order to be effective in the real world. The multi-agent adversarial robustness problem is derived using \u000f\\eqsilon-Nash equilibria; the paper purposes ERMAS, and present how ERMAS solves the nested robust optimization problem using dualization, trustregions, and first-order meta-learning techniques; the authors  empirically validate ERMAS by training robust policies in two multi-agent problems: sequential bimatrix games and economic simulations.\n\n##########################################################################\n\nReasons for score: \n\n1. Overall I think the reality gap is a very important problem, and the paper is well organized and presented clearly. The robust optimization formulation provides a way to approach this problem. I have several concerns about this formulation (stated below)\n\n2. The robust optimization formulation requires an input of the size of the uncertainty set. However I did not find discussion in the paper on how such a bound on the uncertainty set can be obtained or estimated in practice. If such a bound is loose, the robust optimization approach might become overly conservative. Following on this conservativeness concern, ERMAS learns the robust policies by finding the worst-cas reality gaps, which corresponding to highly adversarial agents --so a small fraction of highly adversarial agents could be sufficient make the uncertainty set size large.\n\n3. I would like to see more discussions on the approximate feasibility and optimality for solving the robust optimization problem eq(7). \n\n4. In the experiments (figure 2), it is unclear that if the average per-time step regret experienced by the agents increases as the reward slack \\epsilon increases (the trend becomes decreasing as \\epsilon is large)\n\n \n##########################################################################\n\nQuestions during rebuttal period: \n\n \nPlease address and clarify the cons above \n\n \n#########################################################################\n\n--post discussion-- \nThe authors' response are helpful in addressing the concerns in the original review. I decided to update the score to 6.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a multi-agent RL algorithm where the rewards of the other agents are only known up to some accuracy. The setting is somewhat restrictive, in the sense that the transition is assumed to be known. It would perhaps have been more interesting for the paper to also consider unknown transitions, so as to bring it closer to work in single-agent reinforcement learning. It also seems to not be making a very good job of linking the related work to the contribution of this paper (even after looking at the appendix).

- Authors briefly say in the introduction 
"Alternative frameworks improve robustness, e.g., to changes in environment dynamics, observation or action spaces (Pinto et al., 2017; Li et al., 2019; Tessler et al., 2019), but do not address reality gaps due to reward function mismatches, as they use inappropriate metrics on the space of adversarial perturbations"

Authors should try and better explain the differences with those papers. Do  they consider changes in dynamics rather than the reward? It appears that the former is more general than the latter. Couldn't the authors compare with them with an appropriate experiment?

It is also hard to see how this exactly connects with a reality gap. What is the 'training' environment? What is the 'testing' environment? This is simply a robust optimisation algorithm applied to multi-agent games with partially unknown reward functions. 

In addition the experiments themselves are not explained clearly. 

On the plus side, I think the algorithmic details and experimental are interesting. If there was a better explanation and discussion/comparison with related work, then it would have been a good paper. Authors are encouraged to make a stronger effort to compare with other methods both in terms of the algorithm and experimentally.
Decision: Reject