ERMAS: Learning Policies Robust to Reality Gaps in Multi-Agent Simulations

1 INTRODUCTION . Reinforcement learning ( RL ) offers a tool to optimize policy decisions affecting complex , multiagent systems ; for example , to improve traffic flow or economic productivity . In practice , the need for efficient policy evaluation necessitates training on simulations of multi-agent systems ( MAS ) . Agents in these systems can be emulated with fixed behavioral rules , or by optimizing for a reward function using RL ( Zheng et al. , 2020 ) . For instance , the impact of economic policy decisions are often estimated with agent-based models ( Holland & Miller , 1991 ; Bonabeau , 2002 ) . This commonly introduces a reality gap as the reward function and resulting behavior of simulated agents might differ from those of real people ( Simon & Schaeffer , 1990 ) . This becomes especially problematic as the complexity of the simulation grows , for example , when increasing the number of agents , or adding agent affordances ( Kirman , 1992 ; Howitt , 2012 ) . As a result , policies learned in imperfect simulations need to be robust against reality gaps in order to be effective in the real world . We introduce -Robust Multi-Agent Simulation ( ERMAS ) , a robust optimization framework for training robust policies , termed planners , that interact with real-world multi-agent systems . ERMAS trains robust planners by simulating multi-agent systems with RL and sampling worst-case behaviors from the worst-case agents . This form of multi-agent robustness poses a very challenging multilevel ( e.g. , max-min-min ) optimization problem . Existing techniques which could be applied to ERMAS ’ s multi-agent robustness objective , e.g. , naive adversarial robustness ( Pinto et al. , 2017 ) and domain randomization ( Tobin et al. , 2017 ; Peng et al. , 2018 ) , are intractable as they would require an expensive search through a large space of agent reward functions . Alternative frameworks improve robustness , e.g. , to changes in environment dynamics , observation or action spaces ( Pinto et al. , 2017 ; Li et al. , 2019 ; Tessler et al. , 2019 ) , but do not address reality gaps due to reward function mismatches , as they use inappropriate metrics on the space of adversarial perturbations . To solve this problem , ERMAS has three key features : 1 ) It formulates a multi-agent robustness objective equivalent to finding the worst case -Nash equilibria . 2 ) It optimizes a tractable dual problem to the equivalent objective . 3 ) It approximates the dual problem using local solution concepts and first-order meta-learning techniques ( Nichol et al. , 2018 ; Finn et al. , 2017 ) . ERMAS ultimately yields policies that are robust to other agents ’ behavioral deviations , up to a regret of . We show that ERMAS learns robust policies in repeated bimatrix games by finding the worst-case reality gaps , corresponding to highly adversarial agents , which in turn leads to more robust planners . We further consider a challenging , large-scale spatiotemporal economy that features a social planner that learns to adjust agent rewards . In both settings , we show policies trained by ERMAS are more robust by testing them in perturbed environments with agents that have optimized for reward functions unused during ERMAS training . This generalization error emulates the challenge faced in transferring policies to the real world . In particular , we show ERMAS can find AI Economist tax policies that achieve higher social welfare across a broad range of agent risk aversion objectives . In all , we demonstrate ERMAS is effective even in settings where baselines fail or become intractable . Contributions To summarize , our contributions are : • We derive a multi-agent adversarial robustness problem using -Nash equilibria , which poses a challenging nested optimization problem . • We describe how ERMAS efficiently solves the nested problem using dualization , trust- regions , and first-order meta-learning techniques . • We empirically validate ERMAS by training robust policies in two multi-agent problems : sequential bimatrix games and economic simulations . In particular , ERMAS scales to complex spatiotemporal multi-agent simulations . 2 ROBUSTNESS AND REALITY GAPS IN MULTI-AGENT ENVIRONMENTS . We seek to learn a policy πp for an agent , termed the planner , that interacts with an environment featuring N other agents . The planner ’ s objective depends both on its own policy and the behavior of other agents in response to that policy ; this is a multi-agent RL problem in which the planner and agents co-adapt . In practice , evaluating ( and optimizing ) πp requires use of a simulation with agents that emulate those in the environment of interest ( i.e . the real world ) , which might contain agents whose reward function differs from those used in the simulation . Our goal is to train planner policies that are robust to such reality gaps . Formally , we build on partially-observable multi-agent Markov Games ( MGs ) ( Sutton & Barto , 2018 ) , defined by the tuple M : = ( S , A , r , T , γ , o , I ) , where S and A are the state and action spaces , respectively , and I are agent indices . Since the MG played by the agents depends on the choice of planner policy , we denote the MG given by πp as M [ πp ] . MGs proceed in episodes that last H + 1 steps ( possibly infinite ) , covering H transitions . At each time t ∈ [ 0 , H ] , the world state is denoted st. Each agent i = 1 , . . . , N receives an observation oi , t , executes an action ai , t and receives a reward ri , t . The environment transitions to the next state st+1 , according to the transition distribution T ( st+1|st , at ) .1 Each agent observes oi , t , a part of the state st . Agent policies πi are parameterized by θi while the planner policy πp is parameterized by θp . The Nash equilibria of M [ πp ] are agent policies where any unilateral deviation is suboptimal : ANE ( πp ) : = { π | ∀i ∈ [ 1 , N ] , π̃i ∈ Π : Ji ( π̃i , π−i , πp ) ≤ Ji ( πi , π−i , πp ) } , ( 1 ) where Ji ( π , πp ) : = Eπ , πp [ ∑H t=0 γ tr ( i ) t ] denotes the objective of agent i . Hence , a rational agent would not unilaterally deviate from π ∈ ANE ( πp ) . To evaluate a fixed planner policy πp , we simply sample outcomes using policies π ∈ ANE ( πp ) . Also optimizing πp introduces a form of two-level learning . Under appropriate conditions , this can be solved with simultaneous gradient descent ( Zheng et al. , 2020 ; Fiez et al. , 2019 ) . Robustness Objective As noted before , we wish to learn planner policies πp that are robust to reality gaps arising from changes in agent reward functions , e.g. , when agents are boundedly rational.2 We develop a robustness objective for the planner by formalizing such reality gaps as perturbations 1Bold-faced quantities denote vectors or sets , e.g. , a = ( a1 , . . . , aN ) , the action profile for N agents . 2This type of reality gap occurs when the simulated environment ’ s reward function r fails to rationalize the actual behavior of the agents in the real environment , i.e. , when agents in the real world act suboptimally with respect to the simulation ’ s reward function . ξi ∈ Ξ to agent objectives , where the uncertainty set Ξ : ( S , A ) H → R is the space of possible perturbations and represents uncertainty about the objectives of other agents . We extend ANE ( πp , ξ ) to condition the Nash equilibria on perturbations ξ : ANE ( πp , ξ ) : = { π | ∀i ∈ [ 1 , N ] , π̃i ∈ Π : Jξi ( π̃i , π−i , πp ) ≤ J ξ i ( πi , π−i , πp ) } , ( 2 ) Jξi ( π̃i , π−i , πp ) : = Ji ( π̃i , π−i , πp ) + Eτi∼π̃i , π−i , πp [ ξi ( τi ) ] ( 3 ) where τi is a trajectory ( sequence of state-action pairs ) . Following Morimoto & Doya ( 2001 ) , a robust planner optimizes its reward , subject to agents playing a perturbed Nash equilibrium ANE ( πp , ξ ) that maximally penalizes the planner : π∗p = arg max πp min ξ∈Ξ min π∈ANE ( πp , ξ ) Jp ( π , πp ) . ( 4 ) Note that agent policies π ∈ ANE ( πp , ξ ) describes agents that optimize their own reward function , and we assume an adversary chooses ξ. Bounded Uncertainty Set There are two challenges with Equation 4 . First , if the adversary can arbitrarily choose Ξ , the worst case is uninformative.3 Second , depending on the complexity of Π , the uncertainty set Ξ may be high-dimensional and intractable to search . We address these issues by upper-bounding the size of the uncertainty set , L∞ norm of ξi ∈ Ξ , by the term . Thus upper-bounds the difference between the reward functions of agents in the training and testing environments , e.g. , between simulation and the real world . This bounded uncertainty set is : Ξ : = { ξ ∣∣∣∣∣ supπ , πp |ξi ( π , πp ) | < , for all i ∈ I } . ( 5 ) This uncertainty set is equivalent to the -equilibria of M [ πp ] : ANE ( πp , ) : = { π | ∀i ∈ [ 1 , N ] , π̃i ∈ Π : Ji ( π̃i , π−i , πp ) ≤ Ji ( πi , π−i , πp ) + } . ( 6 ) is a tunable hyperparameter—this is the case with most robust RL ( Pinto et al. , 2017 ; Li et al. , 2019 ) —but a good starting value is the anticipated error in reward objective estimates ( applicationspecific ) . Using 6 , the robustness objective becomes the following constrained optimization problem : arg max πp J∗p , min ( πp , ) ︸ ︷︷ ︸ Planner-OPT , where J∗p , min ( πp , ) : = min π∈ANE ( πp , ) Jp ( π , πp ) ︸ ︷︷ ︸ Agent-Adv-Search . ( 7 ) Using ANE ( πp , ) replaces the problem of intractably searching through Ξ with searching through ANE ( πp , ) , and thus merges the two nested min operations in Equation 4 . Conceptually , this transfers the worst-case search problem to the agents : Agent-Adv-Search agents find an adversarial equilibrium ; Planner-OPT optimizes the planner given adversarial agents . Note that the constraint set in Eq 7 is non-empty for ≥ 0 ; the constraints ( Eq 6 ) simply upper-bound the regret of agents . By definition , for non-empty bounded Π , there exists an optimal policy with zero regret . 3 ERMAS : ROBUST POLICIES IN MULTI-AGENT SIMULATIONS . We now introduce ERMAS , an efficient optimization framework to solve the robustness problem in Equation 7 . ERMAS proceeds in three steps . First , it dualizes Equation 7 following constrained RL . Second , it defines a trust-region for the uncertainty set ANE ( πp , ) , approximating the dual problem . Finally , it uses first-order meta-learning with trust-regions to solve the approximate dual problem . See Appendix A.1 for the detailed Algorithm description . Dualizing Agent-Adv-Search The agent search problem in Equation 7 can be formulated similar to a constrained RL problem ( Paternain et al. , 2019 ) , where the primary objective of the agents is to minimize the planner ’ s reward and the secondary objective is to maximize their own reward . While conventional constrained RL enforces a constant lower bound in the constraint , e.g. , 3For instance , by setting ξi such that Jξi = −Jp . Ji ( π , πp ) ≥ C , we enforce a dynamic one : ∀i ∈ 1 . . . N : Ji ( π , πp ) ≥ Ji ( π∗i , π−i , πp ) − , where π∗i is the optimal unilateral deviation for agent i : π ∗ i : = arg maxπ̃i∈Π Ji ( π̃i , π−i , πp ) . Letting λ denote Lagrange multipliers , we can dualize Agent-Adv-Search , i.e. , Equation 4 , as : min π ( Jp ( π , πp ) − N∑ i=1 λi [ Ji ( π ∗ i , π−i , πp ) − Ji ( π , πp ) − i ] ) ︸ ︷︷ ︸ J†p , min ( πp , ) , ( 8 ) This is identical to the dualization of constrained reinforcement learning , whose duality gap is empirically negligible and provably zero under weak assumptions ( Paternain et al. , 2019 ) . We now abuse notation to denote θ : = [ θ1 , . . . , θN , θp ] and Jp ( θ ) : = Jp ( π , πp ) where θ are the parameters of π . To solve Equation 8 , the agents apply gradients : ∇θiJ † p , min ( πp , ) = −∇θiJp ( θ ) − λi∇θi [ Ji ( θ ′ i ( θ ) , θ−i ) − Ji ( θ ) ] , ( 9 ) where θ′i ( θ ) is the parameters of the optimal unilateral deviation π ∗ i for agent i , i.e . the parameters that minimize local regret , which depends on the current policy parameters θi . λi is updated as : ∇λiJ † p , min ( πp , ) = Ji ( π ∗ i , π−i , πp ) − Ji ( π , πp ) − i . ( 10 ) Equation 8 still poses a challenge through the Ji ( π∗i , π−i , πp ) terms , which correspond to unknown agent regret . We now detail the efficient approximation of the value and derivative of agent regret using local and meta-learning approximations , respectively . Trust Regions using Local -equilibria Estimating regret requires knowledge of the optimal unilateral deviation for agent i . We can simplify this problem by proposing a refinement of -equilibria inspired by the notion of local Nash equilibria in differentiable games ( Ratliff et al. , 2014 ) . Definition 3.1 . A strategy π is a local -Nash equilibrium if there exists open sets Wi ⊂ ΠN such that πi ∈ Wi and for each i ∈ { 1 , . . . , N } we have that Ji ( π′i , π−i ) ≤ Ji ( π ) + ′ for all π′i ∈Wi \ { πi } , where ′ : = supπ′i∈Wi KL ( πi||π ′ i ) . By instead performing Agent-Adv-Search on the local -Nash equilibria , we can limit the set of unilateral deviations to consider to a small trust region , Πη ( π ) : ANE ( πp , η ) : = { π | ∀i ∈ [ 1 , N ] , π̃i ∈ Πη ( πi ) : Ji ( π̃i , π−i , πp ) ≤ Ji ( πi , π−i , πp ) + } , ( 11 ) Πη ( π ) : = { π′ ∈ Π | KL ( π||π′ ) ≤ η } , ( 12 ) where η > 0 defines the size of the trust region . For small η , algorithms such as TRPO ( Schulman et al. , 2017 ) can be used to efficiently approximate optimal local deviations of πi , affording reasonable approximations of Ji ( π∗i , π−i , πp ) . Note that our usage of trust region algorithms is not for optimization purposes . ERMAS requires the use of trust region optimization to ensure that the equilibria considered by ERMAS are limited to a local neighborhood of the policy space ( Eq 11 ) . First-Order Meta Learning Approximation The full gradient in Equation 9 is also complicated by the need to estimate the derivative of local regret ∇θi [ Ji ( θ′i ( θ ) , θ−i ) − Ji ( θ ) ] . The second term maximizes the performance of the agent ’ s policy and is simply found with policy gradient . The first term is less straightforward : it minimizes the performance of the best agent policy in the current trust region . We note that this first term corresponds to a meta-learning gradient . We follow REPTILE ( Nichol et al. , 2018 ) to obtain a first-order approximation of a M -step meta-learning gradient : ∇θiJi ( θ′i ( θ ) , θ−i ) = g1 − 1 M M∑ i=1 gi , gi = ∇θiJi θi + i−1∑ j=1 gj , θ−i , θp , ( 13 ) where gi denotes the ith policy gradient in the direction of Ji . In practice , we scale this meta-learning term with the hyperparameter β as β < 1 incorporates a helpful inductive bias where maximizing agent reward leads to local maxima . We can alternatively apply this gradient update periodically , to both mimic β < 1 and reduce computation overhead . First-order meta-learning approximations are known to be empirically effective , and are necessary for ERMAS to efficiently solve Eq 8 . ERMAS By solving the dual problem ( Eq . 8 ) , ERMAS yields robustness to -equilibria and , equivalently , uncertainty in agent objectives . ERMAS solves the dual problem by combining trust regions and meta-learning techniques to estimate and differentiate agent regret . Algorithm 1 and 2 ( Appendix A.1 ) describe an efficient implementation of this procedure for nested policy learning .

Summary: This paper proposes an interesting method for being able to act and plan robustly in a multiagent simulation and be robust to the reality gap between training time and testing time for agents in a marl setting. the method does show improvements in terms of being able to train the policy for this use case and being more robust to some out of distribution configuration of the environment however these improvements appear to be rather limited. in addition, the organization and writing for the paper is very technical and could be improved with additional background information on the uses of metrics and environments as well as better flow between the content in the paper to understand the importance of the different aspects of the method. These improvements could help the reader understand the novelty and important aspects of the method that are difficult to measure. At the moment it comes across as a mix of different methods combined to be able to support this more robust method without a very clear story about the primary problem the paper is trying to solve or the more significant technical aspect of the method that provides this novel solution.