ASYNCHRONOUS MULTI-AGENT GENERATIVE ADVERSARIAL IMITATION LEARNING

1 INTRODUCTION . Imitation learning ( IL ) also known as learning from demonstrations allows agents to imitate expert demonstrations to make optimal decisions without direct interactions with the environment . Especially , inverse reinforcement learning ( IRL ) ( Ng et al . ( 2000 ) ) recovers a reward function of an expert from collected demonstrations , where it assumes that the demonstrator follows an ( near- ) optimal policy that maximizes the underlying reward . However , IRL is an ill-posed problem , because a number of reward functions match the demonstrated data ( Ziebart et al . ( 2008 ; 2010 ) ; Ho & Ermon ( 2016 ) ; Boularias et al . ( 2011 ) ) , where various principles , including maximum entropy , maximum causal entropy , and relative entropy principles , are employed to solve this ambiguity ( Ziebart et al . ( 2008 ; 2010 ) ; Boularias et al . ( 2011 ) ; Ho & Ermon ( 2016 ) ; Zhang et al . ( 2019 ) ) . Going beyond imitation learning with single agents discussed above , recent works including Song et al . ( 2018 ) , Yu et al . ( 2019 ) , have investigated a more general and challenging scenario with demonstration data from multiple interacting agents . Such interactions are modeled by extending Markov decision processes on individual agents to multi-agent Markov games ( MGs ) ( Littman & Szepesvári ( 1996 ) ) . However , these works only work for synchronous MGs , with all agents making simultaneous decisions in each turn , and do not work for general MGs , allowing agents to make asynchronous decisions in different turns , which is common in many real world scenarios . For example , in multiplayer games ( Knutsson et al . ( 2004 ) ) , such as Go game , and many card games , players take turns to play , thus influence each other ’ s decision . The order in which agents make decisions has a significant impact on the game equilibrium . In this paper , we propose a novel framework , asynchronous multi-agent generative adversarial imitation learning ( AMAGAIL ) : A group of experts provide demonstration data when playing a Markov game ( MG ) with an asynchronous decision-making process , and AMAGAIL inversely learns each expert ’ s decision-making policy . We introduce a player function governed by the environment to capture the participation order and dependency of agents when making decisions . The participation order could be deterministic ( i.e. , agents take turns to act ) or stochastic ( i.e. , agents need to take actions by chance ) . A player function of an agent is a probability function : given the perfectly known agent participation history , i.e. , at each previous round in the history , we know which agent ( s ) participated , it provides the probability of the agent participating in the next round . With the general MG model , our framework generalizes MAGAIL ( Song et al . ( 2018 ) ) from the synchronous Markov games to ( asynchronous ) Markov games , and the learned expert policies are proven to guarantee subgame perfect equilibrium ( SPE ) ( Fudenberg & Levine ( 1983 ) ) , a stronger equilibrium than the Nash equilibrium ( NE ) ( guaranteed in MAGAIL Song et al . ( 2018 ) ) . The experiment results demonstrate that compared to GAIL ( Ho & Ermon ( 2016 ) ) and MAGAIL ( Song et al . ( 2018 ) ) , our AMAGAIL model can better infer the policy of each expert agent using their demonstration data collected from asynchronous decision-making scenarios . 2 PRELIMINARIES . 2.1 MARKOV GAMES . Markov games ( MGs ) ( Littman ( 1994 ) ) are the cases of N interacting agents , with each agent making a sequence of decisions with strategies only depending on the current state . A Markov game1 is denoted as a tuple ( N , S , A , Y , ζ , P , η , r , γ ) with a set of states S andN sets of actions { Ai } Ni=1 . At each time step t with a state st ∈ S , if the indicator variable Ii , t = 1 , an agent i is allowed to take an action ; otherwise , Ii , t = 0 , the agent i does not take an action . As a result , the participation vector It = [ I1 , t , · · · , IN , t ] indicates active vs inactive agents at step t. The set of all possible participation vectors is denoted as I , namely , It ∈ I . Moreover , ht−1 = [ I0 , · · · , It−1 ] represent the participation history from step 0 to t−1 . The player function Y ( governed by the environment ) describes the probability of an agent i being allowed to make an action at a step t , given the participation history ht−1 , namely , Y ( i|ht−1 ) . ζ defines the participation probability of an agent at the initial time step ζ : [ N ] 7→ [ 0 , 1 ] . Note that , the player function can be naturally extended to a higher-order form when the condition includes both previous participation history and previous state-action history ; thus , it can be adapted to non-Markov processes . The initial states are determined by a distribution η : S 7→ [ 0 , 1 ] . Let φ denotes no participation , determined by player function Y , the transition process to the next state follows a transition function : P : S×A1∪ { φ } ×· · ·×AN ∪ { φ } 7→ P ( S ) . Agent i obtains a ( bounded ) reward given by a function ri : S×Ai 7→ R2 . Agent i aims to maximize its own total expected return Ri = ∑∞ t=0 γ tri , t , where γ ∈ [ 0 , 1 ] is the discount factor . Actions are chosen through a stationary and stochastic policy πi : S ×Ai 7→ [ 0 , 1 ] . In this paper , bold variables without subscript i denote the concatenation of variables for all the agents , e.g. , all actions as a , the joint policy defined as π ( a|s ) = ∏N i=1 πi ( ai|s ) , r as all rewards . Subscript −i denotes all agents except for i , then ( ai , a−i ) represents the action of all N agents ( a1 , · · · , aN ) . We use expectation with respect to a policy π to denote an expectation with respect to the trajectories it generates . For example , Eπ , Y [ ri ( s , ai ) ] , Est , a∼π , It∼Y [ ∑∞ t=0 γ tri ( st , ai ) ] , denotes the following sample process as s0 ∼ η , I0 ∼ ζ , It ∼ Y , a ∼ π ( ·|st ) , st+1 ∼ P ( st+1|st , a ) , for ∀i ∈ [ N ] . Clearly , when the player function Y ( i|ht−1 ) = 1 for all agents i ’ s at any time step t , a general Markov game boils down to a synchronous Markov game ( Littman ( 1994 ) ; Song et al . ( 2018 ) ) , where all agents take actions at all steps . To distinguish our work from MAGAIL and be consistent with the literature Chatterjee et al . ( 2004 ) and Hansen et al . ( 2013 ) , we refer the game setting discussed in MAGAIL as synchronous Markov games ( SMGs ) , and that of our work as Markov games ( MGs ) . 2.2 SUBGAME PERFECT EQUILIBRIUM FOR MARKOV GAMES . In synchronous Markov games ( SMGs ) , all agents make simultaneous decisions at any time step t , with the same goal of maximizing its own total expected return . Thus , agents ’ optimal policies are interrelated and mutually influenced . Nash equilibrium ( NE ) has been employed as a solution concept to resolve the dependency across agents , where no agents can achieve a higher expected reward by unilaterally changing its own policy ( Song et al . ( 2018 ) ) . However , in Markov games ( MGs ) allowing asynchronous decisions , there exist situations where agents encounter states ( subgames ) resulted from other agents ’ “ trembling-hand ” actions . Since the NE does not consider the “ trembling-hand ” resulted states and subgames , when trapped in these situations , agents are not able to make optimal decisions based on their polices under NE . To address this problem , Selten firstly proposed subgame perfect equilibrium ( SPE ) ( Selten ( 1965 ) ) . SPE ensures NE for every possible 1Note that Markov games defined in MAGAIL ( Song et al . ( 2018 ) ) are in fact synchronous Markov games , with all agents simultaneously making decisions in each turn . We follow the rich literature ( Chatterjee et al . ( 2004 ) ; Hansen et al . ( 2013 ) ) to define Markov games , which allow both synchronous and asynchronous decision-making processes . 2Because of the asynchronous setting , the rewards only depend on agents ’ own actions . subgame of the original game . It has been shown that in a finite or infinite extensive-form game with either discrete or continuous time , best-response strategies all converge to SPE , rather than NE ( Selten ( 1965 ) ; Abramsky & Winschel ( 2017 ) ; Xu ( 2016 ) ) . 2.3 MULTI-AGENT IMITATION LEARNING IN SYNCHRONOUS MARKOV GAMES . In synchronous Markov games , MAGAIL ( Song et al . ( 2018 ) ) was proposed to learn experts ’ policies constrained by Nash equilibrium . Since there may exist multiple Nash equilibrium solutions , a maximum causal entropy regularizer is employed to resolve the ambiguity . Thus , the optimal policies can be found by solving the following multi-agent reinforcement learning problem . MARL ( r ) = arg max π N∑ i=1 ( βHi ( πi ) + Eπi , πE−i [ ri ] ) , ( 1 ) whereHi ( πi ) is the γ-discounted causal entropy of policy πi ∈ Π , Hi ( πi ) , Eπi [ − log πi ( ai|s ) ] = Est , ai∼πi [ − ∑∞ t=0 γ t log πi ( ai|st ) ] , and β is a weight to the entropy regularization term . In practice , the reward function is unknown . MAGAIL applies multi-agent IRL ( MAIRL ) below to recover experts ’ reward functions , with ψ as a convex regularizer , MAIRLψ ( πE ) = arg max r −ψ ( r ) + N∑ i=1 ( EπE [ ri ] ) − ( max π N∑ i=1 ( βHi ( πi ) ) + Eπi , πE−i [ ri ] ) . ( 2 ) Moreover , MAGAIL solves MARL ◦ MAIRLψ ( πE ) to inversely learn each expert ’ s policy via applying generative adversarial imitation learning ( Ho & Ermon ( 2016 ) ) to each expert i ∈ [ N ] : min θ max w Eπθ [ N∑ i=1 logDwi ( s , ai ) ] + EπE [ N∑ i=1 log ( 1−Dwi ( s , ai ) ) ] . ( 3 ) Dwi is a discriminator for agent i that classifies the experts ’ vs policy trajectories . πθ represent the learned experts ’ parameterized policies , which generate trajectories with maximized the scores from Dwi for i ∈ [ N ] .

Summary: In this work, a multi-agent imitation learning algorithm for extensive Markov Games is proposed. Compared to Markov Games (MGs), extensive Markov Games (eMGs) introduces indicator variables, which means whether agents will participate in the game at the specific time step or not, and player function, which is a probability distribution of indicator variables given histories and assumed to be governed by the environment, not by the agents. Such a model allows us to consider asynchronous participation of agents, whereas MGs only consider synchronous participation, which is assumed in the existing multi-agent imitation learning algorithms such as MA-GAIL and MA-AIRL.