Multi-Agent Interactions Modeling with Correlated Policies

1 INTRODUCTION . Modeling complex interactions among intelligent agents from the real world is essential for understanding and creating intelligent multi-agent behaviors , which is typically formulated as a multiagent learning ( MAL ) problem in multi-agent systems . When the system dynamics are agnostic and non-stationary due to the adaptive agents with implicit goals , multi-agent reinforcement learning ( MARL ) is the most commonly used technique for MAL . MARL has recently drawn much attention and achieved impressive progress on various non-trivial tasks , such as multi-player strategy games ( OpenAI , 2018 ; Jaderberg et al. , 2018 ) , traffic light control ( Chu et al. , 2019 ) , taxi-order dispatching ( Li et al. , 2019 ) etc . A central challenge in MARL is to specify a good learning goal , as the agents ’ rewards are correlated and thus can not be maximized independently ( Bu et al. , 2008 ) . Without explicit access to the reward signals , imitation learning could be the most intuitive solution for learning good policies directly from demonstrations . Conventional solutions such as behavior cloning ( BC ) ( Pomerleau , 1991 ) learn the policy in a supervised manner by requiring numerous data while suffering from compounding error ( Ross & Bagnell , 2010 ; Ross et al. , 2011 ) . Inverse reinforcement learning ( IRL ) ( Ng et al. , 2000 ; Russell , 1998 ) alleviates these shortcomings by recovering a reward function but is always expensive to obtain the optimal policy due to the forward reinforcement learning procedure in an inner loop . Generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ) leaves a better candidate for its model-free structure without compounding error , which is highly effective and scalable . However , real-world multi-agent interactions could be much challenging to imitate because of the strong correlations among adaptive agents ’ policies and rewards . Consider if a football coach wants to win the league , he must make targeted tactics against various opponents , in addition to the situation of his team . Moreover , the multi-agent environment tends to give rise to more severe compounding errors with more expensive running costs . Motivated by these challenges , we investigate the problem of modeling complicated multi-agent interactions from a pile of off-line demonstrations and recover their on-line policies , which can regenerate analogous multi-agent behaviors . Prior studies for multi-agent imitation learning typically limit the complexity in demonstrated interactions by assuming isolated reward structures ( Barrett et al. , 2017 ; Le et al. , 2017 ; Lin et al. , 2014 ; Waugh et al. , 2013 ) and independence in per-agent policies that overlook the high correlations among agents ( Song et al. , 2018 ; Yu et al. , 2019 ) . In this paper , we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework with correlated policies by approximating opponents ’ policies , in order to reach inaccessible opponents ’ actions due to concurrently execution of actions among agents when making decisions . Consequently , with approximated opponents model , we develop a Decentralized Adversarial Imitation Learning algorithm with Correlated policies ( CoDAIL ) suitable for learning correlated policies under our proposed framework , which allows for decentralized training and execution . We prove that our framework treats the demonstrator interactions as one of -Nash Equilibrium ( -NE ) solutions under the recovered reward . In experiments , we conduct multi-dimensional comparisons for both the reward gap between learned agents and demonstrators , along with the distribution divergence between demonstrations and regenerated interacted trajectories from learned policies . Furthermore , the results reveal that CoDAIL can better recover correlated multi-agent policy interactions than other state-of-the-art multi-agent imitation learning methods in several multi-agent scenarios . We further illustrate the distributions of regenerated interactions , which indicates that CoDAIL yields the closest interaction behaviors to the demonstrators . 2 PRELIMINARIES . 2.1 MARKOV GAME AND -NASH EQUILIBRIUM . Markov game ( MG ) , or stochastic game ( Littman , 1994 ) , can be regarded as an extension of Markov Decision Process ( MDP ) . Formally , we define an MG with N agents as a tuple 〈N , S , A ( 1 ) , . . . , A ( N ) , P , r ( 1 ) , . . . , r ( N ) , ρ0 , γ〉 , where S is the set of states , A ( i ) represents the action space of agent i , where i ∈ { 1 , 2 , . . . , N } , P : S × A ( 1 ) × A ( 2 ) × · · · × A ( N ) × S → R is the state transition probability distribution , ρ0 : S → R is the distribution of the initial state s0 , and γ ∈ [ 0 , 1 ] is the discounted factor . Each agent i holds its policy π ( i ) ( a ( i ) |s ) : S × A ( i ) → [ 0 , 1 ] to make decisions and receive rewards defined as r ( i ) : S×A ( 1 ) ×A ( 2 ) ×· · ·×A ( N ) → R. We use−i to represent the set of agents except i , and variables without superscript i to denote the concatenation of all variables for all agents ( e.g. , π represents the joint policy and a denotes actions of all agents ) . For an arbitrary function f : 〈s , a〉 → R , there is a fact that Eπ [ f ( s , a ) ] = Es∼P , a∼π [ f ( s , a ) ] , E [ ∑∞ t=0 γ tf ( st , at ) ] , where s0 ∼ ρ0 , at ∼ π , st+1 ∼ P ( st+1|at , st ) . The objective of agent i is to maximize its own total expected return R ( i ) , Eπ [ r ( i ) ( s , a ) ] = E [ ∑∞ t=0 γ tr ( i ) ( st , at ) ] . In Markov games , however , the reward function for each agent depends on the joint agent actions . Such a fact implies that one ’ s optimal policy must also depend on others ’ policies . For the solution to the Markov games , -Nash equilibrium ( -NE ) is a commonly used concept that extends Nash equilibrium ( NE ) ( Nash , 1951 ) . Definition 1 . An -NE is a strategy profile ( π ( i ) ∗ , π ( −i ) ∗ ) such that ∃ > 0 : v ( i ) ( s , π ( i ) ∗ , π ( −i ) ∗ ) ≥ v ( i ) ( s , π ( i ) , π ( −i ) ∗ ) − , ∀π ( i ) ∈ Π ( i ) , ( 1 ) where v ( i ) ( s , π ( i ) , π ( −i ) ) = Eπ ( i ) , π ( −i ) , s0=s [ r ( i ) ( st , a ( i ) t , a ( −i ) t ) ] is the value function of agent i under state s , and Π ( i ) is the set of policies available to agent i . -NE is weaker than NE , which can be seen as sub-optimal NE . Every NE is equivalent to an -NE where = 0 . 2.2 GENERATIVE ADVERSARIAL IMITATION LEARNING . Imitation learning aims to learn the policy directly from expert demonstrations without any access to the reward signals . In single-agent settings , such demonstrations come from behavior trajectories sampled with the expert policy , denoted as τE = { ( st , a ( i ) t ) } ∞t=0 . However , in multi-agent settings , demonstrations are often interrelated trajectories , that is , which are sampled from the interactions of policies among all agents , denoted as ΩE = { ( st , a ( 1 ) t , ... , a ( N ) t ) } ∞t=0 . For simplicity , we will use the term interactions directly as the concept of interrelated trajectories , and we refer to trajectories for a single agent . Typically , behavior cloning ( BC ) and inverse reinforcement learning ( IRL ) are two main approaches for imitation learning . Although IRL theoretically alleviates compounding error and outperforms to BC , it is less efficient since it requires resolving an RL problem inside the learning loop . Recent proposed work aims to learn the policy without estimating the reward function directly , notably , GAIL ( Ho & Ermon , 2016 ) , which takes advantage of Generative Adversarial Networks ( GAN ( Goodfellow et al. , 2014 ) ) , showing that IRL is the dual problem of occupancy measure matching . GAIL regards the environment as a black-box , which is non-differentiable but can be leveraged through Monte-Carlo estimation of policy gradients . Formally , its objective can be expressed as min π max D EπE [ logD ( s , a ) ] + Eπ [ log ( 1−D ( s , a ) ) ] − λH ( π ) , ( 2 ) where D is a discriminator that identifies the expert trajectories with agents ’ sampled from policy π , which tries to maximize its evaluation from D ; H is the causal entropy for the policy and λ is the hyperparameter . 2.3 CORRELATED POLICY . In multi-agent learning tasks , each agent imakes decisions independently while the resulting reward r ( i ) ( st , a ( i ) t , a ( −i ) t ) depends on others ’ actions , which makes its cumulative return subjected to the joint policy π . One common joint policy modeling method is to decouple the π with assuming conditional independence of actions from different agents ( Albrecht & Stone , 2018 ) : π ( a ( i ) , a ( −i ) |s ) ≈ π ( i ) ( a ( i ) |s ) π ( −i ) ( a ( −i ) |s ) . ( 3 ) However , such a non-correlated factorization on the joint policy is a vulnerable simplification which ignores the influence of opponents ( Wen et al. , 2019 ) . And the learning process of agent i lacks stability since the environment dynamics depends on not only the current state but also the joint actions of all agents ( Tian et al. , 2019 ) . To solve this , recent work has taken opponents into consideration by decoupling the joint policy as a correlated policy conditioned on state s and a ( −i ) as π ( a ( i ) , a ( −i ) |s ) = π ( i ) ( a ( i ) |s , a ( −i ) ) π ( −i ) ( a ( −i ) |s ) , ( 4 ) where π ( i ) ( a ( i ) |s , a ( −i ) ) is the conditional policy , with which agent i regards all potential actions from its opponent policies π ( −i ) ( a ( −i ) |s ) , and makes decisions through the marginal policy π ( i ) ( a ( i ) |s ) = ∫ a ( −i ) π ( i ) ( a ( i ) |s , a ( −i ) ) π ( −i ) ( a ( −i ) |s ) da ( −i ) = Ea ( −i ) π ( i ) ( a ( i ) |s , a ( −i ) ) .

Summary: In this work, a multi-agent imitation learning algorithm with opponent modeling is proposed, where each agent considers other agents’ expected actions in advance and uses them to generate their own actions. Assuming each agent can observe other agents’ actions, which is a reasonable assumption in MARL problems, a decentralized algorithm called CoDAIL is proposed. For each iteration of CoDAIL, (1) each agent trains opponent models (other agents’ policies) by minimizing either MSE loss (continuous actions) or CE loss (discrete actions), (2) samples actions from those opponent models, (3) updates individual rewards (discriminators) and critics and (4) updates policies with multi-agent extention of ACKTR (which is used in MA-GAIL and MA-AIRL as well).