Multi-Agent Imitation Learning with Copulas

1 INTRODUCTION . Recent years have witnessed great success of reinforcement learning ( RL ) for single-agent sequential decision making tasks . As many real-world applications ( e.g. , multi-player games ( Silver et al. , 2017 ; Brown & Sandholm , 2019 ) and traffic light control ( Chu et al. , 2019 ) ) involve the participation of multiple agents , multi-agent reinforcement learning ( MARL ) has gained more and more attention . However , a key limitation of RL and MARL is the difficulty of designing suitable reward functions for complex tasks with implicit goals ( e.g. , dialogue systems ) ( Russell , 1998 ; Ng et al. , 2000 ; Fu et al. , 2017 ; Song et al. , 2018 ) . Indeed , hand-tuning reward functions to induce desired behaviors becomes especially challenging in multi-agent systems , since different agents may have completely different goals and state-action representations ( Yu et al. , 2019 ) . Imitation learning provides an alternative approach to directly programming agents by taking advantage of expert demonstrations on how a task should be solved . Although appealing , most prior works on multi-agent imitation learning typically assume agents make independent decisions after observing a state ( i.e. , mean-field factorization of the joint policy ) ( Zhan et al. , 2018 ; Le et al. , 2017 ; Song et al. , 2018 ; Yu et al. , 2019 ) , ignoring the potentially complex dependencies that exist among agents . Recently , Tian et al . ( 2019 ) and Liu et al . ( 2020 ) proposed to implement correlated policies with opponent modeling , which incurs unnecessary modeling cost and redundancy , while still lacking coordination during execution . Compared to the single-agent setting , one major and fundamental challenge in multi-agent learning is how to model the dependence among multiple agents in an effective and scalable way . Inspired by probability theory and statistical dependence modeling , in this work , we propose to use copulas ( Sklar , 1959b ; Nelsen , 2007 ; Joe , 2014 ) to model multi-agent behavioral patterns . Copulas are powerful statistical tools to describe the dependence among random variables , which have been widely used in quantitative finance for risk measurement and portfolio optimization ( Bouyé et al. , 2000 ) . Using a copulas-based multi-agent policy enables us to separately learn marginals that capture the local behavioral patterns of each individual agent and a copula function that only and fully captures the dependence structure among the agents . Such a factorization is capable of modeling arbitrarily complex joint policy and leads to interpretable , efficient and scalable multi-agent imitation learning . As a motivating example ( see Figure 1 ) , suppose there are two agents , each with one-dimensional action space . In Figure 1a , although two joint policies are quite different , they actually share the same copula ( dependence structure ) and one marginal . Our proposed copula-based policy is capable a2 π ( a1 , a2|s ) da2 ) as well as the copula c ( F1 ( a1|s ) , F2 ( a2|s ) ) on the unit cube . Here Fi is the cumulative distribution function of the marginal πi ( ai|s ) and ui : = Fi ( ai|s ) is the uniformly distributed random variable obtained by probability integral transform with Fi . More details and definitions can be found in Section 3.2. of capturing such information and more importantly , we may leverage such information to develop efficient algorithms for such transfer learning scenarios . For example , when we want to model teamplay in a soccer game and one player is replaced by his/her substitute while the dependence among different roles are basically the same regardless of players , we can immediately obtain a new joint policy by switching in the new player ’ s marginal while keeping the copula and other marginals unchanged . On the other hand , as shown in Figure 1b , two different joint policies may share the same marginals while having different copulas , which implies that the mean-field policy in previous works ( only modeling marginal policies and making independent decisions ) can not differentiate these two scenarios to achieve coordination correctly . Towards this end , in this paper , we propose a copula-based multi-agent imitation learning algorithm , which is interpretable , efficient and scalable for modeling complex multi-agent interactions . Extensive experimental results on synthetic and real-world datasets show that our proposed method outperforms state-of-the-art multi-agent imitation learning methods in various scenarios and generates multi-agent trajectories close to expert demonstrations . 2 PRELIMINARIES . In this work , we consider the problem of multi-agent imitation learning under the framework of Markov games ( Littman , 1994 ) , which generalize Markov Decision Processes to multi-agent settings , where N agents are interacting with each other . Specifically , in a Markov game , S is the common state space , Ai is the action space for agent i ∈ { 1 , . . . , N } , η ∈ P ( S ) is the initial state distribution and P : S × A1 × . . . × AN → P ( S ) is the state transition distribution of the environment that the agents are interacting with . Here P ( S ) denotes the set of probability distributions over state space S. Suppose at time t , agents observe s [ t ] ∈ S and take actions a [ t ] : = ( a1 [ t ] , . . . , aN [ t ] ) ∈ A1× . . .×AN , the agents will observe state s [ t+1 ] ∈ S at time t+1 with probability P ( s [ t + 1 ] |s [ t ] , a1 [ t ] , . . . , aN [ t ] ) . In this process , the agents select the joint action a [ t ] by sampling from a stochastic joint policy π : S → P ( A1 × . . . × AN ) . In the following , we will use subscript −i to denote all agents except for agent i . For example , ( ai , a−i ) represents the actions of all agents ; πi ( ai|s ) and πi ( ai|s , a−i ) represent the marginal and conditional policy of agent i induced by the joint policy π ( a|s ) ( through marginalization and Bayes ’ s rule respectively ) . We consider the following off-line imitation learning problem . Suppose we have access to a set of demonstrations D = { τ j } Mj=1 provided by some expert policy πE ( a|s ) , where each expert trajectory τ j = { ( sj [ t ] , aj [ t ] ) } Tt=1 is collected by the following sampling process : s1 ∼ η ( s ) , a [ t ] ∼ πE ( a|s [ t ] ) , s [ t+ 1 ] ∼ P ( s|s [ t ] , a [ t ] ) for t ∈ { 1 , . . . , T } . The goal is to learn a parametric joint policy πθ to approximate the expert policy πE such that we can do downstream inferences ( e.g. , action prediction and trajectory generation ) . The learning problem is off-line as we can not ask for additional interactions with the expert policy or the environment during training , and the reward is also unknown . 3 MODELING MULTI-AGENT INTERACTIONS WITH COPULAS . 3.1 MOTIVATION . Many modeling methods for multi-agent learning tasks employ a simplifying mean-field assumption that the agents make independent action choices after observing a state ( Albrecht & Stone , 2018 ; Song et al. , 2018 ; Yu et al. , 2019 ) , which means the joint policy can be factorized as follows : π ( a1 , . . . , aN |s ) = N∏ i=1 πi ( ai|s ) ( 1 ) Such a mean-field assumption essentially allows for independent construction of each agent ’ s policy . For example , multi-agent behavior cloning by maximum likelihood estimation is now equivalent to performing N single-agent behavior cloning tasks : max π E ( s , a ) ∼ρπE [ logπ ( a|s ) ] = N∑ i=1 max πi E ( s , ai ) ∼ρπE , i [ log πi ( ai|s ) ] ( 2 ) where the occupancy measure ρπ : S × A1 × . . . × AN → R denotes the state action distribution encountered when navigating the environment using the joint policy π ( Syed et al. , 2008 ; Puterman , 2014 ) and ρπ , i is the corresponding marginal occupancy measure . However , when the expert agents are making correlated action choices ( e.g. , due to joint plan and communication in a soccer game ) , such a simplifying modeling choice is not able to capture the rich dependency structure and coordination among agent actions . To address this issue , recent works ( Tian et al. , 2019 ; Liu et al. , 2020 ) propose to use a different factorization of the joint policy such that the dependency among N agents can be preserved : π ( ai , a−i|s ) = πi ( ai|s , a−i ) π−i ( a−i|s ) for i ∈ { 1 , . . . , N } . ( 3 ) Although such a factorization is general and captures the dependency among multi-agent interactions , several issues still remain . First , the modeling cost is increased significantly , because now we need to learn N different and complicated opponent policies π−i ( a−i|s ) as well as N different marginal conditional policies πi ( ai|s , a−i ) , each with a deep neural network . It should be noted that there are many redundancies in such a modeling choice . Specifically , suppose there are N agents and N > 3 , for agent 1 and N , we need to learn opponent policies π−1 ( a2 , . . . , aN |s ) and π−N ( a1 , . . . , aN−1|s ) respectively . These are potentially high dimensional and might require flexible function approximations . However , the dependency structure among agent 2 to agent N − 1 are modeled in both π−1 and π−N , which incurs unnecessary modeling cost . Second , when executing the policy , each agent i makes decisions through its marginal policy πi ( ai|s ) = Eπ−i ( a−i|s ) ( ai|s , a−i ) by first sampling a−i from its opponent policy π−i then sampling its action ai from πi ( ·|s , a−i ) . Since each agent is performing such decision process independently , coordination among agents are still impossible due to sampling randomness . Moreover , a set of independently learned conditional distributions are not necessarily consistent with each other ( i.e. , induced by the same joint policy ) ( Yu et al. , 2019 ) . In this work , to address above challenges , we draw inspiration from probability theory and propose to use copulas , a statistical tool for describing the dependency structure between random variables , to model the complicated multi-agent interactions in a scalable and efficient way . 3.2 COPULAS . When the components of a multivariate random variable x = ( x1 , . . . , xN ) are jointly independent , the density of x can be written as : p ( x ) = N∏ i=1 p ( xi ) ( 4 ) When the components are not independent , this equality does not hold any more as the dependencies among x1 , . . . , xN can not be captured by the marginals p ( xi ) . However , the differences can be corrected by multiplying the right hand side of Equation ( 4 ) with a function that only and fully describes the dependency . Such a function is called a copula ( Nelsen , 2007 ) , a multivariate distribution function on the unit hyper-cube with uniform marginals . Intuitively , let us consider a random variable xi with continuous cumulative distribution function Fi . Applying probability integral transform gives us a random variable ui = Fi ( xi ) , which has standard uniform distribution . Thus one can use this property to separate the information in marginals from the dependency structures among x1 , . . . , xN by first projecting each marginal onto one axis of the hyper-cube and then capture the pure dependency with a distribution on the unit hyper-cube . Formally , a copula is the joint distribution of random variables u1 , . . . , uN , each of which is marginally uniformly distributed on the interval [ 0 , 1 ] . Furthermore , we introduce the following theorem that provides the theoretical foundations of copulas : Theorem 1 ( ( Sklar , 1959a ) ) . Suppose the multivariate random variable ( x1 , . . . , xN ) has marginal cumulative distribution functions F1 , . . . , FN and joint cumulative distribution function F , then there exists a unique copula C : [ 0 , 1 ] N → [ 0 , 1 ] such that : F ( x1 , . . . , xN ) = C ( F1 ( x1 ) , . . . , FN ( xN ) ) ( 5 ) When the multivariate distribution has a joint density f and marginal densities f1 , . . . , fN , we have : f ( x1 , . . . , xN ) = N∏ i=1 fi ( xi ) · c ( F1 ( x1 ) , . . . , FN ( xN ) ) ( 6 ) where c is the probability density function of the copula . The converse is also true . Given a copula C and marginals Fi ( xi ) , then C ( F1 ( x1 ) , . . . , FN ( xN ) ) = F ( x1 , . . . , xN ) is a N -dimensional cumulative distribution function with marginal distributions Fi ( xi ) . Theorem 1 states that every multivariate cumulative distribution function F ( x1 , . . . , xN ) can be expressed in terms of its marginals Fi ( xi ) and a copula C ( F1 ( x1 ) , . . . , FN ( xN ) ) . Comparing Eq . ( 4 ) with Eq . ( 6 ) , we can see that a copula function encoding correlations between random variables can be used to correct the mean-field approximation for arbitrarily complex distribution .

Summary: The paper proposes a multi-agent imitation learning method that learns a joint policy for all agents from offline demonstrations. The key idea is to first learn a marginal policy for each agent using behavioral cloning, then fit a copula function that captures dependencies between the agents' policies. Experiments with particle simulations, simulated driving, and simulated RoboCup suggest that this approach could potentially outperform prior methods.