Multi-Agent Imitation Learning with Copulas

1 INTRODUCTION . Recent years have witnessed great success of reinforcement learning ( RL ) for single-agent sequential decision making tasks . As many real-world applications ( e.g. , multi-player games ( Silver et al. , 2017 ; Brown & Sandholm , 2019 ) and traffic light control ( Chu et al. , 2019 ) ) involve the participation of multiple agents , multi-agent reinforcement learning ( MARL ) has gained more and more attention . However , a key limitation of RL and MARL is the difficulty of designing suitable reward functions for complex tasks with implicit goals ( e.g. , dialogue systems ) ( Russell , 1998 ; Ng et al. , 2000 ; Fu et al. , 2017 ; Song et al. , 2018 ) . Indeed , hand-tuning reward functions to induce desired behaviors becomes especially challenging in multi-agent systems , since different agents may have completely different goals and state-action representations ( Yu et al. , 2019 ) . Imitation learning provides an alternative approach to directly programming agents by taking advantage of expert demonstrations on how a task should be solved . Although appealing , most prior works on multi-agent imitation learning typically assume agents make independent decisions after observing a state ( i.e. , mean-field factorization of the joint policy ) ( Zhan et al. , 2018 ; Le et al. , 2017 ; Song et al. , 2018 ; Yu et al. , 2019 ) , ignoring the potentially complex dependencies that exist among agents . Recently , Tian et al . ( 2019 ) and Liu et al . ( 2020 ) proposed to implement correlated policies with opponent modeling , which incurs unnecessary modeling cost and redundancy , while still lacking coordination during execution . Compared to the single-agent setting , one major and fundamental challenge in multi-agent learning is how to model the dependence among multiple agents in an effective and scalable way . Inspired by probability theory and statistical dependence modeling , in this work , we propose to use copulas ( Sklar , 1959b ; Nelsen , 2007 ; Joe , 2014 ) to model multi-agent behavioral patterns . Copulas are powerful statistical tools to describe the dependence among random variables , which have been widely used in quantitative finance for risk measurement and portfolio optimization ( Bouyé et al. , 2000 ) . Using a copulas-based multi-agent policy enables us to separately learn marginals that capture the local behavioral patterns of each individual agent and a copula function that only and fully captures the dependence structure among the agents . Such a factorization is capable of modeling arbitrarily complex joint policy and leads to interpretable , efficient and scalable multi-agent imitation learning . As a motivating example ( see Figure 1 ) , suppose there are two agents , each with one-dimensional action space . In Figure 1a , although two joint policies are quite different , they actually share the same copula ( dependence structure ) and one marginal . Our proposed copula-based policy is capable a2 π ( a1 , a2|s ) da2 ) as well as the copula c ( F1 ( a1|s ) , F2 ( a2|s ) ) on the unit cube . Here Fi is the cumulative distribution function of the marginal πi ( ai|s ) and ui : = Fi ( ai|s ) is the uniformly distributed random variable obtained by probability integral transform with Fi . More details and definitions can be found in Section 3.2. of capturing such information and more importantly , we may leverage such information to develop efficient algorithms for such transfer learning scenarios . For example , when we want to model teamplay in a soccer game and one player is replaced by his/her substitute while the dependence among different roles are basically the same regardless of players , we can immediately obtain a new joint policy by switching in the new player ’ s marginal while keeping the copula and other marginals unchanged . On the other hand , as shown in Figure 1b , two different joint policies may share the same marginals while having different copulas , which implies that the mean-field policy in previous works ( only modeling marginal policies and making independent decisions ) can not differentiate these two scenarios to achieve coordination correctly . Towards this end , in this paper , we propose a copula-based multi-agent imitation learning algorithm , which is interpretable , efficient and scalable for modeling complex multi-agent interactions . Extensive experimental results on synthetic and real-world datasets show that our proposed method outperforms state-of-the-art multi-agent imitation learning methods in various scenarios and generates multi-agent trajectories close to expert demonstrations . 2 PRELIMINARIES . In this work , we consider the problem of multi-agent imitation learning under the framework of Markov games ( Littman , 1994 ) , which generalize Markov Decision Processes to multi-agent settings , where N agents are interacting with each other . Specifically , in a Markov game , S is the common state space , Ai is the action space for agent i ∈ { 1 , . . . , N } , η ∈ P ( S ) is the initial state distribution and P : S × A1 × . . . × AN → P ( S ) is the state transition distribution of the environment that the agents are interacting with . Here P ( S ) denotes the set of probability distributions over state space S. Suppose at time t , agents observe s [ t ] ∈ S and take actions a [ t ] : = ( a1 [ t ] , . . . , aN [ t ] ) ∈ A1× . . .×AN , the agents will observe state s [ t+1 ] ∈ S at time t+1 with probability P ( s [ t + 1 ] |s [ t ] , a1 [ t ] , . . . , aN [ t ] ) . In this process , the agents select the joint action a [ t ] by sampling from a stochastic joint policy π : S → P ( A1 × . . . × AN ) . In the following , we will use subscript −i to denote all agents except for agent i . For example , ( ai , a−i ) represents the actions of all agents ; πi ( ai|s ) and πi ( ai|s , a−i ) represent the marginal and conditional policy of agent i induced by the joint policy π ( a|s ) ( through marginalization and Bayes ’ s rule respectively ) . We consider the following off-line imitation learning problem . Suppose we have access to a set of demonstrations D = { τ j } Mj=1 provided by some expert policy πE ( a|s ) , where each expert trajectory τ j = { ( sj [ t ] , aj [ t ] ) } Tt=1 is collected by the following sampling process : s1 ∼ η ( s ) , a [ t ] ∼ πE ( a|s [ t ] ) , s [ t+ 1 ] ∼ P ( s|s [ t ] , a [ t ] ) for t ∈ { 1 , . . . , T } . The goal is to learn a parametric joint policy πθ to approximate the expert policy πE such that we can do downstream inferences ( e.g. , action prediction and trajectory generation ) . The learning problem is off-line as we can not ask for additional interactions with the expert policy or the environment during training , and the reward is also unknown . 3 MODELING MULTI-AGENT INTERACTIONS WITH COPULAS . 3.1 MOTIVATION . Many modeling methods for multi-agent learning tasks employ a simplifying mean-field assumption that the agents make independent action choices after observing a state ( Albrecht & Stone , 2018 ; Song et al. , 2018 ; Yu et al. , 2019 ) , which means the joint policy can be factorized as follows : π ( a1 , . . . , aN |s ) = N∏ i=1 πi ( ai|s ) ( 1 ) Such a mean-field assumption essentially allows for independent construction of each agent ’ s policy . For example , multi-agent behavior cloning by maximum likelihood estimation is now equivalent to performing N single-agent behavior cloning tasks : max π E ( s , a ) ∼ρπE [ logπ ( a|s ) ] = N∑ i=1 max πi E ( s , ai ) ∼ρπE , i [ log πi ( ai|s ) ] ( 2 ) where the occupancy measure ρπ : S × A1 × . . . × AN → R denotes the state action distribution encountered when navigating the environment using the joint policy π ( Syed et al. , 2008 ; Puterman , 2014 ) and ρπ , i is the corresponding marginal occupancy measure . However , when the expert agents are making correlated action choices ( e.g. , due to joint plan and communication in a soccer game ) , such a simplifying modeling choice is not able to capture the rich dependency structure and coordination among agent actions . To address this issue , recent works ( Tian et al. , 2019 ; Liu et al. , 2020 ) propose to use a different factorization of the joint policy such that the dependency among N agents can be preserved : π ( ai , a−i|s ) = πi ( ai|s , a−i ) π−i ( a−i|s ) for i ∈ { 1 , . . . , N } . ( 3 ) Although such a factorization is general and captures the dependency among multi-agent interactions , several issues still remain . First , the modeling cost is increased significantly , because now we need to learn N different and complicated opponent policies π−i ( a−i|s ) as well as N different marginal conditional policies πi ( ai|s , a−i ) , each with a deep neural network . It should be noted that there are many redundancies in such a modeling choice . Specifically , suppose there are N agents and N > 3 , for agent 1 and N , we need to learn opponent policies π−1 ( a2 , . . . , aN |s ) and π−N ( a1 , . . . , aN−1|s ) respectively . These are potentially high dimensional and might require flexible function approximations . However , the dependency structure among agent 2 to agent N − 1 are modeled in both π−1 and π−N , which incurs unnecessary modeling cost . Second , when executing the policy , each agent i makes decisions through its marginal policy πi ( ai|s ) = Eπ−i ( a−i|s ) ( ai|s , a−i ) by first sampling a−i from its opponent policy π−i then sampling its action ai from πi ( ·|s , a−i ) . Since each agent is performing such decision process independently , coordination among agents are still impossible due to sampling randomness . Moreover , a set of independently learned conditional distributions are not necessarily consistent with each other ( i.e. , induced by the same joint policy ) ( Yu et al. , 2019 ) . In this work , to address above challenges , we draw inspiration from probability theory and propose to use copulas , a statistical tool for describing the dependency structure between random variables , to model the complicated multi-agent interactions in a scalable and efficient way . 3.2 COPULAS . When the components of a multivariate random variable x = ( x1 , . . . , xN ) are jointly independent , the density of x can be written as : p ( x ) = N∏ i=1 p ( xi ) ( 4 ) When the components are not independent , this equality does not hold any more as the dependencies among x1 , . . . , xN can not be captured by the marginals p ( xi ) . However , the differences can be corrected by multiplying the right hand side of Equation ( 4 ) with a function that only and fully describes the dependency . Such a function is called a copula ( Nelsen , 2007 ) , a multivariate distribution function on the unit hyper-cube with uniform marginals . Intuitively , let us consider a random variable xi with continuous cumulative distribution function Fi . Applying probability integral transform gives us a random variable ui = Fi ( xi ) , which has standard uniform distribution . Thus one can use this property to separate the information in marginals from the dependency structures among x1 , . . . , xN by first projecting each marginal onto one axis of the hyper-cube and then capture the pure dependency with a distribution on the unit hyper-cube . Formally , a copula is the joint distribution of random variables u1 , . . . , uN , each of which is marginally uniformly distributed on the interval [ 0 , 1 ] . Furthermore , we introduce the following theorem that provides the theoretical foundations of copulas : Theorem 1 ( ( Sklar , 1959a ) ) . Suppose the multivariate random variable ( x1 , . . . , xN ) has marginal cumulative distribution functions F1 , . . . , FN and joint cumulative distribution function F , then there exists a unique copula C : [ 0 , 1 ] N → [ 0 , 1 ] such that : F ( x1 , . . . , xN ) = C ( F1 ( x1 ) , . . . , FN ( xN ) ) ( 5 ) When the multivariate distribution has a joint density f and marginal densities f1 , . . . , fN , we have : f ( x1 , . . . , xN ) = N∏ i=1 fi ( xi ) · c ( F1 ( x1 ) , . . . , FN ( xN ) ) ( 6 ) where c is the probability density function of the copula . The converse is also true . Given a copula C and marginals Fi ( xi ) , then C ( F1 ( x1 ) , . . . , FN ( xN ) ) = F ( x1 , . . . , xN ) is a N -dimensional cumulative distribution function with marginal distributions Fi ( xi ) . Theorem 1 states that every multivariate cumulative distribution function F ( x1 , . . . , xN ) can be expressed in terms of its marginals Fi ( xi ) and a copula C ( F1 ( x1 ) , . . . , FN ( xN ) ) . Comparing Eq . ( 4 ) with Eq . ( 6 ) , we can see that a copula function encoding correlations between random variables can be used to correct the mean-field approximation for arbitrarily complex distribution .

Summary: The paper proposes the use of "Copulas" to capture dependencies among agents in multi-player environments. The authors argue that prior work(eg GNN, VAE, LSTM etc based) do not leverage the common structure among agent behaviour. They explain how the copula can more efficiently encode the dependency among policies, as compared to simply factorizing. The authors provide empirical results on real world and synthetic datasets, showcasing superior performance against baseline approaches.