A Principled Permutation Invariant Approach to Mean-Field Multi-Agent Reinforcement Learning

1 INTRODUCTION . Multi-Agent Reinforcement Learning ( Littman , 1994 ; Zhang et al. , 2019 ) generalizes Reinforcement Learning ( Sutton and Barto , 2018 ) to address the sequential decision-making problem of multiple agents maximizing their individual long term rewards while interacting with each other in a common environment . With breakthroughs in deep learning , MARL algorithms equipped with deep neural networks have seen significant empirical successes in various domains , including simulated autonomous driving ( Shalev-Shwartz et al. , 2016 ) , multi-agent robotic control ( Matarić , 1997 ; Kober et al. , 2013 ) , and E-sports ( Vinyals et al. , 2019 ) . Despite tremendous successes , MARL is notoriously hard to scale to the many-agent setting , as the size of the state-action space grows exponentially with respect to the number of agents . This phenomenon is recently described as the curse of many agents ( Menda et al. , 2018 ) . To tackle this challenge , we focus on cooperative MARL , where agents work together to maximize their team reward ( Panait and Luke , 2005 ) . We identify and exploit a key property of cooperative MARL with homogeneous agents , namely the invariance with respect to the permutation of agents . Such permutation invariance can be found in many real-world scenarios with homogeneous agents , such as distributed control of multiple autonomous vehicles and team sports ( Cao et al. , 2013 ; Kalyanakrishnan et al. , 2006 ) , but also in scenarios with heterogeneous agent groups , where invariance holds within each group ( Liu et al. , 2019b ) . More importantly , we find that permutation invariance has significant practical implications , as the optimal value functions remain invariant when permuting the joint state-action pairs . Such an observation strongly advocates a permutation invariant design for learning , which helps reduce the effective search space of the policy/value functions from exponential dependence on the number of agents to polynomial dependence . Several empirical methods have been proposed to incorporate permutation invariance into solving MARL problems . Liu et al . ( 2019b ) implement a permutation invariant critic based on Graph Convolutional Network ( GCN ) ( Kipf and Welling , 2017 ) . Sunehag et al . ( 2017 ) propose value decomposition , which together with parameter sharing , leads to a joint critic network that is permutation invariant over agents . While these methods are based on heuristics , we are the first to provide theoretical principles for introducing permutation invariance as an inductive bias for learning value functions and policies in homogeneous systems . In addition , we adopt the DeepSet ( Zaheer et al. , 2017 ) architecture , which is well suited for handling homogeneity of agents , with much simpler operations to induce permutation invariance and greater parameter efficiency . To scale MARL algorithms in the presence of a large number , even infinitely many , agents , mean-field approximation has been explored to directly model the population behavior of the agents . Mean-field game considers large populations of rational agents that play a noncooperative game . Yang et al . ( 2017 ) consider a mean-field game with deterministic linear state transitions , and show that it can be reformulated as a mean-field MDP , where the mean-field state lies in finite-dimensional probability simplex . Yang et al . ( 2018 ) take a mean-field approximation over actions , such that the interaction for any given agent and the population is approximated by the interaction between the agent ’ s action and the averaged actions of its neighboring agents . However , the motivation for averaging over local actions remains unclear , and it generally requires a sparse graph over agents . In practice , properly identifying such structure also demands extensive prior knowledge . Mean-field control instead considers a central controller who aims to compute strategy to optimize the average payoff across the population . Carmona et al . ( 2019 ) motivate a mean-field MDP from the perspective of mean-field control . The mean-field state therein lies in a probability simplex and is thus continuous in nature . To enable the ensuing Q-learning algorithm , discretization of the joint state-action space is necessary . In addition , the dynamic programming principles of such mean-field control problem has been studied in ( Gu et al. , 2019 ) . Gu et al . ( 2020 ) also propose a Q-learning type algorithm , where the state-action space is first discretized into an epsilon-net . The kernel regression operator is used to construct an estimate of the unknown Q-function from samples . Gu et al . ( 2021 ) propose a localized training , decentralized execution framework by locally grouping homogenous agents using their states . Wang et al . ( 2020 ) motivate a mean-field MDP from permutation invariance , but assume a central controller coordinating the actions of all the agents , and hence is restricted to handling the curse of many agents from the exponential blowup of the joint state space . Our formulation of mean-field approximation allows agents to make their own local actions without resorting to a centralized controller . We propose a mean-field Markov decision process motivated from the permutation invariance structure of cooperative MARL , which can be viewed as a natural limit of finite-agent MDP by taking the number of agents to infinity . Such a mean-field MDP generalizes traditional MDP , with each state representing a distribution over the state space of a single agent . The mean-field MDP provides us a tractable formulation to model MDP with many agents , including an infinite number of agents . We further propose the Mean-Field Proximal Policy Optimization ( MF-PPO ) algorithm , at the core of which is a pair of permutation invariant actor and critic neural networks . These networks are implemented based on DeepSet ( Zaheer et al. , 2017 ) , which uses convolutional type operations to induce permutation invariance over the set of inputs . We show that with sufficiently many agents , MF-PPO converges to the optimal policy of the mean-field MDP with a sublinear sample complexity independent of the number of agents . To support our theory , we conduct numerical experiments on the benchmark multi-agent particle environment ( MPE ) and show that our proposed method requires a smaller number of model parameters and attains better performance than multiple baselines . Notations . W denote P ( X ) as the set of distribution on set X . δx denotes the Dirac measure supported at x . For s = ( s1 , . . . , sN ) , we use s i.i.d.∼ p to denote that each si is independently sampled from distribution p. For f : X → R and a distribution π ∈ P ( X ) , we write 〈f , π〉 = Ea∼πf ( a ) . We write [ m ] in short for { 1 , . . . , m } , and ∆d for the standard probability simplex in Rd . 2 PROBLEM SETUP . We focus on studying multi-agent systems with cooperative , homogeneous agents , where the agents within the system are of similar nature and hence can not be distinguished from each other . Specifically , we consider a discrete time control problem with N agents , formulated as a Markov decision process ( SN , AN , P , r ) . We define the joint state space SN to be the Cartesian product of the finite state space S for each agent , and similarly define the joint action space AN . The homogeneous nature of the system is reflected in the transition kernel P and the shared reward r , which satisfies : r ( st , at ) = r ( κ ( st ) , κ ( at ) ) , P ( st+1|st , at ) = P ( κ ( st+1 ) |κ ( st ) , κ ( at ) ) ( 2.1 ) for all ( st , at ) ∈ SN × AN and the permutation mapping κ ( · ) ∈ SN , where SN is the set of all one-to-one mapping from [ N ] to itself . In other words , it is the configuration , rather than individual identities , that affects the team reward , and the transition to the next configuration solely depends on the current configuration . See Figure 1 for detailed illustration . Such permutation invariance finds applications in many real-world scenarios , including distributed control of autonomous vehicles , and social economic systems ( Zheng et al. , 2020 ; Cao et al. , 2013 ; Kalyanakrishnan et al. , 2006 ) . Our goal is to find the optimal policy ν , where ν ( s ) ∈ ∆|AN | for all s ∈ SN , and maximize the expected discounted reward V ν ( s ) = ( 1 − γ ) E { ∑∞t=0 γtr ( st , at ) |s0 =s , at∼ν ( st ) , ∀t≥0 } . Our first result shows that learning with permutation invariance advocates invariant network design . Proposition 2.1 . For cooperative MARL satisfying ( 2.1 ) , there exists an optimal policy ν∗ that is permutation invariant , i.e. , ν∗ ( s , a ) = ν∗ ( κ ( s ) , κ ( a ) ) for any permutation mapping κ ( · ) . In addition , for any permutation invariant policy ν , the value function V ( · ) and the state-action value function Q ( · ) is also permutation invariant , i.e. , V ν ( s ) = V ν ( κ ( s ) ) , Qν ( s , a ) = Qν ( κ ( s ) , κ ( a ) ) , where Qν ( s , a ) = Es′ { r ( s , a ) + γV ν ( s′ ) } . Proposition 2.1 has an important implication for architecture design , as it states that it suffices to search within the permutation invariant policy and value function classes . To the best of our knowledge , this is the first theoretical justification of permutation invariant network design for learning with homogeneous agents . We focus on the factorized policy class with a parameter sharing scheme , where each agent makes its own decision without consolidating with others . Specifically , the joint policy ν can be factorized as ν ( a|s ) =∏N i=1 µ ( ai|oi ) , where µ ( · ) denotes the shared local mapping and oi denotes the local observation . Such a policy class is widely adopted in the celebrated centralized training – decentralized execution paradigm ( Lowe et al. , 2017 ) , due to its light overhead in the deployment phase and favorable performances . However , directly learning such factorized policy remains challenging , as each agent needs to estimate its state-action value function , denoted as Qν ( s , a ) . The search space during learning is ( |S| × |A| ) N , scaling exponentially with respect to the number of agents . The large search space poses as a significant roadblock for efficient learning , and is coined as the curse of many agents . To address the curse of many agents , we exploit the homogeneity of the system and take the mean-field approximation . We begin by taking the perspective of agent i , which is arbitrarily chosen from the N agents . We denote its state as s and the states of the rest of the agents by sr. One can verify that when permuting the state of all the other agents , the value function remains unchanged ; additionally , we can further characterize the value function as a function of the local state and the empirical state distribution over the rest of agents . Proposition 2.2 . For any permutation mapping κ ( · ) , the value function satisfies V ν ( s , sr ) = V ν ( s , κ ( sr ) ) . Additionally , there exists gν such that : V ν ( s , sr ) = gν ( s , p̂sr ) , where p̂sr = 1 N ∑ s∈sr δs is the empirical distribution over the states of rest of the agents sr. For a system with a large number of agents ( e.g. , financial markets , social networks ) , the empirical state distribution can be seen as the concrete realization of the underlying population distribution of the agents . Motivated from this observation and Proposition 2.2 , we formulate the following mean-field MDP that can be seen as the limit of finite-agent MDP in the presence of infinitely many homogeneous agents . Definition 2.1 ( mean-field MDP ) . The mean-field MDP consists of elements of the following : state ( s , dS ) ∈ S × P ( S ) ; action a ∈ A ⊆ AS ; reward r ( s , dS , a ) ; transition kernel P ( s′ , d′S |s , dS , a ) . The mean-field MDP has an intimate connection with our previously discussed finite-agent MDP . Since the the agents are homogeneous , the system is the same from any agent ’ s perspective . We choose any agent ( referred to as representative agent ) , the state information of such an agent includes the local state s , and the mean-field state dS . With state information , the agent selects a meta action a ∈ A ∈ AS , and uses such a meta action to make local decision a = a ( s ) ∈ A . We remark that such a modeling of decision process allows the agent to make decision on both its local information ( local state s ) and the global information ( mean-field state dS ) . From homogeneity we assume all the rest of the agents uses the same meta action a to make their local actions . Note that different agents can still make different local actions due to their different local states , i.e. , a ( z ) 6= a ( z′ ) in general for z 6= z′ ∈ S . The joint state at the next timestep ( s′ , d′S ) naturally depends on the current global state ( s , dS ) and the meta action a ( since all the other agents use a to generate their local actions ) , and is specified by the transition kernel P ( s′ , d′S |s , dS , a ) . In addition , the representative agents receives a reward r ( s , dS , a ) , which depends on the local state and mean-field sate , and the meta action a . Our goal is to learn efficiently a policy π , where π ( ·|s , dS ) ∈ ∆|A| for all ( s , dS ) ∈ S × P ( S ) , for maximized expected discounted reward . To facilitate discussions , we define the value function V π ( s , dS ) = ( 1−γ ) E { ∑∞ t=0 γ tr ( st , dS , t , at ) } , where ( s0 , dS,0 ) = ( s , dS ) , at ∼ π ( st , dS , t ) , ∀t ≥ 0 ; and Q-function Qπ ( s , dS , a ) = ( 1 − γ ) E { ∑∞ t=0 γ tr ( st , dS , t , at ) } , where ( s0 , dS,0 ) = ( s , dS ) , a0 = a , at ∼ π ( st , dS , t ) . The optimal policy is denoted by π∗ ∈ argmaxV π ( s , dS ) . Despite the intuitive analogy to finite-agent MDP , solving the mean-field MDP poses some unique challenges . In addition to having an unknown transition kernel and reward , the mean-field MDP takes a distribution as its state , which we do not have complete information of during training . In the following section , we propose our mean-field Neural Proximal Policy Optimization ( MF-PPO ) algorithm that , with a careful architecture design , can solve such mean-field MDP in a model-free fashion efficiently .

Summary: This paper deals with a class of cooperative MARL problems with permutation invariance. It first shows that, for such problems, there exists an optimal policy that is permutation invariant, and the value function can be characterized as a function of the local state of one agent and the empirical state distribution over the rest of agents. Based on these observations, the authors introduce the mean-field MDP as the limit of the MARL problem with infinitely many homogeneous agents and design a mean-field proximal policy optimization (MF-PPO) algorithm to solve it. It shows with permutation invariance, the search space of the actor/critic network polynomially depends on the number of agents $N$ and establishes the global convergence of MF-PPO. Some numerical results show better performance compared with some existing algorithms.