COLLABORATIVE MULTIAGENT REINFORCEMENT LEARNING  IN HOMOGENEOUS SWARMS

1 INTRODUCTION . We consider problems in which groups of agents learn to collaborate . The setup consists of a world , or environment , in which individual agents observe the state of the world , their own state , and possibly some partial information about the state of other agents . Out of the available information agents settle on a choice of action that determines the collection of an instantaneous reward . Our interest is in long term collaborative setups in which the team as a whole acts to maximize the sum of discounted rewards collected by all the members of the group . We restrict attention to homogeneous teams where all agents have identical dynamics and reward structures and , even if our methods work for small number of agents , we focus the effort on large swarms with up to a few hundred agents . Although having a homogeneous team is perhaps the simplest multiagent learning problem we can conceive , the challenges are still significant because the action choices of any agent can have , and most likely will have , some bearing on the actions chosen by other agents . Indeed , agents ’ actions have a direct effect on the evolution of their own state , the state of the world , and the states of other agents . But this generates an indirect coupling between the actions of different agents as they reason about the collective effects of their individual choices ( Eksin et al . ( 2013a ; b ) ) . Collaborative homogeneous swarms are as common in robotics as they are complex . A somewhat canonical example is a collaborative navigation problem in which a group of n autonomous agents must stake positions in a set ofm surveyed points ( Berman et al . ( 2009 ) ) . More practical examples are autonomous warehouse management ( Enright & Wurman ( 2011 ) ) , collaborative assembly ( Knepper et al . ( 2013 ) ) , and concurrent control and communication for teams of robots Stephan et al . ( 2017 ) . A large literature exists to generate the necessary autonomous collective behavior but the inherent hardness of the problem limits their applicability to relatively small teams ( Solovey & Halperin ( 2016 ) ) . Our goal here is to leverage the increasing success of deep neural networks in reinforcement learning ( Mnih et al . ( 2013 ; 2016 ) ; Levine et al . ( 2015 ) ; Pathak et al . ( 2017 ) ; Khan et al . ( 2018 ) ; Gupta et al . ( 2017 ) ) to develop a generic methodology for learning in the context of large scale collaborative homogeneous swarms . In principle , we can think of the team as a whole as a system to which we apply some reinforcement learning technique . This is certainly a viable approach with interesting variations of actor-critic models having found particular success ( Foerster et al . ( 2017 ) ; Lowe et al . ( 2017 ) ) . However , these techniques are limited in their applicability to small teams with a few agents . This is not unexpected because the sample inefficiency of conventional reinforcement learning gets compounded with the exponential complexity of having agents learning with respect to each other . To circumvent this limitation we exploit the homogeneity of the team . Under this assumption it is natural to conclude that two different agents faced with the same state and information about the state of other agents must have the same policy . Therefore , we can force the learning procedure to find a common policy for all agents , an observation that constitutes the first contribution of this paper : ( C1 ) We formulate collaborative learning for homogeneous teams as a constrained dynamic program in which different agents are required to find a shared common policy . In forcing all agents to find a common policy we mitigate the sample inefficiency of reinforcement learning as the experiences of any given agent inform the learning process of all other agents . The situation is not unlike meta-learning where we cross-pollinate different tasks to effectively enlarge the size of the training set ( Al-Shedivat et al . ( 2017 ) ; Finn et al . ( 2017 ) ) . This similarity of purpose notwithstanding the differences are sufficiently important to warrant the development of a custom learning technique . We do so by working on a variation of the policy gradient method which we adapt to solve the collaborative learning constrained dynamic program mentioned in ( C1 ) . Interestingly , the equivalence between policies can be incorporated into the problem formulation in several different ways ( see Section 2 ) . Our experimental results show that the most convenient formulation is one in which we introduce a ( fictitious ) common policy and require all agents to find policies that agree with this common policy . In order to solve this optimization we utilize a coordinate projected policy gradient algorithm whose development is the second contribution of this paper : ( C2 ) We develop a coordinate projected policy gradient algorithm . At each step of this iterative algorithm agents roll out trajectories according to the common policy and utilize the experience to update their individual policies . This is a policy gradient descent step along individual agent coordinates . After this update we roll out trajectories in which individual policies are pitched out against the common centralized policy . This is used to implement a policy gradient descent step on the central policy . After these two separate coordinate descent steps we project the disparate policies into a common policy and proceed to the next iteration . We provide experimental evaluations in a collaborative navigation problem and a multi-predatormulti-prey game ( from Lowe et al . ( 2017 ) ) , and a multiagent survival game ( from Zheng et al . ( 2017 ) . The numerical experiments show marked improvements in rewards relative to methods that do not force policy equivalence ( Lowe et al . ( 2017 ) ) . 1.1 RELATED WORK . Multiagent reinforcement learning problems have a long history Littman ( 1994 ) ; Hu & Wellman ( 2003 ) ; Busoniu et al . ( 2006 ) ; Conitzer & Sandholm ( 2007 ) . Early approaches focused on tabular methods to compute Q-values for Markov games ( e.g , Hu & Wellman ( 2003 ) ) and developed several solution methodologies for finding competitive and collaborative equilibrium ( e.g. , Conitzer & Sandholm ( 2007 ) ) . Building on the success of deep reinforcement learning there has been a recent buildup of interest in using high capacity neural network models . These include the use of neural networks in two-player games ( Tampuu et al . ( 2017 ) ) and in generic Bayesian games ( Da Silva et al . ( 2006 ) ; Eksin et al . ( 2013a ; b ) ; Hong et al . ( 2018 ) ) in which each of the agents must build a model for the behavior of other agents . In terms of novel algorithmic development , ( Foerster et al . ( 2017 ) ) and ( Lowe et al . ( 2017 ) ) propose a variation of actor-critic methods in which each agent is modeled as a decentralized actor working in conjunction with a centralized critic with parameter sharing among the agents . In Bayesian games formulations as well as in actor-critic approaches , it is possible to have a heterogeneous agent mix but it is difficult to scale the number of agents . This is different from our interest which is on leveraging homogeneity to provide scalability to large swarms . An alternative approach to achieve scalability is to work with mean field game models in which we assume the number of agents to be infinite and train policies that can be deployed on individual robots while operating on summary statistics of the team ’ s state ( Mguni et al . ( 2018 ) ; Yang et al . ( 2017 ) ) . Our work differs in that we operate on information about the state of individual agents which can be incomplete and typically involves a few agents only . 2 MARKOV COLLABORATIVE REINFORCEMENT LEARNING . We consider policy learning problems in a collaborative Markov team ( Littman ( 1994 ) ) . The team is composed of N agents generically indexed by n which at any given point in time t occupy a position xnt ∈ X in configuration space and must choose an action ant ∈ A in action space . Agents operate in a world , or environment , whose state at time t we denote as wt . The team and environment are assumed Markov so that if we collect all agents ’ configurations in the vector xt : = [ x1t ; . . . ; xNt ] and all actions in the vector at : = [ a1t ; . . . ; aNt ] ∈ AN the evolution of the system is completely determined by the conditional transition probability p ( xt+1 , wt+1 ∣∣xt , at , wt ) . We further assume that agents are statistically identical in that the probability transition kernel is invariant to agent permutations . This implies the transition dynamics are the same for all agents so that if we swap two of them in configuration and action space we expect to see the same statistical evolution . The assumption is justified because the robotic swarm is assumed homogeneous . In order to choose their actions ant , agents have access to their own states xnt , the state of the world wt and some possibly partial information I ( x−nt ) about the state of other agents x−nt = [ xmt ] m6=n . All of these variables are grouped in the local state snt = ( xnt , wt , I ( x−nt ) ) . The action ant is chosen to be Markov with respect to this state . Therefore , the policy of agent n is a function πn that chooses actions according to ant = πn ( snt ) = πn ( xnt , wt , I ( x−nt ) ) . ( 1 ) Observe that the policy is chosen to be Markov even though there may be advantages in keeping track of past states and past information . The team is Markov but from the perspective of an individual agent the evolution of the system is not necessarily Markov unless the information I ( x−nt ) is a complete description of the states of other agents . For future reference we define π : = [ π1 ; . . . ; πN ] to group the policies of all agents and π−n = [ πm ] m6=n to group the policies of all agents except n. As agents operate in their environment , they collect individual rewards rn ( xt , wt , ant ) which depend on the configuration of the team xt , the state of the world wt and their own individual action ant . The quantity of interest to agent n is not this instantaneous reward but rather the long term reward accumulated over a time horizon T as discounted by a factor γ , Rn : = T∑ t=0 γtrn ( xt , wt , ant ) . ( 2 ) The reward Rn in equation 2 is stochastic as it depends on the trajectory ’ s realization . In conventional reinforcement learning , agent n would define the cost L̃n ( πn ) : = Eπn ( Rn ) and search for a policy πn that maximizes this long term expected reward . Naturally , the expectation Eπn ( Rn ) implicitly depends on the policies of other agents since this would affect the transition probability p ( xt+1 , wt+1 ∣∣xt , at , wt ) . To emphasize this fact we write L̃n ( πn ) = Eπn , π−n ( Rn ) where , we recall , π−n = [ πm ] m 6=n represents the policies of all agents except n. Since we are interested in collaborative teams , we consider a different formulation in which agents strive to maximize the sum of accumulated rewards across all members of the team Eπ [ N∑ n=1 Rn ] = N∑ n=1 Eπn , π−n [ Rn ] : = N∑ n=1 Ln ( πn , π−n ) , ( 3 ) where we have defined Ln ( πn , π−n ) : = Eπn , π−n [ Rn ] which is the component of the reward that is collected by agent n. To clarify ideas we discuss an example . Example 1 ( Collaborative Navigation ) To illustrate the problem formulation consider a collaborative navigation task whereby a team of N agents are tasked with reaching N preassigned goals ; see Figure 2 ( left ) . The goals are fixed and we denote their locations as yn . The agents move to approach these goals and at each point in time t , the position of agent n is denoted as xnt ∈ R2 . Agent n observes its location in space perfectly . This is the agent ’ s position in configuration space [ cf . equation 1 ] . Agent n also observes perfectly the location of all goals . Thus , the state of the world available to all agents is wt = [ yn ; . . . ; yn ] . Observe that this is an static state of the world . A time varying world state is obtained if the goals are moving around . In addition to this , agent n also observes the positions of other agents relative to its own position . I.e. , the information I ( x−nt ) is a collection of relative locations xmt − xnt for all m 6= n. This is a full information example . Partial information is obtained if the location of some agents is unknown or if the location of the agents is known with some error . The reward collected by agent n varies inversely with the distance ‖xnt − yn‖ to its assigned target and incorporate a negative reward for colliding with other agents . We wish to learn a policy that maximizes the sum of accumulated rewards across all members of the team [ cf . equation 3 ] . We expect that this will induce a behavior in which all agents move to their assigned targets while avoiding collisions with each other .

Summary: The authors present an approach to training collaborative swarms of agents based around giving all agents identical (or near identical) policies. The training regime involves individual agents rolling out trajectories based on slight perturbations of an agent of focus keeping the policy of other agents fixed. This is repeated for each agent, then these trajectories are used to batch update the joint policy with an average gradient.