Learning to Simulate Self-driven Particles System with Coordinated Policy Optimization

1 Introduction . Self-Driven Particles ( SDP ) describe a wide range of multi-agent systems ( MAS ) in nature and human society . In SDP , individual agent pursues its own goal and interacts with each other following simple local alignment , and then the population exhibits complex collective behaviors [ 47 ] . We commonly see the phenomena of collective behaviors , such as the flocking birds [ 4 ] , molecular motors [ 6 ] , human crowd [ 17 , 16 ] , and the traffic system [ 20 ] . To understand and simulate such phenomena , researchers have developed a number of SDP models . For example , simple-rule based models [ 7 ] or Hydrodynamic equations based models [ 2 , 19 ] can simulate the SDP very well in an unconstrained environment with random movement and resemble complex behaviors such as schooling fish [ 12 ] and marching locusts [ 5 ] . However , in a more structured environment such as a particular traffic scene where the interactions of agents are time-varying and the environment is non-stationary , it is difficult to design manual controllers or use rules to recover the underlying collective behaviors . When the interactive environment is available , reinforcement learning becomes a promising approach to learn the controllers for actuating the SDP . Recently , many multi-agent reinforcement learning ( MARL ) methods have been developed to play competitive multi-player games , such as Hide and Seek [ 1 ] , Football [ 23 ] , Go and other board games [ 34 ] , and StarCraft [ 33 ] . However , it is challenging to apply the existing MARL to simulate SDP systems . One essential issue is that each 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . constituent agent in a SDP system is self-interested and the relationship between agents is constantly changing . As illustrated in Fig . 1A , the vehicles in the traffic system demonstrate social behaviors that are either cooperative or competitive depending on the situation . Meanwhile , the cooperation and the competition naturally emerge as the outcomes of the self-driven instinct and the multiagent interaction . Thus it is difficult to generate those social behaviors through a top-down design . Furthermore , most of the MARL methods assume that the role of each agent is pre-determined and fixed within one episode . In cooperative MARL methods [ 32 , 41 , 9 , 8 ] , all agents need to cooperate to maximize a joint scalar reward . However , the credit decomposition problem centered in such setting is not the major issue of simulating SDP systems , because each agent in SDP has its individual objective and the credit decomposition is no longer needed . On the other hand , directly applying independent learning [ 43 , 35 ] will lead to a group of agents with unreasonable behaviors that are aggressive or egocentric , since each agent is trained to maximize its own reward . In order to simulate SDP systems , the coordination of the self-interested constituents is the major issue . The coordination of agents in MARL has been previously studied in the context of mixed cooperative and competitive tasks [ 25 , 38 ] . However , seldom works focus on the simulation of SDP systems . Taking the microscopic traffic simulation as an example , existing works focus on learning high-level controller to indirectly actuate vehicles [ 48 ] , or using independent learner to train policies executing in the environments engaging a limited set of agents on simple scenes [ 31 , 56 ] , without considering the coordination problem . Instead , we aim to control all the vehicles in the scene by operating on the low-level continuous control space . This allows a higher degree of freedom to learn diverse behaviors but poses a challenging continuous control problem . In this work we propose a novel MARL method called Coordinated Policy Optimization ( CoPO ) to facilitate the coordination of agents at both local and global levels . To evaluate the proposed method , as illustrated in Fig . 1 , we construct five typical traffic environments as the testing bed . These environments contain rich interactions between agents and complex road structures . Besides , we develop three task-agnostic metrics to characterize different aspects of the learned populations . We show that the proposed CoPO method can achieve superior performance compared to the baselines . Noticeably , collective social behaviors that resemble real-world patterns emerge in the trained population . 2 Related Work . Multi-Agent Reinforcement Learning . Three typical task settings are explored in MARL [ 18 ] : the fully cooperative task , the fully competitive task , and the mixed cooperative and competitive task . Many works focus on the problems in cooperative tasks such as credit assignment [ 32 , 41 , 9 , 8 , 55 ] and communication [ 10 , 46 ] . In competitive tasks , the challenges include modeling the opponents [ 11 ] and generating meaningful opponent via self-play [ 15 , 49 , 1 ] . In mixed cooperative and competitive task [ 29 , 25 , 38 ] , the roles of the agents are mostly predefined and fixed at run-time . Instead , in this work we hope to simulate SDP systems without imposing any constraint on the roles of the agents thus there is more space for emergent social behaviors . Apart from the fixed role problem , previous methods like MADDPG [ 29 ] , QMIX [ 32 ] , QTRAN [ 40 ] , and COMA [ 9 ] are only applicable to discrete action space . We instead focus on learning policies for continuous control . The independent PPO [ 35 ] and the Mean Field MARL [ 52 ] are two recent methods applied for continuous control in MARL . The Mean Field MARL method proposed by Yang et al . [ 52 ] approximates the value of global reward Q ( s , a ) , a = [ a1 , ... , aN ] by agent-independent value functions Qi ( s , ai , ā N i ) , where ā N i is the average action of the nearby agents around agent i . Following similar design but in the context of simulating SDP , we use the mean field to compute neighborhood reward and use that reward to enforce local coordination , instead of using mean field to factorize joint value via averaging actions as done in [ 52 ] . Traffic simulation . Simulating the macroscopic and microscopic traffic systems has been studied for decades [ 21 ] . As for the macroscopic traffic system where the dynamics of individuals is simple and the high-level characters of flow such as the traffic throughput are more concerned , early works focus on designing controllers to direct a fleet of vehicles into a dense and stable platoon [ 39 , 3 ] . CityFlow [ 53 ] and FLOW [ 51 ] use RL agents to steer the low-level controllers of vehicles in order to investigate the traffic phenomenon in large-scale macroscopic traffic flow . For microscopic system , researchers also use behavioral model [ 45 ] or hand-designed control laws [ 22 , 50 ] to control individual vehicles . Zhou et al . [ 56 ] propose SMARTS simulator to investigate the interaction of RL agents and social vehicles in atomic traffic scenes . Pal et al . [ 31 ] find the emergent road rules of MARL agents trained by simple independent learner , which are also shown in our experiments . In this work , we work on the traffic environments with complex real-world scenarios such as tollgate and parking lot . The traffic flow can contain more than 40 vehicles . In contrast to previous work , we also propose novel method to improve the collective motions that better simulate SDP systems . 3 Preliminaries . 3.1 Problem Setting . SDP system has two distinguishable properties : ( 1 ) each individual agent is self-interested , ( 2 ) the individual agent can vary to be cooperative or competitive with others within one episode and ( 3 ) the individual agent only actively interacts with its nearby agents . To accommodate these properties , we formulate a SDP system as a set of Decentralized Partially Observable Markov Decision Processes ( Dec-POMDPs ) [ 14 ] represented by a tuple G = < E , S , A , P , R , ρ0 , O , Z , γ > . E is the set of the agent indices . We define the active agent set at each environmental time step t as Et = { i1 , t , ... , ikt , t } ⊂ E wherein kt is the number of existing agents at this step . We consider a partially observable setting in which each agent can not access the environmental state st and instead draws individual observation oi , t according to the local observation function oi , t = Zi ( st ) : S → O . The joint action space At ⊂ A is composed of the union of all active agents ’ action spaces ⋃kt i=1Ai , t . Each agent i chooses its action ai , t ∈ Ai , t following its policy ai , t ∼ πi ( ·|ot ) . These actions form a joint action at = { ai , t } kti=1 ∈ At causing a transition on the environmental state subjected to the state transition distribution P ( st+1|st , at ) . Each agent receives their individual reward from its reward function ri , t = Ri ( st , at ) . ρ0 is the initial state distribution and γ is the discount factor . So in test time our system is decentralized , where each agent is fed with its local observation and actuated by its policy and interacts with others . Here we define the environmental episode τ = { ( st , at , r1 , t , ... , rkt , t ) } Tt=0 , where T is the environmental horizon . An environmental episode thus contains a set of agent episodes : { ( oi , t , ai , t , ri , t ) } tei t=tsi , wherein tsi and t e i denote the environmental steps when agent i enters and terminates , respectively . Suppose the policy πi is parameterized by θi , the individual objective for each agent is defined as the discounted sum of individual reward ( namely the individual return ) : JIi ( θi ) = Eτ [ Ri ( τ ) ] , wherein Ri ( τ ) = ∑tei t=tsi γt−t s i ri , t . 3.2 Individual Policy Optimization . To simulate SDP systems , a simple approach is the independent policy optimization ( IPO ) [ 35 ] where each agent maximizes individual objective as if in the single-agent environment . Denote the individual value function is V Ii , t = V I i ( st ) = E [ ∑tei t′=t γ ( t′−t ) ri , t ] and the corresponding advantage function is AIi , t = A I i ( st , ai , t , a¬i , t ) = ri , t + γV I i ( st+1 ) − V Ii ( st ) , wherein a¬i , t = { ai } kt j=1 , j 6=i is the shorthand of other agents ’ actions , the policy gradient method computes the gradient of individual objective as [ 42 ] : ∇θiJIi ( θi ) = E ( s , a ) [ ∇θi log πθi ( ai|s ) AIi ( s , ai , a¬i ) ] . ( 1 ) As a common practice , in PPO algorithm [ 37 ] the clipped importance sampling factor ρ = πi , new ( ai|s ) /πi , old ( ai|s ) is used to mitigate the distribution shift occurred after the policy is updated for several epochs , wherein πi , old is the behavior policy that generate samples and πi , new is the latest policy parameterized by θi . We call the resulting objective as the surrogate objective : ĴIi ( θi ) = E ( s , a ) min ( ρAIi , clip ( ρ , 1− , 1 + ) AIi ) . ( 2 ) is hyper-parameter . By conducting the stochastic gradient ascent on the surrogate objective w.r.t . the policy parameters , the expected individual return can be improved . In continuous control RL problem , both the state space and action space are high-dimensional . Therefore V Ii is approximated using a neural network whose input is typically the local observation . In the centralized training and decentralized execution ( CTDE ) framework [ 29 ] , the value function takes global information as input by concatenating all agents ’ local observations .

Summary: In this paper the authors propose a multi-agent reinforcement learning approach to finding controllers in self-driven particles systems. The problem of SDP is formulated as dec-POMDP where each agent has an individual reward function that is wishes to maximize using local observations only.  The algorithm consists of two components: local coordination relying on neighboring rewards and global coordination.   The local coordination part modifies the objective to be a combination of individual rewards and neighbor reward using a Local Coordination Factor. The LCF is used to compute a coordinated objective.   The global coordination component is responsible to find the best LCF, for tractability this objective is factorized into individual global objective.  The algorithm is evaluated on a set of traffic scenarios against multiple baselines: independent policies, mean field methods and curriculum learning.   The reward model using LCF and the split between local and global coordination is sound and interesting. However, the derivation lack of clarity, and it would have been useful to study the algorithm on other domains, especially given that the claim is on general SDP systems.   The authors should revise the claim formulation and provide more justification of the global coordination method.