Learning to Simulate Self-driven Particles System with Coordinated Policy Optimization

1 Introduction . Self-Driven Particles ( SDP ) describe a wide range of multi-agent systems ( MAS ) in nature and human society . In SDP , individual agent pursues its own goal and interacts with each other following simple local alignment , and then the population exhibits complex collective behaviors [ 47 ] . We commonly see the phenomena of collective behaviors , such as the flocking birds [ 4 ] , molecular motors [ 6 ] , human crowd [ 17 , 16 ] , and the traffic system [ 20 ] . To understand and simulate such phenomena , researchers have developed a number of SDP models . For example , simple-rule based models [ 7 ] or Hydrodynamic equations based models [ 2 , 19 ] can simulate the SDP very well in an unconstrained environment with random movement and resemble complex behaviors such as schooling fish [ 12 ] and marching locusts [ 5 ] . However , in a more structured environment such as a particular traffic scene where the interactions of agents are time-varying and the environment is non-stationary , it is difficult to design manual controllers or use rules to recover the underlying collective behaviors . When the interactive environment is available , reinforcement learning becomes a promising approach to learn the controllers for actuating the SDP . Recently , many multi-agent reinforcement learning ( MARL ) methods have been developed to play competitive multi-player games , such as Hide and Seek [ 1 ] , Football [ 23 ] , Go and other board games [ 34 ] , and StarCraft [ 33 ] . However , it is challenging to apply the existing MARL to simulate SDP systems . One essential issue is that each 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . constituent agent in a SDP system is self-interested and the relationship between agents is constantly changing . As illustrated in Fig . 1A , the vehicles in the traffic system demonstrate social behaviors that are either cooperative or competitive depending on the situation . Meanwhile , the cooperation and the competition naturally emerge as the outcomes of the self-driven instinct and the multiagent interaction . Thus it is difficult to generate those social behaviors through a top-down design . Furthermore , most of the MARL methods assume that the role of each agent is pre-determined and fixed within one episode . In cooperative MARL methods [ 32 , 41 , 9 , 8 ] , all agents need to cooperate to maximize a joint scalar reward . However , the credit decomposition problem centered in such setting is not the major issue of simulating SDP systems , because each agent in SDP has its individual objective and the credit decomposition is no longer needed . On the other hand , directly applying independent learning [ 43 , 35 ] will lead to a group of agents with unreasonable behaviors that are aggressive or egocentric , since each agent is trained to maximize its own reward . In order to simulate SDP systems , the coordination of the self-interested constituents is the major issue . The coordination of agents in MARL has been previously studied in the context of mixed cooperative and competitive tasks [ 25 , 38 ] . However , seldom works focus on the simulation of SDP systems . Taking the microscopic traffic simulation as an example , existing works focus on learning high-level controller to indirectly actuate vehicles [ 48 ] , or using independent learner to train policies executing in the environments engaging a limited set of agents on simple scenes [ 31 , 56 ] , without considering the coordination problem . Instead , we aim to control all the vehicles in the scene by operating on the low-level continuous control space . This allows a higher degree of freedom to learn diverse behaviors but poses a challenging continuous control problem . In this work we propose a novel MARL method called Coordinated Policy Optimization ( CoPO ) to facilitate the coordination of agents at both local and global levels . To evaluate the proposed method , as illustrated in Fig . 1 , we construct five typical traffic environments as the testing bed . These environments contain rich interactions between agents and complex road structures . Besides , we develop three task-agnostic metrics to characterize different aspects of the learned populations . We show that the proposed CoPO method can achieve superior performance compared to the baselines . Noticeably , collective social behaviors that resemble real-world patterns emerge in the trained population . 2 Related Work . Multi-Agent Reinforcement Learning . Three typical task settings are explored in MARL [ 18 ] : the fully cooperative task , the fully competitive task , and the mixed cooperative and competitive task . Many works focus on the problems in cooperative tasks such as credit assignment [ 32 , 41 , 9 , 8 , 55 ] and communication [ 10 , 46 ] . In competitive tasks , the challenges include modeling the opponents [ 11 ] and generating meaningful opponent via self-play [ 15 , 49 , 1 ] . In mixed cooperative and competitive task [ 29 , 25 , 38 ] , the roles of the agents are mostly predefined and fixed at run-time . Instead , in this work we hope to simulate SDP systems without imposing any constraint on the roles of the agents thus there is more space for emergent social behaviors . Apart from the fixed role problem , previous methods like MADDPG [ 29 ] , QMIX [ 32 ] , QTRAN [ 40 ] , and COMA [ 9 ] are only applicable to discrete action space . We instead focus on learning policies for continuous control . The independent PPO [ 35 ] and the Mean Field MARL [ 52 ] are two recent methods applied for continuous control in MARL . The Mean Field MARL method proposed by Yang et al . [ 52 ] approximates the value of global reward Q ( s , a ) , a = [ a1 , ... , aN ] by agent-independent value functions Qi ( s , ai , ā N i ) , where ā N i is the average action of the nearby agents around agent i . Following similar design but in the context of simulating SDP , we use the mean field to compute neighborhood reward and use that reward to enforce local coordination , instead of using mean field to factorize joint value via averaging actions as done in [ 52 ] . Traffic simulation . Simulating the macroscopic and microscopic traffic systems has been studied for decades [ 21 ] . As for the macroscopic traffic system where the dynamics of individuals is simple and the high-level characters of flow such as the traffic throughput are more concerned , early works focus on designing controllers to direct a fleet of vehicles into a dense and stable platoon [ 39 , 3 ] . CityFlow [ 53 ] and FLOW [ 51 ] use RL agents to steer the low-level controllers of vehicles in order to investigate the traffic phenomenon in large-scale macroscopic traffic flow . For microscopic system , researchers also use behavioral model [ 45 ] or hand-designed control laws [ 22 , 50 ] to control individual vehicles . Zhou et al . [ 56 ] propose SMARTS simulator to investigate the interaction of RL agents and social vehicles in atomic traffic scenes . Pal et al . [ 31 ] find the emergent road rules of MARL agents trained by simple independent learner , which are also shown in our experiments . In this work , we work on the traffic environments with complex real-world scenarios such as tollgate and parking lot . The traffic flow can contain more than 40 vehicles . In contrast to previous work , we also propose novel method to improve the collective motions that better simulate SDP systems . 3 Preliminaries . 3.1 Problem Setting . SDP system has two distinguishable properties : ( 1 ) each individual agent is self-interested , ( 2 ) the individual agent can vary to be cooperative or competitive with others within one episode and ( 3 ) the individual agent only actively interacts with its nearby agents . To accommodate these properties , we formulate a SDP system as a set of Decentralized Partially Observable Markov Decision Processes ( Dec-POMDPs ) [ 14 ] represented by a tuple G = < E , S , A , P , R , ρ0 , O , Z , γ > . E is the set of the agent indices . We define the active agent set at each environmental time step t as Et = { i1 , t , ... , ikt , t } ⊂ E wherein kt is the number of existing agents at this step . We consider a partially observable setting in which each agent can not access the environmental state st and instead draws individual observation oi , t according to the local observation function oi , t = Zi ( st ) : S → O . The joint action space At ⊂ A is composed of the union of all active agents ’ action spaces ⋃kt i=1Ai , t . Each agent i chooses its action ai , t ∈ Ai , t following its policy ai , t ∼ πi ( ·|ot ) . These actions form a joint action at = { ai , t } kti=1 ∈ At causing a transition on the environmental state subjected to the state transition distribution P ( st+1|st , at ) . Each agent receives their individual reward from its reward function ri , t = Ri ( st , at ) . ρ0 is the initial state distribution and γ is the discount factor . So in test time our system is decentralized , where each agent is fed with its local observation and actuated by its policy and interacts with others . Here we define the environmental episode τ = { ( st , at , r1 , t , ... , rkt , t ) } Tt=0 , where T is the environmental horizon . An environmental episode thus contains a set of agent episodes : { ( oi , t , ai , t , ri , t ) } tei t=tsi , wherein tsi and t e i denote the environmental steps when agent i enters and terminates , respectively . Suppose the policy πi is parameterized by θi , the individual objective for each agent is defined as the discounted sum of individual reward ( namely the individual return ) : JIi ( θi ) = Eτ [ Ri ( τ ) ] , wherein Ri ( τ ) = ∑tei t=tsi γt−t s i ri , t . 3.2 Individual Policy Optimization . To simulate SDP systems , a simple approach is the independent policy optimization ( IPO ) [ 35 ] where each agent maximizes individual objective as if in the single-agent environment . Denote the individual value function is V Ii , t = V I i ( st ) = E [ ∑tei t′=t γ ( t′−t ) ri , t ] and the corresponding advantage function is AIi , t = A I i ( st , ai , t , a¬i , t ) = ri , t + γV I i ( st+1 ) − V Ii ( st ) , wherein a¬i , t = { ai } kt j=1 , j 6=i is the shorthand of other agents ’ actions , the policy gradient method computes the gradient of individual objective as [ 42 ] : ∇θiJIi ( θi ) = E ( s , a ) [ ∇θi log πθi ( ai|s ) AIi ( s , ai , a¬i ) ] . ( 1 ) As a common practice , in PPO algorithm [ 37 ] the clipped importance sampling factor ρ = πi , new ( ai|s ) /πi , old ( ai|s ) is used to mitigate the distribution shift occurred after the policy is updated for several epochs , wherein πi , old is the behavior policy that generate samples and πi , new is the latest policy parameterized by θi . We call the resulting objective as the surrogate objective : ĴIi ( θi ) = E ( s , a ) min ( ρAIi , clip ( ρ , 1− , 1 + ) AIi ) . ( 2 ) is hyper-parameter . By conducting the stochastic gradient ascent on the surrogate objective w.r.t . the policy parameters , the expected individual return can be improved . In continuous control RL problem , both the state space and action space are high-dimensional . Therefore V Ii is approximated using a neural network whose input is typically the local observation . In the centralized training and decentralized execution ( CTDE ) framework [ 29 ] , the value function takes global information as input by concatenating all agents ’ local observations .

Summary: The authors introduce a novel MARL learning algorithm, CoPO, that is designed to facilitate coordination in SDP systems. In addition to independent value functions, CoPO learns a neighborhood value function according to a return based on the normalized sum of neighboring agent rewards.  Both independent and neighborhood returns are mixed together according to a popular measure of social value orientation. Using a centralized bi-level optimization process, the agent social value distribution is optimised at the same time as the optimal low-level continuous decentralized control policies. The authors evaluate CoPO on a testbed of 5 common road traffic coordination tasks and according to three novel collective metrics. CoPO is shown to outperform other state-of-the-art CTDE MARL algorithms.