Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization
Year: 2021

Abstract:
Self-Driven Particles (SDP) describe a category of multi-agent systems common in everyday life, such as flocking birds and traffic flows. In a SDP system, each agent pursues its own goal and constantly changes its cooperative or competitive behaviors with its nearby agents. Manually designing the controllers for such SDP system is time-consuming, while the resulting emergent behaviors are often not realistic nor generalizable. Thus the realistic simulation of SDP systems remains challenging. Reinforcement learning provides an appealing alternative for automating the development of the controller for SDP. However, previous multiagent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand, which fail to capture the essence of SDP where the role of each agent varies to be cooperative or competitive even within one episode. To simulate SDP with MARL, a key challenge is to coordinate agents’ behaviors while still maximizing individual objectives. Taking traffic simulation as the testing bed, in this work we develop a novel MARL method called Coordinated Policy Optimization (CoPO), which incorporates social psychology principle to learn neural controller for SDP. Experiments show that the proposed method can achieve superior performance compared to MARL baselines in various metrics. Noticeably the trained vehicles exhibit complex and diverse social behaviors that improve performance and safety of the population as a whole. Demo video and source code are available at: https://decisionforce.github.io/CoPO/.

heading: 1 Introduction
text: Self-Driven Particles (SDP) describe a wide range of multi-agent systems (MAS) in nature and human society. In SDP, individual agent pursues its own goal and interacts with each other following simple local alignment, and then the population exhibits complex collective behaviors [47]. We commonly see the phenomena of collective behaviors, such as the flocking birds [4], molecular motors [6], human crowd [17, 16], and the traffic system [20]. To understand and simulate such phenomena, researchers have developed a number of SDP models. For example, simple-rule based models [7] or Hydrodynamic equations based models [2, 19] can simulate the SDP very well in an unconstrained environment with random movement and resemble complex behaviors such as schooling fish [12] and marching locusts [5]. However, in a more structured environment such as a particular traffic scene where the interactions of agents are time-varying and the environment is non-stationary, it is difficult to design manual controllers or use rules to recover the underlying collective behaviors.
When the interactive environment is available, reinforcement learning becomes a promising approach to learn the controllers for actuating the SDP. Recently, many multi-agent reinforcement learning (MARL) methods have been developed to play competitive multi-player games, such as Hide and Seek [1], Football [23], Go and other board games [34], and StarCraft [33]. However, it is challenging to apply the existing MARL to simulate SDP systems. One essential issue is that each
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
constituent agent in a SDP system is self-interested and the relationship between agents is constantly changing. As illustrated in Fig. 1A, the vehicles in the traffic system demonstrate social behaviors that are either cooperative or competitive depending on the situation. Meanwhile, the cooperation and the competition naturally emerge as the outcomes of the self-driven instinct and the multiagent interaction. Thus it is difficult to generate those social behaviors through a top-down design. Furthermore, most of the MARL methods assume that the role of each agent is pre-determined and fixed within one episode. In cooperative MARL methods [32, 41, 9, 8], all agents need to cooperate to maximize a joint scalar reward. However, the credit decomposition problem centered in such setting is not the major issue of simulating SDP systems, because each agent in SDP has its individual objective and the credit decomposition is no longer needed. On the other hand, directly applying independent learning [43, 35] will lead to a group of agents with unreasonable behaviors that are aggressive or egocentric, since each agent is trained to maximize its own reward.
In order to simulate SDP systems, the coordination of the self-interested constituents is the major issue. The coordination of agents in MARL has been previously studied in the context of mixed cooperative and competitive tasks [25, 38]. However, seldom works focus on the simulation of SDP systems. Taking the microscopic traffic simulation as an example, existing works focus on learning high-level controller to indirectly actuate vehicles [48], or using independent learner to train policies executing in the environments engaging a limited set of agents on simple scenes [31, 56], without considering the coordination problem. Instead, we aim to control all the vehicles in the scene by operating on the low-level continuous control space. This allows a higher degree of freedom to learn diverse behaviors but poses a challenging continuous control problem. In this work we propose a novel MARL method called Coordinated Policy Optimization (CoPO) to facilitate the coordination of agents at both local and global levels. To evaluate the proposed method, as illustrated in Fig. 1, we construct five typical traffic environments as the testing bed. These environments contain rich interactions between agents and complex road structures. Besides, we develop three task-agnostic metrics to characterize different aspects of the learned populations. We show that the proposed CoPO method can achieve superior performance compared to the baselines. Noticeably, collective social behaviors that resemble real-world patterns emerge in the trained population.

heading: 2 Related Work
text: Multi-Agent Reinforcement Learning. Three typical task settings are explored in MARL [18]: the fully cooperative task, the fully competitive task, and the mixed cooperative and competitive task. Many works focus on the problems in cooperative tasks such as credit assignment [32, 41, 9, 8, 55] and communication [10, 46]. In competitive tasks, the challenges include modeling the opponents [11] and generating meaningful opponent via self-play [15, 49, 1]. In mixed cooperative and competitive task [29, 25, 38], the roles of the agents are mostly predefined and fixed at run-time. Instead, in this work we hope to simulate SDP systems without imposing any constraint on the roles of the agents thus there is more space for emergent social behaviors. Apart from the fixed role problem, previous
methods like MADDPG [29], QMIX [32], QTRAN [40], and COMA [9] are only applicable to discrete action space. We instead focus on learning policies for continuous control.
The independent PPO [35] and the Mean Field MARL [52] are two recent methods applied for continuous control in MARL. The Mean Field MARL method proposed by Yang et al. [52] approximates the value of global reward Q(s,a), a = [a1, ..., aN ] by agent-independent value functions Qi(s, ai, ā N i ), where ā N i is the average action of the nearby agents around agent i. Following similar design but in the context of simulating SDP, we use the mean field to compute neighborhood reward and use that reward to enforce local coordination, instead of using mean field to factorize joint value via averaging actions as done in [52].
Traffic simulation. Simulating the macroscopic and microscopic traffic systems has been studied for decades [21]. As for the macroscopic traffic system where the dynamics of individuals is simple and the high-level characters of flow such as the traffic throughput are more concerned, early works focus on designing controllers to direct a fleet of vehicles into a dense and stable platoon [39, 3]. CityFlow [53] and FLOW [51] use RL agents to steer the low-level controllers of vehicles in order to investigate the traffic phenomenon in large-scale macroscopic traffic flow. For microscopic system, researchers also use behavioral model [45] or hand-designed control laws [22, 50] to control individual vehicles. Zhou et al. [56] propose SMARTS simulator to investigate the interaction of RL agents and social vehicles in atomic traffic scenes. Pal et al. [31] find the emergent road rules of MARL agents trained by simple independent learner, which are also shown in our experiments. In this work, we work on the traffic environments with complex real-world scenarios such as tollgate and parking lot. The traffic flow can contain more than 40 vehicles. In contrast to previous work, we also propose novel method to improve the collective motions that better simulate SDP systems.

heading: 3 Preliminaries

heading: 3.1 Problem Setting
text: SDP system has two distinguishable properties: (1) each individual agent is self-interested, (2) the individual agent can vary to be cooperative or competitive with others within one episode and (3) the individual agent only actively interacts with its nearby agents.
To accommodate these properties, we formulate a SDP system as a set of Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) [14] represented by a tuple G =< E ,S,A,P,R, ρ0,O,Z, γ >. E is the set of the agent indices. We define the active agent set at each environmental time step t as Et = {i1,t, ..., ikt,t} ⊂ E wherein kt is the number of existing agents at this step. We consider a partially observable setting in which each agent can not access the environmental state st and instead draws individual observation oi,t according to the local observation function oi,t = Zi(st) : S → O. The joint action space At ⊂ A is composed of the union of all active agents’ action spaces ⋃kt i=1Ai,t. Each agent i chooses its action ai,t ∈ Ai,t following its policy ai,t ∼ πi(·|ot). These actions form a joint action at = {ai,t}kti=1 ∈ At causing a transition on the environmental state subjected to the state transition distribution P(st+1|st,at). Each agent receives their individual reward from its reward function ri,t = Ri(st,at). ρ0 is the initial state distribution and γ is the discount factor. So in test time our system is decentralized, where each agent is fed with its local observation and actuated by its policy and interacts with others.
Here we define the environmental episode τ = {(st,at, r1,t, ..., rkt,t)}Tt=0, where T is the environmental horizon. An environmental episode thus contains a set of agent episodes: {(oi,t, ai,t, ri,t)}
tei t=tsi , wherein tsi and t e i denote the environmental steps when agent i enters and
terminates, respectively. Suppose the policy πi is parameterized by θi, the individual objective for each agent is defined as the discounted sum of individual reward (namely the individual return): JIi (θi) = Eτ [Ri(τ)], wherein Ri(τ) = ∑tei t=tsi γt−t s i ri,t.

heading: 3.2 Individual Policy Optimization
text: To simulate SDP systems, a simple approach is the independent policy optimization (IPO) [35] where each agent maximizes individual objective as if in the single-agent environment. Denote the individual value function is V Ii,t = V I i (st) = E[ ∑tei t′=t γ (t′−t)ri,t] and the corresponding advantage
function is AIi,t = A I i (st, ai,t,a¬i,t) = ri,t + γV I i (st+1)− V Ii (st), wherein a¬i,t = {ai} kt j=1,j 6=i is the shorthand of other agents’ actions, the policy gradient method computes the gradient of individual objective as [42]:
∇θiJIi (θi) = E (s,a) [∇θi log πθi(ai|s)AIi (s, ai,a¬i)]. (1)
As a common practice, in PPO algorithm [37] the clipped importance sampling factor ρ = πi,new(ai|s)/πi,old(ai|s) is used to mitigate the distribution shift occurred after the policy is updated for several epochs, wherein πi,old is the behavior policy that generate samples and πi,new is the latest policy parameterized by θi. We call the resulting objective as the surrogate objective:
ĴIi (θi) = E (s,a) min(ρAIi , clip(ρ, 1− , 1 + )AIi ). (2)
is hyper-parameter. By conducting the stochastic gradient ascent on the surrogate objective w.r.t. the policy parameters, the expected individual return can be improved. In continuous control RL problem, both the state space and action space are high-dimensional. Therefore V Ii is approximated using a neural network whose input is typically the local observation. In the centralized training and decentralized execution (CTDE) framework [29], the value function takes global information as input by concatenating all agents’ local observations.

heading: 4 Coordinated Policy Optimization
text: In a SDP system each agent has its individual objective. However, if we simply maximize each individual reward, the system will have sub-optimal solutions where, for example, the agents will become aggressive and egocentric, jeopardizing the performance of the population and leading to critical failures. On the contrary, if we apply cooperative learning schemes [32, 41] and consider the summation of individual reward as the joint objective, the trained agents will exhibit unreasonable behaviors such as sacrificing oneself to improve group reward, which is not expected in SDP systems. To find a balance, as illustrated in Fig. 2 and Algorithm in the Appendix, we devise a novel MARL algorithm called Coordinated Policy Optimization (CoPO) to facilitate the bi-level coordination of agents to learn the controllers of the SDP systems. Following the CTDE framework, during centralized training, we first propose individual learning objectives by local coordination (Sec. 4.1), a mechanism inspired by the fact that each individual agent is mostly affected by its nearby agents [52]. We coordinate agents’ objectives in neighborhood by mixing rewards following a common social psychological principle. We further design the meta-learning technique to optimize the local coordination process, leading to the global coordination (Sec. 4.2) which can improve collective performance of the population.

heading: 4.1 Local Coordination
text: IPO only maximizes the individual objective (Eq. 1) which leads to egocentric policies that damage the overall performance of the population. However, in natural SDP systems, e.g. animal groups, there exists a certain level of cooperative behavior that not only benefits others, but also results better individual utility. To resemble such feature, we incorporate the ring measure of social value orientation [28, 38], a common metric from social psychology, into our formulation of local coordination. Concretely, we define the Local Coordination Factor (LCF) as a degree φ ∈ [−90◦, 90◦] describing an agent’s preference of being selfish, cooperative, or competitive.1 We then define the neighborhood reward as:
rNi,t =
∑ j∈Ndn (i,t) rj,t
|Ndn(i, t)| , wherein Ndn(i, t) = {j : ||Pos(i)− Pos(j)|| ≤ dn}. (3)
Ndn(i, t) defines the neighborhood of agent i within the radius dn at step t. Note that the neighborhood of agent is time-varying, therefore the the neighborhood reward can not be simply computed from a fixed set of agents across the episode. Weighted by LCF, the coordinated reward is defined as:
rCi,t = cos(φ)ri,t + sin(φ)r N i,t. (4)
1The social value orientation can be in range [−180◦, 180◦]. We only take the meaningful half of the range.
The LCF enables the local coordination in the optimization process. Supposing the agents maximize the coordinated reward with φ = 0◦, 90◦,−90◦, the agents should become egoistic, altruistic, or sadistic respectively. Illustratively, Fig. 3 demonstrates an interesting preliminary result on a simple version of Roundabout environment. During IPO training, we randomly select φ ∼ U(0◦, 90◦) and assign it to each agent in each episode. Current φ is fed to policy as extra observation. In test time, we run the trained population for multiple runs. In each run we set a fixed LCF for all agents. The test results show that a peak of success rate at certain LCF at test time. This suggests that increasing the awareness of neighbors’ interests when training individual agents will propose better policies that increase the collective performance. IPO might lead to sub-optimal policies since the individual objectives never consider local coordination with other agents.
In order to incorporate the LCF into training process to improve the population performance, apart from the value function approximator for the individual reward (Sec. 3.2), we use another value function to approximate the discounted sum of the neighborhood reward V Ni,t = E[ ∑tei t γ
t−tsi rNi,t] and the neighborhood advantage is therefore calculated as: ANi,t = r N i,t + γV N i,t+1 − V Ni,t . The coordinated advantage can be derived as: ACΦ,i,t = cos(φ)A I i,t + sin(φ)A N i,t. We now have the coordinated objective, which takes the utilities of neighbors into consideration, as follows:
JCi (θi,Φ) = E (s,a) [min(ρACΦ,i, clip(ρ, 1− , 1 + )ACΦ,i)]. (5)
We follow a similar training procedure as IPO in Sec. 3.2 with the advantage replaced by the coordinated advantage. To improve diversity, we use distributional LCF φ ∼ N (φµ, φ2σ), where Φ = [φµ, φσ] > are the learnable parameters of the LCF distribution.

heading: 4.2 Global Coordination
text: The major challenge of local coordination is the selection of LCF. Though the experiment in Fig. 3 shows that there exists an optimal LCF that can maximize the population success rate, it is laborious to search for such LCF. We therefore introduce global coordination which enables automatic search of the best LCF distribution to improve population efficiency.
In this section, we introduce how we accomplish global coordination with bi-level optimization method that computes a meta-gradient of the global objective. We first define the global objective that indicates the performance of a population:
JG(θ1, θ2, ...) = E τ [ ∑ i∈Eτ tei∑ t=tsi ri,t] = E τ [ T∑ t=0 ∑ i∈Eτ,t ri,t], (6)
wherein Eτ,t denotes the active agents at environmental time step t of the episode τ . Here we remove γ since it is ambiguous when the start steps of agents are not aligned. Eq. 6 can not be directly optimized when computing policy gradients of individual policies’ parameters.
We introduce the concept of individual global objective to make the optimization of the global objective feasible. By this way, we can turn the system-level optimization process into agent-level optimization processes, so that the individual agent’s data can be easily accessed. We factorize
JG(θ1, ...) to individual global objectives JGi (θi) such that maximizing the objective for each agent i is equivalent to maximizing Eq. 6. The individual global objective is defined as:
JGi (θi|θ1, ...) = E τ [ tei∑ t=tsi
∑ j∈Eτ,t rj,t
|Eτ,t| ]. (7)
If we introduce the global reward rG = ∑ j∈Eτ,t rj,t/|Eτ,t| as the average reward of all active agents at step t, then this objective is the cumulative global reward in the time interval when agent i is alive.
Proposition 1 (Global objective factorization). Suppose each agent i ∈ E can maximize its individual global objective JGi , then the original global objective J G is maximized.
We prove that such factorization does not affect the optimality of global objective in Appendix.
To improve global objective via optimizing LCF, we need to compute the gradient of Eq. 7 w.r.t. Φ. Denote the parameters of policies before and after optimizing Eq. 5 as θoldi and θ new i respectively, we can compute the gradient applying the chain rule as:
∇ΦJGi (θnewi ) = ∇θnewi J G i (θ new i )︸ ︷︷ ︸
1st Term
∇Φθnewi︸ ︷︷ ︸ 2nd Term . (8)
The 1st term resembles the policy gradient as Eq. 2 with the objective replaced by JGi :
1st Term = E (s,a)∼θoldi [∇θnewi min(ρA G(s,a), clip(ρ, 1− , 1 + )AG(s,a))]. (9)
Here we use an extra global value function V G to estimate the value of global reward rG, then compute the global advantage AG. The samples (s,a) are generated by the behavior policy θoldi .
By following first-order Taylor expansion, the 2nd term can be computed as follows. Note that ∇θoldi J C i (θ old i ,Φ) recovers the vanilla policy gradient in Eq. 1.
2nd Term = ∇Φ(θoldi + α∇θoldi J C i (θ old i ,Φ)) = α E (s,a)∼θoldi [∇θoldi log πθoldi (ai|s)∇ΦA C Φ,i]. (10)
Averaging over all agents and combining Eq. 9 and Eq. 10, we can derive the LCF objective as:
JLCF (Φ) = E i,(s,a)∼θoldi [∇θnewi min(ρA G, clip(ρ, 1− , 1 + )AG)][∇θoldi log πθoldi (ai|s)]A C Φ,i. (11)
We then conduct stochastic gradient ascent on JLCF w.r.t. Φ with learning rate α.
The detailed procedure of CoPO is given in Appendix. In our implementation, we follow the commonly used parameter sharing technique [44] so that all agents share the same set of neural networks, which guarantees the scalability of our method. As a summary, our method totally employs 4 neural networks: the policy network, the individual value network, the neighborhood value network, and the global value network. The input to all networks is always the local observation of the agent. Our further study (see Sec. 5.5) shows that centralized critic method, which feeds global information to the value networks, does not yield better performance and usually causes unstable training. The reason might be the time-varying input to the centralized critics, that the combined observation of nearby agents is always changing and doesn’t hold a temporal consistency across the episodes.

heading: 5 Experiments

heading: 5.1 Evaluation Platform
text: We develop a set of 3D traffic environments to evaluate MARL methods for simulating SDP systems based on MetaDrive [26], a lightweight and efficient microscopic driving simulator2. The environments are developed in Panda3D [13] and Bullet Engine, which can run 80 FPS on a standard
2MetaDrive can be found at: https://github.com/decisionforce/metadrive.
workstation. Examples of the environments are shown in Fig. 1. The descriptions and typical settings of the five traffic environments are as follows:
Roundabout: A four-way roundabout with two lanes. 40 vehicles spawn during environment reset. This environment includes merge and split junctions.
Intersection: An unprotected four-way intersection allowing bi-directional traffic as well as U-turns. Negotiation and social behaviors are expected to solve this environment. We initialize 30 vehicles.
Tollgate: Tollgate includes narrow roads to spawn agents and ample space in the middle with multiple tollgates. The tollgates create static obstacles where the crashing is prohibited. We force agent to stop at the middle of tollgate for 3s. The agent will fail if they exit the tollgate before it is allowed to pass. 40 vehicles are initialized. Complex behaviors such as deceleration and queuing are expected. Additional states such as whether vehicle is in tollgate and whether the tollgate is blocked are given.
Bottleneck: Complementary to Tollgate, Bottleneck contains a narrow bottleneck lane in the middle that forces the vehicles to yield to others. We initialize 20 vehicles.
Parking Lot: A compact environment with 8 parking slots. Spawn points are scattered in both parking lots and external roads. 10 vehicles spawn initially. In this environment, we allow agents to back their cars to spare space for others. Maneuvering and yielding are the key to solve this task.
Those environments cover several scenarios in previous simulators [56, 31] but are extended with real-world scenarios rather than atomic scenes. Besides, our environments support dense traffic flow where all vehicles are controlled through the low-level continuous signals. All the environments can be further concatenated to form more complex tasks for the future study of multi-task learning.
Evaluating a population simulated by MARL is an open question. In this work, we define a set of general-purpose metrics that can characterize different aspects of the SDP systems. We choose three episodic metrics with minimal assumptions on the tasks. We evaluate the competence of population with success rate. It is the ratio of the number of agents that reach their goals over the total number of agents in one episode. We also evaluate the efficiency of the population. It indicates the difference between successes and failures in a unit of time (Nsuccess −Nfailure)/T . It is possible for a population to achieve high success rate but has low efficiency, because the agents in the population are running in very low velocity. The third metric we adopt here is the safety. It is defined as the total number of critical failures for all agents in one episode.

heading: 5.2 Experiment Setting
text: We compare the independent policy optimization (IPO) method [35] using PPO [37] as the individual learners. We further encode the nearby agents’ states into the input of value functions (centralized critics) following the idea of Mean Field MARL [52] and form the mean field policy optimization (MFPO). We examine different variants of centralized critics, but find MFPO yields best performance (see Sec. 5.5). The curriculum learning (CL) [30] is also a baseline, where we chunk the training into 4 phases and gradually increase initial agents in each episode from 25% to 100% of the target number.
We conduct experiments with the aforementioned environments and algorithms using RLLib [27], a distributed learning system which allows large-scale parallel experiments. Generally, we host 4 concurrent trials in an Nvidia GeForce RTX 2080 Ti GPU. Each trial consumes 2 CPUs with 4 parallel rollout workers. Each trial is trained over roughly 1M environmental steps, which corresponds to about 55 hours in real-time transportation system and +2,000 hours of individual driving experience (assume averagely 40 vehicles running concurrently). All experiments are repeated 8 times with different random seeds. Other hyper-parameters are given in Appendix.

heading: 5.3 Main Results
text: Table 1 shows the main results of all the evaluated methods. Because of the bi-level coordination in policy optimization, our CoPO achieves superior success rate in all 5 environments. Noticeably in the most difficult environment, the unprotected Intersection, our method outperforms baselines with a substantial margin. That environment requires agents to cooperate frequently in order to determine who should go fist. The populations generated from other methods cause severe congestion in the
middle of intersection which prevents any agent going through. See Sec. 5.4 and demo video for detailed visualization.
We also find the MFPO performs worse than the independent PPO in several environments. That might be due to the neighborhood changing all the time. Simply concatenating or averaging neighbors’ states as the additional input to the value functions makes the training unstable. A detailed comparison of different design choices in centralized critics is given in Sec. 5.5.
Fig. 4 presents the radar plots under the three metrics. In each environment, we normalize each metric to its maximal and minimal values in all evaluated episodes for all algorithms. For safety measurement, we use negative values of total crashes in normalization. We can see that CoPO achieves superior results across all three metrics. Noticeably, CoPO has a substantial improvement in safety, showing the importance of the bi-level coordination.
To further evaluate the generalization of the populations, we run the trained policies in the test environments with different initial numbers of agents. As shown in Fig. 5, we find that in Roundabout and Intersection, where the interactions are extremely intensive, IPO and MFPO overfit to the initial number of agents. In the environment with sparse traffic, their performance is inferior to that in the environment with more agents. On the contrary, our method does not overfit these two environments. In Parking Lot, it seems that all methods has some degree of overfitting to the initial number of agents. The results are consistent with the finding in [24], showing that strategies learned via a single instance of MARL algorithm can overfit to the policies of other agent in the population. In Appendix, we conduct an experiment using a simple heuristic to control part of the vehicles in the population to test the generalizability of different training methods. The experiment shows that CoPO trained agents are robuster than IPO trained ones when interacting with vehicles controlled by unseen policies. It is left to future work on improving the generalization of MARL methods.

heading: 5.4 Behavioral Analysis
text: Fig. 6 plots the visualization of all trajectories occurred in one environmental episode, where the agents spawning at same entries share the same colors. The population trained from CoPO exhibits collective behaviors, while the critical failures reduce drastically compared to other populations. The
population trained from other methods tend to directly rush into the center of Intersection, causing severe congestion and frequent crashes. On the contrary, the population from CoPO tends to drive collectively and successfully avoid crashes through social behaviors such as yielding, with superior success rate and high safety. The qualitative result suggests that CoPO can resemble the collective motions widely existing in SDP systems [54].
Fig. 7 further visualizes the temporal behaviors of CoPO agents in 5 environments. Each vehicle in the population tends to drive socially and react to its nearby vehicles. A set of diverse behaviors emerge through the interactions, including socially compliant behaviors such as yielding, queuing, and even reversing to leave more room to others, as well as aggressive behaviors such as cutting in and rushing. The population trained from CoPO successfully reproduces diverse interaction behaviors in traffic systems.

heading: 5.5 Ablation Studies
text: We evaluate different variants of centralized critics in Roundabout environment. The Concat variant refers to simply concatenating all states of nearest K=4 agents as a long vector fed to the value networks. The Mean Field method uses the average of the states from nearby agents within a given radius (we use 10 meters). The CF refers to “counterfactual”, namely adding the neighbors’ actions accompanied with their observations as the input to value functions. Note that the Mean Field w/ CF method follows the Mean Field Actor Critic proposed in [52], the difference is that we use the training pipeline of PPO [36] in order to operate in continuous action space. As illustrated in Fig. 9, we find that the design with Mean Field is more stable compared to Concat. CF also stabilizes the training. We therefore report the results of Mean Field w/ CF as the MFPO method in Sec. 5.3.
Fig. 8 plots the learning curves of CoPO with and without centralized critics. We find that CoPO with centralized critics yields inferior success rate and is unstable in some environments compared to the native CoPO. This might because nearby agents are always changing and doesn’t hold the temporal consistency within episode.
(a) sample φ fromN (0, 0.12) 62.38 ±7.26 70.23 ±2.72 60.47 ±5.80 71.29 ±3.27 59.14 ±1.29 (b) maximize global reward 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 (c) maximize neighborhood reward 65.70 ± 2.21 66.83 ± 2.16 57.30 ± 3.46 71.62 ± 1.22 6.34 ± 1.44 CoPO: sample φ fromN (φµ, φ2σ) 73.67 ±3.71 78.97 ±4.23 86.13 ±1.76 79.68 ±2.91 65.04 ±1.59
In Table 2(a), we validate the effectiveness of global coordination by replacing the LCF distribution with a given normal distribution during the course of training. The proposed method can yield better success rate than the result of manually defined fixed LCF distributions. In (b), we use IPO to maximize the global reward directly. Unfortunately, the training fails because it is hard to build the causal connection between individual’s behavior with the average reward of all agents. In (c), we use IPO to maximize the neighborhood reward. This method performs poorer than CoPO since we balance the individual reward and the neighborhood reward.

heading: 6 Conclusion
text: In this work, we develop a novel MARL method called Coordinated Policy Optimization (CoPO) to incorporate social psychology principle to learn neural controller of SDP systems. Experiments on microscopic traffic simulation show that the proposed method can learn populations which achieve superior performance over the previous MARL methods in three general-purpose metrics. Interestingly, we find the trained vehicles exhibits complex and socially compliant behaviors that improve the efficiency and safety of the population as a whole.

heading: 7 Social Impact and Limitations
text: This work could help the development of intelligent transportation systems which would have a wide impact on society. We can analyze the emergent behaviors of the traffic under different scene structures and optimize the road structure or traffic light control to improve the traffic efficiency. Moreover, CoPO can simulate diverse traffic flow, which can be used to benchmark the generalizability of the autonomous driving systems. Besides simulating traffic flow, the proposed method is applicable for pedestrian crowd simulation to study social crowd behaviors as well as potential human stampedes and crushes.
This work focuses on decision-making problem, so we simplify the perception of the vehicles as one-channel LiDAR and assume the acquisition of accurate sensory data. In reality, the precise perception of the surroundings in self-driving remains very challenging. This work adopts 5 typical traffic scenarios in a simulator as the testbed, which are still far from emulating the complexity of real-world traffic scenes. In the future we will import real-world HD maps in the simulator to create more realistic scenarios.
Acknowledgments and Disclosure of Funding
This project was supported by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Fund.

Review summaries:
In this paper the authors propose a multi-agent reinforcement learning approach to finding controllers in self-driven particles systems. The problem of SDP is formulated as dec-POMDP where each agent has an individual reward function that is wishes to maximize using local observations only.  The algorithm consists of two components: local coordination relying on neighboring rewards and global coordination.   The local coordination part modifies the objective to be a combination of individual rewards and neighbor reward using a Local Coordination Factor. The LCF is used to compute a coordinated objective.   The global coordination component is responsible to find the best LCF, for tractability this objective is factorized into individual global objective.  The algorithm is evaluated on a set of traffic scenarios against multiple baselines: independent policies, mean field methods and curriculum learning.   The reward model using LCF and the split between local and global coordination is sound and interesting. However, the derivation lack of clarity, and it would have been useful to study the algorithm on other domains, especially given that the claim is on general SDP systems.   The authors should revise the claim formulation and provide more justification of the global coordination method. 

The paper proposes a coordinated policy optimization algorithm to simulate self-driven particles and showcases its power in the autonomous driving application of traffic simulation. The algorithm proposes both local and global coordination mechanisms to tradeoff cooperation and competitiveness at the scene-level. The paper benchmarks the efficiency and safety of their method against a few baselines on a newly proposed simulator of heavily interactive scenarios.

The authors introduce a novel MARL learning algorithm, CoPO, that is designed to facilitate coordination in SDP systems. In addition to independent value functions, CoPO learns a neighborhood value function according to a return based on the normalized sum of neighboring agent rewards.  Both independent and neighborhood returns are mixed together according to a popular measure of social value orientation. Using a centralized bi-level optimization process, the agent social value distribution is optimised at the same time as the optimal low-level continuous decentralized control policies. The authors evaluate CoPO on a testbed of 5 common road traffic coordination tasks and according to three novel collective metrics. CoPO is shown to outperform other state-of-the-art CTDE MARL algorithms. 

The authors aim to design controllers for traffic simulation agents in the mixed-motive reinforcement learning setting. They introduce a smooth interpolation between individual and mean global reward (social value orientation) which agents locally optimize while using a meta-optimization process to optimize the degree of interpolation. They show that in a suite of 5 traffic scenarios, their method generally produces agent populations that succeed at higher rates, are more efficient, and produce a smaller amount of critical failures.