Learning to Simulate Self-driven Particles System with Coordinated Policy Optimization
Authors: Zhenghao Peng
Quanyi Li
Ka Ming Hui
Chunxiao Liu
Bolei Zhou
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multi-agent Reinforcement Learning
Traffic Simulation

Abstract: Self-Driven Particles (SDP) describe a category of multi-agent systems common in everyday life, such as flocking birds and traffic flows. In a SDP system, each agent pursues its own goal and constantly changes its cooperative or competitive behaviors with its nearby agents. Manually designing the controllers for such SDP system is time-consuming, while the resulting emergent behaviors are often not realistic nor generalizable. Thus the realistic simulation of SDP systems remains challenging. Reinforcement learning provides an appealing alternative for automating the development of the controller for SDP. However, previous multi-agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand, which fail to capture the essence of SDP where the role of each agent varies to be cooperative or competitive even within one episode. To simulate SDP with MARL, a key challenge is to coordinate agents' behaviors while still maximizing individual objectives. Taking traffic simulation as the testing bed, in this work we develop a novel MARL method called Coordinated Policy Optimization (CoPO), which incorporates social psychology principle to learn neural controller for SDP. Experiments show that the proposed method can achieve superior performance compared to MARL baselines in various metrics. Noticeably the trained vehicles exhibit complex and diverse social behaviors that improve performance and safety of the population as a whole. Demo video and source code are available at: https://decisionforce.github.io/CoPO/

Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization
Zhenghao Peng", Quanyi Li Ka Ming Hui", Chunxiao Liu', Bolei Zhou'
*The Chinese University of Hong Kong, 'SenseTime Research Cenntee for Perceptual and Interactive Intelligence
Abstract
Self-Driven Particles (SDP) describe category of multi-agent systems common in everyday life, such as flocking birds and traffic flows. SDP system, each agent pursues its own goal and constantly changes its cooperative competitive behaviors with its nearby agents. Manually designing the controllers for such SDP system is time-consuming while the resulting emergent behaviors often not realistic nor generalizable. Thus the realistic simulation of SDP systems remains challenging. Reinforcement learning provides appealing alternative for automating the development of the controller for SDP. However, previous multi- agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand, which fail to capture the essence of SDP where the role of each agent varies to be cooperative or competitive even within one episode. To simulate SDP with MARL, a key challenge is to coordinate agents' behaviors while still maximizing individual objectives. Taking traffic simulation the testing bed. in this work we develop a novel MARL method called Coordinated Policy Optimization (CoPO). which incorporates social psychology principle to learn neural controller for SDP. Experiments show that the proposed method can achieve superior performance compared MARL baselines in various metrics. Noticeably the trained vehicles exhibit complex and diverse social behaviors that improve performance and safety of the population as a whole. Demo video and source code are available at: httt::://decisieeeee
1 Introduction
Self-Driven Particles (SDP) describe a wide range of multi-agent systems (MAS) in nature and human society. In SDP, individual agent pursues its own goal and interacts with each other following simple local alignment, and then the population exhibits complex collective behaviors [47]. We commonly the phenomena of collective behaviors, such the flocking birds 4 molecular motors (6).
human crowd 17 16 and the traffic system (20.. To understand and simulate such phenomena, researchers have developed number of SDP models. For example, simple-rule based models (7) Hydrodynamic equations based models 2 19 can simulate the SDP very well in an unconstrained environment with random movement and resemble complex behaviors such schooling fish [12] and marching locusts (5). However, in a more structured environment such as particular traffic scene where the interactions of agents are time-varying and the environment non-stationary, is difficult to design manual controllers or use rules to recover the underlying collective behaviors.
When the interactive environment is available, reinforcement learning becomes promising approach to learn the controllers for actuating the SDP. Recently, many multi-agent reinforcement learning (MARL) methods have been developed
play competitive multi-player games, such Hide and Seek [I) Football (3), Go and other board games 34,, and StarCraft [33] However, it is challenging to apply the existing MARL to simulate SDP systems. One essential issue is that each
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Competition
Cooperation
Pypesseng
Intersection
Tollgate
Roundabout
Bottleneck
Parking Lot
The five environments used in our evaluation. We highlight two cooperative and competitive events the vehicle interactions.
Figure I:
constituent agent in a SDP system self-interested and the relationship between agents constantly changing. As illustrated in Fig. [A the vehicles in the traffic system demonstrate social behaviors that are either cooperative or competitive depending on the situation. Meanwhile, the cooperation and the competition naturally emerge the outcomes of the self-driven instinct and the multi- agent interaction. Thus is difficult to generate those social behaviors through a top-down design. Furthermore, most of the MARL methods assume that the role of each agent is pre-determined and fixed within one episode. In cooperative MARL methods [3 9 to maximize joint scalar reward. However, the credit decomposition problem centered in such setting is not the major issue of simulating SDP systems, because each agent SDP has its individual objective and the eredit decomposition no longer needed. On the other hand, directly applying independent learning
8 all agents need to cooperate
will lead to a group of agents with unreasonable behaviors that are aggressive or egocentric, since each agent is trained to maximize its own reward.
43 351
In order to simulate SDP systems, the coordination of the self-interested constituents is the major issue. The coordination of agents in MARL has been previously studied in the context of mixed cooperative and competitive tasks 225 38). However, seldom works focus on the simulation SDP systems. Taking the microscopic traffic simulation
example, existing works focus learning high-level controller to indirectly actuate vehicles [48], or using independent learner to train policies executing in the environments engaging limited set of agents simple scenes K1 56), without considering the coordination problem. Instead, we aim to control all the vehicles in the scene by operating the low-level continuous control space. This allows higher degree of freedom to learn diverse behaviors but poses a challenging continuous control problem. In this work we propose novel MARL method called Coordinated Policy Optimization (CoPO) facilitate the coordination of agents at both local and global levels. To evaluate the proposed method, as illustrated in Fig. [ we construct five typical traffic environments as the testing bed. These environments contain rich interactions between agents and complex road structures. Besides, we develop three aagn-tgoottic metrics to characterize different aspects of the learned populations. We show that the proposed CoPO method can achieve superior performance compared to the basclines. Noticeably, collective social behaviors that resemble real-world patterns emerge the trained population.
2
Related Work
Multi-Agent Reinforcement Learning. Three typical task settings are explored in MARL [8] the fully cooperative task, the fully competitive task, and the mixed cooperative and competitive task. Many works focus on the problems in cooperative tasks such as credit assignment [32
551 and
communication [10 [66]. In competitive tasks, the challenges include modeling the opponents [I]] and generating meaningful opponent via self-play [15 I. mixed cooperative and competitive task [29] 25 38 the roles of the agents are mostly predefined and fixed at run-time. Instead, in this work we hope to simulate SDP systems without imposing any constraint on the roles of the agents thus there is more space for emergent social behaviors. Apart from the fixed role problem, previous

---

methods like MADDPG [29], QMIX B2) QTRAN 40] and COMA 9 are only applicable to discrete action space. We instead focus on learning policies for continuous control.
The independent PPO [35] and the Mean Field MARL 152 are two recent methods applied for continuous control in MARI.. The Mean Field MARL method proposed by Yang et al. (52) approx- imates the value of global reward Q(s, a), a = [a1...., an] by agent-independent value functions Qi a4 a. where aj is the average action of the nearby agents around agent i. Following similar design but in the context of simulating SDP, we use the mean field to compute neighborhood reward
and use that reward to enforce local coordination, instead of using mean field to factorize joint value via averaging actions as done in 12
Traffic simulation. Simulating the macroscopic and microscopic traffic systems has been studied for decades [21] As for the macroscopic traffic system where the dynamics of individuals is simple and the high-level characters of flow such as the traffic throughput
concerned, early works focus on designing controllers to direct a fleet of vehicles into a dense and stable platoon [39 3]
CityFlow (53) and FLOW [51] use RL agents steer the low-Ievel controllers of vehicles in order to investigate the traffic phenomenon in large-scale macroscopic traffic flow. For microscopic system, researchers also use behavioral model [55] hand-designed control laws 22 50
control individual vehicles. Zhou el al. (56) propose SMARTS simulator to investigate the interaction of RL agents and social vehicles in atomic traffic scenes. Pal et al. 31 find the emergent road rules of MARL agents trained by simple independent learner, which are also shown in our experiments. In this work, we work on the traffic environments with complex real-world scenarios such as tollgate and parking lot. The traffic flow can contain more than 40 vehicles. In contrast to previous work, we also propose novel method to improve the collective motions that better simulate SDP systems.
3 Preliminaries
3.1 Problem Setting
SDP system has two distinguishable properties: (I) each individual agent is self-interested, (2) the individual agent can vary to be cooperative or competitive with others within one episode and (3) the individual agent only actively interacts with its nearby agents.
To accommodate these properties, we formulate a SDP system as a set of Decentralized Par- tially Observable Markov Decision Processes (eccPPOMPP) [14] represented by tuple G =< A, A, R. Po: 0,2,7 >.
the set of the agent indices. We define the active agent set at each environmental time step l. as & = (ii,t,..., ) C & wherein k; is the number of existing agents this step. We consider a partially observable setting in which each agent can not access the environmental state 8, and instead draws individual observation o,, according to the local observation function O,,t = (x() S + O. The joint action space Ae C AA is composed of the union of all active agents' action spaces - Ai,t. Each agent i chooses its action di,t E Ai, following its policy ar,, ~ (-loe). These actions form joint action a, = J E Ar causing transition on the environmental state subjected to the state transition distribution P(se+ 18e,, ax). Each agent receives their individual reward from its reward function Ti., (, at). Po is the initial state distribution and y is the discount factor. So in test time our system is decentralized, where each agent fed with its local observation and actuated by its policy and interacts with others.
Here we define the environmental episode т = {(s,. a,, Ti.i ak. To= where T is the environmental horizon. An environmental episode thus contains set of agent episodes: {(o,,e di,t, rr.)) wherein to and t denote the environmental steps when agent enters and terminates, respectively. Suppose the policy т. is parameterized by 0,, the individual objective for each agent is defined as the discounted sum of individual reward (namely the individual return): (0,) = Er (R(()), wherein R&(T) = Ei = 1-1 Ti,t-
3.2 Individual Policy Optimization
To simulate SDP systems, simple approach is the independent policy optimization (IPO) 35 where each agent maximizes individual objective as if in the single-agent environment. Denote the individual value function is ViA Vi (st)
EE (t'-0) Drael and the corresponding advantage
3

---

function is A!, = A/ (st. ai,t, a-i,t) = ria + (s - (st), wherein a-i.t = ي= the shorthand of other agents' actions, the policy gradient method computes the gradient of individual objective as [42]:
نزنز is
V Ve, (0.)
E VV. log ne. (ails) (s, az a-)]]
(s.a)
[37 the clipped importance sampling factor p Tieeew (a: ola is used to mitigate the distribution shift occurred after the policy is up- dated for several epochs, wherein old is the behavior policy that generate samples and ninnew is the latest policy parameterized by 0, We call the resulting objective as the surrogate objective:
As a common practice, in PPO algorithm
(0.) = E mii(ppf.ccip( (s.a)
clip(p: 1 - e, 111++0 +)
é is hyper-parameter. By conducting the stochastic gradient ascent on the surrogate objective w.r.t. the policy parameters, the expected individual return can be improved. In continuous control RL problem, both the state space and action space are high-dimensional. Therefore V, is approximated using a neural network whose imput is typically the local observation. In the centralized training and decentralized execution (CTDE) framework (29), the value function takes global information as input by concatenating all agents" local observations.
4 Coordinated Policy Optimization
In a SDP system each agent has its individual objective. However, if we simply maximize each individual reward, the system will have sub-optimal solutions where, for example, the agents will become aggressive and egocentric, agooentric.jeopprdiiing the performance of the population and leading to critical failures. On the contrary, if apply cooperative learning schemes [32 41] and consider the summation of individual reward as the joint objective, the trained agents will exhibit unreasonable behaviors such as sacrificing oneself to improve group reward, which is not expected in SDP systems. To find a balance, as illustrated novel MARL algorithm called Coordinated Policy Optimization (CoPO) to facilitate the bi-level coordination of agents to learn the controllers of the SDP systems. Following the CTDE framework, during centralized training, first propose individual learning objectives by local coordination (Sec. a mechanism inspired by the fact that each individual agent is mostly affected by its nearby agents (52). We coordinate agents' objectives in neighborhood by mixing rewards following common social psychological principle. We further design the meta-learning technique to optimize the local coordination process, leading the global coordination (Sec. 4.2) which can improve collective performance of the population.
Fig. 2 and Algorithm in the Appendix, we devise a
4.1 Local Coordination
IPO only maximizes the individual objective (Eq. [T which leads to egocentric policies that damage the overall performance of the population. However, in natural SDP systems, e.g. animal groups, there exists certain level of cooperative behavior that not only benefits others, but also results better individual utility. To resemble such feature, we incorporate the ring measure of social value orienta- tion (28 38) a common metric from social psychology, into our formulation of local coordination. Concretely, we define the Local Coordination Factor (LCF) as degree ő E [-90", 90%] describing an agent's preference of being selfish, cooperative, competitive !1 We then define the neighborhood reward as:
ΣJENs, (6.1 Tha 8,1 Wa, (1,
wherein Na, (i, = ( ||Pos(i) - Pos(j)ll S d.
(3)
Na. (i, t) defines the neighborhood of agent within the radius d,, at step f. Note that the neighbor- hood of agent is tire-narying, therefore the the neighborhood reward can not be simply computed from a fixed of agents across the episode. Weighted by LCF, the coordinated reward is defined as:
cos(o)r,, (
'The social value orientation can be range |-800, 180%). We only take the meaningful half of the range.

---

V?
Success Rate
0.8
Local Coordination
Fixed
0.4
Unilorm
JLCP
pc
20
40
60
80
Execution
+
test time (degree)
Global Coordination
Figure 3: The test-time performance of pop- ulation trained with uniformly sampled LCF.
Figure 2: The framework of the CoPO method.
The LCF enables the local coordination in the optimization process. Supposing the agents maximize the coordinated reward with á 0% 90 -90%, the agents should become egoistic, altruistic, or F demonsttatee an interesting preliminary result on a simple version of Roundabout environment. During IPO training, assign to each agent in each episode. Current e is fed to policy as extra observation. In test time, the trained population for multiple runs. In each test results show that peak of success rate at certain LCF at test time. This suggests that increasing the awareness of neighbors' interests when training individual agents will propose better policies that increase the collective performance. IPO might lead to sub-optimal policies since the individual objectives never consider local coordination with other agents.
sadistic respectively. Illustratively. Fig.
randomly select 2/(0,, 90%) and
fixed LCF for all agents. The
In order to incorporate the LCF into training process to improve the population performance, apart from the value function approximator for the individual reward (Sec. (3.2), function to approximate the discounted sum of the neighborhood reward V. E| g f N and the neighborhood advantage is therefore calculated as: A = rie 7 1.++1 ViY The coordinated advantage can be derived as: A objective, which takes the utilities of neighbors into consideration, as follows:
another value
cos( cos()) A + sin( A We now have the coordinated
J, (0,, +) = E mpin p clip(p: ,, 4) (s,a)
(5)
We follow a similar training procedure as IPO in Sec. 3.2 with the advantage replaced by the coordinated advantage. To improve diversity,
distributional LCF & NN oz), where + I Øo] are the learnable parameters of the LCF distribution.
4.2 Global Coordination
The major challenge of local coordination is the selection of LCF Though the experiment in Fig. shows that there exists optimal LCF that can maximize the population success rate, it is laborious to search for such LCF, therefore introduce global coordination which enables automatic search of the best LCF distribution to improve population efficiency.
In this section, we introduce how we accomplish global coordination with bi-level optimization method that computes a meta-get gratd-grrdent of the global objective. We first define the global objective that indicates the performance population:
(0,, 02,...
E YEE-
PiAd
E
Σ rı
(6)
E=
t= iEE.,
wherein E, denotes the active agents at environmental time step t of the episode т. Here we remove y since it is ambiguous when the start steps of agents are not aligned. Eq. can not be directly optimized when computing policy gradients of individual policies' parameters.
We introduce the concept of individual global objective to make the optimization of the global objective feasible. By this way, we can turn the system-level optimization process into agent-level optimization processes, that the individual agent's data can be easily accessed. We factorize
5

---

Jo (o......) to individual global objectives (0:) such that maximizing the objective for each agent equivalent maximizing Eq. 6 The individual global objective defined as:
Ejee. TJn E7,1
(0.10:
I
we introduce the global reward y== -jEc,, TT,e/ as the average reward of all active agents at step i, then this objective is the cumulative global reward in the time interval when agent is alive. Proposition 1 (Global objective factorization). Suppose each agent i can maximize its individual global objective Jú then the original global objective j is maximized.
We prove that such factorization does not affect the optimality of global objective in Appendix.
To improve global objective via optimizing LCF, we need to compute the gradient of Eq. 7 w.r.t. Φ. Denote the parameters policies before and after optimizing Eq. 5 as 0ot and 0mew respectively, we can compute the gradient applying the chain rule as:
J/ (0,T0) Vggen Jf (0)o0))
V
(8)
Term
2nd Torm
The 1st term resembles the policy gradient as Eq. with the objective replaced by J,C:
1st Term-
E (H.
V one min(pA a), clip(p, c, -)
Here we extra global value function VG estimate the value of global reward G then compute the global advantage AG, The samples (s, a) are generated by the behavior policy 0old.
By following first-order Taylor expansion, the 2nd term be computed follows. Note that V Vouad! (0 2 recovers the vanilla policy gradient in Eq.
2nd Term = V (Bpld + a Jf (a (fulh, +)) = a
V
E
V log g (a V
(10)
(s.n)~0pt
Averaging over all agents and combining Eq. 9 and Eq.
10
we derive the LCF objective
JLCF
E
V Ve min miinppp clip(pp - A) yau log naga (a: [ (11)
We then conduct stochastic gradient ascent on JLCF w.r.t. with learning rate a.
The detailed procedure of CoPO is given in Appendix. In our implementation. we follow the commonly used parameter sharing technique 44 so that all agents share the same set of neural networks, which guarantees the scalability of our method. As a summary, our method totally employs 4 neural networks: the policy network, the individual value network, the neighborhood value network, and the global value network. The input to all networks is always the local observation of the agent. Our further study (see Sec. [5.5) shows that centralized critic method, which feeds global information to the value networks, does not yield better performance and usually causes unstable training. The reason might be the time-varyee varying input the centralized critics, that the combined observation of nearby agents is always changing and doesn't 't hold temporal consistency across the episodes.
5 Experiments
5.1
Evaluation Platform
We develop set of 3D traffic environments to evaluate MARL methods for simulating SDP sys- tems based on MetaDrive (26). a lightweight and efficient microscopic driving simulato The environments are developed in Panda3D [13 and Bullet Engine, which can run 80 FPS on a standard
MetaDrive can be found at: httpa///ttua.ccoocccccaaaatttttt

---

workstation. Examples of the environments are shown in Fig. of the five traffic environments are as follows:
! The descriptions and typical settings
Roundabout: A four-way roundabout with two lanes. 40 vehicles spawn during environment reset. This environment includes merge and split junctions.
Intersection: An unprotected four-way intersection allowing bi-directional traffic as well as U-turns. Negotiation and social behaviors are expected to solve this environment. We initialize 30 vehicles.
Tollgate: Tollgate includes narrow roads to spawn agents and ample space the middle with multiple tollgates. The tollgates create static obstacles where the crashing is prohibited. We force agent to stop at the middle of tollgate for 3s. The agent will fail if they exit the tollgate before is allowed to pass. 40 vehicles are initialized. Complex behaviors such as deceleration and queuing are expected. Additional states such as whether vehicle is in tollgate and whether the tollgate blocked are given.
Complementary to Tollgate, Bottleneck contains a narrow bottleneck lane in the middle that forces the vehicles to yield others. We initialize 20 vehicles.
Bottleneck:
Parking Lot: A compact environment with parking slots. Spawn points are scattered in both parking lots and extemnl roads. 10 vehicles spawn initially. In this environment, allow agents to back their cars to spare space for others. Maneuvering and yielding are the key to solve this task.
Those environments cover several scenarios in previous simulators [36] 31 but are extended with real-world scenarios rather than atomic scenes. Besides, our environments support dense traffic flow where all vehicles are controlled through the low-level continuous signals. All the environments can be further concatenated to form more complex tasks for the future study of multi-task learning.
Evaluating a population simulated by MARL is an open question. In this work, we define set of general-purpose metrics that can characterize different aspects of the SDP systems. We choose three episodic metrics with minimal assumptions on the tasks. We evaluate the competence of population with success rate. is the ratio of the number of agents that reach their goals over the total number of agents in one episode. We also evaluate the efficiency of the population. It indicates the difference between successes and failures a unit of time (NVsssesss Niaulure) / is possible for population to achieve high success rate but has low efficiency, because the agents in the population are running in very low velocity. The third metrie we adopt here is the safety. defined as the total number of critical failures for all agents in one episode.
5.2 Experiment Setting
We compare the independent policy optimization (IPO) method 355 using PPO 37 as the individual learners. We further encode the nearby agents' states into the input of value functions (centralized critics) following the idea of Mean Field MARL [52) and form the mean field policy optimization (MFPO). We examine different variants of centralized crities, but find MFPO yields best performance (see Sec. (.5). The curriculum learning (CL) [30] is also a baseline, where we chunk the training into 4 phases and gradually increase initial agents in each episode from 25% 100% of the target number.
We conduct experiments with the aforementioned environments and algorithms using RLLib (27). a distributed learning system which allows large-scale parallel experiments. Generally, we host concurrent trials in Nvidia GeForce RTX 2080 Ti GPU. Each trial consumes 2 CPUs with 4 parallel rollout workers. Each trial is trained over roughly 1M environmental steps, which corresponds to about 55 hours in real-time transportation system and +2,000 hours of individual driving experience (assume averagely 40 vehicles running concurrenly). All experiments are repeated 8 times with different random seeds. Other hyper-parameters are given in Appendix.
5.3 Main Results
Table the main results of all the evaluated methods. Because of the bi-level coordination in policy optimization, our CoPO achieves superior success rate all environments. Noticeably in the most difficult environment, the unprotected Intersection, our method outperforms baselines with substantial margin. That environment requires agents to cooperate frequently in order to determine who should go fist. The populations generated from other methods cause severe congestion in the
7

---

Poundaatent
Intertation
Tokgetn
Betlwnedk
Parking Lot
Ducoms Sare
Sucor Face
Sucasa
Sucoun
Succes
GPaineyy
Sably
Sably
Qaldy
Saley
Saley
Figure 4: Performance of the trained populations from different MARL methods.
Roundaoout
Into secdon
Tolgana
Softl mocd
Parting Lot
30
3.7
94
MFRO
8.5
02
(Dun))
Wnde/nunberra Aggar
1n06 Narberr Agents
1stal Number stAgera
1stel humber A'Arra
Intl of
Figure 5: The success rate of different trained populations with varying number of initial agents at test time. The gray lines denote the initial number of agents in the training environments.
Table 1: Success rate of different approaches.
Roundabout
Intersection
Tollgate
Bottleneck
Parking Lot
IPO
70.81 +1995
60.47 +5.79
82.90 +281
72.43 #3.79
61.05 #281
MFPO
64.27 1368
67.74
14.19
81.05 13007
67.40 24.77
53.96 2465
CL
65.48 1396
62.03
14.1
73.72 13446
68.81
60.62 =2:5
CoPO (Ours)
73.67
1371
78.97
1:423
86.13 1:776
79.68
#2.91
65.04 11.59
middle of intersection which prevents any agent going through. See Sec. 5.4 and demo video for
detailed visualization.
We also find the MFPO performs worse than the independent PPO in several environments. That might be due to the neighborhood changing all the time. Simply concatenating averaging neighbors' states as the additional input to the value functions makes the training unstable, A detailed comparison of different design choices in centralized critics is given in Sec. 5.5
Fig. presents the radar plots under the three metrics. In each environment, we normalize each metric to its maximal and minimal values in all evaluated episodes for all algorithms. For safety measurement, we use negative values of total crashes in normalization. We can see that CoPO achieves superior results across all three metrics. Noticeably, CoPO has substantial improvement in safety, showing the importance of the bi-level coordination.
To further evaluate the generalization of the populations, we run the trained policies in the test environments with different initial numbers of agents. As shown in Fig. 5 and Intersection, where the interactions are extremely intensive, IPO and MFPO overfit to the initial number of agents. In the environment with sparse traffic, their performance is inferior that in the environment with more agents. On the contrary, our method does not overfit these two environments. In Parking Lot, it seems that all methods has some degree of overfitting the initial number of agents. The results are consistent with the finding in (24] showing that strategies learned via single instance of MARL algorithm can overfit to the policies of other agent in the population. conduct an experiment using simple heuristic to control part of the vehicles in the population to test the generalizability of different training methods. The experiment shows that CoPO trained agents are robuster than IPO trained ones when interacting with vehicles controlled by unseen policies. It is left to future work on improving the generalization of MARL methods.
we find that in Roundabout
Appendix, we
5.4 Behavioral Analysis
Fig. plots the visualization of all trajectories occurred in one environmental episode, where the agents spawning at same entries share the same colors. The population trained from CoPO exhibits collective behaviors, while the critical failures reduce drastically compared other populations. The
8

---

(a) IPO
(b) MFPO
(c) CL
(d) CoPO (Ours)
Figure 6: Plot of trajectories of trained populations in Intersection. The dark spots indicate the locations where critical failures happen. CoPO produces safer and more coordinated traffic flow.
Wadinng
Roundabout
Intersection
Pusking
Tollgate
Ousssigg
princhs
OMiing
Parking Lot
Bottleneck
Mevarang
Yeeking
Figure 7: Visualizing the social behaviors of CoPO populations. Diverse behaviors emerge in CoPO populations, which highlighted with black dots in the plot. We plot the past 25 and future 25 positions of each vehicle with different colors, where the luminosity of a trajectory decreases from light dark, representing its past future positions.
population trained from other methods tend to directly rush into the center of Intersection, causing severe congestion and frequent crashes. On the contrary, the population from CoPO tends drive collectively and successfully avoid crashes through social behaviors such as yielding, with superior success rate and high safety. The qualitative result suggests that CoPO can resemble the collective motions widely existing in SDP systems [54].
Fig. further visualizes the temporal behaviors of CoPO agents in 5 environments. Each vehicle in the population tends to drive socially and react to its nearby vehicles. A set of diverse behaviors emerge through the interactions, including socially compliant behaviors such as yielding, queuing, and even reversing to leave more room to others, as well as aggressive behaviors such as cutting in and rushing. The population trained from CoPO successfully reproduces diverse interaction behaviors in traffic systems.
5.5 Ablation Studies
We evaluate different variants of centralized critics in Roundabout environment. The Concat variant refers to simply concatenating all states of nearest K=4 agents as long vector fed to the value networks. The Mean Field method uses the average of the states from nearby agents within given radius (we use 10 meters). The CF refers to "counterfactual", namely adding the neighbors' actions accompanied with their observations as the input to value functions. Note that the Mean Field w/ CF method follows the Mean Field Actor Critic proposed in [52], the difference is that we use the training pipeline of PPO (66 in order to operate continuous action space. As illustrated in Fig. 9 we find that the design with Mean Field is more stable compared to Concat. CF also stabilizes the training. We therefore report the results of Mean Field w/ CF as the MFPO method Sec. 5.3
Fig. g plots the learning curves of CoPO with and without centralized critics. We find that CoPO with centralized critics yields inferior success rate and is unstable in some environments compared the native CoPO. This might because nearby agents are always changing and doesn' 't hold the temporal consistency within episode.

---

Roundabout
Infertctionn
Tolgab
Botteneck
Parking Lot
0.55
0.0
CuPO
Figure 8: Comparison of CoPO with or without centralized crities.
Table 2: Ablation study on the effectiveness of global coordination. Average success rate is provided.
Experiment
Roundabout
Intersection
Tollgate
Bottleneck
Parking Lot
(a) sample from (0, 0.1²) (b) maximize global reward
62.38 17.26
70.23 1:722
60.47 =5.00
71.29 43.27
59.14 1:29
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
0.00 + 0.0
0.00 +: 0.00
(c) maximize neighborhood reward
65,70 +: 221
66.83 2.16
57,30 = 3:46
71.62 +: 1.22
6,34 +: L.44
CoPO: sample from (o )
73.67 #3.71
78.97 4:4:33
86.13 1.77
79.68 4:
65.04 1:.59
In Yable([a) [a), validate the effectiveness of global coordi- nation by replacing the LCF distribution with a given normal distribution during the course of training. The proposed method can yield better success rate than the result of man- ually defined fixed LCF distributions. In (b), we use IPO to maximize the global reward directly. Unfortunately. the training fails because is hard build the causal connection between individual's 's behavior with the average reward of all agents. In (c), we use IPO maximize the neighborhood reward. This method performs poorer than CoPO since we balance the individual reward and the neighborhood reward.
Goncat wio CF
Mean Fieid CF
Concat
w/
Sucsess
0.4
0.0
0.0
0.2
0.4
0.8
0.8
1,9
Environmental Step
Comparison of design choices of centralized critic methods.
Figure 9:
Conclusion
In this work, we develop novel MARL method called Coordinated Policy Optimization (CoPO) to incorporate social psychology principle learn neural controller of SDP systems. Experiments on microscopic traffic simulation show that the proposed method can learn populations which achieve superior performance over the previous MARL methods in three general-purpose metrics. Interestingly, we find the trained vehicles exhibits complex and socially compliant behaviors that improve the efficiency and safety of the population as a whole.
7 Social Impact and Limitations
This work could help the development of intelligent transportation systems which would have wide impact on society. We can analyze the emergent behaviors of the traffic under different scene structures and optimize the road structure or traffic light control to improve the traffic efficiency. Morcover, CoPO can simulate diverse traffic flow, which can be used to benchmark the generalizability of the autonomous driving systems. Besides simulating traffic flow, the proposed method is applicable for pedestrian crowd simulation to study social crowd behaviors as well as potential human stampedes and crushes.
This work focuses on decision- (ecision-making problem, so we simplify the perception of the vehicles as one-channel LiDAR and assume the acquisition of accurate sensory data. In reality, the precise perception of the surroundings in self-driving remains very challenging. This work adopts 5 typical traffic scenarios in simulator as the testbed, which are still far from emulating the complexity of real-world traffic scenes. the future we will import real-world HD maps in the simulator to create more realistic scenarios.
Acknowledgments and Disclosure of Funding
This project was supported by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Fund.
10

---

References
[1] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint (xXxv:19990075222 2019.
[2] Eric Bertin, Michel Droz, and Guillaume Grégoire. Hydrodynamic equations for self-propelled particles: microscopic derivation and stability analysis. Journal of Physics A: Mathematical and Theoretical, 42544):44500,, 2009.
[3] Bart Besselink and Karl H Johansson. String stability and delay-based spacing policy for vehicle platoons subject to disturbances. IEEE Transactions on Automatic Control, (2(9):4376-4391 2017.
[4] Kunal Bhattacharya and Tamás Viesek. Collective decision making cohesive flocks. New Journal of Physics, 02(9)) 2010.
[5] Jerome Buhl, David JT Sumpter, lain Couzin, Joe J Hale, Emma Despland, Edgar Miller, Steve Simpson. From disorder to order in marching locusts. Science, 402--176) 2006.
[6] Debashish Chowdhury. Collective effects in intra intra-cellular molccular motor transport: Coordination, cooperation and competition. Physica A: Statistical Mechanics and its Applications, 372(1):84 95, 2006.
[7] András Czirók and Tamás Viesek. Collective behavior of interacting self- ceff-ropeelled particles. Physica A: Statistical Mechanics and its Applications, 281
2000.
[8] Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 32:4403-4414, 2019.
[9] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Coun- terfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[10] Jakob N Foerster, Yannis M Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. arXiv preprint arXiv:1605.06677 2016.
[11] Jakoh N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learingg awareness, arXiv preprint arXiv:1709.04320 2017.
[12] Jacques Gautrais, Christian Jost, and Guy Theraulaz. Key behavioural factors a self-organised fish school model. In Annales Zoologici Fennici, volume 45, pages 415-428. BioOne, 2008.
[13] Mike Goslin and Mark R Mine. The panda3d graphics engine. Computer, 37(10): 12-1 2004.
[14] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66-83. Springer, 2017.
[15] Johannes Heinrich and David Silver. Deep reinforcement learning from self play in imperfeet-innoomaatoon games. arXiv preprint MeXiv:1603.01121 2016.
[16] Dirk Helbing, IlIés Farkas, Tamas Viesek. Simulating dynamical features of escape panic. Nature, 407(6803) 2000.
[17] Dirk Helbing. Joachim Keltsch, and Peter Molnar. Modelling the evolution of human trail systems. Nature, 388(6637):47-50, 1997.
[18] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement leamning Autonomous Agents and Multi-Agent Systems, 33(6):750-79) 2019.
[19] Thomas lhle. Kinetic theory of tlocking: Derivation of hydrodynamic equations. Physical Review E. 83(3)0030001, 2011.
[20] Venkatesan Kanagaraj and Martin Treiber. Self-driven particle model mixed traffic and other disordered flows. Physica A: Staristical Mechanics and its Applications, 509:1-11, 2018.
[21] Matthew G Karlaftis and Eleni Vlahogianni. Statistical methods versus neural networks in transporta- tion research: Differences, similarities and some insights. Transportation Research Part C: Emerging Technologies, 19(3):387-39) 2011.
11

---

[22] Florian Knomr, Daniel Baselt, Michael Schreckenberg, and Martin Mauve. Reducing traffic jams via vanets. IEEE Trunsactions on Vehicular Technology. 61(88:3490-3498, 2012.
[23] Karol Kurach, Anton Raichuk, Piotr Stanczyk. Michat 7ajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research foothall: A novel reinforcement learning environment. In Proceedings of the AAAI Conference Artificial Intelligence. volume 34, pages
2020.
[24] Marc I Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach multiagent reinforcement learning. arXiv preprint arXiv:1711.00822, 2017.
[25] Joel Z Leibo, Vinicius Zambaldi, Mare Lanetot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv: /00.::07700003737 2017.
[26] Quanyi Li, Zhenghao Peng, Zhenghai Xue, Qihang Zhang, and Bolci Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. arXiv preprint arXiv:2109. 12674, 2021.
[27] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning, In International Conference on Machine Learning, pages 3053-3062, 2018.
[28] Wim BG Liebrand. The effect of social motives, communication and group size on behaviour in an n-person multi-stage mixed-motive game. European journal of social psychology. 14(3):239-25) 264,, 1984.
[29] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiiee environments. Advances in neural information processing systems, 2017.
[30] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-50 2020.
[31] Avik Pal. Jonah Philion, Yuan-Hong Liao, and Sanja Fidler. Emergent road rules in multi-agent driving environments. arXiv preprint arXiv:201 1. 10753, 2020.
[32] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295 4304. PMLR, 2018.
[33] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint 11.ivv1992.000444 2019.
[34] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7833)):004-609) 2020.
[35] Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv e-prints, pages arXiv-2011, 2020.
[36] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 1206.02438 2015.
[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. optimization algorithms. arXiv preprint arXiv:1707.066 06347, 2017.
Proximal policy
[38] Wilko Schwarting, Alyssa Pierson, Javier Alonso-Mora, Sertac Karaman, and Daniela Rus. Social behavior for autonomous vehicles. Proceedings of the National Academy of Sciences, 1 149020)24972-224788 2019.
[391 Steven E Shladover. Review of the state of development of advanced vehicle control systems (aves). Vehicle System Dynamics, 24(6-7):551-595 1995.
[40] Kyunghwan Son, Daewoo Kim, Wan Ju Kang. David Earl Hostallero, and Yung Yi. Quran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896. PMLR, 2019.
[41] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnekki Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls. et al. deloppsccooooiiiion networks for cooperative multi-agent learning. arXiv preprint arXiv: 170. 2017.
12

---

[42] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[43] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pages 330-337, 1993.
[44] Justin K Terry, Nathanicl Grammel, Ananth Hari, Luis Santos, and Benjamin Black, Revisiting parameter sharing in multi-agent deep reinforcement learning. 2021.
[45] Martin Treiber and Ame Kesting. The intelligent driver model with stochasticity-new insights into traffic flow oscillations. Transportation research procedia, 23:174-187, 2017.
[46] Simon Vanneste, Astrid Vanneste, Siegfried Mercelis, and Peter Hellinckx. Learning communicate using counterfactual reasoning. arXiv preprint arXiv:2006.07200 2020.
[47] Tamás Viesek, András Czirók, Eshel Ben-Jacob, Inon Cohen, and Ofer Shochet. Novel type of phase transition in a system self-driven particles. Physical review letters, 75(6):1226, 1995.
[48] Eugene Vinitsky, Aboudy Kreidich, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M Bayen. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Conference on Robot Learning, pages 399-409, 2018.
[49] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, (75(7782):350-3555 2019.
[50] Meng Wang, Winnie Daamen, Serge P Hoogendoorn, and Bart van Arem. Connected variable speed limits control and car-following control with vehhicle-infrastrucuure communication to resolve stop-and-go waves. Journal of Intelligent Transportation Systems, 20(6):559-572, 2016.
[51] Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen. Flow: Archi- tecture and benchmarking for reinforcement learning in traffic control. arXiv preprint arXiv:7110.05463 page 10, 2017.
[52] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mcan field multi-agent reinforcement learning. In International Conference Machine Learning, pages 5571-5580. PMLR, 2018.
[53] Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. Cityflow: multi-agent reinforcement learning environment for large scale city traffic scenario. In The World Wide Web Conference, pages 3620-3624. 2019.
[54] Bolei Zhou, Xiaoou Tang, and Xiaogang Wang. Mcasuring crowd collectiveness. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3049 3056, 2013.
[55] Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. 2020.
[56] Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang. David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar. Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhengbang Zhu. Nhat Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezace, Peyman Yadmellat, Mohsen Rohani, Nicolas Perez Nieves, Yihan Ni, Scyedershad Banijamali, Alexander Cowen Rivers, Zheng Tian, Daniel Palenicek, Haitham bou Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. Smarts: Scalabie multi- agent reinforcement learning training school for autonomous driving, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
2. If you including theoretical results...
13

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proof's of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URI.)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report enror bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "hI09HHvjrG", "reviewer": "Reviewer_5BQc", "summary": "The authors introduce a novel MARL learning algorithm, CoPO, that is designed to facilitate coordination in SDP systems. In addition to independent value functions, CoPO learns a neighborhood value function according to a return based on the normalized sum of neighboring agent rewards.  Both independent and neighborhood returns are mixed together according to a popular measure of social value orientation. Using a centralized bi-level optimization process, the agent social value distribution is optimised at the same time as the optimal low-level continuous decentralized control policies. The authors evaluate CoPO on a testbed of 5 common road traffic coordination tasks and according to three novel collective metrics. CoPO is shown to outperform other state-of-the-art CTDE MARL algorithms.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors' choice of topic fills a gap in MARL literature. While prior work on social value orientation in autonomous driving [e.g. Schwarting 2019] seeks to estimate the participants' social values from observed trajectories, CoPO seems to be the first work that tries to learn optimal social value orientations end-to-end in a CTDE MARL setting. I believe that this represents an important change of perspective: While prior work has focused on social value orientations being inherent properties of human drivers that cannot be influenced by autonomous drivers but have to be anticipated and respected, CoPO is based on the idea that incorporating social value functions in fact yields an effective inductive bias that allows end-to-end MARL methods to effectively solve certain general-sum games (also known as non-zero-sum or mixed cooperative-competitive) for purely autonomous drivers as well. Unlike human drivers, autonomous drivers can tune their \"social\" value distributions. The fact that this results in such a significant impact on task completion across a range of common traffic coordination problems is, in my view, a surprising insight and validates the original approach of this work.\n\nThe submission is written clearly. The bi-level optimization method CoOP is clearly described and well-motivated. Both evaluation environments and metrics used are well described, and experimental results are presented clearly and indicate significant outperformance.\n\nOverall, I recommend this work for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It would be great if the authors could connect their findings concerning the suboptimal performance of centralized value functions over decentralized ones in the light of recent developments in the field [Lyu et al 2021].\n\nI would be interested to see what happens if we put a multimodal prior on the Phi-proposal - do we observe optimal multi-modal distributions for agent social value orientations? \n\nAnother interesting line of research would be to study mixed autonomous-human traffic. For this purpose, one might want to consider dynamically varying Phis depending on what social value orientation is detected for a given traffic neighborhood. \n\nThe authors would do well to check grammar and spelling in isolated instances.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "cbHWxsJxWO4", "reviewer": "Reviewer_xK7J", "summary": "In this paper the authors propose a multi-agent reinforcement learning approach to finding controllers in self-driven particles systems. The problem of SDP is formulated as dec-POMDP where each agent has an individual reward function that is wishes to maximize using local observations only.\n\nThe algorithm consists of two components: local coordination relying on neighboring rewards and global coordination. \n\nThe local coordination part modifies the objective to be a combination of individual rewards and neighbor reward using a Local Coordination Factor. The LCF is used to compute a coordinated objective. \n\nThe global coordination component is responsible to find the best LCF, for tractability this objective is factorized into individual global objective.\n\nThe algorithm is evaluated on a set of traffic scenarios against multiple baselines: independent policies, mean field methods and curriculum learning. \n\nThe reward model using LCF and the split between local and global coordination is sound and interesting. However, the derivation lack of clarity, and it would have been useful to study the algorithm on other domains, especially given that the claim is on general SDP systems. \n\nThe authors should revise the claim formulation and provide more justification of the global coordination method.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clear and well written and the topic is interesting.\n\nThe experiments are focused on traffic scenarios which weakens a bit the claim of the paper. The challenges addressed by the author are valid, it is in practice very difficult to hand-engineer controllers for those systems.  The utility of the solution found by the algorithm needs to be better justified, can we really draw conclusions from the output of this algorithm? \n\nThe authors propose a general claim in the introduction, however at the core of the algorithm lies the computation of a neighbor set for each agent. In the traffic domain, neighbors are computed based on distances, how would one compute neighbors in other domain, how is this heuristics affecting the results?\n\nFigure 3: how is success rate defined? It is not clear if the agents are sharing the same LCF, in practice it seems like every individual would have different LCF.\n\nIt is understandable that the authors need to factorize the global objective for tractability. However, it is not very clear what the individual global objective represents (the term is contradicting itself), the authors should provide some intuition. \n\nIn the proof there seem to be a hidden assumption, a priori, the expectation over trajectories depends on all the agents. Why would the term inside the sum be only depending on \\theta_i? This step needs to be clarified.\n\nThe experiment show that the proposed algorithm outperforms the baselines on multiple objective on most of the traffic scenarios.\n\nOne of the main usage of this training procedure seems to generate a variety of behavior to perform analysis. The interpretation of the generated behavior is very subjective, the author should comment on how to quantitively assess the quality of those behaviors. \n\nIn the evaluation procedure, it would be useful to evaluate the generalization of the policy in terms of interaction. What if part of the agent in the environment follows a different policy trained by another algorithm or following a heuristic?\n\nFigure 3 from the appendix should make it in the main paper. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The conclusions of this paper are limited to the traffic domain. It does have societal impact as such solution could help the development of intelligent transportation systems which would have wide impact on our society.", "review_text": "", "extra_scores": {"ethical_concerns": "No strong concern, one could wonder the social implication of the LCF parameters and if this model correctly represent human behavior but this would be future work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "AlOqQ6gspW9", "reviewer": "Reviewer_oVAt", "summary": "The authors aim to design controllers for traffic simulation agents in the mixed-motive reinforcement learning setting. They introduce a smooth interpolation between individual and mean global reward (social value orientation) which agents locally optimize while using a meta-optimization process to optimize the degree of interpolation. They show that in a suite of 5 traffic scenarios, their method generally produces agent populations that succeed at higher rates, are more efficient, and produce a smaller amount of critical failures.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I enjoyed reading this paper. However, I think it could improve its positioning relative to existing work as well as compare to more related baselines. For these reasons I would suggest rejecting the paper for this conference, but I would look forward to seeing an improved version in a future venue!\n\nIn the abstract the authors write \"However, previous multi-agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand, which fail to capture the essence of SDP where the agents vary to be cooperative or competitive even within one episode.\" — However, in the introduction and related work the do note some mixed-motive RL works which study exactly this problem. Furthermore, I think the authors missed some key and very related works in their literature review for example \"Social Diversity and Social Preferences\nin Mixed-Motive Reinforcement Learning\" McKee et. al also experiments with social value orientation explicitly. Furthermore, recent works such as Baker et. al 2020, Hughes et. al 2018, Jaques et. al 2019 all work in mixed motive RL in temporally extended games. For these reasons, I would suggest the authors reframe their claims to be more focused on traffic simulations as opposed to general mixed-motive RL. I would also suggest they experiment with some of these prior methods as baselines since they are designed for the same setting.\n\nIf I understand correctly, equation 6 is the mean reward of all agents in the episode. If the meta-optimization process is optimizing this value, doesn't that contradict one of the goals of the method which is to not have individuals sacrifice themselves for the good of the whole? I ask because Phi is allowed to go up to 90 (altruistic) so it seems that this would optimize the global reward best. Are there suboptimal local minima the agents fall into when optimizing with phi=90? I think it would be interesting to see Figure 3 but also plotted with equation 6 (global mean reward) as the y-axis in order to understand this better.\n\nIs it possible that there is a different local reward function that would optimize the global efficiency, success rate, and critical failure rate? Is this reward function intractable?\n\nAre there any links to existing game theory literature? It may be a nice addition to this paper to add experiments to smaller more well understood domains to more easily see what types of equilibria it induces (e.g. iterated prisoners dilemma, the coins game, etc).\n\nIn the introduction you mention that that your work is unique because it operates in the continuous control space. However, there are many high fidelity robotics works that actually use a discretized action spaces (e.g. see the robotic hand work from OpenAI). Maybe you should consider discretizing the action space and running this baseline if you'd like to make this claim stronger? Or I'd suggest dropping the language about continuous action spaces since I don't think it's a very important part of your work here and maybe distracts from the main point about coordination and social value orientation.\n\nI would be interested to see how results change with varying d_n.\n\nConcatenation is 1 parameterization of a global centralized value function. There are many others (e.g. attention, mean, max, etc). I notice you say that your centralized critic performed worse (L209). Concatenation is not entity invariant, but attention/mean/max are. I would suggest trying these instead.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There was limited to no discussion of limitations and societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "5v16m-Q6XUM", "reviewer": "Reviewer_6S1x", "summary": "The paper proposes a coordinated policy optimization algorithm to simulate self-driven particles and showcases its power in the autonomous driving application of traffic simulation. The algorithm proposes both local and global coordination mechanisms to tradeoff cooperation and competitiveness at the scene-level. The paper benchmarks the efficiency and safety of their method against a few baselines on a newly proposed simulator of heavily interactive scenarios.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated from the perspective of the challenges. It is clear that the key challenge in simulating microscopic traffic is the agents' coordination, while things like dynamic feasibility etc. at the agent level are easier. However, I missed a more detailed motivation in terms of potential applications. I think this type of multi-agent simulation (or planning) have multiple applications such as testing/training a separate autonomy stack or the more hypothetical scenario in which all autonomous cars are driving with the same, share autonomy system. While the latter is well covered by the proposed framework, I believe there are no studies or intuition about the suitability of the proposed algorithm on the former. Would be great if the authors can clarify during rebuttal.\n\nIn terms of related work, the paper has pretty good coverage but I found 2 rather relevant works missing:\n[A] Suo, Simon, et al. \"TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. --> This is based on Imitation Learning but very focused on the interaction modeling of realistic, scene-level trajectories.\n[B] Tang, Yichuan. \"Towards learning multi-agent negotiations via self-play.\" Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019. --> This one leverages RL and a zoo of models to iteratively achieve more complex and diverse simulated agents.\nI would like to see the revision of the paper including a discussion of these, and it would also be great to include them as baselines in the future.\n\nRegarding the method, I believe it is sound and intuitive. While the local coordination is a simple extension of previous approaches, I believe the global coordination mechanism and its learning is original. I like the fact that the proposed method is decentralized at inference despite learning being coordination-aware, and that the local coordination factor is modeled as a continuous degree between cooperation and competitiveness. The write-up is also pretty clear overall. Nonetheless, I have a few questions and comments:\n- In the preliminaries, it says that \"the policy graddit methods are the appropriate solution\", but that seems like a big claim without much backup.\n- In Eq. 1, would the intuition that optimizing the individual advantage is hard given that we only observe a single action from the rest of the agents and thus the effect of the agent action cannot be well identified?\n- The radial neighborhood is overly simplistic. I believe this is probably the reason why there are still critical failures happening in intersection (as seen in Fig. 6), since the conflicting vehicles come from different branches that are far away.\n- I am confused about the statement in L168 about a shared and fixed LCF sinc in Section 4.2 it seems that the LCF is optimized per actor, which makes more sense to me.\n- Why is $\\phi$ uniformly sampled between [0, 90] (L167) instead of [-90, 90] as stated in L164?\n- In Eq. 6, should it be $r_{i, t}^C$ instead of $r_{i, t}$? If not, why?\n- In L185, it says that $\\gamma$ is removed because the start steps of agents are not aligned. Couldn't the discount factor just go inside the sum over actors? Or do the authors mean it is implemented this way but simplified in the Eq. 6?\n- How is the output parameterized, more concretely?\n\nFinally, regarding the experiments, I believe they provide good insights into the difference between the baselines and the proposed method, although more baselines would have made the submission stronger. Aside from the scarcity of baselines, I have the following questions:\n- Why are the new traffic environments required? By looking at Figure 1, it looks like a similar environment such as HighwayEnv could be sufficient.\n- Since the networks are shared across all agents, I wonder how different goals are encoded for different agents. For instance, in Parking Lot, the agents can navigate toward external roads or enter parking lots. Would be great if the authors could share some details about how this is achieved.\n- Why is a basic LiDAR simulated? Is there realistic background? If not, I do not really understand how this contributes to better experiments as opposed to just having access to past tracks of other agents.\n- In Fig. 4, why does CoPO achieve less efficiency?\n- I missed an experiment where we train a simple planner on the simulated agents for the proposed method and baselines, and evaluate in a common set of scenarios/agents. This would be similar to Table 3 in TrafficSim [A].\n- A bigger concern is the claim about greater diversity in the abstract. I believe this is not backed up by any metric or qualitative results. If anything, Fig. 6 suggests that CoPO generates __less__ diverse traffic than the baselines as the lines of traffic flow are \"thinner\".\n- In the video, it seems that most CoPO collisions in roundabout happen before entering it, as opposed to when merging as in most of the cases for the baselines. Any intuitions on why?\n\nMinor comments:\n- L101: even if well-known, include the full name of the Dec-POMDP abbreviature.\n- In the footnote at the end of page 4, how is the meaningful half range enforced? Clipping?\n- L240: sentence is oddly phrased. Perhaps replace \"being terminated\" for \"are terminated\".\n\nPS. Figures 6 and 7 are great and the visualizations on the video too! Really enjoyed looking at the results, although would have appreciated narration on the video while highlighting particular cases. Please address as many comments as possible during the rebuttal.\n\n==========\n\nPost rebuttal comments:\n- Thanks for the new experiments where IDM is used to control a percentage of the agents. The results are interesting and do show better robustness for CoPO than IPO. However, the results are still concerning in that the effectiveness and applicability of this method seems to be a bit limited to the coordinated agents setting, and the variance of the impact across scenarios seems high. Also, I wanted to make a comment that Tang et. al, does not just use IDM. IDM is used only as an initialization, then a policy is fully learned over multiple training iterations where the other agents policy get picked from an increasing model zoo.\n- Thanks for clarifying about the goal encoding. Please include this information in the final revision.\n- Overall, the responses and other reviewer comments were aligned with my expectations and thus I maintain my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not see a discussion about the limitations or societal impact. Please comment on the rebuttal and include this in later revisions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method that uses bi-level optimization to overcome social dilemmas amongst large groups of self-interested agents. Specifically, each agent considers a weighted sum of its own reward and the reward of its neighbours. The weighting term, phi, trading off between these two is optimized using meta-gradients of the global objective. 

Importantly, by assuming that the method can undertake reward-shaping for the individual agents, the actual social dilemma is no longer relevant and the problem can be considered fully cooperative. 


There is a clear divide between the reviewers: While 3 of the reviewers provided very positive reviews, one reviewer remained critical. While the authors addressed the concerns and provided additional (favorable) experimental comparison at short notice, this did not change the score.


Given the positive reviews, the discussion and the overall contributions of the paper, I recommend this paper to be accepted.

However, here are two requests: The problem is currently introduced as a Dec-POMDP. This is inaccurate since different agents have different reward functions. 

Furthermore, two necessary baselines need to be added: 1)  Comparison to a fully cooperative setting where the global reward (sum of all agents’ reward) is maximized. 2) Each agent maximizes the total local reward (i.e. neighbors’ and own reward). This is very close to phi=90 degrees but includes the agent’s own reward. 

Decision: Accept (Poster)