Emergent Road Rules In Multi-Agent Driving Environments
Authors: Avik Pal
Jonah Philion
Yuan-Hong Liao
Sanja Fidler
Venue: ICLR 2021 Poster (2021)

Abstract: For autonomous vehicles to safely share the road with human drivers, autonomous vehicles must abide by specific "road rules" that human drivers have agreed to follow. "Road rules" include rules that drivers are required to follow by law – such as the requirement that vehicles stop at red lights – as well as more subtle social rules – such as the implicit designation of fast lanes on the highway. In this paper, we provide empirical evidence that suggests that – instead of hard-coding road rules into self-driving algorithms – a scalable alternative may be to design multi-agent environments in which road rules emerge as optimal solutions to the problem of maximizing traffic flow.  We analyze what ingredients in driving environments cause the emergence of these road rules and find that two crucial factors are noisy perception and agents’ spatial density.  We provide qualitative and quantitative evidence of the emergence of seven social driving behaviors, ranging from obeying traffic signals to following lanes, all of which emerge from training agents to drive quickly to destinations without colliding. Our results add empirical support for the social road rules that countries worldwide have agreed on for safe, efficient driving.

Published as a conference paper at ICLR 2021
EMERGENT ROAD RULES IN MULTI-AGENT DRIVING ENVIRONMENTS
Avik Pal', Jonah Philion2:3.4, Yuan-Hong Liao?.+, Sanja F.d ITT Kanpur, Uniivessity of Toronto, SNVIDIA, Veector Institute avikpal@cse aitk.ac..a (iphilion, andrew, Fidler)ecs.
ABSTRACT
For autonomous vehicles to safely share the road with human drivers, autonomous vehicles must abide by specific "road rules" that human drivers have agreed to follow. "Road rules" include rules that drivers are required to follow by law such the requirement that vehicles stop at red lights rules - such as the implicit designation of fast lanes on the highway. In this paper, we provide empirical evidence that suggests that instead of hard-coding road rules into self-driving algorithms environments in which road rules emerge as optimal solutions the problem of maximizing traffic flow. We analyze what ingredients in driving environments cause the emergence of these road rules and find that two crucial factors are noisy perception and agents' spatial density. We provide qualitative and quantitative evidence of the emergence of seven social driving behaviors, ranging from obeying traffic signals to following lanes, all of which emerge from training agents drive quickly to destinations without colliding. Our results add empirical support for the social road rules that countries worldwide have agreed on for safe, efficient driving.
well more subtle social
scalable alternative may be to design multi-agent
INTRODUCTION
Public roads significantly more safe and efficient when equipped with conventions restricting how one may use the roads. These conventions are, some extent, arbitrary. For instance, "drive on the left side of the road" con- vention is. practically speaking. no better worse than "drive on the right side of the road" convention. However, the decision to reserve some orientation as the canonical orientation for driving is far from arbitrary in that estab- lishing such convention improves both safety (doing so decreases the probability of head-on collisions) and efficiency (cars can drive faster without worrying about dodging oncoming traffic).
In this paper, we investigate the extent to which these road rules mules-like like the choice of a canonical heading orientation can be learned in multi-agent driving environments which agents are trained drive different destinations quickly as possible without colliding with other agents, As visualized in Figure our agents are initialized in random positions in different maps (either synthetically generated or scraped from real-world intersections from the nuScenes dataset (Caesar et al., 2019)) and tasked with reaching randomly sampled feasible target destination. Intuitively, when agents have full access to the map and exact states of other agents, optimizing for traffic flow leads the agents to drive in qualitatively aggressive and nn-humanlike ways. However, when perception is imperfect and noisy, we show in Section that the agents begin to rely on constructs such as lanes, traffic lights, and safety distance to drive safely at high speeds.
Figure 1. Multi-agent Driving Environ- ment We train agents to travel from a a->b as quickly as possible with limited perception while avoiding collisions and find that "road rules" such as lane following and traffic light usage emerge.
Notably, while prior work has primarily focused on building driving simulators with realistic sensors that mimic LiDARs and cameras (Dosovitskiy et al., 2017; Manivasagam et al., 2020; Yang et al., 2020; Bewley et al., 2018), definition of reward and perception noise that determine agents trained the simulator exhibit realistic behaviors. We hope that the lessons in state space, action space, and reward design gleaned
focus the high-level design choices for the simulator such the

---

Published as a conference paper at ICLR 2021
from this paper will transfer to simulators in which the prototypes for perception and interaction used in this paper are replaced with more sophisticated sensor simulation. Code and Documentation for all experiments presented in this paper can be found in our Project Page'.
Our main contributions are as follows:
We define multi-agent driving environment in which agents equipped with noisy LiDAR sensors are rewarded for reaching given destination as quickly as possible without colliding with other agents and show that agents trained in this environment learn road rules that mimic road rules common in human driving systems.
We analyze what choices in the definition of the MDP lead to the emergence of these road rules and find that the most important factors are perception noise and the spatial density of agents in the driving environment.
We release a suite of 2D driving environments" with the intention of stimulating interest within the MARL community to solve fundamental self-driving problems.
2 RELATED WORKS
Reinforcement Learning Deep Reinforcement Learning (DeepRL) has become an popular frame- work that has been successfully used solve Atari (Mnih al., 2013), Strategy Games (Peng et al., 2017; OpenAI, 2018), 1)), and Traffic Control (Wu et al., 2017a; Belletti et al., 2018). Vanilla Policy Gradient (Sutton et al., 2000) is an algorithm that optimizes an agent's policy by using conte-carlo estimates of the expected return. Proximal Policy Optimization (Schulman et al., 2017) which we use in this work - is an on-policy policy gradient algorithm that alternately samples from the environment and optimizes the policy using stochastic gradient descent. PPO stabilizes the Actor's training by limiting the step size of the policy update using a clipped surrogate objective function.
Multi-Agent Reinforcement Learning The central difficulties of Multi-Agent RL (MARL) include environment non stattiation (Hernandez-Leal et al., 2019; 2017; Busoniu et al., 2008; Shoham et al., 2007), credit assignment (Agogino and Tumer, 2004; Wolpert and Tumer, 2002), and the curse of dimensionality (Busoniu et al., 2008; Shoham et al., 2007). Recent works (Son et al., 2019; Rashid et al., 2018) have attempted to solve these issues in centralized training decentralized execution framework by factorizing the joint action-value Q function into individual Q functions for each agent. Alternatively, MADDPG (Lowe al., 2017) and PPO with Centralized Critic (Baker et al., 2019) have also shown promising results in dealing with MARL Problems using policy gradients.
Emergent Behavior Emergence of behavior that appears human-like in MARL (Leibo al., 2019) has been studied extensively for problems like effective tool usage (Baker et al., 2019), ball passing and interception in 3D soccer environments (Liu et al., 2019), capture the flag (Jaderberg 2019), hide and seek (Chen et al., 2020; Baker et al., 2019). communication (Foerster et al., 2016; Sukhbaatar et al., 2016), and role assignment (Wang et al., 2020). For autonomous driving and traffic control (Sykora et al., 2020), emergent behavior has primarily been studied in the context of imitation learning (Bojarski et al., 2016; Zeng et al., 2019; Bansal et al., 2018; Philion and Fidler, 2020; Bhattacharyya et al., 2019). Zhou et al. (2020) solve similar problem as ours from the perspective of environment design but fail to account for real-world aspects like noisy perception, which are inherent for emergent rules. In contrast to works that study emergent behavior in mixed-traffie autonomy (Wu et al., 2017b), embedded rules in reward functions (Medvet et al., 2017; Talamini et al., 2020) and traffic signal control (Stevens and Yeh, 2016), we consider a fully autonomous driving problem in decentralized execution framework and show the emergence of standard traffic rules that are present in transportation infrastructure.
al.,
PROBLEM SETTING
We frame the task of driving as a discrete time Multi-Agent Dec-POMDP (Oliehoek et al., 2016), Formally, a Dec-POMDD is tuple G = (S, A,P,r, Po. O, n, 7, 7). S denotes the state space of the environment, A denotes the joint action space of the n agents s.t. Ui_1 ag E A, P is the state transition probability P: S x A S + I is bounded reward function r S x a + R. po is the initial state distribution, O is the joint observation space of the n agents s.t. Uh oz E O E (0, 1] is the discount factor, and T is the time horizon.
wttp://.ooooooaaooooooooooaaaiat Ptttps://gi thub. htt.://cooooeeeeeeeeeeeee watpsocaa
2

---

Published as a conference paper at ICLR 2021
We parameterize the policy ne o a R+ of the agents using neural network with parameters In all our experiments, the agents share a common policy network. Let the expected return for the 1thh agent be (=()))) Er
EE y (rr,, a.t)). where t (su, ai,u ST-1, as,r-1) is the trajectory of the agent, So Po. Oi, 30(0,,100,1) and $141 P(s st, Ui_1 as,.). Our objective is to find the optimal policy which maximizes the utilitarian objective Σ 1/s.
Reward We use high-level generie rewards and avoid any extensive reward engineering. The agents receive reward of +1 if they successfully reach their given destination and -1 if they collide with any other agent or go off the road. In the event of a collision, the simulation for that agent halts. In an inter-agent collision, we penalize both agents equally without attempting to determine which agent was responsible for the collision. We additionally regularize the actions of the agents to encourage smooth actions and add normalized penalty proportional to the longitudinal distance of the agent from the destination to encourage speed. We ensure that the un-discounted sum of each component of the reward for agent over the entire trajectory is bounded 0 S 1. We combine all the components to model the reward ri, received by agent at timestep /. as follows:
(St, dg,z) = I [Reached the Destination at timestep t]
I [Coolidde for the first time with an obstacle/agent timesetep
1 04.s (..((e1) (a:)
Distance to goal from current position
T Distance to goal from initial position where (as)max is the maximum magnitude of action that agent can take
T
Map and Goal Representation We use multiple environments: four-way intersection, highway tracks, and real-world road patches from nuScenes (Caesar et al., 2019), to train the agents. The initial state distribution Po is defined by the drivable area in the base environment. The agents "sense" traffic signal if they are near a traffic signal and facing the traffic signal. These signals are represented discrete values- 0, 0.5, 1.0 for the 3 signals and 0.75 for no signal available in the observation space. In all but our communication experiments, agents have the ability to communicate exclusively through the motion that other agents observe. In our communication experiments, we open a discrete communication channel designed to mimic turn signals and discuss the direct impact on agent behavior. Additionally, to mimic a satnav dashboard, the agents observe the distance from their goal, the angular deviation of the goal from the current heading, and the current speed.
LiDAR observations We simulate a LiDAR sensor for each agent by calculating the distance to the closest object dynamic or static along each of equí-angular rays. We restrict the range of the LiDAR to 50m. Human eyes themselves are imperfect sensors and are easily thwarted by weather, glare, visual distractions; in experiments,
study the importance of this "visual" sensor by introducing noise in the sensor. We introduce 2% lidar noise by random uniformly dropping (assigning a value of 0) x% of the rays at every timestep (Manivasagam et al., 2020). To give agents the capacity to infer the velocity and acceleration of nearby vehicles, we concatenate the LiDAR observations from T' = 5 past timesteps.
4 POLICY OPTIMIZATION
4.1 POLICY PARAMETERIZATION
In our experiments we consider the following two parameterizations for our policy network(s):
1.
Fixed Track Model: We optimize policies that output a Multinomial Distribution over fixed set of discretized accelcration values. This distribution is defined by TT(a))), where Te our policy network, a the acceleration, and o the observation. This acceleration used to drive the vehicle along fixed trajectory from the source to target destination. This model trains efficiently but precludes the emergence of lanes.
2. Spline Model:
To train agents that are capable of discovering lanes, we use a two-stage formulation inspired by Zeng et al. (2019) in which trajetories shapes are represented by clothoids and time-dependence is represented by a velocity profile. Our overall policy is factored into two "subpolicies" spline subpolicy is tasked with predicting the spline along which the vehicle is supposed to be driven. This subpolicy conditions initial local observation of the environment and
spline subpolicy and an acceleration subpolicy. The
3

---

Published as a conference paper at ICLR 2021
predicts the spline (see Section 5). We use Centripetal Catmull Rom Spline (Catmull and Rom, 1974) to parameterize this spatial path. The acceleration subpolicy follows the same parameterization from Fixed Track Model, and controls the agent's 's motion along this spline. We formalize the training algorithm for this bilevel problem in Section 4.4.
Note that the fixed track model is special case of the spline model parameterization, where the spline is hard-coded into the environment. These splines can be extracted from lane graphs such as those found in the HD maps provided by the nuScenes dataset (Caesar et al., 2019).
4.2 PROXIMAL POLICY OPTIMIZATION USING CENTRALIZED CRITIC
We consider Centralized Training with Decentralized Execution approach in our experiments. During training. the critic has access to all the agents' observations while the actors only see the local observations. We Proximal Policy Optimization (PPO) (Schulman al., 2017) and Generalized Advantage Estimation (GAE) (Schulman et al., 2015) to train our agents. Let Va denote the value function. To train our agent, optimize the following objective:
L (6) E min (f)) clip(F(o), - 1- e) - e1 (Vo((s))) - Vtarget) C2 no)]
where F(Ø) <a(u]o)
*eats (alo) Å is the Estimated Advantage, H--) measures entropy, (c, c1, c2) are hyperparameters, and Viarget is the value estimate recorded during simulation. Training is performed using a custom adaptation of SpinningUp (Achiam, 2018) for MARL and Horovod (Sergeev and Balso, 2018). The agents share common policy network in all the reported experiments. In our environments, the number of agents present can vary over time, as vehicles reach their destinations and new agents spawn. To enforce permutation invariance across the dynamic pool of agents, the centralized critic takes as input the mean of the latent vector obtained from all the observations.
Algorithm 1: Alternating Optimization for Spline and Acceleration Control
Result:
Trained Subpolicies and IT.
Na f Spline Subpolicy, Te f Acceleration Subpolicy, V + Value Function for Accelcration Control; for N do
/* Given Te optimize T
for =
Ki do Collect set of Partial Trajectories D1, using a. (o.))) and o. arg max (a(o)).):
Compute and store the normalized rewards R:
end
Optimize the parameters @ using the objective Lz(0) and stored trajectories D1; Given optimize ze Te and vfe for I K3 do Collect set Partial Trajectories D2,x using a, f- arg max mo(alos) and aa ~ To((a);
Compute and store the advantage estimates À using GAE;
end
Optimize the parameters using the objective La(ø) and stored trajectories Da;
end
4.3 SINGLE-STEP PROXIMAL POLICY OPTIMIZATION
In a single-step MDP, the expected return modelled by the critic is equal to the reward from the environment as there are no future timesteps. Hence, optimizing the critic is unnecessary in this context. Let R, denote the normalized reward. The objective function defined in Sec 4.2 reduces to:
L2( = E [i))( clip(F(0), - e, e) - c2ld (T)
(I)
4.4 BILEVEL OPTIMIZATION FOR JOINT TRAINING OF SPLINE/ACCELERATIUN SUBPOLICIES
In this section, present the algorithm we use to jointly train two RL subpolicies where one subpolicy operates in a single step and the other operates over a time horizon 2 1. The subpolicies operate oblivious of each other, and cannot interact directly. The reward for the spline subpolicy is the undiscounted sum of the rewards received by the acceleration subpolicy over the time horizon. Pseudocode is provided in Algorithm 1, The algorithm runs for total of N iterations. For each
4

---

Published as a conference paper at ICLR 2021
iteration, collect K1 and K₂ samples from the environment train the spline and acceleration
subpolicy respectively. We denote actions and observations using (ag, 0,) and (aa: 0a) (not to be confused with the notation used in Section 3) for the spline and acceleration subpolicy respectively.
EMERGENT SOCIAL DRIVING BEHAVIOR
We describe the various social driving behaviors that emerge from experiments and analyze them quantitatively. Experiments that do not require an explicit emergence of lanes - -,.1, 5.3, 5.4, 5.7 use the fixed track model. We use the Spline Model for modeling lane emergence in Experiments 5.2 and 5.5. The results Section 5.6 are a consequence of the increased number training agents, and observed using either agent type. Qualitative rollouts are provided in project page.
Derance ta Intersction led D.E.
Ditemce tnterstcoon - a., 25, 1531 Desarcet brosectes rnd IEA.
Dteancs Insereioo tnd 122.5
-10
Figure Traffic Light Usage Actions taken by the agents varying amount perception noise and traffic signal (represented hy color coding the hox plots). A strong correlation exists hetween acceleration, traffic signal and distance from the intersection. As the agent approaches the intersection, the effect of red signal on the actions is more prominent (characterized by the reduced variance). However, the variance for green signal increasing, since agents need to marginally slow down upon detecting agent in front of them. With more LiDAR noise, detecting location the intersection becomes diffcult far from the intersection. Agents with better visibility can potentially safely cross the intersection on a red light, hence the mcan accelcration near the intersection goes down with increasing lidar noise. Finally, with increasing number of agents, the variability of the actions increase due presence of leading vehicles on the same path.
agents prematurely
down
Number of Training Agents
Number Training Agents
Number Training Agents
6,52
0.29
Green Siginl
0,
8.54
0.00
8.40
Fraction
9.73
0.38
0.80
0.42
0.92
44
0.40
0.57
5.38
Figure 3. Traffic light usage Spatial 2D histogram of synthetic intersection showing the fraction of agents that see a green signal. Once agents have entered the intersection we consider signal they saw just before entering it. A darker shade of green in the intersection shows that fewer agents violated the traffic signal. In a single agent environment, there is no need to follow the signals. Agents trained on environment with higher spatial density (top row) violate the signal less frequently. Agents also obey the signals more with increased perception noise (bottom row).

---

Published as a conference paper at ICLR 2021
Time Step
Step
Time
0.30
2.020
der Notee (853
030
0.66
Narmal und
Postion
Narma Lans Pox
Lane lran
Figure Lanes emerge with more perception noise When perception noise increased agents follow lanes more consistently (higher peaks for 25% and 50% LiDAR models). However, after a certain threshold imperfect perception leads poor convergence can be seen for the 75% LiDAR model.
5.1 STOPPING AT A TRAFFIC SIGNAL
Traffic Signals are used to impose an ordering on traffic flow direction in busy 4-way intersections. We simulate 4-way intersection, where agents need reach the opposite road pocket in minimum time. We constrain the agents to move along a straight line path. We employ the fixed track model and agents learn to control their accelerations reach their destination. We study the agent behaviors by varying the number of training agents and their perception noise (Figure 2 & 3).
Note that the agents merely observe ternary value representing the traffic light's state, not color. To make the plots in this section, we visually inspect rollouts for each converged policy to find permutation of the ternary states that align with human red/yellow/green traffic light conventions.
Time
Number Training Agents
20
0.4
3.9
13
Shep
3.
-1.0
-0.75
-1.53
-1.25
0.00
0.25 0.50 0.75
Normaltze Lane Puitoss
Figure 5. Lanes emerge with more agents With low spatial agent density. the subpolicies converge to roundabout motion. On increasing the spatial density, the agents learn to jointly obey the traffic signals and lanes. Agents trained with higher spatial den- sity follow two lanes exclusively on one side of the road.
Figure 6. Spatial Positions on Intersection Environment Agents trained in 8 agent environment cross the intersection in two discrete lanes the right hand side of the road. The chosen lane depends on the starting positions of the agents. Agents starting towards the left tend to take the inner lane to allow faster traffic flow.
5.2 EMERGENCE OF LANES
To analyze lane emergence, we relax the constraint on the fixed agent paths in the setup of Section 5.1. We use the Bilevel PPO Algorithm (Section 4.4) train the two subpolicies. The spline subpolicy predicts a deviation from path along the roads' central axis connecting the start position to the destination, similar to the GPS navigation maps used by human drivers. The acceleration subpolicy uses the same formulation as Section 5.1.
To empirically analyze the emergence of lanes, plot the "Normalized Lane Position" of the agents over time. "Normalized Lane Position" is the directional deviation of agent from the road axis. We consider the right side of the road (in the ego frame) have positive "Normalized Lane Position".

---

Published as a conference paper at ICLR 2021
Figures 4 & 5 show the variation with lidar noise and number of training agents respectively. Figure 6 shows the spatial positions of the agents for the 8 agent perfect perception environment.
5.3 RIGIT OF WAY
For tasks that can be performed simultaneously and take approximately equal time for completion, First In First Out (FIFO) scheduling strategy minimizes the average waiting time. In the context of driving through an intersection where each new agent symbolizes a new task, the agent that arrives first the intersection should also be able leave the intersection first. In other words, given any two vehicles, the vehicle arriving at the intersection first has the "right of way" over the other vehicle.
Let the time at which agent [n] arrives at the intersection be (ta); and leaves the intersection be (ta);. If 3j E [n] {i}, such that (fa); < (ta); and (ta); > (ta)j, we say that agent i doesn't It respect j's right of way. We evaluate this metric on a model trained on a nuScenes intersection (Figure 8). We observe that, at convergence, the agents follow this right of way 85.25 8.9% of time (Figure 7).
X
-4
78-
0.00
0.66
82-
Number Training
Figure 8. nuScenes In- tersection used for Right of Way (see Section 5.3) and Communication (see Sec- tion 5.4) Training and Evalu- ation. The red and green dots mark the location of the traf- fic signals and their current ternary state.
LOGAI Nuzee (%6
Figure 7. Right of Way Agents are inereasing able to successfully reach
Figure 9. Emergent Communica- tion with Perception Noise Speaker Consistency & Pearson coefficient be- tween the agent's heading and its sent message increase with increase in Per- ception Noise. Since all agents have faulty sensors, agents each other to navigate by propagating their head- ing to their trailing car.
their destinations (denoted by the de- creasing red line) with more training episodes. The agents also increasing obey the right of way (denoted by the
increasing green line). The error bars are constructed using μ
5.4 COMMUNICATION
One way to safely traverse an intersection is to signal one's ' intention to nearby vehicles. We analyze the impact of perception noise on emergent communication at an intersection (Figure 8). In particular, we measure the Speaker Consistency (SC), proposed in Jaques et al. (2019). SC can be considered as the mutual information between an agent's 's message and its future action. We report the mutual information and the Pearson coefficient (Freedman et al., 2007) between the agent's heading and its sent message. We limit the communication channel to one bit for simplicity. Each car only receives a signal from the car in the front within -30 and 30 Figure 9 shows that agents rely more heavily on communication at intersections when perception becomes less reliable.
5.5 FAST LANES ON A HIGHWAY
Highways have dedicated fast lanes to allow a smooth flow of traffic. In this experiment, we empirically show that autonomous vehicles exhibit a similar behavior of forming fast lanes while moving on a highway when trained to maximize traffic flow. We consider a straight road with a uni-directional flow of traffic. Agents are spawned random positions along the road's axis.
Recobbratoon salt hainn
Figure 10. Spatial Positioning in Highway Environment Agents start from one of the positions marked by
and need to reach the end position represented by the solid black line. The agents with a higher acceleration rating move the right hand side while the lower rated ones drive on the left hand side.
7

---

Published as a conference paper at ICLR 2021
Every agent is assigned scalar value called "Acceleration Rating," which scales the agent's accelera- tion and velocity limits. Thus, a higher acceleration rating implies a faster car. The spline subpolicy predicts an optimal spline considering this acceleration rating. Even though the agents can decide to move straight by design, it is clearly not an optimal choice as slower cars in front will hinder smooth traffic flow. Figure 10 shows that agents are segregated into different lanes based on their Acceleration Rating. Figure visualizes this behavior over time.
Nomositod Frokoon
0.5
0.0
-05
05
0.5 0.7
0.8
2.9
10
2.5
06 0.3 08 0.9
1.0 0.5
2.5 0.7 0.8
0.3
3.2
0.5
0.5
6.7 0.8 5.5 1.0
Saring
Figure 11. Fast Lane Emergence Visualization of rollouts from a 10-agent highway environment. In y-axis, we show the agent's ' position relative to the axis of the road normalized by road width. The agents reach consensus where the faster agents end up the right-hand side lane. This pattern ensures that slower vehicles do not obstruct faster vehicles once the traffic flow has reached a steady state.
5.6 MINIMUM DISTANCE BETWEEN VEHICLES
In this task, we evaluate the extent to which the agents learn to respect a minimum safety distance between agents while driving. When agents are too close, they are at a greater risk of colliding; when agents are too far, they are not as efficient traveling a + b.
Sall Divn
To derive human-like "safety distance", we assume agents can change their velocity according to v² the final and initial
Certance to Landing Car (m))
u² + 2ad, where and velocities respectively, a is acceleration, and d is the distance for which the acceleration remains constant. Hence, for an agent to stop entirely from a state with of
Figure 12. Safety Distance Maintained in Nuscenes Environment Agents need to maintain minimum of the Match-Ma Distance to be able to safely stop. 98.45 98.45% of the agents in this plot leam to respect this distance threshold.
velocity vo, needs at least a distance of 2ames front of it, where amer is the maximum possible deceleration of the agent.
Our agents perceive the environment through LiDAR; thus, agents can estimate nearby agents' velocity and acceleration. We define the safe distance the distance needed for trailing agent to have a zero velocity in the leading agent's frame. We assume that the leading agent travels with As²
constant velocity, and as such, the safe distance is defined by 20maz= where As is the relative velocity. Any car having distance greater than this safely slow down. Agents trained nuScenes Intersections obey this safety distance around 98.45% time (Figure 12).
Sali Drinna
Spend Mording Diearon
Agest
Dutrine from Nearst Federsian (m)
Actus Dottres to Noaee Pedeatiin (r)
Figure 13. Safety Distance from Pedestrians We observe that most the agents maintain a distance greater than the recommended speed-matching distance from the pedestrians. The shaded region in the KDE plots indicate the 95% confidence interval and the dotted line is the sample mean
5.7 SLOWING DOWN NEAR A CROSSWALKK
In this task, we evaluate if agents can detect pedestrians and slow down in their presence. We augment the environment setup of Sec. 5.5, to include crosswalk where at most 10 pedestrians spawned at the start of every rollout. The pedestrians cross the road with a constant velocity. If any agent collides with pedestrian, they get collision reward of -1, and the simulation for that agent stops.
8

---

Published as a conference paper at ICLR 2021
The KDE plots in Figure 13 show that the agents indeed detect the pedestrians, and most of them maintain a distance greater than 6m. To determine the agents can safely stop and prevent collision with the pedestrians, we calculate a safe stopping distance of x² 2 where s is the velocity of the agent. In the scatter plot, we observe that most agents adhere to this minimum distance and drive at distance, which lies in the safe driving region.
6 DISCUSSION
Distancs Drivin
Salety Distance Ageet Driving
Friidgg
Speed Manc--Maciir.... Disana
10
Distanc Loading
Leadinx
Figure 14. Safety Distance for Humans vs. Safety Distance for RL Agents The agents trained in our MDP (right) tend violate the safety distance slightly more than human drivers (left) (98.45% % 99.56%) but in both cases safety distance is observed the majority of the time (green triangular region).
6.1 STATISTICS OF HUMAN DRIVING
In some cases, the same statistics accumulated over agent trajectories that we use in Section 5 to quantitatively demonstrate emergence can also be accumulated over the human driving trajectories labeled in the nuScenes dataset. In Figure 14, we visualize safety distance statistics across nuScenes trajectories and safety distance statistics across RL agents trained on nuScenes intersections side- by-side. The nuScenes trainval split contains 64386 car instance labels, each with associated trajectory. For each location along the trajectory, we calculate the safety distance as described in Section 5.6. The same computation is performed over RL agent trajectories. The agents trained in our MDP tend to violate the safety distance more than human drivers. However, in both cases, a safety distance is observed the vast majority of the time (green triangular region).
6.2 FUTURE WORK
By parameterizing policies such that agents must follow the curve generated by the spline subpolicy initialization (see Section 4.4), we prevent lane change behavior from emerging. The of more expressive action space should address this limitation at the cost of training time. Additionally, the fact that our reward is primarily based on agents reaching destinations means that convergence is slow on maps that are orders of magnitude larger than the vehicles' dimensions. One possible solution to training agents to navigate large maps would be generate a curriculum of target destinations, as in Mirowski et al. (2018).
7 CONCLUSION
In this paper. we identify a lightweight multi-agent MDP that empirically captures the driving problem's 's essential features. We equip our agents with a sparse LiDAR sensor and reward agents when they reach their assigned target destination as quickly as possible without colliding with other agents in the scene. We observe that agents in this setting rely on a shared notion of lanes and traffic lights to compensate for their noisy perception. We believe that dense multi-agent interaction and perception noise are critical ingredients designing simulators that seek to instill human-like road rules in self-driving agents.
8 ACKNOWLEDGEMENT
This work was supported by NSERC and DARPA's XAI program. SF acknowledges the Canada CIFAR AI Chair award the Vector Institute. AP acknowledges support by the Vector Institute.

---

Published as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Adrian K Agogino and Kagan Tumer. Unifying temporal and structural credit assignment problems. 2004.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint 1rXiv:1909007528. 2019.
Mayank Bansal, Alex Krizhevsky, and Abhijit S. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. CoRR, abs/1812.03079, 2018. URL hnttp://amte htt.v..wxvvv...../........... org/ abs/18 03079.
F. Belletti, Daniel Haziza, Gabriel Gomes, and A. Bayen. Expert level control of ramp metering based on multi-task deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 19:1 2018.
Alex Bewley, Jessica Rigley, Yuxuan Liu, Jeffrey Hawke, Richard Shen, Vinh-Dieu Lam, and Alex Kendall. Learning to drive from simulation without real world labels. CoRR, abs//812.0033 112.03823 2018. URL http: arxiv. www//bsssss...o..o.............. /tcos/////1. 111111111110 03823.
Raunak Bhattacharyya, Derek Phillips, Changliu Liu, Jayesh K Gupta, Katherine Driggs-Camplee and Mykel Kochenderfer. Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 789-795. JEEE, 2019.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. CoRR, abs/1604.07316, 2016. URL http:://arxiii org/ abs/ / 604 07316.
Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. JEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156-17,, 2008.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuScenes: multimodal dataset for autonomous driving. arXiv preprint
Edwin Catmull and Raphael Rom. A class of local interpolating splines. In Computer aided geometric design, pages 317-326. Elsevier, 1974.
Boyuan Chen, Shuran Song, Hod Lipson, and Carl Vondrick. Visual hide and seek. In Artificial Life Conference Proceedings, pages 645. MIT Press, 2020.
Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: 2017. URL htttp://axxiv..//
an open urban driving simulator. CoRR, abs / / /111111111 03938.
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agen reinforcement learning. In Advances in neural information processing systems, pages 2137-2145, 2016.
David Freedman, Robert Pisani, and Roger Purves. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York, 2007.
Pablo Hernandez-Leal Michael Kaisers, Tim Baarslag, and Enrique Munoz. de Cote. survey of learn ing in multiagent environments: Dealing with non-stationarity. arXiv preprint 1 2017.
Pablo Hernandez-Leal Bilal Kartal, and Matthew E. Taylor. survey and critique of multiagent deep reinforcement leamning Autonomous Agents and Multi-Agent Systems, 33:750- 797, 2019.
10

---

Published as a conference paper at ICLR 2021
Max Jaderberg, Wojciech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population- based reinforcement learning. Science, 364(6443):859-866 -65, 2019.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pages 3040-3049. PMLR. 2019.
Joel Z. Leibo, Edward Hughes, Mare Lanctot, and Thore Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. CoRR, Abss/903.00744, 2019. URL http: / //apvvv.//iiv/aa//aa///a org/ abs 1903. 00742.
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emergent coordination through competition. arXiv preprint 190.00190..7755, 2019.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiv environments. In Advances in neural information processing systems, pages
2017.
Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang. Wei-Chiu Ma, and Raquel Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world, 2020.
Eric Medvet, Alberto Bartoli, and Jacopo Talamini. Road traffic rules synthesis using grammatical evolution. In European Conference on the Applications of Evolutionary Computation, pages 173-188, Springer, 2017.
Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith An- derson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell, Learning to navigate in cities without a map. CoRR, abs/1804.00068. 2018. URL http://arxist
001168.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
5602.
Frans A Oliehoek, Christopher Amato, al. A concise introduction decentralized POMDPs. volume 1. Springer, 2016.
OpenAl. Openai five. https: /blog. httpaaacccomppppaaaeeeeee 2018.
P. Peng. Quan Yuan, Ying Wen, Y. Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-cooriinan nets for learning to play starcraft combat games. ArXiv, abs/1 2017.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision, 2020.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 14885 2018.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 1ee6.000630333 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv: 00020057999 2018.

---

Published as a conference paper at ICLR 2021
Yoav Shoham, Rob Powers, and Trond Grenager. If multi-agen agent learning is the answer, what is the question? Artificial intelligence, 171(7):365-3) 2007.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv: 1905:05000 2019.
Matt Stevens and Christopher Yeh. Reinforcement learning for traffic optimization. Stanford. edu, 2016.
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. In Advances in neural information processing systems, pages 2244 2016.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in neural information processing systems, pages 1057-1063, 2000.
Quinlan Sykora, Mengye Ren, and Raquel Urtasun. Multi-agent routing value iteration network. arXiv preprint arXiv:2007.050000 2020.
Jacopo Talamini, Alberto Bartoli, Andrea De Lorenzo, and Eric Medvet. On the impact of the rules on autonomous drive learning. Applied Sciences, 10(7):2394, 2020.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles, 2020.
David H Wolpert and Kagan Tumer. Optimal payoff functions for members of collectives. In Modeling complexity in economic and social systems, pages 355-369. World Scientific, 2002.
Cathy Wu, Aboudy Kreidich, Eugene Vinitsky, and Alexandre M. Bayen. Emergent behaviors in
auxeddautonomy traffic. volume 78 Proceedings of Machine Learning Research, pages 398-407. PMLR, 13-15 Nov 2017a. URL http: pto.eedooeeeedeeeooooo ngs. mlr www.mmlrrrrerr. press / v wul 7 html.
Cathy Wu, Aboudy Kreidieh, Eugene Vinitsky, and Alexandre M Bayen. Emergent behaviors in mixed-autonone traffic. In Conference on Robot Learning, pages 398-407, 2017b.
Zhenpei Yang. Yuning Chai, Dragomir Anguelov, Yin Zhou, Pei Sun, Dumitru Erhan, Sean Rafferty, and Henrik Kretzschmar. Surfelgan: Synthesizing realistic sensor data for autonomous driving, 2020.
W. Zeng, W. Luo. S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8652 000-8661,,22 2019.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezace, Peyman Yadmellat, Mohsen Rohani, Nicolas Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexan- der Cowen Rivers, Zheng Tian, Daniel Palenicek, Haitham bou Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving, 2020.
12


Reviews:
[{"review_id": "tW1G-dCCy2B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigate how to design simulation environments so the the agent trained with them can master social rules.\nCons:\n1. The paper is well written and easy to read and understand. Thanks!\n2. The experiments are solid and well defined. \n\nMy major concern of this paper are:\n1. The authors seems to only considered the noise of sensors and the number of agents, and these two factors happened to induce social behaviors like following lanes and stopping at traffic signals. In other words, I am not quite convinced that with all vehicles being automated, sensor noise will cause them to formulate rules that what human drivers follow today. Since human driving interactions are complex, I do not think that sensor noise would be enough to induce them. \n2. I would be happy to see how this configuration of simulation environment compares with reward guided social behaviors. For example, we can design a reward to encourage agents follow right of way. I think this way might be more direct and powerful. \n2. The number of agents seems to be too small and may affect the formulation of road social rules.\n3. It seems like the agents did not take traffic signal status as input for the action selection, then how did they formulate the signal control rules. \n4. Figures 2 and 3 needs better explanations for readers to understand. \n\nOverall, I think this paper needs better formulation of the logics and also a deep investigation of how the simulation variations lead to those behaviors. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "k5sMKkdriwB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a bilevel MARL method that can learn road rules and conventions implicitly without hard coding. It is quite interesting to see a simple and straightforward idea that is effective in this task setting. However, this paper needs to be further polished in terms of its delivery, completeness, and evaluation.\n\nMethodology:\nHow is the noise introduced to the mimicked LiDAR perception results?\nIn Spline Model, does the trajectory shape only depends on the initial state without considering the afterward interaction?\nThe clarity and completeness of writing could be significantly improved.\nThe equations, terms, and the algorithm in Section 4 need sufficient explanations. E.g., what are vf^{target}_t, H, K_1, K_2, N? What is the complete reward list?\n\nExperiments:\nIt seems the route is relatively short, and the driving scenarios used for different tasks are distinct. I was wondering whether the generalization of the proposed method and the learned rules could be validated.\n\nNo baselines have been evaluated or compared. Which experimental results are or the fixed track model? What are the differences between the two models in the results? What are the reward values during the training phase?\n\nThe captions and axis labels are difficult to read.\nWhat do the colors mean in Fig. 2?\n\n\n====\nMy rating has been updated after the rebuttal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "U9QpjLeXtLc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The output of the work is an MDP model that is capable of encoding complex traffic rules including traffic signals, lanes, right of way FIFO etc. Various different traffic environments such as intersections, highways, nuScenes are considered.\n\nThe MDP that results from this work is very useful for future research. Currently, there are very few simulators that encode complex traffic rules or provide the flexibility to perform research. Therefore, the possibility of new simulators resulting from this MDP is exciting and useful to encourage and promote research in autonomous driving, especially in dense and heterogeneous environments.\n\nI am glad that the authors have provided the source code **with extensive documentation and instructions** on how to run the code and reproduce the results. \n\nUnless there is some major flaw with this paper (that I may have missed), there is no reason to reject this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8DuD3QuadM9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to learn traffic rules (e.g. traffic lights, speed limits) via multi-agent RL (MARL) from observations rather than hardcoding the rules into the algorithms. To this end, the authors claims contributions in:\n\n-- Defining a multi-agent driving environment, where each agent has incomplete observation (noisy LiDAR), and is rewarded for reaching a destination quickly without colliding. Experiments show that road rules can be learned.\n\n-- Ablation on the choice of the MDP, and insights that perception noise and spatial density of agents are important to successfully learn the environment.\n\n-- Authors promise to release a suite of 2D driving environments for future MARL research in self-driving.\n\nI must admit that I am mainly a computer vision person, and I only have limited experience with RL or MARL. However, I hope that my assessment below is still better than an educated guess. I apologize in advance if there is any obvious misunderstanding.\n\nStrengths:\n\n-- Novelty in problem statement and at a high level: The problem statement of learning hard traffic rules via observing logs seems new. The application of MARL to the problem is new too AFAIK. And as the authors stated, the majority of multi-agent behavior works have been using imitation learning. (Maybe consider citing [1] line of work as well, which deals with multi-agent imitation learning). The proposed PPO-based method seems a good alternative.\n\n-- Extensive experiments: There are 7 small but specific tasks such as \"Traffic lights\", \"Emergence of lanes\", etc., where the authors provided evidences to the claims that perception noise and spatial density of agents are crucial for the method's success.\n\n[1]: Multi-Agent Generative Adversarial Imitation Learning\n\nWeaknesses:\n\n-- No comparison to prior work: The authors acknowledged imitation learning-based (IL) methods but did any quantitative comparison for them. I don't see any evidence why solving the proposed problem using MARL is better than IL. So why are we doing MARL over IL?\n\n- Usefulness: for the traffic light use case, the handling of red / green lights is essentially learned by NOT driving into other cars, which obey the traffic rules. With other words, if there are no other agents to demonstrate how to behave, the agent will always prefer to run over red. This raises the question of the usefulness of the system.\n\n-- Lack of novelty in the method itself: The paper seems to be using an off-the-shelf PPO. The centralized critic, single-step PPO, and bi-level PPO do not provide sufficient novelty in terms of methods. The centralized critic can be seen as a straightforward extension of PPO for the self-driving application. The bi-level extension is a straightforward way to optimize two objectives at the same time, which is quite common even in the ages of convex optimization (https://en.wikipedia.org/wiki/Biconvex_optimization). I think that there __is__ novelty in the method, but I am not sure if it's enough for this venue.\n\n- Dataset: NuScenes dataset is a perception dataset, and probably does not contain many interesting interactive scenarios. Still, the method does not seem to perform very well in the experiments. E.g., according to Fig 2., many cars are still accelerating despite the red light...\n\nDetails:\n\n- All axes names and titles in all diagrams are way too small. No way people can decipher them if printed on paper.\n\n- No legends in the diagrams. Barely any caption. Please make diagrams self-contained if possible.\n\nConclusion:\n\nThis works provides an interesting new problem statement and explores the area of using MARL for autonomous driving. My main concerns regarding this paper are the lack of comparison to prior works in the experiments, and the work provides any usefulness and improvements over current autonomous systems, which are mostly based on imitation learning.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows how "road rules" (e.g., implicit designation of fast lanes on a highway) naturally emerge in a multi-agent MDP. The paper shows that interesting traffic rules do emerge, and it presents a detailed analysis of the factors that lead to this emergence. The paper is complemented by documented source code, with the aim to encourage the community to further work on the topic.

The reviewers agreed that this is original work, and appreciated its simplicity. Two concerns that were recurrently voiced were that 1) there is no algorithmic innovation and 2) there is no comparison to baseline models, or more generally a better placement in the context of existing literature.

The authors provided a detailed and, to my eyes, convincing response. With respect to the two concerns above, I would go as far as saying that 1) (no algorithmic innovation) is a feature, not a bug. The paper is interesting exactly because it studies emergent phenomena after framing multi-agent driving as a standard RL problem. Concerning 2) (lack of baselines), it seems to me somewhat besides the point: The paper is not claiming state of the art on some benchmark for a new algorithm, but studying how certain implicit rules emerge in a given setup. In this sense, as the authors point out, rather than looking at alternative baselines, it is informative to look at which aspects of the setup contribute to rule emergence, which is what the paper does.

Although I realize that in proposing this I am going beyond the reviewers' ratings, I found this to be an original and exciting paper, that I would strongly like to see accepted at the conference.

Decision: Accept (Poster)