Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper investigate how to design simulation environments so the the agent trained with them can master social rules.
Cons:
1. The paper is well written and easy to read and understand. Thanks!
2. The experiments are solid and well defined. 

My major concern of this paper are:
1. The authors seems to only considered the noise of sensors and the number of agents, and these two factors happened to induce social behaviors like following lanes and stopping at traffic signals. In other words, I am not quite convinced that with all vehicles being automated, sensor noise will cause them to formulate rules that what human drivers follow today. Since human driving interactions are complex, I do not think that sensor noise would be enough to induce them. 
2. I would be happy to see how this configuration of simulation environment compares with reward guided social behaviors. For example, we can design a reward to encourage agents follow right of way. I think this way might be more direct and powerful. 
2. The number of agents seems to be too small and may affect the formulation of road social rules.
3. It seems like the agents did not take traffic signal status as input for the action selection, then how did they formulate the signal control rules. 
4. Figures 2 and 3 needs better explanations for readers to understand. 

Overall, I think this paper needs better formulation of the logics and also a deep investigation of how the simulation variations lead to those behaviors. 
review_title: Solid paper for potential accept

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes to learn traffic rules (e.g. traffic lights, speed limits) via multi-agent RL (MARL) from observations rather than hardcoding the rules into the algorithms. To this end, the authors claims contributions in:

-- Defining a multi-agent driving environment, where each agent has incomplete observation (noisy LiDAR), and is rewarded for reaching a destination quickly without colliding. Experiments show that road rules can be learned.

-- Ablation on the choice of the MDP, and insights that perception noise and spatial density of agents are important to successfully learn the environment.

-- Authors promise to release a suite of 2D driving environments for future MARL research in self-driving.

I must admit that I am mainly a computer vision person, and I only have limited experience with RL or MARL. However, I hope that my assessment below is still better than an educated guess. I apologize in advance if there is any obvious misunderstanding.

Strengths:

-- Novelty in problem statement and at a high level: The problem statement of learning hard traffic rules via observing logs seems new. The application of MARL to the problem is new too AFAIK. And as the authors stated, the majority of multi-agent behavior works have been using imitation learning. (Maybe consider citing [1] line of work as well, which deals with multi-agent imitation learning). The proposed PPO-based method seems a good alternative.

-- Extensive experiments: There are 7 small but specific tasks such as "Traffic lights", "Emergence of lanes", etc., where the authors provided evidences to the claims that perception noise and spatial density of agents are crucial for the method's success.

[1]: Multi-Agent Generative Adversarial Imitation Learning

Weaknesses:

-- No comparison to prior work: The authors acknowledged imitation learning-based (IL) methods but did any quantitative comparison for them. I don't see any evidence why solving the proposed problem using MARL is better than IL. So why are we doing MARL over IL?

- Usefulness: for the traffic light use case, the handling of red / green lights is essentially learned by NOT driving into other cars, which obey the traffic rules. With other words, if there are no other agents to demonstrate how to behave, the agent will always prefer to run over red. This raises the question of the usefulness of the system.

-- Lack of novelty in the method itself: The paper seems to be using an off-the-shelf PPO. The centralized critic, single-step PPO, and bi-level PPO do not provide sufficient novelty in terms of methods. The centralized critic can be seen as a straightforward extension of PPO for the self-driving application. The bi-level extension is a straightforward way to optimize two objectives at the same time, which is quite common even in the ages of convex optimization (https://en.wikipedia.org/wiki/Biconvex_optimization). I think that there __is__ novelty in the method, but I am not sure if it's enough for this venue.

- Dataset: NuScenes dataset is a perception dataset, and probably does not contain many interesting interactive scenarios. Still, the method does not seem to perform very well in the experiments. E.g., according to Fig 2., many cars are still accelerating despite the red light...

Details:

- All axes names and titles in all diagrams are way too small. No way people can decipher them if printed on paper.

- No legends in the diagrams. Barely any caption. Please make diagrams self-contained if possible.

Conclusion:

This works provides an interesting new problem statement and explores the area of using MARL for autonomous driving. My main concerns regarding this paper are the lack of comparison to prior works in the experiments, and the work provides any usefulness and improvements over current autonomous systems, which are mostly based on imitation learning.
review_title: Interesting work, hope rebuttal can shed further clarity

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper proposes a bilevel MARL method that can learn road rules and conventions implicitly without hard coding. It is quite interesting to see a simple and straightforward idea that is effective in this task setting. However, this paper needs to be further polished in terms of its delivery, completeness, and evaluation.

Methodology:
How is the noise introduced to the mimicked LiDAR perception results?
In Spline Model, does the trajectory shape only depends on the initial state without considering the afterward interaction?
The clarity and completeness of writing could be significantly improved.
The equations, terms, and the algorithm in Section 4 need sufficient explanations. E.g., what are vf^{target}_t, H, K_1, K_2, N? What is the complete reward list?

Experiments:
It seems the route is relatively short, and the driving scenarios used for different tasks are distinct. I was wondering whether the generalization of the proposed method and the learned rules could be validated.

No baselines have been evaluated or compared. Which experimental results are or the fixed track model? What are the differences between the two models in the results? What are the reward values during the training phase?

The captions and axis labels are difficult to read.
What do the colors mean in Fig. 2?


====
My rating has been updated after the rebuttal.
review_title: Review from R2

final_score: rating: 7: Good paper, accept
review_content: Summary: The output of the work is an MDP model that is capable of encoding complex traffic rules including traffic signals, lanes, right of way FIFO etc. Various different traffic environments such as intersections, highways, nuScenes are considered.

The MDP that results from this work is very useful for future research. Currently, there are very few simulators that encode complex traffic rules or provide the flexibility to perform research. Therefore, the possibility of new simulators resulting from this MDP is exciting and useful to encourage and promote research in autonomous driving, especially in dense and heterogeneous environments.

I am glad that the authors have provided the source code **with extensive documentation and instructions** on how to run the code and reproduce the results. 

Unless there is some major flaw with this paper (that I may have missed), there is no reason to reject this paper.
review_title: Useful to research in autonomous driving

Meta-review: This paper shows how "road rules" (e.g., implicit designation of fast lanes on a highway) naturally emerge in a multi-agent MDP. The paper shows that interesting traffic rules do emerge, and it presents a detailed analysis of the factors that lead to this emergence. The paper is complemented by documented source code, with the aim to encourage the community to further work on the topic.

The reviewers agreed that this is original work, and appreciated its simplicity. Two concerns that were recurrently voiced were that 1) there is no algorithmic innovation and 2) there is no comparison to baseline models, or more generally a better placement in the context of existing literature.

The authors provided a detailed and, to my eyes, convincing response. With respect to the two concerns above, I would go as far as saying that 1) (no algorithmic innovation) is a feature, not a bug. The paper is interesting exactly because it studies emergent phenomena after framing multi-agent driving as a standard RL problem. Concerning 2) (lack of baselines), it seems to me somewhat besides the point: The paper is not claiming state of the art on some benchmark for a new algorithm, but studying how certain implicit rules emerge in a given setup. In this sense, as the authors point out, rather than looking at alternative baselines, it is informative to look at which aspects of the setup contribute to rule emergence, which is what the paper does.

Although I realize that in proposing this I am going beyond the reviewers' ratings, I found this to be an original and exciting paper, that I would strongly like to see accepted at the conference.

Decision: Accept (Poster)