Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposed a method that combines Traffic Coordinator Network (TraCo), Counterfactual Advantage Function (CAF), and attention mechanism.
The proposed method is based on multi-agent reinforcement learning and aims to solve problems related to automated driving in multi-vehicle environments.
On busy roads, vehicles need to operate efficiently, which is also related to safety.
The experiment was compared to several baseline algorithms.
A simulator environment called METADRIVE was used.
Results showed the effectiveness of the proposed method.
review: This paper seems to be about how to control a large number of vehicles on a busy road.
It is interesting to note that the methodology approach in this study is based on multimodal reinforcement learning, where the overall reward can be decomposed as the reward of individual vehicles.
However, the environmental settings used in the experiments are simple (such as a nondescript crossroads or a bottlenecked power line road) and may not be sufficiently validated to be effective in a wide variety of traffic situations.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: There are some concerns with the lack of clarity in the claims made in the introduction of this paper and in the experiments.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper introduces the Traffic Coordinator Network (TraCo), a novel method for addressing Multi-agent Reinforcement Learning (MARL) limitations and the context of autonomous driving. Traditional MARL systems fail to accurately evaluate an agent's contribution to the team, which may change over time. The TraCo approach in this study resolves this issue by focusing on team needs and interests. TraCo uses a cross-attention mechanism and a Counterfactual Advantage Function (CAF) to understand distinctive characteristics of domain agents and accurately quantify an agent's contribution. This approach facilitates improved decision-making in traffic flow simulations. While there are still gaps in replicating real-world vehicle behaviors due to reinforcement learning's random exploration, this approach significantly outperforms existing models in various metrics.
review: This paper is a prime example of high-quality academic writing. All figures are informative and plotted clearly. Reading this paper is an intellectual pleasure. I believe this is a high-quality paper in clarity, originality, and significance. And I don’t have many suggestions.

Strengths:
1. The logical flow of this paper is well-executed, facilitating a comprehensive understanding of the presented ideas.

Suggestions:
1. The authors could mention some details in the TraCo model training process, including the training environment setting.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper presents the Traffic Coordinator Network, a pioneering solution to address the shortcomings of Multi-agent Reinforcement Learning in autonomous driving. This is a high-quality paper, and I recommend a strong acceptance.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors consider the problem of managing traffic flow and propose an algorithm with a virtual agent that serves as a traffic coordinator, sending orders to other individual agents. The authors introduce the Counterfactual Advantage Function (CAF) to motivate agents to work in line with the team's interests. Additionally, they utilize cross-attention to effectively handle dynamic numbers of observations and agents. In their experiments, they conduct experiments using RLLib and demonstrate that their algorithm outperforms existing approaches in four different traffic flow simulations
review: Originality) The paper introduces a novel approach to modeling traffic flow by utilizing the Counterfactual Advantage Function and cross-attention mechanism. This approach surpasses existing solutions and promotes social behavior, effectively reducing collision rates.

Quality) The submission is technically convincing. The experiment results are well-described and persuasive.

Clarity ) The paper does not provide enough information about problem formulation, action and observation space as well as algorithm and hyperparameters, however authors state that there will be more details about it the in appendix. I found a few typos, which will be listed in the questions section

Significance) The paper and its results are important and can be used by other researchers to further improve traffic flow simulation. The paper improves the safety and success rate of vehicle agents.

Relevance) I think it is a relevant paper because it outperforms existing solutions in traffic flow simulation. This paper brings us closer to a future where such experiments can be conducted in real-world scenarios.

Limitations) I think limitations are addressed well

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: It is an important paper that demonstrates the successful approach to modeling traffic flow and showcases the emergence of social behavior among vehicles. It is written clearly but certain sections that are crucial for understanding are missing. Additionally, I noticed a few typos and proposed to write more information about problem setting and experiments. After all, I find this paper significant and relevant to the conference

After author response: I appreciate the author's rebbutal. However I miss the final draft to change my score.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes TraCo, which utilizes a combination of cross-attention and counterfactual advantage function to learn a virtual traffic coordinator for cooperation with multi-agent reinforcement learning. The goal is to accurately quantify the contribution that a team needs from an agent in the field of autonomous driving and to enable the emergence of rich and diverse social behaviors among vehicles within the traffic flow. There are many details that need to be explained in detail by the author.

review: There are many details that need to be explained in detail by the author.

1. What are the functional differences between the coordinator and the traffic light? Why not train the signal light directly？

2. Overall, the method description is quite confusing. Please provide a detailed description of the method and explain why it can solve the problem of "cannot express the contribution that the team needs from the agent.

3. The method is similar to HRL, where the top layer provides goal z, while the bottom layer makes decisions based on z from individual rewards and group rewards. The author needs to introduce the top-level part in detail, i.e., the blue section of TraCo in Figure 2. The author provided a relatively detailed definition of Ai, but from the structure in Figure 2, it is not the main part of TraCo.

4. As it is an application-oriented problem, the author also needs to introduce the model definition of the environment in traffic scenarios, including observation, actions, and two rewards. 

5. Are the two actors and two Qtots in Figure 2 the same? How to estimate Q(s, a(z-))? Where does its data come from? How to obtain r and s' corresponding to a(z-)?

6. The definitions of Qtot and Qnet in the article need to be introduced. Different symbols are used for the same definition before and after the article.

7. How to set the hyperparameter alpha in formula 6

8. Why is Figure 4 only showing three scenarios without another one?

9. Need ablation experiment to demonstrate the effectiveness of the proposed counterfactual strategy in training the Q(s,a(z))-Q(s,a)
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: There are many details that need to be explained in detail, thus leaning to a rejection.
review_title: null

Meta-review: This paper presentes a multi-agent RL method for autonomous driving in busy enviornment, featuring a traffic coordinator network. 

The reviewers have agreed that the paper presents in interesting approach to a timely issue in the context of autonous driving. Most reviewers have found the approach being technically sound and well-described. However, there is a concern on the reality of the test scenarios in the paper, in the sense that the simulated environment does not capture the chain reaction of accidents in real driving conditions; this concern has not been completely resolved in the rebuttal phase. 

While the AC thinks that the proposed scheme has limitation to be directly applied to real situations, he/she also thinks that the presented idea and the verification on a (limited) virtual environment would also be useful for the audience of CoRL. The authors need to incorporate the comments provided in their rebuttal in a final version.
Decision: Accept (Poster)