TraCo: Learning Virtual Traffic Coordinator for Cooperation with Multi-Agent Reinforcement Learning
Authors: Weiwei Liu
Wei Jing
lingping Gao
Ke Guo
Gang Xu
Yong Liu
Venue: CoRL 2023 Poster (2023)
Keywords: autonomous driving
multi-agent reinforcement learning
counterfactual reasoning

Abstract: Multi-agent reinforcement learning (MARL) has emerged as a popular technique in diverse domains due to its ability to automate system controller design and facilitate continuous intelligence learning. For instance, traffic flow is often trained with MARL to enable intelligent simulations for autonomous driving. However, The existing MARL algorithm only characterizes the relative degree of each agent's contribution to the team, and cannot express the contribution that the team needs from the agent. Especially in the field of autonomous driving, the team changes over time, and the agent needs to act directly according to the needs of the team. To address these limitations, we propose an innovative method inspired by realistic traffic coordinators called the Traffic Coordinator Network (TraCo). Our approach leverages a combination of cross-attention and counterfactual advantage function, allowing us to extract distinctive characteristics of domain agents and accurately quantify the contribution that a team needs from an agent. Through experiments conducted on four traffic tasks, we demonstrate that our method outperforms existing approaches, yielding superior performance. Furthermore, our approach enables the emergence of rich and diverse social behaviors among vehicles within the traffic flow.

TraCo: Learning Virtual Traffic Coordinator for Cooperation with Multi-Agent Reinforcement Learning
Weiwei Liu
Wei Jing
Huzhou Institute of Zhejiang University Zhejiang University China 1 ww33032 ju. www.ju..eee..ooom...oooooo...... edu.cn cn
Netease Fuxi Robotics China 21 wlwjf melingggmal com
Lingping Gao
Ke Guo
Autonomous Driving Lab Alibaba DAMO Academy China glp p. dl @ma com
Alibaba DAMO Academy University of Hong Kong China kguoêcs.. hhu.
Gang Xu
Yong Liu
College of Control Science and Engineering Zhejiang University China wuuyaθzju. edu.cn on
College of Control Science and Engineering Zhejiang University China yongliueiipe. jju. www.ed..ccmmm en
Abstract: Multi-agent reinforcement learning (MARL) has emerged as popular technique in diverse domains due to its ability to automate system controller de- sign and facilitate continuous intelligence learning. For instance, traffic flow of- ten trained with MARL to enable intelligent simulations autonomous driving. However, The existing MARL algorithm only characterizes the relative degree of each agent's contribution the team, and cannot express the contribution that the team needs from the agent. Especially in the field of autonomous driving, the team changes over time, and the agent needs to act directly according to the needs of the team. To address these limitations, we propose an innovative method inspired by realistic traffic coordinators called the Traffic Coordinator Network (TraCo). Our approach leverages a combination of cross-attention and counterfactual advantage function, allowing
extract distinctive characteristics of domain agents and accurately quantify the contribution that team needs from agent. Through experiments conducted on four traffic tasks, demonstrate that our method out- performs existing approaches, yielding superior performance. Furthermore, our approach enables the emergence of rich and diverse social behaviors among vehi- cles within the traffic flow.
Keywords: autonomous driving, multi-agent reinforcement learning, counterfac- tual reasoning
Introduction
There are numerous Multi-Agent Systems (MAS) [1] present nature and human society. Self- Driven Particles (SDP) [2] have been proposed describe these systems. In SDP, each agent interacts with its surrounding agents, considers the interests of others while pursuing its goals, and ultimately exhibits complex collective behavior. For instance, traffic flow used in au- tonomous driving is often considered a typical example of SDP [3]. While early SDP mod- els were relatively simple, some have been based on philosophical ideas, such as the Belief-
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Desire-Intention (BDI) model [4, 5], which improves the reasoning and decision decision-making abil- ities of agents. However, traditional control methods can struggle to describe actual group behavior due to the complexity of reasoning required for artificially designed rules [6. 7]-
Multi-agent reinforcement learning (MARL) [8, 9, 10] has emerged promising algorithm for learning con- trollers to simulate SDP behaviors. the SDP of autonomous driving traffic flow, reward decomposition is partic- ularly crucial. Each agent must strike delicate balance between its own interests and the team's, which serves as the foundation of appropriate social behavior. However, achieving this balance challenging problem [11], especially complex scenarios such as navigating busy intersections. Even experienced drivers may require the assistance of traffic coordinator to safely navigate such environments, vehicle interaction is constantly changing, making it difficult to measure each vehicle's contribution from its perspective. Thus, agents must take practical actions that prioritize the team's 's interests. Though previous approaches [12, 13] have attempted evaluate an agent's contribution to the team, they often fail to address the reward balancing problem and perform poorly traffic flow simulation applications.
Traffic Coordinator
Ego Car
Order
In-domain Observation
Gradient
Policy
Gradient
Figure 1: Example of traffic flow managed by the traffic coordinator: the vehicle operates based on its driving ca- pabilities and follows the commands issued by the traffic coordinator.
We propose a novel benefit trade-off scheme inspired by the task of traffic coordination. Unlike existing schemes, we start from the needs of the team's interests and make the agent act directly according to the team's interests. As shown Figure I, the cars the intersection need follow the traffic coordinator's orders and their driving capabilities. To implement this scheme, we introduce the Traffic Coordinator (TraCo) network. TraCo learns the local interactions within dynamic number of surrounding agents, through the Cross-Attenticn mechanism [14]. Therefore, TraCo is able to issue traffic order embedding based on the importance of surrounding agents relative to the feature correlation of ego agents, without altering the network structure. The agents then act according to the received orders and their states. In addition, We design the Counterfactual Advantage Function (CAF) to measure the impact of the team's orders on the team's interests, and the agent's advantage function to measure its ability to act.
The main contributions summarized as follows:
We propose the TraCo network to capture the interactions better and evaluate the reward decomposition of agents from the perspective of team benefits. TraCo uses a virtual traffic coordinator with a cross-attention mechanism to capture features and interactions, and issue TraCo commands within the traffic scene, thereby assisting agents in generating strategies to act according to the interests of the team.
We incorporate CAF TraCo to measure the impact of TraCo's commands to the agent on the team's interests. This allows agents directly in line with the team's interests, while also promoting TraCo's ability move beyond the limitations of feature extraction network, and thus significantly improve the performance of the simulated traffic flow.
We conduct extensive experiments. The results show that TraCo performs superior in mul- tiple metrics, and the agents exhibit diverse social behaviors.

---

2 Related Work
2.1 MARL and Value Decomposition
In multi-agent system, the agents share the same reward function. As result, the rewards they obtain may not accurately reflect their behaviors, leading to inaccurate policy updates. This is known as the credit assignment problem [11] in MARL.
solution. VDN [15] uses a simple summation calculate the joint action-value function for decomposing the team's reward signal to each agent. Compared with [QL[16], this centralized training method can guarantee the optimality of the joint action value function to certain extent. Nevertheless, the simple summation significantly limits the fitting ability of the joint action-value function. QMIX [13] improves the fitting process of simple summation in VDN to non-linear fitting subject to monotonic constraints. QTRAN [17] introduces an intermediate action-value function that approximates the real action-value function and then decomposes the intermediate action-valle value functtin,, which avoids the monotonic constraint of QMIX. In addition, given the limitation of the QMIX action-value function fitting ability with monotonic constraints, the agent cannot explore the entire joint action space. MAVEN [18] shares hidden variable in the value function of each agent, and uses the mutual information obtained by maximizing the trajectory information of the agent and the hidden variable information to increase the divergence of the policy and make the actions more diverse.
In discrete action spaces, value decomposition
For continuous action state spaces, existing work often learns the joint state-value function directly. One such algorithm that does this is MADDPG [19], which extends the DDPG [20] algorithm to the multi-agent system. MADDPG adopts a scheme of centralized training and decentralized ex- ecution, where the joint value function is computed and then used evaluate the policy of each agent. Another algorithm that follows a similar path is MAPPO [21], which extends PPO [22] to the multi-agent domain. Interestingly, MAPPO requires only a minimal hyperparameter search to achieve comparable performance to state-of-the-arr algorithms. Both MADDPG and MAPPO im- plicitly solve the multi-agent credit assignment problem. In contrast, COMA [23] tackles the credit assignment problem head-on by using a counterfactual baseline to evaluate the contribution of each agent the team. In addition, simple independent learning [24] that solely pursues self-interest may make traffic vehicles aggressive and display irrationally selfish behavior. Unlike existing algorithms that measure their contribution to the team starting from the agent, distributing the team's 's needs directly to the agents could be an interesting solution.
2.2 Autonomous Traffic Flow
Collecting data solely from the real world for autonomous driving is impractical and expensive. Therefore, traffic flow simulations have emerged as a popular alternative for modeling vehicle inter- actions. Traditional approaches [25] employed predetermined rules to control traffic flow. Recent research has introduced RL as a way to control vehicles traffic flow. For example, CityFlow [26] employs RL algorithms study traffic flow city-wide scale. RL has also been used to train individual vehicles [27] in controlled environments and examine vehicle-to-vehice social interac- tions [28]. Additionally, SMARTS [29] studies the interaction capabilities between agents in differ- ent environments. Nevertheless, the study of interactions between agents diverse environments remains an area of interest. In this context, our research investigates four common autonomous driv- ing scenarios, with a focus on applying the traffic coordinator network in continuous action spaces. Our approach starts with teams, which directly ask agents in favor of the team, explicitly ad- dressing the trade-off between team and self-interest, which allows model the traffic flow more efficiently than previous work.
3

---

8; {ot ob....,ppp 1p0
TraCo
Cross-
%
PC
+
FC
Traffic
Atten
Courdinator
*
Actor
a: (z:)
+
Actor
Actor
as
at E R.
Iteration
Biy a
Sig
Native Render
MSE Las
tot
Q
Q a(z)) Q(si, a(2;))
minus
Critic,
Criticue
(P())
The architecture of the Traffic Coordinator network. The left image showcases the simulated environment, with the third-person perspective visible in the bottom left corner. The ob- servations of agents around agent are denoted as {ox. ,00,p while o, represents the observation of agent i. We incorporate two evaluation networks (Critic, and Criticto:) to individual and joint state value functions, respectively. Equations (10) and (11) illustrate how the joint action-valie value function can be computed the joint state-value function, followed by using MSE loss to train the action-value network.
Figure 2:
3 Methods
Autonomous vehicles must be vigilant avoid collisions while reaching their destination. In real- ity, drivers drive carefully under the guidance of the traffic coordinator, which provides clear driving directions based on the surrounding situation, reducing the driving difficulty. This section provides detailed overview of the TraCo network, which enhances the social behavior of traffic flow. To en- sure that compliance with TraCo's orders improves the team's 's interest, we utilize the counterfactual advantage function. The TraCo network differs from traditional centralized algorithms by reduc- ing the information processing difficulties on each agent and providing clear centralized vehicle guidance.
3.1 Traffic Coordinator
As shown in Figure 2, We introduce virtual agent, the traffic coordinator, to model the traffic flow. The traffic coordinator has in-domain observations and distributes order vectors = E IRd (=* is the order issued by the traffic coordinator to agent i, and d2 is the order dimension) to each agent. The orders issued by the traffic coordinator network can be represented by z = z 2 E N, where N the number of agents. The function f generates the order z and is parameterized by 4,, with zi ~ [p (o,, S;), where o; the observation of agent and s; is obtained by aggregating agent information in the domain, as illustrated in Figure 2. Upon receiving the traffic coordinator order z; each agent i takes actions according to its observation o,, In an episode, the traffic coordinator observes the state in the domain, calculates, and distributes orders z, to the agent. At each moment, any agent will act based on its individual observation and order 2;, Ps = x (0;, 24), ag ~ P:- The objective function formulated using PPO as follows:
To (a | 0i, zi) A,, cilp( To(a; 0i z;) Tg (a, o o,,22)
((0) Σ. min(
+
(I)
Tg' (a; o 24)
Since the information closer the vehicle may be more important, as shown in Figure 2, In order to capture the interactions between agents, utilize Cross-Attention network [14] in the traffic coordinator network to extract relevant information.

---

3.2 Counterfactual Advantage Function
Updating the policy solely based on Equation (1) results in the traffic coordinator network issuing orders only to extract in in-domain features, with the agent action network making more comprehensive decisions based on these features. The advantage function remains unchanged as follows:
A(o,, F4, 4.) Q (o,, 2,, 0.) V (o); 24) =E,, 1pp_111pppu111s0s. ( -V
(2)
($ >V
V (s
However, this approach fails address the critical interests balance problem. Since traditional TD emrors only consider agent rewards, the contribution of each agent the team is still not measured. Therefore, inspired by the difference rewards[30], we design counterfactual rewards to measure the impact whether each agent acts according to the team's 's order on the team's 's interests.
T (ioo (z()), o,-, a.-) (oo (2(2; ), a.-),
Here Oi- and a,- represent the joint observations and joint actions of agents other than agent a,(=,)) represents the action of the agent only based on its own observation when there is no traffic
coordinator order. Ttor represents the team reward in the domain, Ttot rx ... + p {k . E domaing). At this point, the reward for agent is:
(4)
At this point, agent i's reward comes from its reward, and the team's reward. According to Equation (3), obtaining r! needs to perform actions with or without traffic coordinator orders so that the environment gives different reward signals, which requires modeling the environment. Obviously, this process overly complicated. Researchers [31] suggest using function approximations instead of simulators to estimate differential rewards.
Similar to COMA [23], design centralized critic, which is used to estimate the joint actions value function Qiot of all agents in the domain. Then, for each agent i, with this value function Qtor, we can compute Counterfactual Advantage Function (CAF), keeping the actions a,- of
other agents fixed in the process:
AP a) Qtor a(=:)) - Qior a(z,))
(5)
In Equation (5), centralized critic is employed reason about counterfactuals. Specifically, the critic considers the scenario where only the actions of agent change and computes the counter- factual advantage function (s, which reflects the contribution of agent to the team. This approach enables learning directly from the agent's experience without relying on an additional environment model.
Based this, the advantage function of agent at this time is:
A A + a A!
(6)
A Q (0,, a.) - V, (o,)
Eir ,vV V171
where represents the coefficient of the counterfactual advantage function. A is the advantage function obtained by agent based on its reward function. Please see the appendix for additional methods and pseudocode.
Experiments
4.1 Baseline Algorithms and Experiment Setup
Experimental baseline algorithms include IPPO [24], MEPO [32], and CoPO [3]. IPPO uses PPO as an independent learner; MFPO encodes the state of surrounding agents as a mean state, which
5

---

Bottlenck
Tollgate
0.40
040
C2PD
0.35
0.4
Success Rate
0.50
0.25
0.3
0.20
0.2
0.30
0.1
0.00
0.0
0.4
1.0
Timesteps
Timesteps
Parking Lot
Intersection
0.25
IPFO
0.7
METNO
Success itate
0.20
0.5
0.35
0.4
0.30
0.2
CuPD
0.05
0.1
0.0
0.25
0.75 Timesteps
1.25
1.75
0.6
1.0
Timesteps
les
Figure 3: Performance comparison of TraCo and other baseline algorithms in four task scenarios. Note that the time interval step of the features extracted by the traffic coordinator network is 1.
is used as auxiliary input the value function. CoPO is to split the source of the agent into its reward and the average reward of the surrounding agents, and the global reward controls the proportion of the two.
We ran experiments using RLlib [33] with the aforementioned environments and algorithms on Nvidia GeForce RTX 2080Ti GPUs. Each trial was trained on over million environment steps, equivalent to approximately 55 hours in real-time traffic system over 2,000 hours of individual driving experience, assuming average of 40 vehicles running at the same time. For the introduc- tion of the experimental scenarios and tasks, please refer to the appendix.
4.2 Results
In Figure 3, we compare the success rates of our TraCo algorithm to those of the baseline algorithms across all tasks. As a result of the virtual traffic coordinator's order, our TraCo algorithm outper- forms the baseline in three tasks and performs comparably to the baseline only in the Intersection task. Notably, in the Tollgate task, which involves the most agents, TraCo outperforms the strong baseline CoPO by a significant margin. As shown Figure 5, this task requires agents to exhibit active cooperation behaviors, such as queuing and giving way, and strong interaction abilities among agents. Populations generated by other algorithms fail to exhibit such behavior, leading to conges- tion. Interestingly, IPPO performs comparably to MFPO and even outperforms CoPO in the Parking Lot task, despite MFPO and CoPO having more intra-domain information. This is because the value estimated by IPPO's 's critic network includes noise perturbations that improve the algorithm's 's explo- ration performance. Additionally, the Parking Lot task is continually changing due to community factors. Simply averaging or concatenating neighbors' states as an additional input to the value function makes training unstable, a phenomenon also observed MARL from StarCraft [24, 34].
https://gituubb cow//ntaadrivvees//tttaaaii httpra/rieeeeeeiidddebeeeenrrarn MbuncinaaaattetemaaaAAAAALLL As described repo link, CoPO measures performance using the maximum value of each set of experimental data with different ran- dom seed. However, for more comprehensive comparison, we use the average performance across 8 random seeds in this study.

---

Bottleneck
Tollgate
Safety
0.4
0.3
cano
0.0
uccess
D
01
Efficiency
Efficiency
Parking Lot
Intersection
Safety
Safety
18 g2
0.6 0.a
0.08
Don
aecess Rate
Trudk
Efficiency
Efficiency
Figure 4: Performance comparison among TraCo and baseline algorithms with three metries.
Cle
roking
Orirgin
Visualization of the social behavior of the population with TraCo. The social behavior of each agent is denoted with black dots, while the subsequent trajectory of each vehicle is indicated by a decreasing intensity color, with brighter colors representing more recent steps.
Figure 5:
Figure 4 depicts the lidar chart with three metrics after normalization. Despite the comparable success rate, TraCo outperforms baseline algorithms on safety and efficiency metrics. To render the vehicle running track, MateDrive employs PyGame. Figure illustrates that TraCo generates populations that exhibit social behaviors, such reversing, cutting in line, queuing, waiting, and following, in all four tasks complete their goals. This demonstrates that the vehicle selects differ- ent driving styles based on the situation and simulates range of interactive behaviors in the traffic system.
4.3 Generalization
To evaluate the generalization ability, we vary the initial number of agents the test phase to deter- mine their converged policies. Figure illustrates that as the number agents increases, the popu- lation success rate decreases due road congestion and a higher likelihood of collisions. However, observe that having too few agents did not improve the algorithm's performance in the Intersec- tion task. We suspect that in the multi-agent algorithm, each agent's policy may overfit the behavior
7

---

Bonleweck
GAO
CuFD
MITO
HIPO
IPP
03-
Success Rate
0.4
0.3
0.2
0.2
0.0
12
15
21
24
10
15
20
25
30
35
40
45
50
Initial Number of Agents
Initial Number of Agents
Panking
Intersection
0.6
Success Rate
0.3
0.00
0.2
I7PO
TraCo
0.55
TiaCo
Initial Number Agents
Number Agents
Figure 6: Success rate for different initial numbers of vehicles test time. The gray vertical line represents the initial number of vehicles during training. from which the algorithm policy is trained.
of other agents, resulting in failure. This overfitting may occur because reduced number of agents leads to fewer encountered situations, which can limit the model's ability to generalize. Additionally, we find that inputting the reward distribution coefficient, which is learned during CoPO training, as prior knowledge into the agent observation may interfere with the generalization ability of the algo- rithm once the number of agents changes during the test phase. Notably, TraCo outperforms baseline algorithms, even when the number of agents in the population changes. This is because TraCo uses cross-attention network to process the dynamic number of agent information in the domain, allow- ing its model to adapt to the community environment of the dynamic number of agents. Finally, we have designed ablation experiments, which are detailed in the Appendix.
Limitation
TraCo has demonstrated its ability in traffic flow simulation by facilitating team instructions, while agents exhibit complex social behaviors. However, there are still gaps in its ability to replicate real-world vehicle behaviors, which can be attributed the random exploration of reinforcement learning. To address this shortcoming, plan integrate real data into our approach in the future, in order to constrain vehicle behavior and improve overall performance. In addition, TraCo still lacks the ability to handle traffic lights, which may limit its application to certain urban driving scenarios.
6 Conclusions
We present a novel approach to model traffic flow using the Traffic Coordinator Network (TraCo) with the Counterfactual Advantage Function (CAF) and an attention mechanism. TraCo models real traffic coordination to enhance vehicle decision-making unlike traditional feature extraction networks. Our experiments demonstrate that TraCo-trained vehicles exhibit lower collision rates and higher success rates than baseline models while demonstrating diverse range of social behaviors.
7 Acknowledgment
This work is supported by NSFC 62088100 101 Autonomous Intelligent Unmanned Systems.

---

References
[1] A. Dorri, S. S. Kanhere, and R. Jurdak. Multi-agent systems: A survey. Ieee Access, 6: 28573-28593, 2018.
[2] T. Vicsek, A. Czirók, E. Ben-Jacob, I. Cohen, and O. Shochet. Novel type of phase transition in system of self-driven particles. Physical review letters, 75(6):1226, 1995.
[3] Z. Peng, Q. Li, K. M. Hui, C. Liu, and B. Zhou. Learning to simulate self-driven particles system with coordinated policy optimization. Advances in Neural Information Processing Systems, 34:10784-100 10797, 2021.
[4] M. Georgeff, B. Pell, M. Pollack, M. Tambe, and M. Wooldridge. The belief-desire-inten model of agency. In Intelligent Agents V: Agents Theories, Architectures, and Languages: 5th International Workshop, ATAL '98 Paris, France, July 4-7, 1998 Proceedings pages 1-10. Springer, 1999.
[5] G. Simari and S. D. Parsons. Markov Decision Processes and the Belief-Desire-Intie Model: Bridging the Gap for Autonomous Agents. Springer Science & Business Media, 2011,
[6] A. Czirók and T. Viesek. Collective behavior of interacting self-propelled particles. Physica
A: Statistical Mechanics and its Applications, 281(1-4):17-29, 2000.
[7] E. Bertin, M. Droz, and G. Grégoire. Hydrodynamic equations for self-propelled particles: microscopic derivation and stability analysis. Journal of Physics A: Mathematical and Theo- retical, 42(44))445001, 2009.
[8] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782)3300-344 2019.
[9] S. Bhalla, S. Ganapathi Subramanian, and M. Crowley. Deep multi agent reinforcement learn- ing for autonomous driving. In Canadian Conference Artificial Intelligence, pages 67-78. Springer, 2020.
[10] A. Malus, D) Kozjek, et al. Real-time order dispatching for a fleet of autonomous mobile robots using multi-agent reinforcement learning. CIRP annals, 69(1):397-400, 2020.
[11] M. Zhou, 7. Liu, P. Sui, Y. Li, and Y. Y. Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. Advances in neural information processing systems, 13:11853-1181 1864, 2020.
[12] T. Rashid, G. Farquhar, B. Peng, and S. Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 33:10199-10210, 2020.
[13] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. Mono- tonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research,
2020.
[14] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu. Cenet: Criss-cross attention for semantic segmentation. Proceedings of the IEEE/CVF international conference on computer vision, pages 603-612, 2019.
[15] Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, al. Value-decomposition networks for cooperative multi- agent learning, arXiv preprint arXiv: 00777700006666 2017.
[16] A. Tampuu, T. Matiisnn D. Kodelja, Kuzovkin, K. Korjus, Aru, J. Aru, and R. Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4): c0172395, 2017.

---

[17] K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pages 5887-5896. PMLR, 2019.
[18) A. Mahajan, T. Rashid, M. Samvelyan, and S. Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
[19] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor- critic for mixed cooperative-compeiityy environments. Advances in neural information pro- cessing systems, 30, 2017.
[20] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:/509.0000 2015.
[21] C. Yu, Velu, Vinitsky, Gao, Wang, A. Bayen, and Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
[22] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: hatd:::777.63444 2017.
[23] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[24] C. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. Torr, M. Sun, and S. Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.0953: 2020.
[25] S. E. Shladover. Review of the state of development of advanced vehicle control systems (aves), Vehicle System Dynamics, 24(6-7):551 (11--7):551-955 1-595, 1995.
[26] H. Zhang, S. Feng, Liu, Y. Ding, Y. Zhu, Z. Zhou, W. Zhang, Y. Yu, H. Jin, and Z. Li. Cityflow: A multi-agent reinforcement learning environment for large scale city traffic sce- nario. In The world wide web conference, pages 3620-3624, 2019.
[27] A. Kendall, Hawke, D. Janz, Mazur, D. Reda, J.-M. Allen, V.-D. Lam, A. Bewley, and A. Shah. Learning to drive in day. In 2019 International Conference Robotics and Automation (ICRA), pages 2254.. IEEE, 2019.
[28] S. Shalev-Shwartz, Shammah, and Shashua. Safe, multi-agent, reinforcement learning 2016.
for autonomous driving. arXiv preprint
[29] M. Zhou, J. Luo, J. Villella, Y. Yang, D. Rusu, J. Miao, W. Zhang, M. Alban, I. Fadakar, Z. Chen, et al. Smarts: Scalable multi-agent reinforcement learning training school for au- tonomous driving. arXiv preprint arXiv:2010.09776, 2020.
[30] D. H. Wolpert and K. Tumer. Optimal payoff functions for members of collectives. In Modeling complexity in economic and social systems, pages 355-369. World Scientific, 2002.
[31] M. K. Colby, W. J. Curran, and K. Tumer. Approximating difference evaluations with local information. In AAMAS, pages 1659-1660, 2015.
[32] Y. Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and Wang. Mean field multi-agent reinforce- ment learning. In International conference machine learning, pages 5571-5580. PMLR, 2018.
[33] E. Liang, Liaw, R. Nishihara, Moritz, R. Fox, K. Goldberg, Gonzalez, M. Jordan, and I. Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference Machine Learning, pages 3053-3062. PMLR, 2018.
10

---

[34] J. Hu, S. Hu, and S.-w. Liao. Policy regularization via noisy advantage values for cooperative multi-agent actor-critic methods. arXiv preprint arXiv:2106, 1402240061334,,, 2021.
[35] R. Varma. Picking loss functions-a comparison between mse, cross entropy, and hinge loss, 2018.
[36] Q. Z. Peng, I.. Feng, Q. Zhang, 7. Xue, and Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
8 Appendix
8.1 Method Supplement
In order to calculate the counterfactual advantage function AP, the joint action value function Qtor must be estimated, although only the state value function Vior estimated with PPO. Thus, we have:
tot (st) = E 10000000000000
hot 1+1
(8)
and:
D0
Qia (st, a.) E Ee44 :0:44100
From equations (8, 9), get:
Qiol (St, 0i) = lt not 22004
tot rrr4
(10)
1=
rrto y tot V (8++1)
In this way, can obtain the oction-value function Qtor from the state-value function Vzat in the PPO architecture. However, since state-value functions cannot evaluate agent actions, we design an action-value function network that takes input approximating Equation (10). To train the network, we utilize Mean Square Error (MSE) loss [35]:
MSE
(y: -1)2²
1=1
(11)
(Quat rt 1ot - Viai (1141))
where 6 denotes the batch size. Algorithm shows the overall process of TraCO in the appendix.
At this time, the partial differential of the counterfactual advantage function for agent i to take an action under instruction z; is:
0 14 Q(s;, a(2;))
0z, Q(s,, a(27))
0z, dz,
(12)
Q(s, a(z,))
oz,
The above equation reveals that the agent i's utility with counterfactual instructions aligns with the global learning objective, and maximizing the counterfactual reward can enhance the joint action- value function. Consequently, the agent's advantage function can decomposed into its individual and team contributions, shown in Equation (6), which completes the value decomposition opera- tion.

---

Algorithm TraCo for agent i
1: Input: Randomly initialize TraCo, actor and critic network and V with weights yp, 0, and
0g
2: for episode=1, T do
3: Get agents' observations for,.... Jn
4:
Get 8; = {oR,-.................... ....................... Op} according to the distance
5:
Compute = for 8t), ag π [0;;
6:
Compute counterfactual advantage function Al according to equations (5, 10, 11)
7:
Compute A; according to equation (6)
8: Update with PPO rules
9: end for
8.2 Experiment Platform and Scenarios
We MetaDrive [36] simulator, which is capable for generating infinite scenarios with var- ious road maps and traffic settings to enable generalizable RL. In setup, use current state, navigation info, and surrounding data encoded in vector of 72 lidar-like measurements as agent observations, while the policy output is the acceleration and steering of the vehicle. As the mutual influence between vehicles decreases with distance, we define the in-domain state for the traffic coordinator as the information splicing of different vehicles within 40-meter radius of the ego- vehicle.
As shown in Figure 5, we benchmark our method in four common autonomous driving tasks, which are described in detail as follows:
Bottleneck:
The Bottleneck is to set up a narrow bottleneck lane between the eight lanes, forcing vehicles to give way and queue up to pass. The environment is initialized with 20 cars.
The Tollgate environment models the real-world behavior of vehicles passing through tollgate, where agents are required to wait for permission signal for 3 seconds before continuing. Failure to comply with this rule results in failed episode. The environment is initialized with 40 cars.
Tollgate:
Parking lot: The parking lot scenario in simulation consists of parking spaces. Spawn points for vehicles are scattered both within and outside the parking lot, leading simultaneous entry and exit of vehicles and thereby increasing the level of difficulty. The environment is initialized with 10 cars.
Intersection: an unprotected intersection scenario, vehicles are required to negotiate and judge the potential intentions of other parties in order complete the task. The environment initialized with 30 cars.
In this paper, we use three indicators to evaluate the performance of multi-agent algorithms. suc- cess rate is the ratio of vehicles successfully reaching the destination, safety is the vehicle non- collision rate, efficiency >= indicates the difference between successes and failures in a unit of time (N &Nsucesss Naaiture)/TT Vehicles may travel low speeds for the safety of driving, but this conducive the effective passage of vehicles.
8.3 Ablation Studies
In our previous experiments, we employed the traffic coordinator network solely as feature ex- traction network, without considering the counterfactual advantage function. Therefore, it is crucial to verify the validity of this function. As illustrated in Figure TraCo w/o CAF performs worse than TraCo w/ CAF in all four autonomous driving tasks. This is because the traffic coordinator network, when equipped with counterfactual advantage function, not only extracts in-domain fea- tures but also evaluates the agent's 's behavior based on these features. This evaluation allows for the
12

---

Tollgate
0.40
0.35
Tolawi CAT
0.4
Success Rate
0.25
0.3
0.20
0.2
0.15
0.10
0.1
0.00
0.2
0.8
Timesteps
Timesteps
Purking Lot
Intersection
CAF
TraCo CAF
0.25
TaCC Mr CAF
80.20
Success Rat
30.20
0.5
cesso.1
0.4
10.10
0.1
0.00
0.0
0.00
0.25 0.50
0.75
1.25 Timesteps
1.00
1.50
1.75
2.00
0.0
0.2
0.4
0.6
0.8
1.0
Timesteps
Figure 7: Performance comparison of TraCo with and without counterfactual advantage functions.
Table I: The traffic coordinator network re-issues the command according to the current situation different time intervals, and the command z remains unchanged during this time interval.
Bottleneck
Tollgate
Parking lot
Intersection
Success Rate Elliciency Sufety
Success Rute Elliciency Sufety Success Rate Efficiency Safety Success Rate Elliciency Salety
TraColl 0.36 13.13
0.26
0.36
0.36
0.19
0.22
0.38
0.27
0.04
0.21
0.27
0.73 + 0.05
0.51
0.73
TraCo/2 0.37 0.09
0.27
0.37
0.32
0.15
0.18
0.34
0.15
0.07
0.11
0.16
0.72 = 0.01
0.51
0.72
TraCo/4 0.38 0.07
0.28
0.17 0.16
0.09
0.2
0.14
0.15
0.72 =
0.72
TraCol6 0.09
0.3
0.25
TraCo88 0.34 0.12
0.25
0.34
0.29
0.18
0.16
0.31
0.21 0.04
0.16
0.21
0.74 1: 0.02
0.53
0.74
measurement of the agent's contribution to itself and the surrounding team, effectively addressing the interests balance problem.
Taking inspiration from the behavior of real-life traffic coordinators, who issue commands based on vehicle behavior and intersection information at time intervals rather than continuously directing vehicles, designed different time intervals for the Traffic Coordinator Network (TroCo) to extract features. As shown in Table 1, our experiments reveal that in complex traffic environments such as Tollgate and Parking lot, where obstacles are numerous, roads are congested, and the behavior of domain agents difficult to predict, frequent direction is necessary to ensure optimal vehicle decision-making However, Bottleneck and Intersection tasks, where the purpose of the vehicle clear, and the behavior is more predictable, frequent direction may interfere with the agent's decision-making. In such cases, appropriate time interval enhance the consistency of the agent's is behavior.
13


Reviews:
[{"review_id": "v-dr4FvsOg", "reviewer": "Reviewer_nkFo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors consider the problem of managing traffic flow and propose an algorithm with a virtual agent that serves as a traffic coordinator, sending orders to other individual agents. The authors introduce the Counterfactual Advantage Function (CAF) to motivate agents to work in line with the team's interests. Additionally, they utilize cross-attention to effectively handle dynamic numbers of observations and agents. In their experiments, they conduct experiments using RLLib and demonstrate that their algorithm outperforms existing approaches in four different traffic flow simulations", "summary_of_recommendation": "It is an important paper that demonstrates the successful approach to modeling traffic flow and showcases the emergence of social behavior among vehicles. It is written clearly but certain sections that are crucial for understanding are missing. Additionally, I noticed a few typos and proposed to write more information about problem setting and experiments. After all, I find this paper significant and relevant to the conference\n\nAfter author response: I appreciate the author's rebbutal. However I miss the final draft to change my score.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "•\tClose round-bracket in Equation (1)\n\n•\tAt line 204 – “fro”\n\n•\tI miss at least a brief description of observation and action space as well as problem formulation and experiment description\n\n•\tI would suggest moving equations in the appendix and addressing missing parts", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality) The paper introduces a novel approach to modeling traffic flow by utilizing the Counterfactual Advantage Function and cross-attention mechanism. This approach surpasses existing solutions and promotes social behavior, effectively reducing collision rates.\n\nQuality) The submission is technically convincing. The experiment results are well-described and persuasive.\n\nClarity ) The paper does not provide enough information about problem formulation, action and observation space as well as algorithm and hyperparameters, however authors state that there will be more details about it the in appendix. I found a few typos, which will be listed in the questions section\n\nSignificance) The paper and its results are important and can be used by other researchers to further improve traffic flow simulation. The paper improves the safety and success rate of vehicle agents.\n\nRelevance) I think it is a relevant paper because it outperforms existing solutions in traffic flow simulation. This paper brings us closer to a future where such experiments can be conducted in real-world scenarios.\n\nLimitations) I think limitations are addressed well\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "j1QAV2kaUK", "reviewer": "Reviewer_qaUa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces the Traffic Coordinator Network (TraCo), a novel method for addressing Multi-agent Reinforcement Learning (MARL) limitations and the context of autonomous driving. Traditional MARL systems fail to accurately evaluate an agent's contribution to the team, which may change over time. The TraCo approach in this study resolves this issue by focusing on team needs and interests. TraCo uses a cross-attention mechanism and a Counterfactual Advantage Function (CAF) to understand distinctive characteristics of domain agents and accurately quantify an agent's contribution. This approach facilitates improved decision-making in traffic flow simulations. While there are still gaps in replicating real-world vehicle behaviors due to reinforcement learning's random exploration, this approach significantly outperforms existing models in various metrics.", "summary_of_recommendation": "This paper presents the Traffic Coordinator Network, a pioneering solution to address the shortcomings of Multi-agent Reinforcement Learning in autonomous driving. This is a high-quality paper, and I recommend a strong acceptance.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Questions:\n1. The authors could elaborate on the agents' observations, states, and actions under different experiment settings. What are the observations, states, and actions for the four experiment scenarios?\n2. The authors mentioned three metrics in Section 4.2 and Fig 4. However, the authors should have explained how the other two metrics, safety and efficiency, are calculated.\n3. While the authors cited the original paper, the authors could explain the details of the cross-attention mechanism.\n4. Can the model handle external impacts on traffic patterns? And what could be the corresponding changes in MARL agent configurations?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is a prime example of high-quality academic writing. All figures are informative and plotted clearly. Reading this paper is an intellectual pleasure. I believe this is a high-quality paper in clarity, originality, and significance. And I don’t have many suggestions.\n\nStrengths:\n1. The logical flow of this paper is well-executed, facilitating a comprehensive understanding of the presented ideas.\n\nSuggestions:\n1. The authors could mention some details in the TraCo model training process, including the training environment setting.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "RduFOL3So9f", "reviewer": "Reviewer_9NNb", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes TraCo, which utilizes a combination of cross-attention and counterfactual advantage function to learn a virtual traffic coordinator for cooperation with multi-agent reinforcement learning. The goal is to accurately quantify the contribution that a team needs from an agent in the field of autonomous driving and to enable the emergence of rich and diverse social behaviors among vehicles within the traffic flow. There are many details that need to be explained in detail by the author.\n", "summary_of_recommendation": "There are many details that need to be explained in detail, thus leaning to a rejection.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please refer to the pros and cons part.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There are many details that need to be explained in detail by the author.\n\n1. What are the functional differences between the coordinator and the traffic light? Why not train the signal light directly？\n\n2. Overall, the method description is quite confusing. Please provide a detailed description of the method and explain why it can solve the problem of \"cannot express the contribution that the team needs from the agent.\n\n3. The method is similar to HRL, where the top layer provides goal z, while the bottom layer makes decisions based on z from individual rewards and group rewards. The author needs to introduce the top-level part in detail, i.e., the blue section of TraCo in Figure 2. The author provided a relatively detailed definition of Ai, but from the structure in Figure 2, it is not the main part of TraCo.\n\n4. As it is an application-oriented problem, the author also needs to introduce the model definition of the environment in traffic scenarios, including observation, actions, and two rewards. \n\n5. Are the two actors and two Qtots in Figure 2 the same? How to estimate Q(s, a(z-))? Where does its data come from? How to obtain r and s' corresponding to a(z-)?\n\n6. The definitions of Qtot and Qnet in the article need to be introduced. Different symbols are used for the same definition before and after the article.\n\n7. How to set the hyperparameter alpha in formula 6\n\n8. Why is Figure 4 only showing three scenarios without another one?\n\n9. Need ablation experiment to demonstrate the effectiveness of the proposed counterfactual strategy in training the Q(s,a(z))-Q(s,a)", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "0ud6c6WrSp", "reviewer": "Reviewer_TCct", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a method that combines Traffic Coordinator Network (TraCo), Counterfactual Advantage Function (CAF), and attention mechanism.\nThe proposed method is based on multi-agent reinforcement learning and aims to solve problems related to automated driving in multi-vehicle environments.\nOn busy roads, vehicles need to operate efficiently, which is also related to safety.\nThe experiment was compared to several baseline algorithms.\nA simulator environment called METADRIVE was used.\nResults showed the effectiveness of the proposed method.", "summary_of_recommendation": "There are some concerns with the lack of clarity in the claims made in the introduction of this paper and in the experiments.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The critical factors in adapting multi-agent reinforcement learning to self-driving vehicles appear to be learning speed and safety.\nA large data set would be needed for learning. It is unclear how the proposed approach would work for this problem.\nCould it sustain better performance with less data than other existing methods?\n\nIn addition, there are questions about whether multi-agent reinforcement learning is really appropriate for solving the challenges of automated driving in traffic situations like this one.\nThe proposed method has the advantage of being able to split the reward into individual vehicles.\nHowever, some disadvantages may also exist.\nFor example, there is the issue of quantity and variation of the data set.\nIn addition, the proposed method seems to only compare with other reinforcement learning methods, and no comparison has been made with methods that control the entire system centrally or with other heuristic methods.\nWouldn't it be imperative for the authors to compare the proposed method with these other approaches in order to evaluate it from a broader perspective?\n\nI have some questions about the experiment. As supplementary material, the authors have attached a video. In this video, the vehicles were seen crashing. Afterward, the colliding vehicle completely disappeared.\nThis is very different from the real case. Does the disappearance of the car all at once after the collision cause any problems in learning the model?\nIt is also unclear how the data that had been acquired from the vanished car is handled.\n\nAdditionally, cars were seen crossing the white line. In automated driving, it would be very important to ensure traffic laws are observed.\nThe proposed method does not seem to have a mechanism to ensure compliance with the law.\nIs this a method that could be extended in the future?\nI have questions about how these cars can comply with the law.\n\nThere appear to be several minor typographical and formatting errors in this paper.\nThe author could review the paper in detail and correct them. For example, the paper may not include a space before the citation number. A space between the citation number and the word preceding it should be included.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper seems to be about how to control a large number of vehicles on a busy road.\nIt is interesting to note that the methodology approach in this study is based on multimodal reinforcement learning, where the overall reward can be decomposed as the reward of individual vehicles.\nHowever, the environmental settings used in the experiments are simple (such as a nondescript crossroads or a bottlenecked power line road) and may not be sufficiently validated to be effective in a wide variety of traffic situations.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presentes a multi-agent RL method for autonomous driving in busy enviornment, featuring a traffic coordinator network. 

The reviewers have agreed that the paper presents in interesting approach to a timely issue in the context of autonous driving. Most reviewers have found the approach being technically sound and well-described. However, there is a concern on the reality of the test scenarios in the paper, in the sense that the simulated environment does not capture the chain reaction of accidents in real driving conditions; this concern has not been completely resolved in the rebuttal phase. 

While the AC thinks that the proposed scheme has limitation to be directly applied to real situations, he/she also thinks that the presented idea and the verification on a (limited) virtual environment would also be useful for the audience of CoRL. The authors need to incorporate the comments provided in their rebuttal in a final version.
Decision: Accept (Poster)