DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving

Authors: Xuemeng Yang, Licheng Wen, Tiantian Wei, Yukai Ma, Jianbiao Mei, Xin Li, Wenjie Lei, Daocheng Fu, Xing Gao, Pinlong Cai, Tao MA, Min Dou, Hongsheng Li, Liang He, Yong Liu, Botian Shi
Primary area: applications to robotics, autonomy, planning
Keywords: Autonomous Driving, Diffusion Model, Closed-loop Simulation

Abstract:
This paper introduces DriveArena, the first high-fidelity closed-loop simulation system designed for driving agents navigating real-world scenarios. DriveArena comprises two core components:  Traffic Manager, a traffic simulator capable of generating realistic traffic flow on any global street map, and World Dreamer, a high-fidelity conditional generative model with infinite auto-regression. DriveArena supports closed-loop simulation using road networks from cities worldwide, enabling the generation of diverse traffic scenarios with varying styles. This powerful synergy empowers any driving agent capable of processing real-world images to navigate in  DriveArena's simulated environment. Furthermore,  DriveArena features a flexible, modular architecture, allowing for multiple implementations of its core components and driving agents. Serving as a highly realistic arena for these players, our work provides a valuable platform for developing and evaluating driving agents across diverse and challenging scenarios.  DriveArena takes a significant leap forward in leveraging generative models for driving simulation platforms, opening new avenues for closed-loop evaluation of autonomous driving systems.

Decision: reject
Reviewer scores: 3, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730259191695

thread_type:
official_review

<thread_object>
creator: reviewer_i9pt

<content>
summary:
The paper proposes a traffic simulation platform for testing autonomous driving algorithms. The simulation architecture is built based on LimSim, using Monte Carlo tree search for vehicle motion planning. A diffusion-based renderer is applied to achieve realistic and controllable surround images for vision-based driving agents. The platform supports driving scenario construction from nuScenes and OSM; codes and tools are open-source for the community to use.

strengths:
Compared to previous work using video generation methods as world models to achieve realistic traffic simulation, this work uses a two-step pipeline, including rule-based traffic simulation in geometric space and diffusion-based video generation conditioned on trajectory layouts of vehicles. I believe this approach can achieve better physical realism and temporal consistency.

Text prompts are introduced to achieve diverse driving scenarios and plenty of demos are presented to clearly show the generation results.

The codes are well organized with a modularized design and the whole platform is open-source to better support downstream tasks in research of autonomous driving.

weaknesses:
The author says that one of the main contributions is scalability, which means simulation on any region can be achieved by using map info from OSM. As far as I know, OSM only contains road-level map information, and extra efforts like completing lane topology and planning vehicle OD are needed to construct simulations based on it, this part of the work seems unclear in this paper.

As the dreamer is the most important part of this paper, it would be better if the author could provide some indicators that can directly evaluate the generated results, like FID.

Minor note: it seems that Figure 3 is not in vector format.

questions:
+ About constructing traffic scenarios from OSM.
  + How is the HD map built from the OSM data, and how is traffic demand generated in this kind of scenario? 
  + OSM maps are not high quality in many areas, is there a way to solve this?
  + Is this part of the work mainly based on the tools provided by SUMO or LimSim?
+ Are there any indicators to evaluate the generated results directly, like FID or FVD?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730504000256

thread_type:
official_review

<thread_object>
creator: reviewer_uuyz

<content>
summary:
This work presents DriveArena, an image-based high-fidelity simulator for closed-loop simulation of agents for autonomous driving. DriveArena consists of a Traffic Manager and a World Dreamer, and its modular design allows for easy replacement of each of the components. Traffic Manager enables the dynamic control of all traffic agents and supports various HD maps, both of which are inputs to World Dreamer, which uses conditional diffusion to generate realistic images. The diffusion model is conditioned on map and object layout and generates images autoregressively for temporal consistency and variable length simulations.

DriveArena is evaluated in two ways. First, its fidelity is evaluated, among others, with UniAD's performance. Then, open- and closed-loop evaluation of VAD and UniAD models is performed.

strengths:
* Highly relevant research direction: The work correctly argues for the need of closed-loop evaluation of autonomous driving behavior models.
* Novelty: High-fidelity closed-loop image-based simulation with clear controllability (e.g. via text prompts).
* Performance: Evaluation of sim-to-real gap shows superiority over MagicDrive and reasonable results for open-loop and closed-loop behavior eval.
* Well presented: The paper is easy to follow and all information is well presented.

weaknesses:
* Evaluation of sim-to-real gap: The presented evaluation is pretty short and for example lower L2 distances do not necessarily imply higher quality images. Additional evaluation of the fidelity would be helpful. Are there perception metrics such as FID that could be used or other metrics that compare statistics between ground truth and generated images? Otherwise, user studies are another possibility to judge the quality.
* Unclear takeaway from VAD vs. UniAD open- and closed-loop comparison: In open-loop, UniAD performs better on nuScenes but worse on DriveArena than VAD. This difference is explained with better open-loop generalization of VAD. However, it's unclear what role the fidelity of DriveArena plays. Is it possible to e.g. run an experiment with different DriveArena datasets, some that are closer and some that are further from nuScenes? In closed-loop eval, UniAD outperforms VAD in DriveArena. It's unclear whether these differences are due to open- / closed-loop model gaps or issues in DriveArena. I acknowledge that this difficulty of correct attribution is inherent to research in this area but you might be able to shed more light on this. For example, would it be possible to evaluate the models on various levels of fidelity in DriveArena to disentangle open- / closed-loop eval from it?

questions:
* Evaluation of sim-to-real gap: How are the videos generated? Is this in open- or closed-loop? For correctness, it should be closed-loop. If so, do you notice any suffering from DAgger issues?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681997162

thread_type:
official_review

<thread_object>
creator: reviewer_jafa

<content>
summary:
This work presents a generative simulation framework for autonomous driving. In particular, a layout-conditional diffusion model is proposed as a sensor simulator, with bounding boxes and road graphs serve as underlying world state.

strengths:
The topic of generative simulation is an especially timely one for the AV field, with better and better generative video models coming out regularly and frameworks such as this one being able to capitalize on these parallel investments.

While all of the individual pieces of this framework have existed before, connecting them all together into a usable simulation framework for the broader community to use is appreciated and presents a potential path towards practical generative simulation.

The paper is written well and it is easy to follow the core ideas as presented.

Leveraging layout-conditional generation is a sound and sensible idea for maintaining consistency across time.

weaknesses:
The core weakness is that any geometric or semantic consistency is not guaranteed. Layout conditioning certainly helps, but in the linked webpage's videos there are clear inconsistencies across timesteps (e.g., car color and types changing over time). This is something that is not brought up in Lines 190 - 198, but perhaps should be as it is the core reason why works leverage NeRF or 3D Gaussian Splatting (for their geometric/semantic/temporal consistency over purely-2D generative models).

While static images of generations appear to be of good quality, there are significant temporal consistency issues when viewed as part of a video on the linked project page (most videos appear to be static even with the ego-vehicle theoretically moving forward in the world). Do the authors have any idea for why that is? It almost appears that the video diffusion model suffers from mode collapse when tasked with generating building walls (taking example from the linked webpage).

The AV results in Tables 1 and 2 still show a significant gap to real data, indicating that, while the core points of DriveArena are sensible, there is still much work to be done to leverage it for practical AV development.

questions:
In Lines 261-263 it is written "We also verify that extending to a multi-frame auto-regressive version (using multiple past frames as reference and outputting multi-frame images) and adding additional temporal modules can enhance temporal consistency." - How does this work in practice? In closed-loop, the ego-vehicle can drive however it wants and so there can still be inconsistencies between generated videos at time t and t+1, right? Further, how are the generated videos used? Are T frames predicted but only the 1st one is shown to the ego-policy? And then a new prediction is made (similar to model-predictive control)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730738855349

thread_type:
official_review

<thread_object>
creator: reviewer_pxgv

<content>
summary:
The submission introduces DriveArena, a high-fidelity closed-loop simulation platform designed for testing and developing autonomous driving agents in real-world scenarios. The platform consists of two main components: the Traffic Manager and the World Dreamer. The Traffic Manager is responsible for generating realistic traffic flow on any global street map, while the World Dreamer is a high-fidelity conditional generative model that creates infinite autoregressive simulations. DRIVEARENA enables the generation of diverse traffic scenarios with varying styles and allows driving agents that can process real-world images to navigate within its simulated environment.

strengths:
1. This submission targets an important problem in the field of autonomous driving: how to properly evaluate the performance of end-to-end systems. The submission introduces a closed-loop evaluation method, which is more reflective of real-world driving conditions compared to open-loop evaluations.  It will be useful for practical applications.
2. The platform utilizes road networks from cities worldwide and allows for the generation of diverse traffic scenarios with varying styles, which is essential for training and evaluating driving agents across different driving environments. 
3. The submission provides a clear and detailed explanation of the technical aspects of DriveArena. The figures, tables, and appendices enhance the understanding of the system's components and their interactions.

weaknesses:
1. The submission lacks novelty. The proposed DriveArena is just a combination of several existing methods: LimSim for traffic simulation and condition generation; DriveDreamer/Vista for generating images from the conditions; NAVSIM and Carla for closed-loop evaluation. 
2. The World Dreamer model is trained primarily on the nuScenes dataset, which may not capture diverse driving scenarios. To improve the model's generalizability, it would be beneficial to incorporate additional datasets that represent different geographical locations, driving cultures, and road conditions.
3. This submission fails to address an important issue for closed-loop evaluation: the model should be able to generate the same scene captured from different positions (similar to actual scenarios of driving differently in the same scene). No visualization was found addressing such an issue. 
4. Experiments are not enough. The submission primarily focuses on the simulation platform itself rather than an in-depth evaluation of various driving agents within the platform. Expanding the experimental section to include a broader range of driving agents and more extensive testing can help provide a clearer picture of DRIVEARENA's capabilities and limitations.
5. Minor issues in writing and presentation. For example, The figures are not vectorized for zooming in and they are suggested to be replaced.

questions:
Can authors provide visualizations of generating consistent scenes based on slightly different conditions (e.g., the ego car moves differently), which is a key aspect for closed-loop evaluation?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes a closed-loop simulator for autonomous driving. The simulation involves two components: a traffic simulation system that generates traffic flow simulations and a multi-view image generation system for creating images based on the generated traffic and text prompt. The topic of AV simulation is timely and important, and the present work is clearly written. However, it is limited, as the underlying design choices limit geometric and semantic consistency, which ultimately puts the usefulness and contribution of such a platform into question. Given that the authors position this work from an overall system perspective, I am weighing the lack of this consistency higher. The authors are correct in that consistency is not the only thing needed for meaningful AV simulation, but it does seem to be an important requirement for such a work to be broadly useful for the community.