SimGen: Simulator-conditioned Driving Scene Generation
Authors: Yunsong Zhou
Michael Simon
Zhenghao Peng
Sicheng Mo
Hongzi Zhu
Minyi Guo
Bolei Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: Autonomous Driving
Generative Models
Simulators
Primary area: machine_vision
TL;DR: This paper introduces a simulator-conditioned diffusion model, which follows the layout guidance from the simulator and cues of the rich text prompts to generate images of diverse driving scenes.

Abstract: Controllable synthetic data generation can substantially lower the annotation cost of training data. Prior works use diffusion models to generate driving images conditioned on the 3D object layout. However, those models are trained on small-scale datasets like nuScenes, which lack appearance and layout diversity. Moreover, overfitting often happens, where the trained models can only generate images based on the layout data from the validation set of the same dataset. In this work, we introduce a simulator-conditioned scene generation framework called SimGen that can learn to generate diverse driving scenes by mixing data from the simulator and the real world. It uses a novel cascade diffusion pipeline to address challenging sim-to-real gaps and multi-condition conflicts. A driving video dataset DIVA is collected to enhance the generative diversity of SimGen, which contains over 147.5 hours of real-world driving videos from 73 locations worldwide and simulated driving data from the MetaDrive simulator. SimGen achieves superior generation quality and diversity while preserving controllability based on the text prompt and the layout pulled from a simulator. We further demonstrate the improvements brought by SimGen for synthetic data augmentation on the BEV detection and segmentation task and showcase its capability in safety-critical data generation.

SimGen: Simulator-conditioone Driving Scene Generation
Yunsong Zhou
Michael Simon'
Zhenghao Peng'
Sicheng Mo
Hongzi Zhu
Minyi Guo Bolei Zhou'
University of California, Los Angeles Shanghai Jiao Tong University
https://netadriverse. gitbeb...it............. gitb...... ho/simgen/
Rich Appearance
Text Prompt
Christmos night
Streal Talayau
Snowy rood
Simulator
Rainy day
SimGGn
Challenging Layout
Rear-end Cutting collision
in
Control Conditions
SimGen is a controllable scene generation paradigm conditioned on a simulator. learns from real-world and simulated data and then generate diverse driving scenes based on the simulator's ' control conditions and text prompt.
Figure 1:
Abstract
Controllable synthetic data generation substantially lower the annotation cost of training data. Prior works use diffusion models to generate driving images conditioned the 3D object layout. However, those models are trained small- scale datasets like nuScenes, which lack appearance and layout diversity. Moreover, overfitting often happens, where the trained models can only generate images based on the layout data from the validation set of the same dataset. In this work, we scene generation framework called SimGen that can learn to generate diverse driving scenes by mixing data from the simulator and the real world. uses novel cascade diffusion pipeline address challenging sim-to-real gaps and multi-condition conflicts. A driving video dataset DIVA is collected enhance the generative diversity of SimGen, which contains over 147.5 hours of real-world driving videos from 73 locations worldwide and simulated driving data from the MetaDrive simulator. SimGen achieves superior generation quality and diversity while preserving controllability based the text prompt and the layout pulled from simulator. We further demonstrate the improvements brought by SimGen for synthetic data augmentation the BEV detection and segmentation task and showcase its capability in
introduce
data generation.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
A high-quality and diverse training data corpus is crucial for autonomous driving research and development. However, is costly and laborious to annotate the data. Synthetic data generation is a promising alternative to harvest annotated training data, which brings realistic images and notable performance improvements across tasks like object detection [10] and semantic segmentation [75]. Besides the realism of the generated images, there are two necessary conditions to consider for practical synthetic data generator for autonomous driving: 1) Appearance diversity, which ensures the synthetic data can cover a spectrum of weather, environmental, and geographical conditions. 2)
Layout diversity, namely the distribution of objects, should cover different traffic scenarios, including safety-critical situations that are rare to collect in the real world.
Recent diffusion-based generative models show promising results to generate realistic driving images from text prompts [78], BEV road maps [67], and object boxes [20, 70, 72, 79]. Despite generating coherent images, these attempts lack the generalizability of generating new and diverse real-world appearances and traffic scenarios due to data limitations. They are confined to learning on small-
datasets [29, 32, 44, 71] with limited scenarios such as only urban streets [5] or restricted weather conditions [52]. In addition, the driving behaviors in the available driving datasets like nuScenes are tedious and lack complex or safety-criticn
situations. Another option for collecting synthetic data is from driving simulators, which can effortlessly generate scenes encompassing various behaviors with its physics and graphics engines [16, 36, 58, 61, 66]. Simulators also provide accurate control over all objects and their spatial locations, thus can easily generate huge amount of traffic layout maps. However, open-soure simulators usually only contain a limited amount of 3D assets, and they lack a realistic visual appearance. Thus, the models trained on Aimulator-generatiee data can easily overfit, also known as the Simulation to Reality (Sim2Real) gap.
We take the best of two worlds by integrating the data-driven generative models with simulator to obtain both the appearance diversity of real-world data and the layout controllability of simulated data. To this end, we introduce SimGen, a cimulator-conditione diffusion model, which follows the layout guidance from the simulator and rich text prompts to generate diverse driving scene images. One naive approach is to guide an image generation model with the depth and semantic images from the simulator via training a control branch through ControlNet [81]. Yet, as the simulator has limited assets and cannot fully capture the variations in the real world, the simulated conditions and the underlying real-world conditions that guide a diffusion model to generate real-world images might have conflicts. To tackle this, SimGen adopts cascade design. The model first injects noise-added simulated conditions such as depth and semantic images into the intermediate sampling process of a diffusion network. The network then converts simulated conditions
pre- pre-trained tit--to-rell-codiie into more realistic conditions via continuous denoising, free of additional training on simulated conditions beyond this diffusion network. After that, second diffusion module utilizes adapter to integrate multimodal conditions and uses masks to filter conflicting data. SimGen thus achieves outstanding generation quality and diversity while preserving layout controllability by connecting with the simulator.
We construct dataset called DIVA to obtain the appearance and layout diversity of the training data. DIVA comprises two parts: the web data and the synthesized data from the simulator. On the one hand, web data covers worldwide range of geography, weather, scenes, and traffic elements, preserving the appearance diversity of a wide range of traffic participants. We design a data curation pipeline to collect and label YouTube driving videos. On the other hand, virtual driving videos with the traffic flow replayed from trajectory datasets or generated by a safety-criiical scenario generator [80] are collected from driving simulator [36]- In short, DIVA dataset blends real-workd appearances and virtual layouts, consisting of 147.5 hours of Diverse In-the-wild and Virtual driving datA.
We summarize our contributions as follows: 1) novel controllable image generation model SimGen incorporating a driving simulator to generate realistic driving scenarios with appearance and layout diversity; 2) new dataset DIVA containing massive web and simulated driving videos that ensures diverse scene generation and advances simulation-to-realityy research; 3) SimGen improves over counterparts like BEVGen [67]. MagicDrive [20], Panacea [72], DrivingDiffusion [38], Le., in terms of image quality, diversity, and controllability of scene generation.
'The work was done when YZ was visiting student UCLA.

---

2 Appearance Diversity and Layout Diversity from DIVA Dataset
We introduce large-scale DIVA dataset containing diverse driving scenes in the real world and the simulation. It facilitates the training of generative models and tackles the simulation-to-reality (Sim2Real) challenge. Tab. displays the statistics, composition, and annotation of the data, which comprises about 147.5 hours of driving videos. The data is collected from a vast corpus of high- quality YouTube driving videos and simulation environments in the MetaDrive simulator [36]. We use DIVA-Rest and DIVA-Sim to denote the web data downloaded from YouTube and the data from the MetaDrive simulator, respectively. Comparisons with other datasets, license, and privacy considerations are detailed in Appendix B.
2.1 DIVA-Real: Appearance Diversity in Web Data
Collecting web videos. As shown in Fig. 2 (left), to streamline the process and minimize manual effort, we begin by searching for relevant keywords on YouTube to identify a batch of driving video channels. The videos are downloaded from these identified YouTube channels. We filter out unsuitable videos hased on their length and resolution and proceed to download the appropriate ones. This yields hundreds of first-person driving videos, each with an average duration of one hour. Next, we sample the videos into frames at 10Hz, excluding the initial and final 30 seconds to eliminate user channel information. This process yields over 4.3 million frames, awaiting further data cleaning.
Data cleaning and autolabeling. Data cleaning is vital for ensuring data quality, but manual inspection of each image is impractical. In- spired by [78], we implement an auto- mated data-cleaning workflow to ex- pedite the process. With the remark- able image understanding capabilitiess of the vision- Laaion-a model (VLM), i.e. -aaMttd V2 [19], we are able to conduct the quality checks via VLM with checklist including cri- teria such as non-front view, video transition, black screens, etc, to iden- tify nonconforming images. Driving videos are chunked into five-frame batches. For each batch, the VLM chooses and assesses random image; this single image fails to pass checks, the entire batch of five frames will be discarded. In the autolabeling process, pre-trained models for various tasks, including BLIP2-flant5 [56]. ZoeDepth [3], and Segformer [76], are used to generate annotations of text, depth, and semantic segmentation, respectively. Eventually, over 1 120 hours of driving videos with rich annotations collected.
Table 1: Comparing DIVA with relevant datasets on scale, diversity, and annotations. * perception subset. *: includ- ing procedural generation [36] and safety-critical [80] data. Cts: countries; Seg: segmentation; Virt: virtual image.
Datasel
Time
Frames
Cities
Annolations
(haurs)
Depth Seg. Virt.
KITTI 1221
1.4
15kk
1
SitySeapes [13]
0,5
25k
50
Waymo" [65]
390k
3
Argoverse 2" [74]
4.2
300k
6
nuPlan" [7]
120
4.0M
2
4
Honda-HAD HAD [31]
1.2M
nuScenes [6]
5,5
241k
2
2
Real
4.3M
19
71
DIVA-Sim
9983.
5
DIVA (AII)
147.5
5.3M
22
76
2.2 DIVA-Sim: Layout Diversity from the Simulator
Simulators are capable of faithfully reconstructing real-world scenes and hence obtaining training data with layout diversity. Also, after loading the driving scenarios such as map topology from the dataset, the simulator allows changing the motions and states of the traffic participants with pre-defined rules or interactive policies that differ from the original ones. This inspires us to build Sim2Real data from the simulator. The Sim2Real data is induced from the same real-world scenarios, in which we can obtain real-world map topology, layout, and raw sensor data. At the same time, we can reconstruct the paired data from those scenarios but with reconstructed sensor data and even with altered layout and traffic flows. DIVA-Sim utilizes the MetaDrive simulator [36] and ScenarioNet [37] to gather 5.5 hours of virtual driving videos from nuScenes layouts [6] and another 22 hours from procedurally generated behaviors. It includes set of safety-critcaal driving data through interactions introduced by an adversarial traffic generation method [80], further improving the diversity of our dataset.
Scene layout construction. We utilize ScenarioNet [37] to transform scenes into unified description format suitable for simulators, known as scene records, logging map elements and objects. As illus- trated by the example scene in Fig. 2 (right), loading scene records, MetaDrive [36] can reconstruct
3

---

Youtube Data Pool
Scene Record
Control Policy
Diccrded Data
nuscenes + Waymo +GG
ReployPolicy Safaty critical
YouTuber Selection
Spilting & Procesiigg
Vo.
METADRIVE
Example Scene
Video Collecting
Simulator
Checkist
Mapp Bements Objects
Pleass evaluate af this image scrviiits any Blumines??
Blok screers
Nurthes
Sensors
Va this image of landscole?
Autolabelling
subilling
Landscape
Rendering
DIVA-Real
DIVA-Sim
Image
Seg
Messoyclitt -Wountddroodd...
RGB
Depth
Seg
Instance
Figure 2: Constructing DIVA dataset. DeVA-Real (left) comprises driving videos collected from YouTube. We apply Vision Language Model to filter out noisy images via a checklist and utilize off-the-shelf models to annotate text, depth, and semantic labels. Meanwhile, DIVA-Sim (right) employs scene records and control policies in simulator to create map elements and objects. It can generate digital twins of real-world data and safety-critical scenes. Then various kinds sensors placed in the simulation produce multimodal images. Ren. renderee::::;; T.D.: top- view. Numbers and letters indicate the sequence of processes.
roads, blocks, and intersections, and place corresponding 3D models like vehicles, cyclists, and pedestrians, based the recorded positions and orientations. We will reasonably select representative 3D models based on the category and dimensions of the objects. And the model's 's shape is scaled based on the real dimensions replicate the objects in the nuScenes dataset accurately. By doing so, the digital twin scenario can be faithfully reconstructed in the simulator.
Obtaining images via trajectory replay and rendering pipeline. The control policy determines the motion dynamics, while the sensors generate multimodal image data at any desired location. To create nuScenes digital twins, ReplayPolicy is applied to replay logged trajectories of all objects. Our cameras are placed in the exact pose of the nuScenes front camera, with the camera's 's field of view adjusted to match that of nuScenes closely. The camera attribute be set multiple types obtain a variety of sensor data. In summary, we can obtain the following conditions through the simulator: rendered RGB, depth, semantic segmentation, instance segmentation, and top-down views.
Creation safety-critical data. Besides building digital twins the real-world data, we can harness the simulator to continue growing the safety-critical data and enhance layout diversity. We apply the CAT method [80] to generate safety-critical data based on real-world scenarios. Specifically, we first randomly sample one scenario from the Waymo Open dataset [65]. A traffic vehicle is perturbed to attempt colliding with the ego-vehicle via adversarial interaction learning [80]. Thus, we harvest many safety-critica, scenarios with adversarial driving behaviors, which might be challenging to collect in the real world. This scalable creation of the safety-critical data from the simulator is also one of the strengths of our method.
SimGen Framework
SimGen aims to generate realistic driving images based on the text prompt and the spatial conditions including semantic and depth maps from real-world datasets and the driving simulator. We incorporate driving simulator into the data generation pipeline achieve controllable and diverse image generation. Incorporating the simulator provides access to diverse layouts and behaviors of traffic participants, thus better closing the Sim2Real gap. However, if just conditioning the diffusion model on synthesized data from a simulator, the diffusion model will result in bad image quality due to the limited assets and the artificial rendering. We propose cascade generative model that first transforms the simulated spatial conditions to realistic conditions as those in the dataset, then uses those realistic conditions to guide the first-view image diffusion model.
Illustrated in Fig. 3, SimGen first samples a driving scenario and a text prompt from the dataset and invokes the driving simulator MetaDrive [36] to render simulated conditions (SimCond), the

---

SimGen
Condbrr
Text
recr building
cor criiing
Adopter
Fred
Bihudion
MearCornd
Scone Recoro
IggOiff
CondDift
Qutput Image
ImgDif
Simulatio
SimCond
Cond
Mask
Denowing
Text
Stepe
SchoCond
Chrpul VINOOA
Condition Generation vo Simuator
Transformation
Redi Condition
Controlabie Image Generation
Module Training
Figure 3: Illustration of SimGen. SimGen processes text and scene record as inputs. The text is feature-encoded and utilized in the subsequent modules, whereas the scene record undergoes a simu- lator rendering into simulated depth and segmentation (SimCond) and extra conditions (ExtraCond). SimCond, coupled with the text features, is fed into the CondDiff module that converts SimCond into RealCond, representing real depth and segmentation. Eventually, the text features, RealCond, and ExtraCond are inputted into the ImgDiff module, where an Adapter merges multi-source conditions into unified control condition and generates driving scene images.
synthesized depth and segmentation images. Then, the SimCond and text features are fed into lightweight diffusion model CondDiff (Sec. 3.1) that converts simulated conditions into realistic conditions (RealCond), that resembles the real-world depth and segmentation images from YouTube and nuScenes datasets. Finally, diffusion model called ImgDiff (Sec. 3.2) generates a driving scene according to multi-modal conditions, including RealCond, textual prompts, and optional simulated spatial conditions, including RGB images, instance maps, and top-down views, etc.
3.1 Sim-to-Real Condition Transformation
While strive to align the simulator settings with real data, such intrinsic and extrinsic parameters of the camera, there is still a disparity between RealCond and SimCond. The disparity arises from image mismatch, inherent flaws of the 3D models, and the simulator's 's lack of background details (Appendix C.1.1). Consequently, simulator conditions require transformation to closely resemble real ones. An easy solution is use domain adaptation [48] and consider the SimCond and RealCond as different image styles. However, training a domain transfer model that can generalize to novel scenarios requires paired SimCond and RealCond data far exceeding public datasets like nuScenes. Thus, it's necessary to have an adaptation-fit free approach for Sim2Real transformation without additional training on SimCond. To achieve that, we first use data from DIVA-Real train diffusion model, CondDiff, that generates RealCond purely from text prompts. The training does not contain data rendered from simulators. During inference, CondDiff injects noise-added SimCond into the intermediate sampling process and converts it into realistic conditions via continuous denoising.
Learning generate conditions from text inputs. To facilitate the learning process of CondDiff. we initiate this stage with text-to- -exa-to-Realoond generation. Concretely, we utilize Stable Diffusson 2.1 (SD-2.1) [60], a large-scale latent diffusion model for text-to-image to--geg generation. denoising UNet, denoted by cp. with multiple stacked convolutional and attention blocks, which learns synthesize images by denoising latent noise. Let Xg E X' represents latent feature from the data distribution p(x). Starting from Xo, the training process involves gradually adding noise to procedure x, for E (0, 1] until x, transforms into Gaussian noise, namely forward stochastic differential equation (SDE) [27]. The model is optimized by minimizing the mean-square error:
implemented as
X; = 0g + oeC, ~ N I), Xo ~ p(x),
Vt, min Blle Eq (x 1)]
(2)
where is scalar function that describes the magnitude of the noise at denoising step f, is scalar function that denotes the magnitude of the data Xo, 0 parameterizes the denoiser model ep, e the added noise, and is the text condition that guides the denoising process. The learning occurs in a compressed latent space X' instead of the pixel space [60]. During sampling, the model iteratively denoises the final step prediction from the standard Gaussian noise to generate images.
5

---

The original SD-2.1 is trained on data from various domains unrelated to the depth and semantic images in driving scenes. As depicted in the CondDiff in the upper right of Fig. 3, we fine-tune the SD-2.1 to be text-to- text-to-RealCon RealCendCC model using the triplets of text, depth and segmentation data from DIVA-Real and nuScenes, with the objective of Eq. (2). After loading the SD-2.1 checkpoint, all parameters 0 of the UNet are fine-tuned at this stage, while the CLIP text encoder [55] and autoencoder
[18] remain frozen. The depth and segmentation data is autolabelled by a set of perception models as discussed in Sec. 2.1.
Adee sim-to-real transformation. Now, we have a model CondDiff that generates RealCond purely from text prompts. We will then use the conditions from simulator SimCond to guide the sampling process so that we can transform SimCond to RealCond. According to SDEdit [45], the reverse SDE, where the diffusion model iteratively denoises standard Gaussian noise to generate images, can start from any intermediate time. This inspires us to insert added SimCond into the intermediate time of the sampling process, and the model will use them as guidance to generate RealCond with the SimCond layouts. In detail, the module first encodes the SimCond into latent space to get x It selects specific time to E (0, 1) and perturbs the input x using Gaussian noise of standard deviation of, as follows:
Sample x NNx"iim oo.
The perturbing process will effectively remove low-level details like pixel information while pre- serving high-level cues like rough color strokes [45]. The noise- poise-processed image x scamlessly substitutes the diffusion model's state at time to during denoising. Thus, the intermediate state X1, = x poi serves as a guidance to solve the corresponding reverse SDE as follows:
- (xt, -1 100 4), Ex(xi, (4)),
where μg and Eg are determined by CondDiff Eg. The above equation iterates until the model generates a synthesized image Xo like RealCond at t. = O. Throughout this process, all parameters of CondDiff remain frozen, with only SimCond xim, text c, and noise affecting the sampling process.
3.2 Controllable Image Generation with Multimodal Conditions
In the second stage, we will use ceffusioo-bbaeed model to synthesize diverse driving images by integrating various con- trol conditions (Tab. 2), including the RealCond from the data or generated from SimCond by CondDiff, the textual prompt, and some extra conditions ExtraCond such as rendered RGB, instance segmentation, and top-down views from the simula- tor. ExtraCond offers additional information for the output image, including road typology and object attributes (orienta- tion, outlines, and 3D locations), highlighting the necessity of incorporating them into model control.
Table 2: Formats of conditions. Deal/SimCond: depth and segmen- tation; ExtraCond: rendered RGB, instance maps, and top-down views.
Dataset
RealConnd
SimCond ExtraCond
nuScenes
DIVA-Real
DIVA-Sim
However, there exist conflicts among multimodal conditions (Appendix C.1.2); 1) Modal discrepancy: The nuScene dataset contains full set of RealCond, SimCond, and ExtraCond, while YouTube only includes RealCond. This might impact the quality of images generated based on nuScenes layouts due to the data bias for diffusion models [33]. 2) Condition disparity: The lack of rich background information in simulated conditions compared to real ones results in struggle between the two modalities. In real-world images, the background might contain urban buildings with drastically different facades and street trees of different species. Although CondDiff can convert SimCond to RealCond, the domain gap prevents the same transformation for ExtraCond (e.g., rendered RGB, instance segmentation, and top-down views) from the simulator. Thus, we propose using unified adapter in Diff to address these issues. Its essence lies in mapping variable conditions into fixed-length vectors, overcoming the misalignment of low-level features, and enabling unified control input interface for the diffusion model.
Mitigating condition conflicts with adapters. Adapters are essential at the guiding branch of image generation to ensure the model learns necessary, unique, non-conflicting information from all conditions. Inspired by UniControl [54], we devised a set of convolutional modules as the adapters to capture features from various modalities, as shown in the ImgDiff in the lower right of Fig. 3. For a set of input conditions X {xt, x². ...." J. cach condition undergoes feature extraction via the x"

---

unified adapter Fada represented as:
K
Fada (x*) = Σ
Flo covl (F M)),
(S)
where the indicator function, x² the k-th condition image, and F Foo2 are the convolution layers of the i-th module the adapter. M is the valid mask for each condition.
The valid mask is the key to mitigating conflicts. The entire mask will be padded with 0 if a condition missing or not provided. For simulator- genirrreg conditions, we set the masks of backgrounds to based on the semantic labels, preventing unwanted constraints on background generation. Since top- down view conditions don't belong to the frontal perspective, all information retained. Ultimately, two convolutional layers process the concatenated condition features, max pooling them into fixed-length feature vector for control.
Controllable image generation. We utilize the ControlNet [81] to guide image generation. After the feature extraction by Fadas conditions are encoded into the UNet model. Then, the model injects control information into each UNet layer through residual connections. All parameters in UNet's input and middle layers are frozen, and we only fine-tune the output layers and the control branch.
4 Experiments
Setup and protocols. SimGen is learned in two stages on DIVA and nuScenes dataset [6]. The performance is evaluated based on image quality, controllability, and diversity. The Frame-wise Fréchet Inception Distance (FID) evaluates the synthesized data's quality. SimGen's controllability corresponds to how well the generated images align with ground truths from the nuScenes validation set. The controllability is measured by the 3D detection metrics (AP) and BEV segmentation metrics (mloU) when applying out-of-the- off-the-box perception models the generated images. Lastly, diversity measured using the pixel variance of the generated images. More details on training, sampling, and evaluation metries are provided in Appendix C and Appendix D.
4.1 Comparison to State-of-the-arts
Comparison to nuScenes-specífic models. We compare SimGen with the most recently avail- able data generation approaches exclusively trained nuScenes. Tab. 3 shows that SimGen surpasses all previous methods in image quality (FID) and diversity pDxx)) Specifically, Sim- Gen significantly increases pix by +6.5 com- pared DrivingDiffusion [38]. For fair compar- isons, we train a model variant (SimGen-nuSc)) on the nuScenes dataset only. We find that al- though SimGen-nuSe performs on par with Sim- Gen on nuScenes, its performance in diversity is less than ideal, and it struggles to generalize to novel appearances like Desert, Mountains, and Bl Blizzard, where the generation degrades to the nuScenes visual pattern. In contrast, SimGen trained on DIVA exhibits strong generalization ability across appearances shown in Fig. 4.
Table 3: Generation quality and diversity com- pared to nuScenes experts. The FID and Dpi indicate the image quality and pixel diversity, re- spectively. gray main metric. bold: best results.
Method
Dataset
FID
Dpix
BEVGen [67]
25.5
17.0
BEVControl [79]
24.9
MagicDrive [20]
nuScenes
16.6
19.7
Panacea [72]
17.0
DrivingDiffusion [38]
15.9
20.1
SimGen-nuSe
nuScenes
15.6
20.5
SimGen
DIVA
15.6
26.6
Controllability for autonomous driving. The controllability of our method is quantitatively assessed based on the perception performance metrics obtained using single-frame version of BEVFu- sion [40]. We feed the data from nuScenes validation set into SimGen and generate the driving images. Then, the perception performance of pre-trained BEVFusion, involving map segmentation (mloU) and 3D object detection (AP), recorded. Compared to the perception scores on the raw nuScenes data, the relative performance metrics serve as the indicators of the alignment between the generated images and the conditions. As depicted in Tab. 4, SimGen achieves relative performance of -3.3 map segmentation of vehicles, underscoring a robust alignment of the generated samples.
7

---

Conditions
Reference
SimGen
London
Desert
Barcelona
SimGen-nuSe
Miami
Columbla
Chicago
Conditions
Reference
SimGen
Storm
Downtown Atlanta
Berlin
SimGen-nuSe
Switzerland
Mountains
At dusk
Conditions
Reference
SimGen
Big Trees
Vegas
sports car
SimGen-nuSc
Small Town
Bangkok
City street
Conditions
Reference
SimGen
In the fall
Manil
Micinight
SimGen-nuSe
Blue sedan
Kuala lumpur
Blizzard days
Figure 4: Generating diverse appearances conditioned simulator's conditions and texts. We show the generation results SimGen (blue boxes) and SimGen-nuSe (gray boxes) under the same conditions. Compared to models confined to limited datasets, SimGen exhibits stronger ability to generate more realistic and diverse driving scenarios. Reference is not used for inference.
Data augmentation via synthetic data. SimGen can produce augmented data with accurate anno- tation controls, enhancing the training for perception tasks, e.g., map segmentation, and 3D object detection. For these tasks, we augment an equal number of images as in nuScenes dataset, ensuring consistent training iterations and batch sizes for fair comparisons to baseline. Tab. 5 indicates that blending generated with real data can elevate the singe-frame version of BEVFusion's vehicle mloU 39.0, a +4.4 uptick compared to models trained purely on real data. These outcomes reinforce SimGen's 's validity controllable synthetic data generator for enhancing perception models.

---

Table 4: Generation controllability for perception Table Comparison involving data aug- tasks. Oracle: single-frame version of BEVFu- mentation using synthetic data. The sion [40]. In blue the relative drop compared to Baseling ine single-frame version of BEV- standard nuScenes validation data.
Fusion [40] trained on nuScenes train set.
Map Seg
Object Detection
Map Seg
Object Det
Method
Method
mloUReed
mloUwaice
APor
APrack
mfoUsae
mloUwa
APcar
APrac
Oracle
72.2
34.6
47.0
21.4
Baseline
72.2
34.6
47.0
21.4
BEVGen [67] 50.1 (:211)
5.9 (-2877)
24.7 (:22.)
9.1 (-15.0)
BEVGen [67]
71.9
34.2
47.3
21.1
MagieD. [20] 58.6 (-136)
29.5 65.1)
37.3 (9.7)
17.3 (4..)
MagicD, [20]
77.4
37.7
48.0
22.8
SimGen-nuSe 60.6
29.9 (4:3)
39.1 (:7.)
18.1 (3.3)
SimGen-nuSe
77.7
38.0
48.3
23.0
SimGen
62.9 (-9.3)
31.2 (3.4)
41.0 (-6.0)
19.6
SimGen
78.9
39.0
49.1
23.6
Sudden broking
Crossroad meeting
Merging
Sharp turning
Figure 5: Generating safety-critical scenes. SimGen can also recreate image sequences of safety- critical scenes where risky driving behaviors like sudden braking and merging happen.
4.2 Ablation Study
The ablation is conducted by training each variant of our model on a DIVA subset with 30K frames, and report FID and av- erage precision of cars (APcar) as the quality and controllability metrics. We gradually introduce our proposed components and conditions, starting with a ControlNet baseline [81] that directly takes SimCond as input. As shown in Tab. 6, by introducing a cascade pipeline to transform SimCond into RealCond, the FID significantly reduces by -2.3, as the transformed conditions closely resemble real scenarios. Including simulator pulleed ExtraCond to the control conditions improves the alignment of the generated images with the target layouts, effectively en- hancing the AP.ar by +1.3. However, slight deterioration in the FID metric (+0.5) may result from condition conflicts. Lastly, using a Unified Adapter helps alleviate conflicts, significantly improving generated image quality by -0.8. The effectiveness of ExtraCond is exhibited in Fig. 6, where the addition of instance map, rendered RGB, and top-down view enables the model to better handle object boundaries, orientation angles, and occlusions in these cases. To the best of image quality, we only use depth and segmentation conditions in subsequent experiments.
Table 6: Ablation designs in SimGen. All proposed designs con- tribute to the final performance.
Ablation
FID/
APca f
Baseline
19.5
45.7
Cascade Pipeline
17.2
46.3
ExtraCond
17.7
47.6
Unified Adapter
16.9
48.2
4.3 Discussions
Extension to video generation. SimGen is not designed for video generation. But the high-quality image generation brings a potential for video generation, which is important for interactive scene generation and closed-loop planning. We have a preliminary attempt by integrating temporal attention layers into UNet similar to [78], and then conducting subsequent training stages focusing solely on learning the newly added layers while freezing the original parameters. This shows a promising result of temporal consistency across frames, as compared with video generation models in Appendix D.2.
Generating safety-critical scenarios. The key innovation SimGen the controllability of layouts brought by connecting driving simulator. Building upon video generation, we showcase SimGen's

---

Instance Map
Rendered RGB
Top-down View
w/o
Figure 6: Ablation study simulator conditions.
generalization capabilities in novel layouts, specifically in safety-critical scenarios in Fig. 5. The visualized layout is initialized from scenario sampled from the Waymo Open dataset [65] and then populated with risky behaviors via an adversarial interaction traffic flow generation method [80]. SimGen can transform safety-critical driving scenarios from the simulator into realistic sequential images, including risky behaviors like sudden braking, crossroad meeting, merging, sharp turning, etc. This application impossible with existing models, which are only trained and conditioned on given static real-world dataset that lacks records of dangerous driving behaviors. This brings new opportunities for closed-lood looed-lopp data generation capabilities (Appendix D.2).
5 Related Work
cartusion-basee generative Models. Diffusion models have made significant strides in image generation [15, 45, 49, 57, 62] and video generation [4, 24]. Recent works incorporate additional control signals beyond text prompts [23, 39, 47]. ControlNNet [81] integrates a trainable copy of the SD encoder for control signals. Studies like Uni Nottrollll [83] and UniControl [54] have also focused on fusing multimodal inputs into a unified control condition using input-level adapter structures. Our method distinguishes itself in its capability of multimodal conditioned generation by addressing the sim-to-real gap and condition conflicts in the complex realm of driving scenarios.
Controllable generation for autonomous driving. Autonomous driving research heavily relies on paired data and layout ground truths, spurring numerous studies on their generation [11, 44]. Some works [21, 28, 78] utilize diffusion models to generate future driving scenes based historical information, but they lack the ability to control scenes through layout. Other generative methods, like BEVGen [67] and BEVControl [79], use BEV layouts create synthetic single multi-view images. Recent innovative method Panacea [72] generates panoramic and controllable videos, while Magic- Drive [20] offers diverse 3D controls and tailored encoding strategies. Lastly, DriveDreamer [70] and Driving [38] employ diffusion models for realistic multi-view video generation and environment representation. Yet, these works confined to limited appearances and layouts of static datasets, restraining their real-world applicability and the controllability over the layouts that deviate from the dataset, such as the safety-critical scenarios.
Scenario generation via simulators. Driving simulators [16, 36] are fundamental to autonomous driving development, providing controlled simulations that mimic reality. Notable studies include SYNTHIA [61], AIODrive [73], and GTA-V [58] that generate vitural images and annotations. SHIFT [66] diversifies with environmental changes, while CAT [80] creates safety-critical scenarios for targeted training from real- world logs. Despite their layout diversity and attempts at photorealism enhancement [59], the simulated images lack realism. In this work, we bridge the two worlds obtain both the appearance diversity from diffusion models and the layout controllability from simulators.
6 Conclusion
We propose a simulator-conditions diffusion model, SimGen, that learns to generate diverse driving scenarios by mixing data from the simulator and real world. A novel dataset containing massive web and simulated driving videos is collected to ensure diverse scene generation and mitigate simulation- to-rliiiy gap. By obtaining diversity in appearance and layout, SimGen exhibits superior data augmentation and zero-shot generalization capabilities in generating diverse and novel scenes.
Limitations and future work. SimGen currently does not support multi-view generation, limiting its application in Bird's Eye View models. Inheriting the drawbacks of diffusion models, SimGen suffers from long inference time, which may impact the applications like closed-loop training. The study of extending SimGen to video generation is left for future work.
10

---

Acknowledgements
The project was supported by the NSF Grants CCRI-2235012 and RI-2339769, and the Sony Focused Research Award. YZ. HZ, and MG were supported by the National Natural Science Foundation of China (No. 62432008). ZP is supported by the Amazon Fellowship via the Science Hub for Humanity and Artificial Intelligence UCLA.
References
[1] Sami Abu-Fl-Haija, Nisarg Kothari, Joonscok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan, Youtube-8m: large-scale video classification benchmark. arXiv preprint Xrv: X6X:::60000000000 (609006655 2016. 23
[2] Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. Advances in Neural Information Processing Systems, 36, 2024. 17
[3] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint 2xx::2302.122888 2023. 3, 20, 21
[4] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable vidco diffusion: Scaling latent vidco diffusion models to large datasets. arXiv preprint ar arXiv:2311.1512, 2023. 10
[5] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023. 2
[6] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yuxin Pan, Giancarlo Baldan, and Oscar Beijbom, nuScenes: A multimodal dataset for autonomous driving. 2020. 3, 7,
[7] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijhom, and Sammy Omari. nuplan: closed-loop mi-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106. 11810, 2021. 3
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset.
2017. 25
[9] Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01399 2024. 18
[10] Kai Chen, Enze Xie, Zhe Chen, Langing Hong, Zhenguo Li, and Din Yeung. Integrating geometric control into text-to-ine tot-tto-mmae diffusion models for high-quaitty detection data generation via text prompt. arXiv preprint arXiv:2306. 046077 2023. 2
[11] Jiaxin Cheng, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, and Mu Li. Layoutdiffuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint arXiv:2302.08908, 2023. 10
[12] O Contributors. Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving, 2023. 23
[13] Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3223,, 2016. 20
[14] Boyang Deng, Richard Tucker, Zhengqi Li, Lconidas Guibas, Noah Snavely, and Gordon Wetzstein. Streetscapes: Large-scale consistent street view generation using autoregressive video diffusion. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024. 18
[15] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advunces in neural information processing systems, 34:8780-8794, 2021. 10
[16] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pages 1-16. PMLR, 2017. 2. 10, 27
11

---

[17] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum. Josh Tenenbaum, Dale Schuurmans. and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances Neural Information Processing Systems, 36, 2024.
[18] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12888 2021.
[19] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijic Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, al. LLaMA-Adapier v2: Parameter--fficient visual instruction model. arXiv preprint arXiv:2304.5501
2023. 3,
[20] Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02600 2023. 2, 7, 9, 10, 18, 27
[21] Shenyuan Gao, Jiazhi Yang. Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint (rxXi:2440..73999 2024. 10
[22] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. The kitti vision benchmark suite. URL http://www. cylibs. net/datassts//iiri, 1(5):1113, 2015. 3
[23] Yuwei Guo, Ceyuan Yang, Anyi Rao. Yaohui Wang, Yu Qiao. Dahua Lin. and Bo Dai. Animated- iff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint S.Ki::3307.04725, 2023. 10, 17
[24] Yingging He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. 2022. 10
[25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. 2017. 25
[26] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 2021. 25
[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information proessing systems, 13:6840-68511 2020.
[28] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev. Alex Kendall, Jamie Shotton. and Gianluea Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint
2023. 10
[29] Fan Jia, Weixin Mao. Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving. arXiv preprint arXiv:2311.13549, 2023.
[30] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint
2017. 23
[31] Jinkyu Kim, Teruhisa MIsu, Yi-Ting Chen, Ashish Tawari, and John Canny. Grounding human-to-vehicle advice for self-driving vehicles. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10591-10599, 2019.
[32] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5820-5829, 2021. 2, 26
[33] Yeongmin Kim, Byeonghu Na, Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, and I1-Chul Moon. Training unbiased diffusion models from biased dataset. arXiv preprint arXiv:2403.011 189, 2024. 6
[34] Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Scung Wook Kim, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Dreamteacher: Pretraining image backbones with deep generative models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16698-16708, 2023. 17
[35] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping anguage-image pre- training with frozen image encoders and large language models. 2023. 18
12

---

[36] Quanyi Li, Zhenghao Peng. Lan Feng. Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Iransactions on pattern analysis and machine intelligence, 45()):361-3345,, 2022. 2, 4, 10, 21, 22, 27
[37] Quanyi Li, Zhenghao Peng, Lan Feng, Chenda Duan, Wenjie Mo, Bolei Zhou, al. Scenarionet: Open- source platform for large-scale traffic scenario simulation and modeling. arXiv preprint arXiv:2306. 12241, 2023. 3, 21
[38] Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusion: Layout- view driving seene vidco generation with latent diffusion model. arXiv preprint arXiv:23 2023, 2, 7, 10, 18, 26
[39] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Procedingss of the IERF/CVF Conference on Computer Vision and Pattern Recognition, pages 2251 1-22521. 2023. 10
[40] Tingting Liang. Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang. Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust tidar-cament fusion framework. Advances in Neural Information Procesiing Systems, 35:1 15:10421-,,434, 2022. 7,9, 26
[41] Quan Liu, Yunsong Zhou, Hongzi Zhu, Shan Chang, and Minyi Guo. Apr: online distant point cloud registration through aggregated point cloud reconstruction. arXiv preprint arXiv:2305.0289) 2023. 21
[42] Quan Liu, Hongzi Zhu, Yunsong Zhou, Hongyang Li, Shan Chang. and Minyi Guo. features for distant point cloud registration. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 18215-18225, 2023. 21
invariant
[43] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018. 24
[44] Jiachen Lu. Ze Huang. Jiahui Zhang, Zeyu Yang, and Li Zhang. Wovogen: World volume aware diffusion for controllable multi-camera driving scene generation. arXiv preprint 2r 2023. 2, 10
[45] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations, arXiv preprint arXiv:2108.01073. 2021. 6, 10
[46] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023. 18
[47] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-iman diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, pages -304, 2024. 10
[48] Koustav Mullick, Harshil Jain, Sanchit Gupta, and Amit Arvind Kale. Domain adaptation driving datasets for real-workd autonomous driving. arXiv preprint arXiv:2302.04149 2023. 5
of synthetic
[49] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv: 12. 12. 2021, 10
[50] Zhenghao Peng. Wenjie Mo, Chenda Duan, Quanyi Li, and Bolei Zhou. Reward-free policy learning through active human involvement. 2022. 17
[51] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023. 10
[52] Xiaojuan Qi, Zhengzhe Liu, Qifeng Chen, and Jiaya Jia. 3d motion decomposition for rgbd future dynamic scene synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7673 7682, 2019. 2
[53] Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu- Yu-Gang Jiang. NuScenes- QA: multi-modal visual question answering benehmark for autonomous driving scenario. arXiv preprint Xiv: 14836, 2023. 22
[54] Can Qin, Shu Zhang, Ning Yu. Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang. Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unitied diffusion model for controllable visual generation in the wild, arXiv preprint arXiv:2305.11147. 2023. 10
13

---

[55] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021. 6
[56] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. 2020. 3
[57] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022. 10
[58] Stephan R Richter, Vibhav Vinect, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 102-118. Springer, 2016. 2, 10
[59] Stephan R Richter, Hassan Abu AlHaija, and Vladlen Koltun. Enhancing photorealism enhancement. JEEE Transactions on Puttern Andlyis and Machine Intelligence, 45(2):1700-1711 2022. 10
[60] Robin Rombach, Andreas Blattmann, Dominik Lorenz. Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022. 5, 17
[61] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3234-3243, 2016. 2, 10
[62] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding, Advances in neural information processing systems, 35:36479-36494, 2022. 10, 17
[63] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.0055: 00512, 2022. 18
[64] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025500 2020. 25
[65] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446 2454, 2020. 3, 4, 10, 21, 22
[66] Tao Sun, Mattia Segu, Janis Postels, Yuxuan Wang, Lue Van Gool, Bernt Schiele, Federico Tombari, and Fisher Yu. Shift: a synthetic driving dataset for continuous multi-tasi task domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 21372221382, 2022. 2,
10
[67] Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. fiew image generation from sird's view layout. IEEE Robotics and Automation Letters, 2024. 2, 7, 9. 10
[68] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models video: new metric & challenges. arXiv preprint arXiv: 1812:1112. 2018. 25
[69] Patrick Von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models, 2022. 17,35
[70] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards real- world- world models for autonomous driving. arXiv preprint arXiv:2009009777, 2023. 2, 10,
26
[71] Yuqi Wang. Jiawei Hc, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. arXiv preprint arXiv:2311.17918 2023. 2
[72] Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang. Chong Luo, Chi Zhang, Tiancai Wang. Xiaoyan Sun, and Xiangyu Zhang, Panacea: Panoramic and controllable video generation for autonomous driving. arXiv preprint 211.12231,,16813, 2023. 2, 7, 10, 18
14

---

[73]
Xinshuo Weng, Yunze Man, Jinhyung Park, Ye Yuan, Matthew O'Toole, and Kris M Kitani. All-in-one drive: A comprehensive perception dataset with high-density long-range point clouds. 2023. 10
[74] Benjamin Wilson, William Qi Tanmay Agarwal, John Lambert, Jagject Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, ct al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint (xXix:2301.0049) 2023. 3
[75] Weijia Wu, Yuzhong Zhao. Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems, 36, 2024." 2
[76] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12099, 2021. 3, 20, 21
[77] Ning Xu, Linjie Yang, Yuchen Fan, Dingeheng Yue, Yuchen Liang, Jianchao Yang. and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint 1rXiv:180900333 2018. 23
[78] Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia 7eng. Ping Luo, et al. Generalized predictive model for autonomous driving. arXiv preprint arXiv:2403.09M 2024. 2,3, 9, 10, 24, 26, 30
[79] Kairui Yang. Enhui Ma, Jibin Peng. Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via hev sketch layout. arXiv preprint arXiv:2308.011661 2023. 2, 7, 10
[80] Linrui Zhang, Zhenghao Peng. Quanyi Li, and Bolei Zhou. Cat: Closed-loop adversarial training for safe end driving. In 7th Annual Conference on Robot Learning, 2023. 2, 3, 4, 10, 17, 21, 22
[81] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 2, 7,9, 9, 10
[82] Qihang Zhang. Zhenghao Pong, and Bolci Zhou. Learning to drive by watching youtube vidcos: Action- conditioned contrastive policy pretraining. In European Conference on Computer Vision, pages 111-128. Springer, 2022. 23
[83] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024. 10
[84] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor-comrector framework for fast sampling of diffusion models. Advances in Neural Information Procexsing Systems, 36, 2024. 18
[85] Yunsong Zhou, Yuan He, Hongzi Zhu, Cheng Wang. Hongyang Li, and Qinhong Jiang, Monocular 3d object detection: An extrinsic parameter free approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7556-7566. 2021.
[86] Yunsong Zhou, Yuan Hc, Hongzi Zhu, Cheng Wang, Hongyang Li, and Qinhong Jiang. Monoef: Extrinsic parameter free monocular 3d object detection. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(12):10144-0012
2021. 25
[87] Yunsong Zhou, Hongzi Zhu, Chunqin Li, Tiankai Cui, Shan Chang, and Minyi Guo. Tempnet: Online semantic segmentation on large-scale point cloud series. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7118-7127, 2021. 21
[88] Yunsong Zhou, Quan Liu, Hongzi Zhu, Yunzhe Li. Shan Chang, and Minyi Guo. Mogde: Boosting mobile monocular 3d object detection with ground depth estimation. Advances in Neural Information Processing Systems, 35:2033-208 2045, 2022. 25
[89] Yunsong Zhou, Hongzi Zhu, Quan Liu, Shan Chang, and Minyi Guo, Monoatt: Online monocular 3d object detection with adaptive token transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17493-17503, 2023. 25
[90] Yunsong 7hou, Linyan Huang, Qingwen Bu, Jia Zeng. Tianyu I.i, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied understanding of driving scenarios. arXiv preprint 240453340. 2024. 23
[91] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang. Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. In ECCV, 2022. 23
15

---

Appendix
Discussions
17
DIVA Dataset
18
B.1
DIVA-Real
18
B.1.1 Data Collection and Cleaning
18
B.1.2
Multimodal Annotation
18
B.1.3
Appearance Diversity Highlights
20
B.2
DIVA-Sim
B.2.1 Data Collection
B.2.2
Layout Diversity Highlights
21
B.3 Extensions on Public Dataset
B.4 License and Privacy Considerations
22
C
Implementation Details of SimGen
23
C.1 Empirical Study
23
C.1.1 Cascade Diffusion Scheme
23
C.1.2 Unified Adapter
24
C.2 Model Design
24
C,3 Training Details
24
C,4 Sampling Details
25
Experiments
25
D.1 Metrics
25
D.2 More Ablations
26
D.3 Qualitative Results
27
D.4 Failure Cases
27
16

---

A Discussions
SimGen project page provides links to the YouTube videos (DIVA-Real.Video_LLii used in DIVA-Real, as well as digital twins of nuScenes dataset (DIVA-Sim mwic__iigiial_tiiiiiiii and safety-critical video clips (DIVA-Sim www.-aaaaattttiiseeeee Deciv_do_ee__iideoss) included in DIVA-Sim.
To better understand our work, we supplement with the following question-anssering.
Q1. What makes SimGen stand out compared to pixel-to-pixé transformation models?
Recent GAN-based and Diffusion-based works in image transformation can generate images that are controllable based on specific conditions [23, 62]. Yet, their limitations lie in the fact that the content they generate is strictly tethered these input conditions. If these conditions, derived directly from a simulator, are missing important contextual information like backgrounds and buildings, then output images may similarly lack background details Consequently, SimGen employs cascade structure, permitting the CondDiff model to conceptualize different background scenarios through text, thereby enriching the visual composition of the rendered driving scenes. Detailed analysis is shown in Appendix C.I.
Q2. What is the criteria to demonstrate good generalization and diversity of your model? How much data do we need?
Currently, it's challenging to define specific standard to assess the diversity and generalization abilities of the models, as quality evaluation is subjective and fair comparison can be difficult. However, by utilizing publiely available data, have found that scaling up the data size proves beneficial for the zero-shot generation on novel scenarios. Equally important to note is that our approach is easily scalable, and by leveraging massive in-the-wild data, we offer continuing opportunity to strengthen its generalization capabilities.
Q3. What is the definition of safety-critical scenarios and how to ensure they are realistic and feasible?
safety-critical scenario is a situation where one or more vehicles collide with the ego vehicle, which rare collect in from logged scenarios to ensure reality and feasibility, which uses a data-driven motion prediction model that predicts several modes of possible trajectories of each traffic vehicle. Please refer back to [80] for a detailed description of safety- cafety-critical scenarios.
datasets like Waymo. We utilize CAT [80] to generate risky behaviors
Q4. Broader impact. What are potential applications and future directions with the provided DIVA data and the SimGen model, for both academia and industry?
Datasets. DIVA collects massive data from YouTube and simulators, significantly enhancing the appearance and layout diversity driving video clips. This provides the community with extensive high-quality resources for exploring open avenues in autonomous driving and Sim2Real research.
Models. Beyond data augmentation, we hope our model can also benefit the community by enabling wider applications. In this work, we demonstrate SimGen's capability as closed-loop data generator. holds promise to adapt to downstream tasks like closed-loop evaluation of autonomous driving agents [50], which is showcased in Appendix D.2. To boost deployment efficiency, distilling knowledge from the generative model worth exploring [34]. Besides,
scene generation also provides opportunities to achieve physically grounded real-world generation [2, 17]. Please note that our model will be publicly released to benefit the community and can be further fine-tuned flexibly according to custom data within the industry.
Negative societal impacts. The potential downside of SimGen could be its unintended use in generating counterfeit driving scenarios. Our code includes the Diffusers [69] safety checker to screen for NSFW outputs. Besides, we plan regulate the effective use of the model and mitigate possible societal impacts through gated model releases and monitoring mechanisms for misuse.
Q5. Limitations. What are the issues with current designs, and corresponding preliminary solutions?
1) Panoramic image generation is necessary for current Bird's Eye View perception models in autonomous driving. Yet, to utilize scaled web data, which consists of front-facing single-camera footage, SimGen does not engage multi-view image generation. This may limit the application of SimGen in wolldwrrdd deployments. 2) We chose SD-2.1 [60] as our base diffusion model. inheriting
17

---

its advantages of high visual quality and better rendering capabilities of the text encoder. On the other hand, we noted that it has slow sampling speed and high computation costs. Our model does indeed suffer from this issue.
However, pioneering work exploring how introduce simulators into generative models for diverse driving scene generation, the primary focus of this work is the simulator-conditione alization ability across unseen driving scenarios rather than multi-view designs and computational overhead. Future work might include trying cross-frame attentions [20, 38, 72, 14], faster sampling methods [9, 46, 63, 84], and transferring our general method more efficient diffusion models.
gener-
B DIVA Dataset
Our dataset, DIVA, contains 147.5 hours of driving video along with diverse multimodal conditions, including text, segmentation, depth, and virtual images. In this section, we detail the YouTube and simulator video collection process, annotation method, more examples, and analysis to illustrate the diversity of the DIVA dataset.
B.1 DIVA-Real
B.1.1 Data Collection and Cleaning
Data preparation. We first searched for driving videos on YouTube using keywords such driving videos, 4X, and HD. We then identified a selection of YouTubers who consistently upload high-quality driving videos. We further inspected the quality of these videos in terms of resolution, resulting in 130 high-quality front-view driving videos, including Barcelona 4K Driving Downtow,, Cairo 4K Pyramid Expressway Sunrise, Las Vegas 4K Sunset Drive and Istanbul 4X Night Drive Turkey. We used videos from 10 selected clips as the validation set and the other videos for training. The diversity of DIVA-Real is illustrated in Fig. 7. Additionally, cut off the first and last 30 seconds of each video to remove any solicitations or other edited footage.
Data format. We segmented the video data into images at a rate of 10Hz, with a resolution of 1080p (1960x 1080) for each image.
Data cleaning. To automate the process of filtering out low-quality images from the dataset, we utilize vision language model (VLM), LLaMA-Adapter V2 [19]. First, we group the images at rate of 2Hz and randomly select one image from each group to feed into the VLM. We provide checklist, asking the VLM sequenced questions about the image quality. The checklist includes items such non-front view, video transition, black screens, etc. The VLM then uses its acquired world knowledge to infer and assist in automatically eliminating low-quality images. The checklist organized as a set of text prompts given to the VLM, specified below.
Text Prompt Examples:
"Is the driving scenario image presented from POV other perspective rather than front view?".
" "Does the driving scenario image exhibit gradual transition due video transition?",
"Is the image almost completely black, distinguishable from those depicting night driving?",
"Is this image excessively blurry, rendering any foreground object information indistinguish- able?",
"Does this image feature subtitles, distinct from signs or markers in driving scenarios?",
"Does this image depict scenic view or bird's bird's-eye perspective, distinct from front-view driving footage?", etc.
B.1.2 Multimodal Annotation
Our OpenDV-Wild features three types of annotations: text, depth, and semantics. We leverage the established BI.IP-fllant5 [35] to describe each frame's main objects or scenarios with the following prompt.
18

---

Figure 7: Various video samples from Deva-Real. Due to space limitations, we only showcase certain frames from the videos. covers a wide range of diversity across multiple axes, including geographical location, traffic scenarios, time periods, weather conditions, elc.
19

---

Table Location distribution of nuScemes and DIVA-Real.
Dataset
North America
South America
Europe
Asian
Africa
nuScenes
44.1%
0.0% 8.5%
0.0%
55.9%
0.0%
DIVA-Real
56.9%
16.9%
14.6%
3.1%
Table 8: Time period distribution of nuScenes and DIVA-
Dataset
Daytime
Dawn
Dusk
Nighttime
nuScenes 88.4%
0.0%
0.0%
11.6%
DeVA-Real
55.8%
16.3%
10.1%
17.8%
Text Prompt Example: "Question: Describe the image of a driving scenario concisely. Answer:".
We present content query to the VLM according to each example's 's text prompts. If the VLM responds with no to all the queries, then the set of images can successfully pass the review.
For depth and semantic segmentation, we employ pre-trained ZoFDepth [3] and Segformer [76] for automated label generation. Segformer is previously trained on the CityScapes dataset [13]. The examples of text annotations are shown as follows.
Text Annotation Examples:
"A car is driving down street in Rio De Janeiro." motorcyclist on road with a speed limit sign.",
" "A car driving down street in Madrid, Spain.", ",
"A view of city street with tall buildings and TV tower in the background.",
"A snowy road with trees on both sides of the road and red car is driving.",
" A car driving on highway at dusk in Las Vegas, Nevada.", etc.
B.1.3 Appearance Diversity Highlights
Diversity over prior datasets. Beyond its large data scale, our dataset outshines competitors in terms of appearance diversity. YouTube, known for its diverse content, is regarded as global mosaic and crucial source of data. DIVA- -eal leverages this by comprising 120 hours of publicly available videos from over 71 citics across more than 19 countries. Our dataset exhibits a globe-wise geographic distribution compared other open datasets collected in limited regions. On top of that, DIVA-Real covers a rich variety of driving scenarios, including bridges, bays, wilderness, deserts, dusk, fog. and more. Unlike datasets restricted dull scenes, dataset empowers models to capture wealth of visual appearance diversity. Lastly, ours boasts a richness annotations on par with other datasets.
In this section, we provide detailed data analysis about the diversity of DIVA-Real. For simplicity's sake, we assume that all clips within a video are shot at same location and time, with any single frame from segment representative of the geographical location, lighting conditions, weather, and other informational aspects concerning the video. Thus, we manually review each video's 's title and random frame from the video and assess its geographic location, time, and weather conditions, among other things, for statistical analysis. The following diversity analysis has been derived from this process in three aspects.
Location distribution.
According to statistical results, YouTube videos are derived from 71 cities in 19 countries, covering a much larger area than any existing public driving dataset, as shown in Tab. 7. For example, the most popular regions, DeVl-Real includes hours video data in the United States, covering cities like Los Angeles, New York, Las Vegas, Miami, Boston, Atlanta, New Orleans, etc., and encompasses geographical areas such as urban, rural, coastal, wilderness, mountainous, and port regions.
20

---

Table Weather distribution of nuScenes and DeVA-Real.
Dataset
Normal
Rainy
Cloudy
Foggy
Snowy
nuScenes
80.5%
19.5%
0.0%
0.0%
0.0%
DIVA-Real
58.2%
1.0%
28.6%
2.1%
10.2%
Table 10: Layout distribution of nuScenes and DIVA-Sim.
Left
Right
Left Lane
Right Lane
Intersection Passing
Dataset
Forward
U-Tum
Stop
Turn
Turn
Change
Change
nuScenes
47.1%
18.0%
10.2%
5.0%
2.5%
13. 13.1%
0.0%
4.1 %
DIVA-Sim
36.2%
14.3%
10.0%
10.7%
17.4%
6.6%
1.2%
3.6%
Time period and weather variance. The DIVA-Real dataset also includes variety of times and weather conditions. As shown Tab. 8, in addition to daytime, the dataset covers a considerable proportion of dawn, dusk, and nighttime scenarios. Tab. 9 presents the weather distribution in the dataset, including rainy, cloudy, foggy, and snowy conditions. These diverse times and weather conditions ensure variety of appearances.
Corner cases. YouTube videos also include extreme cases and safety-critical scenarios. Fig. presents several special cases from DIVA-Real such as crowded pedestrian-fille intersections (fourth one in the first row), roads with a lot of parked vehicles (fifth one in the first row), a country road in the sunset (third one in the fifth row), and passing under an overhead structure with limited light (fifth one in the seventh row).
B.2
DIVA-Sim
B.2.1 Data Collection
The DIVA-Sim dataset is collected through the MetaDrive simulator [36]. DIVA-Sim accumulated total of 27.5 hours of virtual driving data, including 5.5 hours from the digital twins of the nuScenes dataset [6] via ScenarioNet [37], and 22 hours dangerous driving scenarios collected initially based on the Waymo Open dataset [65] by adversarial interventions [80]. For data from nuScenes and Waymo Open, each vidco lasts seconds and 8 seconds, respectively.
For each scenario, DIVA-Sim provides a variety of labels, including rendered RGB, depth, segmen- tation, instance map, and top-down view. All camera outputs are synthesized using the OpenGL rendering backend from the Panda3D game engine, allowing us to incorporate depth map and semantic colormap similar ZoeDepth [3] and Segformer [76].
Data format. We segmented the video data into images at rate of 10Hz, with a resolution of x960 1080 for each image.
B.2.2 Layout Diversity Highlights
We randomly sample 500 safety-critical videos generated based Waymo Open dataset [65] and manually reviewed the contents of the top-down view each video, collecting the data shown in Tab. 10. Beyond forwarding, turning, and stopping, DIVA-Sim also covers cases like changing lanes, passing through intersections, and making U-turns. Fig. 8 visually displays the top-down views of various dangerous driving scenarios, including collisions, quick stops, and reckless merging. These illustrate the diversity of layouts in DIVA-Sim.
B.3 Extensions Public Dataset
In addition to collecting YouTube data and simulation data, we also annotate data of public datasets promote research from simulation reality and enable fair comparisons.
nuScenes dataset. The nuScenes dataset [6] is a public driving dataset that includes 1000 scenes from Boston and Singapore for diverse driving tasks [87, 42, 41]. Each scene comprises 20-second
21

---

Figure 8: Various safety-criticat layouts from DIVA-Sim. The yellow rectangular represents the ego-car and the other cars interacting with it. All scenarios are initialized by the Waymo Open dataset [65] and generated by adversarial interactions [80] within simulators.
video, approximately 40 frames. It provides 700 training scenes, 150 validation scenes, and 150 test scenes. Similarly, we utilize BLIP2-flant5, ZoeDepth, and Segformer to provide textual, depth, and semantic labels for this dataset.
B.4 License and Privacy Considerations
All the data is under the CC BY-NC-SA 4.0 license. Other datasets (including nuScenes [53], Waymo Open [65], Metadrive [36]) inherit their own distribution licenses.
http://ccctiiccoommoooooolleeeee
22

---

1a Monotony categories
1.b Variation posttions
1.c Occlusion
1. Mismatches
2&3. Inherent flaws of 3D models and mising backgrounds
Figure 9: Gaps between conditions in simulators and reality. In each group of images, the first
row represents the conditions in the simulator, and the second row represents the real conditions.
We place high value on license and privacy protection, following the precedent from YouTube- 8M
[1], YouTube- VOS [77], AOC [82], CelebV-HQ [91], ELM [90], OpenScene [12], and Kinetics [30], etc. For videos from YouTube, permission to access the video content is received through Creative Commons license. Besides, we skip channel-related content at the beginning and end of the videos during data processing to ensure we do not infringe upon the rights of logos, channel owner information, or other copyrighted materials. We do not provide video content; users are redirected to original YouTube videos via a link. The platform safeguards personal info with encryption, access limits, and identity checks to prevent unauthorized video access. We will credit the source, provide link to the license, and state that no modifications have been made to the video itself, and the data will be used for commercial purposes. All the data we obtain complies with regulations and YouTube's 's Privacy Policy. In addition, we comply with any limitations required by applicable law and any requests submitted by users. For instance, users may have the right to view, correct, and delete personal information we possess about them, such as deleting text labels, unlinking videos, and de-identifying data.
C Implementation Details of SimGen
C.1 Empirical Study
C.1.1 Cascade Diffusion Scheme
It's worth noting that, despite our efforts to replicate real scenes closely possible in the simulator, gaps are inevitably present. Gaps between the conditions in simulators and reality can affect the accuracy of model training. These gaps primarily originate from: 1) mismatches, 2) inherent flaws of 3D models, and 3) missing backgrounds. In Fig. 9, we provide some visualizations to illustrate these gaps.
The mismatch between simulated and real images can exist in multiple aspects. Firstly, given that the models within the simulator are finite, it is unrealistic to accurately represent wide variety of
23

---

object categories through it. As shown in example l.a, for excavators, the simulator uses dump trucks based on size, resulting in some mismatches in object shape. Secondly, slight differences between the camera positions in the simulator and real datasets can cause objects to be displaced in the simulator, as in example I.b, where Ay is larger than Ay2- Thirdly, the simulator lacks physical entities such as buildings or trees, making it difficult to replicate scenarios where vehicles are obstructed by the environment, as shown in example I.c. Beyond mismatches, inherent flaws may also exist within the simulator models. Although the simulator can reflect the real size of vehicles, achieving perfect consistency in shape is still challenging. Furthermore, the transparency of the model's s window parts can lead to discrepancies between depth and semantic segmentation results in those areas. Finally, the simulator lacks elements such as buildings, trees, and street lamps, which are essential conditions for controlling image generation.
Through the above analysis and visualization results, combined with the outcomes of our ablation study, we infer that real depth and semantic segmentation images are more suited to control real image generation than simulated equivalents. Therefore, SimGen's 's cascade diffusion network is necessary, as it allows the decoupling of the introduction of SimCond in the CondDiff' module from the controlled image generation in the ImgDiff module.
It's worth noting that this gap doesn't 't mean simulated conditions can't be used for training image generation models. Instead, suggests that employing real depth and semantics is better choice, and better aligned for joint training with data from YouTube. Indeed, the simulator's 's ExtraCond can also contribute the model's accuracy, For instance, the instance map can indicate the object count and occlusion relationships in the scene to the model, and the top-down view can provide spatial location information. These insights can guide the use of unified adapter to merge multi-modal conditions.
C.1.2 Unified Adapter
SimGen employs unified adapter to address the two obstacles in multi-moda condition conflicts: Modal Discrepancy and Condition Disparity. Modal discrepancy refers to the inconsistent number of modalities between data from nuScenes and YouTube (the latter lacks ExtraCond), which might lead the model to establish a statistical shortcut, such as outputting nuScenes-style images when ExtraCond is present and YouTubeestyyle images when it's absent. This shortcut can significantly impact the model's 's instruct-following ability and diversity at inference time. On the other hand, condition disparity refers to the lack of background information in the ExtraCond condition, which can result in conflicting control information with RealCond. Thus, our proposed solution is to use an adapter to merge various modalities into unified control feature, employing a mask during the fusion process to eliminate conflicts arising from absent background information ExtraCond.
C.2 Model Design
Realismi-cottrollaabittty trade-off. Apart from the discretization steps of the CondDiff solver, the critical hyperparameter for sim-to-real transformation is t,, the starting time of the image synthesis process in the reverse SDE. We notice that with a fixed CondDiff model, there's a trade-off between Realism and Controllability when choosing different L, values. Smaller l, values lead to fewer denoising steps, giving SimCond more control over image generation but potentially compromising realism. Generally, we find ta E [0.4, 0.65] to work well for the foreground, and we ultimately select te as 0.5 for foregrounds.
Extension on Video Generation Having acquired the single-frame variant, we lock the original blocks within the denoising UNet and intersperse them with temporal reasoning blocks, mirroring the strategy of GenAD [78], thereby facilitating video sequence modeling.
C.3 Training Details
SimGen is trained in two modules: CondDiff, which converts simulated conditions to real ones, and ImgDiff, which generates images from multimodal conditions. In the first stage, we fine-tun tne-tune the pre-trained SD-2.1-V on per-image denoising with 1.1B trainable parameters of its denoising UNet. is trained on 4.5M text-depth-segmeentaiie for 30K iterations on 8 GPUs with a batch size of 96 with AdamW [43]. We linearly warm up the learning rate for 10 steps in the beginning, then keep constant at 10-5 The default GPUs in most of our experiments are NVIDIA Tesla A6000 devices unless otherwise specified.
pairs of DIVA-Real and nuScenes. We train the model
24

---

In the second stage, we train the model via unified adapter and ControlNet using text text-condition image pairs, lifting to generate realistic images during inference. The training data consists of DIVA and nuScenes, with conditions confined to ExtraCond and RealCond. Following the design of ControlNet, we freeze the input and middle layers of the UNet, training only the parameters of the control branch and the Adapter. This stage is trained for 50,000 iterations on 8 GPUs, with 295 batch size of 96.
For the extension of video generation, we freeze all blocks of the single-frame version and only optimize the introduced temporal reasoning blocks, resulting in 418M trainable parameters this stage. To maximize the data efficiency for constructing video clips, we take each frame of a 10Hz YouTube and nuScenes video a starting frame to form a 3s training sequence at 2Hz. The text condition is structured in the same way as the first stage, and we acquire the context from the middle frame of the sequence. SimGen is trained on 8 GPUs for 30K iterations with a total batch size of 24. The learning rate is set as 1 10-5 after 10 warm-up steps.
In both stages, the input frames are resized 256 448, and the text condition is dropped at probability Yo = 0.1 to enable classifier- guidance [26] in sampling. Both CLIP text encoders and the autoencoder are kept frozen throughout our experiments.
For effective classifier-free free guidance [26], ImgDiff random drops conditions during training at a rate of % = 0.1. Additionally, enhance the model's ' robustness by randomly masking the background real conditions with a fixed probability of % = 0.5. To address potential cumulative errors from the CondDiff process during ImgDiff denosing, introduce slice noise with a probability of %, 0.25. Slice noise entails partitioning the image into n x n. patches and randomly masking them with a probability of 7/p 0.25.
C.4 Sampling Details
Given conditions from simulators, SimGen first has reverse SDE process in CondDiff. It starts from SimCond added with standard Gaussian noises. The sampling step of this stage is 25 (1, 0.5). After that, the second sampling process is involved in Img starting with random Gaussian noises. Both sampling processes are performed by Denoising Diffusion Implicit Models (DDIM) [64]. We use 50 sampling steps and set the scale of classifier-free guidance to 9.5. The sampling speed is 1.13 seconds per step per batch. The image resolution is 256 x 448, and the video sequence is at 2Hz.
D Experiments
We conduct extensive experiments on multiple datasets to evaluate the performance of our method. For comparison convenience, we trained two models on the nuScenes and DIVA datasets, respectively, namely SimGen-nuSe and SimGen, adopting the same training strategy.
D.1 Metrics
One of the roles of synthesized images is to augment existing perception models of autonomous driving [85, 86, 89, 88]. We use various metrics in multiple aspects for quantitative evaluation. For generation quality metries, we use Fréchet Inception Distance (FID) [25] and Fréchet Video Distance (FVD) [68]. For generation diversity, the pixel diversity Dpis metric is leveraged. The controllability of the model is reflected by the alignment between the generated image and the conditioned BEV sequences. For the video generation task, all frames are at 2Hz. The specific metrics are described as follows.
FID: It evaluates the generation quality of images, which video frames in our experiments, by measuring the distribution distance of features between the predictions and original frames in the dataset. The features are extracted by pre-trained Inception model. For quantitative comparison on nuScenes, FID is evaluated on 6019 generated frames and ground-truth frames. For experiments on YouTube, FID calculated on 18000 frames from both generation and the dataset.
FVD: It measures the semantic similarity between real and synthesized videos with a pre-trained I3D action classification model [8] as the feature extractor. We evaluate 4369 video clips from nuScenes and 3000 video clips from YouTube.
Dpix: To gauge the diversity of the generated data, we compute the standard deviation of the pixel values in the generated images. A higher value indicates greater diversity of colors in the generated
25

---

Bad exampies ww/o CondDift)
Good examples /w/ CondDiff
Figure 10: Comparison of proposed cascade diffusion model (blue boxes) naïve approaches (gray boxes).
Conditions
Moutains
Busy street London
City street
Crossood
Street night
Small town
Figure 11:
Visualization of text-grounded Sim-to-Real condition transformation.
data. The conditions for the evaluation come from the nuScenes validation set, and text prompts are collected from both the nuScnes validation set and randomly selected 18,000 frames of YouTube data. For each condition, we randomly select a text from the collected prompts as input, and the model generates the corresponding image. To reduce randomness, we test each model 3 times with the same random seed and take the average.
Controllability Metries:
To affirm the consistency between the generated images and original data layout, we employ metries such as Average Precision (AP), and Mean Intersection over Union (mloU) to evaluate the perception performance on the nuScenes dataset. Our evaluation comprises two aspects: firstly, we use pre-trained perception models compare the validating performance of generated data with real data. Secondly, we explore the potential of employing augmented training sets as a strategy for performance improvement.
We adopt BEVFusion [40], state-of-the-ar perception method, as our primary evaluation tool. Specifically, we utilize BEVFusion implementation that only incorporates a front view, masking other camera perspectives and ground truth during evaluation.
D.2 More Ablations
Results of video generation. It's worth noting that the innovation of SimGen doesn't focus video generation. However, high-quality image generation implies the po-
Table 11: Quality of video generation.
Method
FVDJ
tential for video generation, which is crucial for interac-
DriveGAN [32]
502
tive scenario generation and closed-loop planning. We made a preliminary attempt at video generation based GenAD [78]. Tab. 11 compares SimGen with other video generation models. Thanks to its commendable image generation quality, SimGen achieves performance that is on par with other models.
DriveDreamer [70]
452
Driving [38]
332
GenAD [78]
184
SimGen
271
Effectiveness Sim2Real condition transformation. validate the effectiveness of the cascade diffusion model, we display set of comparative images in Fig. 10. The blue boxes represent the cascade diffusion structure we adopt, it transforms SimCond from the simulator into RealCond in CondDiff, and then generates realistic images through the ImgDiff model. The grey boxes signify that
26

---

Snowy road
Sudden braking
Keeping safe distance
uiod (oret)
Collsion
K Moving to the right slowly
Figure 12: Preliminary attempt closed-loop evaluation. For two scenarios, the IDM behavior [36] (gray boxes) leads to hazardous driving situations, while manual control (blue boxes) adopts measures evade risks.
we removed CondDiff, and ImgDiff directly generates images using SimCond. It observable that removing CondDiff causes the generative model to introduce certain distortions from the simulator conditions into the produced images, seen in the overly narrow wheels and deformed rear part shown in the left set of pictures. The introduction of CondDiff transforms these distortions into realistic conditions after resampling, thus greatly enhancing image generation quality. Fig. 11 further demonstrates that CondDiff can transform depth and segmentation from the simulator into real ones based on different texts.
Closed-loop evaluation. We further explore applying our dimulator-conditione generative models to the closed-loop evaluation in Fig. 12. The evaluation focuses two driving behaviors, namely IDM [36] (gray boxes) and manual control (blue boxes) in different scenarios. IDM could lead to risks like sudden braking or collision in these cases. Conversely, manual control promotes safety by maintaining distance and slowing down. The video data is generated by SimGen, using conditions pulled from simulator interactions, with one-second frame interval.
Generalization on novel simulators. As CondDiff can convert simulated conditions into real condi- tions in an adaptation-frre approach. SimGen possesses the ability to perform zero-shot generalization to other simulators. In Fig. 13, we exhibit a case study of generating realistic images using depth and semantic segmentation conditions provided by CARLA [16]. This provides the possibility for SimGen to utilize and integrate by various simulation platforms generate diverse driving scenarios.
the diverse layouts, driving policies, and physical engines provided
D.3 Qualitative Results
Text-grounded image generation. SimGen is a capable text-to-image diffusion model for driving scenarios, especially when examining text controllability compared to other works. In Fig. 14, demonstrate SimGen's 's exceptional ability to generate images from different text prompts. Thanks to the relatively simple simulator conditions and comprehensive DIVA dataset, the text prompt can effectively influence the resulting image, even changing the surrounding building and background with reference specific cities. Whereas as other diffusion-basee generative models struggle to generate images with characteristics that were not originally present in nuScenes
[20], like unseen weather background settings, Simgen's 's text-grounding can influence both foreground objects like cars and match the background to cities not present in nuScenes.
conullato-conditione image generation. Fig. 15 displays additional examples of SimGen generat- ing diverse images based on conditions provided by the simulator. The far-left column presents the mamulator-renderee RGB, while the right side shows images generated by SimGen following different text prompts. This further validates SimGen's 's potent ability to adhere to the simulator conditions while maintaining rich appearance diversity.
Video generation. Our preliminary SimGen video generation model able maintain the ability to create driving images with a wide range of backgrounds while also incorporating temporal consistency as shown in Fig. 16.
D.4 Failure Cases
We show some failure cases of SimGen in Fig. 17. The first column represents the conditions from the simulator, with each generated image accompanied by a text prompt. The failure cases of SimGen are included as follows: 1) Text comprehension error: as in the first image, where the adjective "green" is not assigned to any discernible "traffic light" but instead as a vehicle color. 2) Condition conflict: the second image provides a text prompt of a tow truck, but it's challenging for the model to generate such vehicle based on the shape of a sedan. 3) Background subsumption: the third image demonstrates case where the background subsumes the smaller car. 4) Generation instability: in the
27

---

last image, SimGen occasionally produces distortion and blur in background generation, likely due to cumulative model error and overabundance of nighttime images DIVA.
image
Tunnel
SPoot broes
Cloudy down
Country rood
Snowy shoel
City street of right
Figure 13: Case study of zero-shot image generation CARLA. We randomly select scenario CARLA, for which SimGen generates various driving scenarios through the conditions (depth and segmentation) produced by the simulator and different textual prompts.
City al right
Highway with cily Skylie
Busy city street right
City street of dluk
Stue van in Landon
Snowyy mountain rood
los Angeles, Calfomo
Santiago. Chile
Double decker bus
Downlown San francisco
Motorcyclst
Cedarwood
Tra/fic Jomn right
Cairo. Egypt
Troffic jam
Wet city street
Mami
White truck
Yellow toxt
New York city
City Parkod bike
Stopped red light
Strect
Slopped tralicc light
Ari intersection
London
Los Angeles at dusk
Figure 14: Text-grounded image generation. Each image is generated by SimGen using a randomly selected text prompt and simulator conditions. The rich appearance diversity is reflected through the wide range of generated content.
28

---

Figure 15: Diverse generated images from simulator scenarios From the same original simulated driving scenario (left column), we show a diverse range of generated images (columns through 7). SimGen is capable of generating driving scenes in wide variety of settings based the same simulator conditions.
29

---

Figure 16: Preliminary attempt at video generation. Notably, SimGen is not designed for video generation. We simply follow some practices in [78] temporal consistency, and video generation will be our future work.
Condition
city strent with a green traffic light
truck drivig highway at night
highway with builings in the background
Image of a driving cown a street at dusk
Figure 17: Failure cases SimGen.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, all the claims accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we discuss the limitations in Sec. 6 and Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
31

---

This paper does not include theoretical results. We appropriately cite statements and proofs from other papers.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, details implementation and training are shown in Appendix C.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification: Our codes and data are available in https://gtt http//gttttubbb com/metadriverree SimGen, and we show full implementation details in Appendix C.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: cenollc/guidess..
/ /nips...
sssisssnn wwwssssoiiiyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, all the training and test details are shown in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: This paper does not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, this paper provides information on computer resources in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ete://neeisseeeeeeeee wwweeriss.cppuullllleeeee htte//ublccceee cettiiissCieellnennn h ines?
Answer: [Yes]
Justification: Yes, this research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we have discussion on broader impacts in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our code (when released) includes the Diffusers [69] default safety checker to screen for NSFW outputs and remove them. We recognize that proper safeguards for image generators are still an open research problem and are far from perfect, and our method will use any available safeguards when publically released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the data we use follows the CC BY-NC-SA 4.0 license, as explained in Appendix B.4.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hcoe...c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
35

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We detail our dataset in Sec. and Appendix We plan to release the dataset alongside the public release of our code, where there will be documentation provided alongside the dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not include experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
36

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "Kzc88mKksH", "reviewer": "Reviewer_iVy1", "summary": "The paper presents SimGen, a framework for generating diverse and realistic driving scenes using a cascade diffusion pipeline, aiming to generate controllable, diverse, and realistic autonomous driving images. It also introduces the DIVA dataset, including real-world and simulated data to enhance training diversity for autonomous driving tasks.", "questions": "Following the previous points:\n\n- Have you tried conventional Sim2Real methods? A more detailed discussion on why CondDiff performs better is helpful.\n\n- I’m curious how many diffusion steps (t_s) on average were needed to map from synthetic to real (CondDiff)?\n\n- Why doesn’t SimGen work as well as GenAD for video generation? Any hypotheses? \n\n- Is fig11 a failure case of Sim-to-Real transformation? The semantic map of the street at night is very different; the pedestrians got disappeared when given the \"Mountains\" text, …\n\n**Minor issues**: \n\nThe presentation needs improvement. There are ambiguities in organization of sections and some sentences like L257, L159, and L214. \n- L174, L766, L891, …“It’s” and L789 “doesn’t” “can’t”\n- L295 “singe-frame”\n- Fig3 caption: “Eventually, the text features, SimCond, and ExtraCond are inputted into the ImgDiff module” —> RealCond?", "limitations": "The authors have mentioned several limitations in their work but their discussion is somewhat general. \n\nWhile the authors mention some failure cases, there is no discussion on the possible reasons for these failures or how they could be avoided.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem is interesting and timely and the dataset could be useful for the community. Collecting videos from YouTube and the engineering efforts to create a proper dataset are valuable contributions.", "weaknesses": "While the paper tackles an intriguing problem, there are some concerns about it:\n\n- **DIVA Dataset quality**: The quality of the provided dataset is unclear. L112 states that “videos with rich annotations are collected.” What are the performance metrics for the used VLM, depth estimator, and segmentator? In other words, what is the quality level of the annotated data? Additionally, Tab1 is misleading; it should include a column indicating whether the annotations are human-labeled or pseudo-labeled. \n\n- **SimGen performance**: The performance of the SimGen framework is not clearly demonstrated. In Tab3, the performance is on par with DrivingDiffusion. If other methods were trained on DIVA, would their performance be the same as SimGen’s? For a more comprehensive comparison, I suggest training SimGen on DIVA only (not on nuScenes) and then evaluating it on nuScenes. The current improvement might be due to having more data (simple augmentation). \n\n- **The utility of the generated scenarios**:  Tab5 is a nice informative table but it shows limited utility of the generated scenarios in other tasks, especially when the simulator is trained only on nuScenes. Why don't the generated scenarios show higher improvements? Could authors report complete data for Tabs 4, 5, and 6 (for all classes)? \n\n- **Dataset diversity**:  The diversity of the dataset is unclear. While Tabs 7-10 provide some information, a quantitative comparison with other datasets is needed. The same applies to “Corner cases”—some statistics are necessary beyond the four qualitative figures provided. \n\n- **Safety-critical scenarios**: The ability to generate those is one of the main claims of the paper but what is the authors' definition of safety-critical scenarios? How can these be compared with scenarios generated by other methods or datasets? How do they ensure these scenarios are realistic / feasible?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***W1: DIVA Dataset quality.***\n\nSupp Tab. 1 includes the evaluation of the annotation quality. To evaluate the VLM, we utilize the DIVA-Real and nuScenes datasets and employ the widely used ROUGE-L metric (**84.4** and **85.2**) [1] to assess the similarity between the annotated data and pseudo-labels. The pseudo-labels are derived from GPT4V and manually checked to ensure reasonableness. For the evaluation of the depth estimator and segmentator on the nuScenes dataset, we leverage the metrics of absolute relative error (**0.118**) and mIoU (**82.4**), respectively. The results indicate that our annotated data exhibits a sound data quality level. Besides, we follow your advice to include a column indicating whether the annotations are human-labeled or pseudo-labeled in the revision.\n\n***W2: SimGen performance.***\n\nFor a fair comparison, we have set up the SimGen-nuSc within the nuScenes dataset in Tab. 3. SimGen-nuSc surpasses DrivingDiffusion in terms of image quality and diversity by **-0.3** and **+0.4** respectively under the same training data. If other methods were trained on DIVA, there might be a performance boost similar to that from SimGen-nuSc to SimGen (**+6.1**). However, methods like BEVGen and MagicDrive require explicit labels like 3D boxes and lane lines for training, preventing them from utilizing the web data. Moreover, training SimGen on DIVA only and testing on nuScenes would be unfair as it raises domain generalization issues in those works. \n\n***W3: The utility of the generated scenarios.***\n\nThe DIVA dataset is an essential part of our contribution. Aside from the improvements resulting from the framework (demonstrated by SimGen-nuSc), Tab. 5 also shows that SimGen trained on the DIVA dataset brings higher improvements and outperforms baselines by a large margin. In Supp Tab. 2-4, we report complete data for SimGen's performance in 3D object detection tasks. Tab. 3 witnesses the benefits of SimGen on all categories in nuScenes, including motorcycle (**+2.9**), cone (**+1.6**), and barrier (**+1.7**). The results indicate that SimGen can generate a diverse range of objects (**10** categories) covering common driving scenarios, thereby confirming its utility in practical applications.\n\n***W4: Dataset diversity.***\n\nIn Supp Tab. 5-6, we provide a comparison between DIVA and nuScenes. nuScenes data is only from Boston and Singapore, while DIVA includes data from South America (**8.5%**), Europe (**16.9%**), Asia (**14.6%**), and Africa (**3.1%**). In nuScenes, **88.4%** of the data is during the daytime, and **80.5%** is on normal days. In contrast, DIVA collects data at dawn (**16.3%**) and dusk (**10.1%**), covering cloudy (**28.6%**), snowy (**10.2%**), and many other weathers (**3.1%**). Also, **75.3%** of nuScenes data is keeping forward and turning, while DIVA includes lane changing (**28.1%**), intersection passing (**6.6%**), and U-turns (**1.2%**), reflecting more complex traffic layouts. The results show that DIVA has a more diverse and balanced data distribution than nuScenes. In addition, Tab. 7 reports statistics on corner cases, including **9** kinds of behaviors of other vehicles that lead to safety-critical driving scenarios. The statistics report that the proportions of crash back and cutting in are **41.2%** and **19.0%**, respectively.\n\n***W5: Safety-critical scenarios.***\n\nA safety-critical scenario is a situation where one or more vehicles collide with the ego vehicle, which is rare to collect in real-world datasets like Waymo. We utilize CAT [2] to generate risky behaviors from logged scenarios to ensure reality and feasibility, which uses a data-driven motion prediction model that predicts several modes of possible trajectories of each traffic vehicle. Please refer back to [2] for a detailed description of safety-critical scenarios. \n\n***Q1: Detailed discussion on CondDiff.***\n\nWe have tried the conventional Sim2Real method which focuses on photorealism enhancement [3]. As discussed in Appendix A and C.1, the pixel-to-pixel transformation only brings limited appearance diversity. It is like applying an image filter or style transfer to images rather than generating new content, with the outputs strictly constrained by input conditions. If input conditions derived from a simulator lack background information (e.g., trees, buildings, etc.), the output images may completely ignore generating any background content. Moreover, such a model struggles to alter visual appearances via text prompts in generating novel scenarios. \n\n***Q2: Diffusion steps for Sim2Real transformation.***\n\nAppendix C.2 shows the diffusion steps ($t_s$) for Sim2Real transformation is **0.5**.\n\n***Q3: Comparision of SimGen to GenAD.***\n\nSimGen aims for controllable image generation in driving scenes, so it is not specifically designed for video generation. We only have a preliminary attempt to achieve temporal consistency. GenAD incorporates spatiotemporal optimizations and is designed for video generation, but it has much less flexibility and controllability in content generation compared to SimGen.\n\n***Q4: Explanations on Sim2Real transformation.***\n\nFig. 11 visualizes Sim2Real transformation under varying text prompts rather than a failure case. The real conditions generated by the model may be related to the training data distribution. Despite \"street at night\" not mentioning trees, the model still envisions the presence of far trees in the scene and generates a corresponding semantic map. Due to the scarcity of pedestrian pixels in the simulated condition, the model may overlook this condition when the text prompt is \"Mountains\" where there are supposed to be no pedestrians, resulting in their disappearance.\n\n***Minor issues.***\n\nWe will reorganize the sentences and correct the typos in the revision.\n\n[1] ROUGE: A Package for Automatic Evaluation of Summaries.\n\n[2] CAT: Closed-loop Adversarial Training for Safe End-to-End Driving.\n\n[3] Enhancing Photorealism Enhancement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjfSfLyKm2", "reviewer": "Reviewer_azzq", "summary": "This paper presents a simulator-conditioned scene generation framework, SimGen, which generates diverse driving scenes conditioned on simulator controls and texts. To support the model training, the authors also introduce a driving video dataset, DIVA, comprising 147.5 hours of both real-world and simulator-based driving data. SimGen can be controlled by both text prompts and layouts from the simulator. The synthetic data generated by SimGen can be used for data augmentation on BEV detection and segmentation tasks. The authors also demonstrate its ability to create safety-critical data.", "questions": "See weakness 1 about DIVA-Sim dataset.", "limitations": "The authors have included a limitation section.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The idea is interesting and significant for the field of autonomous driving. This method generates realistic visual data based on simulator conditions, offering flexibility to data collection in rare and safety-critical scenarios.\n2. Flexible conditioning signals. The authors greatly utilize 2d visual representations like depth and segmentation as the conditioning signals for sim2real image generation. These 2d representations are more flexible to transfer and easy to collect with well-established visual models. This is especially important for front-view YouTube videos, where 3D conditioning signals, e.g., 3D bounding boxes, are hard to estimate.\n3. The proposed dataset, DIVA, would be helpful for the community. DIVA is a large mixed dataset comprising both real-world and simulated driving data. It leverages the advantages of two different data sources to improve appearance and layout diversity.\n4. The qualitative visualizations of generated images are abundant and intuitive.", "weaknesses": "[Major] Unclear usage of DIVA-Sim dataset in training. Even though it is clear how the SimGen works given the layout from the simulator, it is still confusing to me how the DIVA-Sim is incorporated during training. From lines 176-177, it says “The training does not contain data rendered from simulators.”, showing that DIVE-SIM is not used for the training of CondDiff (Sec. 3.1). So is it only used for the training of ImgDiff? If so, it means that the ImgDiff could process both SimCond and RealCond in training to generate simulated images and realistic images respectively, would that cause some optimization issues like unstable training since these two condition- and image distributions are significantly different? The authors are highly suggested to specify the usage of each part of the dataset in the main paper for readability. Adding a subsection in Sec 3. might be helpful.\n\n[Major] Lack of scalability analysis for data augmentation experiments. In Table 5, the author incorporates the equal-sized synthesized data as an augmentation to improve the performance of perception models. However, it is important to investigate how the performance goes with more synthesized data, such as 200% or even more.\n\n[Minor] Lack of high-level system workflow for readability and better understanding. The authors are encouraged to add a small yet concise figure to distinguish the training and inference procedure, where each node in the figure should present one stage in the training/inference pipeline, such as the Sim-to-Real Condition Transformation in Sec 3.1. Different from the architectural design in Figure 3, this figure focuses on the high-level workflow of the system instead of module details. Note that it’s not required for rebuttal, but should be considered for future revisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***W1: Unclear usage of DIVA-Sim dataset in training.***\n\nAs stated in L158, Tab.2, and Appendix C.3, DIVE-Sim is only used in the training of ImgDiff. ImgDIff takes RealCond from DIVA and nuScenes and ExtraCond from DIVA-Sim as inputs, without including SimCond, and outputs real images. Consequently, there are no optimization issues. We will add more descriptions for Tab. 2 in the revision.\n\n***W2: Lack of scalability analysis for data augmentation experiments.***\n\nIn Supp Tab. 8, we assess the various proportions of synthesized data as an augmentation strategy to boost the perception model (BEVFusion). The results indicate that performance improves with an increase in synthesized data, from **47.7** to **51.9** on $\\text{AP}_{3D}$. This validates SimGen’s data scaling capability for data augmentation in driving applications.\n\n***W3: Lack of high-level system workflow for readability and better understanding.***\n\nWe will add a figure to illustrate the system workflow in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s9QmFHLEY9", "reviewer": "Reviewer_xuJP", "summary": "The paper introduces SimGen, a framework for generating diverse driving scenes to reduce annotation costs in autonomous driving. SimGen combines simulator and real-world data using a novel cascade diffusion pipeline to address sim-to-real gaps and multi-condition conflicts. It is enhanced by the DIVA dataset, which includes extensive real-world and simulated driving data. SimGen improves synthetic data augmentation for BEV detection and segmentation tasks, with code, data, and models to be released.", "questions": "- Without videos demonstrating 3D consistency, the proposed method is less practical for autonomous driving applications.\n- I am confused about the conditions used in SimGen. Figure 3 indicates that DIVO synthetic images are not used in the generation process, whereas Figure 4 shows them being used as conditions. Could you clarify where the synthetic images are actually utilized?", "limitations": "The limitations are discussed in the paper. However, the social impact is not discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well organized.\n- The proposed DIVA dataset is designed in an elegant manner and could be beneficial to the research community.\n- The generated images are of good quality and diversity.", "weaknesses": "- Despite achieving superior results in single image generation, the 3D consistency has not been evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***W1: Despite achieving superior results in single image generation, the 3D consistency has not been evaluated.***\n\nWe agree with the reviewer that achieving 3D consistency will be an important future direction. However, it is not the main focus of this submission. We have preliminary attempts at video generation and design the temporal consistency in Sec. 4.3 and Appendix D.2. Fig. 5, 12, and 16 provide visualization examples, and Tab. 11 indicates that SimGen performs comparably to other driving video generation models. Nevertheless, ensuring and evaluating 3D geometry consistency in generated driving scenes poses a challenge, with all existing work (e.g., Panacea, DrivingDiffusion, GenAD) struggling to achieve this. Even powerful video generation models such as Sora and SVD have similar difficulties.\n\n***Q1: Without videos demonstrating 3D consistency, the proposed method is less practical for autonomous driving applications.***\n\nAlthough SimGen cannot guarantee 3D consistency, the capability to generate diverse driving images and videos with annotations is crucial to many applications. Tab. 5 demonstrates how SimGen can generate diverse driving scenarios for data augmentation in map segmentation and 3D object detection models. Fig. 5 and 12 showcase its applications in generating hazardous driving scenarios and closed-loop evaluation. Other applications like lane detection and BEV detection can also benefit from synthetic image-annotation pairs. \n\n***Q2: I am confused about the conditions used in SimGen. Figure 3 indicates that DIVA synthetic images are not used in the generation process, whereas Figure 4 shows them being used as conditions. Could you clarify where the synthetic images are actually utilized?***\n\nAs illustrated in Fig. 3 and Tab. 2, a scene record is fed into the simulator to generate DIVA synthetic images, which are further grouped into SimCond (simulated depth and segmentation) and ExtraCond (rendered RGB images, instance maps, and top-down views). SimCond is transformed into RealCond by the CondDiff module and then, along with ExtraCond and text, is fed into the ImgDiff module to generate real images. More descriptions on this part will be included in the revision.\n\n***Limitations.***\n\nWe have discussed the social impacts in Appendix A.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RURIzPNgQJ", "reviewer": "Reviewer_vm6z", "summary": "This paper proposes a novel framework named SimGen, which aligns the web-scaled unannotated driving footage with the simulator-generated images and annotations to obtain the appearance diversity and traffic layout diversity for generating novel driving scene images.", "questions": "1.\tI am curious about the solution of the statistical shortcut that the authors have mentioned in Appendix C.1.2. The core concern is that YouTube data has no ExtraCond but nuScenes data has, such that “outputting nuScenes-style images when ExtraCond is present and YouTube-style images when it’s absent.”  I am wondering why this can be solved by an adapter that merges various modalities into a unified control feature.\n2.\tIf I understand correctly, in DIVA-sim, the safety-critical data originating from the Waymo Open Dataset are only used to showcase the proposed SimGen’s ability to generate realistic images for diverse traffic layouts qualitatively. These data are not used for training SimGen since no ground truth real images can be obtained. Also, these data are not used for data augmentation or evaluation for the pre-trained perception models. I am wondering whether the authors can illustrate the usage of the safety-critical data originating from Waymo more clearly.\n3.\tIn terms of future work to extend SimGen to multi-view image generation, is there any idea that the author is willing to discuss since the multi-view video footage are not that abundant on the internet?", "limitations": "The authors have discussed the limitations of this paper in the main paper.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper tackles a challenging but under-explored problem: how can we best use the real-world and simulator data for autonomous driving together? The real-world driving footage is abundant on the internet and has great appearance diversity but lacks task annotations. On the other hand, the simulated data can easily obtain the task annotations but are still visually distinct from the real-world data. In this paper, the authors propose a novel image-generation framework for driving scenes named SimGen, which can utilize both real-world data and the flexible simulator to generate diverse driving scene images both for appearance and traffic layout. This work is remarkably valuable for the autonomous driving community.\n\nThe proposed SimGen framework designs a two-stage cascade diffusion process to alleviate the domain gap between the simulation and the real-world, as well as tackle the misaligned modalities of conditions from different data sources. This framework is wisely designed and can easily be extended to other robotics tasks, even beyond the scope of autonomous driving.\n\nIn the experiment, the proposed SimGen has shown the best image generation quality and diversity compared to existing data, which proves the usefulness of the proposed SimGen framework.\n\nMoreover, the proposed DIVA dataset in this paper contains great diversity in appearance, weather, lighting, and location. This can significantly enhance the existing autonomous driving datasets, in which only a few cities are involved.", "weaknesses": "I just have few content presentation suggestions for this paper. \n\nI think moving the Related Work to the earlier part of the paper will give the readers a better understanding of the contributions of this paper compared to existing works. Besides, I think the generalizability of the trained SimGen shown in Appendix D.3  “Simulator-conditioned image generation,” is quite impressive and may be better to move to the main paper. The trained SimGen can generate diverse images based on the conditions provided by the novel simulator CARLA in zero-shot can surely illustrate its generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***W1: Content presentation suggestions for this paper.***\n\nThank you for the suggestion. We follow your advice to reorganize the Related Work section and move the content in Appendix D.3 to the main paper.\n\n***Q1: The solution of the statistical shortcut.***\n\nIntegrating input conditions through an adapter is a common practice in training image generation models with multi-modal conditions [1], which serves as our foundation. Randomly dropping conditions within the adapter helps the model balance various condition modalities, preventing overfitting to specific ones. The adapter merges information from different modalities to reduce uncertainty caused by randomly dropping. Furthermore, we mitigate conflicts between multi-modal conditions by masking out simulator background information, preventing the model from learning associations between simulator backgrounds and nuScenes-style images.\n\n***Q2: The usage of the safety-critical data originating from Waymo.***\n\nThe collection of safety-critical data is a part of our contributions. The DIVA-sim dataset includes simulated driving videos and annotations of hazardous behaviors initialized from Waymo, ensuring diverse scene generation and advanced simulation-to-reality research. Annotations such as object attributes, lane lines, trajectories, etc., can be derived from the simulator. In addition to showcasing how SimGen can generate safety-critical scenarios qualitatively based on DIVA-Sim, the labeled driving scenes can also be utilized in map segmentation and object detection tasks for data augmentation. Furthermore, agents learned from safety-critical data have the potential to achieve superior driving safety [2]. We leave these parts as our future work.\n\n***Q3: Future work to extend SimGen to multi-view image generation.***\n\nThank you for the suggestion. Regarding extending SimGen to multi-view image generation, we could consider leveraging 360-degree images from Google Street View [3], which have recently been used to train diffusion models for generating realistic multi-view street scene images. Additionally, we might incorporate cross-view attention mechanisms to ensure consistency across different views, similar to the approaches demonstrated in DrivingDiffusion [4] and Panacea [5].\n\n\n[1] UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild.\n\n[2] CAT: Closed-loop Adversarial Training for Safe End-to-End Driving.\n\n[3] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion.\n\n[4] DrivingDiffusion: Layout-Guided Multi-view Driving Scene Video Generation with Latent Diffusion Model.\n\n[5] Panacea: Panoramic and Controllable Video Generation for Autonomous Driving.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vC5HDjthvE", "reviewer": "Reviewer_u5Ry", "summary": "This paper aims to provide a photorealistic appearance to conventional graphics engine. The basic idea is to use controlnet to convert the rendered semantic mask + depth to real images, similar to the setting of image-to-image translation and style transfer. This paper also introduces the DIVA dataset that contains driving logs with diverse appearances and layouts.\n\n---\nUpdate after rebuttal: I've raised my score from Borderline reject to Weak accept. See comments for details.", "questions": "1. How many categories can SimGen generate? It seems that not all categories in nuscene will be generated as discussed in line 128.\n2. What is the difference between the proposed DIVA dataset and the OpenDV-YouTube dataset from GenAD? It is also collected from YouTube.\n3. As the CFG scale (classifier-free guidance scale) impacts the conditioning strength, I am wondering how the CFG scale impacts the metric of pixel diversity.", "limitations": "The authors have discussed their work's limitations and potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper presents a novel application that enables graphics engine with realistic appearances using diffusion prior. It can create driving scenarios with various appearances such as different illumination and weather.\n2. To provide a diverse and realistic generation, SimGen introduces the DIVA Dataset collected from high-quality YouTube driving videos to help the training.\n3. The overall flow of the presentation, including writing and figure demonstration, is smooth and easy to follow.", "weaknesses": "1. Modern perception systems in self-driving cars rely on surround-view cameras to perceive the 360 degrees of the world. This paper seems to neglect this problem and only simulates front-view cameras.\n2. Conditioning on simulator may not be necessary. We know that simulators can render sharp and accurate semantics and depth. However, this paper does not leverage these advantages and instead proposes CondDiff, which trains a diffusion model to convert SimCond to RealCond, as simulated conditions have a distinct distribution from real-world conditions. Meanwhile, simulator data is sometimes converted from real-world data using its lane layout and 3D box. So, why not reconstruct the mesh or 3D occupancy from real-world data, and then perform conditional generation? In this way, there will be no domain gap between real conditions and simulated conditions, and it also maintains controllability. \n3. This paper only uses certain interfaces of the simulator, which can be trivial in implementation, as it can easily replaced by low-level traffic simulation and then convert the projected 3D box masks to RealCond. Thus leveraging simulators is questionable, and underlying technical contribution can be limited.\n4. The generation result of SimGen is restricted to the asset bank of the simulator itself. It can only generate certain categories that are available in the simulator. As a result, SimGen only demonstrates improvement over certain categories (road, car, and truck) in downstream tasks, as shown in Table 4 and Table 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Before diving into the details, we reclaim the benefits of our simulator-conditioned pipeline.\n\nSimulators can reconstruct scenarios from public datasets, real-world driving logs (e.g., Waymo, nuPlan), and program-generated scenes to obtain data with diverse layouts and annotations. Also, simulators allow for editing the motions and states of traffic participants with rules or interactive policies [1], enabling the generation of risky scenarios from driving records. Finally, simulators provide control over all objects and spatial positions, facilitating the controllable generation.\n\n***W1: Neglection of surround-view cameras.***\n\nWe agree that multi-view generation is meaningful for driving systems, but it is diagonal to the problem we tackle here. Our goal is to achieve appearance and layout diversity in generating novel scenarios, while multi-view generation emphasizes consistency between adjacent views. They are two interrelated yet distinct. Rather than just overfitting on nuScenes [2], SimGen learns from both datasets and web data to ensure content diversity. The front-view setup provides SimGen with a unified IO for utilizing data with various sensor setups. We plan to achieve multi-view consistency via cross-view attention [2] and learning from 360-degree images [3] in our future work.\n\n***W2: Necessity of conditioning on simulators.***\n\nConditioning on the simulator has significant advantages.\n\n1. The MetaDrive simulator provides control over all objects and their 3D locations in a traffic scenario. It also provides access to 100K real-world traffic scenarios imported from Waymo and nuPlan. Besides, it can effortlessly generate scenes encompassing risky behaviors with its physics engines. \n\n2. While mesh and occupancy can be utilized for conditional generation [4], existing annotators struggle to reconstruct scenes from web data. Thus, the model's generalization capability is limited to public datasets. Differently, semantics and depth can be rendered from simulators, and pseudo-labels can be obtained from datasets and web data. Therefore, SimGen achieves superior generation diversity while preserving controllability by corporating the simulator.\n\n3. It also paves the path for agent-based interaction to unify the generated perception with downstream decision-making and simulation applications. \n\n***W3: Implementation of incorporating simulators.***\n\n1. Incorporating simulators is not trivial. Beyond implementing interfaces, we customize and extend the functions within MetaDrive, including importing attributes from real logs, modifying asset shapes, customizing camera parameters, designing instance cameras, etc. All codes will be made publicly available.\n\n2. The usage of simulators cannot be replaced by traffic simulations like LimSim [5], as spatial conditions help enhance the quality of generated images. It's not feasible to export the first-view conditions (e.g., the semantic shapes) from a traffic simulator. While projected 3D boxes offer basic control over object positions and sizes, simulator assets closely resemble real objects in terms of 3D shapes and provide details like vehicle doors, windows, wheels, pedestrian limbs, etc. Moreover, annotating 3D boxes from web data is challenging, restricting models based on traffic simulation to generate scenarios within a limited amount of fully annotated data.\n\n***W4: The generation is restricted to simulator assets.***\n\nThe simulator's assets include **12** foreground objects, such as vehicles, pedestrians, cones, traffic signs, etc., not limited to just roads and cars. For a specific category, SimGen can generate diverse appearances based on texts, not confined to the assets. Tab. 15 (9th row) shows SimGen generates different types of vehicles (vans, trucks, and buses), differing from the predefined asset. Besides, we report the per-class results of Tab. 4-5 in Supp Tab. 2-3. Tab. 2 shows SimGen's excellent controllability, with only a **-4.8** decrease in the bus category. Tab. 3 validates SimGen's benefits across all nuScenes categories, including motorcycle (**+2.9**), cone (**+1.6**), and barrier (**+1.7**).\n\n***Questions.***\n\n***1.***\nAligned with semantic labels of Cityscapes, SimGen can generate **19** categories, covering all classes in nuScenes and **97%** pixels in daily driving scenes. Fig. 14 shows the diverse categories that SimGen can generate, including poles (r1c2), buses (r3c1), motorcycles (r3c3), barriers (r4c4), etc. Besides, the Supp Tab. 3 confirms SimGen generates all categories in nuScenes.\n\n***2.***\nPart of DIVA dataset, DIVA-Real, shares similarities to OpenDV as both use YouTube data. But DIVA targets different tasks and has different data-preparing pipelines.\n**Task objective**: DIVA is designed for controllable image generation. To support multimodal conditions, DIVA offers depth and segmentation labels. These labels enable SimGen to learn and control the layouts and behaviors of generated scenarios. OpenDV is collected mainly for predictive video generation, so there is no need to collect these labels.\n**Automated pipeline**: The collection of DIVA is fully automated, facilitating data scaling while ensuring quality. In contrast, OpenDV requires manual quality checks, which is less efficient.\n\n***3.***\nIn Supp Tab. 9,  as the CFG scale increases from 5 to 14, pixel diversity rises from **21.3** to **28.2** before reaching saturation. The rise is due to changes in image contrast and sharpness; as the scale hits a threshold, the image distinctions diminish.\n\n[1] CAT: Closed-loop Adversarial Training for Safe End-to-End Driving.\n\n[2] DrivingDiffusion: Layout-Guided Multi-view Driving Scene Video Generation with Latent Diffusion Model.\n\n[3] Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion.\n\n[4] Wovogen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generation.\n\n[5] LimSim: A Long-term Interactive Multi-scenario Traffic Simulator.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and ACs,

We sincerely thank all reviewers for their detailed and constructive comments. It is encouraging that reviewers acknowledge our pioneering efforts in establishing a simulator-conditioned generative model in driving scenarios. We have taken each comment into consideration, added more requested ablative studies in the rebuttal, and clarified some implementation details.

The attached PDF includes tables with quantitative results. We will refer the reviewer to the corresponding component in the following detailed responses. Additionally, we will also incorporate these results into our revised paper.

Please refer to the rebuttal blocks below for our point-by-point responses to each reviewer.

Thank you for your time and effort once again! We hope our rebuttal can address your concerns, and you are more than welcome to ask any further questions. Looking forward to your reply!

Best regards,
The authors of Submission2300.

Meta-review: The submission received uniformly positive feedback from the reviewers, all of whom highlighted the strengths of the paper and its contribution to the field. The authors have addressed some of the raised concerns in the rebuttal phase. We invite the authors to address all the weak points for the camera-ready version. Thanks.
Decision: Accept (poster)