MagicDrive: Street View Generation with Diverse 3D Geometry Control
Authors: Ruiyuan Gao
Kai Chen
Enze Xie
Lanqing HONG
Zhenguo Li
Dit-Yan Yeung
Qiang Xu
Venue: ICLR 2024 poster (2024)
Keywords: diffusion models
street view generation
3D control
Primary area: generative models
TL;DR: MagicDrive generates street-view images & videos with enhanced 3D control. The generated data can facilitate various downstream 3D perception tasks.

Abstract: Recent advancements in diffusion models have significantly enhanced the data synthesis with 2D control. Yet, precise 3D control in street view generation, crucial for 3D perception tasks, remains elusive. Specifically, utilizing Bird's-Eye View (BEV) as the primary condition often leads to challenges in geometry control (e.g., height), affecting the representation of object shapes, occlusion patterns, and road surface elevations, all of which are essential to perception data synthesis, especially for 3D object detection tasks. In this paper, we introduce MagicDrive, a novel street view generation framework, offering diverse 3D geometry controls including camera poses, road maps, and 3D bounding boxes, together with textual descriptions, achieved through tailored encoding strategies. Besides, our design incorporates a cross-view attention module, ensuring consistency across multiple camera views. With MagicDrive, we achieve high-fidelity street-view image & video synthesis that captures nuanced 3D geometry and various scene descriptions, enhancing tasks like BEV segmentation and 3D object detection. Project Website: https://flymin.github.io/magicdrive

Published as a conference paper at ICLR 2024
MAGICDRIVE: STREET VIEW GENERATION WITH DIVERSE 3D GEOMETRY CONTROL
Ruiyuan Gao Kai Chen?*, Enze Xte Lanqing Hong?, Zhenguo Li Dit-Yan Yeung' Qiang Xu
'The Chinese University of Hong Kong "Hong Kong University of Science and Technology Huawei Noah's Ark Lab (rygao, qxu30cse cucck cuhk, edu. kai. conneco .usttteeetttut............. .us. .k, [xie. enze, honglanging, 1i. zhenguo]ehuawei con, dyyeungecse. ust hk
ABSTRACT
Recent advancements in diffusion models have significantly enhanced the data synthesis with 2D control. Yet, precise 3D control in street view generation, cru- cial for 3D perception tasks, remains elusive. Specifically, utilizing Bird's-Ei E-ye View (BEV) as the primary condition often leads to challenges in geometry control (e.g., height), affecting the representation of object shapes, occlusion patterns, and road surface elevations, all of which are essential to perception data synthesis, es- pecially for 3D object detection tasks. In this paper, we introduce MAGICDRIVE, novel street view generation framework, offering diverse 3D geometry controls including camera poses, road maps, and 3D bounding boxes, together with textual descriptions, achieved through tailored encoding strategies. Besides, design incorporates a cross-view view attention module, ensuring consistency across multiple camera views. With MAGICDRIVE, & video synthesis that captures nuanced 3D geometry and various scene descrip- tions, enhancing tasks like BEV segmentation and 3D object detection.
achieve high-fidelity street-view image
BEV map
3D box
Camera
Ponning tik
Front Left
Front
Front Right
Figure 1: Multi-camera street view generation from MAGICDRIVE. MAGICDRIVE generate continuous camera views with controls from the road map, object boxes, and text (e.g., weather).
INTRODUCTION
The high costs associated with data collection and annotation often impede the effective training of deep learning models. Fortunately, cutting-edge generative models have illustrated that synthetic data can notably boost performance across various tasks, such as object detection (Chen et al., 2023c) and semantic segmentation (Wu et al., 2023b). Yet, the prevailing methodologies are largely tailored 2D contexts, primarily relying on 2D bounding boxes (Lin et al., 2014; Han et al., 2021) or segmentation maps (Zhou et al., 2019) as layout conditions (Chen et al., 2023e; Li et al., 2023b).
In autonomous driving applications, thorough grasp of the 3D environment is essential. This demands reliable techniques for tasks like Bird's-Eye View (BEV) map segmentation (Zhou & Krähenbühl, 2022; et al., 2023) and 3D object detection (Chen et al., 2020; Huang et al., 2021; Liu et al., 2023a; Ge et al., 2023). A genuine 3D geometry representation crucial for capturing intricate details from 3D annotations, such as road elevations, object heights, and their occlusion pat-
"qqual contribution. 'Corresponding authors. Project Page: https:/ :/ff n.github 10/mag

---

Published as a conference paper at ICLR 2024
boxes
BEV Map
distance
No height
x ocelusion
x No view-eenssseenney
(b) Road surfice elevation guided by 3D bounding boxes. Figure 2: 3D bounding boxes are crucial for street view synthesis. Two examples show that 2D boxes or BEV maps lost distance, height, and elevation. Images are generated from MAGICDRIVE.
3D bounding boxes show position relationships.
terns, as shown in Figure 2. Consequently, generating multi-camera street-view images according to 3D annotations becomes vital to boost downstream perception tasks.
For street-view data synthesis, two pivotal criteria are realism and controllability. Realism requires that the quality of the synthetic data should align with that of real data; and in given scene, views from varying camera perspectives should remain consistent with one another (Mildenhall et al., 2020). On the other hand, controllability emphasizes the precision in generating street-view images that adhere to provided conditions: the BEV map. 3D object bounding boxes, and camera poses for views. Beyond these core requirements, effective data augmentation should also grant the ficxibility to tweak finer scenario attributes, such as prevailing weather conditions the time of day. Existing solutions like BEVGen (Swerdlow et al., 2023) approach street view generation by encapsulating all semantics within BEV. Conversely, BEVControl (Yang et al., 2023a) starts by projecting 3D coordinates to image views, subsequently using 2D geometric guidance. However, both methods compromise certain geometric hemensions--heiht is lost in BEVGen and depth in BEVControl.
The rise of diffusion models has significantly pushed the boundaries of controllable image gener- ation quality. Specifically, ControlNet (Zhang al., 2023a) proposes flexible framework in- corporate 2D spatial controls based on pre-trained Text-to-Image (T2I) diffusion models (Rombach al., 2022). However, 3D conditions are distinct from pixel-level conditions text. The challenge of seamlessly integrating them with multi- multi-camera view consistency in street view synthesis remains.
In this paper, we introduce MAGICDRIVE, a novel framework dedicated to street view synthesis with diverse 3D geometry controls'. For realism, we harness the power of pre-trained stable dif- al., 2022), further fine-tuning it for street view generation. One distinctive component of our framework is the cross-view attention module. This simple yet effective com- ponent provides multi-view consistency through interactions between adjacent views. In contrast to previous methods, MAGICDRIVE proposes separate design for objects and road map encoding to improve controllability with 3D data. More specifically, given the sequence-like, variable-length nature of 3D bounding boxes, employ cross-attention akin text embeddings for their encod- ing. Besides, we propose that an addictive encoder branch like ControlNet (Zhang et al., 2023a) can encode maps in BEV and is capable of view transformation. Therefore, our design achieves geometric controls without resorting to any explicit geometric transformations or imposing geomet- ric constraints on offering attribute control such as weather conditions and time of day.
fusion (Rombach
consistency. Finally, MAGICDRIVE factors in textual descriptions,
Our MAGICDRIVE framework, despite its simplicity, excels in generating strikingly realistic images & videos that align with road maps, 3D bounding boxes, and varied camera perspectives. Besides, the images produced can enhance the training for both 3D object detection and BEV segmentation tasks. Furthermore, MAGICDRIVE offers comprehensive geometric controls at the scene, back- ground, and foreground levels. This flexibility makes it possible to craft previously unseen street views suitable for simulation purposes. We summarize the main contributions of this work as:
The introduction of MAGICDRIVE, an innovative framework that generates multi perspective camera views & videos conditioned on BEV and 3D data tailored for autonomous driving-
The development of simple yet potent strategies manage 3D geometric data, effectively ad- dressing the challenges of multi-camera view consistency in street view generation.
Through rigorous experiments, we demonstrate that MAGICDRIVE outperforms prior street view generation techniques, notably for the multi-dimeensionl controllability. Additionally, our results reveal that synthetic data delivers considerable improvements in 3D perception tasks.
In this paper, our 3D geometry controls contain control from road maps, 3D object boxes, and camera poses. We do not consider others like the exact shape of objects background contents.
2

---

Published as a conference paper at ICLR 2024
2
RELATED WORK
Diffusion Models for Conditional Generation. Diffusion models (Ho et al., 2020; Song et al., 2020; Zheng et al., 2023) generate images by learning a progressive denoising process from the Gaussian noise distribution to the image distribution. These models have proven exceptional across diverse tasks, such rext-to-image synthesis (Rombach et al., 2022; Nichol et al., 2022; Yang et al., 2023b), inpainting (Wang et al., 2023a), and instructional image editing (Zhang et al., 2023b; Brooks al., 2023), due to their adaptability and competence in managing various form of controls (Zhang et al., 2023a; Li et al., 2023b) and multiple conditions (Liu et al., 2022a; Gao et al., 2023). Besides, data synthesized from geometric annotations can aid downstream tasks such as 2D object detec- tion (Chen et al., 2023c; Wu et al., 2023b). Thus, this paper explores the potential of T2I diffusion models in generating street-view images and benefiting downstream 3D perception models.
Street View Generation. Numerous street view generation models condition on 2D layouts, such as 2D bounding boxes et,,, 2023b) and semantic segmentation (Wang et al., 2022). These methods leverage 2D layout information corresponding directly to image scale, whereas the 3D information does not possess this property, thereby rendering such methods unsuitable for leveraging 3D infor- mation for generation. For street view synthesis with 3D geometry, BEVGen (Swerdlow et al., 2023) is the first to explore. It utilizes a BEV map as a condition for both roads and vehicles. However, the omission of height information limits its application in 3D object detection. BEVControl (Yang et al., 2023a) amends the loss of object's 's height by the height-lifting process. Similarly, Wang et al. (2023b) also projects 3D boxes to camera views to guide generation. However, the projection from 3D to 2D results in the loss of essential 3D geometric information, like depth and occlusion. In this paper, we propose encode bounding boxes and road maps separately for more nuanced control and integrate scene descriptions, offering enhanced control over the generation of street views.
Multi-camera Image Generation of a 3D scene fundamentally requires viewpoint consisteney. Several studies have addressed this issue within the context of indoor scenes. For instance, MVDif- fusion (Tang et al., 2023) employs panoramic images and cross-vier viee attention module to maintain global consistency, while Tseng et al. (2023) leverage epipolar geometry as constraining prior. These approaches, however, primarily rely on the continuity of image views, a condition not always met in street views due to limited camera overlap and different camera configurations (e.g., expo- sure, intrinsic). Our MAGICDRIVE introduces extra cross- view attention modules to UNet, which significantly enhances consistency across multi-camera views.
3 PRELIMINARY
Problem Formulation. In this paper, we consider the coordinate of the LiDAR system as the ego car's coordinate, and parameterize all geometrie information according to it. Let S {M, B, L} be the description of driving scene around the ego vehicle, where M E 13 1 is the binary map representing a w X h meter road area in BEV with e semantic classes, B {(c, ((c,,,))) ) represents the 3D bounding box position (b. = {(z,, Uj. 25)1 E R883) and class (c: E C) for each object in the scene, and L is the text describing additional information about the scene (e.g., weather and time of day). Given a camera pose P IK, R, T) (i.e., intrinsics, rotation, and translation), the goal of street-view image generation is to learn a generator G()) which synthesizes realistic images I E R #xxx33 corresponding to the scene S and camera pose P as, I = G(S,P, z), where 2 ~ N 1) is a random noise from Gaussian distribution.
Conditional Diffusion Models. Diffusion models (Ho et al., 2020; Song et al., 2020) generate data (æa) by iteratively denoising a random Gaussian noise (xrr for T steps. Typically, to learn the denoising process, the network is trained to predict the noise by minimizing the mean-square error:
(I) where co is the network to train, with parameters 0, c is optional conditions, which is used for the conditional generation, t E [o, is the time-step, e E (O 1) is the additive Gaussian noise, and ā is a scalar parameter. Latent difffusion models (LDM) (Rombach et al., 2022) is a special kind of diffusion model, where they utilize a pre-trained Vector Quantized Variational AutoEncoder (VQ- VAE) (Esser et al., 2021) and perform diffusion process in the latent space. Given the VQ-VAE encoder as 2 = E(x), one can rewrite (.). in Equation I as ep (())) + vV-āā,e t, c) for LDM. Besides, LDD considers text describing the image condition
Laimple Ezo.c,e,t [||e <a(V + vI - ā,e, c
3

---

Published as a conference paper at ICLR 2024
Prompe: A Arsving soone <dsseriphiow.>>
Biest
UNet
"A driving
Pigars
"A driving Sc5ne
RAi
Camera Pose
Eajos
Seq Embbedding:
Object Box
Emap
Road Map
Orensvviww Amention
+ Truinable
KV
Choss astenton injection
@ Prozen
Addictive encoder injecion
Figure 3: Overview of MAGIEDRIVE for street-view image generation. MAGICDRIVE generates highly realistic images, exploiting geometric information from 3D annotations by independently encoding road maps, object boxes, and camera parameters for precise, geometry-guider synthesis. Additionally, MAGICDRIVE accommodates guidance from descriptive conditions (e.g., weather).
STREET VIEW GENERATION WITH 3D INFORMATION
The overview of MAGICDRIVE is depicted in Figure 3. Operating on the LDM pipeline, MAG- ICDRIVE generates street-view images conditioned both scene annotations (S) and the camera pose (P) for each view. Given the 31) geometric information in scene annotations, projecting all to BEV map, akin to BEVGen (Swerdlow et al., 2023) or BEVControl (Yang al., 2023a). doesn't ensure precise guidance for street view generation, as exemplified in Figure 2. Consequently, MAG- ICDRIVE categorizes conditions into three levels: scene (text and camera pose), foregrondd (3D bounding boxes), and background (road map); and integrates them separately via cross-attention and an additive encoder branch, detailed Section 4.1. Additionally, maintaining consistency across different cameras is crucial for synthesizing street views. Thus, we introduce simple yet effective cross-vicrs attention module in Section 4.2. Lastly, we elucidate our training strategies in Section 4.3, emphasizing ClassffierFFree Guidance (CFG) in integrating various conditions.
4.1 GROMETRIC CONDITIONS ENCODING
As illustrated in Figure 3, two strategies employed for information injection into the UNet of diffusion models: cross-attention and additive encoder branch. Given that the attention mecha- nism (Vaswani al., 2017) is tailored for sequential data, cross-attention is apt for managing vari- able length inputs like text tokens and bounding boxes. Conversely, for grid-like data, such as road maps, the additive encoder branch is effective in information injection (Zhang al., 2023a). There- fore, MAGICDRIVE employs distinct encoding modules for various conditions.
Scene-level Encoding includes camera pose sequence L. For text, we construct the prompt with template as "A driving scene image at ilocati on)
[K E 3 R R3x3 E R331), and text
(descrips pti on] " and leverage a pre-trained CLIP text encoder (Frert) as LDM (Rombach al., 2022), shown by Equation 2, where L is the token length of L. As for camera pose, we first concat each parameter by their column, resulting in P [K, R. T E 7x3 Since P contains values from sin/cos functions and also 3D offsets, to have the model effectively interpret these high-frequency variations, we apply Fourier embedding (Mildenhall et al., 2020) to each 3-dim vector before leveraging Multi-Layer Perception (MLP, Ecam) to embed the camera pose parameters, as in Equation 3. To maintain consistency, we set the dimension of he the same that of hŞ. Through the CLIP text encoder, each text embedding hy already contains positional information (Radford et al., 2021). Therefore, we prepend the camera pose embedding he to text embeddings, resulting in scene-level embedding h² = [h', h'].
h' [hj h4 Etext (L),
(2)
he Ecom Fourier(P)) = Ecam (Fourier ([kr R, ))
(3)
3D Bounding Box Encoding. Since each driving scene has a variable length of bounding boxes, we inject them through the cross-attention mechanism similar to scene-level information. Specifically, we encode each box into a hidden vector h², which has the same dimensions as that of h', Each
3D bounding box (c, by) contains types of information: class label e; and box position b, For
4

---

Published as a conference paper at ICLR 2024
without cross-view attention
Tansformer Block
I Trainable
Frozen
self-attn
ECAMP <TXF> <beoxss>
cross-attn
with cross-view attention
Left Neighbor
cross-view alin
Right Neighbor
zero-imil
Figure 4: Cross-view Attention. left: we introduce cross-view attention to the pre-trained UNet after the cross-attention module. right: we highlight some areas for comparison between without and with cross-view attention. Cross-view attention guarantees consistency across multiple views.
class labels, we utilize the method similar to Li et al. (2023b), where the pooled embeddings of class names (Tee.) are considered as label embeddings. For box positions b, RR83 represented by the coordinates of its 8 corner points, we utilize Fourier embedding to each point and pass through an MLP for encoding, as in Equation 4. Then, we use an MLP to compress both class and position embedding into one hidden vector, as in Equation 5. The final hidden states for all bounding boxes of each scene are represented as h = [h. h'yl. where N the number of boxes.
(i) AvgPool( (Le)) c2(0) = MLP,, (Fouriero))))()))) h = Bror (ci, b.) = MLP (L)
(5) Ideally, the model learns the geometric relationship between bounding boxes and camera pose through training. However, the distribution of the number of visible boxes to different views is long-tailed. Thus, we bootstrap learning by filtering visible objects to each view (v:), i.e., Juia in Equation 6. Besides, we also add invisible boxes for augmentation (more details in Section 4.3).
h = (h) E felsess Re: T.,) > 03.
(6)
Road Map Encoding, The road map has 2D-grid format. While Zhang et al. (2023a) shows the addictive encoder can incorporate this kind of data for 2D guidance, the inherent perspective differences between the road map's BEV and the camera's First-Person View (FPV) create discrep- ancies. BEVControl (Yang et al., 2023a) employs a back-projection to transform from BEV to FPV but complicates the situation with an ill- ill-posed problem. In MAGICDRIVE, view transformation is unnecessary, as sufficient 3D cues (e.g., height from object boxes and cam- integrate pose) allow the addictive encoder accomplish view transformation. Specifically, scene-level and 3D bounding box embeddings into the map encoder (see Figure 3). Scene-level em- beddings provide camera poses, and box embeddings offer road elevation cues. Additionally, incor- porating text descriptions facilitates the generation of roads under varying conditions (e.g., weather and time of day). Thus, the map encoder can synergize with other conditions for generation.
propose that explicit
4.2 CROSS-VIES ATTENTION MODULE
In multi multi-camera view generation, is crucial that image synthesis remains consistent across different perspectives. To maintain consistency, we introduce a cross-view attention module (Figure 4). Given the sparse arrangement of cameras in driving contexts, each cross- cross-view attention allows the target view to access information from its immediate left and right views, as in Equation 7; here, t, I, and r are the target, left, and right view respectively. Then, the target view aggregates such information with skip connection, as in Equation 8, where h² indicates the hidden state of the target view.
Attention (Q,, K2, V,) softmax QiKI Ve, i E (1,r),
Vd
(8) We inject cross-view attention after the cross-attention module in the UNet and apply zero- initialization (Zhang al., 2023a) to bootstrap the optimization. The efficacy of the cross-view attention module is demonstrated in Figure 4 right, Figure 5, and Figure 6. The multilayered struc- ture of UNet enables aggregating information from long-range views after several stacked blocks. Therefore, using cross- view attention on adjacent views is enough for multi-view consistency, fur- ther evidenced by the ablation study in Appendix C.
h huu = h", + Attention + Attention
5

---

Published as a conference paper at ICLR 2024
4.3 MODEL TRAINING
Classifier-free Guidance reinforces the impact of conditional guidance (Ho & Salimans, 2021; Rombach et al., 2022). For effective CFG, models need discard conditions during training occa- sionally. Given the unique nature of each condition, applying a drop strategy is complex for multiple conditions. Therefore, our MAGICDRIVE simplifies this for four conditions by concurrently drop- ping scene-level conditions (camera pose and text embeddings) at rate of y². For boxes and maps, which have semantic representations for null (i.e., padding token in boxes and in maps) in their encoding, we maintain them throughout training. At inference, we utilize null for all conditions, enabling meaningful amplification to guide generation.
Training Objective and Augmentation. With all the conditions injected inputs, we adapt the training objective described in Section to the multi-condition scenario, as in Equation
= Beg (S.P))
[||== c ( + vT 4, PH)]]
Besides, emphasize two essential strategies when training our MAGICDRIVE. First, to counteract our filtering of visible boxes, we randomly add 10% invisible boxes as an augmentation, enhancing the model's geometric transformation capabilities. Second, to leverage cross-view attention, which facilitates information sharing across multiple views, we apply unique noises to different views in each training step, preventing trivial solutions Equation 9 (e.g., outputting the shared component across different views). Identical random noise is reserved exclusively for inference.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Dataset and Baselines. We employ the nuScenes dataset (Caesar et al., 2020), a prevalent dataset in BEV segmentation and detection for driving, as the testing ground for MAGICDRIVE. We adhere to the official configuration, utilizing 700 street-view scenes for training and 150 for validation. Our baselines are BEVGen (Swerdlow et al., 2023) and BEVControl (Yang et al., 2023a), both recent propositions for street view generation. Our method considers 10 object classes and road classes, surpassing the baseline models in diversity. Appendix B holds additional details.
Evaluation Metries. We evaluate both realism and controllability for street view generation. Real- ism is mainly measured using Fréchet Inception Distance (FID), reflecting image synthesis quality. For controllability, MAGICDRIVE is evaluated through two perception tasks: BEV segmentation and 3D object detection, with CVT (Zhou & Krähenbühl, 2022) and BEVFusion (Liu et al., 2023a) as perception models, respectively. Both of them are renowned for their performance each task. Firstly, we generate images aligned with the validation set annotations and perception models pre-trained with real data to assess image quality and control accuracy. Then, data is generated based on the training set to examine the support for training perception models as data augmentation.
Model Setup. Our MAGICDRIVE utilizes pre-trained weights from Stable Diffusion v1.5, train- ing only newly added parameters. Per Zhang et al. (2023a), a trainable UNet encoder is created for Emap. New parameters, except for the zero-init module and the class token, are randomly initialized. We adopt resolutions reconeile discrepancies in perception tasks and baselines: 200x (0.25x down-sample) following BEVGen and for CVT model support, and a higher 272 736 (0.5x down-sample) for BEVFusion support. Unless stated otherwise, images are sampled using the UniPC (Zhao et al., 2023) scheduler for 20 steps with CFG at 2.0.
5.2 MAIN RESULTS
Realism and Controllability Validation. We assess MAGICDRIVE'S capability to create realistic street-view images with the annotations from the nuScenes validation set. As shown by Table 1, MAGICDRIVE outperforms others in image quality, yielding notably lower FID scores. Regarding controllability, assessed via BEV segmentation tasks, MAGICDRIVE equals or exceeds baseline re- sults at 224 x 400 resolution due to the distinct encoding design that enhances vehicle generation precision. At 736 resolution, our encoding strategy advancements enhance vehicle mloU per- formance. Cropping large areas negatively impacts road mloU on CVT. However, our bounding box encoding efficacy is backed by BEVFusion's results in 3DD object detection.

---

Published as a conference paper at ICLR 2024
Table I: Comparison of generation fidelity with driving-view generation methods. Conditions for data synthesis are from nuScenes validation set. For each task, we test the corresponding models trained on the nuScenes training set. MAGICDRIVE surpasses all baselines throughout the evalua- tion. 14 indicates that a higher/lower value better. The best results are in bold, while the second best results are in underlined italic (when other methods are available).
BEV segmentation
3D object detection
Synthesis resolution
Method
FID
Road mloU Vehicle mIoU f
mAP 1
NDS f
Oracle Oracle
72.21
33.66
35.54
41.21
224x400
72.19
33.61
23.54
31.08
BEVGen BEVControl
204x400
25.54
50.20
5.89
24.85
60.80
26.80
MAGICDRIVE
204
16.20
61.05
27.01
12.30
23.32
MAGICDRIVE
272x736
16.59
54.24
31.05
20.85
30.26
Ground Truth
BEVControl
Inconsistent Wrong place
MAGICDRIVE
Figure 5: Qualitative comparison with BEVControl on driving scene from nuScenes validation set. We highlight some areas with rectangles to ease comparison. Compared with BEVControl, genera- tions from MAGICDRIVE appear more consistent in both background and foreground.
Training Support for BEV Segmentation and 3D Object Detection. MAGIC can produce augmented data with accurate annotation controls, enhancing the training for perception tasks. For BEV segmentation, we augment an equal number of images as in the original dataset, ensuring con- sistent training iterations and batch sizes for fair comparisons to the baseline. As shown in Table 3, MAGICDRIVE significantly enhances CVT in both settings, outperforming BEVGen, which only marginally improves vehiele segmentation. For 3D object detection, we train BEVFusion models with M MAGICDRIVE 's synthetic data as augmentation. To optimize data augmentation, we randomly exclude 50% of bounding boxes each generated scene. Table 2 shows the advantageous impact of MAGICDRIVE data in both CAM-only (C) and CAM+LiDAR (C+L) settings. It's crucial to note that CAM+LiDAR settings, BEVFusion utilizes both modalities for object detection, requiring more precise image generation due to LiDAR data incorporation. Nevertheless, MAGICDRIVE'S synthetic data integrates seamlessly with LiDAR inputs, highlighting the data's high fidelity.
5.3 QUALITATIVE EVALUATION
Comparison with Baselines. We assessed MAGICDRIVE against two baselines, BEVGen and BEVControl, synthesizing multi-camera views for the same validation scenes (the comparison with BEVGen is in the Appendix D). Figure 5 illustrates that MAGICDRIVE generates images markedly superior in quality to BEVControl, particularly excelling in accurate object positioning and main- taining consistency in street views for backgrounds and objects. Such performance primarily stems from MAGICDRIVE bounding box encoder and its cross-view attention module.
Multi-level Controls. The design of MAGICDRIVE introduces multi-level controls to street-view generation through separation encoding. This section demonstrates the capabilities of MAGIC- DRIVE by exploring three control signal levels: scene level (time of day and weather), background level (BEV map alterations and conditional views), and foreground level (object orientation and dele- tion). As illustrated in Figure I, Figure 6, and Appendix E, MAGICDRIVE adeptly accommodates alterations at each level, maintaining multi-camera consistency and high realism in generation.

---

Published as a conference paper at ICLR 2024
Table 2: Comparison about support for 3D object
Table 3: Comparison about support for
detection model (i.e. BEVFusion). MAGICDRIVE generates 272x736 images for augmentation. Re-
BEV segmentation model (i.e. CVT). Results are reported by testing on the nuScenes validation set.
sults are reported on the nuScenes validation set.
Modality
Data
1
NDS
Vehicle mIoU
Road mloU
Data
w/o synthetic data
32.88
37.81
w/ MAGICDRIVE
35.40 12.52
39.76 -195
w/o synthetic data
36,00
74.30
w/o synthetic data
65.40
69.59
w/ BEVGen
36.60
71.90
C+L
MRIVE
67.86 +2.46
70.72 .1.3
MAGICDRIVE
40.34 +4.34
79.56 +526
Generation from DRIVEDRIVE
Background level: conuiition given. view (annotated with red box)
Generation from MAGIc DRIVE
Foreground level: votats each vehiols 180 degrse (nead to tail) (we highlight some rotated vehicles for compatison)
Foreground level: delete 50t objects (we show the DELETED boxes and highlight areas for comparison)
Figure 6: Showcase for multi-level control with MAGICDRIVE. We show background level and foreground-level control separately with different conditions. All scenes based on the nuScenes validation set. More results are Appendix E.
5.4 EXTENSION TO VIDEO GENERATION
We demonstrate the extensibility of MAGICDRIVE video generation by fine-tuning it nuScenes videos. This involves modifying self-attention to ST-Attn (Wu et al., 2023a), adding a temporal attention module each transformer block (Figure 7 left), and tuning the model 7-frame clips with only the first and the last frames having bounding boxes. We sample initial noise independently for each frame using the UniPC (Zhao al., 2023) sampler for 20 steps and illustrate an example in Figure right.
Furthermore, by utilizing the interpolated annotations from ASAP (Wang et al., 2023c) like Drive- Dreamer (Wang et al., 2023b), MagicDrive can be extended to 16-frame video generation at 12Hz trained on Nvidia V100 GPUs. More results (e.g., video visualization) can be found on our website.
6 ABLATION STUDY
Bounding Box Encoding, MAGICDRIVE utilizes separate encoders for bounding boxes and road maps. To demonstrate the efficacy, we train a ControlNet (Zhang et al., 2023a) that takes the BEV map with both road and object semantics as a condition (like BEVGen), denoted as "w/o Foor" in Table Objects BEV maps are relatively small, which require separate Froz for accurate vehicle annotations, shown by the vehicle mloU performance gap. Applying visible object filter fuiz significantly improves both road and vehicle mIoU by reducing the optimization burden. A MAG- IcDRIVE variant incorporating Ebor with BEV of road and object semanties didn't 't enhance perfor- mance, emphasizing the importance of integrating diverse information through different strategies.
Effect of Classifier-free Guidance. We focus the two most crucial conditions, í.e. object boxes and road maps, and analyze how CFG affects the performance of generation. We change CFG from 1.5 to 4.0 and plot the change of validation results from CVT in Figure 8. Firstly, by increasing
8

---

Published as a conference paper at ICLR 2024
Tansformer Block
K Keyframe
Trainable Frozenn
ST-Att (VV
<iocxsss>
Left Neighbor
éross-view
Right Neighbor
6094
Key + Prev. frames
temp altn
Keyframe
Figure 7: Extend MAGICDRIVE to video generation. Left: changes in transformer block for video generation. Right: result of video generation. Only keyframes have bounding box control.
Table 4: Ablation of the separate box encoder. Evaluation results are from CVT on the synthetic nuScenes validation set, without M {0} in CFG scale = 2. MAGICDRIVE has better controllability and keeps image quality.
20
Road
61.05
Road mloU f
Vehicle
Method
FID +
mloU f
Vettcce
27
27.25
w/o Eboz 18.06
58.31
5.50
27.01
no map CPG
mip- CFG
w/o Juiz
14.67
56.46
24.73
26
2.0
2.5
3.0
w/ EboX & mapobj
14.70
56.04
26.20
CPG
Figure 8: Effect of CFG on different conditions to each metrics.
Ours
14.46
59.31
27.13
in previous stud- ies (Chen et al., 2023c). Secondly, retaining the same map for both conditional and unconditional inference eliminates CFG's effect the map condition. As shown by blue lines of Figure 8, increas- ing CFG scale results in the highest vehicle mloU at CFG=2.5, but the road mloU keeps decreasing. Thirdly, with M {0} for unconditional inference in CFG, road mloU significantly increases. However, it slightly degrades the guidance on vehicle generation. As mentioned in Section 4.3, CFG complexity increases with more conditions. Despite simplifying training, various CFG choices exist during inference. We leave the in-depth investigation for this case as future work.
CFG scale, FID degrades due notable changes in contrast and sharpness,
7 CONCLUSION
This paper presents MAGICDRIVE, novel framework to encode multiple geometric controls for high-quality multi-camera street view generation. With the separation encoding design, MAGIC- DRIVE fully utilizes geometric information from 3D annotations and realizes accurate semantic control for street views. Besides, the proposed cross-view attention module is simple yet effective in guaranteeing consistency across multi-camera views. As evidenced by experiments, the generations from MAGICDRIVE show high realism and fidelity to 3D annotations. Multiple controls equipped MAGICDRIVE with improved generalizability for the generation of novel street views. Meanwhile, MAGICDRIVE can be used for data augmentation, facilitating the training for perception models on both BEV segmentation and 3D object detection tasks.
Limitation and Future Work. We show failure cases from MAGICDRIVE in Figure 9. Although MAGICDRIVE can generate night views, they are not as dark as real images (as Figure 9a). This may be due to that diffusion models are hard to generate too dark images (Guttenberg, 2023). Fig- 9b shows that MAGICDRIVE cannot generate unseen weathers for nuScenes. Future work may
GroundTrrth
Ground
MAGICDRIVE wight
MAGICDRIVE!
weather Figure 9: Failure cases of MAGICDRIVE. focus how improve the cross-domain generalization ability of street view generation.
dark enough

---

Published as a conference paper at ICLR 2024
Acknowledgement. This work is supported in part by the General Research Fund (GRF) of Hong Kong Research Grants Council (RGC) under Grant No. 14203521, in part by the CUHK SSFCRS funding No. 3136023, and in part by the Research Matching Grant Scheme under Grant No. 7106937, 8601 130, and 8601440. We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend Al Processor used for this research. This research has been made possible by funding support from the Research Grants Council of Hong Kong through the Research Impact Fund project R6003-21.
REFERENCES
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
Kai Chen, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit- Yan Yeung. Multisiam: Self-supervised multi-instande siamese representation learning for autonomous driving. In ICCV, 2021.
Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit-Yan Yeung. Mixed autoencoder for self-supervised visual representation learning. In CVPR, 2023a.
Kai Chen, Chunwei Wang, Kuo Yang, Jianhua Han, Lanqing Hong. Fei Mi, Hang Xu, Zhengying Liu, Wenyong Huang, Zhenguo Li, Rin Yeung, Lifeng Shang, Xin Jiang, and Qun Liu. Gain- ing wisdom from setbacks: Aligning large language models via mistake analysis. arXiv preprint arXiv:2310.1044 2023b.
Kai Chen, Enze Xie, Zhe Chen, Lanqing Hong, Zhenguo Li, and Dit- Yan Yeung. Integrating geo- metric control into text-to-image diffusion models for high-quality detection data generation via text prompt. arXiv preprint arXiv:2306.. 04607, 2023c.
Yaran Chen, Haoran Li, Ruiyuan Gao, and Dongbin Zhao. Boost 3-d object detection via point clouds segmentation and fused 3-d giou-11 loss. IEEE TNNLS, 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Ruiyuan Gao, Chenchen Zhao, Lanqing Hong, and Qiang Xu. DiffGuard: Semantic mismatch- guided out-of-distributtio detection using pre-trained diffusion models. In ICCV, 2023.
Chongjian Ge, Junsong Chen, Enze Xie, Zhongdao Wang. Lanqing Hong. Huchuan Lu, Zhenguo Li, and Ping Luo. MetaBEV: Solving sensor failures for bev detection and map segmentation. In ICCV, 2023.
Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit- Yan Yeung, James T Kwok, and Yu Zhang. Mixture of tuning. arXiv preprint arX1v:2312.12379, 2023.
lora experts for vision-language instruction
Nicholas Guttenberg. Diffusion with offset noise. https:// http::/wwww.rrolaaabb...o//oo/// filffusionwwit ofstt-oooonoooooo 2023.
Jianhua Han, Xiwen Liang, Hang Xu, Kai Chen, Lanqing Hong, Chaoqiang Ye, Wei Zhang, Zhen- guo Li, Xiaodan Liang, and Chunjing Xu. Sodal0m: Towards large-scale object detection bench- mark for autonomous driving. arXiv preprint arXiv:2106. 11188,111111111111111111111 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Junjie Huang, Guan Huang, Zheng Zhu, Ye Yun, and Dalong Du. Bevdet: High performnroom multi- camera 3d object detection in bird-eye-view. arXiv preprint 12. 2021.
10

---

Published as a conference paper at ICLR 2024
Yuanfeng Ji, Zhe Chen, Enze Xie Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. DDP: Diffusion model for dense visual prediction. In ICCV, 2023.
Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit- Yan Yeung. et al. Coda: A real-world road comner case dataset for object detection in autonomous driving. arXiv preprint Mokiv:2203.07722 2022.
Pengxiang Li, Zhili Liu, Kai Chen, Lanqing Hong, Yunzhi Zhuge, Dit- Yan Yeung, Huchuan Lu, and Xu Jia. Trackdiffusion: Multi-object tracking data generation via diffusion models. arXiv preprint arXiv:2312.00063 2023a.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jac Lee. Gligen: Open-set grounded text-to-image generation. In CVPR, 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In ECCV, 2014.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Zhijian Liu, Haotian Tang, Alexander Amini, Xingyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In ICRA, 2023a.
Zhili Liu, Jianhua Han, Kai Chen, Lanqing Hong, Hang Xu, Chunjing Xu, and Zhenguo Li. Task- customized sulervupe pre-training with scalable dynamic routing. In AAAI, 2022b.
Zhili Liu, Kai Chen, Yifan Zhang. Jianhua Han, Lanqing Hong, Hang Xu, Zhenguo Li, Dit-Yan Ye- ung. and James Kwok. Geom-erasing: Geometry-drin removal of implicit concept in diffusion models. arXiv preprint arXiv:2310.05873, 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishknn, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. resolution image synthesis with latent diffusion models. In CVPR, 2022.
High-
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2020.
Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird's- eye view layout. arXiv preprini urXiv:2301.0444
2023.
Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: En- abling holistic multi-view image generation with correspondence-awarr diffusion. arXiv preprint arXiv:2307.01009, 2023.
Hung- Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Con- sistent view synthesis with pose-guided diffusion models. In CVPR, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
11

---

Published as a conference paper at ICLR 2024
Su Wang. Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In CVPR, 2023a.
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Semantic image synthesis via diffusion models. arXiv preprini arXiv:2207.0000 2022.
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards real-world-driver world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023b.
Xiaofeng Wang, Zheng Zhu, Yunpeng Zhang, Guan Huang, Yun Ye, Wenbo Xu, Ziwei Chen, and Xingang Wang. Are we ready for vision-centric driving streaming perception? the asap bench- mark. In CVPR, 2023c.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tunc-a-video: One-shot tuning of image diffusion models for text-to-video generation. CVPR, 2023a.
Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffu- sion models. arXiv preprint arXiv:2308.06 2023b.
Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout. arXiv preprint 2.X1v:2308801661,, 2023a.
Yijun Yang, Ruiyuan Gao, Xiaosen Wang, Nan Xu, and Qiang Xu. Mma-diffusion: Multimodal attack on diffusion models. arXiv preprint arXiv:2311.175// 2023b.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023a.
Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. arXiv preprint arXiv:2303.0966
2023b.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: unified predictor- corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867, 2023.
Ziyang Zheng, Ruiyuan Gao, and Qiang Xu. Non-cross diffusion for semantic consistency. arXiv preprint arXiv:23/2.00820. 2023.
LIU Zhili, Kai Chen, Jianhua Han, HONG Lanqing, Hang Xu, Zhenguo Li, and James Kwok. Task- customized masked autoencoder via mixture of ctuster-conditional experts. In ICLR, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. In IJCV, 2019.
Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmentation. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A
OBJECT FILTERING
e T
230 366
(3) All object boxes
(b) visible boxes for Front Left view (c) visible boxes for Back Right view
Figure 10: Bounding box filtering to each view. The dashed lines in (b-c) represent the r-axis of the camera's ss coordinates. Boxes retained only if they have at least point in the positive half of the z-axis in each camera's coordinate.
In Equation 6, we employ foiz for object filtering to facilitate bootstrap learning. We show more details of fuiz here. Refer Figure 10 for illustration. For the sake of simplicity, each camera's Field Of View (FOV) is not considered. Objects are defined as visible if any corner of their bounding boxes is located in front of the camera (i.e., 2 > 0) within each camera's coordinate system. The application of fuiz significantly lightens the workload of the bounding box encoder, evidence for which can be found in Section
B MORE EXPERIMENTAL DETAILS
Semantic Classes for Generation. To support most perception models on nuScenes, we try to include semanties commonly used in most settings (Huang al., 2021; Zhou & Krähenbühl, 2022; Liu et al., 2023a; Ge et al., 2023). Specifically, for objects, ten categories include car, bus, truck, trailer, motorcycle, bicycle, construction vehicle, pedestrian, barrier, and traffic cone. For the road map, eight categories include drivable area, pedestrian crossing, walkway, stop line, car parking area, road divider, lane divider, and roadblock.
Optimization. We train all newly added parameters using AdamW (Loshchilov & Hutter, 2019) optimizer and constant learning rate at 86-5 and batch size 24 (total 144 images for views) with linear warm-up of 3000 iterations, and set y² 0.2.
C ABLATION ON NUMBER OF ATTENDING VIEWS
Table 5: Ablation number of attending views. Evaluation results from CVT the synthetic nuScenes validation set, without M {0} in CFG scale = 2.
Attending Num.
FID +
Road mloU f
Vehicle mloU f
13.06
58.63
26.41
(ours)
14.46
59.31
27.13
5 (all)
14.76
58.35
25.41
In Table 5, we demonstrate the impact of varying the number of attended views on evaluation re- sults. Attending a single view yields superior FID results; the reduced influx of information from neighboring views simplifies optimization for that view. However, this approach compromises mloU, also reflecting less consistent generation, as depicted in Figure 11. Conversely, incorporat- ing all views deteriorates performance across all metrics, potentially due to excessive information causing interference in cross-attention. Since each view has an intersection with both left and right views, attending one view cannot guarantee consistency, especially for foreground objects, while attending to more views requires more computation. Thus, we opt for 2 attended views in our main paper, striking balance between consistency and computational efficiency.
13

---

Published as a conference paper at ICLR 2024
Attend to 2 views
Attend to I view
Comparison between different numbers of attending views. Only attending to one view results in worse multi-camera consistency.
Figure 11:
D QUALITATIVE COMPARISON WITH BEVGEN
Figure 12 illustrates that MAGICDRIVE generates images with higher quality compared to BEV- Gen (Swerdlow al., 2023), particularly excelling in objects. Such enhancement can be attributed to MAGICDRIVE'S utilization of the diffusion model and the adoption of customized condition injection strategy.
Ground Truth
BEVGen
to gen. Low quality Wrong place
MAGIC
Qualitative comparison with BEVGen driving scene from nuScenes validation set. We highlight some areas with rectangles to ease comparison. Compared with BEVGen, image quality of objects from MAGICDRIVE is much better.
Figure 12:
E MORE RESULTS WITH CONTROL FROM DIFFERENT CONDITIONS
Figure 13 shows scene level control (time of day) and background level control (BEV map alter- ations). MAGICDRIVE can effectively reflect these changes in control conditions through the gener- ated camera views.
Generation from MAGIC Darve
Scene level: prompt "night, diffioutt lighting"
Background level: flpp wep boxcs from Lefe to right world coorslinace))
Figure 13: Showcase for scene-level control with MAGICDRIVE. The scene is from the nuScenes validation set.
14

---

Published as a conference paper at ICLR 2024
F MORE EXPERIMENTS WITH 3D OBJECT DETECTION
Table 6: Comparison about support for 3D object detection model (i.e., BEVFusion). MAGICDRIVE
generates 272x736 images for augmentation. Results are from tests on the nuScenes validation set.
CAM-Only
CAM+LiDAR
Epoch
Data
mAP
NDS f
mAP
NDS 1
0.5x
w/o synthetic data
30.21
32.76
too few epochs
too few epochs
w/ MAGICDRIVE
33.29 (+3.08)
36.69 (+3.93)
1x
w/o synthetic data
32.88
37.81
65.40
69.59
w/ MAGICDRIVE
35.40 (+2.52)
39.76 (+1.95)
67.86 (+2.46)
70.72 (+1.13)
2x
w/o synthetic data
35.49
40.66
68.33
71.31
w/ MAGICDRIVE
35.74 (+0.25)
41.40 (+0.74)
68.58 (+0.25)
71.34 (+0.03)
In Table 6, we show additional experimental results on training 3D object detection models using synthetic data produced by MAGICDRIVE. Given that BEVFusion utilizes a lightweight backbone (i.e., Swin-T (Liu et al., 2021)), model performance appears plateau with training through 1- 2x epochs (2x: 20 for CAM-Only and 6 for CAM+LiDAR), Reducing epochs can mitigate this saturation, allowing more varied data to enhance the model's perceptual capacity in both settings. This improvement is evident even when epochs for 3D object detection are further reduced to 0.5x x. Our MAGICDRIVE accurately augments street-view images with the annotations. Future works may focus annotation sampling and construction strategies for synthetic data augmentation.
G
MORE DISCUSSION
More future work. Note that MAGICDRIVE-gone generrted street views can currently only perform as augmented samples to train with real data, and is exciting to train detectors solely with generated data, which will be explored in the future. More flexible usage of the generated street views beyond data augmentation, especially incorporation with generative pre-training (Chen et al., 2023a; Zhili et al., 2023), contrastive learning (Chen et al., 2021: Liu et al., 2022b) and the large language models (LLMs) (Chen et al., 2023b; Gou et al., 2023), an appealing future research direction. It is also interesting utilize the geometric controls in different circumstances beyond 3D scenarios (e.g., multi-object tracking (Li et al., 2023a) and concept removal (Liu et al., 2023b)).
H DETAILED ANALYSIS ON 3D OBJECT DETECTION WITH SYNTHETIC DATA
Table 7: Per-class performance comparison with BEVFusion for 3D object detection with 1x set- ting. Results are tested on the nuScenes validation set.
Data
mAP
car
cone
barrier
bus
ped. motor. truck bicycle trailer constr.
BEVFusion
32.88
50.67
50.46 48.62 37.73 35.74 30.40 27.54
24.85
15.56
7.28
+ MAGICDRIVE
35.40
51.86
53.56
51.15
40.43 38.10
33.11 29.35
27.85
18.74
9.83
Difference
+2.52 +1.20 +3.10 +2.53 +2.70 +2.36 +2.71 +1.11
+
+3.00
+3.19
+2.55
We provide per-class AP for 3D object detection from the nuScenes validation set using BEVFusion in Table 7. From the results, we observe that. firstly, the improvements for large objects are sig- nificant, for example, buses, trailers, and construction vehicles. Secondly, objects with less diverse appearances, such as traffic cones and barriers, show more improvement, especially compared to trucks. Thirdly, we note that the improvement is marginal for cars, while significant for pedestri- ans, motorcycles, and bicycles. This may be because the baseline already performs well for cars. For pedestrians, motorcycles, and bicycles, even though distant objects from the ego car are gen- erated less faithfully, MAGICDRIVE can synthesize high-quality objects near the ego car, as shown in Figure 1 17-18. Therefore, more accurate detection of objects near the ego contributes to im- provements for these classes. Overall, mAP improvement comes with promotion in all classes' AP, indicating MAGICDRIVE can indeed help the training of perception models.
15

---

Published as a conference paper at ICLR 2024
I MORE RESULTS FOR BEV SEGMENTATION
BEVFusion also capable of BEV segmentation and considers most of the classes we used in the BEV map condition. Due the lack of baselines, we present the results in Table 8 to facilitate com- parison for future works. As can be seen, the x resolution does not outperform the x20x400 resolution. This is consistent with the results from CVT in Table on the Road segment. Such results confirm that better map controls rely on maintaining the original aspect ratio for generation training (i.e., avoiding cropping on each side).
Table 8: Generation fidelity to BEV map conditions. Results are tested with BEVFusion for BEV segmentation on the nuScenes validation set.
mIOU for classes
Methods
resolution
CAM-only
CAM+LiDAR
Oracle Oracle
57.09
62.94
224x400
52.72
58.49
MAGICDRIVE
224x400
30.24
48.21
MAGICDRIVE
272x736
28.71
47.12
J GENERALIZATION OF CAMERA PARAMETERS
To improve generalization ability, MAGICDRIVE encodes raw camera intrinsic and extrinsic pa- rameters for different perspectives. However, the generalization ability is somewhat limited due nuScenes fixing camera poses for different scenes. Nevertheless, we attempt to exchange the intrinsic and extrinsic parameters between the three front cameras and three back cameras. The comparison is shown Figure 14. Since the positions of the nuScenes cameras are not symmetrical from front to back, and the back camera has 120 FOV compared to the 70 FOV of the other cam- eras, clear differences between front and back views can be observed for the same 3D coordinates.
70
120
120
70%
Figure 14: To show the generalization ability of learned camera encoding, exchange the camera parameters between front cameras and back cameras. The 3D position is the same for two generations. We highlight some areas (with box boxes)
16

---

Published as a conference paper at ICLR 2024
K MORE GENERATION RESULTS
We show some corner-case (Li et al., 2022) generations in Figure 15, and more generations in Figure 16-Figure 18.
Figure 15: Generation from MAGICDRIVE with corner-case annotations.
17

---

Published as a conference paper at ICLR 2024
Figure 16: Generation from MAGICDRIVE with annotations from nuScenes validation set.
18

---

Published as a conference paper at ICLR 2024
B
A
Figure 17: Generation from MAGICDRIVE with annotations from nuScenes validation set.
19

---

Published as a conference paper at ICLR 2024
SHIICENIEERSS
MI
Figure 18: Generation from MAGICDRIVE with annotations from nuScenes validation set.
20


Reviews:
[{"review_id": "anGYCnTWqM", "reviewer": "Reviewer_dbzg", "summary": "The paper proposes MagicDrive - a Bird's-eye-view(BEV)-to-street-view image generation method. Given a BEV road map, 3D bounding boxes for objects, the camera pose, and an input prompt it generates a consistent, multi-view image set for autonomous driving purposes. It is capable of scene, background and foreground control by prompting - lighting conditions, weather, object orientation, object deletion are available. \n\nThe main paper contributions are the a view-consistent image generation and 3D bounding box encoding for objects, as opposed to previous approaches that used only the BEV map.\n\nThe algorithm yields favorable visual results compared to similar methods (15-16 FID vs 20+) and the augmented data it generates improves upon the BEVFormer 3D object detection (\\~+2mAP, depending on input modality) and Cross View Transformer vehicle and road mIoU (\\~4-5%) on the nuScenes validation dataset.", "questions": "1. Why are the BEVFusion numbers much lower? Why not use BEVFusion for the BEV segmentation as well?\n2. If the aim is just to generate novel views, why not add additional elements to the bounding box images and use controlNet image encoding? See [1*] for inspiration.\n3. If the data augmentation strategy works so well, why not start with a state-of-the art method such as [2*] and see what it can be improved from there?\n4. Why not present other methods for consistent view generation? Arguably [4*] deals with the same problem; the scope is different, but they also have reasonable depth maps.\n5. The method is heavily reliant on nuScenes; how would you consider improving generalization?\n\n___\n[1*]Li, X., Zhang, Y., & Ye, X. (2023). DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model. arXiv preprint arXiv:2310.07771.\n[2*] Hu, H., Wang, F., Su, J., Hu, L., Feng, T., Zhang, Z., & Zhang, W. EA-BEV: Edge-aware Bird’s-Eye-View Projector for 3D Object Detection.\n[3*]Höllein, L., Cao, A., Owens, A., Johnson, J., & Nießner, M. (2023). Text2room: Extracting textured 3d meshes from 2d text-to-image models. arXiv preprint arXiv:2303.11989. https://github.com/lukasHoel/text2room\n[4*]Bahmani, S., Park, J. J., Paschalidou, D., Yan, X., Wetzstein, G., Guibas, L., & Tagliasacchi, A. (2023). Cc3d: Layout-conditioned generation of compositional 3d scenes. arXiv preprint arXiv:2303.12074. https://github.com/sherwinbahmani/cc3d", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- consistent cross-camera image generation\n    - a cross-view attention model with neighboring views \n- better problem modelling compared to older methods\n    - 3D object bounding box and camera pose allows a wider array of edits and more accurate terrain representation\n    - prompting the diffusion model allows for more diverse output images", "weaknesses": "## Summary ##\nA view-consistent UNet generation method and bounding box inputs for a controlNet BEV-to-RGB are the main contributions. Apart from the benefit of encoding bounding boxes, unclear whether the chosen consistency method is ideal.\n## Details ##\n\n- engineering work / limited contributions\n    - ControlNet stable diffusion pipeline coupled with a multi-view conditional UNet  \n        - there are other consistency methods - inpainting, panorama input, feature volumes - why is this cross-view attention module the best choice?\n- limited comparisons, different baseline numbers\n    - the authors use BEVFormer for some comparisons and CVT for others \n        -  for BEVFormer the reported numbers are significantly lower compared to the original paper and I don't believe it's only the resolution; no numbers match\n- method not mature enough\n    - to the best of my knowledge, neither of the two baselines (BEVGen/BEVControl) have been accepted at a major conference; furthermore, MagicDrive disregards other practical considerations such as temporally-consistent frames [1*]\n___\n[1*] DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model. arXiv preprint arXiv:2310.07771.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IINcdFVyM7", "reviewer": "Reviewer_jXFZ", "summary": "The goal of this paper is to have multi-modality control on street scene generation process. Overall idea of this paper is to use ControlNet framework on top of pre-trained stable diffusion model to support conditional generation over street view dataset. The complexity comes in terms of how to design multi-modality conditions for the ControlNet condition. For this purpose, the authors introduce various cross attention over their conditions to fuse their conditions onto scene representation (they also feed non-scene related condition directly to stable diffusion). The training follows ControlNet paradigm with classifier-free guidance to encourage output more aligned with conditions. Result-wise, they compare with BEVGen and BEVControl on nuScenes. The experiment aims to reveal they produce more realistic images and have better control over output space for street view generation task.", "questions": "1. Do you have different CFG weights for different conditions? If so, I am curious on how you make that work.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is very nicely organized and written. \n2. The quality of the generated street view is realistic\n3. We can see MagicDrive has more precise control on street generation than baselines", "weaknesses": "1. The main concern is for their marginal technical contribution. The proposed method is ControlNet applied into street view generation setting with multi-modality condition. The novelty probably lies in how to organize the condition into controlNet setting, which might not sufficient for acceptance. \n2. MagicDrive does not ensure consistency across adjacent frames after checking their website demo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "go0meXlWEb", "reviewer": "Reviewer_UU4E", "summary": "This paper introduces a novel framework for generating street view imagery with diverse 3D geometry controls such as camera poses, road maps, and 3D bounding boxes, using tailored encoding strategies. Existing methods primarily focus on 2D control which limits their utility in 3D perception tasks essential for autonomous driving. This paper consider a BEV view input, and input these control through encoding each of the information and insert these in the cross attention inside the diffusion UNet. In order to ensure the consistency between different views. It also introduce the cross-view attention for the training.", "questions": "Diffusion models are costly, Could the framework be extended or modified to handle real-time or near real-time massive generation requirements, which are crucial for applications in autonomous driving systems? This also related to handle dynamic entities in the scene such as moving vehicles or pedestrians, especially when synthesizing views over a period of time?\n\nHow closely does the synthetic data correlate with real-world data and what measures were taken to ensure the accuracy and reliability of this synthetic data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Adopting diffusion for driving view synthese which trying to solve the data limitation in corner cases for self-driving is important. \n\n- The overall strategy is sound and the paper proposed reasonable ways to encoding different information. Including Scene-level Encoding,  3D Bounding Box Encoding,  Road Map Encoding, these encoding are well organized and normalized in inserting to the cross attention module.  It also enables the final multi-level control of the generation. \n\n- Other modules such as cross-view module help in image synthesis consistency. \n\n- The experiments, show that it outperforms the other baselines such as BEVGen and BEV-Control, for synthesizing multi-camera views.", "weaknesses": "- the synthesized views are impressive,  the experiments are conducted in 700 street-view scenes for training and 150 for validation, which is a much smaller scale than the real-world senario. Wonder how to possiblly make it generalizable for real world domain. Does this be helpful to improve the detection & other understanding tasks when the data is large.\n\n- In addition, not only for the dark scene, many generated instances such as human can be distorted with diffusion models. Wonder how that affects the detection accuracy for each subclass. The author provides overall accuracy in 3D object detection, may also analysis the details how to mix the synthesized images and real-images for training the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BfWtEzzmL7", "reviewer": "Reviewer_TY6p", "summary": "This work focus on street view generation conditioned on BEV layout.\nThe authors harness the power of pretrained stable diffusion model and ControlNet to generate realistic images.\nCross view attention is applied to achieve better multi-view consistency.\nExperiments are conducted on nuscenes to demonstrate superior realism over the baselines.\nThe generative results can serve as data augmentation for downstream task to boost the performance.", "questions": "Equation 9 is not closed in $||$\n\nWhat if apply the data augmentation to the SOTA camera 3D detection models, can you achieve the new SOTA on nuscenes?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Decent and realistic results. Some images are hard to distinguish unless zoom in.\n2. The overall pipeline is sound to me, and compare to the baselines, it shows improved realism and better multi-camera consistency.\n3. It can boost downstream perception performance.\n4. Clear writing, easy to understand", "weaknesses": "1. The paper’s claim of “geometry control” appears to be somewhat overstated. Geometry encompasses more than just pose and Box3D; it also includes shape, topology, and the ability to modify any object within the scene.\n2. The consistency aspect of the results is not fully realized. While I acknowledge that the multi-camera consistency is superior to that of the baselines, the broader aspect of consistency, such as consistency from novel viewpoints (e.g., moving cameras away from the original view, rotating it 360 degrees, modifying the focal length to zoom into details in distant regions), seems to be lacking. Based on my observations from the website and deductions from the approach, achieving such consistency with the current representation seems highly unlikely.\n3. The novelty of this work is unclear to me, as I am not very familiar with this topic. Upon a quick review of BEVGen and BEVControl, it appears that the main difference lies in the new modeling of 3D bounding boxes (in this work, the authors decouple the 3D boxes and road maps and model them separately), the use of Stable Diffusion, and cross-view attention. However, none of these elements seem to be significantly innovative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)