Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
Authors: Kai Zeng
Zhanqian Wu
Kaixin Xiong
Xiaobao Wei
Xiangyu Guo
Zhenxin Zhu
Kalok Ho
Lijun Zhou
Bohan Zeng
Ming Lu
Haiyang Sun
BING WANG
Guang Chen
Hangjun Ye
Wentao Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Autonomous Driving
Driving World Model
Perception Tasks
Synthetic Data
Primary area: applications to robotics, autonomy, planning

Abstract: Recent advancements in driving world models enable controllable generation of high-quality RGB videos or multimodal videos. 
Existing methods primarily focus on metrics related to generation quality and controllability. 
However, they often overlook the evaluation of downstream perception tasks, which are {\bf really crucial} for the performance of autonomous driving. 
Existing methods usually leverage a training strategy that first pretrains on synthetic data and finetunes on real data, resulting in twice the epochs compared to the baseline (real data only). 
When we double the epochs in the baseline, the benefit of synthetic data becomes negligible.
To thoroughly demonstrate the benefit of synthetic data, we introduce Dream4Drive, a novel synthetic data generation framework designed for enhancing the downstream perception tasks.
Dream4Drive first decomposes the input video into several 3D-aware guidance maps and subsequently renders the 3D assets onto these guidance maps.
Finally, the driving world model is fine-tuned to produce the edited, multi-view photorealistic videos, which can be used to train the downstream perception models.
Dream4Drive enables unprecedented flexibility in generating multi-view corner cases at scale, significantly boosting corner case perception in autonomous driving. 
To facilitate future research, we also contribute a large-scale 3D asset dataset named DriveObj3D, covering the typical categories in driving scenarios and enabling diverse 3D-aware video editing.
We conduct comprehensive experiments to show that Dream4Drive can effectively boost the performance of downstream perception models under various training epochs. 
Project website: \url{https://wm-research.github.io/Dream4Drive/}.

Published as a conference paper at ICLR 2026
RETHINKING DRIVING WORLD MODEL AS SYN- THETIC DATA GENERATOR FOR PERCEPTION TASKS
Kai Zeng!.2. 1.2. Zhanqian Wu Kaixin Xiong Xiaobao Wei Xiangyu Guo
Zhenxin Zhu? Kalok Ho Lijun Zhou² Bohan Zeng' Ming Lu
Haiyang Sun"i, Bing Wang² Guang Chen" Hangjun Y Wentao Zhang 1.4.52
Peking University Xiaomi EV Huazhong University of Science and Technology
4 Beijing Key Laboratory of Data Intelligence and Security (Peking University)
5 Zhongguancun Academy
Detection Performance with Scaling Training Epochs
Tracking Performance with Scaling Training Epochs
0.51
0.35
0.50
0.34
0.33
0.49
AMOTA
NDS
0.32
0.48
Real (28130))
0.31
Real (281330)
Panacca (+28130))
Panacea (+28133)
0.47
SubjectDrive (+2881
0.30
SubjectDrive (0)
Ours (+420) -2%)
(-420, <2%)
1x
1x
3x
Figure 1: Dream4Drive demonstrates the effectiveness of synthetic data: with fewer than 2% syn- thetic samples, consistently improves detection and tracking across epochs, outperforming previ- data augmentation baselines under fair evaluation. 1x denotes the baseline training epochs; 2x and 3x represent twofold and threefold increases, respectively.
ABSTRACT
Recent advancements driving world models enable controllable generation high-quality RGB videos multimodal videos. Existing methods primarily fo- cus on metrics related to generation quality and controllability. However, they often overlook the evaluation of downstream perception tasks, which are really crucial for the performance of autonomous driving. Existing methods usually leverage training strategy that first pretrains synthetic data and finetunes real data, resulting in twice the epochs compared to the baseline (real data only). When double the epochs in the baseline, the benefit of synthetic data becomes negligible. To thoroughly demonstrate the benefit of synthetic data, we introduce Dream4Drive, a novel synthetic data generation framework designed for enhane- ing the downstream perception tasks. Dream4Drive first decomposes the input video into several 3D-aware guidance maps and subsequently renders the 3D as- sets onto these guidance maps. Finally, the driving world model is fine-tuned to produce the edited, multi-viow photorealistic vidcos, which can be used to train the downstream perception models. Dream4Drive enables unprecedented flexi- bility in generating multi-view corner cases at scale, significantly boosting corner case perception in autonomous driving. To facilitate future research, also con- tribute a large-scale 3D asset dataset named DriveObj3D, covering the typical cat- egories in driving scenarios and enabling diverse 3D-aware video editing. We con- duct comprehensive experiments to show that Dream4Drive can effectively boost the performance of downstream perception models under various training epochs. Project website: reseer/rc github. i o Dream4Drive/
"Equal Contribution. Email: awweforekke@gmiil.ccco
Project Leader.
EEqual Corresponding Author. Email: wentac. zhang@pku.edu.co

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Perception tasks such as 3D object detection (Li et al. 2022 2024b Wang et al., 2023a: 2025) and 3D tracking (Wang et al.] 2023a Zhang et al.] 2023c Han et al. 2025 Li et al. 2025b). which support planning and decision making (Jiang et al.) 2023 Hu et al. 2023 Shan et al., 2025 Hao ct al. 2024), are extremely important in autonomous driving. The performance of perception mod- els, however, is highly dependent on large-scale annotated training datasets (Caesar et al., 2020) Wang et al. 2023b). To ensure reliability in rare but critical safety scenarios, it is essential to gather adequate long-tail data. Although the autonomous driving community has developed a thorough 3D) annotation pipeline to facilitate data acquisition (Zhao et al. 2025b), collecting long-tail data remains highly time-consuming and labor-intensive.
Driving world models based diffusion and ControlNet (Rombach et al. 2022 Zhang et al. 2023b)
generate synthetic data from scene layouts and text (Gao et al.) 2023] Wen et al. 2024 Guo et al. 2025), but offer limited control over object pose and appearance. reducing data diversity (Li al. 2025ap. Editing-based methods (Singh et al. 2024 Liang et al. (2225)] Yu et al., (025) improve this by inserting objects using reference images and 3D boxes, yet their single- view nature restricts use in multi-view BEV perception. Reconstruction based approaches (NeRF, 3DGS) (Chen et al. [021]] Zanjani et al. 2025) provide geometric control but suffer from artifacts due to sparse views and lack illumination modeling, causing inconsistencies between inserted objects and the background.
More importantly, we argue that the data augmentation experiments of previous methods (Wen et al.) 2024 Li et al. (024a) are unfair, as they usually leverage training strategy that first pretrains on synthetic data and finetunes on real data, resulting in twice the epochs compared to the baseline (real data only). We find that, under the same number of training epochs, large amounts of synthetic datasets offer little to no advantage and can even perform worse than using real data alone. As shown in Fig. T under the 2x epoch setting, models trained exclusively and NDS compared to those trained on real and synthetic data. Given the importance of downstream perception tasks for autonomous driving, we believe it is essential to rethink the effectiveness of the driving world model as a synthetic data generator for these tasks.
real data achieve higher mAP
To reevaluate the value of synthetic data, introduce Dream4Drive, a novel 3D-aware syn- thetic data generation framework designed for downstream perception tasks. The core idea of Dream4Drive is to first decompose the input video into several 3D-aware guidance maps and subse- quently render the 3D assets onto these guidance maps. Finally, the driving world model is fine-tuned to produce the edited, multi-view photorealistic videos, which can be used to train the downstream perception models. Consequently, we can incorporate various assets with different trajectoriesse.gg views, poses, and distance) into the same scene, significantly improving the diversity of the synthetic data. As shown in Fig. M under identical training epochs (Ix, 2x, 3x), our method requires only 420 synthetic samples less than 2% of real samples- to surpass prior augmentation methods. To be best of our knowledge, are the first to demonstrate under fair comparisons that synthetic data can provide real benefits beyond training solely on real data.
Specifically, Dream4Drive leverages a multi-view video inpainting model finetuned from the Diffu- sion Transformer (Peehles & Xie, 2023). Unlike prior methods that rely on sparse spatial controls (e.g.. BEV maps and 3D bounding boxes), Dream4Drive uses dense 3D-aware guidance maps (Yu) & Smith 2019 Liang et al. 2025a) (e.g., depth, normal, edge, cutout, and mask) to preserve the geometry and appearance of the original video, while editing them by rendering 3D assets into these maps. This design enables instance-level, cross-view consistent video editing, ensuring both visual realism and geometric fidelity. The generated videos not only achieve superior quality but can also be directly used to train state-of-the-art perception models (Wang et al., 2023a).
To facilitate diverse 3D-aware video editing, we design a pipeline that automatically acquires high- quality 3D assets based on target scene image or video of desired asset category. We first apply the image segmentation model (Ren et al. 2024 Lin et al. 2024) to localize and crop objects of the specified category, then employ the image generation model (Wu et al. 2025) to generate multi-view consistent images of the target object. These images are fed into a mesh generation model (Hun- yuan3D et al.) 2025) to generate high-quality 3D assets. We present DriveObj3D, large-scale 3D asset dataset that encompasses typical categories found in driving scenarios, to support future research. Our main contributions are:
2

---

Published as a conference paper at ICLR 2026
DriveObj3D (3D Asset Library)
+
Real Data
Downstream Tasks
Dream4Drive
Dream4Drive-Ein
Detection
Geo. Transformation
Precise Annotation
Tracking
Geometric Variation
Appearance Variation
Synthetic Data
Dream4Drrive-Reinn
3D
Adapter
Condition
Diffusion Transformer
sdfru
Figure 2: The illustration of Dream4Drive, which provides precise annotations, geometric variety, and appearance diversity to improve downstream perception tasks.
We find that previous data augmentation methods are evaluated unfairly: under the same training epochs, hybrid datasets do not show any advantage over real data alone.
We propose Drive, a 3D-aware synthetic data generation framework that edits the video with dense guidance maps, producing synthetic data with diverse appearances and geometric consistency.
We contribute large-scale dataset named DriveObj3D, covering the typical categories in driving scenarios for 3D-aware video editing.
Extensive experiments across different training epochs show that adding less than 2% syn- thetic data can significantly improve perception performance, highlighting the effectiveness Drive4Drive.
RELATED WORK
Video Generation Autonomous Driving. High-quality data is crucial for training perception
models in autonomous driving, motivating growing interest in driving video generation. Early ap-
proaches (Yang et al.] 2023; Gao et al.) [023 Wen et al. (024) employ diffusion models with Con- trolNet, conditioned
BEV maps (Ma al. 2024b) and 3D bounding boxes, generate paired
image data (Zhou et al. 2024 An et al. 2024 2025b Recent works (Gao et al., 2024a Jiang 2026 Li et al. (025c) adopt powerful Diffu- sion Transformers (DiTs) to further enhance generation quality. SubjectDrive (Huang et al. 2024a) uses an external subject bank enhance vehicle appearance diversity. However, these methods de- pend on original scene layouts, which limit geometric diversity and struggle to generate high-quality long-tail corner cases, thereby restricting their effectiveness for downstream perception.
al. 2024 Li al. 2024a Ji al. 2025 An
Video Editing in Autonomous Driving. To enrich scene diversity, object-level editing methods insert new objects into existing videos using reference images and 3D bounding boxes (Singh al.,
2024
Liang et al. 2025b] Buburuzan et al. [2025] |nn al. 2025a). More recent NeRF- (Mildenhall
ct al. 2021 and 3DGS- (Kerbl et al. (2023) approaches (Chen et al. 2021 Huang et al. 2024b Chen et al. 2024 Wei et al.) 2024 Zhu al. 2025) improve geometric fidelity, yet remain limited by sparse views, inconsistent lighting and restricted background diversity. In contrast, our approach builds on generative models and introduces novel 3D-aware video editing mechanism, enabling seamless insertion diverse 3D assets and producing geometrically and visually diverse data that effectively boosts downstream performance.

---

Published as a conference paper at ICLR 2026
3 DREAM4DRIVE
Our goal is to generate high-quality synthetic videos based on real video with ground-truth 3D box annotations and target 3D asset, creating synthetic data for training downstream perception models. We first introduce the preliminaries Sec. Sec.[3.I] then explain how to conduct 3D-aware scene editing in Sec. 3.2 and finally describe how render the edited video from the guidance maps in Sec. 3.3 The overall framework is shown in Fig.
3.1 PRELIMINARIES
Latent Diffusion Models (LDMs) (Rombach et al., 2022) address the high computational cost of diffusion models by operating in a lower- dimensionn latent space. Given an image an encoder & used to obtain the corresponding latent representation z = {x)). The forward diffusion process in the latent space is defined as gradual noising process:
((z, zn) (M, vanzo -.,)
where ar = II' (1 B.) with 3; being variance schedule. The reverse process is modeled by neural network co and parameterized as:
pe (2) = 1N t), Eo Oz(z,.)) t)),
(2)
Finally, decoder D maps the denoised latent variable back the image space, i.e., x' = -(zo).
ControlNets (Zhagg et al., 2023a) extend LDMs by incorporating additional conditioning signals c to provide finer control over the generation process. In particular, the reverse diffusion process is modified to condition on c:
Po (o c) = ([ t, c), Eo (z, c)).
The conditioning variable incorporate various forms of guidance, including spatial maps, semantic layouts, and other task-specific signals. By integrating c, ControlNets allow for more precise manipulation of the generation process and improving the quality of synthesized images.
3.2 3D-RWAR SCENE EDITING
C- N0,,N,
Inserted Asset
Anythina
Iprert E Bllonic
Joscs gllaasc
Ego
Vehicle
Space
Figure 3:
The illustration of 3D-aware scene editing. Given the input images, we first obtain the depth map, normal map, and edge map for the background and then render the object image and object mask for the target 3D asset.
To overcome the limitations of implicit 3D controls (like bounding boxes or BEV maps) in precisely placing assets, we introduce a new form of 3D-aware guidance maps.
For an input RGB image I E #xxx3 x3, we use Depth Anything (Yang et al. 2024) to obtain the depth map E H x1, The normal map N IR H x3 is then derived from the depth map. We also use OpenCV's Canny edge detector to get the edge map E E IR # x1, It is important to note that the depth, normal, and edge information within the foreground object regions are masked since our model needs to learn to generate an edited video based on the target 3D asset.
For target 3D asset in our DriveObj3D, we position within the 3D space of the original video based the provided 3D bounding boxes {B.) For each frame and each view, then use calibrated camera intrinsics K,, and extrinsics E, to render the target 3D asset. Therefore, we can obtain the object image O R #x x3 and object mask M R #xxx1
Once we have obtained the depth map. normal map, and edge map for the background, as well as the rendered object image and object mask for the target 3D asset, utilize fine-tuned driving
4

---

Published as a conference paper at ICLR 2026
world model to render the edited video based the 3D-aware guidance maps. This 3D 3D-rware scene editing pipeline effectively utilizes the accurate pose, geometry, and texture information provided by 3D assets, ensuring geometric consistency in the results generated. Notably, method does not depend on 3D bounding box embeddings for controlling object placement. Instead, we directly edit in 3D space, offering more intuitive and reliable way manage control.
3.3 3D-AWARE VIDEO RENDERING
guidanee maps
FEN
Encoder
Temporal
Patchify
Control Blool
Cross View Aftn
Gated
Lineur Unit
Fusion Net
Cross-Mttention
Base- Block
Cony ID
Spatial
Multi-Condition
Attention
FEN
Noise
weight copy
1
Fusion Adapter
Multi-Condition
(a) Diffusion Transformer
(b) Fusion Adapter
Figure 4: The illustration 3D-aware video rendering. Given the 3D-aware guidance maps, we employ multi-condition fusion adapter control the video generation of diffusion transformer, rendering the edited video.
The video generation process in recent methods (Wen et 2024 Li al. 2024a Guo et al. 2025) relies on sparse spatial controls, such as bird's eye view (BEV) maps and projected 3D bounding boxes. While this approach allows for some control over the synthesized content, often falls short sparse spatial controls, fine-tune
in achieving high-quality generation. Instead of relying driving world model to generate edited videos from the dense 3D-aware guidance maps, including depth map (D), normal map (N), and edge map (E) for the background and the image (O) and mask (M) for the foreground object.
The architecture of the multi-condition fusion adapter is shown in Fig. We first encode the five conditions using a VAE, and then apply different 3D embedders to patchify the latents. A FusionNet module then combines these five sets of features, described by the following equation:
Cx ),
Frusion Nut 3DEmbeddera (Varmbe (VAB(Ck))
where D, N, F, M denote the depth, normal, edge, object, and mask, respectively, and @ indi- cates concatenation along the channel dimension. The fused features are incorporated into the con- trol blocks of the DiT architecture, enriching semantic information and thereby facilitating instance- level spatial alignment, temporal consistency, and semantic fidelity. The outputs of the control blocks are further integrated into the base block. Additionally, spatial view attention mechanism is introduced to enhance cross-view coherence, which is especially beneficial in driving scenes.
We adopt rectified flow (Liu al. 2022) for stable sampling and classifier-free guidance (Ho & Salimans 2022) to balance text with multiple 3D geometric conditions, thereby improving control- lability. The main training objective is simplified diffusion loss that predicts the noise component at each timestep;
(5) where Gaussian noise, eg the predicted noise, z, the noisy latent at timestep t, and all condi- tioning signals. To achieve precise instance- control, we introduce Foreground Mask Loss as Ji al. 2025) and an LPIPS loss (Zhang al. 2018). The final training objective
Laitiusson = -E.....eA/0o.) [lle eg (Zr. t, c)
Liaut
Adiifusson Laittusion
Amask Cmask Appip CLPIPP
5

---

Published as a conference paper at ICLR 2026
In experiments, the weights empirically set Adiftusion 1.0, Amask 0.1, and Appips 0.1. Notably, our training framework requires no expensive 3D annotations, relying solely on RGB videos and their 3D-aware guidance maps, which can be generated in real time using off-the-shelf tools. This approach significantly reduces training costs. More details are included in the Appx.
DRIVEOB13D
The diversity of the synthesized scenes is directly determined by the richness the asset library used for insertion. To build large-scale 3D assets for diverse 3D-aware video editing, we design simple yet effec- tive pipeline. The core idea is to de- compose the asset generation process into three steps: (i) 2D instance seg- mentation; (ii) multi-view image gen- eration; (iii) 3D mesh generation. As shown in Fig. [S] the pipeline inputs a video or image and the target as- category, and generates 3D mesh for downstream applications.
Image Seg
Image Gen
Mexh Gen
Front
Lefl view
Back view
Figure 5: The illustration of creating 3D asset in DriveObj3D. We first apply a segmentation model to seg- ment the target object, then generate multi-view images, and finally create a 3D mesh from those images.
Concretely, we first localize and segment the target object. Given an input image I and category label class, Grounded-SAM (Ren et al. (2024) is applied to segment the object Iuggtt- To overcome the occlusion of target object, we employ a multi-view image generation model Qwen-Image (Wu
et al. 2025). Conditioned on Junget. Qwen-Image synthesizes set of novel views {I, Tumi- 1. which are subsequently fed into multi-view reconstruction model Hunyuan3D (Hunyuan3D et 2025)
to recover the final 3D mesh.
Barrier
Hlcycle
Bus
Car
Con vehicle
Motoreycle
Pedes
cone
Trailer
Trick
Figure 6: Comparison of 3D asset generation different methods. Our simple yet effective method produces better 3D assets across diverse categories in autonomous driving, outperforming existing baselines. Cohive construction vehicle; Pdes is Pedestrian: T_cone traffic.com ccone.
As shown in Fig. 6, assets generated by Text-to-3D methods (Xiang et al. 2025 often exhibit style inconsistencies with the original data, while single-view approaches (Hunyuan3D et al. 2025) tend to produce incomplete assets. In contrast, our simple yet effective pipeline leverages multi-view synthesis generate complete and high-fidelity assets, large-scale downstream driving tasks, we construct a diverse asset dataset, DriveObj3D, covering a wide range of categories in driving scenarios in Fig. 7 All assets will be released publicly.
under severe occlusions. To support

---

Published as a conference paper at ICLR 2026
Cal 2001
ENDD D07) CLARN
Figure 7: DriveObj3D dataset. large-scale collection of diverse 3D assets across typical driving categories, supporting scalable video editing and synthetic data generation.
Iv
2x
Dream (+420, <2%)
Dremeer Wang al. 20244
Macic
Panacza Wan
SubtectDrive Huung
Dream/(Drive (Ous)
(28130)
(28130)
2023
2004
2024g
MAP
34.5
36.1
35,4
37
MAYE
29.1
28.9
21.7
123.4
27.3
26.4
26.8
NDS 1
46.9
47.8
50.4
39.5
18.1
39.8
49.
50.2
50.6
Table 1: Comparison of detection under different training epochs. Wo only on the vehicle classes of cars, trucks, and buses.
indicates the evaluation of
5
EXPERIMENTS
5.1 SETUP
Datasets. We utilize the nuScenes dataset (Caesar et al. 2020) for building 3D assets and finetun- ing our video generation model. It comprises 1000 scenes in total, with 700 designated for training, 150 for validation, and 150 for testing. Each scene contains a 20-second multi-view video captured by 6 cameras. More details on inserted scene and asset selection are given in the Appx. A
Metrics. Following Panacea (Wen et al. 2024) and SubjectDrivve (Huang et al.) 2024a), we pri- marily evaluate how the generated data improves perceptual model performance on detection and tracking tasks. Detection metries include nuScenes Detection Score (NDS), mean Average Preci- sion (mAP), mean Average Orientation Error (mAOE), and mean Average Velocity Error (mAVE). Tracking metrics include Average MultiObject Tracking Accuracy (AMOTA), Precision (AMOTP), Recall, and Accuracy (MOTA).
5.2 MAIN RESULTS
Effectiveness for Downstream Tasks. We evaluate the effectiveness of Dream4Drive against prior driving world models, with detection and tracking results reported in Tab. Tab. and Tab. 2 While Panacea and SubjectDrive outperform the real dataset baseline at double training epochs, align- ing the training epochs shows minimal gains over using real data alone. In contrast, our method explicitly edits objects at specified 3D positions and leverages 3D-aware guidance maps to guide synthesis, generating accurately annotated videos that consistently improve downstream perception models. Remarkably, with only 420 inserted samples, our approach outper- forms prior methods that used the full set of synthetic data. Moreover, for the first time, synthetic data achieves performance that surpasses real data when training epochs are equal.
foreground-backgron
Effectiveness for Various Resolutions. As generative models continue to advance, the ability to synthesize high-resolution videos has become achievable (Gao et al. 2024b]a). To investigate the effect of high-resolution synthetic data downstream perception models, further conduct experiments for detection and tracking tasks at a resolution of 512 x 768, as reported in Tab, 3nd
Under both Ix and 2x epochs, real data and Dream4Drive significantly outperform their low- resolution counterparts (256 x 512, Tab. and 2 Remarkably, with high-resolution augmentation, Dream4Drive requires only 420 samples to achieve 4.6 point (12.7%) mAP increase and 4.1 point (8.6%) NDS improvement. Most of the gains come from large vehiele categories, includ- ing bus, construction vehicle, and truck; detailed AP for each category is provided in the Appx.

---

Published as a conference paper at ICLR 2026
1x Epochs
2x Epochs
Real (28130)
Dream4Drive
Real (28130)
Panacea Wen al. 2024
SubjectDrive (Hunn 2024a
Dream4Drive (Ours)
(+420, <2%)
AMOTA
30.1 137.9
31.2 135,4
34.1 134.1
33.7 135.3
33.7 135.3
34,4
AMOTP
133.5
Table 2: Comparison of tracking under different training epochs.
Epochs
2x Epochs
Epochs
Real
Naive Insert
Dream4Drive
Real
Naive Insert
Dream4Drive
Real
Naive Insert
Dream4Drive
MAP1
36.1
40.1
40.7
42.2
42.9
43.6
43.1
43.1
44.5
MATE
69.2
64.7
64.2
61.6
62.4
61.6
60.5
61.5
59.8
MAOR
56.7
49.0
48.0
43.2
37.5
39.4
45.7
38.9
40.1
mAVE
28.5
28.4
27.1
27.5
27.3
27.4
27.4
27.4
27.2
NDS
47.9
51.3
52.0
53.2
54.0
54.3
53.6
54.2
$5.0
Table Detection performance under different training epochs (Ix, 2x, 3x).
"Naive Insert" denotes
the direct projection of 3D assets into the original scene. Results are reported at 512x768 resolution.
Unlike prior augmentation paradigms, Dream4Drive consistently outperforms training on real data alone, regardless of the number of epochs, highlighting the value of high-quality synthetic data.
Quantitative and Qualitative Comparison with Naive Insertion. After extracting 3D assets, we can directly generate edited videos with projection. To assess the impact of 3D-aware video rendering versus direct insertion downstream tasks, conduct comprehensive evaluations, as reported in Tab and While direct insertion improves performance over real data alone, its results remain inferior our generative method due missing realism, such shadows and reflections. Interestingly, direct insertion achieves the highest mAOE, likely because the inserted assets perfectly align with the original bounding box orientations. Fig. presents visual comparisons between naive insertion and our generative approach across multiple scenes.
Naive Insertion
Ours
Naive Insertion
Ours
Figure 8: Comparison with naive insertion. As can be seen, Dream4Drive generates more realistic edited videos than naive insertion.
Comprehensive visualization of asset insertion results. More asset categories, insertion loca- tions, and corner case scenarios are comprehensively presented in the Appx. D
5.3 ABLATION STUDIES
Effect of Insertion Pusition. Dream4Drive edit videos by projecting 3D assets anywhere in a scene. To systematically evaluate the impact of insertion position on downstream model perfor- mance, categorize positions front, back, left, and right, shown in Tab.
8

---

Published as a conference paper at ICLR 2026
1x Epochs
2x Epochs
3x Epochs
Real
Naive Insert
Ours
Real
Naive Insert
Ours
Real
Naive Insert
Ours
AMOTA
32.8
36.5
37.9
39.7
42.2
42.6
41.3
42.2
43,5
AMOTP
134.0
128.7
128.0
125.1
124.0
123.3
124.1
123.7
121.3
MOTA f
28.1
31.7
33.1
35.6
37.3
37.4
36.8
37.5
38.5
RECALL 1
44.0
45.4
46.9
50.7
51.8
52.4
51.8
52.5
Table 4: Tracking performance under different training epochs (Ix, 2x, 3x).
"Naive Insert" denotes the direct projection of 3D assets into the original scene. Results reported 512x668 resolution.
Views
Distances
3D Asset Generation Methods
Speed
Front
Back
Left
Right
Close
Mid
Far
Trellis
Hunyuan3D
Ours
Low
Middle
High
mAP1
40.2
40.2
40.2
39.8
39.7
40.3
40.5
39,8
40.2
40.7
40,4
40.7
40.9
MATE
66.2
66.0
64.6
66.2
63.7
65.4
65.1
65.6
65.1
64.2
64.5
64.2
63.9
55.1
45.7
52.2
49.7
48.0
48.3
48.1
48.0
mAVE.
4
27.7
27.5
28.5
27.9
28.1
28.0
27.9
27.8
28.0
27.1
27.1
27.1
27.0
NDS
51.0
50.6
50.7
50.5
50.9
51.3
50.8
50.9
52.0
51.8
52.1
Table 5: Ablation Studies. We report detection performance across insertion positions and asset, with best results per block (Views, Distances, 3D Methods) in bold, at 512x768 resolution.
Results indicate that inserting assets in the front back yields similar performance, whereas left- side insertions outperform right-side ones, with 0.4 point increase in mAP, 0.9 point increase in NDS, and a 5.7 point reduction mAOE. This likely indicates dataset bias: most vehicles appear on the ego vehicle's left side, enhancing such corner cases improves model predictions, while right-side corner cases yield limited gains on the validation set.
We further examine the effect of insertion distance performance in Tab. Close insertions tend to perform poorly, likely due to the asset blocking the camera view, which interferes with the training of other instances. Distant insertions offer more effective augmentation, detectors often struggle with distant objects. Increasing their prevalence enhances detection performance.
Effect of 3D Asset Source. We observe that the source of inserted assets affects the quality of synthetic data, consistent with (Ljungbergh et al., 2025). We therefore investigate how different asset sources influence downstream perception performance.
As shown in Tab. 5 although Trellis (Xiang et al. 2025) can generate high-quality assets, its style does not fully match autonomous driving scenarios, which can lead artifacts and degraded quality when assets are inserted. negatively affecting downstream tasks. Hunyuan3D (Hunyuan3D et al.) 2025)'s 's single-view generation also underperforms our multiview approach, since single-image as- sets may be incomplete, whereas our method produces complete, high-quality 3D) assets.
5.4 TAKEAWAYS
We summarize the main observations from our experiments with perception model augmentation as follows:
Duplying original layouts to create synthetic data does not improve performance; instead, enhancing scenes by inserting new 3D assets is an effective strategy for augmentation.
High-resolution synthetic data offers greater benefits for data augmentation.
The placement of inserted assets influences the effectiveness of augmentation, highlighting biases present in the dataset.
Insertions at farther distances generally improve performance, while close-range insertions may introduce strong occlusions that hinder training.
Using assets from the same dataset reduces the domain gap between synthetic and real data, benefiting downstream model training.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
In this paper, we find that previous driving world models inaccurately assess the effectiveness of synthetic data for downstream tasks. To address this, we present Dream4Drive, 3D-aware syn- thetic data generation pipeline that synthesizes high-quality multi-view corner cases. To facilitate future research, we also contribute a large-scale 3D asset dataset named DriveObj3D, covering the typical categories in driving scenarios. Extensive experiments demonstrate that with less than 2% additional synthetic data, Dream4Drive consistently improves downstream perception, validating the effectiveness of synthetic data for autonomous driving.
ACKNOWLEDGMENTS
This work is supported by National Natural Science Foundation of China (92470121, 62402016), National Key R&D Program of China 2024YFA11110003) Zhongguaneun Academy, (Grant No.s C20250204, C20250602), and High-perfone perforreform Computing Platform of Peking University.
ETHICS STATEMENT
This work focuses on synthetic data generation for autonomous driving, aiming to improve down- stream perception through the proposed Dream4Drive framework. We build upon publicly available driving datasets and a curated 3D asset library, without collecting new human-subject data or per- sonally identifiable information. Potential risks include the misuse of synthetic driving scenarios for surveillance or malicious applications, as well as biases introduced by the underlying datasets and assets. We encourage responsible use of this research strictly for advancing safe and reliable autonomous driving.
REPRODUCIBILITY STATEMENT
We ensure reproducibility by providing detailed descriptions of algorithms, metrics, and experimen- tal settings in the main paper and appendix. All assets and code will be released upon publication.
REFERENCES
Ruichuan An, Sihan Yang. Ming Lu, Renrui Zhang. Kai Zeng. Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, et al. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11706 2024.
Ruichuan An, Sihan Yang. Renrui Zhang, Zijun Shen, Ming Lu, Gaole Dai, Hao Liang, Ziyu Guo, Shilin Yan, Yulin Luo, et al. Unictokens: Boosting personalized understanding and generation via unified concept tokens. arXiv preprint arXiv:2505. 14671, 2025a.
Ruichuan An, Kai Zeng, Ming Lu, Sihan Yang, Renrui Zhang, Huitong Ji, Qizhe Zhang, Yulin Luo, Hao Liang, and Wentao Zhang. Concept-as- tree: Synthetic data is all you need for vim personalization. arXiv preprint arXiv:2503. 12999, 2025b.
Ruichuan An, Sihan Yang, Ziyu Guo, Wei Dai, Zijun Shen, Haodong Li, Renrui Zhang, Xinyu Wei, Guopeng Li, Wenshan Wu, and Wentao Zhang. Genius: Generative fluid intelligence evaluation suite, 2026. URL /attaa:/
org/ httpgg/aasbe2....22/ 221./2222.......
Alexandru Buburuzan, Anuj Sharma, John Redford, Puneet K Dokania, and Romain Mueller. Mobi: Multimodal object inpainting using diffusion models. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1974-1984, 2025.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1621-11631 2020.
10

---

Published as a conference paper at ICLR 2026
Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Manivasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun. Geosim: Realistic video simulation via geometry-aware composition for self-driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7230-7240, 2021.
Ziyu Chen, Jiawei Yang. Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojeie, Sanja Fidler. Marco Pavone, et al. Omnire: Omni urban scene reconstruc- tion, arXiv preprint arXiv:2408. /6760, 2024.
Ruiyuan Gao, Kai Chen, Enze Xie. Lanqing Hong, Zhenguo Li, Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601, 2023.
Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrivedit: High- resolution long video generation for autonomous driving with adaptive control. arXiv preprint arXiv:2411.13807 2024a.
Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang. and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint urXiv:2405. 17398, 2024b.
Xiangyu Guo, Zhanqian Wu, Kaixin Xiong. Ziyang Xu, Lijun Zhou, Gangwei Xu, Shaoqing Xu, Haiyang Sun, Bing Wang, Guang Chen, et al. Genesis: Multimodal driving scene generation with spatio-temporal and cross-modal consistency. arXiv preprint arXiv:2506.074477 2025.
Liuyang Han, Jun Wang, Chen Li, Fazhan Tao, and Zhumu Fu. A novel multi-object tracking frame- work based on multi-sensor data fusion for autonomous driving in adverse weather environments. IEEE Sensors Journal, 2025.
Xiaoshuai Hao, Ruikai Li, Hui Zhang, Dingzhe Li, Rong Yin, Sangil Jung, Seung-In Park, Byun- gln Yoo, Haimei Zhao, and Jing Zhang. Mapdistill: Boosting efficient camera- based hd map construction via camera-lidar fusion model distillation. In European Conference on Computer Vision, pp. 166-183. Springer, 2024.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598. 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang. et al. Planning-oriee -rienned autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 17853-17861 2023.
Binyuan Huang, Yuqing Wen, Yucheng Zhao, Yaosi Hu, Yingfei Liu, Fan Jia, Weixin Mao, Tiancai Wang, Chi Zhang, Chang Wen Chen, et al. Subjectdrive: Scaling generative data in autonomous driving via subject control. arXiv preprint MXXx:22403.19438 2024a.
Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. S3gaussian: Self-supervised street gaussians for au- tonomous driving. arXiv preprint arXiv:2405.20323, 2024b.
Team Hunyuan3D, Shuhui Yang, Mingxin Yang, Yifei Feng, Xin Huang, Sheng Zhang, Zebin He, Di Luo, Haolin Liu, Yunfei Zhao, et al. Hunyuan3d 2.1: From images to high-fidelity 3d assets with production-ready pbr material. arXiv preprint arXiv:2506. 15442, 2025.
Yishen Ji, Ziyue Zhu, Zhenxin Zhu, Kaixin Xiong, Ming Lu, Zhiqi Li, Lijun Zhou, Haiyang Sun, Bing Wang, and Tong Lu. Cogen: 3d consistent video generation via adaptive conditioning for autonomous driving. arXiv preprini arXiv:2503.22233 2025.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient au- tonomous driving. Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pp. 8350- 2023.
11

---

Published as a conference paper at ICLR 2026
Junpeng Jiang, Gangyi Hong, Lijun Zhou, Enhui Ma, Hengtong Hu, Xia Zhou, Jie Xiang, Fan Liu, Kaicheng Yu, Haiyang Sun, et al. Dive: Dit-based video generation with enhanced control. arXiv preprint arXiv:2409.01595, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 12(4):139-1, 2023.
Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centric driving scene genera- 2024a.
tion. arXiv preprint arXiv:2412.0544
Jiusi Li, Jackson Jiang, Jinyu Miao, Miao Long, Tuopu Wen, Peijin Jia, Shengxiang Liu, Chunlei Yu, Maolin Liu, Yuzhan Cai, al. Realistic and controllable 3d gaussian-guide guided object editing for driving video generation. arXiv preprint arXiv:2508. 2025a.
Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit- Yan Yeung, et al. Coda: A real-world road corner case dataset for object detection autonomous driving. In European Conference on Computer Vision, pp. 406-423. Springer, 2022.
Ruikai Li, Xinrun Li, Mengwei Xie, Hao Shan, Shoumeng Qiu, Xinyuan Chang, Yizhe Fan, Feng Xiong, Han Jiang, Yilong Ren, al. Amap: Distilling future priors for ahead- aware online hd map construction. arXiv preprint 2rXiv:2512.191500 2025b.
Yongkang Li, Kaixin Xiong. Xiangyu Guo, Fang Li, Sixu Yan, Gangwei Xu, Lijun Zhou, Long Chen. Haiyang Sun, Bing Wang, et al. Recogdrive: A reinforced cognitive framework for end-to- end autonomous driving. arXiv preprint arXiv:2506.80055 2025c.
Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird's s-eee-view representation from lidar-camera via spatiotemporal transformers. JEEE Transactions Pattern Analysis and Machine Intelligence, 2024b.
Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Chih-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, al. Diffusion renderer: Neural inverse and forward rendering with video diffusion models. In Proceedings of the Computer Vision and 2025a.
Pattern Recognition Conference, pp.
Yiyuan Liang, Zhiying Yan, Liqun Chen, Jiahuan Zhou, Luxin Yan, Sheng Zhong, and Xu Zou. Driveeditor: A unified information-guisn framework for controllable object editing in driving scenes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 5164- 5172, 2025b.
Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shang- hang Zhang, and Hongsheng Li. aatwaand-understaae Leveraging visual prompts to enable milms to comprehend what you want. arXiv preprint arXiv:2403..0221, 2024.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209. 000003, 2022.
William Ljunghergh, Bernardo Taveira, Wenzhao Zheng. Adam Tonderski, Chensheng Peng. Fredrik Kahl, Christoffer Petersson, Michael Felsberg, Kurt Keutzer, Masayoshi Tomizuka, et al. R3d2: Realistic 3d asset insertion via diffusion for autonomous driving simulation. arXiv preprint arXiv:2506.07880 2025.
Jiachen Lu, Ze Huang, Zeyu Yang, Jiahui Zhang, and Zhang. Wovogen: World volume aware diffusion for controllable multi-camera driving scene generation. In European Conference Computer Vision, pp. 329-345. Springer, 2024.
Enhui Ma, Lijun Zhou, Tao Tang, Zhan Zhang, Dong Han, Junpeng Jiang, Kun Zhan, Peng Jia, Xianpeng Lang. Haiyang Sun, et al. Unleashing generalization of end-to-end autonomous driving with controllable long video generation. arXiv preprint arXiv:2406.00 01349, 2024a.
12

---

Published as a conference paper at ICLR 2026
Yuexin Ma, Tai Wang, Xuyang Bai, Huitong Yang. Yuenan Hou, Yaming Wang, Yu Qiao, Ruigang Yang, and Xinge Zhu. Ciston-centrie bev perception: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T' Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Q. Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Ass- ran, Nicolas Ballas, Wojciech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve. Huijiao Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning ro- bust visual features without supervision. ArXiv, abs/2304.07193, 2023. URL https://api.
semanticscho
2281700777777
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. Interna- tional Conference Machine Learning, 2021. URL https: http:/aapp.eennnccccchh..e.e Btts/CoopuII::2315910
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.1411 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp.
2022.
Hao Shan, Ruikai Li, Han Jiang, Yizhe Fan, Ziyang Yan, Bohan Li, Xiaoshuai Hao, Hao Zhao, Zhiyong Cui, Yilong Ren, et al. Stability under scrutiny: Benchmarking representation paradigms for online hd mapping. arXiv preprint arXiv:2510. 10660, 2025.
Bharat Singh, Viveka Kulharia, Luyu Yang, Avinash Ravichandran, Ambrish Tyagi, and Ashish Shrivastava. Genmm: Geometrically and temporally consistent multimodal data generation for video and lidar. arXiv preprint arXiv:2406. 10722, 2024.
Hai Wang, Junhao Liu, Haoran Dong, and Zheng Shao. A survey of the multi-sensor fusion object detection task autonomous driving. Sensors, 25(9)47794, 2025.
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3621-36333 2023a.
Xiaofeng Wang, Zheng Zhu, Yunpeng Zhang, Guan Huang, Yun Ye, Wenbo Xu, Ziwei Chen, and Xingang Wang. Are we ready for vision-centric driving streaming perception? the asap bench- mark. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9600-9610, 2023b.
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards world-vord world models for autonomous driving. In European Conference on Computer Vision, pp. 55-72. Springer, 2024a.
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14749-14759) 2024b.
13

---

Published as a conference paper at ICLR 2026
Xiaobao Wei, Qingpo Wuwu, Zhongyu Zhao, Zhuangzhe Wu, Nan Huang, Ming Lu, Ningning Ma, and Shanghang Zhang. Emd: Explicit motion modeling for high-quality street gaussian splatting. arXiv preprint ar arXhv:2411.15582, 2024.
Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9902-6912, 2024.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan. Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang. Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang, Yonggiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-image technical report, 2025. URL pnttps://arxiv. /bss
2508. 02324
Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. In Proceedings of the Computer Vision and Putlern Recognition Conference, pp. 21469-21480, 2025.
Kairui Yang. Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with multi nusti-perspecctiv consistency via bev sketch layout. arXiv preprint arXiv:2308. 01661, 2023.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng. and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10371-10381, 2024.
Beike Yu, Dafang Wang, Jiang Cao, Pengyu Zhu, and Yifei Zhao. Vehiclesim: realistic and 3d-aware video editing with one image for autonomous driving. Multimedia Systems, 31(4):316, 2025.
Ye Yu and William AP Smith. Inverserendernet: Learning single image inverse rendering. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3155- 3164, 2019.
Farhad Zanjani, Davide Abati, Auke Wiggers, Dimitris Kalatzis, Jens Petersen, Hong Cai, and Amirhossein Habibian. Gaussian splatting is an effective data generator for 3d object detection. arXiv preprint arXiv:2504. 16740, 2025.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3836-3847, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023b.
Peng Zhang, Xin Li, Liang He, and Xin Lin. 3d multiple object tracking on autonomous driving: A literature review. arXiv preprint arXiv:2309. 15411, 2023c.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang. Xiaoyi Bao, and Xingang Wang- Drivedreamer-2:: LIm-enhanced world models for diverse driving video generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 10412-10420, 2025a.
Jingyuan Zhao, Yuyan Wu, Rui Deng, Susu Xu, Jinpeng Gao. and Andrew Burke. A survey of autonomous driving from a deep learning perspective. ACM Computing Surveys, 5710))1660, 2025b.
14

---

Published as a conference paper at ICLR 2026
Yunsong Zhou, Michael Simon, Zhenghao Peng, Sicheng Mo, Hongzi Zhu, Minyi Guo, and Bolei Zhou. Simgen: Simulator-conditioned driving scene generation. ArXiv, abs/2406.09386, 2024.
URL https:///aap wtt.aa/w....ccoo..aaaaoooooaaaap
Ziyue Zhu, Zhanqian Wu, Zhenxin Zhu, Lijun Zhou, Haiyang Sun, Bing Wan, Kun Ma, Guang Chen, Hangjun Ye, Jin Xie, et al. Worldsplat: Gaussian-centric feed-forward 4d scene generation for autonomous driving. arXiv preprint 4rXiv:2509.23402, 2025.
15

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A.1 MODEL AND TRAINING DETAILS
DriveObj3D Generation Pipeline. We adopt Grounded-SAM for image segmentation, Qwen- Image-Edit for image generation, and Hunyuan3D 2.0 for 3D asset generation.
For video synthesis, Dream4Drive is constructed upon a DiT-based architecture. The backbone is initialized with pretrained weights from MagicDriveDiT (Gao et al., 2024a), which originally conditioned on BEV maps and 3D bounding boxes. We extend its Con- trolNet branch by incorporating novel 3D-aware guidance maps, and fine-tune the model on the nuScenes dataset. The generated videos have a resolution of 512 x 768 with 33 frames.
Inpainting Diffusion Model.
Training is conducted in PyTorch using 8 NVIDIA H200 GPUs with mixed-precision acceleration for 2000 iterations. We employ the Adam W optimizer with a weight decay of 0.01 and adopt a cosine annealing learning rate schedule with linear warm-up over the first 10% of steps. The learning rate is set to 2 x 10-4, and the batch size per GPU is 1. During training, we introduce additional Mask Loss and LPIPS Loss to enhance perceptual consistency and local reconstruction quality.
Mask Loss. In the VAE-decoded space, we compute the mean squared error (MSE) between pre- diction and ground truth only within the foreground mask region. Let & and denote the predicted and ground-truth decoded images, and M be a binary mask (1 for constrained pixels, 0 otherwise). The masked MSE defined as:
|M (x x
maakk
EM-
where @ denotes element-wise multiplicatton, and the denominator normalizes over valid pixels.
LPIPS Loss. To further improve perceptual quality, we adopt the LPIPS (Learned Perceptual Im- age Patch Similarity) loss (Zhang al. 2018), which measures feature-level perceptual differences between prediction and ground truth:
LLPPP = LPIPS(X, x),
where the LPIPS network pretrained on large-scale perceptual similarity data.
A.2 DETAILED SYNTHESIS PROCEDURE AND TIME-COST ANALYSIS FOR GENERATING THE 420 SAMPLES
We begin by clarifying the construction of the 420 samples, then present a detailed, step-by-step account of the time required, and conclude by describing the operations carried out at each stage.
Construction of the 420 samples. We randomly select seven annotated scenes and insert ten object categories (one asset per category). Each video contains 33 frames, from which we extract six key frames for downstream training, resulting in a total of 7 x x 6 = 420 samples.
Step
Automation (8) 700 x 10
Manual effort (8)
Scene selection & initial position annotation (MLLM-assisted) Scene verification
7 x 20
Scene orientation annotation
7 x 15
Asset selection
10 x 5
Video generation (DiT) Total
70 x 4
9800s
2.72h
295s
Table 6: Automation and manual efforts involved in synthesizing the 420 samples.
Detailed synthesis steps and time analysis (Tab.
Scene selection and initial position annotation. An MLLM identifies scenes containing at least one collision-free straight-driving trajectory (front, rear, left, or right). For eligible scenes, it annotates the asset's 's initial insertion position in ego-vehicle coordinates (zx, y, 2) for the first frame. The nuScenes training set contains 700 videos, and each MLLM query takes about 10 seconds, resulting in a total of 700 x 10 seconds. The full prompt is provided in Listing
16

---

Published as a conference paper at ICLR 2026
Human verification of MLLM-filtered scenes. Only seven scenes needed. We manu- ally verify the MLLM- --gnerrttddddd initial positions, requiring approximately 20 seconds per scene, totaling 7 x 20 seconds.
Scene orientation annotation. Orientation calibrated using the pyrender package. All assets are pre-standardized to ensure consistent orientation. For each scene, render an initial asset insertion using nuScenes intrinsics and extrinsics to determine the correct yaw angle. The first rendering takes seconds, the correction takes seconds, and the second rendering takes another 6 seconds, totaling roughly 15 seconds per scene (i.c., 7 15 seconds).
Asset selection. We require only one asset per object category. A random choice per category suffices (about seconds each), giving a total of 10 x seconds.
Video generation. We generate 70 videas for the 420 samples. Each video uses 10 infer- ence steps; on an H200 GPU each step takes roughly 4 seconds, producing a total cost of 70 x 10 x 4 seconds.
Summary. We adopt partially automated pipeline. The full synthesis process for all 420 samples takes under 3 hours, with approximately 300 seconds of actual human labor, which is effectively negligible.
A.3 MORE EXPERIMENT RESULTS
Quantitative Evaluation of Asset Quality. We evaluate synthesized 3D assets by measuring image similarity to their corresponding ground-truth originals using CLIP and DINO metrics (Fig. [7.. For CLIP model, we adopt cl aali-vittlarge-paa model, we use dinov2 base Oquab et al. (2023). The results Tab. 7 show that our pipeline consistently outperforms Trellis and Hunyuan across all object categories.
3--33 Radford et al. (202T), and for DINO
car
miick
bus
trailer
consruaction vehicle
pedescrian
motorcycle
bicycle
ratlic cone
barrier
038913
0.8870
0.8753
0.9045
0.9275
0.9254
0.9270
ILIP
Hunyuan3
0.8929
0.9043
0.9031
0.9022
0.8787
0.8863
0.9121
0.9170
0.9288
0.9101
Ours
0.9048
0.9291
0.9197
0.9270
0.8996
0.9061
0.9201
0.9351
0.9305
0.9310
Trellis
0.7008
0.7256
0.6074
0.6090
0,7055
DINO IS
Hunyuan3D
0.7360
0.7631
0.7365
0.7344
0.7204
0.7074
0.7369
0.7405
0.7610
0.7652
Ours
0.8117
0.7979
0,8082
0.8040
0.7960
0.7835
0.8065
0,8129
0.8365
0.839
Table 7: Quantitative evaluation of synthesized asset quality using CLIP Image Similarity (CLIP IS) and DINO Image Similarity (DINO IS). Higher scores indicate closer resemblance to the ground- truth assets.
These results align with the downstream perception improvements demonstrated earlier in Tab. confirming the value of high-fidelity asset synthesis.
Scaling Behavior of OOD Samples. We further study how downstream performance changes when increasing the amount of OOD synthetic data. As shown in Tab, [ expanding OOD samples from to 35 scenes does not yield proportional improvements. degrade performance, likely because excessive OOD content dilutes in-distribution signal. Small amounts OOD data improve robustness (Tab. 3. but aggressive scaling is not beneficial.
fact, too many OOD samples slightly
Epochs
2x Epochs
Epochs
Scenes
14 Scenes
35 Scenes
Scenes
14 Scenes
35 Scenes
Scenes
Scenes
Scenes
mAP 1
40.7
39.7
43.6
43.1
42.3
44.5
44.1
43.6
mATE
64.2
64.4
64.5
61.6
62.0
62.5
59.8
$9.5
60.3
mAOE 1
48.0
49.7
51.6
39.4
39.8
40.2
40.1
40,3
40.4
mAVE
27.1
27.8
28.4
27.4
28.1
28.4
27.2
27.7
27.6
NDS f
52.0
51.6
50.9
54.3
53.8
53.1
55.0
54.6
54.2
Table 8: Scaling analysis with increasing numbers of OOD scenes under different training epochs. More OOD samples do not necessarily yield better performance.
To examine whether more diverse OOD conditions (rather than more scenes) provide additional gains, we apply style transfer to generate new synthetic data (e.g., rain or nighttime) and augment the original set. Tab. shows consistent improvement across all metries.
17

---

Published as a conference paper at ICLR 2026
1x Epochs
2x Epochs
3x Epochs
Original (120)
Style Transfer (120+120)
Original (120)
Style Transfer (420+120)
Original (-20)
Style Transfer (420+120)
44.2
MATE
64.2
63.7
51.6
61.2
59.8
59.7
MADE
48.0
47.5
39.4
39.0
40.1
39.8
26.8
26.8
NDS
52.0
52.4
54.3
54.7
55.0
55.2
Table 9: Effect of styee-transeerred OOD synthetic data. Adding environmental diversity improves all downstream perception metrics.
Conclusion. Our findings indicate that downstream perception benefits most from OOD scene lay- outs and from environmental diversity, rather than merely increasing the volume of synthetic data. Additional visualization examples style transfer are included in the Fig. 20
T9
21
22
A.4 MORE ABLATION STUDIES
We conduct additional ablation studies on trajectory speed, asset categories, and scene selection diversity. Unless otherwise specified, all experiments are conducted under the 1x training epoch and 512x768 resolution setting.
Ablation on Trajectory Speed. Inserted vehicles always follow straight trajectories, and we vary speed across three levels (2 5 8 m/s). Higher speeds consistently yield larger downstream gains (Tab. 5) consistent with Table 5 in the main paper. Faster inserted vehicles leave the vehicle's' vicinity earlier, helping the perception model learn to detect distant objects more effectively.
Ablation Asset Categories. For each object category, randomly select three assets and split them into three groups (A / / C). Performance is nearly identical across groups (Tab. [T0), demonstrating that visual style differences across assets have negligible influence long the OOD) layout is preserved.
pedestrian
motoncycle
bicycle
Assets group A
0.600
0.354
0.341
0.106
0.135
0.468
0,402
0.411
0.626
0.565
group
0.594
0.350
0.343
0.110
0.135
0.471
0.405
0.414
0.625
0.563
Asscts group
0.601
0.352
0.341
0.108
0.133
0.470
0.401
0.411
0.626
0.565
Table 10: Ablation on selected asset categories, reporting AP metrics.
Ablation on Scene Selection Diversity. We randomly sample two additional sets insertion scenes, forming three groups (A B C) including the original. Downstream results remain similar across groups (Tab. 11 indicating that seene-selection-innueed bias is minimal.
1x Epochs
2x Epochs
3x Epochs
A
B
B
C
mAP 1
40.7
40.8
40.7
43.6
43.5
43.6
44.5
44.3
44.5
MATE L
64.2
64.1
64.2
61.6
61.5
61.4
59.8
59.8
59.9
mAOE 1
48.0
48.0
48.1
39.4
39.4
39.5
40.1
40.2
40.1
MAVE 4
27.1
26.9
27.1
27.4
27.5
27.3
27.2
27.3
27.2
NDS
52.0
52.1
52.1
54.3
54.2
54.3
55.0
55.0
55.1
Table 11: Ablation on insertion-scene selection.
B DETAILED AP METRICS
Tab. compares detection metrics across different training epochs. Detailed per-class AP scores provided in Tab. 12
13 and 14
C ADDITIONAL EXPERIMENTS ON GENERATION QUALITY
18

---

Published as a conference paper at ICLR 2026
Method
car
truck
bus
trailer
construction vehicle pedestrian motorcycle
bicycle
traffic.com
barrier
Real
0,562
0.323
0.296
0.067
0.111
0,422
0.377
0.371
0.569
0.515
Ours
0.600
0.354
0,341
0.106
0.135
0,468
0.402
0.411
0.626
0.565
Table 12: AP comparison across different categories for Real and Ours (+420) at Ix training epoch.
Method
truck
bus
trailer
construction, vehicce pedestrian motorcycle
bicycle
traffic_cone
barrier
Real
0.622
0.371
0.375
0.154
0.132
0.493
0.430
0,400
0.651 0.647
0.589
Ours
0.633
0.383
0.389
0.168
0.147
0.497
0.446
0.434
0.604
Table 13: AP comparison across different categories for Real and Ours (+420) at 2x training epoch.
Controllability. The controllability of our approach is quantitatively evaluated using perception metrics from StreamPETR Wang et al. (2023a). We first generate the entire nuScenes validation set with Drcam4Drive, and then measure perception performance using pre-trained StreamPETR model. The relative metries, compared to those obtained on real data, indicate how well the gen- erated samples align with the original annotations. As shown in Tab, T6 Drive achieves relative performance of 82%, demonstrating precise control over foreground object locations.
Method
FVD
FID
No cutout&mask
66.36
14.38
No depth&normal
34.64
7.33
No edge
49.45
8.44
Table 15: Ablation study on 3D-aware guidance maps.
Generation Quality. As shown in Tab. 17 our method achieves FVD 31.84 and FID 5,80, out- performing layout- and 3D comantics-guides methods Gao al.] (2023 2024a): et al. (2024));
Ji et al. (2025). The results indicate improved motion consistency and preserved visual fidelity, confirming that our 3D-aware guidance maps effectively generate both foreground and background content. Notably, no post-processing or selective filtering was applied to the generated videos.
Additional Ablation Studies. We perform ablation experiments on the components of the Multi- condition Fusion Adapter, with results presented in Tab. 15] The findings indicate that all compo- nents contribute significantly to overall performance.
D ADDITIONAL VISUALIZATION
We present more comparisons between the naive insert and ours in Fig. 9 T0 asset insertion videos across different categories in Fig. 11 12 highlighted with red bounding boxes. In addition, we showcase several corner-case examples, such
13 14
15 and 16 where all inserted assets
as imminent collision and close-following scenarios, also illustrated in Fig. 17 and Fig. [8
E LIMITATION AND FUTURE WORKS
Although Dream4Drive capable of inserting arbitrary assets into diverse scenes, automatically ensuring that the inserted trajectories remain within drivable areas and avoid collisions with pedes- trians or other vehicles remains an open challenge. Addressing this issue would enable more flexible generation of diverse
cases.
F STATEMENT ON LLM USAGE
During the preparation of this manuscript, large language models (LLMs) were used solely for language refinement. All scientific ideas, experiments, analyses, and conclusions are the authors' original contributions.
19

---

Published as a conference paper at ICLR 2026
Method
car
truck
bus
trailer
construction,veiin pedestrian
motorcycle
bicycle
traffic_comd cone
barrier
Real
0.627
0.362
0.367
0.154
0.132
0.488
0.436
0.454
0.656 0.654
0.632
Ours
0.639
0.377
0.408
0.190
0.164
0.501
0.446
0.439
0.621
Table 14: AP comparison across different categories for Real and Ours (+420) at 3x training epoch.
Method
mAP1 t
mATE 4
mAOE 4
mAVE +
NDS 1
Real
36.1
69.2
56.7
28.5
47.9 39.1 (82%)
Gen-nuScenes
24.4
78.5
66.1
34.9
Table 16: Domain gap. Comparison of detection performance Real and Gen-nuScenes validation sets at 512x768 resolution. Values are reported in percentage.
Method
FPS
Resolution
FVD4
FID
MrgicDrive Gao et al. (2023)
12Hz
224x400
218.12
16.20
Panacea Wen et al. 22024
2Hz
256x512
139.00
16.96
Suvject Huang et al. (2024a)
2Hz
256x512
124.00
15.98
DriveWM Wang et al. (2024b)
2Hz
192x384
122.70
15.80
Delphi Ma et al.
(2024a)
2Hz
512x512
113.50
15.08
Magic Gao et (002aa)
12Hz
224x400
94.84
20.91
DiVE Jiang et al. (2024)
12Hz
480x854
94.60
UniScene Li et al. (2024a)
12Hz
256x512
70.52
6.12
CoGen Ji al. (2025)
12Hz
360x640
68.43
10.15
DriveDream-2 Zhao et al.
2025a
12Hz
512x512
55.70
11.20
Ours
12Hz
512x768
31.84
5.80
Table 17: Quantitative comparison on video generation quality with other methods. Our method achieves the best FVD and FID score.
are expert autanomous-driviig scene understarding.
Based on the Input video, perform the following tasks without blas prefecence toward specific scone types le.g., do not favor simple scenes over complex anes, and do not based converience). observable evidence in the vides.
decisiona
strictly derived
[Task objective]
Tvaluate the driving scene and determine whether there exists least one safe, ruered, straight driving trajectory any of the Tour directions relative to the vehicle: front, rear, laft, or right.
If such a direction exists, identify the direction and provide the recommended initis: asset. pesition must given the ago-vehicle coordinata system and muat corraspond to the first frame of the video.
insersion position
[Ego Vehicle Coordinate System]
The ego vehicle is located at the origin 18, D, 03
exle: forward
y-axis: left
z-axis: upwerd
[Definition of "Sate Straight Tra Jeccory"]
direction is considered valid for 3D-asset insertion only if all the following conditions of vehicles, pedestsians, and any that direction consinuous and drivable (not leading into ancoming traffic lanes, sidewalká, barriers, buildings, etc.).
aze met:
Throughout entire video,
direction
static or dynamic obstacles blocking straight motion.
3. There is sufficient spatiel clearance for inserting typical vehicle sized 3D asset approx. length 4m, widch 1.3m, height 1.5m).
4. The insertion location must not collide with any existing chject in the first frome.
alloved judge disection solely based on objective video evidence,
assuming simplicity or difficulty, and witheut preferring any type of scene.
20

---

Published as a conference paper at ICLR 2026
[output format)
Reapond strictly in the following JSON structure:
"is_insertable": true/false,
"diraction": "front" "rear" "left" "right" null, "initfal_poattion
"x": number null,
"y": number or null, "2" number or null
"reason"; "Brief explanation of the decision"
[Examples)
If the front disection is clear and drivable:
"is_insertable": true, "direction" ion": "Front", "iaitial_position" (*x*: 8,0, "y": 0.0, *t"; 0.01, unobatrusted drivable long distance."
"reason": "The
if direction is suitable:
"ia_insertable": false, odirectionn: null,
positioll (*x*; mull, "y": mull, *2"; mull),
"resson"; "All disections contain obstacles that bleck safe straight Lrajectory."
Listing 1: Prompt for instructing MILM to identify safe driving scenarios.
21

---

Published as a conference paper at ICLR 2026
Naive Insertion
Ours
Figure 9: A case study of contrast between reflection and shadow.
Naive Insertion
Ours
Figure 10: A case study of contrast between reflection and shadow.
22

---

Published as a conference paper at ICLR 2026
FRONT
FRONT RIGHT
BACK RIGHT
BACK
BACK LEFT
FRONT LEFT
3D Asset
Frames
Figure 11: Insertion of car in the right-side region.
FRONT
RIGHT
BACK LEFT
3D Asset
Frames
Figure 12: Insertion of a truck in the left-side region.
23

---

Published as a conference paper at ICLR 2026
FRONT
FRONT RIGHT
BACK RIGHT
BACK
BACK LEFT
FRONT LEFT
3D Asset
Frames <
Figure 13: Insertion of a barrier in the left-side region.
FRONT
FRONT RIGHT
BACK RIGHT
BACK
FRONT LEFT
3D Asset
Frames
Figure 14: Insertion of bus in the back-side region.
24

---

Published as a conference paper at ICLR 2026
FRONT
FRONT RIGHT
BACK RIGHT
BACK
BACK LEFT
3D Asset
Frames
Figure 15: Insertion of a traffic cone in the left-side region.
FRONT
RIGHT
BACK
BACK LEFT
FRONT LEFT
3D Asset
Frames
Figure 16: Insertion of construction vehicle in the back-side region.
25

---

Published as a conference paper at ICLR 2026
FRONT
FRONT RIGHT
BACK RIGHT
BACK
BACK LEFT
FRONT LEFT
Imminent Collision Scenario
Frames
Figure 17: Corner case. Insertion of a barrier in the front-side region, where the ego vehicle is about to collide with it.
FRONT
FRONT RIGHT
BACK RIGHT
BACK
BACK LEFT
FRONT LEFT
Close- Following Scenario
Frames
Figure 18: Corner case. Insertion of truck in the close-range front-side region.
26

---

Published as a conference paper at ICLR 2026
FRONT
BACK RIGHT
BACK
BACK LEFT
FRONT
3D Asset
Frames
Figure 19: Style transfer. Insertion of a bus in the rain.
FRONT
FRONT RIGHT
BACK RIGHT
BACK
BACK LEFT
FRONT LEFT
3D Asset
Frames
Figure 20: Style transfer. Insertion of a truck in the rain.
27

---

Published as a conference paper at ICLR 2026
FRONT
FRONT RIGHT
FRONT LEFT
Assel
Frames
Figure 21: Style transfer. Insertion of a barrier in the night.
FRONT
FRONT RIGHT
BACK RIGHT
BACK
BACK LEFT
FRONT
3D Asset
Frames
Figure 22: Style transfer. Insertion of constructive vehicle in the night.
28


Reviews:
[{"review_id": "izC6kwriC6", "reviewer": "Reviewer_dArx", "summary": "This paper primarily investigates the limitations of existing generation methods in downstream tasks, demonstrating that significant improvements can be achieved using only 2% of generated data. It also introduces DriveObj3D, a large-scale 3D asset dataset covering typical categories in driving scenarios and enabling diverse 3D-aware video editing.", "questions": "This work demonstrates the effectiveness of out-of-distribution (OOD) scene data generation for downstream task training, focusing specifically on *scene-level* OOD—namely, inserting vehicles into originally empty scenes. However, the scope of OOD scenarios could be further expanded. For instance, one could explore modifying existing scenes by altering vehicle trajectories or swapping vehicle categories to investigate how such diverse OOD variations impact downstream task performance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Identifies and analyzes the limitations of current generative methods in downstream tasks, showing that even a small amount (as little as 2%) of high-quality generated data can lead to significant performance gains.\n- Introduces DriveObj3D, a large-scale 3D asset dataset encompassing typical object categories in driving scenarios, which supports diverse and 3D-aware video editing applications.", "weaknesses": "- The method relies on scene insertion and asset insertion that currently lack sufficient automation and are primarily dependent on manual effort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q0Ykqghm0O", "reviewer": "Reviewer_FTf2", "summary": "The paper presents Dream4Drive, a 3D-aware synthetic data generation framework designed to improve downstream perception tasks such as 3D object detection and tracking in autonomous driving. Dream4Drive employs dense 3D-aware guidance maps to edit real driving videos by rendering high-quality 3D assets, ensuring multi-view geometric consistency and photorealism. A new dataset, DriveObj3D, is introduced to provide diverse 3D vehicle and traffic assets for large-scale, geometry-consistent scene editing. Dream4Drive demonstrates that injecting fewer than 2% synthetic samples consistently improves detection and tracking metrics, outperforming existing augmentation baselines.", "questions": "Please refer to the weaknesses above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Dream4Drive introduces a combination of dense 3D-aware guidance maps with generative video editing, enabling realistic, geometry-consistent, multi-view scene synthesis.\n\n2. Demonstrating that fewer than 2% synthetic samples can enhance real data training is a strong and practically relevant result, suggesting high efficiency in data augmentation.\n\n3. The proposed DriveObj3D fills a gap in open 3D asset resources for driving research, supporting reproducibility, and future extension in video-level generative modeling.", "weaknesses": "1. The paper does not clarify how inserted 3D assets are constrained to remain physically valid: ensuring no collisions, adherence to drivable areas, and correct orientations, nor how occlusion layers between objects are resolved across depth, normal, and edge maps.\n\n2. There is no qualitative criterion for evaluating 3D asset realism or consistency, raising uncertainty about the reliability of the generated dataset and edited scenes.\n\n3. As shown in Table 3, the method’s performance fluctuates across epochs; in some settings, direct insertion outperforms Dream4Drive, and differences between methods are within statistical noise, suggesting the need for repeated trials with mean–variance reporting.\n\n4. The introduction is verbose, and the method section lists procedural steps without sufficient motivation or justification. Figures contain minor layout inconsistencies.\n\n5. Missing Related Work: SimGen (NeurIPS 2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZQfdkrSssa", "reviewer": "Reviewer_yb3h", "summary": "This paper proposes Dream4Drive, a framework for generating synthetic training data for autonomous driving perception tasks. The key insight is that previous work evaluated synthetic data unfairly by pretraining on synthetic data and then finetuning on real data (2x training epochs), obscuring true benefits. Dream4Drive decomposes videos into 3D-aware guidance maps (depth, normal, edge, object image, mask) and uses a fine-tuned diffusion transformer to render edited multi-view videos after inserting 3D assets. The authors contribute DriveObj3D, a dataset of 3D assets across driving scenarios. With only 420 synthetic samples (<2% of training data), they demonstrate improvements in detection and tracking at 1x, 2x, and 3x training epochs.", "questions": "Additional questions:\n\nQ1: Given that naive insertion performs within 0.6-0.7 mAP of your full method (Tables 3-4), can you provide a compute or latency cost-benefit analysis? The qualitative improvement in image quality is clear (Figure 8) but the actual metric benefits less so.\n\nQ2: How much manual effort is required to select 420 valid insertion positions and trajectories? Can you quantify the human time required and discuss path to full automation?\n\nQ3: Building off Q2, do the authors have an evidence or intuition as to whether the approach will scale with more augmented scenes using their library? W2 suggests that the increase in training data doesn't necessarily correspond with a proportional increase in performance.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper identifies a key shortcoming in most generative simulators for autonomous driving data: improvements in downstream task from training on both real and synthetic data have in the past been attributed only to the addition of synthetic data while ignoring that increasing training epochs improves performance even just using the base dataset (nuScenes).\n- The paper provides a large-scale dataset (DriveObj3D) along with a method to insert assets from the dataset in a natural manner to match scene lighting and shadows.\n- The results show some improvement over existing methods in downstream detection tasks while being sample efficient (only adding <2% data over the original dataset)", "weaknesses": "W1. The paper lacks novelty beyond the Epoch related finding. The actual data generation framework is a composition of existing models and methods rather than a technical contribution. While adding multiview images as conditions to generate a 3D mesh for 3D assets does improve asset quality over meshes generated from single view, it seems like an unsurprising improvement. Moreover, the actual improvement in scores from Table 5 compared to other 3D asset generation methods is relatively small, despite using multiview images vs Hunyuan3D's single view method.\n\nW2. The detection performance results using the method are marginal improvements over previous works. In Tables 1, 2, and 3, the relative improvement in metrics like mAP in the 2x and 3x Epochs are small, even over just using the original nuScenes dataset. In fact, this result shows that even the DriveObj3D falls prey to the core discovery the authors note about normalizing for # of Epochs trained; the improvement over training with only real data diminishes with increasing Epochs. The increase in performance in mAP vs Real data in Table 1 (2x Epochs) is <1% improvement despite including >1% increase in training data. \n\nW3. Appendix E acknowledges that \"automatically ensuring insertions are in drivable areas and avoid collisions remains an open challenge.\" This is a significant limitation, with manual effort required preventing this method from generating corner cases on a significant scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XNRUbwVNji", "reviewer": "Reviewer_yxwm", "summary": "This paper critically re-evaluates the utility of synthetic data generated by driving world models for downstream perception tasks. The authors argue that previous works often rely on an unfair evaluation protocol where models trained on hybrid (real + synthetic) data use twice the training epochs of real-data-only baselines. They demonstrate that when training epochs are matched, the benefits of existing synthetic data augmentation methods become negligible or even negative.\n\nTo address this, the paper introduces **Dream4Drive**, a novel framework for generating high-quality, 3D-aware synthetic data. Instead of relying on sparse conditioning signals like BEV maps, Dream4Drive first decomposes a real video into a set of dense, 3D-aware guidance maps (depth, normal, edge, etc.). It then renders 3D assets into these maps and fine-tunes a Diffusion Transformer-based world model to generate photorealistic, multi-view videos that incorporate these new objects. This approach allows for precise, instance-level control and ensures geometric and visual consistency.\n\nFurthermore, the authors contribute **DriveObj3D**, a large-scale 3D asset dataset tailored for driving scenarios, along with an automated pipeline for its creation. Through extensive experiments on detection and tracking tasks, the paper shows that augmenting the training set with a very small fraction (<2%) of data generated by Dream4Drive consistently and significantly improves perception performance, even under fair, epoch-matched comparisons.", "questions": "1. Regarding the `DriveObj3D` creation pipeline: Could you quantify the robustness of this pipeline? What is the approximate failure rate, and what are the common failure modes? How much manual filtering or intervention was required to curate the final high-quality dataset?\n2. The scene selection for insertion is described as choosing frames \"where no other vehicles appear along the insertion trajectory.\" How is this selection performed (manually or automatically)? Could this process introduce a bias, for example, by preferentially selecting less cluttered scenes, thereby limiting the complexity of the generated data?\n3. The paper highlights improved realism via shadows and reflections. Does the generative model also learn to synthesize more complex physical interactions? For example, does an inserted vehicle generate splashes when moving through a puddle, or create dust on a dirt road? To what extent can the model handle nuanced lighting effects beyond direct shadows, such as colored light from traffic signals reflecting on the vehicle?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Important and Timely Critique:** The central argument regarding the unfair evaluation of synthetic data is a significant contribution. By demonstrating that simply increasing training epochs on real data can match or exceed the performance of hybrid-data training, the paper forces the community to reconsider how the value of synthetic data is measured. This provides a strong and compelling motivation for the proposed work.\n2. **Comprehensive and Convincing Experiments:** The experimental validation is thorough and directly supports the paper's claims.\n    - The head-to-head comparisons under 1x, 2x, and 3x epoch settings provide clear evidence for the effectiveness of Dream4Drive over baselines.\n    - The ablation studies are insightful, systematically analyzing the impact of insertion position, distance, 3D asset source, and different components of the guidance maps. This provides a deeper understanding of what makes the synthetic data effective.\n    - The evaluation on both detection and tracking tasks, at multiple resolutions, demonstrates the general applicability of the generated data.", "weaknesses": "1. **Limited Scope of Generated Corner Cases:** The paper defines \"corner cases\" primarily as the insertion of new objects into a scene. While this is an important class of long-tail events, it does not cover other critical scenarios, such as adverse weather conditions (heavy rain, snow, fog), unusual lighting (lens flare, low sun), complex multi-agent interactions, or environmental changes (e.g., road construction). The framework's applicability to these other types of corner cases is not explored.\n2. **Potential Bottlenecks in the Asset Generation Pipeline:** The `DriveObj3D` pipeline is a cascade of multiple sophisticated models (segmentation, multi-view generation, mesh generation). The final asset quality is contingent on the successful execution of every step. This pipeline may be brittle; for instance, a failure in multi-view generation could lead to an incomplete or distorted 3D mesh. A discussion on the robustness, failure modes, and potential need for manual curation of this pipeline would strengthen the paper.\n3. **Limited Technical Contribution.** While the specific application and the system-level design are effective in research application, the work does not introduce a new core generative modeling technique or a new research framework. For some reviewers, the novelty might be perceived as incremental, lying more in the clever combination of existing parts than in foundational innovation.\n4. **Computational Cost Analysis is Missing:** The proposed data generation process is multi-staged and involves fine-tuning a large Diffusion Transformer model. This suggests a significant computational cost. While the paper argues for fairness in terms of training *epochs*, a discussion on fairness in terms of total *compute* (cost of data synthesis + cost of training) would provide a more complete picture. For instance, how does the cost of generating 420 synthetic samples compare to the cost of training the perception model for an additional epoch?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)