Glad: A Streaming Scene Generator for Autonomous Driving
Authors: Bin Xie
Yingfei Liu
Tiancai Wang
Jiale Cao
Xiangyu Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Video Generation; Autonomous Driving
Primary area: generative models

Abstract: The generation and simulation of diverse real-world scenes have significant application value in the field of autonomous driving, especially for the corner cases. Recently, researchers have explored employing neural radiance fields or diffusion models to generate novel views or synthetic data under driving scenes. However, these approaches suffer from unseen scenes or restricted video length, thus lacking sufficient adaptability for data generation and simulation. To address these issues, we propose a simple yet effective framework, named Glad, to generate video data in a frame-by-frame style. To ensure the temporal consistency of synthetic video, we introduce a latent variable propagation module, which views the latent features of previous frame as noise prior and injects it into the latent features of current frame. In addition, we design a streaming data sampler to orderly sample the original image in a video clip at continuous iterations. Given the reference frame, our Glad can be viewed as a streaming simulator by generating the videos for specific scenes. Extensive experiments are performed on the widely-used nuScenes dataset. Experimental results demonstrate that our proposed Glad achieves promising performance, serving as a strong baseline for online video generation. We will release the source code and models publicly.

Published as a conference paper at ICLR 2025
GLAD: A STREAMING SCENE GENERATOR FOR AUTONOMOUS DRIVING
Bin Xie Yingfei Liu²t, Tiancai Wang", Jiale Cao Xiangyu Zhang?3
'Tianjin University, MEEGVII Technology. *StepFun (bin (ben.xie, connor]@tju. edu. (liuyingfei, wangtiancel (aanngxiangyuujemeeee
edu, en
ABSTRACT
The generation and simulation of diverse real-world scenes have significant ap- plication value in the field of autonomous driving, especially for the corner cases. Recently, researchers have explored employing neural radiance fields or diffusion models to generate novel views or synthetic data under driving scenes. However, these approaches suffer from unseen scenes or restricted video length, thus lack- ing sufficient adaptability for data generation and simulation. To address these issues, we propose a simple yet effective framework, named Glad, to generate video data in frame-by-frand style. To ensure the temporal consistency of syn- thetic video, we introduce a latent variable propagation module, which views the latent features of previous frame as noise prior and injects it into the latent fea- tures of current frame. In addition, we design a streaming data sampler to orderly sample the original image in a video clip continuous iterations. Given the ref- erence frame, our Glad can be viewed as a streaming simulator by generating the videos for specific scenes. Extensive experiments are performed on the widely- used nuScenes dataset. Experimental results demonstrate that our proposed Glad achieves promising performance, serving as strong baseline for online video generation. We will release the source code and models publicly.
INTRODUCTION
driving tasks are usually data-intensive, relying on vast amounts of data to learn in- formed models. In recent years, autonomous driving has achieved considerable progress, partic- ularly in the field of Bird's Eye View (BEV) perception (Huang et al., 2021; Liu et al., 2022; Li al., 2022h; Liu al., 2023; Li et al., 2023b; Liao et al., 2022) and end-to-end planning (Shi al., 2016; Jiang et al., 2023; Chen et al., 2024), thanks to the availability of public datasets such as nuScenes (Caesar al., 2019), CARLA (Dosovitskiy al., 2017), Waymo (Ettinger et al., 2021), and ONCE (Mao et al., 2021). However, these real-world driving datasets exist several limitations. On one hand, it expensive and labor-intensive to collect large-scale real-world driving data. On the other hand, although corner-cases comprise only small portion of dataset, they are more important for evaluating the safety of autonomous driving. Therefore, the scale and diversity of real-world datasets constrain the further development of autonomous driving.
Autonomous
Instead of collecting large-scale real-world datasets, researchers have explored generating synthetic street-view data. Some approaches focus on leveraging Neural Radiance Fields (NeRF) (Yang al., 2023a; Yan et al., 2024a; Yang et al., 2023c; Tonderski et al., 2023) and 3D Gaussian Splatting (3DGS) (Yan al., 2024b) for rendering the novel views in driving scenes. These cutting-edge ap- proaches offer a powerful tool for rendering highly realistic and detailed images from various view- points. However, shown in Fig. 1(a), these approaches struggle reconstruct the non-seen streets when the source trajectory and simulated trajectory are different. Recently, some approaches (Gao al., 2023; Yang et al., 2023b; Swerdlow et al., 2024: Li al., 2023a; Jia et al., 2023; Hu al., 2023) attempt to employ Stable Diffusion (Rombach et al., 2022) to generate the synthetic data of driving scenes. For instance, Panacea (Wen et al., 2023) and 2riveDreamer-2 (Zhao al., 2024)
* "This work done during the internship MEGVII Technology
I 'quual contribution
1 *oorrspondiig author; Jiale Cao

---

Published as a conference paper at ICLR 2025
Soutre
Retorence
Realistic, Constrained scene
Offine, Fixed lengths
Streaming, A Arbibrary lengths
Source tra eecoorryy O +
O Source injectory
Source trajectory O +
+0
Simulated trajectory
Simulated trajectory
Simulated trajectory
O O +
+O +O + "c.
Readered scene
Offline Generated scene + + inconsistent
Online Genernted scene O +O +0 flexible
O O
wnssen
(a) Unisim
(b) Panacca
(c) Glad
Comparison of Unisim, Panacea, and our proposed Glad: (a) The barf-based Unisim (Yang et al., 2023c) struggles to render unseen scene and objects when the simulated ego trajectory deviates from the source data. (b) The diffusion-based Panacea (Wen et al., 2023) generates fixed- length video data in offinn manner. It suffers from relatively high memory consumption, and lacks the adaptability to accommodate variations in dynamic simulated trajectory. (c) Our Glad is designed for fame-by-frame generation, enabling to generate videos of arbitrary lengths and exhibit- ing good flexibility in the variations of simulated trajectory.
Figure/ I:
have delved into the ability of Stable Diffusion to generate synthetic video data from given initial frame. However, as shown in Fig. 1(b), these diffusion models (e.g., Panacea) are limited to offline video generation and require high memory consumption. Moreover, fails to dynamically generate video data in response to the changes in simulated trajectory. For flexible generator, it requires the ability of generating dynamic scenes corresponding to the simulated trajectory.
To address the issues mentioned above, we propose Glad, simple yet effective framework that generates and forecasts synthetic video data an online (i.e., frame-by-frame manner. Our Glad is based on Stable Diffusion, and introduces the latent variable propagation (LVP) to maintain temporal consistency when performing frame-by sauee-b--fuee by-frame generation. In LVP, the latent features of previous frame viewed as the noise prior injected into the latent features of current frame. In addition, introduce a streaming data sampler (SDS), which aims to keep the consistency between training and inference, and present efficient data sampling for training. In SIS,, we sample original frames in video clip one by one at continuous iterations, and save the generated latent features in cache used for the noise prior of next iteration.
With such streaming generation manner, our Glad is capable of generating videos of arbitrary lengths in theory. Specifically, it can generate the videos of the novel scene from the noise when serving as the data generator. In addition, given the reference frame, it can produce the video data of specific scene. We perform the experiments on the public autonomous driving dataset nuScenes, which demonstrates the efficacy of our Glad. The contributions and merits can be summarized as follows:
A simple yet effective framework, named Glad, is proposed to generate video data in an online manner, instead of offline fixed-length manner. Theoretically, our proposed Glad is able to generate videos of arbitrary lengths, showcasing substantial potential in data generation and simulation.
A latent variable propagation strategy introduced ensure the temporal consistency for frame-by-frre -r-mame video generation, where the latent features of previous frame are viewed as the noise fed to the latent features of current frame.
We further design a streaming data sampler to provide efficient training. In the streaming data sampler, we sample the frame in video clip one by one at several continuous iterations, and save the latent features in cache for next iteration.
The experiments are performed the widely-used dataset nuScenes. Our Glad is able to generate high-quality video data as generator. Further, our generated video data can significantly improve the perception, tracking, and HD map construction performance.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORKS
2.1 VIDEO GENERATION MODELS
Recent years have seen considerable advancements in video generation, focusing on improving the quality, diversity, and controllability of generated content. Initial methods (Yu et al., 2022; Clark et al., 2019; Tulyakov et al., 2018) extend the success of Generative Adversarial Networks (GANs) from image synthesis to video generation. However, these approaches often face challenges in maintaining temporal consistency.
The introduction of diffusion models (Ho et al., 2020; Rombach et al., 2022) represents a paradigm shift, providing a robust alternative for high-fidelity video synthesis. VDM (Ho et al., 2022b) uti- lizes factorized spatio-temporal attention blocks to generate 16-frame, 64 64 pixels videos, which are then upscaleable to 128 x 128 pixels and 64 frames using enhanced model. Imagen Video (Ho et al., 2022a) further progresses the field through a cascaded diffusion process, beginning with a base model that produces videos of x0x24 24 pixels and 16 frames, and sequentially upsampling through six additional diffusion models to reach 1280 x 768 pixels and 128 frames. MagieVider (Zhou et al., 2022) employs Latent Diffusion Models (LDM) enhance efficiency in video generation. VideoLDM (Blattmann et al., 2023), leveraging a similar architecture, incorporates temporal at- tention layers into pre-trained text-to-image diffusion model, excelling in text-to-video synthesis. TRIP (Zhang et al., 2024) proposes a new recipe of image-to-video diffusion paradigm that pivots on image noise prior derived from static image. In contrast, our Glad uses latent features from the previous frame as noise prior, replacing Gaussian noise, while TRIP predicts a "reference noise" for the entire clip, risking inaccuracy over time. Vista (Gao et al., 2024) presents a generalizable driving world model with high fidelity and versatile controllability. Additionally, Sora (Tim et al., 2024) sig- nificantly enhances video quality and diversity, while introducing capabilities for text-prompt driven control of video generation.
Despite these advancements, video generation still faces several challenges, including maintaining temporal consistency, generating longer videos, and reducing computational costs. A streaming video generation pipeline could present an elegant solution. Notably, several advanced works (Yan al., 2021; Hong et al., 2022; Huang al., 2022; Henschel et al., 2024) have adopted autore- gressive approach to generate video frames. VideoGPT (Yan et al., 2021) predicts the current latent code from the previous frame, Autoregressive GAN (Huang al., 2022) generates frames based on a single static frame, and Cog Video (Hong et al., 2022) employs a GPT-like transformer architec- ture. - (Weng et al., 2024) firstly explores autoregressive text-to-video generation and adopts multi-reference frames and anchor frame as condition, while our Glad achieves streaming video gen- eration by using previous latents as initial noise in the diffusion process. Nonetheless, these methods do not yet fully satisfy the stringent requirements for generation quality, controllability, and motion dynamics essential in autonomous driving applications.
2.2 DRIVING DIFFUSION MODELS
In the realm of driving scenario generation, early research predominantly focuses synthesizing individual images. BEVGen (Yan et al., 2024b) explores the generation of multi-view street images from BEV layout. BEVControl (Yang et al., 2023b) incorporates cross-view attention to enhance visual consistency and ensure a cohesive scene representation. MagicDrive (Gao et al., 2023) high- lights the challenges associated with the loss of 3D geometric information after the projection from 3D to 2D. In addition to the cross-view consistency, cross-frame consistency remains crucial for temporal modeling. Based on this point, DrivingDiffusion (Li et al., 2023a) and Panacea (Wen et al.. 2023) introduce sequences of BEV layouts to generate complex urban videos in control- lable manner. However, their dlddng-window approach to video generation proves inefficient and unsuitable for extended durations. Moreover, DriveDreamer-2 (Zhao et al., 2024) and ADriver-I (Jia autonomous
et al., 2023) use the initial frame as a reference, demonstrating significant potential driving simulation by predicting subsequent video sceness. GAIA-I (Hu et al., 2023) generates long- duration, realistic video data conditioned on diverse inputs such as images, text, and action signals. Unfortunately. the capability to manipulate motion trajectories of other vehicles through control signals is limited in (Hu et al., 2023). To meet the demands of data generation and simulation, the challenges such extended video length, controllability and consistency still hold significant value.
3

---

Published as a conference paper at ICLR 2025
Dannising TiNe
Danoising UNer
Triirabie
CLIP
Latent Variahle Pripagation
Latent Variable Propagation
Q
Net
Frozen
seene...
z'
Text prompt BEV layout
Guussian Noise
VAE
VAE
VAE
VAE
Inpul
Outrutt
I Inpur
OOtpul
Latest Space
Mullt-iiin
Training
unly
Latent reference frame
Frmme N
Figure 2: Overall architecture of our proposed Glad. Glad based on Stable Diffusion, and can take random noise reference frame as input generate Glad generates video sequences from frame to frame N orderly. We employ the proposed latent variable propagation module feed the denoised latent features previous frame to current frame as the noise prior, which can maintain video temporal consistency. This frame-by-frame generation strategy enables to generate videos of arbitrary lengths. In addition, employ ControlNet (Zhang et al., 2023) to introduce BEV layout for fine- ane-grained control on data generation.
specific scenes. Afterwards,
3 METHOD
Here we introduce our proposed Glad, a simple yet effective approach that denoises the current frame from the fully denoised latent feature of the previous frame instead of Gaussian noise, which generates and simulates video data in an online manner. The proposed Glad is based on Stable Diffusion (Rombach et al., 2022), and introduces two novel modules for generating video data, cluding latent variable propagation and streaming data sampler. The latent variable propagation aims to maintain temporal consistency when performing frame-by-frame generation, while stream- ing data sampler provides efficient data sampling pipeline and keeps data consistency between training and inference. Our proposed Glad is capable of both generating an entirely new scene from Gaussian noise and simulating specific scene based on given reference frame.
Fig. presents the overall architecture of our proposed Glad. Given the Gaussian noise or latent features z of the reference frame, introduce latent variable propagation (LVP) process first and then view it as the noise-added latent features z of frame 1. Afterwards, we employ denoising UNet recover the latent features z² which fed to VAE decoder for image generation and LVP module for latent features propagation. By analogy, we can generate synthetic images from frame to frame N. which together form a continuous video sequence. Compared the fixed-length video generation in existing approaches (Wen et al., 2023; Zhao et al., 2024), our frame-by- by-ree-b-fameee video generation approach is more efficient and flexible, which can theoretically generate the videos of arbitrary lengths. In addition, similar to most existing approaches, we employ ControlNet (Zhang et al., 2023) and CLIP (Radford et al., 2021) to condition feature extraction denoising UNet by given text prompt and BEV semantic layout.
Overview,
3.1 LATENT VARIABLE PROPAGATION
Our latent variable propagation (LVP) aims to maintain the temporal consistency in frame- frame-by-frame video generation. The distribution of noise plays an essential role in conditioning image synthesis of diffusion models. When extending image-level diffusion model Stable Diffusion (Rombach et al., 2022) to generate video sequences, how to generate the noise distribution across time is crucial to maintain video temporal consistency. One straightforward way is to sample the noise from the same distribution for image generation at different frames similar to offline approaches (Wen et al., 2023). However it cannot maintain temporal consistency in our frame-by-frame generation design. To address this issue, our proposed LVP views the denoised latent features previous frame as the noise prior for image generation at current frame. Based on the LVP module, we can orderly generate video sequences from frame to frame N, while maintaining good temporal consistency.
4

---

Published as a conference paper at ICLR 2025
Specifically, perform the forward diffusion process of frame based the latent features of frame n as
g(zi,r 2)) II 2122)21 g(2)) = = (z / 11 (()((2()))))) n 1,,,, (1)
where z represents the noisy latent features at frame n at time-step and zn 2 represents the denoised latent features at frame n - 1 generated by denoising UNet. The function ip aims to normalize the latent features zn-1 as the noise, where adopt layer normalization operation. The reverse process at frame n can be written as
Po (0 = p(27)) II (o 12"), po(2
Eo(z" ()), I,...,N N
(2) where p(z7) () is the Gaussian noise for generator latent features of reference frame for simulator when n = 1, and represents the denoised latent features z==1 of generated image at frame when
N.
3.2 STREAMING DATA SAMPLER
As mentioned above, our proposed Glad re- quires the denoised latent features n-1 at frame - noise prior to generate the image at frame n. To ensure the consistency between training and inference, we need to calculate the denoised latent features z"-1 at frame n when performing image generation frame n during training. However, is consuming and unnecessary to generate the denoised latent features ==1 at frame
Original image
Latent fonteres
Grussian moise
Diffissoon model
Video clip (N frames)
Streaming sampling
tenche
Leache
N
from scratch (i.e., frame 1). To address this issuc, we introduce streaming data sampler that samples video clip orderly at continuous iterations, which can im- prove the efficiency of training.
Iteration
Iteration 2
Iteration N
Figure 3: Illustration of streaming data sam- pler. The streaming data sampler samples video clip from frame to frame N at continuous it- erations. At each iteration, latent features generated by diffusion model the cache, and reuse the noise at next iteration.
Fig. presents the pipeline our streaming data sampler module. Given video clip having N frames for each clip, we orderly sample the frame one by one at N continuous iterations. At each iteration, we employ diffusion model to generate denoised latent features at current frame, and save the denoised latent features at current frame in the cache. The saved denoised latent features at current iteration will be used as the noise prior at next iteration. In this streaming way, we just generate the synthetic image each frame in video clip only once, resulting in more efficient training.
the denoised
3.3 TRAINING AND INFERENCE STRATEGY
Layout and text control. It is crucial to freely manipulate the agents within the scenes when performing image generation. Following existing works (Wen al., 2023; Gao et al., 2023), we employ BEV layout and text prompt to precisely control the scene composition, which includes 10 common object categories and 3 different types of map elements. For BEV layout, we firstly convert them into camera view perspective and extract the control elements as object bounding boxes, object depth maps, road maps, and camera pose embeddings. Afterwards, we integrate them into denoising UNet using the ControlNet (Zhang et al., 2023) framework. The text prompts fed to denoising UNet via pre-trained CLIP (Radford et al., 2021).
Training. Inspired by Panacea (Wen et al., 2023), adopt two-stage training strategy and employ the same data and processing rules. First, we perform image-level pre-traine training. Our Glad can also be seen image synthesis model, where only employ Stable Diffusion architecture generate
5

---

Published as a conference paper at ICLR 2025
the image of one frame from Gaussian noise. Second, we perform video-level fine-tuning the autonomous driving dataset nuScenes by using our proposed streaming data sampler. To obtain images with resolution of 512x256 pixels, we randomly resize the original images to proportion of 0.3 to 0.36, and then perform a random crop. This ensures consistency in processing with the downstream StreamPETR method.
Inference. Our Glad can be used as generator and simulator, which respectively generate video sequence from Gaussian noise and reference frame. Specifically, given the reference frame Gaus- sian noise, we first employ VAE encoder to generate the latent features, Subsequently, we feed it to the latent variable propagation module as noise prior, and employ denoising UNet and VAE decoder to predict the denoised latent features and generate the image at frame 1. Then, the denoised latent features at frame are fed to the latent variable propagation module for image generation at next frame. In this way, our Glad is able to generate video sequence of arbitrary lengths.
4 EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
nuScenes dataset. The nuScenes dataset was collected from 1000 different driving scenes in Boston and Singapore. These scenes are split into training, validation, and test sets. Specifically, the training set contains 700 scenes, the validation set contains 150 scenes, and the test set contains 150 scenes. The dataset has totally 10 object classes, and provides accurate 3D bounding-bos annotations for each object. In every scene, there are 6 camera views and each view records length of about 20 second driving video.
Evaluation metrics. We evaluate the quality of the generation and simulation, respectively. To quantitatively evalute the generation quality, we adopt the frame-wise Fréchet Inception Distance (FID) (Parmar al., 2022) and Fréchet Video Distance (FVD) (Unterthiner al., 2018), which calculate the Fréchet distance between generated content (image or video) and ground truth. To evaluate the model's ability adhere the given input conditions, such as the BEV layout, we fol- low (Liao et al., 2022) and report the performance of online vectorized HD map construction. This includes the average precision (AP) for constructing pedestrian crossings (AP, ped), lane dividers (AP ), road boundaries (APboundary). and their weighted average (mAP). To evaluate the po- tentiality of Glad as simulator, we assess the 3D detection performance on the nuScenes dataset, including nuScenes Detection Score (NDS) and mean Average Precision (mAP). To further evaluate the temporal consistency of the generated videos, we also report the multi-objeet tracking results, including average multiple object tracking accuracy (AMOTA) and average multiple object tracking precision (AMOTP).
Implementation details. Our Glad is implemented based on Stable Diffusion 2.1 (Rombach al., 2022). We train our models on NVIDÍA A100 GPUs with the mini-batch of 2 images. During training, we first perform leaee pre-training. Constant learning rate 4x 10-5 has been adopted, and there are 1. 1.25M iterations totally. Afterwards, we fine-tune our Glad on nuScenes dataset with same settings for 48 epochs. We split each video into clips to balance video length and data diversity. During inference, we utilize the DDIM (Song et al., 2020) sampler with 25 sampling steps and scale of the CFG as 5.0. The image is generated at a spatial resolution of 256 x 3072 pixels with 6 different views, and split it to 6 images of 256 x 512 pixels for evaluation. We adopt StreamPETR (Wang et al., 2023a) with ResNet-50 (He et al., 2016) backbone as perception model. Regarding the map construction model, we employ MapTR (Liao et al., 2022) with ResNet-50 and retrain it under a 512 x 256 pixels resolution setting. The inference time of complete denoising process is reported in single NVIDIA A100 GPU.
4.2 MAIN RESULTS
Generation quality. We first online generate video data on nuScenes validation set, and employ the sliding window strategy to evaluation synthetic video quality similar to existing approach (Wen et al., 2023). Tab. 1 compares data generation quality of our proposed Glad and some state-of-the-an approaches. We present the results of our Glad respectively using Gaussian noise from given reference frame. When generating video using Gaussian noise, Glad achieves an FID of 12.57 and an FVD of 207. Furthermore, when generating videos starting from reference frame, Glad achieves

---

Published as a conference paper at ICLR 2025
Table I: Comparison of video generation with state-of-the-art approaches in terms FID and FVD metrics on nuScenes validation set. Compared to the Single frame approaches, our online approach Glad supports high-quality multi-frame generation. Compared to the offline approaches, our Glad achieves promising performance in an online manner.
Type
Method
Ref-Frame
Multi- View
Multi-Frame
FID
FVD
BEVGen (Swerdlow et al., 2024)
25.54
Single
BIVContml (Yang al., 2023h)
24.85
N/A
MagicDrive (Gao et al., 2023)
16.20
N/A
DriveDreamer (Wang al., 2023b)
52,60
452
Wo (Lu ct 2023)
27.60
418
Offline
Panacea (Wen al., 2023)
16.96
MagieDrive (Gao al., 2023)
16.20
DrivingDiffusion (Li et al., 2023a)
15.83
332
Drive-WM (Wang et ul., 2024)
15.80
123
Glad (Ours)
12.57
207
Online
Glad (Ours)
11.18
lower FID of 11.18 and FVD of 188. Compared the offline approaches, our online Glad achieves promising performance in terms of both FID) and FVD. For instance, our Glad without reference frame outperforms MagicDrive by 3.63 and in terms of FID and FVD respectively. We evaluate the generation speed of our method and some open-source methods. To generate multi-view frames, BEVGen requires 6.6s, MagicDrive takes 11.2s, and our Glad takes 9.4s. Namely, our Glad has a comparable generation speed with these approaches, but has better generation quality.
Table 2: Performance on downstream tasks of the generated data on the validation set. We em- ploy pre-trained perception model StreamPETR to perform 3D detection and multi-object tracking on generated and real data. We report the results of StreamPETR when applied to the real nuScenes validation set as Oracle. The input image size is set as 512x256 pixels.
Method
nulmage
Video Length
mAPt
NDSt
AMOTA
AMOTPL
Oracle
<41
34,5
46.9
30,2
1,384
Oracle
<41
37.8
49.4
33,9
1.325
Panacea (Wen et al., 2023)
8
18.8
32.1
Punacea (Wen et al., 2023)
19.9
32.3
DriveWM (Wang 2024)
8
20.7
1 26.8
40.0
19.7
1.563
8
26.3
39.6
20.0
1.563
Glad (Ours)
28.3
41.3
22.7
1.526
16
27.7
40.5
21.9
1.535
Practicality for generation. To validate the practicality of method in driving scenes, we gen- erate all frames the nuScenes validation set using BEV layout. Instead of using sliding window strategy, generate the video per 8 frames. We employ the video-based 3D object detection ap- proach, StreamPETR (Wang et al., 2023a), to perform 3D object detection on both ground-trun -trtth and synthetic validation Tab. 2 reports 3D detection and multi-object tracking results. It can be observed that, compared to Panacea (Wen et al., 2023), our Glad performs better on all 3D object detection metries, which reflect that our Glad can generate more realistic videos for 3D object de- tection. Further, our Glad achieves an mAP of 28.3 and an NDS of 41.3, which correspond to 74.9% and 83.6% of the Oracle, respectively. We also report the results when generating the video per 16 frames, the performance of the 3D object detection and tracking drop slightly.
Another crucial application is the expansion of driving dataset to boost the performance of percep- tion model. We first generate synthetic nuScenes training set using Gaussian noise as the reference frame and BEV layout. Then, we first pre-train StreamPETR on this synthetic training set, and then fine-tune it on real training set. Tab. 3 presents the results of baseline, Panacea (Wen et al., 2023), and our Glad. Compared to the offline Panacea, our online approach Glad achieves the compa- rable performance, significantly outperforming the baseline. We further validate the effectiveness using nulmage pre- tee-rrainnng that can provide a better initialization. The baseline achieves the NDS of 49.4 and AMOTA of 33.9, having 2.5 and 3.7 improvements compared to using ImageNet (Deng et al., 2009) pre-tree mre-trainigg. respectively. Compared to the stronger baseline, our Glad also has 1.9 improvement terms of NDS, and 2.6 improvement in terms of AMOTA.

---

Published as a conference paper at ICLR 2025
Table Impact training augmentation using generated synthetic data. In top part, we employ generated data to pre-train the perception model StreamPETR, and then train it on real data. We set the official reported results where StreamPETR trains on real data only as Baseline. bottom part, orderly train the StreamPETR on nulmage, generated data, and real data.
Method
nulmage
Generated
Real
MAPt
NDSt
AMOTA
AMOTPL
Bascline
34.5
46,9
30.2
1.384
Panacea
37.1 (+2.6)
49.2 (+2.3)
33.7 (+3.5)
1.353
Glad (Ours)
37.1 (+2.6)
49.2 (+2.3)
33.4 (+3.2)
1.356
Baseline
37.8
49.4
33.9
1.325
Glad (Ours)
39.8 (+2.0)
51.3 (+1.9)
36.5 (+2.6)
1.295
Table 4: Performance on downstream tasks of the online vectorized HD map construction. We retrain the Map - under 512x256 pixels with ResNet-50 backbone, and report the results on real data as Oracle or Baseline.
Method
Vidco Length
APdiciderT
AP APpoundary?
mAPt
(a)
Oracle Glad (Ours)
41
42.6
48.4
50.6
47.2
8
25.7
32.1
38.1
32.0
Method
Generated
APped
APaicider
APbonndary
mAPt
(b)
Baseline
42.6
48.4
50.6 53.9 (+3.3)
47.2
Glad (Ours)
49.4 (+6.8)
52.9 (+4.5)
$2.1 (+4.9)
Furthermore, we report the performance of our model on online vectorized HD map construction. As shown in Tab. 4(a), for the generated validation set, Glad achieves mAP of 32.0, while the Oracle performance is 47.2. In Tab. 4(b), after pre-training on the generated training set, compared the baseline, our Glad demonstrates an improvement of 4.9 in mAP.
Potentiality for simulation. To assess the ca- pability of Glad simulator, design experiment to evaluate simulation stability over time. Specifically, we generate video data us- ing reference frame, and perform 3D detection from frame to frame respectively, and com- pare the detection performance data and real data. Fig. 4 gives the relative detection performance between simulated data and real data. The lower relative detection per- formance is, the simulated data collapse. Al- though the performance begins to decline, quickly converges high percentage, such the NDS around 85%. It demonstrates that our Glad has a good potentiality for simulation.
NDS: MAP:
90%
69%
simulated
Frames
Figure The relative detection performance of simulated data compared to real data, as the number simulated frames increases. tends to slightly decrease first two frames and then be- comes stable.
4.3 ABLATION STUDY
Table Ablation study of proposed method.
(a) The effectiveness of Latent Variable Propagation
GPU memory consumption
Model
FID.
FVD.
mAPt
NDS
Method
0
2
8
Baseline 20.85
26.8
40.0
SWS SDS
31GB
+14GB
+28GB
OOM
+LVP
12.57
207
28.3
41.3
31GB
+7GB
+7GB
+7GB
(c) The impact of training video chunks
(d) The impact inference video length
Chunk
FID|
FVD.
mAP
NDSt
Length
mAPt
NDSt
AMOTAt
AMOTP
1
14.43
198
27.8
40.2
4
28.1
41.2
22.1
1.540
2
12.57
207
28.3
41.3
8
28.3
41.3
22.7
1.526
5
14.13
208
27.7
41.0
16
27.7
40.5
21.9
1.535
8

---

Published as a conference paper at ICLR 2025
Here employ StreamPETR (Wang et al., 2023a) to evaluate detection and tracking performance for ablation study.
On the LVP module. Tab. 5(a) compares baseline and our Glad. The bascline directly generates video data from Gaussian noise, while our Glad uses latent variable propagation (LVP), Compared to the baseline, our Glad with LVP has the improvement of 1.3 and 1.5 in terms of NDS and mAP. We also report the quality of image generation, where the FID reduces from 20.85 to 12.57. These results demonstrate that the simple yet effective LVP module is vital for our Glad.
On the SDS module. Tab. 5(b) compares memory usage on single NVIDIA A 100 GPU of streaming data sampler (SDS) in our Glad and sliding window sampler (SWS) in Panecea. Compared to SWS, our SDS does not increase memory usage with the increasing video length. The reason is that we sample video orderly continuous iterations.
Impact of the number of video chunks during training. During training, we split original training video into several video chunks balance video length and data diversity. Tab. 5(c) gives the impact of different length of video chunks. When the number of chunks is equal to 2, it has the best performance.
Impact of the length of generated video during inference. During inference, we can generate video data of different lengths. Tab. 5(d) gives the impact of different video lengths during inference. We observe that it has the best performance when the length is equal to 8. When the length is larger, the performance slightly drops.
Impact of LN Layer. We employ LN layer to normalize the latent features as in Eq. 1. Without LN layer, we observe the training loss escalating to NAN, highlighting the importance of LN layer.
Frane
Frone
Figure 5: Visualization of data generation examples. We generate video clip by feeding Gaussian noise to our Glad, with BEV layout sequences starting at index 2208 of the nuScenes validation set.
deccttittdeed
Beicc bec
Figure 6: Demo of editing input conditions. The BEV's perspective visualization of input condi- tions is shown in the first column, where we have selected the conditions at index 74 of the nuScenes validation set.

---

Published as a conference paper at ICLR 2025
4.4 QUALITATIVE ANALYSIS
Here we provide multi-view generation examples of Glad. As in Fig. 5, as a generator, our Glad generates vieti-view images (corresponding to each row) with strong cross-view consisteney. More- over, Glad maintains strong temporal consistency from frame I to frame 8. In Fig. 6, we demonstrate the capability of generating outputs under edited input conditions using our Glad. The first row dis- plays the ground truth as a reference. Subsequent rows show frames selected from a video based on the same reference frame, with modified conditions for each current frame. In the second row, frames are generated under original conditions. In the third row, a car following the ego is removed. In the fourth row, a new branch road is added, and the last row, the ego vehicle switch lanes to the adjacent one. Our Glad effectively manages these conditions while maintaining consistency across frames under the same reference.
5 CONCLUSION
In this paper, we introduce simple framework, named Glad, to generate video data in online framework. Our proposed Glad extends Stable Diffusion for video generation by introducing two novel modules, including latent variable propagation and streaming data sampler. The latent variable propagation views the denoised latent features of previous frame as noise prior for image generation current frame, leading to maintain good temporal consistency. The streaming data sampler samples the video frame orderly at continuous iterations, enabling efficient training. We perform the experiments on the widely-used dataset nuScenes, which demonstrates the efficacy our proposed method as a strong baseline for generation tasks.
We observe that our proposed method struggles to generate high- quality video data under high dynamic scenes. In these scenes, the temporal consistency of objects still needs to be improved. In future, we will explore improving high dynamic object generation.
Limitations and future work:
6 ACKNOWLEDGE
The work was supported by National Science and Technology Major Project of China (No. 20232D0121300), and also supported by National Natural Science Foundation of China (No. 62271346).
REFERENCES
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563- 22575, 2023.
Holger Caesar, Varun Bankiti, Alex H. Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. arXiv preprint arXiv: 1903. 1027, 2019.
Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint 2rXiv:2402.13244 2024.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv: arXiv: pet.::9070011511 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database, In JEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference Robot Learning, pp. 1-16, 2017.
10

---

Published as a conference paper at ICLR 2025
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In IEEE/CVF International Conference on Computer Vision, pp. 9710-9719, 2021.
Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Din Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601.20 2023.
Shenyuan Gao, Jiazhi Yang. Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405. 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprini arXiv:2403. 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, pp. 6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv: 10.02303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633- 8646, 2022b.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205. 2022.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. Gaia-1: generative world model for autonomous driving. arXiv preprint arXiv:2309. 17080, 2023.
Jiahui Huang, Yew Ken Chia, Samson Yu, Kevin Yee, Dennis Küiseer, Eva G Krumhuber, Dorien Herremans, and Gemma Roig. Single image video prediction with auto-regressive gans, Sensors, 22(9):3533, 2022.
Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi- camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.1199 2021.
Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao. Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving. arXiv preprint arXiv:2311.13549 2023.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient au- tonomous driving. In IEEE/CVF International Conference on Computer Vision, pp. 8340-8350, 2023.
Xiaohui Jiang, Shuailin Li, Yingfei Liu, Shihao Wang, Fan Jia, Tiancai Wang, Lijin Han, and Xi- angyu Zhang. Far3d: Expanding the horizon for surround-view 3d object detection. In AAAI Conference on Artificial Intelligence, pp. 2561-2569, 2024.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114 2013.
Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit- Yan Yeung, et al. Coda: A real-world road corner dataset for object detection in autonomous driving. In European Conference on Computer Vision, pp- 406-423 Springer, 2022a.
11

---

Published as a conference paper at ICLR 2025
Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusion: sayout-guided multi-view driving scene video generation with latent diffusion model. arXiv preprint arXIv:2310.0771, 2023a.
Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for muewi-view 3d object detection. In AAAI Conference Arlificial Intelligence, pp. 1477-1485, 2023b.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning seeee--wee representation from multi-camera images via spa- tiotemporal transformers. In European Conference on Computer Vision, pp. 1-18, 2022b.
Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construc- tion. arXiv preprint arXiv:2208. 144377 21443,2002 2022.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for viewi-view 3d object detection. In European Conference on Computer Vision, pp. 531-548. Springer, 2022.
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi nalti-camera images. In IEEE/CVF International Conference on Computer Vision, pp. 3262-3272, 2023.
Jiachen Lu, Ze Huang. Jiahui Zhang, Zeyu Yang, and Li Zhang. Wovogen: World volume-aware dif- fusion for controllable multi-camera driving scene generation. arXiv preprint arXiv:2312.02994, 2023.
Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, al. One million scenes for autonomous driving: Once dataset. arXiv preprint arXiv: 106. 1037, 2021.
Gaurav Parmar, Richard Zhang, and Jun- Zhu. On aliased resizing and surprising subtleties in gan evaluation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11410-11420, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, and al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695 2022.
Xiaogang Shi, Bin Cui, Gillian Dobbic, and Beng Chin Ooi. Uniad: A unified ad hoc data processing system. ACM Transactions on Database Systems, 42(1):1-42, 2016.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint NXX::2010000502. 2020.
Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird's- eye view layout. IEEE Robotics and Automation Letters, 94):35578 -3585, 2024.
Brooks Tim, Peebles Bill, Holmes Connor, DePue Will, Guo Yufei, Jing Li, Schnurr David, Taylor Joe, Luhman Troy, Luhman Eric, Ng Clarence, Wang Ricky, and Ramesh Aditya. Video genera- tion models as world simulators, 2024.
Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson. Neurad: Neural rendering for autonomous driving. arXiv preprint arXiv:2311.15260, 2023.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recogni- fion, pp. 1-26-1535. 2018
12

---

Published as a conference paper at ICLR 2025
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. arXiv preprint arXhh:1812.017111 2018.
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In /EEE/CVF International Con- ference on Computer Vision, pp. 3621-3631, 2023a.
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards real- world-driven world models for autonomous driving. arXiv preprint wrXiv:2309.09777 2023b.
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14749-1444 14759, 2024.
Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. arXiv preprint arXiv:2311.16811 2023.
Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jianmin Bao, Yuhui Yuan, al. Art-v: Auto-regressive text-to-video generation with diffusion models. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7395-7405, 2024.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khan- delwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Ar- goverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.000 00493, 2023.
Guohang Yan, Jiahao Pi, Jianfei Guo, Zhaotong Luo, Min Dou, Nianchen Deng, Qiusheng Huang, Daocheng Fu, Licheng Wen, Pinlong Cai, et al. Oasim: open and adaptive simulator based on neural rendering for autonomous driving. arXiv preprint arXiv:2402.03830 2024a.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104. 10157, 2021.
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians for modeling dynamic urban scenes. arXiv preprint arXiv:2401.01333 2024b.
Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al. Emernerf: Emergent spatial-temporal scene de- composition via self-supervision arXiv preprint arXiv:2311.02077 2023a.
Kairui Yang. Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with multi-perspectiv consistency via bev sketch layout. arXiv preprint arXiv:2308. N30.:2308..0661,, 2023b.
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma. Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1389-1399, 2023c.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awan implicit generative adversarial networks, arXiv preprint arXiv:2202. 10571, 2022.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Zhongwei Zhang, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Ting Yao, Yang Cao, and Tao Mei. Trip: Temporal residual learning with image noise prior for image-to-video diffusion models. In IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 8671-8661 1-8681, 2024.
13

---

Published as a conference paper at ICLR 2025
Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: LIm-enhanced world models for diverse driving video generation. arXiv preprint arXiv:2403.06845. 2024.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11016 2022.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PRELIMINARY: LATENT DIFFUSION MODEL
The diffusion model (Ho et al., 2020) is sophisticated probabilistic approach for image generation. typically encompasses two phases: the forward diffusion process which the data is gradually injected with Gaussian noise, and the reverse process that learns to reconstruct the original data from the noise.
Specifically, in the forward diffusion process, the data Xo is transformed into a sequence of latent variables x1,...; Xy following a predefined noise schedule , Sr:
:....::yy
T II ((xx g(xe(xx-1) N(x; / 1, (=1
Suppose that p(xr) = (xt; O, I). The reverse process modeled Markov chain with learned
transitions 0, aiming to recover the original data from the noisy latent variables:
T
po (x(:r) = p(xx) II (o (o(x (V(xx11; t), Eo (x, (x,,t)) t)).
(4)
To maintain generation quality with limited computational resources, latent diffusion models like Stable Diffusion (Rombach et al., 2022) perform the forward and reverse processes in the latent space of pretrained VAE (Kingma & Welling, 2013). As in Fig, 2, the encoder function & maps the input data X₂ to latent space representation Zg = {xxo), and the decoder reconstructs the data from the latent space: Xg D(zo).
A.2 MORE EXPERIMENTS AND VISUALIZATION RESULTS
Table 6: Impact of multi-scale training on video generation.
Mult-scale Training
FID 4
FVD 4
mAPt
NDS1
AMOTA
AMOTPL,
12.78
211 207
27.9
41.2
22.4
1.529
12.57
28.3
41.3
22.7
1.526
Impact of multi-scale training. Following the training configuration established Panacea, multi-
scale training adopted the default strategy. Specifically, the original images are randomly resized to scales ranging from 0.3 to 0.36 of their original dimensions. As demonstrated in Tab. 6, multi-scale training exhibits negligible effects on both perception and generation performance. To streamline the implementation, our proposed framework, Glad, retains multi-scale training as its default configuration.
Table 7: Performance gains from external trajectory-based scene generation.
Training Data
Image Size
Backbone
Split
mAP
NDS
Real nuScenes
512x256
VoV-99
val 46.18
55.58
Generated AV2 Real nuScenes
512x256
Vo
val
46.92
56.64
Enhancing model performance with external trajectory generation. We explore cross-domain scenario generation by incorporating trajectories from the Argoverse 2 (AV2) dataset (Wilson al., 2023), which contains 360-degree perception data and diverse driving scenarios. Using Far3D (Jiang et al., 2024) as the unified evaluation framework (compatible with both AV2 and nuScenes datasets), observe 1.06 improvement in terms of NDS on the nuScenes validation set (Tab. 7). This demon- strates that trajectory-bast based scene generation can enhance model generalization despite domain gaps.
15

---

Published as a conference paper at ICLR 2025
Table Comparison when only using generated data for training.
Method
nulmage
Generated
MAPt
NDS1
AMOTA
AMOTP
J
Panacea
22.5
36.1
Glad
27.1
40.8
21.6 24.9
1.569
Glad
30.0
42.8
1.500
Table 9: Per-class detection performance. The abbreviations 'CV' and "TC' denote Construction Vehicle and Traffic Cone, respectively.
Monoccycle
Bicycle
Buriic
Ocacle Onucle (R)
0.308
0.201
0.292
0,495
72.3
538
91.0
Performance comparision with exclusively generated data. In Tab. 8, we present the performance of Glad only using the generated data for training. When trained solely on synthetic data, Glad outperforms Panacea by 4.6 in terms of mAP and 4.7 in terms of NDS.
Detail the detection performance. classes can reflect the generation quality of each category. Since Panacea does not provide results broken down by category, we compare our results with the Oracle results from StreamPETR. It can be observed that common categories such Car and Traffic cone have higher generation quality compared to rare categories like Construction Vehicles and Trailer.
Tab. 9, the detailed detection performance the 10
4009
180%
3003
90%
Num of Objects
2009
1008
18 61%
12
Existinnc
Figure 7: Analysis of object persistence in frames.
Object continuous existence frames analysis. We analyze object persistence in frames to un- derstand why using video frames (such 16 frames instead of does not further improve downstream task performance. As shown in Fig. 7, our analysis of the nuScenes dataset reveals that 96.2% of objects appear in fewer than 8 consecutive frames, and 99.7% appear in fewer than 16 consecutive frames.
Ablation study on error accumulation. As shown in Fig. 8, we evaluate error accumulation by comparing three distinct methods of modifying the noise prior. In the Tab. 10, using denoised latent features achieves the best performance all evaluation metrics. Therefore, we utilize the denoised latent feature.
Ablation study on temporal attention layer. shown in the Tab. 11, using the temporal layer for propagating temporal information yields an FID score of 17.91 and an FVD score of 183. While the FID is slightly worse than that latent variable propagation (LVP), the FVD is marginally better. This suggests that the temporal layer more effectively models temporal propagation but has limitations in spatial modeling of single frames. Overall, LVP achieves a better balance in both spatial modeling and temporal propagation. Moreover, the approach using the temporal layer, which requires modeling multiple frames simultancously, is considerably more expensive in terms of memory consumption and training time.
16

---

Published as a conference paper at ICLR 2025
Keel image n
Keal Image n-
Glad
Sup.
Glad
(b)
Glad
Glad
Training
Cache
Genarati
Caoit
(c)
Glad
Glad
xT) feature
Figure 8: Different noise prior. Here, "Sup." refers the training supervision for Glad, with LayerNorm omitted for clarity. In (a). we directly use the latent feature of real image as the noise prior. (b), we add Gaussian noise the latent feature, while our Glad in (c) directly denoises and caches latent features.
Table 10: The influence for error accumulation under different noise prior. Three strategies correspond to Fig. 8, the strategy (a) is the latent freatures of privious frame, the strategy (b) is the noise added latent freatures of privious frame, and the strategy (c) is newly generated by our Glad.
Strategy
FID +
FVD
MAPt
NDSt
(a)
49.02
607
15.5
30.9
(b)
32.43
362
19.5
35.5
(c)
12.57
207
28.3
41.3
Table 11: Performance with temporal attention layer.
Temporal modeling
FID +
FVD |
Mcmory
Training Time
Latent Variable Propagation (LVP)
12.57
207
38GB
I Days
Temporal Layer (TL)
17.91
183
66GB
5 Days
Visualization results on corner case scenes. To generate corner cases, we utilize the CODA (Li al., 2022a) dataset, which is specifically designed for real-world road corner cases. CODA encom- passes three major autonomous driving datasets, including 134 scenes from nuScenes. this subset of 134 nuScenes scenes. After filtering to identify which scenes belong to the validation set, we retain 30 valid scenes. These 30 scenes serve as our corner case examples, and we present selected generation results in Fig. 9.
focus on
17

---

Published as a conference paper at ICLR 2025
case
case 2
case 3
case 4
case 5
Figure 9: Corner cases generation. From the nuScenes validation dataset, we showcase five corner cases that demonstrate generation capabilities: car making sudden turn in the opposite lane (case 1), traffic cones along the roadside (cases 2 and 3), unusual roadside obstacles (case 4), and unconventional road signs (case 5).
18


Reviews:
[{"review_id": "MthOKtEKJZ", "reviewer": "Reviewer_B154", "summary": "Collecting large-scale real-world driving data is expensive and labor-intensive. Diffusion generative models have achieved remarkable success in creating diverse and  high-quality video  from textual prompts. Therefore, this paper proposes to use diffusion model as a data generator which produce video data frame by frame in an online manner. To maintain temporal consistency, the authors propose latent variable propagation. The Glad can be used as generator and simulator, according to input. Experimental results show Glad is able to to generate videos of arbitrary lengths and exhibiting good flexibility in the variations of simulated trajectory.", "questions": "1. Can the authors clarify with why Glad cannot achieve further improvement by using more video frames in Table 2? I would appreciate if this was clarified.\n\n2. How about the performance of Glad just use the generated data for training.\n\n3. Effective generation of corner cases is essential for providing key data support the model, as mentioned in abstract. I'm wonder if Glab is able to generate such data.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation for using diffusion models for data generation in autonomous driving is clearly a promising direction, reducing the labor-intensity. \n\n2. The paper is well-written and easy to follow. Fig.2 provides a clear illustration of the video data generation pipeline, which makes me easy to understand the method proposed in this paper. \n\n3. The framework is designed similar to the recurrent network, using the previous frame to predict the current frame. This is make sense for achieving arbitrary length video generation.", "weaknesses": "1. From the method and evaluation sections of the paper, it's not very clear whether this method is able to address the error accumulation when the frame is generated one by one. In the method section, it will be useful to clarify this and explain how this method is able to do so. \n\n2. It is unclear to me how many training data used for Glad and other methods in Table 1, such as Panacea and Oracle.\n\n3. It would be interesting if the authors could show the detailed detection performance of 10 classes on the nuScenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R80Gv6NQvB", "reviewer": "Reviewer_8mSK", "summary": "The authors propose a latent propagation method for generating long and consistent driving video in a frame by frame manner.\nThe latent pronation module simply use the denoised latent of frame t as the noise latent for denoting the latent of frame t+1.\nA special streaming data sampler is designed to enable the training of this model.\nExtensive experiments on nuscenes showcases the generation quality and effectiveness of using generated data to improve the downstream perception tasks.", "questions": "1. Intuitively, using the denoised previous frame latent as noise latent seems poses heavy restriction on drastic content change during generation. Could the authors provide more insight about how Glad balance between  frame-to-frame consistency and the quick motion of generated video.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose a simple yet effective framework for video generation in driving scenario, the idea of latent propagation is interesting.\n2. Although missing some comparison, the experiments and ablation studies are generally solid.", "weaknesses": "1. Confusing writing and missing key reference. I think the whole idea of Glad is to denoise frame t+1 from the fully denoised latent of frame t instead of gaussian noise as suggested in Figure 2, but the whole section 4 tells very little about this fact except the eq 3. Also, the idea is not utterly new as this is know as image noise prior in TRIP[1], which also show great FID/FVD performance compared with other video generation models. \n2. Serious issue in paper organization. The section 3 is pretty much redundant as the latent diffusion is introduced in almost 3 years ago and is now well known to the whole AI community and general technique enthusiasts. Also there are tens of works applying stable diffusion or stable video diffusion for video forecasting or world modeling in autonomous driving.\n3. Poor literature review. Why not compare with DriveWM[2] and MagicDrive[3] which also do controlled multi-view multi-frame generation and report results on detection and mapping. Also if the authors want to highlight the streaming generation in driving scene(as indicated in the paper title), Vista[4] is a very important method to compare.\n\n[1] TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models, CVPR 2024\n[2] Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving, CVPR 2024\n[3] MAGICDRIVE: STREET VIEW GENERATION WITH DIVERSE 3D GEOMETRY CONTROL, ICLR 2024\n[4] Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MVi4TJ2LNI", "reviewer": "Reviewer_wGv7", "summary": "This paper proposes “Glad”, to generate visual data from scenarios of autonomous driving in a frame-by-frame manner, where most of the previous works generate the entire sequence at a certain amount of time. It is done by injecting the previous frame noise into later frames. In addition, it designs a continuous sampling the original images into a sequence. Experiments are conducted on nuScenes dataset for video generation, and boosting 3D detection.", "questions": "as above weaknesses", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper is overall easy to read. \n\nUse a previous frame noise to initiate the following frame noise seems to be a good way to maintain the consistency\n\nThe streaming data sampler seems to be a good engineering approach to cache the previous timeframe’s noise and inject into the second iteration instead of recomputing.", "weaknesses": "I have several concerns regarding this paper’s motivation, technical novelty, experiments and presentations. \n\n## Motivation concerns\n\nIn my humble opinion, this paper has makes a seemingly interesting claim for generating scenes in a frame-by-frame manner but unclear reason why we should do this. \n\nFor example, the authors claim in the abstract that “these approaches suffers from unseen scenes or restricted video length”. However, as I understand correctly, the recent diffusion based method can all support such sampling strategy to realize infinite sequence generation. For example, Drive-WM (CVPR2024) and Panacea (CVPR2024) also use the generated frame as the subsequent condition for long video generation. As for unseen scenes, as a layout-dependent generation framework, people in general can pass any unseen layout to generate such unseen scenes. A most recent one [A], although this one is still an Arxiv paper, shows the capability of generating such unseen scenes to boost end-to-end algorithms. Though this cannot be used to directly penalize this work is wrong, it shows that previous approaches can indeed generate unseen scenes, which contradicts to the paper’s core motivation. \n\n[A] Ma et al. Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation, arxiv\n\n\nThe second motivation is depicted in Figure 1 (b-c). This paper claims that previous papers focused on offline generate scenes. If I understand correctly, this means previous method takes a pre-defined trajectory, while this paper can take a dynamic trajectory. However, is this merely an engineering approach? After all, I can also passed a more flexible trajectory to the previous generation method.\n\nIn addition, even though this motivation is true when you try to encode this “Glad” method into some rendering engine to build a close loop simulator, I did not see any of such validation in the latter experiment section. So I am not convinced this is a valid motivation.\n\nAll in all, this paper’s motivations need some further clarification. \n\n\n## Technical novelty concerns\n\nIs this using a single frame as condition to generate next frame already done in DriveWM and Panacea, by setting the sequence length to 1 frame and keep generating? Please clearly discuss what are the differences between their’s approach and this one. \n\nStreaming data sampler is an engineering effort without much technical novelty. It is just a simple caching mechanism during the python programming. \n\n## Empirical validation concerns \n\n\nAs aforementioned, the motivation of injecting dynamic scenes is not well justified during the experiment sections. The only downstream applications other than video quality is 3D object detection. Please at least use end-to-end autonomous driving or 3D object tracking tasks to showcase if the motivation is validated. \n\nIn addition, there is no validation on long sequence video quality. \n\nAlso, does the word “online” meaning that on-the-fly rendering? Can you show the complexity and time benchmarking?\n\n## Presentation concerns\n\nThis paper made simple mistakes in presentation. For example, it should use ‘citep’ when mentioning a method in a third perspective rather than use ‘cite’ like CVPR template.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hGTZEysJgR", "reviewer": "Reviewer_XvyP", "summary": "This paper generates multi-view video data in a frame-by-frame style for autonomous driving.\n\nTo be honest, I reviewed this paper at NIPS, and seeing it again, I am pleased that the authors have taken into account previous reviewers' feedback, such as modifying the wording and adding discussions about previous methods. However, since there were no changes made to the methodology, considering the limited novelty and suboptimal results, I will maintain a negative score.", "questions": "See Weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposed Glad, which generates multi-view video data in a frame-by-frame style and can generate videos of arbitrary lengths theoretically.\n\n2. The paper proposes two training tricks, i.e., Latent Variable Propagation and Streaming Data Sampler, to achieve frame-by-frame generation.\n\n3. The tables and figures are well-presented.", "weaknesses": "1. Latent Variable Propagation and Streaming Data Sampler appear to be simple training tricks. Many previous U-Net-based works with temporal modules (before Sora and SVD), such as ADriver-I, MagicDrive, and Drive-WM, also employ frame-by-frame video generation strategy. Many works in the video generation area also use cross-frame attention of each frame on the first/former frame to preserve the context, appearance, and identity consistency of the video: Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators; ControlVideo: Training-free Controllable Text-to-Video Generation; Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation Using Conditional Image Diffusion Models.\n\n2. The experimental results should provide a comparison between the proposed approach and previous methods, e.g., DriveDreamer-2, considering that DriveDreamer-2 has achieved significantly better results with an FVD of 55.7 compared to the author's Glad approach with 206.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nwBVGlO5DS", "reviewer": "Reviewer_MnsB", "summary": "The paper introduces Glad, a novel framework designed to generate and simulate video data in an online setting, addressing the limitations of existing methods in handling unseen scenarios and short video lengths. Glad ensures temporal consistency in the generated videos by incorporating a hidden variable propagation module, which uses the denoised hidden variable features from the previous frame as a noise prior for the current frame. The framework includes a stream data sampler that sequentially and iteratively samples frames from video clips, enabling efficient training and enhancing the model's ability to generate specific scene videos.", "questions": "My concern revolves around what I perceive as an overclaim by the authors regarding the simulation capabilities of driving video generation. How effectively the authors address this critical point will significantly influence my score on their work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Glad stands out by enabling online video generation, over traditional offline methods that are limited to fixed-length video generation.\n2. By treating the denoised hidden variable features from the previous frame as a noise prior for the current frame, Glad maintains a high level of temporal coherence, which is essential for realistic and seamless video generation.\n3. The Stream Data Sampler (SDS) is designed to improve training efficiency by sequentially sampling frames from video clips across multiple iterations. This method not only ensures consistency between training and inference but also optimizes the training process by caching the generated hidden features for use as noise priors in subsequent iterations.\n4. Glad integrates both data generation and simulation capabilities within a unified framework, offering a comprehensive solution for autonomous driving applications. Whether generating new scene videos from noise or specific scene videos given a reference frame, Glad demonstrates versatility and effectiveness.", "weaknesses": "1. The paper claims significant advancements in online generation of controllable videos for driving simulation, yet the supporting evidence is lacking. The experiments presented in Fig. 5&6 do not convincingly demonstrate the quality of video generation under novel trajectories from a scientific perspective. It is recommended that the authors provide additional qualitative analysis of Glad's performance under different novel trajectories (e.g.,  sharp turns, lane changes, or varying traffic densities) starting from the same initial frame to substantiate their claims.\n2. The paper does not clearly articulate the fundamental benefits of using past frames as priors for current frame generation compared to methods that incorporate temporal transformer layers, such as Panacea and DrivingDiffusion. An ablation study to validate the effectiveness of this approach is warranted, especially given that Tab. 1 shows Panacea achieving better video consistency (FVD) without the use of an initial frame. The authors can compare their approach with temporal transformer layers with Panacea/DrivingDiffusion, focusing on FID, FVD and computational efficiency.\n3. The authors do not explore or discuss the potential performance gains that could be achieved by training models on data generated with new trajectories. This is a significant oversight that limits the understanding of Glad's full potential. The authors may generate an additional batch of data using novel trajectories rather than original trajectories, use it as a training data for Tab. 3, and then see if this dataset produces a gain in the model.\n4. The training data used in the stage one is not detailed in the paper. At L285, the authors fail to specify whether the data comes from public datasets, web sources, self-collected data, or a combination thereof. This lack of transparency raises concerns about privacy issues, the necessity of data annotation, and the distribution of the data. The authors can provide a detailed breakdown of data sources, including preprocessing steps, annotation methods, and measures taken to address privacy concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)