Glad: A Streaming Scene Generator for Autonomous Driving

Authors: Bin Xie, Yingfei Liu, Tiancai Wang, Jiale Cao, Xiangyu Zhang
Primary area: generative models
Keywords: Video Generation; Autonomous Driving

Abstract:
The generation and simulation of diverse real-world scenes have significant application value in the field of autonomous driving, especially for the corner cases. Recently, researchers have explored employing neural radiance fields or diffusion models to generate novel views or synthetic data under driving scenes. However, these approaches suffer from unseen scenes or restricted video length, thus lacking sufficient adaptability for data generation and simulation. To address these issues, we propose a simple yet effective framework, named Glad, to generate video data in a frame-by-frame style. To ensure the temporal consistency of synthetic video, we introduce a latent variable propagation module, which views the latent features of previous frame as noise prior and injects it into the latent features of current frame. In addition, we design a streaming data sampler to orderly sample the original image in a video clip at continuous iterations. Given the reference frame, our Glad can be viewed as a streaming simulator by generating the videos for specific scenes. Extensive experiments are performed on the widely-used nuScenes dataset. Experimental results demonstrate that our proposed Glad achieves promising performance, serving as a strong baseline for online video generation. We will release the source code and models publicly.

Decision: accept (poster)
Reviewer scores: 6, 8, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730351516723

thread_type:
official_review

<thread_object>
creator: reviewer_mnsb

<content>
summary:
The paper introduces Glad, a novel framework designed to generate and simulate video data in an online setting, addressing the limitations of existing methods in handling unseen scenarios and short video lengths. Glad ensures temporal consistency in the generated videos by incorporating a hidden variable propagation module, which uses the denoised hidden variable features from the previous frame as a noise prior for the current frame. The framework includes a stream data sampler that sequentially and iteratively samples frames from video clips, enabling efficient training and enhancing the model's ability to generate specific scene videos.

strengths:
1. Glad stands out by enabling online video generation, over traditional offline methods that are limited to fixed-length video generation.
2. By treating the denoised hidden variable features from the previous frame as a noise prior for the current frame, Glad maintains a high level of temporal coherence, which is essential for realistic and seamless video generation.
3. The Stream Data Sampler (SDS) is designed to improve training efficiency by sequentially sampling frames from video clips across multiple iterations. This method not only ensures consistency between training and inference but also optimizes the training process by caching the generated hidden features for use as noise priors in subsequent iterations.
4. Glad integrates both data generation and simulation capabilities within a unified framework, offering a comprehensive solution for autonomous driving applications. Whether generating new scene videos from noise or specific scene videos given a reference frame, Glad demonstrates versatility and effectiveness.

weaknesses:
1. The paper claims significant advancements in online generation of controllable videos for driving simulation, yet the supporting evidence is lacking. The experiments presented in Fig. 5&6 do not convincingly demonstrate the quality of video generation under novel trajectories from a scientific perspective. It is recommended that the authors provide additional qualitative analysis of Glad's performance under different novel trajectories (e.g.,  sharp turns, lane changes, or varying traffic densities) starting from the same initial frame to substantiate their claims.
2. The paper does not clearly articulate the fundamental benefits of using past frames as priors for current frame generation compared to methods that incorporate temporal transformer layers, such as Panacea and DrivingDiffusion. An ablation study to validate the effectiveness of this approach is warranted, especially given that Tab. 1 shows Panacea achieving better video consistency (FVD) without the use of an initial frame. The authors can compare their approach with temporal transformer layers with Panacea/DrivingDiffusion, focusing on FID, FVD and computational efficiency.
3. The authors do not explore or discuss the potential performance gains that could be achieved by training models on data generated with new trajectories. This is a significant oversight that limits the understanding of Glad's full potential. The authors may generate an additional batch of data using novel trajectories rather than original trajectories, use it as a training data for Tab. 3, and then see if this dataset produces a gain in the model.
4. The training data used in the stage one is not detailed in the paper. At L285, the authors fail to specify whether the data comes from public datasets, web sources, self-collected data, or a combination thereof. This lack of transparency raises concerns about privacy issues, the necessity of data annotation, and the distribution of the data. The authors can provide a detailed breakdown of data sources, including preprocessing steps, annotation methods, and measures taken to address privacy concerns.

questions:
My concern revolves around what I perceive as an overclaim by the authors regarding the simulation capabilities of driving video generation. How effectively the authors address this critical point will significantly influence my score on their work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730541465441

thread_type:
official_review

<thread_object>
creator: reviewer_xvyp

<content>
summary:
This paper generates multi-view video data in a frame-by-frame style for autonomous driving.

To be honest, I reviewed this paper at NIPS, and seeing it again, I am pleased that the authors have taken into account previous reviewers' feedback, such as modifying the wording and adding discussions about previous methods. However, since there were no changes made to the methodology, considering the limited novelty and suboptimal results, I will maintain a negative score.

strengths:
1. This paper proposed Glad, which generates multi-view video data in a frame-by-frame style and can generate videos of arbitrary lengths theoretically.

2. The paper proposes two training tricks, i.e., Latent Variable Propagation and Streaming Data Sampler, to achieve frame-by-frame generation.

3. The tables and figures are well-presented.

weaknesses:
1. Latent Variable Propagation and Streaming Data Sampler appear to be simple training tricks. Many previous U-Net-based works with temporal modules (before Sora and SVD), such as ADriver-I, MagicDrive, and Drive-WM, also employ frame-by-frame video generation strategy. Many works in the video generation area also use cross-frame attention of each frame on the first/former frame to preserve the context, appearance, and identity consistency of the video: Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators; ControlVideo: Training-free Controllable Text-to-Video Generation; Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation Using Conditional Image Diffusion Models.

2. The experimental results should provide a comparison between the proposed approach and previous methods, e.g., DriveDreamer-2, considering that DriveDreamer-2 has achieved significantly better results with an FVD of 55.7 compared to the author's Glad approach with 206.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700318475

thread_type:
official_review

<thread_object>
creator: reviewer_wgv7

<content>
summary:
This paper proposes “Glad”, to generate visual data from scenarios of autonomous driving in a frame-by-frame manner, where most of the previous works generate the entire sequence at a certain amount of time. It is done by injecting the previous frame noise into later frames. In addition, it designs a continuous sampling the original images into a sequence. Experiments are conducted on nuScenes dataset for video generation, and boosting 3D detection.

strengths:
This paper is overall easy to read. 

Use a previous frame noise to initiate the following frame noise seems to be a good way to maintain the consistency

The streaming data sampler seems to be a good engineering approach to cache the previous timeframe’s noise and inject into the second iteration instead of recomputing.

weaknesses:
I have several concerns regarding this paper’s motivation, technical novelty, experiments and presentations. 

## Motivation concerns

In my humble opinion, this paper has makes a seemingly interesting claim for generating scenes in a frame-by-frame manner but unclear reason why we should do this. 

For example, the authors claim in the abstract that “these approaches suffers from unseen scenes or restricted video length”. However, as I understand correctly, the recent diffusion based method can all support such sampling strategy to realize infinite sequence generation. For example, Drive-WM (CVPR2024) and Panacea (CVPR2024) also use the generated frame as the subsequent condition for long video generation. As for unseen scenes, as a layout-dependent generation framework, people in general can pass any unseen layout to generate such unseen scenes. A most recent one [A], although this one is still an Arxiv paper, shows the capability of generating such unseen scenes to boost end-to-end algorithms. Though this cannot be used to directly penalize this work is wrong, it shows that previous approaches can indeed generate unseen scenes, which contradicts to the paper’s core motivation. 

[A] Ma et al. Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation, arxiv


The second motivation is depicted in Figure 1 (b-c). This paper claims that previous papers focused on offline generate scenes. If I understand correctly, this means previous method takes a pre-defined trajectory, while this paper can take a dynamic trajectory. However, is this merely an engineering approach? After all, I can also passed a more flexible trajectory to the previous generation method.

In addition, even though this motivation is true when you try to encode this “Glad” method into some rendering engine to build a close loop simulator, I did not see any of such validation in the latter experiment section. So I am not convinced this is a valid motivation.

All in all, this paper’s motivations need some further clarification. 


## Technical novelty concerns

Is this using a single frame as condition to generate next frame already done in DriveWM and Panacea, by setting the sequence length to 1 frame and keep generating? Please clearly discuss what are the differences between their’s approach and this one. 

Streaming data sampler is an engineering effort without much technical novelty. It is just a simple caching mechanism during the python programming. 

## Empirical validation concerns 


As aforementioned, the motivation of injecting dynamic scenes is not well justified during the experiment sections. The only downstream applications other than video quality is 3D object detection. Please at least use end-to-end autonomous driving or 3D object tracking tasks to showcase if the motivation is validated. 

In addition, there is no validation on long sequence video quality. 

Also, does the word “online” meaning that on-the-fly rendering? Can you show the complexity and time benchmarking?

## Presentation concerns

This paper made simple mistakes in presentation. For example, it should use ‘citep’ when mentioning a method in a third perspective rather than use ‘cite’ like CVPR template.

questions:
as above weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730739014036

thread_type:
official_review

<thread_object>
creator: reviewer_8msk

<content>
summary:
The authors propose a latent propagation method for generating long and consistent driving video in a frame by frame manner.
The latent pronation module simply use the denoised latent of frame t as the noise latent for denoting the latent of frame t+1.
A special streaming data sampler is designed to enable the training of this model.
Extensive experiments on nuscenes showcases the generation quality and effectiveness of using generated data to improve the downstream perception tasks.

strengths:
1. The authors propose a simple yet effective framework for video generation in driving scenario, the idea of latent propagation is interesting.
2. Although missing some comparison, the experiments and ablation studies are generally solid.

weaknesses:
1. Confusing writing and missing key reference. I think the whole idea of Glad is to denoise frame t+1 from the fully denoised latent of frame t instead of gaussian noise as suggested in Figure 2, but the whole section 4 tells very little about this fact except the eq 3. Also, the idea is not utterly new as this is know as image noise prior in TRIP[1], which also show great FID/FVD performance compared with other video generation models. 
2. Serious issue in paper organization. The section 3 is pretty much redundant as the latent diffusion is introduced in almost 3 years ago and is now well known to the whole AI community and general technique enthusiasts. Also there are tens of works applying stable diffusion or stable video diffusion for video forecasting or world modeling in autonomous driving.
3. Poor literature review. Why not compare with DriveWM[2] and MagicDrive[3] which also do controlled multi-view multi-frame generation and report results on detection and mapping. Also if the authors want to highlight the streaming generation in driving scene(as indicated in the paper title), Vista[4] is a very important method to compare.

[1] TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models, CVPR 2024
[2] Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving, CVPR 2024
[3] MAGICDRIVE: STREET VIEW GENERATION WITH DIVERSE 3D GEOMETRY CONTROL, ICLR 2024
[4] Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability, NeurIPS 2024

questions:
1. Intuitively, using the denoised previous frame latent as noise latent seems poses heavy restriction on drastic content change during generation. Could the authors provide more insight about how Glad balance between  frame-to-frame consistency and the quick motion of generated video.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730825406025

thread_type:
official_review

<thread_object>
creator: reviewer_b154

<content>
summary:
Collecting large-scale real-world driving data is expensive and labor-intensive. Diffusion generative models have achieved remarkable success in creating diverse and  high-quality video  from textual prompts. Therefore, this paper proposes to use diffusion model as a data generator which produce video data frame by frame in an online manner. To maintain temporal consistency, the authors propose latent variable propagation. The Glad can be used as generator and simulator, according to input. Experimental results show Glad is able to to generate videos of arbitrary lengths and exhibiting good flexibility in the variations of simulated trajectory.

strengths:
1. The motivation for using diffusion models for data generation in autonomous driving is clearly a promising direction, reducing the labor-intensity. 

2. The paper is well-written and easy to follow. Fig.2 provides a clear illustration of the video data generation pipeline, which makes me easy to understand the method proposed in this paper. 

3. The framework is designed similar to the recurrent network, using the previous frame to predict the current frame. This is make sense for achieving arbitrary length video generation.

weaknesses:
1. From the method and evaluation sections of the paper, it's not very clear whether this method is able to address the error accumulation when the frame is generated one by one. In the method section, it will be useful to clarify this and explain how this method is able to do so. 

2. It is unclear to me how many training data used for Glad and other methods in Table 1, such as Panacea and Oracle.

3. It would be interesting if the authors could show the detailed detection performance of 10 classes on the nuScenes.

questions:
1. Can the authors clarify with why Glad cannot achieve further improvement by using more video frames in Table 2? I would appreciate if this was clarified.

2. How about the performance of Glad just use the generated data for training.

3. Effective generation of corner cases is essential for providing key data support the model, as mentioned in abstract. I'm wonder if Glab is able to generate such data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732352308344

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all reviewers for their valuable and constructive comments. We have tried our best to revise the paper to address all concerns. *All revisions are marked in  **blue.*** Specifically:
***
- **@Reviewer B154**, we have analyzed the influence of noise prior on error accumulation, as shown in Fig.8 in the appendix. The experimental results in Tab. 6 demonstrate that using generated latent features as noise prior achieves the best performance across all evaluation metrics.

- **@Reviewer B154**, we have investigated why using additional video frames does not improve performance. Our analysis of object persistence in frames, shown in Fig.7 in the Appendix, reveals that this limitation is due to the inherent characteristics of the nuScenes dataset.

- **@Reviewer B154**, **@Reviewer wGv7** and **@Reviewer MnsB** ,we have included the generation results for corner case scenes in Fig.9 in the Appendix and provided additional long-form videos at the supplementary material.

- **@Reviewer 8mSK**, we have moved the original Section 3 to the Appendix section and have made the method section more clear.

- **@Reviewer 8mSK** and **@Reviewer XvyP**, we have included the results from contemporary video generation methods Drive-WM[1] and MagicDrive[2] in Tab.2.

- **@Reviewer MnsB**, we have incorporated a temporal layer into our framework for an ablation study. %while maintaining the original Gaussian noise as prior. 
The results are shown in Tab.7 in the Appendix. Compared to using the temporal layer, our Glad achieves a better balance between spatial modeling and temporal propagation.

- **@Reviewer wGv7**, we have replaced all "cite" commands with "citep".

- **@Reviewer MnsB**, to further explore the simulator potential, we have included a video in the supplementary material, demonstrating the end-to-end autonomous driving model VAD[3] performing inference on generated nuScenes validation set by our Glad.

Please let us know if you have any further questions. We are always looking forward to open discussions.

Sincerely,

Authors
***
[1]Wang, Yuqi, et al. "Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2]Gao, Ruiyuan, et al. "Magicdrive: Street view generation with diverse 3d geometry control." arXiv preprint arXiv:2310.02601 (2023).

[3]Jiang, Bo, et al. "Vad: Vectorized scene representation for efficient autonomous driving." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732353473819

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
PDF Revision And Supplementary Material Update

comment:
Dear Reviewers,

We appreciate your feedback and have revised our manuscript accordingly:

- Added more related works, such as TRIP and Vista. (Pg 3)
- Moved the original section 3 (preliminary) to Appendix A.1 (Pg 15)
- Revised method section to focus on our framework and clarify our training details (Pg 4-6)
- Added comparison with MagicDrive and Drive-WM in experiments (Pg 7)
- Fixed the format of the citations
- Added more experiments and visualization results in Appendix A.2 (Pg 15–17)
- Added 40-frame long videos and VAD simulation videos in supplementary material

---

We hope the newly provided results alleviate any confusion. We welcome any further feedback you may have and look forward to the opportunity to continue refining our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1759066142104

thread_type:
public_comment

<thread_object>
creator: ~jiawei_zhou4

<content>
comment:
Why is the temporal consistency of the VAD simulation in the supplementary materials so poor?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper got very diverging scores, so it takes some time to read through all comments and discussions. This paper presents Glad, a novel framework for online generation of multi-view video data for autonomous driving simulation. While reviewers acknowledge some limitations, most reviewers ultimately find the paper's contributions to be promising and worthy of acceptance.
As for some issues raised by Reviewer wGv7 and XvyP, authors have given reasonable explanations in details. To summarize, 
1. The proposed frame-by-frame generation pipeline provides more flexibility in autonomous driving simulation.
2. Latent variable propagation is different from cross-frame attention, noise inversion in training-free video generation works and noise prior in PYoCo and VideoFusion.
3. DriveDreamer-2 is trained and evaluated on the privately annotated videos, 12Hz instead of 2Hz annotation in original nuScenes dataset, which leads to better FVD scores.
4. Delphi is a concurrent work which does not contradicts to the submitted work.