DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving

Authors: Hao LU, Tianshuo Xu, Wenzhao Zheng, Yunpeng Zhang, Wei Zhan, Dalong Du, Masayoshi Tomizuka, Kurt Keutzer, Ying-Cong Chen
Primary area: applications to robotics, autonomy, planning
Keywords: 4D Gaussian Reconstruction;  Autonomous Driving

Abstract:
Photorealistic 4D reconstruction of street scenes is essential for developing real-world simulators in autonomous driving. However, most existing methods perform this task offline and rely on time-consuming iterative processes, limiting their practical applications. To this end, we introduce the Large 4D Gaussian  Reconstruction Model (DrivingRecon), a generalizable driving scene reconstruction model, which directly predicts 4D Gaussian from surround-view videos. To better integrate the surround-view images, the Prune and Dilate Block (PD-Block) is proposed to eliminate overlapping Gaussian points between adjacent views and remove redundant background points. 
To enhance cross-temporal information, dynamic and static decoupling is tailored to learn geometry and motion features better. Experimental results demonstrate that DrivingRecon significantly improves scene reconstruction quality and novel view synthesis compared to existing methods. Furthermore, we explore applications of DrivingRecon in model pre-training, vehicle adaptation, and scene editing. Our code will be made publicly available.

Decision: reject
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729568961352

thread_type:
official_review

<thread_object>
creator: reviewer_ebav

<content>
summary:
This paper proposes a feed-forward 4D reconstruction method that generates 4D scenes from surround-view video inputs in a single feed-forward pass.
The method involves 3D Position Encoding, Temporal Cross Attention, Gaussian Adapter, and Prune and Dilate Block. All these modules consist of the feed-forward 4D reconstruction pipeline. 
The PD-Block learns to prune redundant Gaussian points from different views and background regions and dilate Gaussian points for complex objects, enhancing the quality of reconstruction.
This paper also presents rendering strategies for both static and dynamic components, enabling efficient supervision of rendered images across temporal sequences.

strengths:
1.This paper first explores a feed-forward 4D reconstruction method for surround-view driving scenes, which promotes the development of feed-forward technology in the field of 4D reconstruction.

2.The proposed PD-Block learns to prune and dilate the Gaussian points and allows for Gaussian points that are not strictly pixel-aligned, which is innovative.

weaknesses:
1.The training process requires depth map ground truth, whereas comparison methods like Pixelsplat and MVSpalt can be trained without it. This reliance on depth ground truth during training restricts its practical applicability.

2.The dynamic objects are decomposed through segmentation and have only few categories (vehicles and people). This approach only separates dynamic and static pixels based on semantics, limiting its ability to achieve comprehensive 4D reconstruction of all dynamic objects.

3.Compared to scene-optimized methods, feed-forward reconstruction provides the advantage of generalization, eliminating the need of test-time optimization for each new scene (though it may lead to some decrease in accuracy compared to the scene-optimized method). In the papers of comparing methods MVSplat and PixelSplat, both of them present running time and memory consumption, demonstrating the efficiency of their feed-forward approaches. However, in this paper, while the authors claim their method is feed-forward, they do not provide an analysis of its running time and memory usage. I recommend including this efficiency analysis and comparing it with other methods to strengthen the evaluation. 

Besides, if the authors believe that efficiency is not a concern of this paper, then comparisons with other offline scene-optimized methods (e.g., DrivingGaussian) should be included.

4.If the possible application is to develop real-world simulators in autonomous driving (mentioned in the abstract of the paper), then there is no high requirement for the efficiency of reconstruction, and the existing offline scene-optimized 4D reconstruction method is also acceptable. However, feed-forward does not seem to have an advantage in terms of reconstruction accuracy.

questions:
1.What does “DA-Block” in line 202 refer to? It is not mentioned in the context.

2.Please refer to the questions and suggestions in the Weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730466526235

thread_type:
official_review

<thread_object>
creator: reviewer_3kki

<content>
summary:
This paper proposes a learning-based reconstruction method in a feed-forward manner in driving scenarios. It could predict 4D Gaussian primitives from multi-view temporal input. It is a very early work that explores learning-based generalizable reconstruction and rendering for autonomous driving. This paper also introduces a couple of downstream applications such as model pre-training and vehicle adaptation.

strengths:
-  It is a very early work that explores learning-based generalizable reconstruction methods for autonomous driving, demonstrating this paradigm could work in real-world driving scenarios.
- This paper is comprehensive since it not only develops the methods but also incorporates potential applications such as perception and driving tasks.
- The self-supervised pretraining task is insightful.

weaknesses:
- This paper does not demonstrate the model's generalization to different viewpoints. The authors claim the ability of vehicle adaption. However, only the camera intrinsic is changed. Could the predicted 4D Gaussians produce good rendering quality in viewpoints beyond the driving trajectories (different extrinsic)? A recent work[1] explores this direction.

- The resolution is relatively low.  The produced rendering quality cannot meet the requirements of practical use, such as camera simulation.

- It would be better to show the inference latency.

- The authors do not provide video demonstrations of the rendering results. It is hard to have a intuitive understanding of the actual performance.

[1] Freevs: generative view synthesis on free driving trajectory.

questions:
How does the scene edit (Fig.6) work? This procedure can be more detailed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613361321

thread_type:
official_review

<thread_object>
creator: reviewer_conn

<content>
summary:
Unlike previous methods (e.g., 3DGS/NeRF) that require thousands of iterations to reconstruct a scene, this work aims to predict a 3D scene representation using a neural network.

 The authors make several design choices to make this pipeline work (PD-block, regularization, 3D encoding, etc.). 

Experiments conducted on Waymo demonstrate better performance compared to other generalizable approaches.

strengths:
1. A generalizable and scalable approach that allows training of large models to learn priors from extensive data, generalizing to novel scenes.
2. **Almost** no 3D bounding box labels required for dynamic scenes, enhancing scalability.
3. Detailed explanations and extensive experiments on cross-data evaluation, downstream applications (data augmentation, pretrained model for perception, scene editing).

weaknesses:
1. Overcomplicated design:
   While I appreciate the effort in developing a generalizable model with dynamic-static decomposition, the model seems quite complex, requiring:
   * Multiple modules (image encoder-decoder, temporal cross-attention, Gaussian adapter, PD block, etc.)
   * Numerous regularization terms
   * Several pretrained models (DepthNet, DeepLab, SAM)

   This complexity may hinder downstream applications when used as a pretrained model. For instance, how fast is the model? Is it efficient enough for use in autonomy systems?

2. The realism is still lower compared to optimization-based approaches (e.g., 3DGS), and can only operate on low resolution (256x512) with a limited number of images.

3. (Minor point) The writing seems somewhat rushed, lacking thorough proofreading. Some potential issues:
   * L155, "corresponding intrinsic parameter E" should be K
   * L414 "evaluation on NOTA-DS6" should be Diversity-54

questions:
**Regarding efficiency and comparison with 3DGS**

What is the computational cost to train the model (how many hours on 24 GPUs)? 
How long does it take to reconstruct a 3D scene representation using your approach during inference? How does the efficiency compare to 3DGS, e.g., StreetGaussian on 256x512?

How does the realism compare to 3DGS (e.g., StreetGaussian at 256 × 512)? It's okay if it's worse; I'm just curious. 



**On 3D labels**
What is the performance without using 3D bounding boxes at all? I note that you use 3D bounding boxes as prompts for SAM. A label-free approach would make this work more impactful.

**On downstream applications**
How is UniAD implemented in Waymo? Would it be possible to conduct your experiments on nuScenes to follow the setting/implementation of UniAD?

**Miscellaneous**:
* How many frames are in the input during training?
* In Table 4b, what does "Training Num" refer to? Do you mean number of scenes? The PSNR seems quite high compared to Table 3.

Some questions may require additional experiments; please disregard if they're not feasible. However, I'm particularly interested in the efficiency and comparison with 3DGS.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732325702913

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We fixed some minor typos and uploaded the paper. If you have any further questions or if anything remains unclear,  please don’t hesitate to let us know. We would be more than happy to discuss your concerns in greater detail.  

We have made minor revisions to the paper, all of which are summarized as follows:

**Efficiency comparison:** Our approach is compared with both traditional optimization methods and the latest generalized feedforward networks in terms of latency and memory usage. Results and discussion are in section C of supplementary material.

**Detailed experimental process:** We have fixed some incorrect typos. We give more details of the pre-trained model and the image editing experiment.  We will proofread the paper thoroughly to enhance its writing, presentation, and layout.


Best wishes,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733024579232

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for further feedback

comment:
Dear Reviewers,

We sincerely apologize for troubling you once again, and we deeply appreciate the time and effort you have put into reviewing our submission. We responded further and look forward to discussing with you. As the ICLR discussion period has been extended, we still have approximately some days to continue the discussion. Please let us know if there are any additional points or concerns that you would like us to address before the discussion phase concludes. Your valuable feedback is highly appreciated and will help us further improve our work.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a driving scene reconstruction model, named DRIVINGRECON, which directly predicts 4D Gaussians from surround-view videos. While the idea is somewhat novel and offers a unique approach to scene reconstruction, the performance falls short compared to existing optimized methods. Additionally, the overall pipeline is excessively complex, which may hinder practical implementation and maintenance. Based on these strengths and weaknesses, the decision is not to recommend acceptance at this time.