STORM: Spatio-TempOral Reconstruction Model For Large-Scale Outdoor Scenes

Authors: Jiawei Yang, Jiahui Huang, Boris Ivanovic, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl, Peter Karkus, Danfei Xu, Yue Wang, Marco Pavone
Primary area: applications to robotics, autonomy, planning
Keywords: autonomous driving; reconstruction model; spatiotemporal

Abstract:
We present STORM, a spatio-temporal reconstruction model designed for reconstructing dynamic outdoor scenes from sparse observations. Existing dynamic reconstruction methods often rely on per-scene optimization, dense observations across space and time, and strong motion supervision, resulting in lengthy optimization times, limited generalization to novel views or scenes, and degenerated quality caused by noisy pseudo-labels for dynamics. To address these challenges, STORM leverages a data-driven Transformer architecture that directly infers dynamic 3D scene representations—parameterized by 3D Gaussians and their velocities—in a single forward pass. Our key design is to aggregate 3D Gaussians from all frames using self-supervised scene flows, transforming them to the target timestep to enable complete (i.e., "amodal") reconstructions from arbitrary viewpoints at any moment in time. As an emergent property, STORM automatically captures dynamic instances and generates high-quality masks using only reconstruction losses. Extensive experiments on public datasets show that STORM achieves precise dynamic scene reconstruction, surpassing state-of-the-art per-scene optimization methods (+4.3 to 6.6 PSNR) and existing feed-forward approaches (+2.1 to 4.7 PSNR) in dynamic regions. STORM reconstructs large-scale outdoor scenes in 200ms, supports real-time rendering, and outperforms competitors in scene flow estimation, improving 3D EPE by 0.422m and Acc5 by 28.02%. Beyond reconstruction, we showcase four additional applications of our model, illustrating the potential of self-supervised learning for broader dynamic scene understanding. For more details, please visit our project at https://jiawei-yang.github.io/STORM/.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730387081761

thread_type:
official_review

<thread_object>
creator: reviewer_hxs2

<content>
summary:
This paper introduces a novel pipeline called STORM for dynamic scene reconstruction in real-world environments. 

The core concept involves aggregating reconstructed 3D Gaussians and their motion into a unified representation, enabling the rendering of an amodal representation at any specified time step. 

Additionally, to ensure STORM's effectiveness in diverse scenarios, several significant enhancements have been implemented. 

Experimental results demonstrate the efficacy of the proposed method.

strengths:
1. The paper is well-written and accessible, making it easy to follow the proposed approach.

2. The core idea is straightforward and well-conceived. By integrating 3D Gaussians into a unified representation, STORM enables rendering at any time step, allowing the entire network to be supervised by the color of the target time step.

3. The experiments are comprehensive and informative, demonstrating the effectiveness of the overall pipeline as well as specific enhancements, such as Latent-STORM.

weaknesses:
The experiments are conducted solely on the Waymo Open Dataset, which may raise concerns about STORM’s robustness and generalizability to other datasets.

questions:
The paper highlights that previous reconstruction methods rely on “dense observations across space and time,” whereas STORM “reconstructs dynamic 3D scenes from sparse, multi-timestep, posed camera images.” An insightful ablation study would involve varying the number of camera views and input time steps to assess how effectively STORM performs with different levels of view sparsity. This would provide a clearer understanding of the system's robustness under sparse observation conditions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622257245

thread_type:
official_review

<thread_object>
creator: reviewer_n7na

<content>
summary:
The paper presents STORM, a spatio-temporal reconstruction model that uses a data-driven Transformer architecture to accurately reconstruct dynamic outdoor scenes from sparse camera images without explicit motion supervision. It employs 3D Gaussian Splatting to aggregate per-frame data into an amodal, cohesive representation, enabling consistent scene reconstruction over time. The model is self-supervised, relying on reconstruction loss and incorporating motion tokens to capture motion primitives efficiently. Extensive testing on the Waymo Open Dataset shows that STORM outperforms traditional methods in photorealism and speed, achieving real-time performance within 0.2 seconds per clip.

strengths:
1. The paper presents a clear and well-organized approach to addressing the defined problem of dynamic scene reconstruction.
2. The proposed method achieves state-of-the-art performance, showcasing its practical effectiveness and potential.
3. The video results available on the linked website are highly illustrative, providing clear insights into both the strengths and limitations of the proposed method.

weaknesses:
1. The evaluations are limited to the Waymo dataset, which raises concerns about the comprehensive validation of the method. To strengthen the claims, it would be beneficial to demonstrate the effectiveness of the model across multiple datasets.
2. While the paper targets large-scale outdoor scene reconstruction, existing works such as [1] also address similar challenges. If the primary focus is on dynamic scene reconstruction, the paper’s title should explicitly include the term ‘dynamic,’ and comparative results with dynamic 3D reconstruction methods like [2] should be included to provide context and highlight advancements.
3. One of the main contributions is the introduction of 'motion tokens' in the Transformer’s input sequence, alongside the unique loss function components (reconstruction loss, sky loss, and velocity regularization loss). However, the absence of ablation studies makes it difficult to discern the effectiveness of these specific modules. Adding ablation studies would enhance the paper by showing the impact of each proposed component.

[1] Lin, Jiaqi, et al. "Vastgaussian: Vast 3d gaussians for large scene reconstruction." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.
[2] Fischer, Tobias, et al. "Dynamic 3D Gaussian Fields for Urban Areas." *arXiv preprint arXiv:2406.03175* (2024).

questions:
1. Could the authors include comparisons across multiple datasets to validate the generalizability of the proposed method?
2. Could the authors provide ablation studies to demonstrate the effectiveness of each proposed module in the method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730632811190

thread_type:
official_review

<thread_object>
creator: reviewer_jkrr

<content>
summary:
This paper introduces STORM, a spatio-temporal reconstruction model for rendering dynamic outdoor scenes with sparse images. Combining image tokens, ray tokens, and time tokens, STORM uses a Transformer to predict attributes of 3D Gaussians (position, rotation, scale, opacity, color, velocity) at each timestep for real-time rendering. The training requires ground truth images, depth map and predicted sky mask from pretrained segmentation model.

strengths:
The paper is clearly written.

STORM may be the first method that combines feed-forward 3D Gaussian Splatting and dynamic scenes.

STORM achieves better rendering quality and scene flow accuracy than baselines.

weaknesses:
STORM only evaluates the quantitative performance in short video clips (about 2 seconds on Waymo open dataset). Applying STORM on 20-second videos requires processing in multiple video clips (L428) because of the heavy computation, which is also discussed in limitations. 

There is no quantitative ablation study.

questions:
In Fig.3, can authors add ground truth images for comparison? Based on current visualization, Latent-STORM sometimes performs better and sometimes performs worse than STORM.

Could authors provide some reference for handling exposure mismatches with affine transformation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668859048

thread_type:
official_review

<thread_object>
creator: reviewer_wu7p

<content>
summary:
This paper proposes a method for dynamic 3D reconstruction (4D reconstruction) of urban scenes from temporally and spatially sparse video input.

The model is based on a per-frame transformer architecture that takes image, ray (from given camera intrinsics and extrinsics), time and motion latent tokens as input as well as some special tokens for handling sky and varying camera exposure. The model takes tokens from an image coming from single camera v and timestamp t, and then processes these tokens jointly with a self-attention transformer, to output feature maps F^v_t for that image. These tokens are then decoded into pixel-aligned 3D Gaussian Splatting parameters G^v_t, which are used to reconstruct the input frame as well as to compute the depth map. In addition, the motion tokens are decoded into a set of motion queries and velocity bases, which are employed to estimate the velocity vector of each 3D Gaussian.

Given a set of training frames coming from one or several cameras, the model is run on each frame independently to obtain independent sets of Gaussians. These independent Gaussians can then be warped to any particular timestamp using the velocity estimates, and assuming velocity constancy.

In addition to the vanilla model, the authors propose a latent variant where instead of predicting the Gaussian parameters directly, the transformer first predicts higher dimensional vectors on a lower-dimensional grid, which are then upsampled to full-resolution using a CNN decoder.

The model is trained and evaluated on clips from the Waymo Open Dataset, using the provided camera intrinsics and extrinsics, and three frontal car cameras.

The authors evaluate their method against both per-scene optimization and feed-forward methods. For each 20 frame test clip, 4 frames are provided as inputs for reconstruction to each baseline, and the rest are used for evaluation. The results in Table 1 show that the proposed method obtains SOTA performance on reconstruction metrics compared to both per-scene optimization and feed-forward methods. In addition, the authors evaluate their method on the scene flow estimation task, also obtaining SOTA results.

Finally, the authors link a website with numerous quantitative results which allow to better assess the performance of the proposed method.

strengths:
This is an interesting method which applies per-frame feed-forward 3DGS models to a multi-camera scenario for the urban driving setting. While similar methods exist for object-centric scenes (Splatter Image, LatentSplat) and static objects, or even object-centric scenes and dynamic objects (4DGS Guanjun Wu et al.), I believe the extension to outdoor driving scenes is novel and relevant. The authors successfully devised strategies to deal with the complexities of such scenes, by explicitly modeling the sky and the differences in camera exposure.

Furthermore, the authors adopt a similar strategy for modeling dynamics through scene flow as in NSFF, but which results to be more tractable due to the usage of 3DGS as a discrete representation of the scene. This methods seems to be effective in modelling short term dynamics.

Quantitative and qualitative results show that the method achieves good new-view synthesis results from only a few seen views.

The proposed latent-STORM variant shows improved reconstruction of local details, such as for the articulated bodies of walking pedestrians.

weaknesses:
Some details of the method are not very clear from the paper. For instance, it's not clear how the aggregation and transformation processes are applied on the predictions of the 4 training frames for generating the predicted images \hat{I}. 

In addition, it's not clear whether eq. (4) is correct and how exactly the weighs w are computed as the exact shapes of q and k are not specified. Also, it's not clear how the two different estimates v_t^{-} and v_t^{+} from eq (1) are obtained from what is described in eqs. (3,4). Furthermore, not many details are given about how these weights can be used to generate the motion segmentation figures, and the impact of the number of motion basis. It's also not very clear how the background scene is treated during this motion estimation.

Regarding the quantitative results, it's not clear if the baselines for Table 2 have been retrained specifically on the same training set for fairness. Also, it's specified that the baseline models NSFP and NSFP++ require LiDAR input, but it's not clear whether these methods also benefit from having 3 different input views. 

Regarding the qualitative results, having 3D visualizations in addition to depth-maps would help further assessing the temporal consistency and geometric correctness of the predictions.

Finally some of the proposed applications or extensions in sec. 4.3 are not explained in sufficient detail. For instance, how the point trajectories are chained for point tracking, given that the local scene-flow estimations are sparse and don't cover the whole space. In addition, it's not clear how latents can be manipulated in the latent-STORM model for achieving the edition, control and inpainting effects.

questions:
- Please clarify how the images \hat{I} are computed for a given training batch and which aggregations and transformations are involved.

- Please clarify the shapes of q and k in eq. (4) and how the weights are used for generating the motion segmentation visualizations, as well as how the number of motion basis is determined.

- Please clarify how the baselines are trained and which inputs they require.

- If possible, having 3D visualizations of the results would be beneficial.

- Please provide additional details about how point-tracking can be obtained, and how the editing, control and inpainting behaviours can be obtained in latent-STORM.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732256690955

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We sincerely thank all reviewers for their detailed and constructive feedback, which has been invaluable in refining and improving our work. We are encouraged by the recognition that STORM represents the **first** feed-forward 4D scene reconstruction method for dynamic outdoor environments, a novel and promising direction. Reviewers commended STORM’s state-of-the-art performance in rendering quality and scene flow accuracy, the novel and grounded integration of 3D Gaussian Splatting, and the clear presentation of its pipeline. Below, we summarize the key updates made in response to reviewers' feedback. Our manuscript has been updated accordingly with all major revisions **marked in blue**. Meanwhile, we put the link to our anonymous project page here for easy accessibility: https://anonymousi079j.github.io/STORM_review/ 

### **1. Generalizability Across Datasets**

Reviewers `N7nA` and `HXs2` highlighted the importance of validating STORM’s robustness beyond the Waymo Open Dataset. To address this, we have extended our experiments to include evaluations on two additional datasets: NuScenes [1] and Argoverse2 [2].
STORM achieves state-of-the-art performance compared to other feedforward reconstruction methods on both datasets in terms of full-image PSNR and Depth RMSE, as shown in Table 2 of the revised manuscript. We summarize the results as below for convenience:
| Method   | NuScenes PSNR ↑ | NuScenes D-RMSE ↓ | Argoverse2 PSNR ↑ | Argoverse2 D-RMSE ↓ |
|----------|------------------|-------------------|--------------------|---------------------|
| LGM      | 23.21           | 7.34              | 22.93             | 14.20              |
| GS-LRM   | 24.53           | 7.71              | 24.49             | 14.70              |
| Ours     | **24.90**       | **5.43**          | **24.80**         | **13.51**          |
		

These results validate STORM’s robustness and versatility across diverse real-world scenarios.

We have also added 4D visualizations (3D points and point trajectories) for the Waymo, NuScenes, and Argoverse2 datasets to the project page. Please refer to the “4D Visualization” section in our project page for details. 

### **2. Ablation**
We appreciate the reviewers’ requests for a deeper analysis of STORM’s components and its robustness under sparse observation conditions. In response, we conducted comprehensive ablation studies, summarized below and detailed in Appendix B.2 (Figures B.1–B.2):
- **Velocity regularization coefficient $\lambda_\text{reg}$**: Omitting this term causes gradient instability and NaN loss, making it indispensable. In our ablation, the optimal coefficient value, $5 \times 10^{-3}$, achieves the best balance between dynamic PSNR and flow estimation performance.
- **Motion token count ($M$)**: We evaluated performance with varying numbers of motion tokens ($M=0, 4, 8, 16, 32, 64, 128$. Results show that STORM performs robustly across a range of motion token counts, with the best results achieved at $M=16$. We have updated all our quantitative results in our tables to these best hyperparameter settings ($\lambda_\text{reg}=5 \times 10^{-3}, M=16$) in the revision.
- **Number of Input timesteps:** Encouraged by **Reviewer `HXs2`**, we investigated STORM's robustness to varying numbers of input timesteps. Thanks to the sequence-to-sequence nature of our Transformer-based model, STORM allows flexibility in choosing the number of input timesteps during both training and testing. We conducted two ablation studies (detailed in Fig. B.2): 
    - (1) Testing STORM trained with 4 input timesteps on different number of test-time input timesteps (e.g. 1, 2, 6, 10) without re-training (zero-shot transfer). Results show that STORM generalizes well to unseen configurations, with peak performance achieved at 4 timesteps, as expected. 
    - (2) Retraining STORM with different number of input timesteps (including 1, 2, 4, 6, 10). Results indicate that increasing timesteps improves performance. Notably, when trained with *a single timestep*, STORM generalizes to a future prediction framework (by being able to predict Gaussian velocities given a single timestep), achieving a remarkable **2–4 PSNR** improvement on dynamic regions compared to per-scene optimization methods using *4 timesteps*.
These ablations highlight STORM’s flexibility and robustness under varying conditions, further validating its applicability to real-world scenarios.

### **3. Clarifications**
- **Technical Clarifications**: We revised the manuscript to clarify key technical details, including improved notations for motion group assignments, a brief illustration of how to obtain point trajectories and explanations of latent-STORM editing.
- **References and Related Work**: We have added references to all related works mentioned by reviewers.
- **Visualization Updates**: Our anonymous project page has been updated with additional 3D and tracking visualizations, providing further insights into STORM’s performance.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel pipeline for dynamic scene reconstruction, which addresses several challenges inherent in modeling dynamic scenes with high fidelity and accuracy. By leveraging a novel integration of 3D Gaussians into a unified representation, the method achieves efficient rendering and reconstruction, enabling supervision directly through the reconstruction loss.

Reviewer Wu7P initially expressed concerns about the clarity of certain implementation details and experimental settings. These issues were adequately clarified in the rebuttal, where the authors provided more detailed descriptions of their methodology. Reviewer JKrr noted the lack of a quantitative ablation study in the initial submission, which was a valid critique. In response, the authors included comprehensive ablation studies in their rebuttal, highlighting the contributions of each component in their pipeline. Both Reviewers N7nA and HXs2 emphasized the importance of evaluating the proposed method on additional datasets to validate its generalizability. The authors responded effectively by conducting experiments on diverse datasets, demonstrating consistent performance across varying scenarios. 

Based on the reviewers’ positive feedback, the authors’ effective rebuttal, and the clear contributions of the paper, I recommend its acceptance.