STORM: Spatio-TempOral Reconstruction Model For Large-Scale Outdoor Scenes
Authors: Jiawei Yang
Jiahui Huang
Boris Ivanovic
Yuxiao Chen
Yan Wang
Boyi Li
Yurong You
Apoorva Sharma
Maximilian Igl
Peter Karkus
Danfei Xu
Yue Wang
Marco Pavone
Venue: ICLR 2025 Poster (2025)
Keywords: autonomous driving; reconstruction model; spatiotemporal
Primary area: applications to robotics, autonomy, planning
TL;DR: We present STORM, a spatio-temporal reconstruction model designed to reconstruct space-time scenes and motions from sparse observations for outdoor scenes.

Abstract: We present STORM, a spatio-temporal reconstruction model designed for reconstructing dynamic outdoor scenes from sparse observations. Existing dynamic reconstruction methods often rely on per-scene optimization, dense observations across space and time, and strong motion supervision, resulting in lengthy optimization times, limited generalization to novel views or scenes, and degenerated quality caused by noisy pseudo-labels for dynamics. To address these challenges, STORM leverages a data-driven Transformer architecture that directly infers dynamic 3D scene representations—parameterized by 3D Gaussians and their velocities—in a single forward pass. Our key design is to aggregate 3D Gaussians from all frames using self-supervised scene flows, transforming them to the target timestep to enable complete (i.e., "amodal") reconstructions from arbitrary viewpoints at any moment in time. As an emergent property, STORM automatically captures dynamic instances and generates high-quality masks using only reconstruction losses. Extensive experiments on public datasets show that STORM achieves precise dynamic scene reconstruction, surpassing state-of-the-art per-scene optimization methods (+4.3 to 6.6 PSNR) and existing feed-forward approaches (+2.1 to 4.7 PSNR) in dynamic regions. STORM reconstructs large-scale outdoor scenes in 200ms, supports real-time rendering, and outperforms competitors in scene flow estimation, improving 3D EPE by 0.422m and Acc5 by 28.02%. Beyond reconstruction, we showcase four additional applications of our model, illustrating the potential of self-supervised learning for broader dynamic scene understanding. For more details, please visit our project at https://jiawei-yang.github.io/STORM/.

Published as a conference paper at ICLR 2025
STORM: SPATIO-TEMPORAN RECONSTRUCTION MODEL FOR LARGE-SCALE OUTDOOR SCENES
Jiawei Yang" Jiahui Huang?, 1 Yuxiao Chen', 9 Yan Wang", Boyi Li", Yurong You Maximilian Igl", Apoorva Sharma'.
Peter Karkus", Danfei X Boris Ivanovic Yue Wangl.-.1 Marco Pavone [yangjiaw, yue. @usc. edu, University of Southern California danfei@gatech, edu, Georgia Institute of Technology
pavone@sttafford.ed.. Stanford University
(jiahuih, yokabuih,,yxxiic,, yanwan,, boyil, yurongy, apoorvas, migl, piaaxuss.bivnnorieeevv.... com, NVIDIA Research Equal advising.
ABSTRACT
We present STORM, a spatio-temporal reconstruction model designed for recon- structing dynamic outdoor scenes from sparse observations. Existing dynamic reconstruction methods often rely on per-scene optimization, dense observations across space and time, and strong motion supervision, resulting in lengthy opti- mization times, limited generalization to novel views or scenes, and degenerated quality caused by noisy pseudo-labels for dynamics. To address these challenges. STORM leverages data-driven Transformer architecture that directly infers dy- namic 3D scene
paramettinsspppraaeeeeriieed by 3D Gaussians and their ve- locities-in single forward pass. Our key design is to aggregate 3D Gaussians from all frames using self-supervised scene flows, transforming them to the tar- get timestep enable complete (i.e., "amodal") reconstructions from arbitrary viewpoints at any moment in time. As an emergent property, STORM automat- ically captures dynamic instances and generates high-quality masks using only reconstruction losses. Extensive experiments on public datasets show that STORM achieves precise dynamic scene reconstruction, surpassing state-of-the-an per- scene optimization methods (+4.3 to 6.6 PSNR) and existing feed-forward ap- proaches (+2.1 to 4.7 PSNR) in dynamic regions. STORM reconstructs large-scale outdoor scenes in 200ms, supports real-time rendering, and outperforms competi- tors in scene flow estimation, improving 3D EPE by 0.422m and Ace; by 28.0 28.02%. Beyond reconstruction, illustrating the potential of self-supervised learning for broader dynamic scene understanding. For more details, please visit our project page.
showcase four additional applications of our model,
INTRODUCTION
Understanding and reconstructing dynamic 3D scenes from visual data is fundamental challenge in computer vision, with significant applications in autonomous driving, robotics, and mixed reality, among many others. While static scene reconstruction methods have evolved from per-scene opti- mization (Mildenhall et al., 2021; Kerbl et al., 2023) to more data-driven approaches that leverage generalizable priors for improved performance (Zhang et al., 2024; Tang et al., 2024; Xu al., 2024; Gao et al., 2024b; Wu et al., 2024b), most dynamic scene reconstruction methods still rely heavily on per-scene optimization, dense spatio-temporal observations (Park et al., 2021; Yang et al., 2024b), and strong motion supervision, such as dynamic objects' masks (Li et al., 2b2 Wang et al., 2024b), optical flow (Li et al., 2021b), or point trajectories (Wang et al., 2024b). Consequently, these models suffer from noise in the above pseudo-labels, require lengthy training times that range from hours to days, and cannot benefit from the data-driven advancements (e.g., scaling laws (Zhai et al., 2022)) that are nowadays exploited by generalizable static reconstruction methods.

---

Published as a conference paper at ICLR 2025
Our goal is to develop scalable and data-driven solution for dynamic scene reconstruction ad- dresses the limitations of existing methods. To this end, we present STORM, a self-supervised ap- proach for reconstructing dynamic 3D scene representations and scene motions directly from sparse, multi-timestep, posed camera images. STORM leverages a Transformer model (Vaswani et al., 2017; Dosovitskiy, 2020) to reconstruct dynamic scenes in single feed-forward pass, dramatically ducing reconstruction times from hours to seconds while harnessing data priors learned from large- scale datasets. Crucially, unlike existing methods that require pseudo-labels, STORM utilizes a self- supervised reconstruction loss, enabling significantly more cost-efficient data acquisition.
STORM enabled by our proposed bottom-up amodal aggregation and transformation frame- work. Specifically, for each frame, we predict pixel-aligned or patch-aligned 3D Gaussian Splats (3DGS) (Kerbl et al., 2023) along with their motions, capturing the instantaneous state of the scene at each timestep. Since these image-aligned 3DGS can only represent the observed region from the context frames, we transform the Gaussians predicted from all the input frames to the target timestep, aggregating them into an "amodal" representation of the dynamic scene (Huang et al., 2022). By minimizing the reconstruction loss defined over this aggregated representation, our ap- proach achieves accurate, self-supervised estimation of the temporal changes in the scene, inaccu- racies in dynamics would result in poor aggregation and transformation results, thereby introducing large reconstruction errors-driving our model toward better scene dynamics estimation.
Building upon this foundation, we introduce motion tokens-a a set of learnable tokens prepended to the Transformer's input sequence. These motion tokens interact with image tokens through self- attention operations and are decoded as motion bases at the end of Transformer. They are designed to capture common motion primitives over time while also regularizing the degrees of freedom in pre- dicted motions, motivated by the fact that the scene elements often move cohesively as groups (Wang al., 2024b; Lei al., 2024; Luiten et al., 2023). Concretely, we represent the motion of each 3D Gaussian using 3D velocity vectors, with the final motion computed as weighted combination of shared velocity bases. These weights are determined by the similarity between motion tokens and image tokens. Consequently, motion tokens not only encode scene dynamics but also enable unsupervised segmentation of dynamic instance or motion group segmentation.
Lastly, we introduce few practical techniques to make STORM more robust for in-the-wild captures. We address challenges such as sky modeling and camera exposure mismatches using auxiliary sky and affine tokens, and improve large novel view extrapolation and fine ine-grained human motions, such as leg and arm movements, using latent Gaussians and a latent decoder.
We conduct extensive experiments on the Waymo Open dataset (Sun al., 2020). NuScenes (Caesar et al., 2020) and Argoverse2 (Wilson et al.) to evaluate the performance of STORM. The results demonstrate that STORM accurately reconstructs dynamic scenes in real-time (0.2 seconds for a 2- second clip), significantly surpassing per-scene optimization methods and other generalizable feed- forward models in terms of photorealism, geometry and motion estimation quality. These findings highlight the potential of self-supervised learning for advancing dynamic scene reconstruction and understanding. Our contributions can be summarized as follows:
We propose STORM, the first feed-forward, self-supervised method for fast and accurate recon- struction of dynamic 3D scenes from sparse, multi-timestep. posed camera images.
We propose bottom-up framework that aggregates and transforms per- per-frame 3D Gaussian Splats into a cohesive scene representation, which enables sulf-supervised motion estimation. Further- more, introduce motion tokens that capture common motion primitives and regularize motion predictions, facilitating dynamic motion group segmentation without explicit motion or corre- spondence supervision.
We present several enhancements for in-the-wild scenarios, including sky modeling, camera ex- posure inconsistency handling, large novel-view extrapolation, and fine-grained human motions reconstruction, making STORM well-suited for real-world applications.
2 RELATED WORK
Dynamic scene reconstruction. Derived from neural radiance fields (NeRFs) (Mildenhall et al., 2021), previous beRF-based approaches model scene dynamics either by applying deformations to a canonical volume (Pumarola et al., 2021; Tretschk et al., 2021; Cao & Johnson, 2023; Park et al., 2021; Wu et al., 2022; Fridovich-Keil et al., 2023), or by chaining point-level scene flow mo-
2

---

Published as a conference paper at ICLR 2025
tions (Xian et al., 2021; Gao et al., 2021; Li et al., 2021b; 2023; Liu et al., 2023). These per-scene optimization methods typically require dense temporal and spatial observations or explicit motion supervision, such optical flow (Li et al., 20210; 2023; Yang et al., 2023b; Gao et al., 2024a; Karaev et al., 2023; Fischer et al., 2024a) or dynamic masks (Liu et al., 2023; Li et al., 2023), to overcome the ill-posed nature of reconstructing dynamic scenes from sparse views. More recently, methods based on 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) have adopted similar strategies, apply- ing deformations in canonical space (Wu al., 2024a; Yang et al., 2024b) or rigid transformations to particles (Luiten et al., 2023; Wang et al., 2024b). However, they also depend on dense views or motion supervision, and remain limited to per-scene optimization, lacking the ability leverage data priors, except for e.g. Ren et al. (2024), that only focuses on object-level reconstructions. We instead propose a feed-forward model trained on large-scale datasets, enabling outdoor dynamic scene reconstruction without per-scene optimization or explicit motion supervision.
Feed-forward reconstruction. Feed-forward approaches for 3D reconstruction and rendering aim to generalize across scenes by learning from large datasets. Early works on generalizable NeRFs focus on object-level (Chibane et al., 2021; Johari et al., 2022; Reizenstein et al., 2021; Yu et al., 2021) and scene-level reconstruction (Suhail et al., 2022; Wang et al., 2021; Du et al., 2023; Wang et al., 2024a). These methods typically rely on epipolar sampling or cost volumes to fuse multi-view features, requiring extensive point sampling for rendering, which results in slow speed and often un- satisfactory details. More recently, feed-forward models based on 3DGS have been proposed (Szy- manowiez al., 2024b; Charatan al., 2024; Wewer et al., 2024; Zhang et al., 2024; Szymanowiez et al., 2024a; Tang et al., 2024; Xu et al., 2024). These models leverage large-scale object-centric synthetic datasets (Chang et al., 2015; Deitke et al., 2023; 2024) or indoor datasets (Zhou et al., 2018) for improved speed, view synthesis performance, and generalization. However, these meth- ods are primarily designed for static scenes and struggle with dynamics. Unlike them, our approach is a 3DGS- feed-forward model operated on large-scale outdoor dynamic scenes; more impor- tantly, it also recovers scene motions without explicit motion supervision.
Reconstruction for outdoor urban scenes. Building photorealistic reconstructions of dynamic ur- ban scenes from on-car logs is crucial for autonomous driving, as enables closed-loop training and testing. Recent work has shifted focus from reconstructing static scenes (Guo et al., 2023) to dynamic ones. Most existing methods for dynamic urban scene reconstruction rely on box anno- tations ensure controllability; however, these require expensive ground truth labels (Wu et al., 2023; Chen et al., 2024; Yang et al., 2023a; Tonderski et al., 2024; Fischer et al., 2024b; Zhou et al., 2024a; Ost et al., 2021; Williams al., 2024; Fischer et al., 2024a) and often degrade in perfor- mance when using noisy pseudo-labels (Yan et al., 2024; Zhou et al., 2024b). Methods that do not rely on box annotations typically lack controllability over individual objects (Yang et al., 2024a; Chen et al., 2023; Huang et al., 2024), Furthermore, these approaches are crescene-bbsed do not leverage data priors, and require lengthy training times, ranging from hours (Yan et al., 2024; Yang et al., 2024a) to days (Xie et al., 2023). In contrast, our method is fast, scalable feed-forward model that reconstructs dynamic urban scenes purely through self-supervisson in seconds. By differentiat- ing between different instance groups emerged from our motion tokens, our approach enables better decomposition and controllability.
3 SELF-SUPERVISED
RECONSTRUCTION MODEL
Problem formulation. Our goal is to recover spatiotemporal scene representations from a set of posed images. Specifically, given a set of images I E H x3, with height H and width W. captured at multiple timesteps & and optionally from multiple viewpoints v, along with their corre- sponding camera intrinsic and extrinsic parameters, we aim to reconstruct the underlying appear- ance, geometry, and dynamics of the scene over the observed duration. The core challenge arises from the transient and incomplete nature of the data: each point in the 4D space-time time volume is typi- cally observed only once, making it difficult to infer comprehensive spatiotemporal representation.
3.1 STORM
To address the aforementioned challenges, we propose STORM, as illustrated in Fig. 1. We adopt a Lagrangian representation by modeling scene elements as a set of 3D Gaussians (3DGS) (Kerbl et al., 2023) that translate over time. Specifically, begin by predicting 3DGS for each frame,
3

---

Published as a conference paper at ICLR 2025
Volachy DNSKS
Hodon salens
Tranffrmer
Sceae Paws
Desedher
Blsis AsEyenneat
mar
00000
Contect Toner
Dugget Tume:
Perniteaay SD Geussion Splres (1903)
Agyregsied 1005
Sapmitioo
Figure 1: STORM Overview. From sparsely observed context frames, STORM reconstructs per-frame 3D Gaussian splats (3DGS) and predicts their scene flows using prepended learnable motion tokens and a dynamic mask decoder. The mask decoder computes weights for combining motion bases, derived from the motion tokens, to obtain scene flows. These predicted scene flows enable the ag- gregation and transformation of 3DGS over time, while the predicted weights support unsupervised motion group segmentation. The learning process is guided purely by reconstruction losses.
capturing the instantaneous state of the scene at each timestep. To model dynamics, we task the model with predicting the velocity of each Gaussian. Using these velocities, we transform the 3DGS from their observed context timesteps into any target timestep. This process aggregates the per-frame predictions into cohesive amodal representation that remains consistent over time. Notably, our method trains solely with reconstruction losses, avoiding reliance on external motion supervision, significantly reducing data requirements. Below, we introduce our method in detail.
Network and input. STORM builds upon a standard Transformer model (Vaswani et al., 2017; 7; Doso- vitskiy, 2020), similar Zhang al. (2024). Following standard Vision Transformers (Dosovitskiy, 2020), we divide images into 2D non-overlapping patches. To incorporate 3D spatial information, we extend this patching process to the Plücker ray map (Plucker, 1865), which encodes the ray origins and directions corresponding to each pixel. These ray origins and directions are computed based on the camera's s intrinsic and extrinsic parameters. The concatenated patches are then embed- ded through linear patch embedding layer to obtain image tokens. Lastly, temporal information is infused via a time embedding layer (Peebles & Xie, 2023), which maps frequency-eneoded time vector into time embedding. The resulting input to the Transformer is ID sequence of image tokens, formed by summing the image embeddings, ray embeddings, and time embeddings.
The main output of our model is set of pixel-aligned 3D Gaussians (Kerbl et al., 2023), each defined as g = =(u, R, s, o, c). where 4 E R3 and R E SD(3) represent the center and orientation, Rd indicates the scale, E g. denotes the opacity, and E I corresponds to the color. The orientation is parameterized using a 4D quaternion. The centers of the 3D Gaussians are computed from ray origins and directions, which are pre-computed from camera parameters, along with the ray distance, by μ ray. + d- ray dir. where d is the 1-channel ray distance predicted by the model. By default, our model predicts (G:- E R R[( from the ViT feature map (Fy E g(H/ (ttpx///b// w//))ee using a linear layer, where is the patch size of the Transformer, and is the channel dimension. These 3D Gaussians exist independently in 3D space for each timestep. Next, we describe how their dynamics are obtained and how they are aggregated to form the final amodal, synchronized scene representations. For clarity, the view index v is omitted unless necessary.
Output.
Scene dynamics. To capture the dynamics of a 3D scene, we model the motion of each 3D Gaussian using two velocity vectors, v [V, v IR (we will detail how compute these later), which represent the backward and forward velocities of a Gaussian at timestep f. Empirically, we find that assuming the Gaussians move with constant velocity within the duration of the clip (typically around 2 seconds) achieves good balance between model complexity and representational power. Accordingly, the translation of a Gaussian at time t' is specified as follows:
Hc (t' - t)v, t' <t
Pe-s
Pc + (t' (t'-tje t)vt t''
Amodal aggregation. To create a unified representation of the scene that is consistent over time, aggregate the per-frame 3D Gaussians into amodal, synchronized representation. Specifically, the Gaussians Ge at an arbitrary target timestep t' are defined as the union:
Ge = U G
(2)
4

---

Published as a conference paper at ICLR 2025
With ortrre
Viww vistorte
Veeee dicctionnn
(8)
token
Sky token helps reconstruct unbounded sky
Figure 2:
Effect of afline and sky tokens. (a) The affine token handles exposure mismatches between cameras, eliminating artifacts like the black foggy floaters caused by exposure differences (orange arrows). (b) The sky token enables our method to predict sky colors for every pixel during rendering, even when they are not observed in any context frames.
where G._ contains translated Gaussians with centers Pe-e' derived from the prediction G This amodal representation combines observations from multiple timesteps, capturing the complete geometry and appearance of the scene along with its dynamics. It supports tasks such as rendering the scene from novel viewpoints and moments in time.
Motion tokens and mask decoder. Motivated by the observation that scene dynamics often exhibit low-dimensional structures composed of shared motion patterns (Wang et al., 2024b; Kratimenos et al., 2023; Lei al., 2024), we introduce M learnable motion tokens (indexed by m)) where M << N and N is the number of image tokens. These motion tokens are prepended to the input sequence of the Transformer and interact with other input tokens via self-attention. STORM leverages these to- kens to capture common motion primitives present in the scene over time. At the output of the Trans- former, the motion tokens are decoded into velocity bases vb = (vb vb*)) E RMxx and motion queries q Mxx via a set of Multi-Layer Perceptrons (MIPs). Here, e' denotes the dimen- sion of the motion embedding space. Simultaneously, the image embeddings F E RR/H//xxW//// are mapped into this space produce pixel-aligned motion keys k ( through several deconvolution layers, where each key vector k.,j E RRe corresponds to a spatial location (i,j).
Inspired by SAM (Kirillov et al., 2023), we compute the similarity between motion queries q and motion keys k to derive weights w E (M+("X)))XX M for combining the velocity bases. Specifically, the weights wid) E TR+.M at each spatial location (i,j) are computed as:
4m -m,1
exp
w (i.))
Em= exp
9 -ki
where is a temperature hyperparameter that controls the sharpness of the distribution (we set T = 0.5 in all experiments). The weights w are then used to combine the velocity bases for each Gaussian associated with the pixel at location (i,j), yielding the final velocity v used in Eq. (1):
M
M
(.
(665)) m
vbm:
where Σ
w
(ناا)) m
= and 0 S (لbا < 1.
(4)
m=l
This design captures the now-dimensional structure of scene dynamics and regularizes the motion prediction problem by reducing its degrees of freedom.
3.2 STORM IN THE WILD
Modeling unbounded scenes from multi- videos captured in the wild introduces additional chal- lenges, such as representing the sky, handling exposure differences between cameras, and accurately modeling humans.
Auxiliary tokens for sky and exposure mismatches. In wild video collections, particularly those captured by autonomous vehicles, sky modeling and exposure mismatches are common chal- lenges. Specifically, the sky often lacks well-defined depth, and the same 3D point may appear with
FFlllowing the mask decoder design of SAM (Kirillov et al., 2023), we use distinct MI.P weights for each motion token, this leads to cleaner motion masks.
5

---

Published as a conference paper at ICLR 2025
Time
Figure 3: Latent-STORM Examples. Using latent Gaussians and decoder, Latent-STORM can photorealistically reconstruct human leg movements. Note how the leg angles change over time in Latent-STORM, while they remain static in regular STORM (blue arrows). Please refer to "Human Modeling with Latent- Attent-STORM" section in our project page for video comparisons.
varying colors across different images due to exposure differences between cameras. To address these issues, introduce two types of learnable auxiliary tokens into the input sequence: the sky token and the affine token, designed similarly to the motion tokens.
A single sky token is used to capture sky information. At the output of the Transformer, this sky token conditions a modulated MLP that takes the ray direction as input and outputs the sky color:
(5) where 7-) is a riqueency-based positional embedding function as in Mildenhall et al. (2021). This setup allows query sky colors for every pixel wish render. Given a rendered image Las before sky composition and rendered opacity map Ô, the final image with the sky composed is:
Csky MLP sky (7 (d) sky token).
(6) To handle exposure mismatches between cameras, we introduce v learnable affine tokens, where the number of cameras. These tokens aim to capture exposure variations between cameras. At the Transformer's 's output, each affine token is mapped to a scaling matrix S E R3x3 and bias vector R33 via a linear layer. The final rendered image obtained by applying the affine transformation to every pixel: i SI' + b. The affine transformation has been similarly explored in previous work (Rematas et al., 2022) but only in per-scene optimization setting. Examples illustrating the roles of the sky and affine tokens are shown in Fig. 2.
I' IGs (1 Ô) Csky.
Latent-STORM. As optional enhancement, with a latent decoder to improve STORM's performance on large novel view extrapolation and hu- man body modeling. Instead of predicting pixel-aligned Gaussians with 3-channel color vector, predict patch-aligned Gaussians with e-channel latent vector. This approach reduces the number of Gaussians while increasing the modeling capacity of each Gaussian. Consequently, the model out- put changes from G R ( to G E //////px////)xX////// Transformer. After rasterization, we obtain a px downsampled latent feature map F, which is com- posited with learnable impainting token using the opacity map: = F - - -iipainniing token. tokenn. This composited feature map is then upsampled the original resolution using convolutional de- coder (Rombach et al., 2022) to produce the final color and depth outputs.
introduce the use of latent Gaussians coupled
where p is the patch size of the
This design addresses the limitations of color-based Gaussians handling occluded regions that are not visible in any of the context views, since the decoder can infer and reconstruct these unseen areas from the inpatinting tokens within a reasonable extrapolation range. Additionally, capturing fine-grained human motions, such as leg and arm movements, remains challenging with sparse ob- servations. However, we find that the decoder can photorealistically recover these subtle motions, albeit with a slight compromise in pixel sharpness due to the additional decoding process. Fig. 3 compares this property. We refer to this new variant of our method as Latent-STORA
3.3 IMPLEMENTATION
Model architecture. By default, we use 12-layer Vision Transformer (ViT-B) (Dosovitskiy, 2020)
with full attention and a patch size 8, along with M 16 motion tokens. We study how perfor-
S Significant hallucination would require stronger generative capabilities, which we leave as future work.

---

Published as a conference paper at ICLR 2025
mance scales with respect to the number of motion tokens in Appendix B.2. For the mask decoder and MLP components, we adopt the implementation from SAM (Kirillov et al., 2023), and set the final projected motion embedding space to be 32 dimensional. For the modulated MLP used in sky modeling, we follow DiT' (Peebles & Xie, 2023) to use an adaptive Laye NNorrorm layer for modula- tion. Our GS backend is based on gsplat (Ye al., 2024).
Supervision and loss functions. After aggregating the amodal scene representation from all ob- served timesteps, we transform it into the target timesteps we wish to render. During training, we randomly select starting timestep / and sample 4 target timesteps l' within the range [t, L + 28] for supervision. Using the transformed amodal Gaussians Gv, we minimize combination of recon- struction loss, sky loss, and velocity regularization loss:
L= Lreon
Xuay"
Laky + >veg Gregs
where the reconstruction loss Crecon includes RGB loss, perceptual loss, and depth loss. The sky loss Laky encourages zero opacity for Gaussians located in the sky-region. The velocity regularization loss defined as Lreg |l where we encourage the predicted velocity vectors be small. Aay = 0.1 and Xreg = 0.005 are two hyperparameters that balance different losses. Details regarding training. implementation, and the loss functions are provided in Appendix A.
4 EXPERIMENTS
Datasets. We primarily conduct experiments on the Waymo Open Dataset (Sun et al., 2020), which contains 1,000 sequences of driving logs: 798 sequences for training and 202 for validation. Each sequence consists of a 20-second video recorded at 1OFPS. For training and testing, we use the frontal three cameras at 8x downsampled resolution (160 x 240). The input our model consists of 4 context timesteps, evenly spaced at ++ Os, 0.5.s,, t+ 1.0s, and + 1.5s, where / is randomly chosen starting timestep. Additionally, we evaluate our method on the NuScenes (Caesar et al., 2020) and Argoverse2 (Wilson et al., 2023) datasets. Please refer to Appendix B. for more details.
4.1 RENDERING
Setup. We assess novel view synthesis from sparse view reconstructions using the validation set of the Waymo Open Dataset (Sun al., 2020). Each video sequence is segmented into 10 non- overlapping clips, each 2.0 seconds long and consisting of 20 frames (3 camera views per frame). For reconstruction, we provide the 1st, 5th, 10th, and 15th frames as context frames, and evaluate on the remaining frames. This setup enables evaluation of both interpolation (Os to 1.5s) and extrapolation (1.5s to 2.0s), resulting in 2,019 video clips, or 96,912 total images. We report standard metrics: PSNR, SSIM, and Depth RMSE. Additionally, we analyze performance on both full images and dynamic regions for a more comprehensive evaluation. Lastly, we report the inference time; for per- scene optimization methods, this refers to the test-time fitting time, while for generalizable methods, it refers to the time required for the model to feedforward and output 3D Gaussians.
Baselines. We compare our method against two categories of approaches: per-scene optimization methods and feed-forward models. For per-scene optimization, we evaluate against a NeRF-based approach, EmerNeRF (Yang et al., 2024a), and 3DGS-based approaches, including 3DGS (Kerbl et al., 2023), PVG (Chen et al., 2023), and DeformableGS (Yang et al., 2024b). Since LiDAR data not provided at test time in our setup, train these baselines without LiDAR supervision to ensure fair comparison. In the second category. we compare against recent large reconstruction models, including LGM (Tang et al., 2024) and GS-LRM (Zhang et al., 2024). We notice that the default IGM,, which predicts raw 3DGS coordinates, performs poorly in our datasets. Therefore, we modify it to predict depth and recover positions similar to our approach, and denote it as L.MM". We provide more implementation details of these basclines in Appendix A.3.
We present the quantitative results in Table Compared to per-scene optimization meth- ods, STORM achieves significantly better performance in both dynamic regions and full images in terms of photorealism, geometry, and inference speed. Specifically. in dynamic regions, STORM out- performs the best per-seene method by substantial 5dB in PSNR and 0.346 SSIM. For full images, STORM attains around 0.5 to LdB PSNR gain. Notably, STORM achieves these improvements while reducing inference time from tens of minutes to just 0.18 second, making it suitable for real-time
Results.
7

---

Published as a conference paper at ICLR 2025
Table I: Comparison to state-of-the-art methods on the Waymo Open Dataset. We compare photorealism, geometry and speed metrics against both per-scene optimization methods and gener- alizable feed-forward methods. PSNR, SSIM, and Depth RMSE (D-RMSE) are reported. Speed metrics are estimated on single A100 GPU. reproduced by us. 7: Non-sky region.
Dynamic-only SSIM1
Full imaget
Inferenee speed Real-time rendering (>200FPS)
Methods
PSNR
D-RMSE,
PSNR
SSIM
D-RMSEL
Timel
Per-Soetue Opcinization northods
EmerNeRF (Yang 2024a)
17.79
0.255
40.88
0.738
33,99
14min
3DGS (Kesbl et al., 2023)
17.13
0.267
13.88
25.13
0.741
19.68
23min
PVG (Chen 2023)
15.51
0.128
15.91
22.38
0.661
13.01
27min
DeformableGS (Yang et al., 2024b)
17.10
0.266
12.14
0.761
29min
Generalizable feed-fforword methouts
LGM (Tang sl., 2024)
17.36
0.216
11.09
18.53
0.447
9.07
0.06s
LGM (Tang 2024)
0.443
9.43
23.59
0.06s.
GS-LRMM (Zhang et al,, 2024)
20.02
0.520
9.95
25.18
0.753
7.94
0.02s
LaRent-STORM STORN
21.26
0.535
9.42
25.03
0.750
8.57
0.18s
22.10
0.624
7,50
26.38
0.794
5.48
0.18s
Table 2: Comparison to state-of-the- art methods on more datasets. We report full-image PSNR and Depth RMSE metrics the NuScenes (Cae- sar et al., 2020) and Argoverse2 (Wil-
Table 3: Comparison of scene flow estimation on the Waymo Open Dataset. All competing methods require LiDAR input at test time, whereas our method relies solely on camera images.
son et al.) datasets.
Methods
EPE3D (m) & Acc:(%) 1 Accia 18 (rad) J Inference Time
Method
NuSeenes
Argovere22
eLi 2021a)
NSFP
0.698
42.17
54,26
0.919
~~77//ramm
PSNR
D-RMSEL
PSNR1
D-RMSFL
(Najibi 2022)
NSFP++
0.711
53.10
63.02
0.989
~167s/famme
LGM
23.21
7.34
22.93
14.20
7.71
24.49
Ours
0.276
81.12
85.61
0.658
~0.025e/ramee
Outs
24.90
5.43
24.80
13.51
applications. This confirms our motivation of building data-driven models that excels with data pri- ors, addressing the limitations of per-scene optimization methods. Compared to other generalizable feed-forward models, STORM demonstrates a robust ability to model scene dynamics and process multi-timestep, multi-view images holistically. This enables us to model dynamic scenes better.
Results on additional datasets. We evaluate the applicability of STORM on the NuScenes (Caesar et al., 2020) and Argoverse2 (Wilson et al.) datasets, comparing against other generalizable methods in Table 2. We provide detailed setups Appendix B.1. Our method achieves the best performance in both full-image PSNR and Depth RMSE metrics. Measuring performance on dynamic regions is expected to have more gains. These results validate the generalizability of STORM across datasets.
4.2 FLOW ESTIMATION
Setup and baselines. A unique capability of STORM is scene motion estimation, which we demon- strate using the Waymo Open Dataset (Sun et al., 2020). This dataset provides ground truth 3D scene flows, which we do not use for supervision. We measure 3D scene flow estimation accuracy using standard metrics following Li et al. (2021a): End-Point Error in 3D (EPE3D), Accs, Acc1o- angular error Oerr. and inference time. For baselines, we compare STORM against NSFP (Li al., 2021a), and NSFP++ (Najibi et al., 2022). Since existing methods cannot directly synthesize scene flows novel timesteps, we evaluate the scene flows estimated on the context frames. Specifically, we provide the 1st, 5th, 10th, and 15th sensor observations as input and evaluate on these frames rather than on the remaining ones. Notably, all these competing methods require LiDAR input at test time, whereas STORM relies solely on camera images, making our comparisons highly conservative.
As presented in Table 3, STORM consistently outperforms all methods across all metrics, achieving substantial improvements in EPE3D, Aces and Acc1o despite using only camera images as input. While NSFP (Li et al., 2021a) and NSFP++ (Najibi et al., 2022) excel at scene flow esti- mation with dense space-time observations (10Hz), they struggle with sparse observations (2Hz). In contrast, STORM demonstrates robust performance even with sparse data. To the best of our knowl- edge, STORM is the first scene flow estimation method that does not require depth signals at test time. These results demonstrate the effectiveness of our approach in explicit motion understanding and its potential for scene flow estimation without reliance on additional sensors at test time.
Results.
8

---

Published as a conference paper at ICLR 2025
Jot Sompleel DEN)
Somble
Figure 4: Iterative reconstruction of static scenes. STORM reconstructs 20-second- videos within second in an iterative manner, which can serve as initialization for per-scene optimization methods for further refinement.
Figure 5: Iterative reconstruction of dynamic scenes. Top: STORM reconstructs 20-second-long
videos within 1 second in an iterative manner. Bottom: Furthermore, by chaining scene flows, obtain point trajectories for dynamic Gaussians.
4.3 ABLATION STUDY, QUALITATIVE RESULTS AND APPLICATION
Ablation study. We present detailed ablation study analyze the impact of the velocity regular- ization coefficient Åreg the number of motion tokens M, and the number of input timesteps during training and testing in Appendix B.2. In short, without velocity regularization, training collapses be- cause of gradient explosion, and an optimal A (5e-3) yields the best performance. STORM is robust the choice of M and performs best with M 16 for both rendering and flow estimation tasks. Furthermore, when trained on a fixed number of timesteps (e.g., 4), STORM demonstrates strong zero-shot generalization to varying input timesteps (e.g., 1, 2, 6, 10) at test-time, though re-training for specific configurations achieves optimal results.
Larger scene reconstruction. Figs. 4 and 5 show the results of applying STORM iteratively to 20-second posed videos, each includes 600 images captured across cameras over 200 timesteps. We process videos clip by clip, completing the inference for an entire video within 0.5 seconds single A100 GPU using batch inference and a bf16 6 precision. By merging the Gaussians predicted from each clip, we construct a comprehensive dynamic 3D scene reconstruction in a fully feedforward manner. Although some artifacts are present in overlapping regions when merging clips, these results demonstrate STORM's potential for holistic dynamic scene reconstruction, even for long and complex sequences. Furthermore, the predicted 3D Gaussians can serve as an initialization for per-scene optimization methods for further refinement, which we leave for future work.
the primary focus of our work, STORM models the motion of Gaussians over time, enabling us to derive point trajectories by chaining scene flows. In the bottom row of Fig. 5, we present examples point tracking, demonstrating STORM's potential for applications such motion analysis and object tracking. We hope this approach could inspire further exploration in related tasks, such as 2D pixel or point tracking.
Point tracking. While per-point trajectory estimation
Scene flow estimation and motion segmentation. We visualize the predicted 3D velocities and motion token assignments, i.e., the motion segmentation mask decoded by the mask decoder, in Fig, 6. These masks are derived by applying argmax operation per-pixel assignment weights w along the motion token dimension M (see Eq. (4)). These visualizations demonstrate that STORM captures scene dynamics and groups 3D Gaussians that correspond to the same moving pattern, resulting in motion-leve segmentations. These unsupervised segmentations allow us to select Gaus- sians based their assignments for editing, without using ground truth 3D bounding boxes.

---

Published as a conference paper at ICLR 2025
Bendowebrrgg Brogee
Berdwed mogg!!
Tendenec
Ver
Figure 6: Self-supervieed scene flow estimation and motion segmentation. For each sample, we show the rendered camera images (top), scene flows (middle), and motion assignments (bottom).
(a) Sample-1, original
[e] Sample-3, original
(b) Sample- I. remove dynamic objects
(f) Sample 3. replace cars. add pedestrians
[c] Sample-2 original
(a) Sample-3, add cars
(d) Sample-2 2. clone and affset latent Goussians
Ini S add pedestrions
Figure 7: STORM editing examples. We present examples of removing or cloning vehicles (a-d), as well as adding or replacing pedestrians and vehicles (e-h). Notice how STORM harmonizes the edited images due to the use of the decoder. More examples can be found at our project page.
Editing, control, and inpainting. We demonstrate the capabilities of Latent-STORM in editing scenes. Since each latent Gaussian still represents a physical particle in space, can edit the scene by adding, removing, or modifying these Gaussians before feature map rasterization and decoding. As shown in Fig. Latent-STORM occluded regions, and provide other advantages, such as image harmonization (f, h). For instance, Aatent-STORM can recover leg movements and hand gestures. Furthermore, one common challenge in driving scene reconstruction is the difficulty of inpainting occluded regions or removing static objects without leaving black holes. Latent-STORM addresses this by synthesizing these areas, albeit with slight blurriness. Please refer to our project page for more visualizations if interested. Overall, these results demonstrate the flexibility of STORM as a tool for scene editing and simulation.
reconstruct human movements h), hallucinate
CONCLUSION
In this work, we have introduced STORM, scalable spatio-temporal model designed for dy- namic scene reconstruction from sparse observations without requiring explicit motion supervision. Through extensive experiments, we have demonstrated STORM's ability to reconstruct dynamic out- door scenes and estimate scene dynamics. Our method significantly surpasses existing per-scene optimization and feed-forward approaches, showcasing its versatility for a wide range of applica- tions, including view synthesis, scene editing and point tracking. Looking ahead, we hope STORM will become foundational model for various tasks across multiple domains, enabling more efficient and flexible approaches to 4D scene reconstruction, motion estimation, and beyond. As research in spatio-temporal modeling progresses, we believe STORM has the potential to unlock new possi- bilities for real-time dynamic scene analysis, interactive applications, and further advancements in self-supervised learning.
Our default STORM performs well for vehicle editing but exhibits slightly reduced performance for human modeling.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11221-11631, 2020.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 130-141, 2023.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012. 2015.
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaus- sian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 19457-19467, 2024.
Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, and Li Zhang. Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering. arXiv preprint arXiv:331.118561, 2023.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions, arXiv preprint arXiv:2205.088: 2022.
Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojeic, Sanja Fidler, Marco Pavone, et al. Omnire: Omni urban scene reconstruc- tion. arXiv preprint arXiv:2408. 2024.
Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In Proceedings of the TEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7911-7920, 2021.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information P Processing Systems, 36, 2024.
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views stereo pairs. In Proceedings of the IEEE/CVF |EEE//VFCConference Computer Vision and Pattern Recognition, pp. 4970-4980, 2023.
from
Tobias Fischer, Jonas Kulhanek, Samuel Rota Bulò, Lorenzo Porzi, Mare Pollefeys, and Peter Kontschieder. Dynamic 3d gaussian fields for urban areas. arXiv preprint 200::2206.33155 2024a.
Tobias Fischer, Lorenzo Porzi, Samuel Rota Bulo, Mare Pollefeys, and Peter Kontschieder. Multi- level neural scene graphs for dynamic urban environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21 21125-21135, 2024b.
Sara Fridovich-Keil Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12479-12488, 2023.
Chen Gao, Ayush Saraf. Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5712-5211, 2021.
11

---

Published as a conference paper at ICLR 2025
Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, and Ulrich Neumann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint arXhv:2403.12360 2024a.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405. 10314, 2024b.
Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang, Chenjing Ding, Dongliang Wang. and Yikang Li. Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint arXiv:2306. 300.044888 2023.
Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengiu An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. 8 gaussiam: Self-supe street gaussians for au- tonomous driving. arXiv preprint arXiv:2405.20322 2024.
Shengyu Huang, Zan Gojeic, Jiahui Huang. Andreas Wieser, and Konrad Schindler. Dynamic 3d scene analysis by point cloud accumulation. In European Conference on Computer Vision, pp- 674-690. Springer, 2022.
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18365-18375, 2022.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: better to track together. arXiv preprint arXiv:2307.07635, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139 1, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Agelos Kratimenos, Jiahui Lei, and Kostas Danillidis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting. arXiv preprint arXiv:2312.001 2023.
Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis. Mosca: Dynamic gaussian fusion from casual vidcos via 4d motion scaffolds. arXiv preprint arXiv:2405.17421, 2024.
Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey, Neural scene flow prior. Advances in Neural Information Processing Systems, 34:7838-7851, 2021a.
Zhengqi Li. Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6498-6508, 2021b.
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neu- ral dynamic image-based rendering. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4273-4284, 2023.
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13-23, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:tt////.. http:///ppeeevvviiee net. forum? i d= CRY
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint 2xXiv:2308.09/33 2023.
12

---

Published as a conference paper at ICLR 2025
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 2021.
Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R Qi. Xinchen Yan, Scott Ettinger, and Dragomir Anguelov. Motion inspired unsupervised perception and prediction autonomous driving. In European Conference on Computer Vision, pp. 424 443. Springer, 2022.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2856-2865, 2021.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Bartin-Brualla, and Steven M Seitz. Hypemerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1195-4205, 2023.
Julius Plucker. Xvii. new geometry of space. Philosophical Transactions of the Royal Society of London, (155):725-791, 1865.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer, D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 10318-10327, 2021.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Shordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 10901-10911, 2021.
Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12932-12942, 2022.
Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, and Huan Ling. L4gm: Large 4d gaussian reconstruction model, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference on Computer Vision, pp. 156-174. Springer, 2022.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pp. 2446-2454, 2020.
Stanislaw Szymanowicz. Eldar Insafutdinov, Chuanxia Zheng, Dylan Campbell, João Henriques, Christian Rupprecht, and Andrea Vedaldi. Flash3d: Feed-forward generalisable 3d scene recon- struction from single image. arXiv preprint arXiv:2406.043.44 044440000433, 2024a.
Stanislaw Szymanowicz. Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Patern Recognition, pp. 0228-102177 2024h.
13

---

Published as a conference paper at ICLR 2025
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large muewi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054, 2024.
Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson. Neurad: Neural rendering for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14895-14904, 2024.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Con- 2021.
ference on Computer Vision, pp.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-600 6008, 2017. URL httpps:/pprcceedigg..otot nerripg wec/pappr//11 2017/has////h/////// 3f5ee2435 (ifee243547deeB22 deeel 355 Abbstssaaaabbbaacttcaac html.
Letian Wang, Seung Wook Kim, Jiawei Yang. Cunjun Yu, Boris Ivanovic, Steven L. Waslander, Yue Wang, Sanja Fidler, Marco Pavone, and Peter Karkus. Distillnerf: Perceiving 3d scenes from single-glance images by distilling neural fields and foundation model features. arXiv preprint arXiv:2406. 12095, 2024a.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi- view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1990-4699,. 2021.
Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. 2024b.
Christopher Wewer, Kevin Raj, Eddy Ilg, Bernt Schiele, and Jan Eric Lenssen. latentsplat: Autoencoding variational gaussians for fast generalizable 3d reconstruction. arXiv preprint 1622222403.16292, 2024.
Francis Williams, Jiahui Huang, Jonathan Swartz, Gergely Klar, Vijay Thakkar, Matthew Cong, Xu- anchi Ren, Ruilong Li, Clement Fuji-Tsang, Sanja Fidler, et al. fvdb: A deep-learning framework for sparse, large scale, and high performance spatial intelligence. ACM Transactions Graphics (TOG), 43(4):1-155 2024.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandel- wal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khan- delwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Ar- goverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.0044 00493, 2023.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 203 20310-0,,,, 2024a.
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21551-21561, 2024b.
14

---

Published as a conference paper at ICLR 2025
Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D 2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. Advances in Neural Information Processing Systems, 35:32653- 2022.
Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, et al. Mars: An instance-aware modular and realistic simulator for autonomous driving. In CAAI International Conference on Artificial Intelligence, pp. 15. Springer, 2023.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free- vrewpviewpoint video. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pp. 9421-9431, 2021.
Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Zhang. S-nerf: Neural radiance fields for street views. arXiv preprint (:Xi::3303000749 2023.
Yinghao Xu, Zifan Shi, Wang Yifan, Sida Peng, Ceyuan Yang, Yujun Shen, and Wetzstein Gordon. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arxiv: 2402, 2024.
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians for modeling dynamic urban scenes. arXiv preprint urXiv:2401.01333 2024.
Jiawei Yang. Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, and Yue Wang. Emernerf: Emergent spatial-temporal scene decomposition via self-supervisiin. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net 2024a. URL net htte//ooonddyyo22meeee.
https://cpenreet
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed- loop sensor simulator. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 1389-1399, 2023a.
Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting, arXiv preprint arXiv:2310.1066 2023b.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 20331-20344 2024b.
Vickie Ye, Ruilong Li, Justin Kerr, Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Matthew Tancik, al. gsplat: An open-source library for gaussian splat- ting. arXiv preprint arXiv:2409.06765. 2024.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4578-4587, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12104-12113, 2022.
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404, 19702. 2024.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
15

---

Published as a conference paper at ICLR 2025
Hongyu Zhou, Jiahao Shao, Lu Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugs: Holistic urban 3d scene understanding via gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 21336-21345, 2024a.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely, Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 1rXiv:1805.09%5 2018.
Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan
Hsuan Yang. Driv- inggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 11634-21644 2024b.
16

---

Published as a conference paper at ICLR 2025
A
IMPLEMENTATION DETAILS
In this section, we discuss the implementation details of STORM.
A.1 STORM IMPLEMENTATION DETAILS
Gaussian Parameterizatioon Our Transformer architecture and output mapping largely follow Zhang et al. (2024). The input images are normalized to the range of [-1, 11. Recall that cach Gaussian is defined as g (μ, R, 5, c), where μ E E3 and R E S0(3) represent the center and orientation, s E R indicates the scale, o E R denotes the opacity, and с E I corresponds to the color. Below, describe the activations normalizations applied to the outputs derive these parameters.
For coordinates Ja, first compute ray origins and directions from the camera's intrinsic and trinsic parameters. We then compute = ray, + d ray dird where is the 1-channel ray distance predicted by our model. Depth is computed the sigmoid function, and near and far are hyperparameters. In all our experiments, we set near to 0.1 and far 400.0.
near + (d))= (far-neat) near). where represents
For rotation R, we parameterize it with 4-dieensional quaternion vectors. Note that there is a default normalization step gsplat (Ye al., 2024) that applies L2 normalization to ensure quaternion vectors are unit vectors.
For scale s, we compute s min (exp(s' - 2.3),,.5), following Zhang et al. (2024), where s' rep- resents the outputs before normalization. This regularization limits the maximum size of Gaussians and improves training stability.
For opacity o, we compute = o(o'' - 2.0), again following Zhang et al. (2024).
For color c, we compute acc)) 1. When computing PSNR, we map color back to [0, 1].
Sky MIP. The modulated sky MLP predicts sky color from view directions by conditioning sky token Csky through modulated linear layer. Specifically, rrequency-embedder viewing directional vectors (d) are linearly projected to 64 dimensions, then normalized using LayerNorm without affine parameters. The sky token outputted from the Transformer serves as the conditioning vector C.ky (768 dims) is mapped 64 dimensions and used in adaptive layer normalization (AdaLN) process, where Caky is transformed to produce shift and scale vectors, each size 64, modulating the normalized features by x (1 + scale) + shift. Finally, the modulated features linearly transformed to an output of 3-dimensional color.
Mask Decoder. The convolution layers in the mask decoder are similar those used in SAM (Kir- illov et al., 2023), which is defined as:
self selffouutput.uuccaiing
nn. Sequential nn mn.Conv/rranspose22 (embed dim, 512, (511)) nn.GELU mm.con/Iraaappose221111 256, kernelsize-2, stride-2), LayerNozm2d1(55),
stride=2),
nn Gn.GEL
nn. am.ConvTrranspose2dd22 (256, 128, kernelsize=2, stride=2), Gn.GEL ()
The input to this decoder is the ViT output feature maps. After they are upsampled by the mask decoder. they are projected into 32-dimensional space using linear layer. Additionally, motion tokens are mapped into a 32-dimensional space using set three-layer MLPs, following the design in SAM (Kirillov et al., 2023).
Training. We train our model for 100,000 iterations with a global batch size of 64 on NVIDIA A100 GPUs, using learning rate of 4 10-4 The training process utilizes the AdamW opti- mizer (Loshchilov & Hutter, 2019) along with a cosine learning rate scheduler that includes linear warmup phase over the first 5,000 iterations. We enable the LPIPS loss (Zhang et al., 2018 8) only
17

---

Published as a conference paper at ICLR 2025
after 5,000 iterations, we find this approach stabilizes training. Gradient checkpointing is enabled by default to reduce memory usage. Behind the scene, we observe that STORM benefits from longer training durations and larger model sizes. We maintain the default setup to ensure alignment with our baseline in this work. However, an attractive direction for future work is to explore the scaling laws of STORM (Zhai et al., 2022).
Point trajectory estimation. We visualize the trajectories of dynamic Gaussians. These trajectories are obtained by chaining per-frame scene flows. Specifically, for each frame at L, we use the pre- dicted scene flow to transform Gaussians to its next frame / + to obtain its estimated destination. Then, for every Gaussian at 1, we find its nearest Gaussians transformed from t and connect them to visualize the trajectories. This process is recursively applied to all frames to obtain the final trajectories.
A.2 LOSS FUNCTION
We present more details about our loss function here. Given the rendered images i, depth maps D. opacity maps Ô, and velocities for all 3D Gaussians along with the corresponding observed camera images depth maps D, and sky masks M that are predicted by a pre-trained segmentation model (Chen et al., 2022), we compute the overall loss
L= Lrecon + Asky Laky + Aveg Greg,
(AI)
where the reconstruction loss combines RGB loss, depth loss, and LPIPS loss (Zhang et al., 2018):
| - ||2 |I( D)/ (t//max(DD)1 + Alpips -PIPPS (i.1)
Lrecon
(A2)
and the sky loss and velocity regularization loss are MSE losses that encourage sparsity:
1
Say - (1-0011 Lrgg = 1 /
(A3)
3
Here, the À terms control the relative weighting of each loss component. For the LPIPS loss, utilize a VGG-19-basee (Simonyan & Zisserman, 2014) implementation. We set Alpips to 0.05, Asky to 0.1, and Xreg 5e-3 in all experiments.
A.3 BASELINE IMPLEMENTATIONS
For per-scene optimization 3DGS-based methods, we use the recently open-sourced codebase DriveStudio from Chen et al. (2024), which includes implementations for PVG (Chen et al., 2023), DeformableGS (Yang et al., 2024b), and 3DGS (Kerbl et al., 2023) on the Waymo Open Dataset. For EmerNeRF (Yang et al., 2024a), we directly modify their officially released code. Since our task is to reconstruct short-sequenced dynamic scenes from sparse observations (3 cameras x 4 timesteps), the original training recipes designed for long-sequenced dense views (3 cameras x 200 timesteps) are no longer appropriate. Therefore, we reduce the training iterations for all methods from 20,000 to 5,000 and linearly scale down all iteration-basee hyperparameters. In our prelim- inary experiments, we did not observe significant differences between training for 20,000 versus 5,000 iterations, as there are only limited training views available, while training 5,000 iterations is much faster.
For generalizable approaches, LGM (Tang et al., 2024) has open-sourced their code and pre-trained models, whereas GS-LRM (Zhang al., 2024) has not. However, LGM is originally trained on an object-centric synthetic dataset, which has a significant domain gap compared to our problem. Therefore, we followed their official code to reimplement their model within our codebase elimi- nate potential misalignments due to differences in data processing, learning rate scheduling, super- vision, and optimizers. For GS-LRM (Zhai al., 2022), we implemented the model according to the descriptions provided in their paper. We train these models on the same dataset ours with the same color, depth, perceptual and sky supervision, and the same number of iterations. Since these models do not inherently support sky processing, we modify them to predict the depth of the sky as the far plane, a predefined hyperparameter. This adjustment already enhances the performance of these methods. The number of trainable parameters of these models controlled to be similar, i.e., GS-LRM has 86.68M parameters, LGM has 103.29M parameters, while our default STORM has 100.60M parameters.
18

---

Published as a conference paper at ICLR 2025
Dynanic PENR(11
Acog5(e)
Dynamic PSNR(
Accg5(+>
22.70
0.80
92.00
21.20
$0.02
76.31
E.44
0.20
21.54
0.38
19.66
02.00
21.08
77.98
21.56
76.25
22.30
21.90
75.02
0.81
9.805
8.091
0.9865
0.01
0.885
0,091
0.0935
16
32
326
(a) Effect of flow regularization loss coefficient Areg
(b) Effect of number of motion takens M
Figure B.I: Ablation study on velocity regularization and motion tokens. (a) Effect of velocity regularization coefficient Xreg: We evaluate rendering quality using dynamic PSNR and flow esti- mation performance using EPE3D and Accs. We find that excluding this regularization frequently leads gradient explosion and NaN loss. (b) Impact of motion token count: We study how the number of motion tokens M influences rendering and motion estimation performance.
FSN8 (+)
(PE3D())
Dynamic PENR
EPE3D1
77,08
0.10
20.48
0.62
0.51
79.05
20.00
21.40
8:48
15.50
0.30
20.00
17.35
8:25
6.24
10
6
2B
(a) Zera-shat transfer different number of testing input timesteps
(b) Effect of number of training input timesteps
Figure B.2: Ablation study on input timesteps. (a) Zero-shot transfer evaluation: We test STORM pre-trained with 4 input timesteps with varying test-time input timestep configurations without re- training. (b) Scaling training timesteps: We train STORM with different numbers of input views to assess its adaptability to changes in input timesteps.
B ADDITIONAL RESULTS
B. COMPARISON ON ADDITIONAL DATASETS
NuScenes. The NuScenes dataset (Caesar et al., 2020) contains 1000 driving scenes, each lasting 20 seconds, captured at 12Hz frame rate. These scenes are divided into 700, 150, and 150 scenes for training, validation, and testing, respectively. Similar to our evaluation protocol for the Waymo Open Dataset (Sun al., 2020), we 3 frontal cameras at a roughly 5.5x downsampled resolution (160 288) and leverage both sample (key frames) and sweep data. Models are trained on the training set and evaluated on the validation set with unchanged hyperparameters.
Argoverse2. The Argoverse2 dataset (Wilson et al.) contains 1,000 driving scenes, split into 700 for training, 150 for validation, and 150 testing. includes data from seven ring cameras, providing 360-degree view. For training and evaluation, we use the three frontal cameras, resampled to 192 x 256 resolution. The original central camera resolution 2048 x 1550, while other cameras 1550 x 2048, resulting in reversed aspect ratios. We do not apply special adjustments for this discrepancy and resize all images uniformly to 192 x 256. To simplify processing, performance is measured full images without extracting dynamic instances. Notably, depth RMSE is higher for this dataset compared to NuScenes and Waymo, which is expected due to Argoverse2's larger sensing range of over 200m, in contrast the approximately 80m range of the other datasets.
B.2 ABLATION STUDY
We study the effect of different components of our method here. All experiments here are conducted on the Waymo Open Dataset. To manage the computational demands of these extensive experiments, models studied in Fig. B.1 are trained with a global batch size of 32 for 100k iterations. For the models examined in Fig. B.2 B.2-(b), we use global batch size of 64 over the same number of iterations.
19

---

Published as a conference paper at ICLR 2025
Velocity regularization coefficient Areg. The impact of the velocity regularization coefficient is illustrated Fig. B.1-(a). We observe that omitting this term often results in gradient explosion and NaN loss. Thus, this regularization term is indispensable. In this controlled setting, the optimal coefficient is found to he 5 x 10-3, as achieves the best dynamic PSNR and ranks second in flow estimation performance.
Number of motion tokens M. We evaluate the effect of the number of motion tokens on rendering and flow estimation performance in Fig. B.2-(b). When no motion token (M 0) is used, the dynamic mask decoder directly predicts pixel-aligned velocities from image embeddings, keeping the number of learnalle parameters nearly constant ensure fair comparison. As shown in Fig. B.2-
(b), STORM demonstrates robust performance across different motion token counts, with the best results obtained at M 16.
Number of input timesteps. Our default configuration trains and tests STORM with 4 input timesteps. Leveraging the seuuence-to-squeneee nature of our Transformer-basse model, we can flexibly adjust the number of input timesteps during both training and testing by appending tokens from more input timesteps or dropping tokens from existing input timesteps. This flexibility en- ables to study the effect of input sparsity on STORM's performance. We conduct two ablation studies in Fig. B.2. First, we test STORM trained with 4 input timesteps under varying test-tima input timestep configurations without re- re-training. This zero-shot transfer experiment demonstrates that STORM generalizes well to unseen input configurations, though it achieves peak performance with 4 timesteps, expected. In the second study, we re-train STORM with different numbers of input timesteps and evaluate their performance. Results indicate that increasing the number of input timesteps improves performance. Notably, when trained with single timestep, STORM transitions into a future prediction framework. Even in this configuration, significantly outperforms per-scene optimization approaches that utilize input timesteps for reconstruction, achieving a PSNR improvement on dynamic regions (ef. Table 1).
C LIMITATIONS
While our model benefits from the scalability and flexibility of Transformer architectures, it comes with certain trade-offs. One limitation is the processed sequence length. STORM typically operates on images downsampled by factor of 8, using inputs from three cameras and four timesteps, which results in around 7k tokens. Although we have fine-tuned STORM to handle up to 32,000 tokens in preliminary experiments to enable higher resolution images, longer temporal windows, additional camera views, this comes with a non-trivial increase in computational costs for both training and inference. Another limitation is that model requires camera intrinsic and extrinsic parameters as inputs. While these parameters are readily accessible in autonomous vehicle datasets, they may be more difficult to obtain in other domains, potentially limiting the ability to directly train and test STORM on those data domains without additional effort or preprocessing. Future works to address these limitations include better Transformer architecture with reduced complexity, joint optimization of camera parameters, and the use of geometric foundation models.
20


Reviews:
[{"review_id": "Qk2q4inXtU", "reviewer": "Reviewer_Wu7P", "summary": "This paper proposes a method for dynamic 3D reconstruction (4D reconstruction) of urban scenes from temporally and spatially sparse video input.\n\nThe model is based on a per-frame transformer architecture that takes image, ray (from given camera intrinsics and extrinsics), time and motion latent tokens as input as well as some special tokens for handling sky and varying camera exposure. The model takes tokens from an image coming from single camera v and timestamp t, and then processes these tokens jointly with a self-attention transformer, to output feature maps F^v_t for that image. These tokens are then decoded into pixel-aligned 3D Gaussian Splatting parameters G^v_t, which are used to reconstruct the input frame as well as to compute the depth map. In addition, the motion tokens are decoded into a set of motion queries and velocity bases, which are employed to estimate the velocity vector of each 3D Gaussian.\n\nGiven a set of training frames coming from one or several cameras, the model is run on each frame independently to obtain independent sets of Gaussians. These independent Gaussians can then be warped to any particular timestamp using the velocity estimates, and assuming velocity constancy.\n\nIn addition to the vanilla model, the authors propose a latent variant where instead of predicting the Gaussian parameters directly, the transformer first predicts higher dimensional vectors on a lower-dimensional grid, which are then upsampled to full-resolution using a CNN decoder.\n\nThe model is trained and evaluated on clips from the Waymo Open Dataset, using the provided camera intrinsics and extrinsics, and three frontal car cameras.\n\nThe authors evaluate their method against both per-scene optimization and feed-forward methods. For each 20 frame test clip, 4 frames are provided as inputs for reconstruction to each baseline, and the rest are used for evaluation. The results in Table 1 show that the proposed method obtains SOTA performance on reconstruction metrics compared to both per-scene optimization and feed-forward methods. In addition, the authors evaluate their method on the scene flow estimation task, also obtaining SOTA results.\n\nFinally, the authors link a website with numerous quantitative results which allow to better assess the performance of the proposed method.", "questions": "- Please clarify how the images \\hat{I} are computed for a given training batch and which aggregations and transformations are involved.\n\n- Please clarify the shapes of q and k in eq. (4) and how the weights are used for generating the motion segmentation visualizations, as well as how the number of motion basis is determined.\n\n- Please clarify how the baselines are trained and which inputs they require.\n\n- If possible, having 3D visualizations of the results would be beneficial.\n\n- Please provide additional details about how point-tracking can be obtained, and how the editing, control and inpainting behaviours can be obtained in latent-STORM.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is an interesting method which applies per-frame feed-forward 3DGS models to a multi-camera scenario for the urban driving setting. While similar methods exist for object-centric scenes (Splatter Image, LatentSplat) and static objects, or even object-centric scenes and dynamic objects (4DGS Guanjun Wu et al.), I believe the extension to outdoor driving scenes is novel and relevant. The authors successfully devised strategies to deal with the complexities of such scenes, by explicitly modeling the sky and the differences in camera exposure.\n\nFurthermore, the authors adopt a similar strategy for modeling dynamics through scene flow as in NSFF, but which results to be more tractable due to the usage of 3DGS as a discrete representation of the scene. This methods seems to be effective in modelling short term dynamics.\n\nQuantitative and qualitative results show that the method achieves good new-view synthesis results from only a few seen views.\n\nThe proposed latent-STORM variant shows improved reconstruction of local details, such as for the articulated bodies of walking pedestrians.", "weaknesses": "Some details of the method are not very clear from the paper. For instance, it's not clear how the aggregation and transformation processes are applied on the predictions of the 4 training frames for generating the predicted images \\hat{I}. \n\nIn addition, it's not clear whether eq. (4) is correct and how exactly the weighs w are computed as the exact shapes of q and k are not specified. Also, it's not clear how the two different estimates v_t^{-} and v_t^{+} from eq (1) are obtained from what is described in eqs. (3,4). Furthermore, not many details are given about how these weights can be used to generate the motion segmentation figures, and the impact of the number of motion basis. It's also not very clear how the background scene is treated during this motion estimation.\n\nRegarding the quantitative results, it's not clear if the baselines for Table 2 have been retrained specifically on the same training set for fairness. Also, it's specified that the baseline models NSFP and NSFP++ require LiDAR input, but it's not clear whether these methods also benefit from having 3 different input views. \n\nRegarding the qualitative results, having 3D visualizations in addition to depth-maps would help further assessing the temporal consistency and geometric correctness of the predictions.\n\nFinally some of the proposed applications or extensions in sec. 4.3 are not explained in sufficient detail. For instance, how the point trajectories are chained for point tracking, given that the local scene-flow estimations are sparse and don't cover the whole space. In addition, it's not clear how latents can be manipulated in the latent-STORM model for achieving the edition, control and inpainting effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wyAyhYCxXS", "reviewer": "Reviewer_JKrr", "summary": "This paper introduces STORM, a spatio-temporal reconstruction model for rendering dynamic outdoor scenes with sparse images. Combining image tokens, ray tokens, and time tokens, STORM uses a Transformer to predict attributes of 3D Gaussians (position, rotation, scale, opacity, color, velocity) at each timestep for real-time rendering. The training requires ground truth images, depth map and predicted sky mask from pretrained segmentation model.", "questions": "In Fig.3, can authors add ground truth images for comparison? Based on current visualization, Latent-STORM sometimes performs better and sometimes performs worse than STORM.\n\nCould authors provide some reference for handling exposure mismatches with affine transformation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written.\n\nSTORM may be the first method that combines feed-forward 3D Gaussian Splatting and dynamic scenes.\n\nSTORM achieves better rendering quality and scene flow accuracy than baselines.", "weaknesses": "STORM only evaluates the quantitative performance in short video clips (about 2 seconds on Waymo open dataset). Applying STORM on 20-second videos requires processing in multiple video clips (L428) because of the heavy computation, which is also discussed in limitations. \n\nThere is no quantitative ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jTpXAMIoD7", "reviewer": "Reviewer_N7nA", "summary": "The paper presents STORM, a spatio-temporal reconstruction model that uses a data-driven Transformer architecture to accurately reconstruct dynamic outdoor scenes from sparse camera images without explicit motion supervision. It employs 3D Gaussian Splatting to aggregate per-frame data into an amodal, cohesive representation, enabling consistent scene reconstruction over time. The model is self-supervised, relying on reconstruction loss and incorporating motion tokens to capture motion primitives efficiently. Extensive testing on the Waymo Open Dataset shows that STORM outperforms traditional methods in photorealism and speed, achieving real-time performance within 0.2 seconds per clip.", "questions": "1. Could the authors include comparisons across multiple datasets to validate the generalizability of the proposed method?\n2. Could the authors provide ablation studies to demonstrate the effectiveness of each proposed module in the method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a clear and well-organized approach to addressing the defined problem of dynamic scene reconstruction.\n2. The proposed method achieves state-of-the-art performance, showcasing its practical effectiveness and potential.\n3. The video results available on the linked website are highly illustrative, providing clear insights into both the strengths and limitations of the proposed method.", "weaknesses": "1. The evaluations are limited to the Waymo dataset, which raises concerns about the comprehensive validation of the method. To strengthen the claims, it would be beneficial to demonstrate the effectiveness of the model across multiple datasets.\n2. While the paper targets large-scale outdoor scene reconstruction, existing works such as [1] also address similar challenges. If the primary focus is on dynamic scene reconstruction, the paper’s title should explicitly include the term ‘dynamic,’ and comparative results with dynamic 3D reconstruction methods like [2] should be included to provide context and highlight advancements.\n3. One of the main contributions is the introduction of 'motion tokens' in the Transformer’s input sequence, alongside the unique loss function components (reconstruction loss, sky loss, and velocity regularization loss). However, the absence of ablation studies makes it difficult to discern the effectiveness of these specific modules. Adding ablation studies would enhance the paper by showing the impact of each proposed component.\n\n[1] Lin, Jiaqi, et al. \"Vastgaussian: Vast 3d gaussians for large scene reconstruction.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n[2] Fischer, Tobias, et al. \"Dynamic 3D Gaussian Fields for Urban Areas.\" *arXiv preprint arXiv:2406.03175* (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "foh2pfoj5b", "reviewer": "Reviewer_HXs2", "summary": "This paper introduces a novel pipeline called STORM for dynamic scene reconstruction in real-world environments. \n\nThe core concept involves aggregating reconstructed 3D Gaussians and their motion into a unified representation, enabling the rendering of an amodal representation at any specified time step. \n\nAdditionally, to ensure STORM's effectiveness in diverse scenarios, several significant enhancements have been implemented. \n\nExperimental results demonstrate the efficacy of the proposed method.", "questions": "The paper highlights that previous reconstruction methods rely on “dense observations across space and time,” whereas STORM “reconstructs dynamic 3D scenes from sparse, multi-timestep, posed camera images.” An insightful ablation study would involve varying the number of camera views and input time steps to assess how effectively STORM performs with different levels of view sparsity. This would provide a clearer understanding of the system's robustness under sparse observation conditions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and accessible, making it easy to follow the proposed approach.\n\n2. The core idea is straightforward and well-conceived. By integrating 3D Gaussians into a unified representation, STORM enables rendering at any time step, allowing the entire network to be supervised by the color of the target time step.\n\n3. The experiments are comprehensive and informative, demonstrating the effectiveness of the overall pipeline as well as specific enhancements, such as Latent-STORM.", "weaknesses": "The experiments are conducted solely on the Waymo Open Dataset, which may raise concerns about STORM’s robustness and generalizability to other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)