Instant4D: 4D Gaussian Splatting in Minutes
Authors: Zhanpeng Luo
Haoxi Ran
Li Lu
Venue: NeurIPS 2025 poster ()
Keywords: 4D Reconstruction
Casual Video
Efficient
Primary area: applications
TL;DR: Reconstruct a casual video in less than 2 minutes

Abstract: Dynamic view synthesis has seen significant advances, yet reconstructing scenes
from uncalibrated, casual video remains challenging due to slow optimization and
complex parameter estimation. In this work, we present **Instant4D**, a monocular
reconstruction system that leverages native 4D representation to efficiently process
casual video sequences within minutes, without calibrated cameras or depth sensors.
Our method begins with geometric recovery through deep visual SLAM, followed
by grid pruning to optimize scene representation. Our design significantly reduces
redundancy while maintaining geometric integrity, cutting model size to under **10%**
of its original footprint. To handle temporal dynamics efficiently, we introduce a
streamlined 4D Gaussian representation, achieving a **30×** speed-up and reducing
training time to within two minutes, while maintaining competitive performance
across several benchmarks. Our method reconstruct a single video within 10
minutes on the Dycheck dataset or for a typical 200-frame video. We further
apply our model to in-the-wild videos, showcasing its generalizability. Our project
website is published at https://instant4d.github.io/.

Instant4D: 4D Gaussian Splatting in Minutes
Zhanpeng Luo University of Pittsburgh Zhanpengluo@pitt. edu
Haoxi Ran" Carnegie Mellon University ranhaoxi machaoxx@cmu..co @emme.meeaeee.e
Li Sichuan Univeristy
lul faliiscu.. edu.
ILST
Reediar oin
Siile
Rondering Quality (PANE am
Nedeo I7INE
Pat Mrexest
PAN 12
Reader:
PSNIL
KEALya PANAL
Training Timee Olliates Legarittunk Seaks
Part (a): INSTANT4D achieves better rendering performance with fewer training iterations against the original 4D Gaussian Splatting (4DGS) [41]. Part (b): Visualization on detailed dynamic object like "spinning" apple. After 40-minute optimization, the rendering result of 4DGS remains blurry, while our method achieves better visual quality by 0.8 dB PSNR, faster optimization for convergence by 85%, and lower GPU memory by 69%. Part (c): Bubble chart comparing with most recent art. Note that the bubble size indicates the size of an optimized model.
Figure 1:
Abstract
Dynamic view synthesis has seen significant advances, yet reconstructing scenes from uncalibrated, casual video remains challenging due slow optimization and complex parameter estimation. In this work, we present INSTANT4D, a monocular reconstruction system that leverages native 4D representation efficiently process casual video sequences within minutes, without calibrated cameras or depth sensors. Our method begins with geometric recovery through deep visual SLAMM, followed by grid pruming to optimize scene representation. Our design significantly reduces redundancy while maintaining geometric integrity, cutting model size to under 10% of its original footprint. To handle temporal dynamics efficiently, we introduce a streamlined 4D Gaussian representation, achieving a 30x speed-up and reducing training time to within two minutes, while maintaining competitive performance across several benchmarks. Our method reconstruct single video within 10 minutes on the Dycheck dataset for typical 200-frame video. We further apply our model to in-the-wild videos, showeasing its generalizability. Our project website is published at https: inittannttta.dd... github. io/.
Introduction
Reconstructing dynamic 3D scenes from casually captured, uncalibrated video is fundamental challenge computer vision, critical for applications such as augmented reality (AR), virtual reality (VR), and immersive content creation. While static 3D scene modeling has seen remarkable
Project Lead
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

progress [13, 1, 22, 21, 23,31], extending these techniques to dynamic scene remains challenging, especially when handling moving objects with monocular camera only. This process often requires time-consuming optimization [12, 26] to recover scene geometry and accurate motion. Furthermore, occlusion, deformation, and irregular camera paths add complexity, making efficient and coherent modeling difficult in uncalibrated settings.
Recent approaches leverage optical flow [12], depth [9], point-tracking [29], and pose prior [5] to solve this challenging task. Nevertheless, reconstructing from short, causal video still requires hours of optimization. Inspired by recent advances in deep visual SLAM [11] and real-time rendering [6. 41] we propose INSTANT4D, reconstruction system for dynamic scene reconstruction in only minutes. We employ deep visual SLAM to estimate camera trajectories and refine the monocular depth into video consistent depth. These depth maps are then back projectee into a dense 3D point cloud as 4DGS optimization. Furthermore, we propose a grid pruning strategy, which efficiently reduces redundancy while preserving occlusion structures, reduces the model size to less than 10 % of its original footprint and significantly accelerates the optimization process, achieving 30x acceleration compared to recent works of art.
Then, model the attributes of dynamic scenes with the native 4D Gaussian primitive [40, 41] that captures motion without rigidly segmenting the scene into static and dynamic parts. Unlike previous approaches 5, 9, 29, 33], our method further enables naturally captures on some background variations, However, modeling sparse and temporally inconsistent observations make the 4D Gaussian overfit and prematurely disappear in poorly observed regions. We address this through a carefully crafted initialization scheme and a motion-aware 4D covariance model.
INSTANT4D demonstrates short training time, low peak memory, fast rendering speed, and high rendering quality, as shown in Figure 1. Specifically, we reconstruct scenes on NVIDIA dataset [43] in average 2 minutes and on Dycheck [3] dataset average 7.2 minutes. For typical 5-second, 30 FPS video, our method completes optimization within 8 minutes. We achieve 30x speed-up in reconstruction time. 90% reduction in memory and demonstrate competitive performance on several benchmarks. Our primary contributions are summarized as follows.
We propose INSTANT4D, modern and fully automated pipeline that reconstructs casual monocular videos within minutes, achieving 30x speed up.
We introduce a grid pruning strategy that reduces the number of Gaussians by 92%, preserv- ing the occlusion structures and enabling scalability to long video sequences.
We present a simplified, isotropic, motion-aware 4DGS formulation, in monocular setup, which achieves 29% better performance than current state-of-the-- methods on the Dy- check dataset.
2 Related Work
2.1 Dynamic Novel View Synthesis (NVS)
Earlier methods like [43] used single view and multiview stereo depth to synthesize novel views of dynamic scenes from a single video using explicit depth-based 3D warping. recent line of work [20, 12, 16] extends NeRF [14] to handle dynamic scene by adding a time dimension. In particular, RoDynRF [12] split the scene into static and dynamic parts and used the static radiance field [14] to estimate only the camera poses, which were robustly reconstruct from unposed RGB video. However, limited by Neural Radiance Fields' rendering speed and numerous iteration demands, usually RoDynRF [12] takes over 2 days to reconstruct casual video.
NeRF-based NVS
Gaussian-based NVS Approaches to modeling motion with Gaussian Splatting can be broadly categorized into three types: deformation- trajectory-based and HDDD-Gussiia--ae -ased methods. Deformation-bast based methods [35, 42, 36, 7] employ multi-layer perceptrons (MLPs) or low-rank K K-planes to dynamically adjust the parameters of Gaussians over time. Although expressive, these methods typically suffer from slower training and rendering due to the added complexity of learning continuous deformations. Trajectory-basee Gaussians by pre-computing trajectories, often derived from external motion estimators. Although this enables precise tracking of object movement, it demands substantial preprocessing. For example,
methods [29, 9, 28] explicitly model the motion of

---

Optimization Pipeline
4D Garamiata
Deep Visual SLAM
Geonetrie Recovery
Coddtta Time
Spaating View
Rendetig Lauci
R,, C,a
Video
Geometric Recovery
Grid Pruning
Cloud Cousiaa Initinization
Back Project M T,C00
S= Crn.
G Po
Figure 2: Pipeline of INSTANT4D. We use Deep Visual SLAM model and Unidepth [19] to obtain camera parameters, and metric depth. The metrics depth would be further optimized to consistent video depth. After that we back project from consistent depth to get dense point cloud, further voxel filtered to sparse point cloud, as discuss in Section 3.2. Based on the 4d Gaussians Initialization, we can reconstruct scene 2 minutes. More details about optimization are described in Section 3.3.
3-second video generate trajectory files of more than 100 GB [29]. Furthermore, these methods rigidly partition the scene into static and dynamic components, neglecting subtle background motion that may still be perceptible. This hard segmentation introduces artifacts when background elements exhibit slight temporal shifts. Finally, 4D-Gaussian baase methods [38, 41, 37] extend the standard 3D Gaussian Splatting by including temporal dimension to the representation. At given timestamp, the 4D Gaussian conditioned to a 3D distribution. However, is prone to overfitting in monocular settings where certain regions briefly visible only; without careful temporal management, 4D Gaussians tend to vanish prematurely as they are underconstrained in time.
2.2 Visual SLAM and SIM
Classical cructurr--rom-motiie (SIM) and Simultaneous Localization and Mapping (SLAM)
pipelines recover camera poses and sparse geometry by minimizing re-projection or photometric emoos through bundle adjustment [4, 24J. Deep visual SLAM systems, such as DROID-SLAM [27]
replace handcrafted heuristics with differentiable bundle adjustment layers and data-driven priors, resulting improved robustness in texture- porr scenes and mild dynamics. Recently, MegaSAM [11] has extended the differentiable bundle adjustment to dynamic scenes. MegaSAM leverages data learned before camera and flow supervision, robustly recovers camera parameters, and generates consistent video depth. Another notable recent data-driven method, DUSt3R [32], powered by CroCo encoder [34], learned from vast pretrained data, reconstructs camera poses quickly and robustly. Follow efforts such as MonST3R [44] extend DUSt3R to dynamic scene with additional
dynamic supervision. CUT3R [30] also applies the CroCo [34] encoder and applies continuous learning for static and dynamic reconstruction.
3 Method
Given an unconstrained video sequence V = {4 with resolution H x W, our goal to estimate camera extrinsics G, E SE((), intrinsics K E 3333 and temporally consistent depth maps δ = D. J Leveraging these estimates, we reconstruct a dynamic 4D scene representation and render novel views real-time from arbitrary viewpoints G* given timestamps using Gaussian Splatting. Our pipeline is illustrated in Figure 2 We first briefly summarize the relevant background on

---

deep visual SLAM and Gaussian Splatting (Section 3.1), before detailing our geometric initialization pipeline (Section 3.2) and optimization strategy (Section 3.3).
3.1 Preliminary
MegaSAM [11] extends the differentiable bundle adjustment framework from DROID-SLAM [27] to
handle dynamic monocular videos. Specifically, (initialized by Depth Anything [39]), camera poses
G. E SE(3), and camera intrinsics represented by focal length f (initialized by Unidepth [19]).
During optimization, MegaSAM jointly refines these parameters by iteratively minimizing the weighted reprojection residuals between predicted optical flow and rigidly computed flow from current estimates:
Uij = π GG3 o ग. "
where G.y denotes the relative transformation from frame to frame j, and iT-) denotes the camera projection operation.
MegaSAM optimizes these parameters using the Levenberg-Marquaant (LM) algorithm:
(J T WJ + Adiag(] WJ) A Wr,
(2)
where A (AG, Af) is the parameter update, J is the Jacobian of reprojection residuals with respect to the parameters, and W is diagonal weighting matrix derived from each frame pair. The damping factor À is adaptively predicted by the network during each iteration to stabilize optimization.
4D Gaussian Splatting [41] extends the explicit 3D Gaussian Splatting [6] to dynamic scenes by incorporating temporal dynamics into scene modeling. Specifically. standard 3D Gaussian is parameterized by its mean position μ E R covariance matrix Σ E R3x3, and opacity a E R as follows:
(- 1 (p 11)
G(p, E,a) = exp
(3)
To represent a view- and time-dependent appearance, 4DGS employs a set of 4D sphericylindrical harmonics (SCH), constructed by combining 3D spherical harmonies (SH) with temporal Fourier basis functions:
2Tm T
T (1, 0,
Y (0,
where ym are the standard 3D spherical harmonics indexed by degree 0 and order m with -1 < < 1, n is the temporal frequency index, and T denotes the temporal period.
3.2 Geometric Recovery
Back Projection on Consistent Depth Given the input image sequence V, we first apply MegaSAM [11] to obtain estimates of camera extrinsics G; E SE(3) and intrinsics K E gax3 We then refine the initial monoeular depth estimates to achieve temporally consistent depth maps Đ [ through an additional first-order optimization. Using these refined depths, we back- project each pixel coordinate p; from the image space into 3D world coordinates X; E R3
(5) where - denotes the back projeeroenttoon of pixel p; (in homogeneous coordinates Pa) and depth D, into the camera frame, and G; transforms the 3D point into the world coordinate frame. This procedure yields dense colored point cloud representing the scene geometry. To handle variations depth scale, particularly in outdoor scenes with unbounded regions (e.g., skies), we adaptively increase the voxel size S, during subsequent grid pruning.
X; - - (p., K-))

---

Motion Probability Estimation Separating dynamic foreground objects from the static background remains beneficial, as it allows us efficiently allocate computational resources by representing the static background sparsely while preserving detailed granularity in dynamic regions. To this end, we leverage intermediate predictions from the deep visual SLAM pipeline's low resolution motion probability map in E $ x $ to interpolate to a per- pixel motion probabilities. We then employ Otsu's 's thresholding method [15] on these probability maps generate binary masks that distinguish static from dynamic scene elements. Empirically, we observed that in sequences with large temporal sampling intervals, motion estimation can fail to reliably identify moving objects the sequence boundaries (i.e., first and last frames). To mitigate this issue, we introduce synthetie pseudo-frames at both ends of the sequence, thereby improving motion consistency. Additional visualizations of our motion estimation procedure are provided in the supplementary materials.
Grid Pruning Back- projecctoj depth maps for a 512 512 video sequence of four seconds (30 FPS) yields 30 M raw 3D points. To eliminate redundancy and resolve self-occlusions, we partition the world space into a regular voxel grid and retain only the centroid of points within each occupied voxel. The edge length is adapted to the scene scale,
D
=
(6)
where D, is the mean depth of the frame i. f the estimated focal length, N the number of frames, and X a user-defined scale factor. Each 3D point is assigned to a voxel by integer division with Sp: points in the same cell are aggregated via a hash-map and replaced by their centroid, while other attributes other than position, such as color, timestamp, time scale st, and motion probability i are averaged. Voxels with insufficient support are discarded outliers suppress noise.
In the NVIIDA Dynamic Scene benchmark [43], this pruning reduces the model's 's memory footprint from 10.7 GB to 0.83 GB (92%), reduces the training time from 181 to 42 (4x speed-up), and improves rendering performance from 154 FPS to 981 FPS (see Table 2). With the resulting compact geometric priors, we can reduce the need of the densification stage conventionally applied by 3D Gaussian Splatting [6].
3.3
Optimization
Motion Modeling Once 3D positions and RGB colors are acquired, we optimize the remaining Gaussian attributes such as rotation scaling opacity o, and motion m with lightweight 4D Gaussian formulation. Each Gaussian is described by 4D mean μ = (pzz Py. M2: pe) R
diagonal scale vector s (Suyz: s.) scalar opacity a, and a rotation matrix R Rxx4 Unlike prior 4DGS [41] work that relies on two entangled quaternions and high-order SCH, we model the Gaussian's appearance with simple RGB value, rather than a high-order spherical harmonious function. The simple design cuts the per- Gaussinnn parameter count by over 60% and empirically lessens over-fitting in monocular settings.
We condition a multivariate 4D Gaussian primitive towards a 3D Gaussian primitive at timestamp t during the rendering time, which can be formulated as follows:
Pzzzlt = M1. + E1,3, ΣT.1 (t - p4),
(7)
Ezyzlt Σ1a,13 - Σ1:3, Σi.4 E4,1:3.
(8)
where E E 4 is the full covariance matrix. This conditioned form encodes continuous motion without explicit trajectory storage.
Isotropic Gaussian Although anisotropic Gaussians can model fine finiine shape details, their additional degrees of freedom frequently destabilize optimization in monocular scenarios. Inspired by Gaussian Marbles [25], we therefore adopt an isotropic variant: the orientation matrix is fixed to the identity (R = /), and the covariance is parameterized by two scalars, one shared spatial scale 8ayz and one temporal scale 8,. This compact parameterization improves numerical stability, reduces memory usage, and acts as an implicit regularizer. As evidenced in Table 4, the isotropic model delivers higher robustness without sacrificing rendering quality.
5

---

DImBRsK Neri
Figure 3: Visual comparison on the NVIDIA dataset. [43]
Rendering FPS 1
Method
Calibration
Runtime |
PSNR
480 x 270
860 x 480
HyperNeRF [17]
COLMAP
64
0.40
17.60
DynamicNeRF [3]
COLMAP
74
0.05
26.10
RoDynRF [12]
COLMAP
28
0.42
0.13
25.89
4DGS [35]
COLMAP
1.2
43
29
21.45
Casual-FVS [8]
Video-Depth-Pon
0.25
48
27
24.57
InstantSplat" [2]
Visual SLAM
0.15
117
22.56
4DGS* [41]
Visual SLAM
0.16
98
18.34
Ours
Visual SLAM
0.02
822
676
23.99
Table 1: Quantitative comparison of efficiency and visual quality on NVIDIA dataset following [12].
Our implementation on the same server by replacing the calibration method (COLMAP) from the original paper with Visual SLAM for fair comparison.
Motion-Aware Gaussian In monocular 4DGS primitive modeling, static background primitives can vanish once they leave the camera frustum unless they are explicitly distinguished from moving objects. We apply the mask obtained from 3.2.
To make our 4D primitive aware of underlying motion in the monocular dynamic scene. Considering opacity o,
(V,,, μ4, E4,4) and Equation 7, can find that in the case of isotropic Gaussians, the temporal scaling 8t would be the only term in the covariance affect the Gaussian attribute related with time.
Σ1, St
Therefore, by explicitly setting 8; higher for the static region, those Gaussians that should remain in the 4D space will disappear. Those dynamic Gaussians will change their position and scaling according to the deviation of timestamp t. During rendering. Gaussians farther away from the timestamp will be culled if their opacity o, ( μ4, E1,1) falls below a threshold.
Experiments
4.1 Training and Inference Detail
Implementation Detail On the Dycheck iPhone dataset [3], we followed the evaluation protocol established by Jeong et al [5]. We set the maximum optimization iterations to 5,000 and adopted the standard 3DGS[6] hyperparameters for loss weights and learning rates, with the exception of reducing the position learning rate to le- and extending the learning rate scheduler to 5,000 steps.

---

Paper Windeall
Yodls
Spaa
Block
GS
Oure
Ground Touth
4B GB
Oure
Ground Truth
Figure Visual Comparison on the Dycheck [ataset.[3]
Component
SH
Filter
Densification
PSNR
Runtime (scc)
Memory (MB) Rendering FPS
Ours
23,99
42
832
981
SCH
23.83
59
2806
588
W.o. Voxel Filter
23,38
181
10676
154
Table 2: Ablation study on key components of INSTANT4D's influence on speed and memory. We analyze the effect of spherical harmonics (SH), grid filtering, and Gaussian densification on rendering quality, training runtime, memory usage, and rendering frame rate (FPS). Removing higher-order SCH 3.1 slightly reduces computational cost with minimal impact on PSNR. The grid filtering significantly reduces both memory footprint and runtime while maintaining rendering quality, highlighting its role as an effective regularizer against overfitting.
Our initialization strategy differed between model variants. For the Lite model, we initialized 4D Gaussians with voxel size of X, 4 for static regions and Åa 4 for dynamic regions. In our Full model, we set A = 1 but omitted the grid pruning step for dynamic regions to preserve detail and alleviate some potential underfit caused without densification. During practice,
can still & enable densification for better training view rendering quality. Temporal scaling was set to fps for dynamic regions, while static regions used constant scale equal to the entire video length (8: luiden).
For the NVIDIA Dynamic Scene dataset [43], we reduced the maximum optimization iterations to 1,500 while maintaining the same hyprraarameters only the learning rate scheduler's 's maximum step to match the shorter optimization cycle. The grid pruning and initialization parameters remained consistent across both datasets.
our implementation in Dycheck [3], adjusting
Runtime and Memory The computational requirements for our method scale with input video length, the SLAM system must track additional depth maps. Peak memory usage occurs during consistent video depth optimization, while 4DGS optimization maintains relatively stable runtime regardless of sequence length. Testing on single NVIDIA A6000 GPU, our Lite model completes the full training pipeline in 96 seconds with peak memory usage of 988 MB on the shortest sequence (235-frame windmilll,, and 131 seconds with peak memory of 1,147 MB on the longest sequence (379-frame "apple"). For Full model, geometric recovery processes at approximately 0.8 seconds per frame, requiring about minutes total for depth estimation, video depth consistency optimization, and camera tracking the "apple" sequence. Inference runs over 400 Hz, and the voxel pruning stage completes less than seconds per scene.
4.2 Evaluation on NVIDIA & Dycheck Benchmarks
Evaluation on NVIDIA We evaluated INSTANT4D against several baseline methods in the NVIDIA Dynamic data set following the protocol [12]. This dataset consists of seven scenes, each with 12
7

---

PSNR(1)
Apple
Block
Paper
Spin
Teddy
Average
Runtime(h)
Mem(GB)
D-NeRF [20]
24.23
21.80
21.85
22.15
19.46
21.50
24
RoDynRF [12]
17.38
15.99
20.71
16.66
13.28
16.80
22
15
4DGS [35]
23.24
22.05
21.03
22.99
18.89
21.64
1.2
21
Deform3D [42]
24.82
23.26
20.62
23.51
20.93
22.63
RoDynRF [12] (w.o. pose)
14.50
14.73
17.94
15.75
11.56
14.90
15
RoDyGS [5]
16.79
17.67
19.20
18.47
14.69
17.37
Ours (Lite)
23.48
23.18
23.60
19.96
23.02
1.1
Ours (Full)
26.84
23.98
24.77
25.25
21.78
24.52
0.12
Table DyCheck iPhone benchmark [3]. Methods above the mid-rule are trained with ground-truth camera; those below operate without calibrated poses. Runtime denotes the mean training time per scene and Mem the peak GPU memory during optimization. Runtime for RoDyGS, RoDynRE, and D-NeRF is provided by the authors of [3].
frames captured from 12 camera viewpoints for training, with testing performed from fixed viewpoints at consecutive timestamps. The visualization is shown in Figure 3.
To isolate the contributions of our approach, we developed two comparative baselines with similar training time constraints. The first InstantSplat [2] style baseline, adapt Fan al.'s 's approach [2], with covisible global geometry initialization and joint camera pose optmizaaion. For the previous part, we tested with counterparts against MAST3R [10] such as CUT3R [30] and MonST3R [44], but these models struggle with frequently shifting point clouds when processing long sequence. Therefore, we still use MegaSAM [11] as the visual SL.AM model. shown Table 1, our model achieves a higher rendering quality while maintaining significantly faster training times, demonstrating the effectiveness of our grid pruning and initialization strategy.
Furthermore, we introduce a 4DGS [41] baseline, This baseline isolates the contribution of our 4D Gaussian representation by implementing standard 4DGS approach without our isotropic and motion-aware aware Gaussian strategies. As shown in Table 1, our full model achieves superior rendering quality while maintaining significantly faster training times, demonstrating the effectiveness of our grid pruning and initialization strategy. The ablation results in Table 4 further confirm that omitting motion-awart Gaussians substantially degrades the quality of rendering across both datasets. This degradation likely stems from overlapping dynamic elements in world space, where improperly timestamped Gaussians occlude each other and impede optimization.
While some prior methods such as [12, 3] achieve higher PSNR values, they typically incorporate additional regularization techniques compensate for the limited information available in the 12-frame NVIDIA dataset. Nevertheless, our method offers a compelling trade-off, delivering competitive quality with reconstruction speeds that dramatically outpace previous approaches.
Evaluation on DyCheck The DyCheck iPhone benchmark [3] presents severe motion and parallax, making it stringent test for dynamic reconstruction. Following the RoDyGS [5] evaluation protocol, we report PSNR per-scene together with training time and peak memory (Table Our Lite variant already surpasses all baseline methods that do not require ground truth camera poses input, achieving an average 23.02 dB in just 0.03 with a footprint of 1.1 GB. The Full configuration lifts performance 24.52 dB, outperforming the concurrent RoDyGS by 7.15 dB and exceeding Deform3D[42) (which uses ground-truth poses) by 1.89 dB, yet still trains in only 7.2 minutes.
Both variants maintain real-time rendering (> FPS),,), confirming that our voxel- initialized, simplified 4D Gaussian representa- tion delivers state-of of-the-art quality at a fraction of previous computational cost.
From the visual comparison Figure 4, we can see that compared to the baseline 4DGS model, our method preserves significantly better for statie background as well for the dynamic object.
Bear
Breakdance
Figure 5: Visualization the DAVIS Dataset.
8

---

4.3 Evaluation on in-the-wild video
To assess performance on in-the-wild video, we conduct qualitative experiments on DAVIS Dataset [18]. Figure 5 shows renderings from both novel viewpoints and novel timestamps; complete video results are available on our project website. Our reconstructions exhibit crisp object boundaries and temporally coherent appearance. For example, both the Bear sequence (82 frames) and Break- dance sequence (68 frames) require 2 min for SLAM calibration and min for 4D reconstruction.
Additionally, we further discuss failure cases, In the low-texture Kite-surf sequence, the ocean dominates the field of view, leading to inaccurate visual-SLAM poses; consequently, the surfer occasionally disappears in the rendered output. In our future work, we tend to address such degenerate scenarios for texture-robust pose initialization.
4.4 Ablation and Analysis
We first conduct experiments on the NVIDIA [43] dataset to evaluate the impact of each component on training runtime and memory as seen Table 2. The grid pruning significantly reduces both memory footprint and runtime while maintaining rendering quality, and the simple RGB value we used shows a beneficial trade-off on both performance and training speed.
To assess the impact of each design choice, we conduct an ablation study on the DyCheck iPhone dataset [3] 2 resolution (Table 4). Starting from our Full model, we disable individual components to evaluate their influence rendering quality and temporal consistency. First, we replace the per- Gaussian RGB coefficient with the original time-varying 4D sphericylindrical harmonic basis as used in 4DGS. While this configuration increases the parameter count, it provides no benefit in rendering quality and in fact decreases PSNR by 1.0 This suggests that the simplified RGB representation is not only more efficient, but also better suited for monocular, unconstrained video.
Additionally, we evaluate the impact of our pro- posed design of Isotropic Gaussian. Typically, 3D Gaussian Splatting methods model the scal- ing and rotation of an 3D Gaussian primitive with anisotropic covariance. However, we adopt fixed identity orientation and scalar for 3D scaling, We find that the reduced flexibility in- troduces stability, resulting in a 1.25 dB gain in PSNR. Finally, we investigate the effect of disabling motion-aware Gaussian strategy, setting the temporal scale uniformly for all Gaussians. This configuration fails to differentiate be- tween static and dynamic regions, leading to motion blur and a substantial reduction in PSNR by 3.4 dB. The results demonstrate that each component of INSTANT4D, including the compact RGB representation, isotropic Gaussian formulation, and motion-aware temporal scaling, plays a crucial role in maintaining visual fidelity and temporal stability.
Component
PSNR
SSIM
Ours (Full)
24.52
0.834
w/o Motion Aware Gaussian
21.11
0.721
w/o Isotropic Gaussian
23.00
0.661
w/o zero-degree SH
23.52
0.755
Table 4: Ablation study on each component's ef- fective performance.
5 Diseussion & Conclusion
Discussion While INSTANT4D achieves state-of-the-an efficiency and reconstruction quality, it is currently limited in its scalability to long-duration video sequences. The visual SLAM component retains depth maps for each frame, leading to linear increase in memory consumption the sequence length grows. This constraint hinders the application of our method to extended captures, such as multi-minute scenes continuous video streams. Addressing this bottleneck requires innovations in hierarchical memory management and online depth-map compression, which we consider promising avenues for future research. Furthermore, handling scenes with highly reflective transparent surfaces remains challenge, as depth estimation becomes less stable under such conditions.
Conclusion We present INSTANT4D, a novel system for fast 4D reconstruction from casual, uncal- ibrated monocular video. Our approach leverages grid pruning, motion-aware Gaussian Splatting, and efficient 4D representation to achieve real-time rendering speed with limited memory over- head. Experiments several benchmarks demonstrate its superior rendering visual quality and

---

computational efficiency compared to existing methods. Our future work will focus on extending our framework video sequences with arbitrary length, improving scalability through hierarchical SL.AM representations, and efficient memory management, while also addressing limitations in highly reflective and low-texture scenes.
10

---

References
[1] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19457-19467, 2024. 2
[2] Zhiwen Fan, Kairun Wen, Wenyan Cong, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. Instantsplat: Sparse-view sfm-freee gaussian splatting seconds. arXiv preprint arXiv:2403.2030% 2024. 6, 8
[3] Hang Gao, Ruilong Li, Shubham Tulsiani. Bryan Russell, and Angjoo Kanazawa, Dynamic novel-view synthesis: A reality check. In NeurlPS, 2022. 2, 6, 7, 8, 9, 16
[4] Richard Hartley. Multiple view geometry in computer vision, volume 665. Cambridge university press, 2003. 3
[5] Yoonwoo Jeong, Junmyeong Lee, Hoseung Choi, and Minsu Cho. Rodygs: Robust dynamic gaussian splatting for casual videos. arXiv preprint arXiv:2412.03077 2024. 2, 6,
[6] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2,00:1333.,,,3...... 2, 4, 5,
[7] Mijeong Kim, Jongwoo Lim, and Bohyung Han. 4d gaussian splatting in the wild with uncertainty-awarc regularization. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 129209- 129226.. Curran Associates, Inc., 2024. 2
[8] Yao-Chih Lee, Zhoutong Zhang, Kevin Blackburn-Matzen, Simon Niklaus, Jianming Zhang, Jia-Bin Huang, and Feng Liu. Fast view synthesis of casual videos with soup-of-planee In European Conference Computer Vision, pages 278-296. Springer, 2024. 6
[9] Jiahui Lei, Yijia Weng. Adam Harley, Leonidas Guibas, and Kostas Daniliddis. Mosca: Dynamic gaussian fusion from casual videos via motion scaffolds. arXiv preprint arXiv:2405.17421,
2024. 2
[10] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pages 71-91. Springer, 2024. 8
[11] Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos. arXiv preprint arXiv:2412.044633 2024. 2, 3, 4.
[12] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13-23, 2023. 2, 6, 8
[13] Saswat Subhajyoti Mallick, Rahul Goel, Bernhard Kerbl, Markus Steinberger, Francisco Vicente Carrasco, and Fernando De La Torre. Taming 3dgs: High-quality radiance fields with limited resources. In SIGGRAPH Asia 2024 Conference Papers, pages 1-11, 2024.
[14] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[15] Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 91)):22 66, 1979.
[16] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin- Brualla. Nerfies: Deformable neural radiance fields. Proceedings of the IEEE/CVF international conference computer vision, pages 5865-5874, 2021. 2
11

---

[17] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron. Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla and Steven M Seitz. Hypemerr: A higher-dimensions representation for topologically varying neural radiance fields. arXiv preprint urXiv:2106. 13228, 2021.
[18] Perazzi, Pont-Tuset, McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Puttern Recognition, 2016. 9
[19] Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10106-10116 2024. 3,4
[20] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2021. 2,8
[21] Haoxi Ran, Vitor Guizilini, and Yue Wang. Towards realistic scene generation with lidar diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14738 14748, June 2024. 2
[22] Haoxi Ran, Jun Liu, and Chengjie Wang. Surface representation for point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18942-18952, June 2022.
[23] Haoxi Ran, Wei Zhuo Jun Liu, and Li Lu. Learning inner- geou- relations point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15477-15487, October 2021. 2
[24] Johannes L Schonberger and Jan-Michael Frahm. Structure-fromt ftoo-motion revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[25] Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wet- zstein, and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos. In SIGGRAPHI Asia 2024 Conference Papers, pages 1-11, 2024. 5
[26] Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On- the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint vidcos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20675-20685 2024. 2
[27] Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 14:16558-16569 2021.
[28] Diwen Wan, Yuxiang Wang, Ruijie Lu, and Gang Zeng. Template- articulated gaussian splatting for real-time reposable dynamic view synthesis. arXiv preprint arXiv:2412.05570, 2024.7
[29] Qianqian Wang. Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. arXiv preprint 2rXiv:2407.13764 2024. 2, 3
[30] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state, arXiv preprint arXiv:2501. 12387, 2025.
3, 8
[31] Qianwei Wang, Bowen Li, Zhanpeng Luo, Yifan Xu, Alexander Gray, Tom Silver, Sebastian Scherer, Katia Sycara, and Yaqi Xie. Unifying deep predicate invention with pre-trained foundation models, 2025. 2
[32] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 20697-20709, 2024. 3
12

---

[33] Zihan Wang, Jeff Tan, Tarasha Khurana, Neehar Peri, and Deva Ramanan. Monofusion: Sparse- view 4d reconstruction via monocular fusion. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2025. 2
[34] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud, Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. Advances in Neural Information Processing Systems, 15:3502-3516, 2022.
[35] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEF/CVF conference on computer vision and paltern recognition, pages 203 203200, 2024. 2, 6,
[36] Jiawei Xu, Zexin Fan, Jian Yang, and Jin Xie. Grid4d: 4d decomposed hash encoding for high-fidelity dynamic gaussian splatting. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 123787-123811 1. Curran Associates, Inc., 2024. 2
[37] Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou. 4k4d: Real-time 4d view synthesis at 4k resolution. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 20029 2024. 3
[38] Zhen Xu, Yinghao Xu, Zhiyuan Yu, Sida Peng, Jiaming Sun, Hujun Bao, and Xiaowei Zhou. Representing long volumetric video with temporal gaussian hierarchy. ACM Transactions on Graphics (TOG), 43(6):1-18, 2024. 3
[39] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances Neural Information Processing Systems, 37:21875- 21911, 2024. 4
[40] Zeyu Yang. Zijie Pan, Xiatian Zhu, Li Zhang, Yu-Gang Jiang, and Philip HS Torr. 4d gaussian splatting: Modeling dynamic scenes with native 4d primitives. arXiv preprint arXiv:2412.20770, 2024.
[41] Zeyu Yang, Hongye Yang. Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310. 10642, 2023. 1, 4,
[42] Ziyi Yang. Xinyu Gao. Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scenc reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20331-20341, 2024. 2, 8
[43] Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from monocular camera. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5336-5345, 2020. 2, 5, 6, 9, 16
[44] Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r. A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.0382. 2024. 3,
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We made our main claim, core method and major vantage in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work in the section 5,
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

For our motion aware,, isotropic Gaussian, we give detailed assumption and proof around the equation 7.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have listed the necessary details to reproduce the experiment. will also release our code if accpeted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The data sets available by [43] and [3]. We will provide sufficient instruc- tions to faithfully reproduce the main experiments.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
(eglli//giiee/Codeeeee
onooonnlooollll icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctpublic//aeccccccccccccccc /gwii/es///////eeeeee /Cone//dmddsbiii wwtteSuubissiions. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: We provided detailed information about the implementation detail in the section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report PSNR, SSIM as metric reflecting rendering quality and we also provide comparison on rendering FPS, memory usage and training time.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We reported detailed information in the runtime and memory as in the sec- tion 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: h//errip///////...////// cc wwcppbcccc/aaaeceoccceccceetettt wtwteetthhhhhieeeeeieeeeeeeeeeee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
We faithfully observe the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We talk about the application of our method in section 5 while there might not direct social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that can make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models generate deepfakes for disinformation. On the other hand, it is not needed to point out that generie algorithm for optimizing neural networks can enable people train models that generate Deepfakes faster.
be used to
The authors should consider possible harms that can arise when the technology is being used as intended and functioning correctly, harms that can arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors can also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet can pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we properly cite and credit these assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will provide details of our code and model.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not666 involve LLMs as any important, original, non-standard components
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy btttps:/nneurps... https/////errss..coooeeeeeeeeee 220 for what should or should not be described.
20


Reviews:
[{"review_id": "RcrMU4lnjP", "reviewer": "Reviewer_K6kc", "summary": "This paper proposes a system for efficiently compressing and accelerating precise reconstruction of scenes and moving objects in dynamic environments. The pipeline first performs initialization by separately optimizing camera intrinsics, extrinsics, and depth, simplifying the optimization function to avoid local minima and the computational burden of numerous parameters. Subsequently, it identifies moving objects to allocate more computational resources to them and employs voxel grid pruning to prevent overfitting. Finally, the paper simplifies the optimization formula for 4D Gaussians and increases the temporal scale of the static background to maintain the consistency and stability of the optimization algorithm. The main contributions of this work lie in the initialization with state-of-the-art techniques, a novel pruning strategy, and redundancy reduction in existing methods.", "questions": "Have you attempted to significantly zoom in and out on the constructed 4DGS map to evaluate whether the pruning method maintains good rendering performance for new views?\n\nFor input views with motion blur or defocus blur, have you applied any special processing?\n\nHave you tried depth prediction methods based on video?", "limitations": "The paper effectively discusses its limitations but does not clearly address the robustness to motion blur or defocus blur.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Paper Strengths:\n\n1. The paper proposes a targeted pruning algorithm that reduces the number of scene Gaussians while maintaining accurate scene reconstruction.\n\n2. The paper identifies redundancies in existing works and simplifies the problem setting to maintain consistent performance while accelerating inference speed and reducing model size.\n\n3. The paper proposes a pipeline for depth and camera intrinsic/extrinsic parameter estimation that leverages state-of-the-art techniques.\n\nMajor Weaknesses:\n1. The running time data directly taken from other papers is not very convincing, as these works may not have been run on the same GPU or CPU.\n\n2. The paper only compares different methods based on the same calibration estimation, without designing performance comparisons under the same initial depth, such as depth optimized via the same multi-view approach. This suggests that performance improvements may stem from advancements in the underlying depth estimation algorithm.\n\n3. There is no comparison between the optimized depth and the actual depth, and first-order optimization may be insufficient for ensuring depth consistency.\n\n4. The method is not evaluated under different depth settings, leaving uncertainty about whether the performance improvements are solely dependent on a specific depth estimation algorithm.\n\nMinor Weaknesses:\n- The visual comparisons on the Dycheck dataset appear to be observed from different viewpoints, with positional deviations. The 4D GS and the proposed method (Ours) share the same viewpoint, but the Ground Truth appears to be slightly shifted to the right.\n\n- The sentence \"Those dynamic Gaussians, will change there position and scaling according to the deviation of timestamp\" contains a serious spelling error, where \"there\" should be \"their.\"", "was_revised": true, "final_justification": "Overall, this is a good paper with solid contributions which merit publication.\nThe authors addressed the reviewers concerns well in the rebuttal. Therefore, I would raise the score to 5.", "rebuttal": "Thank you for your in-depth discussions and constructive suggestions for our paper. Our responses to the weaknesses and questions are listed below.\n\n## **Major Weakness:**\n### **1: Runtime Analysis with Other Work**\nBefore clarification, we show a \"detailed\" Table 1 for the comparison of efficiency:\n\n| Method         | Runtime (h) | Rendering FPS (480p) |  Compute |\n|----------------|-------------|----------------------|----------|\n| *Official Results* |\n| HyperNeRF [1]     | 64          | 0.40              | 4 × TPU v4s |\n| DynamicNeRF [2]   | 74          | 0.05              | 1 × V100 |\n| RoDynRF [3]       | 28          | 0.42              | 1 × V100 |\n| 4DGS [4]          | 1.2         | 43                | 1 × RTX 3090 |\n| ~Casual-FVS [5]~     | 0.25        | 48               | 1 × A100 |\n| *Our implementations* |\n| InstantSplat [6]  | 0.15        | 117              | 1 × A6000 |\n| 4DGS [7]         | 0.16        | 98                | 1 × A6000 |\n| Ours           | **0.02**    | **822**              | 1 × A6000 |\n---\n* For fair comparison, we implement a popular and competitive method InstantSplat [6] and the baseline 4DGS [7] and compare our method with them using *the same server settings*.  \n* The implementation of Casual-FVS [5] has not been open-sourced, so we can hardly report the runtime with the same server settings. We will not compare our method with it in Table 1 for our camera-ready version, but we will keep citing it. \n* Our method performs over 1000x faster than some previous NeRF methods like HyperNeRF [1], DynamicNeRF [2] and RoDynRF [3]. Though these methods used different GPUs from ours, we believe the obvious efficiency superiority of our method can show its competitiveness. \n\n\n### **2: Discussion on Depth Estimation in our Work**\nFor a detailed discussion, we list the updated Table 4 below:\n\n| Component               | PSNR ↑ | SSIM ↑ |\n|------------------------|--------|--------|\n| **Ours (Full)**        | **24.52** | **0.834** |\n||\n| w/o Motion Aware Gaussian  | 21.11  | 0.721  |\n| w/o Isotropic Gaussian     | 23.00  | 0.661  |\n| w/o zero-degree SH                     | 23.52  | 0.755  |\n\n---\nWe acknowledge that the speedup and the performance increase are related to the underlying depth estimation algorithm, while we believe our contributions such as the pruning strategy and simplified 4D Gaussian representation are orthogonal to the specifics of the initial depth estimation. \n\nOur ablation study shows that the isotropic Gaussian formulation regularizes the training and increases the rendering quality by 1.52 dB. To solve the problem that the Gaussians outside of the frustum tend to disappear, we design the motion-aware Gaussian strategy, improving the render result by 3.41 dB PSNR compared with the baseline. Experiments show that our method achieves a 30× speed-up and reduces training time to within two minutes. As a result, our improvement does not merely stem from the depth estimation.\n\n### **3: Comparison between the Optimized Depth and the Actual Depth**\nWe conducted a new experiment on the Dycheck dataset to compare the depth estimated by Unidepth [8] with the ground-truth depth. We will also incorporate the table below in our supplementary material for our camera-ready version.\n\n| Method         | Abs-Rel $\\downarrow$|  Log-RMSE $\\downarrow$ | $\\delta_{1.25}$ $\\uparrow$|\n|----------------|------------------|-------------------|-------------------|\n| Unidepth[8] | 0.254       | 0.303         | 68.5          |\n| Optimized   | 0.137        | 0.254          | 89.6         |\n\nThe comparison shows that the first-order optimization reduces error in Abs-Rel by 46.1% and in Log-RMSE by 16.2%, and increases $\\delta_{1.25}$ accuracy by 30.8%.\n\n### **4: Evaluation under Different Depth Setting**\nThank you for emphasizing the importance of depth estimation. As shown in DepthSplat [9], better depth leads to improved novel view synthesis quality. The current evaluations are done under short real-world video and AI generated videos that are featured with foreground moving objects and static background, which does not involve different depth settings. We will add more discussion about the depth setting.\n\n## **Minor Weakness:**\n### **1 & 2**\nWe have fixed this problem and will do a thorough proofreading. \n\n## **Question:**\n### **1: Zoom-in & Zoom-out Effect**\nWith significant zoom-in, we find that our method tends to display clear boundaries without outlier Gaussians that are frequent in 3D Gaussian Splatting and other dynamic reconstruction work. We do not find unusual visual effects other than this. With significant zoom-in, we observe some voidness between Gaussians. This is not observed in the baseline methods.\n\n### **2: Deblurring for Motion and Defocus**\nMotion blurring and defocus blurring are challenging problems. These problems are not quite obvious when evaluating since we typically have a higher video frame rate compared with the camera and object motion. While current dynamic methods don't focus on this problem, we are open to exploring this issue.\n\n### **3: Application with Video Depth Model?**\nWe agree that better depth estimation will improve the result. However, we find that a video-depth prediction model might not be the best solution in our method. For instance, the NVIDIA dynamic dataset is featured with a 12-camera teleporting setup, which is incompatible with the assumptions in the video depth model.\n\nFinally, we sincerely thank you for your questions. Your insights have indeed helped us further refine our paper.\n\n-----------------------------\n## **Reference**\n\n[1] Keunhong Park, Utkarsh Sinha, Peter Hedman *et al.* *HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields*.\n\n[2] Chen Gao, Ayush Saraf, Johannes Kopf *et al.* *Dynamic View Synthesis from Dynamic Monocular Video*.\n\n[3] Yu-Lun Liu, Chen Gao, Andreas Meuleman *et al.* *Robust Dynamic Radiance Fields*.\n\n[4] Guanjun Wu, Taoran Yi, Jiemin Fang *et al.* *4D Gaussian Splatting for Real-Time Dynamic Scene Rendering*.\n\n[5] Yao-Chih Lee, Zhoutong Zhang, Kevin Blackburn-Matzen *et al.* *Fast View Synthesis of Casual Videos with Soup-of-Planes*.\n\n[6] Zhiwen Fan, Wenyan Cong, Kairun Wen *et al.* *InstantSplat: Sparse-view Gaussian Splatting in Seconds*.\n\n[7] Zeyu Yang, Hongye Yang, Zijie Pan *et al.* *Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting*.\n\n[8] Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis *et al.* *UniDepth: Universal Monocular Metric Depth Estimation*.\n\n[9] Haofei Xu, Songyou Peng, Fangjinhua Wang *et al.* *DepthSplat: Connecting Gaussian Splatting and Depth*."}, {"review_id": "WBEiSXpLoX", "reviewer": "Reviewer_wosD", "summary": "This paper presents a 4D reconstruction method that offers fast training speeds and good reconstruction fidelity.\n\nLeveraging the MegaSAM and Unidepth methods, Instant4D initializes the Gaussian primitives through backprojection and voxel pruning. These initializations are then fed into the optimization process of 4DGS, which incorporates a separate temporal dimension in the properties of the 3D Gaussian primitives.\n\nDuring optimization, Instant4D replaces high-order SCH with direct RGB values, uses isotropic Gaussians, and applies higher temporal scaling factors for static regions. The static regions are inferred from MegaSAM's predictions. These techniques contribute to improved reconstruction quality on the NVIDIA dataset.\n\nExperiments conducted on both the NVIDIA and Dycheck datasets demonstrate that the proposed method not only accelerates the training process but also yields superior reconstruction quality on these two datasets.", "questions": "My main concerns stem from the insufficient experimental validation presented in this paper:\n\n1. This method relies solely on the Visual SLAM calibration approach, which makes it unfair to directly compare with mainstream methods that use COLMAP calibration.\n2. The paper lacks experiments on the Neu3D dataset.\n\nIf the authors can provide more comprehensive experimental results that demonstrate the robust performance of the proposed techniques, I would be inclined to raise my score.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n1. The paper is well-written and easy to follow.\n2. The initialization of Gaussian properties using MegaSAM and the Unidepth method is simple and straightforward.\n3. The use of motion masks generated by MegaSAM helps separate dynamic foreground objects, improving the overall reconstruction quality.\n\n### Weakness:\n1. The aggregation process via the hash map in L151 is unclear and requires further explanation.\n2. The paper omits a comparison with the work \"4D Gaussian Splatting for Real-Time Dynamic Scene Rendering\" [A].\n3. The paper also omits experiments on the datasets proposed in \"Neural 3D Video Synthesis from Multi-view Video\" [B]. This omission weakens the overall validation of the proposed techniques, making their effectiveness less convincing.\n4. Since the method aims to reduce training time, it would be beneficial if the authors could provide a detailed comparison of how the proposed techniques improve training speed relative to the baseline method, 4DGS.\n5. The technical contributions in this paper are somewhat limited and lack significant innovation. From my perspective, the core contribution lies in the Gaussian initialization strategy that leverages existing visual SLAM and depth prediction methods. The voxel pruning strategy does not appear to offer novel insights.\n6. As shown in Table 1, the authors differentiate between methods using COLMAP calibration and Visual SLAM calibration. Based on the performance comparison of the 4DGS method, COLMAP calibration seems to result in better reconstruction performance. This raises the question of whether it is worth sacrificing the accuracy of COLMAP calibration in favor of Visual SLAM calibration. It would be more beneficial if the method could combine the accuracy of COLMAP with the useful predictions from Visual SLAM.\n\n\n#### Reference\nA. Wu G, Yi T, Fang J, et al. 4d gaussian splatting for real-time dynamic scene rendering[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024: 20310-20320.\nB. Li T, Slavcheva M, Zollhoefer M, et al. Neural 3d video synthesis from multi-view video[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 5521-5531.", "was_revised": true, "final_justification": "Many thanks to the authors for their response regarding the main contribution. My concerns have been well addressed, and the rebuttal has clarified the method's contributions—specifically, the integration of deep visual SLAM into 4DGS reconstruction, the pruning of initial point clouds, and the introduction of the isotropic Gaussian formulation. The experimental results on the multi-camera Neu3D dataset further demonstrate the effectiveness of the proposed approach. I suggest that the authors include these details and experimental findings in the final version of the paper.", "rebuttal": "Thank you for sharing your precious advice for our paper. Our discussions of the weaknesses and concerns are listed below.\n\n## **Weakness:**\n### **1: More Details about the Grid Pruning Implementation**\n\nOur grid pruning implementation follows the voxel-based filtering approach in Open3D [1] and PointCloudUtils [2]. Specifically, each point is discretized into voxel coordinates (via integer division by voxel size) and inserted into a hash map using these coordinates as keys within constant time complexity. Points within the same voxel are grouped, averaging their positions and attributes (RGB, covariance) into one representative Gaussian. We will clarify this implementation explicitly in our revision.\n\n\n### **2: Comparison with \"4D Gaussian Splatting for Real-Time Dynamic Scene Rendering\" [A]**\n\nThe comparisons to \"4D Gaussian Splatting\" [A] have been presented in our manuscript, denoted as \"4DGS [31]\" in Table 1 and Table 3. \n\n\n**On the NVIDIA dataset (Table 1):**\n\n| Method     | Calibration  | Runtime ↓ | FPS ↑ (480p) | FPS ↑ (860p) | PSNR ↑ |\n|------------|--------------|-----------|--------------|--------------|--------|\n| 4DGS [A]   | COLMAP       | 1.2 h     | 43           | 29           | 21.45  |\n| **Ours**   | Visual SLAM  | **0.02 h**| **822**      | **676**      | **23.99** |\n\n**On the Dycheck dataset (Table 3):**\n\n| Method      | Apple | Block | Paper | Spin | Teddy | Avg PSNR ↑ | Runtime ↓ (h) | Mem ↓ (GB) |\n|-------------|-------|-------|-------|------|-------|------------|---------------|-------------|\n| 4DGS [A]    | 23.24 | 22.05 | 21.03 | 22.99| 18.89 | 21.64      | 1.2           | 21          |\n| Ours (Lite) | 24.90 | 23.48 | 23.18 | 23.60| 19.96 | 23.02      | **0.03**      | **1.1**     |\n| Ours (Full) | **26.84** | **23.98** | **24.77** | **25.25** | **21.78** | **24.52** | 0.12 | 8 |\n\nWe apologize for the confusion regarding the method naming. For clarity, we will rename the 4DGS [37] as \"Real-time 4DGS\" in the camera-ready version to avoid ambiguity.\n\n### **3: Experiment on the Neu3D dataset.**\nWe conducted experiments on Neu3D below:\n\n| Methods       | Coffee Martini | Spinach | Cut Beef | Flame Steak | Sear Steak | Overall |\n|---------------|----------------|---------|----------|-------------|------------|---------|\n| MixVoxels [7] | 29.36          | 31.61   | 31.30    | 31.21       | 31.43      | 30.98   |\n| NeRFPlayer [8]| 31.53          | 30.56   | 29.35    | 31.93       | 29.12      | 30.50   |\n| HyperReel [9] | 28.37          | 32.30   | 32.92    | 32.20       | 32.57      | 31.67   |\n| 4DGS [10]     | 28.33          | 32.93   | 33.85    | 34.03       | 33.51      | 32.53   |\n| **Ours**      | 27.53          | 33.18   | 31.02    | 33.89       | 28.25      | 30.77   |\n---\nThe primary focus of our work is the reconstruction of scenes from *casual monocular* videos, characterized by a single camera undergoing free movement. The Neu3D [6] dataset, however, features static *multi-camera* setups *without camera movement*. Our method tries to handle a general set of casual videos that does not require synchronized multi-view videos.\nThus, some of our designs like initialization strategy and Gaussian simplification formulation are incompatible with Neu3D's static multi-camera setup [6]. However, the performance of our method is still on-par with the most competitive methods. \n\nWe will also add this evaluation in the supplementary material for our camera-ready version.\n\n### **4: Detailed Comparison on how the Proposed Techniques Improve Training Speed Relative to the Baseline method**\n\nWe do analyze how these proposed techniques improve training speed through the ablation study in Table 2. Following your suggestion, we now refactor this table as shown below for better clarity. The two major techniques related to the training speed are grid pruning and the color design. \n\nIn the table below, the Baseline method shares similar methods with Real-time 4DGS. \"+Pruning\" method denotes that instead of starting from the sparse point cloud calibrated from COLMAP, we apply our initialization strategy without requiring densification. As shown in the table, one benefit is that all Gaussians are present at the beginning and can immediately receive gradient updates. The second change is the trade-off for spherical harmonics function design. In this ablation study, experiments show that, given dense point cloud initialization, direct RGB color is sufficient to reconstruct monocular dynamic scenes. It is also worth noting that a third-order spherical harmonics function requires **48** parameters. In comparison, other parameters e.g. (position, rotation, scaling) take **13** parameters in total.\n\n\nUpdated Ablation Study Table 2:\n| Model        | PSNR ↑ | Runtime (sec) | Memory (MB) | Rendering FPS |\n|--------------|--------|---------------|-------------|---------------|\n| Baseline     | 23.38  | 181           | 10676       | 154           |\n| + Pruning    | 23.83  | 59            | 2806        | 588           |\n| - SH         | **23.99** | **42**     | **832**     | **981**       |\n\n\n- Our final version applies both of these techniques and thus achieves 4x less runtime and 13x less memory requirement, compared with the baseline method 4DGS.\n- We will follow your suggestions, update our Table 2 in the camera ready, and include a detailed, reader-friendly analysis on how the proposed techniques improve training speed relative to the baseline method.\n\n\n### **5: Discussion on the contribution of our work.**\n\nOur motivation is rooted in the recent advances in deep visual SLAM systems, which have significantly improved speed [3], robustness [4] and generalization [5] in geometry recovery from casually captured monocular videos. Traditional 4DGS methods initialize from the sparse point cloud calibrated from COLMAP. From the deep visual SLAM system, we are able to acquire the point cloud sequence. However, we find the redundant gaussians greatly slow the overall training speed if we directly input the point cloud sequence into our optimization pipeline. To solve this problem, we apply a simple yet effective grid pruning method, which reduces the number of Gaussians by 92%. Experiments show that our method achieves a 30× speed-up and our method is the first to reduce optimization time for a casual video to minute-level, facilitating numerous downstream applications.\n\nTo further overcome challenges posed in the monocular setting, we also introduce an isotropic gaussian formulation, regularizing the expression and increasing the rendering quality by **1.52 dB**. To solve the problem that the gaussians outside of the frustum tend to disappear, we design the motion-aware gaussian strategy, improving the render result by **3.41 dB** PSNR compared with the baseline. Experiments show that our method achieves a **30×** speed-up and reduces training time to within two minutes, facilitating numerous downstream applications.\n\n\n### **6: Discussion on the COLMAP calibration and Visual SLAM calibration.**\n\nCOLMAP is designed for *static* scenes from *consistent* videos. However, it commonly fails in *dynamic and casually captured* videos. Visual SLAM systems, in contrast, are more robust to motion blurring [4], fast to optimize [5] or feedforward [4] and avoid manual masking.\n\nWe provide the evaluation with COLMAP as calibration as shown below.\n\n| Method      | Calibration | Runtime (s) | FPS (480p) | PSNR ↑ |\n|-------------|-------------|-------------|------------|-----------|\n| Ours-COLMAP | COLMAP      | 252 s       | 103        | 23.45     |\n| **Ours**    | Visual SLAM | 72 s        | **822**    | **23.99** |\n----\n\n\nWe also considered combining Visual SLAM's prediction with COLMAP's pose estimates. However, such pairing is rarely available in practice, as these systems operate independently and rely on different camera intrinsics, scales, and assumptions. WWe will clarify the grid pruning implementation explicitly in the revised manuscript, and we appreciate the opportunity to explain this more clearly.\n\n\n## **Question:**\n\n### **1. Comparison with COLMAP calibration**\nCOLMAP often outputs highly accurate camera poses, but at the same time requires more than 15 minutes optimization. When evaluating, the common practice is to use the COLMAP camera pose as ground truth. Therefore we argue that it is fair to compare with mainstream methods. It is also worth noting that for the runtime reported for COLMAP methods, we do not include the calibration time, while we include the calibration time in our method.\n\n### **2. Evaluation on the Neu3D Dataset**\nPlease refer to our discussion in weakness 3.\n\n\n-----------------------------\n## **Reference**\n\n[1] Qian‑Yi Zhou, Jaesik Park, and Vladlen Koltun. *Open3D: A Modern Library for 3D Data Processing*.\n\n[2] Francis Williams. *Point Cloud Utils*\n\n[3] Jianing Yang, Alexander Sax, Kevin J. Liang et al. *Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass*\n\n[4] Jianyuan Wang, Minghao Chen, Nikita Karaev et al. *VGGT: Visual Geometry Grounded Transformer*\n\n[5] Zhengqi Li, Richard Tucker, Forrester Cole et al. *MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos*\n\n[6] Tianye Li, Mira Slavcheva, Michael Zollhoefer et al. *Neural 3D Video Synthesis from Multi-view Video*\n\n[7] Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian et al. *Mixed Neural Voxels for Fast Multi-view Video Synthesis*.\n\n[8] Liangchen Song, Anpei Chen, Zhong Li et al. *NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields*.\n\n[9] Benjamin Attal, Jia-Bin Huang, Christian Richardt et al. *HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling*. \n\n[10] Zeyu Yang, Hongye Yang, Zijie Pan *et al.* *Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting*."}, {"review_id": "juAAzfJiKe", "reviewer": "Reviewer_vtZE", "summary": "This paper presents Instant4D, a pipeline for view synthesis from casual monocular videos. Compared to the baseline, the proposed approach makes significant modifications in initialization and covariance matrix parameterization. Specifically, it leverages grid-pruned colored point cloud derived from dense depth map to initialize the position and color of Gaussian primitives, and set different temporal scales for Gaussians belonging to static and dynamic regions. To reduce the number of parameters, the Gaussians are constrained to be axis-aligned, while deprecating view-dependent appearance. Experiment results demonstrate that the proposed pipeline achieves improved performance on multiple monocular nvs benchmarks while substantially increasing efficiency and compactness compared to the baseline.", "questions": "See weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n1.\tThe content is well-structured and easy to follow.\n\n2.\tMost adopted components are reasonable and exhibit clear improvements in ablations.\n\n3.\tThe proposed method is evaluated on multiple monocular video datasets as well as in-the-wild AI generated videos.\n\n**Weaknesses**:\n\n1.\tThe ability for novel view synthesis lacks rigorous evaluation. All provided videos seems rendered from training viewpoints. The modification of train/test split for the quantitative evaluation on DyCheck dataset somewhat trivialize the task. It is recommended to provide comparison with more recent methods focused on monocular dynamic reconstruction, such as MoSca and Shape-of-Motion.\n\n2.\tThe proposed simplified representation with a diagonal covariance matrix can be regarded as a \"4D Gaussian Marble\". But in the 4D case, such simplification eliminates the important capability of single Gaussian primitive for modeling motion stand-alone. Actually, the videos shown in supplement exhibit obvious flickering, it appears to stem from this constraint.\n\n3.\tThe proposed framework lacks substantial technical novelty and essentially a natural integration of several existing modules.\n\n4.\tTypos: L154, 215, and 216 \"NVIIDIA\" ->\"NVIDIA\"; L52 “Datast” -> “Dataset”", "was_revised": true, "final_justification": "During the rebuttal, the authors did not provide any substantial new results to dispel my concerns about its novelty, methodology, and capability of novel view synthesis. In fact, their responses tend to avoid directly response to the issues and instead obscure it with some subtle expression.\nFor instance, in response to the concern that “all provided videos seem rendered from training viewpoints,”, they never explicitly confirm or refuted this point in the initial rebuttal, instead offer an indirect statement “For the visualization, we have addressed this issue by adding camera movement.”\nAnd in the latest response, they try to distract with “outputs in the paper are rendered from novel viewpoints.”, but it irrelevant with the discussed video. I note that the videos in the website have been updated. But this does not invalidate the original critique. The original videos, which not can be found in the supplementary material, are obviously rendered from training camera trajectories. \nA similar pattern also appears in their defense of the methodology.  Their latest response seems to implicitly confirm, via “under the same setting”, that the “Anisotropic“ in setting \"w/ Anisotropic Gaussian\" only describes the spatial dimension. Therefore, repeatedly citing the same unrelated result does not help address my concerns.\nEven taking into account the updated videos and new comparison with suggested baseline provided in the discussion period, I still do not think this work significantly exceeds the acceptance threshold.", "rebuttal": "Thank you for your in-depth discussions and suggestions for our paper\n\n## **Weakness:**\n### **1: Qualitative Evaluation & Comparison with Other Methods**\nFor the visualization, we have addressed this issue by adding camera movement. We promise to update the visualization part on our official website very soon after the reviewer discussion (following the rebuttal rules). For the train/test split, we believe Gaussian Splatting/NeRF was originally designed for interpolation between different viewpoints, therefore these per-scene methods naturally do not support extrapolation with large camera movement. Our train/test splitting could be viewed as temporal interpolation.\n\nFor comparison with other methods, Shape-of-Motion [4] requires an additional tracking model to optimize, and the preprocessed file will reach 150 GB even for a 10-second video. Shape-of-Motion [4] also requires users to segment the mask themselves. Our method is fully automated. MoSca [5] utilizes depth, point track, and optical flow to reconstruct the scene and camera parameters. However, this design significantly increases the training time because of the complexity of the optimization. Due to time constraints, we are unable to present a complete evaluation. In the future, we will continue experiments on that benchmark.\n\n### **2: Covariance Design**\nWe choose to simplify the scaling matrix into one shared shape scale $s_{xyz}$ and one temporal scale $s_t$. We find that the isotropic design regularizes optimization and improves rendering quality as shown in Table 4.\n\n| Component               | PSNR ↑ | SSIM ↑ |\n|------------------------|--------|--------|\n| Motion Aware Gaussian  | 21.11  | 0.721  |\n| Isotropic Gaussian     | 23.00  | 0.661  |\n| SH                     | 23.52  | 0.755  |\n| **Ours (Full)**        | **24.52** | **0.834** |\n\n\nFrom our observations, this design can reduce overfitting and improve rendering quality from novel views. We assume the flickering behavior is caused by the lighting effects and texture changes in the bear's fur. We also tried the anisotropic Gaussians formulation, which does not solve this flickering behavior. Based on the overall trade-off, the simplified design is feasible. We will add this comparison for more visualization.\n\n### **3: Technical Contribution**\nOur motivation is rooted in the recent advances in deep visual SLAM systems, which have significantly improved speed [1], robustness [2], and generalization [3] in geometry recovery from casually captured monocular videos. Traditional 4DGS methods initialize from the sparse point cloud calibrated from COLMAP. Our methods show that in the casual video setting, the training speed will be greatly improved by introducing our initialization strategy.\n\nOur work is, to our knowledge, one of the first to tightly integrate deep visual SLAM with 4D Gaussian Splatting, bridging the gap between geometry-centric SLAM research and appearance-centric neural rendering. We fully leverage the deep visual SLAM tool, achieving fast, robust, and efficient 4D reconstruction in monocular settings. We believe this direction is valuable for the community, as it provides a strong baseline, enables real-time rendering, and could be used as a dataset to train feedforward novel view synthesis models for casual video. We will open-source our code to further facilitate our research community.\n\n### **4: Typo**\nWe have corrected them in our manuscript. We will do a second proofreading of our manuscript.\n\nFinally, we sincerely thank you for your questions. Your insights have indeed helped us further refine our paper.\n\n-----------------------------\n## **Reference:**\n[1] Jianing Yang, Alexander Sax, Kevin J. Liang et al. *Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass*\n\n[2] Jianyuan Wang, Minghao Chen, Nikita Karaev et al. *VGGT: Visual Geometry Grounded Transformer*\n\n[3] Zhengqi Li, Richard Tucker, Forrester Cole et al. *MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos*\n\n[4] Qianqian Wang, Vickie Ye and Hang Gao et al. *Shape of Motion: 4D Reconstruction from a Single Video*\n\n[5] Jiahui Lei, Yijia Weng, Adam Harley et al. *MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds*"}, {"review_id": "48SeNA5Hzb", "reviewer": "Reviewer_EE6C", "summary": "This paper proposes Instant4D, a novel 4D Gaussian Splatting method for a monocular camera NVS. Prior methods require time for optimization of the 4D scene. Instant4D leverages deep visual SLAM to determine the camera pose, utilizes depth information from video to obtain an initial 3D point cloud, and optimizes the 4DGS representation. The proposed grid pruning strategy further reduces the redundancy at the initialization of the 3D point cloud. Proposed method reduces memory usage by 92%, training time by 4 times, and improves FPS from 154 to 981 on the NVIDIA dataset and shows improvement on the DyCheck dataset.", "questions": "While the current evaluation is sufficient, is there a plan to add statistics about the result? In the checklist, it is mentioned that statistical significance is tested; however, only the naive metrics are presented. (error bars, confidence interval, statistical significance test)\n\nHow much and what kind of advantage does this method have compared to online 4DGS methods, such as 3DGStream? Their training time is significantly shorter than the typical 4DGS as well, while they are not evaluated on the same dataset. More comparisons with 4DGS methods focusing on such efficiency would make the evaluation more solid.\n\nHow well does this method perform, given the COLMAP-based camera pose and initialization? Integration of SLAM comes with the camera parameters and the proposed depth-based initialization. Current analysis shows the impact of the 4DGS representation and the pruning method. Since SLAM-assisted initialization is an impactful factor for the method's efficiency, further analysis would be beneficial.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n* Evaluation shows a significant speedup and reduced memory usage compared to the state-of-the-art.\n* Proposed pointcloud initialization looks well-adaptable for different 4DGS methods.\n\nWeaknesses:\n* Overall framework adapts existing methods with relatively straightforward novel techniques.", "was_revised": true, "final_justification": "From my side, concerns are addressed and I keep the original rating.", "rebuttal": "Thank you very much for your recognition and valuable feedback. We will revise this paper based on your suggestions. Please feel free to point them out if any of our explanations are unclear.\n\n## **Weakness:**\n###  **1:Our Pipeline Design**\nOur motivation is rooted in the recent advances in deep visual SLAM systems, which have significantly improved speed [1], robustness [2], and generalization [3] in geometry recovery from casually captured monocular videos. Traditional 4DGS methods initialize from the sparse point cloud calibrated from COLMAP. Our methods show that in the casual video setting, the training speed will be greatly improved by introducing our initialization strategy.\n\nTo best our knowledge, our work is one of the first to tightly integrate deep visual SLAM with 4D Gaussian Splatting, bridging the gap between geometry-centric SLAM research and appearance-centric neural rendering. We fully leverage the deep visual SLAM tool, achieving fast, robust, and efficient 4D reconstruction in monocular settings. We believe this direction is valuable for the community, as it provides a strong baseline, enables real-time rendering, and could be used as a dataset to train feedforward novel view synthesis models for casual video. We will open-source our code to further facilitate our research community.\n\n## **Question:**\n### **1: More Statistics about the Result**\nWe agree that more statistical analysis would strengthen our claims. In the revised manuscript, we will include standard deviation and confidence intervals for the ablation study, and test the statistical significance against baseline methods. We believe these additions will better reflect the consistency and statistical reliability of our improvements.\n\nHere, we provide a detailed ablation study with mean and standard deviation on the Dycheck dataset by repeating for 5 times:\n\n| Component               | PSNR (Mean) ↑ | SSIM (Mean) ↑| PSNR (Std) | SSIM (Std) |\n|------------------------|--------------|---------------|------------|------------|\n| **Ours (Full)**        | 24.46        | 0.84          | 0.27       | 0.06       |\n||\n| w/o Motion Aware Gaussian  | 21.34        | 0.77          | 0.35       | 0.07       |\n| w/o Isotropic Gaussian     | 23.12        | 0.74          | 0.53       | 0.04       |\n| w/o zero-degree SH                     | 23.67        | 0.78          | 0.30       | 0.10       |\n\n### **2: Evaluation Compared with Online 4DGS Methods**\nThank you for providing these useful suggestions. Due to limited time, we are unable to provide the quantitative comparison here. We will properly cite related work and include the comparison with 3DGStream [4] in our final version.\n\nFor qualitative comparison, one advantage of our methods is the initialization strategy. 3DGStream [4] requires a 5000-iteration initialization for the first frame, while in our method all Gaussians will receive gradient updates at the beginning. Besides, our other advantage is the native 4D Gaussian representation, which does not require per-frame neural transformation.\n\n### **3: Evaluation with COLMAP Initialization & Analysis**\nHere is the experiment evaluated on the NVIDIA dataset with COLMAP-based camera pose:\n\n| Method      | Calibration | Runtime (s) | FPS (480p) | PSNR ↑ |\n|-------------|-------------|-------------|------------|---------|\n| Ours-COLMAP | COLMAP      | 252 s       | 103        | 23.45   |\n| **Ours**    | Visual SLAM | 72 s        | **822**    | **23.99** |\n----\n\nNote that for our method initialized from COLMAP, we turn off the motion-aware Gaussian design, enable densification, and set the number of iterations to 5000. For the **Ours** runtime, we include the calibration time with deep visual SLAM. For COLMAP's runtime, we exclude the calibration time, which is around 5 times longer than the current runtime. We will add more analysis on the efficiency of our initialization strategy, with the metrics listed in question 1.\n\nFinally, we want to express our sincere gratitude. Our paper will definitely be better with your suggestions. Thank you!\n\n-----------------------------\n## **Reference:**\n[1] Jianing Yang, Alexander Sax, Kevin J. Liang et al. *Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass*\n\n[2] Jianyuan Wang, Minghao Chen, Nikita Karaev et al. *VGGT: Visual Geometry Grounded Transformer*\n\n[3] Zhengqi Li, Richard Tucker, Forrester Cole et al. *MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos*\n\n[4] Jiakai Sun, Han Jiao, Guangyuan Li et al. *3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos*"}]

Author rebuttal:

Meta-review: This paper got two accept and two borderline accept. One reviewer pointed out more experimental results are expected to further validate the proposed approach. Please include more experimental results in the final camera ready paper.
Decision: Accept (poster)