4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos
Authors: Zhen Xu
Zhengqin Li
Zhao Dong
Xiaowei Zhou
Richard Newcombe
Zhaoyang Lv
Venue: NeurIPS 2025 spotlight ()
Keywords: Image-Based Rendering
Large Reconstruction Model
3DGS
4DGS
Self-Supervised Training
Monocular Reconstruction
Primary area: applications
TL;DR: We propose a scalable 4D dynamic reconstruction model trained only on real-world monocular RGB videos.

Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We proposed a novel density control strategy in training, which enables our 4DGT to handle longer space-time input. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can outperform prior Gaussian-based networks significantly in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos
Zhen Xu Zhengqin Li Zhao Dong' Xiaowei Zhou Richard Newcombe' Zhaoyang Lv!
'Reality Labs Research, Meta 'Zhejiang University
Project page: https: //dgtt/dt..// github.tthbii
Mono Video
Supervision
Network Input
Segners
4DGT
Plicker
Puoleer
DineV2
Segment1
Dynamic
Saussion Reconstruction
Figure We propose scalable 4D dynamic reconstruction model trained only on real-world monocular RGB videos. The feed-forward 4DGS (section 3.1 representation enables us to render the geometry and appearance of the dynamic scene from novel views in real-time. Even without explicit supervision, the model can learn to distinguish dynamic contents from the background and produce realistic optical flows. The figure shows an enlarged set of Gaussians for the purpose of visualization. The embedded rendered videos only play in Adobe Reader or KDE Okular.
Abstract
We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We proposed novel density control strategy in training, which enables our 4DGT to handle longer space-time input and remain efficient rendering at time. Our model processes 64 consecutive posed frames in a rolling-window ion, predicting consistent 4D Gaussians in the scene. Unlike ietimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can outperform prior Gaussian-bast -based networks significantly in real-world videos and achieve on-par accuracy with optimization-basae methods on cross-domain videos.
window fash-
Work done during internship at Meta.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Humans record videos digitize interactions with their surroundings. The ability recover persistent geometry and 4D motion from videos has profound impact on AR/VR, robotics, and content creation. Modeling 4D dynamic interactions from general-purpon purpose videos remains a oong-standing challenge. Prior work relying on multi-view synchronized capture or depth sensing is constrained to specific application domains. Recent progress in monocular dynamic video reconstruction via per-video optimization shows promise, but lacks scalability due to its time-consuming inference.
In this paper, we propose 4D Gaussian Transformer (4DGT), a novel transformer-basee model that reconstructs dynamic scenes from posed monocular videos in a feedforward manner. We assume camera calibration and redegree-offfreedon (6DoF) poses are available from on-device SLAM [9] or offline pipelines [1]. Inspired by recent feedforward reconstruction methods for static 3D scenes 66 721, 4DGT learns reconstruction from data and adopts 4D Gaussian Splatting (4DGS) 555 as a unified representation for both static and dynamic content, differing only in lifespan. This design enables fast 4D reconstruction from short videos in seconds. For longer videos with global pose consistency, 4DGT predicts consistent world-aligned 4DGS using 64-frame rolling windows.
Training 4D representation is challenging in defining appropriate supervision. While dynamic monocular videos are abundant, they lack space-time constraints. Multi-view video datasets 4 22 limited in both quantity and diversity, making them insufficient for training models that generalize in the wild. Prior methods [3] trained on synthetic object-level data suffer from a generalization gap when applied complex real-world dynamics.
To address this, we train 4DGT exclusively on posed monocular videos from public datasets. We use two key strategies to mitigate space-time ambiguity. First, we leverage depth and normal predietions from expert models [40] 64 66 as auxiliary supervision for guiding geometry learning. Second, we regularize predicted Gaussian properties to favor longer lifespans and reduce overfitting specific views. These enable 4DGT to effectively disentangle space-time structure, yielding high-quality geometry, novel view synthesis, and emergent motion properties such as segmentation and flow. Our reconstructions also show hetter metric consistency than the expert models used for supervision.
Scaling transformer to predict dense, pixel-aligned 4DGS presents two main challenges. First, dense pixel-aligned 4DGS predictions are computationally expensive for training and rendering. Inspired by density control in 3DGS 19, we introduce a pruning strategy that removes the redundant pixel- aligned 4DGS and further increases tokens a second training with denser space-time samples. This effectively reduces 80% of Gaussians and enables 16x higher sampling rate with the same compute. Second, as space-time samples increase, the number of tokens grows, and vanilla self- attention scales quadratically. To address this, we propose level-of-detail structure via multi-level spatiotemporal attention, achieving an additional 2x reduction in computational cost.
4DGT is the first transformer-basee method for predicting 4DGS in feedforward manner using only real-world posed monocular videos in training. Extensive evaluations across datasets and domains show that 4DGT achieves comparable reconstruction quality to optimization-basee methods while being three orders of magnitude faster, making it practical for long video reconstruction. Compared to prior methods that only train on synthetic object-level data 43]. , 4DGT generalizes better complex real-world dynamics. Compared to the per-frame prediction pipeline [25). it also exhibits emergent motion properties.
In summary, we make the following technical contributions:
We introduce 4DGT, a novel 4DGS transformer trained on posed monocular videos at scale, which produces consistent 4D video reconstructions in seconds at inference.
We propose a training strategy to densify and prune space-time pixel-aligned Gaussians, reducing 80% of predictions, achieving 16x higher sampling rate during training and 5x speed-up in rendering.
We design a multi-level attention module to efficiently fuse space-time tokens, further reducing training time by half.
Our experiments demonstrate strong scalability of 4DGT across real-world domains using mixed training datasets and can outperform the previous Gaussian network significantly. The performance of 4DGT is on par with optimization-bant methods in aceuracy in cross cross-domain videos recorded by similar devices used in training, while being 3 orders of magnitude faster.

---

2 Related Work
Nonrigid reconstruction. Recovering dynamic content from video has long been holy grail challenge in 3D vision. Early approaches demonstrated promising non-rigid shape reconstruction from RGB-D videos [32 B), but relied heavily on depth input and struggled with complex dynamic scenes. Since the seminal work on Neural Radiance Fields (NeRF) [30], several methods have extended NeRF to 4D using multi-view videos [22] 10] or posed monocular videos [36, 37]. However, 4D NeRFs are slow train and render, limiting their scalability for complex dynamic scenes. Recently, generative priors have shown promise in aiding 4D reconstruction (58), offering strong regularization for shape and motion across space and time. Still, optimizing 4D representations remains Lime-consuming and reconstruction quality depends heavily on the generalizability of priors across different scene domains.
Dynamic Gaussian representations. Since the introduction of 3D Gaussian Splatting [9,, several methods have extended it to 4DGS variants [65]
62 55 8) showing promising dynamic scene reconstruction from multi-view videos with faster training and real-time rendering support. However, optimizing dynamic Gaussians for monocular videos remains challenging. Recently, few works have shown that complex 4D scenes composed of moving Gaussians can be recovered by leveraging depth, segmentation, and tracking priors from 2D expert models [57 53 21 Despite strong performance, these methods involve complex processing, including manual annotation on dynamic regions. It further requires lengthy optimization, limiting its scalability in practical applications.
Large reconstruction models. Transformer- 3D large reconstruction models (LRMs) have shown strong potential for learning high-quality 3D reconstruction from data, at the object level 16 35 60 [24] and static scenes [59] (9, [72 61]. LRMs can generate reconstructions in seconds from a few input views, achieving quality comparable to optimization- neural methods. However, training LRMs requires large-scale multi-view supervision of the same instance, which is readily available in synthetic datasets or static scene captures, but remains scarce for real-world videos.
Some recent efforts explored training transformers to predict time-dependent 3D-GS [43 25 63
using animated synthetic data [43, self-curated real-world internet videos (25] and street-level data 633. making them the closest related works in motivation. In contrast to these methods, which predict time-dependent 3D-GS representations, our 4DGT offers holistic 4D scene representation that captures geometry better and enables motion understanding capabilities lacking in prior approaches. input and only focuses street-level scenes. Compared to training on synthetic data in [3 our real-world training approach generalizes better to real-worke scenes. Compared B-Timer (55). which adopts per-frame Gaussian prediction pipeline, our method produces explicit dynamic Gaussians thus can model explicit motion, showing emergent capabilities like motion segmentation. Compared to Pred. 3D Repr. [41), which adopts tri-plane based implicit representation, our Gaussian model enables fast rendering after the feed-forward reconstructrion.
[63] requires
3 Method
Given a posed monocular video, 4DGT uses transformer to predict set of 4DGS, which can be rendered in real time. We first describe the architecture and dynamic scene representation in section 3.1 To enable efficient training and rendering at scale, we introduce pixel density control strategy and a level-of-detail structure based on spatial attention. Both techniques improve space-time sampling rates under fixed compute budgets. In section regularization strategies designed to resolve space-time ambiguities in monocular videos.
detail our training process and
3.1 Feed-Forward Dynamic Gaussian Prediction
Input encoding, Given a posed monocular video, we extract a set of image frames I with camera calibration P; and timestamp T, denoted as {L; E x P. E R x6 Tr E R xxxx I...N., where P; represents the Plücker coordinates [8] and N is the total number of frames. We convert them into patches. For frame i, the patches are denoted as (L., R
1... HW/p2), (Tai) and ) where p is the patch size.
Feature fusion. We use the pretrained DINOv2 image encoder
33 to extract high-level C-
dimensional features Fij E RC. These are concatenated with the temporal and spatial encoding T..
3

---

Mono Video
Wwgg23 and
Stage A
Supervision
Lape
Ded caple Tigh
Reyeed
Opecity
Hooqan
Uigasshlly
Gecasian
4D Goussian Transformrr
Dynamic Gauziion Regresentation
Before Deneification Pruning
Incressing
Stage
B93 Bell
19tar
183
Sumrnni Sman
Spatial-Tempen Attin,
Predicted Opacity - Histogram-> Pruring
After Densification Prusingg
An overview of our method in training and rendering. 4DGT takes a series of monocular frames with poses as input. During training, we subsample the temporal frames different granularity and use all images in training. We first train 4DGT to predict pixel-aligned Gaussians at coarse resolution in stage one. stage two training, we pruned majority of non-activated Gaussians according to the histograms of per-patch activation channels, and densify the Gaussian prediction by increasing the input token samples both space and time. At inference time, we run the 4DGT network trained after stage two. It can support dense video frames input at high resolution.
Figure 2:
and P..y as well as the input RGB image IJ to form the fused transformer input:
(X.,) F((L,) ① Tú, ① Pi, ① Fi, / 1 1.. .........j............. ... HW/p²)),
(1)
where @ denotes the concatenation operation and JF denotes the all-to-all self-attention transformer
module. In contrast to ViT, input used static LRMs in that it uses only Plücker rays [69 or DINO feature [1], our transformer takes timestamp-awarc Plücker rays with DINO feature together as input, which we found to be beneficial provide the best prediction in view synthesis well as geometry prediction.
Dynamic Gaussians. We use variant of 4DGS 665 55 to unify the various components in dynamic scene predictions. To better represent geometry, we adopt the 2DGS [[5] defined by the center R3, scale R2, opacity E R! and orientation q E R (quaternion) of the Gaussians. Compared to 3D) used in previous work [65] [5], 2DGS yields better geometry predictions. To represent motion, use four temporal attributes, namely the temporal center RR life-span I E R' velocity v E R$, and angular velocity w E R (as angle-axis) for each Gaussian. Given a specific timestamp t, for rendering partieular dynamic Gaussian point g {x, q, o, c, 1, v, w), we first calculate the offset to the opacity, location and orientation the Gaussian point from the temporal attributes [65]. Specifically, the life-span I is used to influence the Gaussian opacity time:
2 log(or)) O1, = =0 -0)
(1/2)²
where Ous is the opacity multiplier at the life-span boundary and or is the standard deviation of the Gaussian distribution in the temporal domain. Intuitively, the Gaussian retains its full opacity its temporal center and fades in Gaussian distribution along the temporal axis. At 1/2 time relative to the temporal center c, the opacity of the point reduced by multiplying small factor Oth, which is set to 0.05 for all experiments. The location and orientation of each Gaussian is adjusted by the velocity and angular velocity to account for the motion:
X1, x (t, c), qr, q o(w (t. -
where @ denotes converting the angle-axis representation to a quaternion. For each 2DGS with Xt, St,, qt, Or, timestamp t,, apply 2DGS rasterizer implemented in [67] to render image.

---

Dynamic Gaussian decoding, Given the pixel-aligned features (Xi,)) we use a transformer to decode the pixel-aligned 4DGS for each frame
((X, ((X 1... N,j = 1... HW/p²)),
(4)
where Dx..sq.o.c.Ivv.u denotes the MLP-based decoder head for producing the full suite of dynamic Gaussian parameters {G.j)-
Our proposed 4DGS can unify the prediction of appearance and geometry properties of both static and dynamic elements. For static scenes, the network learn to predict Gaussians with long-living lifespan 1 + oc, v + 0, w + O. For complex dynamic motions with occlusions, it predicts transient dynamic objects with short-living Gaussians + O.
3.2 Multi-level Pixel & Token Density Control
While pixel-aligned Gaussian has been a standard choice in prior work 669 72]. it has a severe limitation in representing video frames that require dense, long-term sampling capture motion effectively. Naively sampling frames ppatial-temporally would result in two key issues. First, the increasing number of aligned Gaussians degrades optimization and rendering performance, leading to suboptimal training and blurry details in dynamic regions. Second, the growing number of input tokens significantly increases computational cost, resulting in under-trained models.
Two-stage training. We introduce a two-stage approach to address these challenges. First, we train on coarsely sampled low-resolution images from scratch until convergence. In the second stage, inspired by 3DGS [19]. we propose to filter pixcl-aligned Gaussians by pruning low-opacity predictions per patch based on the histograms and increasing the token count predict more Gaussians across space-time. Additionally, we introduce a multi-level spatiotemporal attention mechanism to further reduce the computational cost of self-attentio layers.
Pruning. After the initial training stage at coarse resolution, we compute a histogram of activated Gaussians per patch and observe that only few channels are activated. A similar pattern emerges in other pixel-aligned Gaussian methods [69], motivating us to decode only small set of Gaussians using the activated channels. Formally, for each patch of Gaussian parameters Gi we consider the standard deviation of their opacity values oi [O.,,elk = -1...p²):
m(o.,)
Co. (o.,,) = 1=1
(aai)=-Vvooo vu(o?,) 2(
(5)
where u(o;s) is the mean of the opacity values and ((o..) is the standard deviation. A particular pixel is considered activated if it has a value larger than I unit of the standard deviation:
Oz.j.k > p(or.)) + a(05))
Mijjk 0,
(6)
otherwise.
where k is the index of the pixel in the patch and migi indicates whether the pixel is activated. histogram hJ of all activation mask m {ma,a ....pp for the patch output is:
A
N. HW/p
H =
Σ M
where Maj E NP² is the activation mask for patch (i.j). H NP' is the aggregated histogram of all activation masks, and is the patch size. We select S channel from the histogram h; for the patch output for all patches onward training. This effectively implements an S/p² times reduction in the number of Gaussians for each patch, mimicking the pruning strategy of 3DGS [19]. We provide more in-depth analysis for this histogram- based pruning strategy compared to alternatives with visualizations in the appendix.
The predicted Gaussian number can naturally increase with more space-time token inputs, either in resolution per frame temporal frame numbers. The initially trained model provides a good scaffolding for pixel-aligned Gaussians when we increase the input token number in space and time. In the second stage of training, increase the spatial and temporal resolution by factor
Densification.
5

---

of Rs and Rg. respectively. Combining the densification process and pruning strategy, this would result in R² R,- SSpp² times of the Gaussians compared to the first stage. We select R, R, = 10, and p 14 for all experiments, leading to only 80% of the original number of Gaussians while greatly increasing the sampling rate of space-time by 16 times.
Multi-level spatial-temporal attention. The number of patches participating in the self-attention module JF increases by a factor of R² RR, which will slow down optimization and inference significantly. To mitigate this, we propose a temporal level-of-detail attention mechanism to reduce the computational cost. We propose divide the N input frames into M equal trunks in the highest level. This division limits the attention mechanism in the temporal dimension, but reduces
the computation of calculating total tokens to O O( To balance spatial-temporal samples, construct a temporal level-of-detail structure by alternating the temporal range and spatial resolution, achieving much smaller overhead while maintaining the ability to handle long temporal windows. For each level I, we reduce the spatial resolution by a factor of 2' and increase temporal samples by 2. Empirically, we use level L and M 4, which leads to approximately 2 times reduction in the computational cost.
3.3 Training
Loss and regularization. We train 4DGT using segments of W 128 consecutive frames from the
monocular video and subsample every 8 frames as input, resulting in N 16 input frames. Notably.
for the second stage training where we apply techniques mentioned in section 3.2 and section 3.2
we increase the number of input frames to N = 64. After obtaining all Gaussian parameters { (G..)
from each of the N input frames, we render them to all W 128 images for self-supervision and compute the MSE loss. Additionally, we add the perceptual LPIPS loss Lipips and SSIM loss
[56] Lestm for better perceptual quality.
(1.)) + W
W SSTM I.
Cuse
Lipips
Lusim
(8)
W
where I denotes the input image and I is the rendered image, &b is the pre-trained layers AlexNet
(20 and SSIM the SSIM function [56]. To better regularize the training, we encourage the points
to be static and have a long lifespan using:
N.HW/p sp
|vaj,all NHW
N.HW/D²
N.HWIP ww//2.a L1
C. Σ
Lu
Σ
Σ
NHW
NHW
where Vi.j. is the velocity of the Gaussian point, w wij, the angular velocity of the Gaussian point and jig.k is the life-span of the Gaussian point.
Expert guidance. We observe that training can benefit from leveraging monocular export models in geometry prediction. We extract the depth map D; and normal map N, from all W frames using DepthAnything V2 [64] and NtrmleNormal (66) and them pseudo-supervision signal:
W
D
D
N.
N W
CD
LN= =
(10)
where D and N are the predicted depth and normal map rendered using the 2DGS rasterizer [15 The final loss function for training the feed-forward prediction pipeline is:
== Luuse Alpips Lipips Asssim Lusim Xy Au A1 ADCD ANLN, (11)
where Alpips. Assims Av. Aw, d AD and AN are the weights for the corresponding loss functions. We set Appips 2.0, Assim 0.2, X 1.0, A 1.0. X 1.0, \D 0.1 and AN 0.01 for all experiments. All weights for the regularization losses are warmed up linearly from 0 to their final values during the first 2500 iterations training.
4 Implementation Detail
Architecture. We use a modified ViT architecture 7 for our fusion network F. Specifically, we use 12 layers of all-to-all self-attention with 16 heads, each head having hidden dimension of 96,

---

and the fully connected layers have a 4x wider hidden channel size. Since the Plücker coordinates [8] P and timestamps T already provide the 4D position of each pixel. we do not use additional embedding for the positional information. For the second stage training, where we enable the multi- level spatial-temporal attention module, we copy the weights of the first-stage transformer times and train them independently. The 1-th transformer is responsible for the 1-th level of spatial-temporal attention, with classification token for passing information between different levels. For the MLP decoders D, we use 2 fully connected layers with a hidden dimension of 256 for each channel. Both the transformer modules F and use GELU [33 as the activation function and layer normalization [I] as the normalization function. We also disable the bias parameters for all the layers.
Training & Inference. We implement 4DGT in PyTorch framework [38]. We employ FlashAt- tention V3 [45] and the GSplat Rasterizer [67 for efficient attention and Gaussian optimization respectively. For optimization, we use the Adam optimizer [27] with learning rate of 5e and a weight decay of 0.05. For the second stage training. the learning rate is set to le-5. Additionally, we linearly warm-up the learning rate of each stage the first 2500 steps and then apply the cosine decaying schedule [26] for the remaining steps. During the second strange training, we additionally augment the input and output to the network by varying the aspect ratio and field of view of the images. Specifically, we randomly sample an aspect ratio from the uniform distribution on 1 T. 31
and a field of view ratio on the original image on [30%, 100%]. We train our reconstruction model 100k iterations for the first stage and 30k iterations for the second stage, using a total batch size of 64. With 64 Nvidia H100 GPUs, the first stage training takes roughly 9 days and the second stage training takes roughly 6 days. For all other experiments on inference speed, we use single 80 GB A100 GPU.
5
Experiments
Training Datasets. We use the following real-world monocular videos with high-quality calibrations:
Project Aria datasets with closed-loop trajectories: the EgoExo4D and Aria Everyday Activities (AEA) (28)
12] Nymeria 229 Hot3D 2
Video data with COLMAP camera parameters: Epic-Fields 50
5 and Cop3D 44
Phone videos with ARKit camera poses: ARKitTrack 771.
Evaluation datasets. We use the synthetic rendering provided in ADT 34 datasets, which provides metric ground truth depth. To evaluate cross-domain generalization, we use DyCheck [II (DyC) datasets and the dynamic scene in TUM-SLAM [48] (TUM) evaluate novel view synthesis. We further hold out a test split from EgoExo4D, AEA, and Hot3D, which we refer to as the Aria test set.
Metrics. For appearance evaluation, we compare the PSNR and LPIPS 70 metrics on novel view and time rendering results. For geometry evaluation, we compare against the depth RMSE [64] and normal angle error (66) We additionally provide qualitative comparisons of motion rendered in 2D) as optical flow and motion segmentation. All comparison experiments are conducted on 128-frame subsequences of the monocular videos, with 64 frames used as input and the remaining 64 frames used for testing, with images resized to 504 x 504 resolution or similar pixel number for controlled comparison unless specified otherwise. For the DyCheck [1I] dataset, we additionally compare the rendering results on the provided test view cameras, which show signals on extreme view synthesis. We provide more details about evaluation implementations in the appendix.
Baselines. We consider the following baselines as the most relevant work for evaluation.
L4GM 4
It is the closest prior 4D Gaussian model that generalizes to real-world videos. Different from ours trained using real-world data only, they trained on a synthetic dataset and leveraged additional multi-view diffusion priors from ImageDream [2
2. Static-LRM: We trained static scene LRM following [69 on the same real world data as our 4DGT. We use 2DGS instead 3DGS as the representation that shows more similarity to our approach, except that we further model the dynamic content.
Expert monocular models: We compared each individual expert model we used during training in the same setting, including DepthAnythingV2 (44 aligned with the metric scale of UniDepth
[39 and normals provided by StableNormal [66] For novel view evaluation, we unproject the image using the nearest depth and normal frame.
7

---

L4GM
Static-LRM
SoM
4DGT (Ours)
Ground Truth
Figure From left-to-right, we show the novel space-time view comparisons on ADT [44] EgoExo4D [12], DyCheck [I]] and the DyCheck test-view (rightmost). We render the depth (upper right) and normal (below right) next to each synthesized novel view. For ground truth depth and normal on EgoExo4D and DyCheck, we use predictions from the expert models from the ground truth image for reference. Please refer to the appendix for more visual comparisons.
MonST3R 68) We compare to the dynamic point based representation [68] which highlights the representation difference in using 4DGS. We use ground truth camera poses input to their model using the official implementation and using PyTorch3D
42 for normal estimation.
Shape of Motion (SoM) [53]: We use SoM to represent the top-tier per-scene optimization method reference for best dynamic reconstruction quality. We follow SoM's instructions to manually segment the dynamic part. It requires running expert models as input, including mask, depth, and tracking, which do not use. We include the preprocessing time for time comparisons.
We do not make comparisons with CAT4D 158 BulletTimer (25) and Pred. 3D Repr. 41 since they provide neither the source code nor the pre-trained models.
Comparisons to baselines. Table I and Figure 3 present our comparisons to baselines. Compared to L4GM, which also predicts dynamic Gaussians from trained transformer, 4DGT shows much better generalization across real scenes. We found that static-LRM can provide a strong baseline for static scenes but will fail when dynamic motion is present, while 4DGT can do well in both. validated in Table 2c on dynamic regions. The geometry predicted from 4DGT is more consistent with the world coordinate compared expert models when evaluated metric scale. Compared to the optimization-aast method SoM, 4DGT can offer on-par quality in view synthesis well
further

---

Table 1: Comparisons to baselines. We mark SoM in grey as a reference for optimization- based
methods, and rank the other baselines with ours as comparisons in learning based approaches.
Method
PSNR (Render) T
LPIPS (Render). J
RMSE (Depdh)
Deg.
Recon.
ADT
TUM DyC Aria
Avg
ADT TUM DyC Aria
Avg
TUM Avg
ADT
Time 4
SoM 53
30.30
21.03
16.49
26.69
23.63
0.242
0.337
0.392
0.307
0.320
4.158
2.756
3.434
35.05
60000 mss /
L4GM
33
7.348
9.226
8.770
8.617 8,490
0.688
0.670
0.587
0.698 0.661
2,606
1.698
2.094
63.51
200 /1
MoR 68 66
25.13
$1.00
20.61
23.475
51447
11.32
19:90
=3.35
19.24
0.246
30.01
0.273
15,023
0.429
20.08
0.323 0.318
50.125
2.111
20,562
20.201
0.653
20.543
1.382
25.00
22.59
4500 ms /
13:155
13:137
11:358
22-418
10:002
12:073
16.025
1.:046
10:621
10.071
10:311
Experts
40
23.32 18.64
11533
22.32
18.96
0.299
0.318 0.423
0.236 0.319
2.931
0.919
1.925
26.26
350 ms /f
Ours
28.31
21.02
16.12
27.36
23.20
0.243
0.349
0:408
0.230
0.308
0.934
0.394
0.664
25.92
25 ms /1
Table 2: Ablation study on our method components using ADT and DyCheck (DyC).
(a) Ablation on dynamic Gaussian in stage one training.
(b) Ablation on stage two training.
Method
ADT
PSNR1 DyC
ADT DyC
LPIPS
Avg
RMSE
ADTL
ADT]
Deg.
Method
ADT
PSNR1 DyC
Avg
LPIPS ADT
DyC Avg
RMSE ADT.
ADT.
Dec.
Naive
15,49
13,95 14.72
0.612
0:517 0.564
1,156
42.25
Naive
Out
EgoEXo4D
P2
22.72
15.27
0.229 0.385
1.278
41.11
Random.
25.79
20.38
0.333
0.406
0.250
Static LRM
69
19.29 14.21
16.75
0,399
0.463 0.431
0.830
31.96
+ DD&P
28.79
15:52
22.16
0:242
0:434 0.338
0.722
25.95
Per-frame
41
10:07
12.10
11.08
0.748
0.714 0.731
3.117
61:77
Multi-evel
28.71
15.34
22.03
0.242
0.439 0,341
0,783
27.30
L..D.lew
26.45
15.86
21,15
0:120
0.399 0.284
0:773
21.59
Mix. (Ours)
28.31
16.12
22.22
0.243
0:408 0.326
0.934
25.92
(c) Evaluation on the dynamic foreground.
(d) Motion segmentation results.
Method
PSNR1
LPIPSI
RMSE ADT,
Deg. ADT]
ADT DyC Avg
ADT DyC Avg
Method
w/o Cr
MegaSaM 23]
Ours
Static LRM [9) (masked)
17.30 13.56 16.76
0.059 0.220 0.102
0.613
49.35
mloU x100
9.1243
77.124.0
81.2
Ours (masked)
27.29 14,93 22.86
0.030 0.195 0,075
0.388
33.32
as geometry prediction while being 3 orders of magnitude faster in runtime, which makes it more favorable process long-time videos in practice.
Ablation study in stage one training. In Table 2a for stage one training, we start from a Naive training baseline coarse resolution using the image rendering losses in Eq. 8 trained only using the AEA dataset with only 7 hours of data. After further scaling to using the EgoExo4D dataset ( (300 hours), we find that increasing the scale of the dataset can significantly improve the performance. We also compare to a static LRM [69] and per-frame LRM 43 counterpart in the same setting. We can already see the benefits over the baselines at this stage by large margin. We further include the full training loss in Eq. 9 and Eq. |1 and we can see significant improvements in the quality across all metrics. The regularization terms L,,,, can be be categorized into two groups: (I) Cv...1 regularizes the motion of the dynamic Gaussian predictions. Without this term, although the quantitative results are not greatly affected, the model would fall into the trivial local minima of making every Gaussian transient and dynamic, not correctly modeling the scene's 's static or slow- moving parts. This would result a purely- white motion mask. In table 2d] we evaluate the quality of the motion mask on the ADT dataset [34] of our method without this term. Thanks to the explicit modeling of the motion parameters in our representation, our model can produce comparable motion segmentation against MegaSaM [23), which has explicit flow supervision, while being 200 faster (1.5s v.s. 300s). (2) LN.D provides expert guidance for the geometry of the dynamic Gaussian prediction. These terms can greatly improve the quality of the reconstructed geometry. Please refer to the appendix for visual comparisons.
Ablation study in stage two training. Table, 2b shows the ablation study of key design in stage two training. Starting from a Naive model without density control prune and densify (D&P) Gaussians and the multi-level attention proposed in 3.2 naively scaling up resolution and temporal samples will run out of memory in training. Compared with variants using Random sampled Gaussian from predicted patches and further densifying, using the proposed D&P strategy to decode sparse activated Gaussian will lead to better results while being efficient in training. Adding the proposed multi-level attention can further speed up training with only minor sacrifice in quality, but can speed up training two times faster. Finally, we mixed all the proposed datasets in training at stage two. Compared to model training only using EgoExo4D, mixing datasets improves generalization across domains.
9

---

Table 3: Comparison with more comprehensive upper-bound on the ADT dataset
34
Method
PSNR 1
LPIPS +
RMSE
Degree
Recon. Time +
SoM [53]
30.30 1.64 0.24240:004
4.16 3300
34.0743.61
60,000 ms/f
SoM 8 [53.
28.40=2.00 0.281 20.017
4.18#299
18.46#1177
60,000 ms/f
Ours
28.31=1.01
0.243*0.021
0,93#0446
25.94=1.84
25 ms/f
OurSmanet0
31.98=1.01 0.220
0.85#0.45
19.25
25 150 ms/f
Initializing optimization-based methods with 4DG'T Table Sable presents a quantitative comparison on the ADT dataset [34] with stronger baselines and 4DGT-initialized optimization-based method. SoM- and employs the same normal regularization ours to provide stronger geometry upper-bound. This results in improved normal quality, but our feed-forward 4DGT prediction method still achieves comparable-op superior-resus while being orders of magnitude faster. After finetuning the feed-forward prediction for only 10 seconds (100 iterations, 150 ms per frame, denoted as mur5mmet0:), performance further improves beyond all optimization-basee optimization steps per scene (and sophisticated tracking expert like TAPIR [o]), our finetuned feed-forward prediction achieves a 350x speedup, and pure feed-forward performance (Ours) 2, x faster. More finetuning further improves reconstruction quality.
2DGS-Geometry augments SoM [53] with 2DGS 19]
baselines. While SoM and its 2DGS augmented variant require 30,000
6 Conclusion
We introduced 4DGT, a novel dynamic scene reconstruction method that predicts 4DGS from an input posed video frame in feed-forward manner. The representation power of 4DGT enables to handle general dynamic scenes using 4DGS with varying lifespans, and support it to handle complex dynamics in long videos. Different from prior work that heavily depends multi-view supervision from synthetic datasets, 4DGT is trained only using real-world monocular videos. We demonstrate that 4DGT can generalize well videos recorded from similar devices, and the ability of generalization can improve when mixing datasets for training in scale.
Limitations and future work. We do not claim 4DGT can generalize to all videos in the wild. We assume the availability requirement, the training datasets have been limited to data sources from a few egocentric devices and phone captures. We observe that the quality may degrade in videos recorded by an unseen type of device due to the inaccurate metric scale calibrations. We believe this be significantly improved by further scaling up the method using more diverse datasets recorded by different devices with curated calibrations. Similar to most monocular reconstruction methods, we still observe significant artifacts when viewing Gaussians from extreme view angles, departing far from the input trajectory. Future directions propose better representations to address learn to distill more priors from multi-view expert models, such as generative video models.
reliable calibration to train 4DGT and deploy it for inference. For this
10

---

Acknowledgments and Disclosure of Funding
We thank Aljaz Bozic for the insightful discussions. We thank the Project Aria team for their open- source and dataset contributions. This work was also partially supported by NSFC (No. U24B20154).
References
[1] J. L.. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
[2] P. Banerjee. S. Shkodrani, P. Moulon, S. Hampali, S. Han, F. Zhang, L. Zhang, J. Fountain, E. Miller,
S. Basol, et al. Hot3d: Hand and object tracking 3d from egocentric multi-view videos. arXiv preprint arXiv:2411. 1.1667 2024.
[3] A. Bozic, M. Zollhofer. C. Theobalt, and M. Niessner. Deepdeform: Learning non-rigid rgb-d reconstruc- tion with semi-supervised data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2020.
[4] M. Broxton, J. Flynn, R. Overbeck, D. Frickson, P. Hedman, M. Duvall, J. Dourgarian, J. Busch, M. Whalen, and P. Debevec. Immersive light field video with a layered mesh representation, ACM Transactions on Graphics (TOG), 39(4):66-1, 2020.
[5] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari. E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens datasct, In Proceedings of the European conference computer vision (ECCV), pages 720-736, 2018.
[6] C. Doersch, Y. Yang, M. Vecerik, D. Gokay, A. Gupta, Aytar, Carreira, and A, Zisserman. Tapir: Tracking any point with frrmf initialization and temporal refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10061- 2023.
[7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Hcigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[8] Y. Duan, F. Wei. Q. Dai. Y. He. W. Chen, and B. Chen. 4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
191 J. Engel, K. Somasundaram, M. Goesele, A. Sun, A. Gamino, A. Turner, A. Talattof, A. Yuan, B. Souti, B. Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308. 13561, 2023.
[10] S. Fridovich-Kene G. Meanti, F. R. Warburg, B. Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488, 2023.
[11] Н. Gao, R. Li, Tulsiani. B. Russell, and Kanazawa. Monocular dynamic view synthesis: A reality check. Advances in Neural Information Processing Systems, 35:33768-337888 2022.
[12] K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, al. Ego-exoAd: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEF//VF Conference on Computer Vision and Pattern Recognition, pages 19383- 19400, 2024.
[13] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus), arXiv preprint
2016.
[14] Y. Hong, K. Zhang, Gu, S. Bi, Y. Zhou, D. Liu, Liu, K. Sunkavalli, Bui, and H. Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:231 1.04331,104000,. 2023.
[15] B. Huang, Z. Yu, A. Chen, A. Geiger. and S. Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 conference papers, pages 1-11, 2024.
[16] Y. Jiang, L. Zhang, J. Gao, W. Hu, and Y. Yao. Consistent4d: Consistent 360 ('ddeg dynamic object generation from monocular video. arXiv preprint ar3iv:231 1.02848, 2023.
[17] Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[18] P. Julius. On a new geometry of space. Julius Plückers Gesammelte wissenschaftliche Abhandlungen, op. cit,(24) 1:462-545, 1865.
[19] B. Kerbl, G. Kopanas, T. Leimkühher, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42(4):1-14, 2023.
[20] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information procexsing systems, 25, 2012.
[21] J. Lei, Y. Weng, A. Harley, L. Guibas, and K. Daniilidis. MoSca: Dynamic gaussian fusion from casual videos via 4D motion scaffolds. arXiv preprint 174211, 2024.
11

---

[22] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele. R. Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5521-5531, 2022.
[23] Z. Li, R. Tucker, F. Cole, Q. Wang. L. Jin, V. Ye, A. Kanazawa, A. Holynski, and N. Snavely. Megasam: Ac- curate, fast, and robust structure and motion from casual dynamic videos. arXiv preprint arXiv:2412.0446.. 2024.
[24] Ii, D. Wang, K. Chen, 7. Lv, T. Nguyen-Phuoc M. Lee, J.-B. Huang. I. Xiao, C. Zhang, Y. 7hu, et al. Lirm: Large inverse rendering model for progressive reconstruction of shape, materials and view-dependent radiance fields. arXiv preprint arXiv:2504. 20026, 2025.
[25] H. Liang, J. Ren, A. Mirzaci, A. Torralba, Z. Liu, Gilitschenski, S. Fidler, C. Oztireli, H. Ling, Z. Gojeie, et al. Foed-forward bullet-time reconstruction of dynamic scenes from monocular videos. arXiv preprint arXiv:2412.03526 2024.
[26] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint
arXiv:1608.0396 03988, 2016.
[27] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 4rXiv:1711.0100.
2017.
[28] Z. Lv, N. Charron, P. Moulon, A. Gamino, C. Peng, C. Sweeney, E. Miller, H. Tang, J. Meissner, J. Dong, et al, Aria everyday activities dataset. arXiv preprint arXiv:2402. 13349, 2024.
[29] L. Ma, Y. Ye, F. Hong. V. Guzov, Y. Jiang. R. Postyeni. L. Pesqueira, A. Gamino, V. Baiyya, H. J. Kim, et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In European Conference on Computer Vision, pages 445-465. Springer, 2024.
[30] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[31] R. Murai, E. Dexheimer, and A. J. Davison. MASOR-SLAM: Real-time dense SLAM with 3D reconstruc- tion priors. In CVPR, June 2025.
[32] R. A. Newcomhe, D. Fox, and S. M. Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. CVPR, 2015.
[33] M. Oquab, T. Dareet, T. Moutakanni, H. Vo, M. Szaframice, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint
arXiv:2304.07199. 2023.
[34] X. Pan, N. Charron, Y. Yang, S. Peters, T. Whelan, C. Kong, O. Parkhi, R. Newcombe, and Y. C. Ren. Aria digital twin: A new benchmark dataset for egocentrie 3d machine perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20133-20143, 2023.
[35] Z. Pan, Z. Yang, X. Zhu, and L. Zhang. Efficient4d: Fast dynamic 3d object generation from single-view video. arXiv preprint MrXX::2400.00742. 2024.
[36] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Deformable neural radiance fields. In JCCV, 2021.
Nerfies:
[37] K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Bruulla, and S. M. Scitz. Hypernerf: A preprint arXiv:2 13228, 2021.
representation for topologically varying neural radiance fields. arXiv
[38] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein. L.. Antiga, A. Desmaison, A. Kopf, F. Yang, 7. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In NewrIPS, 2019.
[39] L. Piccinelli, Y.HH. Yang. C. Sakaridis, M. Segu, S. Li, L. Van Gool, and F. Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10106-1011 10116, 2024.
[40] L. Piccinelli, C. Sakaridis, Y.H.. Yang, M. Segu, S. Li. W. Abbeloos, and L. Van Gool. Unidepthv2: Universal monocular metric depth estimation made simpler. arXiv preprins arXiv:2502.20118 2025.
[41] D. Qi, T. Yang, B. Wang, X. Zhang, and W. Zhang. Predicting 3d representations for dynamic scenes. arXiv preprint 1rXiv:2501.16611 2025.
[42] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari. Accelerating 3d deep lcarning with pytorch3d. arXiv:2007.088 0085001,,0001...00 2020.
[43] J. Ren, C. Xie, A. Mirzaei, K. Kreis, Z. Liu, A. Torraba, S. Fidler, S. W. Kim, H. Ling, et al. L4gm: Large
4d gaussian reconstruction model. Advances in Neural Information Processing Systems, 37:56828 37756828-56858 2024.
12

---

[44] J. L. Schonberger and J. Frahm. Structurefrom-mmotion reviiitt In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4104-4113, 2016.
[45] J. Shah, G. Bikshandi, Y. Zhang. V. Thakkar, P. Ramani, and T. Dao. Flashattention-3: Fast and accurate attention with asynchrony and low- oow-precision.. Advances in Neural Information Processing Systems, 37: 68658 0888866885.220 2024.
[46] S. Sinha, R. Shapovalov, J. Reizenstein, Rocco, N. Neverova, A. Vedaldi, and D. Novotny, Common pets in 3d: Dynamic new-view synthesis of real-life deformable categories. CVPR, 2023.
[47] M. Slaveheva, M. Baust, D. Cremers, and S. Ilie. Killingfusion: Non-rigid 3d reconstruction without correspondences. In CVPR, July 2017.
[48] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. A henchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages
573-580. IEEE, 2012.
[49] Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pages 202-419. Springer, 2020.
[50] V. Tschernezki, A. Darkhalil, Z. Zhu, D. Fouhey, Laina, D. Larlus, D. Damen, and A. Vedaldi. Epic fields: Marrying 3d geometry and video understanding, Advances in Neural Information Processing Systems, 36: 26485-26500, 2023.
[51] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] P. Wang and Y. Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.00 222201, 2023.
[53] Q. Wang. V. Ye. H. Gao. J. Austin, Z. Li. and A. Kanazawa. Shape of motion: 4d reconstruction from a single video. arXiv preprint arXiv:2407133764 2024.
[54] Q Wang, Y. Zhang, A. Holynski, A. A. Efros, and A. Kanazawa. Continuous 3d perception model with persistent state, 2025.
[55] Y. Wang, Yang. Z. Xu, Sun, Z. Zhang. Yuan, H. Bao, Peng. and Zhou. Freetimegs: Free gaussians at anytime anywhere for dynamic scene reconstruction. In Proceedings of the IEER/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025.
[56] Z. Wang, C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP. 2004.
[57] G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and W. Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08520 2023.
[58] R. Wu, R. Gao, B. Poole. A. Trevithick, C. Zheng. J. T. Barron, and A. Holynski. Cat4d: Create anything 2024.
in 4d with multi-view video diffusion models. arXiv preprint arXiv:2411.1800
[59] D. Xie, Bi, Z. Shu. K. Zhang, Z. Xu, Y. Zhou, S. Pirk, A. Kaufman, X. Sun, and H. Tan. Lrm-zero: Training large reconstruction models with synthesized data. arXiv preprint (xXxx2440600337) 2024.
[60] Y. Xie, C.-H. Yao, V. Voleti, H. Jiang, and V. Jampani. Sv4d: Dynamic 3d content generation with fulti-frame and multi-view consistency. arXiv preprint arXiv:2407. 17470, 2024.
[61] Y. Xu, Z. Shi, W. Yifan, H. Chen, C. Yang, S. Peng, Y. Shen, and G. Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. In European Conference on Computer Vision, pages 1-20. Springer. 2024.
[62] Z. Xu, Y. Xu, Z. Yu, S. Peng, J. Sun, H. Bao, and X. Zhou. Representing long volumetric video with temporal gaussian hierarchy. ACM Transactions on Graphics, 43(6), November 2024. URL https //zju3dv. github. 1o/1ongvolcap
[63] J. Yang, J. Huang, Y. Chen, Y. Wang. B. Li, Y. You, M. Igl. A. Sharma, P. Karkus, D. Xu, B. Ivanovic, Y. Wang, and M. Pavone. Storm: Spatio-temporal reconstruction model for large-scale outdoor scenes. arXiv preprint (xXxx:5501.066/ 2025.
[64] L. Yang. B. Kang, Z. Huang, Zhao, X. Xu, J. Feng, and H. Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875-21911, 2024.
[65] 7. Yang, H. Yang, 7. Pan, and I.. Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. International Conference on Learning Representations (ICLR). 2024.
[66] C. Ye, L. Qiu, X. Gu, Q. Zuo. Y. Wu, Z. Dong, L. Bo, Y. Xiu, and X. Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 43(6):1 2024.
[67] V. Ye, R. Li. J. Kerr, M. Turkulainen, B. Yi, Z. Pan, O. Seiskari, J. Ye, J. Hu, M. Tancik, et al. gsplat: An open-source library for gaussian splatting. Journal of Machine Learning Research, 26(34):1-17, 2025.
13

---

[68] J. Zhang, C. Herrimann, J. Hur, V. Jampani, T. Darrell, F. Cole, D. Sun, and M.-H. Yang. Monstrr: A simple approach for estimating geometry in the presence of motion. arXiv preprint urXiv:2410.03825. 2024.
[69] K. Zhang. S. Bi, H. Tan, Y. Xiangli, N. Zhao, K. Sunkavalli, and Z. Xu. Gs-Irm Large reconstruction model for 3d gaussian splatting, European Conference Computer Vision, 2024.
[70] R. Zhang, Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[71] H. Zhao, Chen, L. Wang, and Lu. Arkittrack: A new diverse dataset for tracking using mobile rgb-d data. CVPR, 2023.
[72] C. Ziwen, H. Tan, K. Zhang, S. Bi, F. Luan, Y. Hong, I.. Fuxin, and 7. Xu. Long-Irm: Long-sequence large reconstruction model for wide-coverage gaussian splats. arXiv preprint arXiv:2410. 12781, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction has clearly stated the contributions and claims we made. We have articulated the technical challenges we have addressed compared to prior work, and stated the novelty of our methods both results end-to-end, and analyzed via careful ablations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have included the limitations of our current methods in the conclusion section. Overall we observe two main limitations for the current methods. First, we observe the generalization capabilities of our method is related the training data. The real world videos are captured from different devices and have various characteristics. We observed that the generalization capability of the model can improve drastic videos captured by the devices when there are training data collected by the same device used in training. our experiments, we have extensively used data from Project Aria, Iphone and GoPro, which however is not representative enough for general in the wild videos. We see domain gap due this reason and we believe it can be resolved the future to further scale our training to videos with larger varicties. Second, we have further highlighted the limitation our predicted 4DGS when viewed from extreme view angles. This has been a persistent challenge in monocular 4D reconstruction and remain challenge.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
15

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided details of methods in implementation with the hyper- parameters. We have full disclosed our training datasets and training settings. We also provided more details in training and evaluation that cannot be sufficiently described in the main paper due to page limit.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
16

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: No
Justification: We cannot provide the code implementation at the submission time due to legal process. We will intend to release our model upon the acceptance of the paper and legal review.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.cc.
guuides/CodeSubmisstt
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ wttppccccccccccccccccccccccccclt http://gweeeeeeeeeeee http:///eeeeuuiassinoonooooeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided details of training datasets, evaluation datasets, our imple- mentation details and hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
17

---

7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate
information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We provided the full evaluations with error bars in standard deviation our evaluation against baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided details of our compute resources in reproducing the model in implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: h/tp:r://///s///// hc/ppbllc//thiicccccaaalaeeeeeee
Answer: [Yes]
Justification:
We have fully reviewed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
18

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We include a short discussion of social impacts the supplementary materials. As an early research efforts, we have not foresee our method will significantly impact the social topics at this stage.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Although we intend to release model if upon approval, we do not foresce misuse of our model in sensitive topics. Our models only reconstruct 4D from input user provider and will not generate or hallucinate sensitive content. Our training data also has strong compliance to privacy.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
19

---

12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited all the work, including data, model and code implemen- tation that influence this work, All datasets, models and code have public research license to be used safely.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wwwode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
20

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M for what should or should not be described.
21

---

Contents
Introduction
2
Related Work
3
Method
3.1
Ferd-rorward Dynamic Gaussian Prediction
3.2
Multi-level Pixel & Token Density Control
3.3
Training
4
Implementation Detail
Experiments
6
Conclusion
10
Additional Details
23
A.1
Additional Details on the Multi-LLevel Attention Module
23
A.2
Additional Details on the Densification & Pruning of the Dynamic Gaussian
23
A.3
Additional Details Datasets and Baselines
24
Additional Details on the Number Gaussians
B
Additional Results
25
B.1
Qualitative Results of the Ablation Study
25
B.2 Results and Discussion on Pruning Pattern Selection
................................
25
B.3
Supplementary Videos
26
Additional Disucssions
26
Additional Discussions on Expert Models
26
Additional Discussions on the Choice of the Dynamic Gaussian Representation
26
Additional Discussions on Metric Scale Cameras
27
Additional Discussions on Limitations
27
D Social Impact
27
22

---

Mogregated Histogram From Patched
Mattogram Thresholding
Random Selection:
Praner
Fome
Fowe
Uniiforn Selectior:
Activeteed
Figure 4: The predicted opacity map (C RN of the pixel-aligned dynamic Gaussians from 4DGT and the computed histogram (E RPXP) of the activation distribution. The right section shows the difference between histogram thresholding (Ours) and other filtering methods (randomly or uniformly selecting the Gaussians to keep) for reducing the number of Gaussians.
A
Additional Details
A.1 Additional Details on the Multi-Level Attention Module
As mentioned in the Method section of the main paper, aside from the number of Gaussians, another efficiency-limitee factor is the number of tokens in the large number of high-resolution images. In the second stage of training, we increase the spatial and temporal resolution by factor of Rg and RG, respectively. The number of patches participating in the self-attention module F increases by factor of R² Rr which will slow down optimization and inference significantly. To mitigate this, we propose temporal level-of-detail attention mechanism to reduce the computational cost. Specifically, noticing that the computational complexity of the self-attention module is Onn²) (simplified from O(n² + n)) where is the number of tokens [51). We propose to divide the N input frames into M equal trunks in the highest level. This division limits the attention mechanism the temporal dimension, but reduces the computation of calculating n total tokens to 0%))) To balance spatial- temporal samples, we construct a temporal offel-ofdettall structure by alternating the temporal range and spatial resolution, achieving much smaller overhead while maintaining the ability to handle long temporal windows. For each level I, we reduce the spatial resolution by a factor of 2 and increase temporal samples by 2. This results in computational complexity of:
0 nn M
n² M 21--1) 0(
n²
1-11
2n² M M- = O
2
+
-
)
(12)
M 21-1
M
1=0
Empirically, we use level I, = 3 and M = 4, which leads to an approximately 2 times reduction in the computational cost.
A.2 Additional Details on the Densification & Pruning of the Dynamic Gaussian
visualize the predicted opacity map of the pixel-aligned dynamic Gaussians. It shows clear patterns of the activation of the pixel inside each patch, especially for the dynamic regions. Notably, randomly or uniformly selecting the Gaussians keep will lead a significant number of active Gaussians being pruned, effectively removing the ability of the model to model the dynamic parts, while our histogram thresholding scheme can effectively keep the Gaussians that are contributing. These strategies blend the densification and pruning strategies of Gaussian representations [19] and the multi-stage training strategy of ViT models 54 effectively introducing density control scheme for the feed-forward prediction pipeline.
In appendix A.2,
23

---

A.3 Additional Details on Datasets and Baselines
For each dataset used training [28
50] we select 99.15% of the sequences as
training set and hold out the rest. For the datasets used in evaluation:
ADT 34
We select subsequences for validating the reconstruction performance:
ceattmeet__rlasee_ee grrlease_multsskeetton_prrr M1292
4_M1299
Apartment release meal battseemaaaassleeaeeaaee
peartment_releasee teleeese_worr_sheeetonneeee
<rreeettonseql3__M12222 132_1
We use all 6 sequences with 3 views, and follow [33 to apply the covisibilitty mask before computing metries novel views:
DyCheck [I]:
-apple, block, space-out, spin, paper-windmill,, teddy
TUM 488: We seclet subsequences for evaluation:
rgbdd [abb_dddaaee__feiburrg22 _es____e____________ pessson
rrbddggdaaassee dataset_freiburg Set____wlal_____ee halfsshhree
rgbd dataset freiburgg Berriburg3_sittinng.or
EgoExo4D [12] We select 3 subsequences from the hold-out sequences:
rmu_bike01 _2, sfw___oooiig015_1111 uniandes_ bouldering_002
Nymeria [29
We select 2 sequences from the hold-out set:
-202.3607 10233067_ss0_jaet L_____sOjaaaa_ james_jjmnssneee actl 7xwm28
__2r088
AEA
28
We select the loc5_ 1oc5_script5... sctpspttssegg__________ seq7_recl sequence from the hold-out set.
Hot3D
We select the PO020 ff530_f555 sequence from the hold-out set.
The testing sequences from EgoExo4D, AEA, and Hot3D denoted Aria in all comparisons.
Note that we do not make comparisons with CAT4D [58), BulletTimer 25 since they provide neither the source code nor the pre-trained models of writing.
A.4 Additional Details on the Number of Gaussians
The number of dynamic Gaussians predicted by the first stage is pixel-aligned, and can computed (derived from eq.
Ng N H W.
(13)
For resolution of 252 252 and 16 images (half spatial resolution and 1/4x temporal resolution of the second stage), this results in 508,032 (0.5M) Gaussians.
In the second stage, the resolution is increased to 504 x 504 and 64 images. With the proposed patch-based pruning strategy, the number of Gaussians can computed as (derived from eq.
S p2
x N NxHx
(14)
which results in total of 229,440 Gaussians.
Finally, the proposed multi-level spatial attention mechanism introduces two additional downsampled outputs with 1/4x and 1/16x the spatial resolution, respectively. This leads to final Gaussian count of:
(1
1 4
1 _
N 829,440
I
1,088,640 (IM).
(15)
16
for the second stage.
Thanks to our proposed selective activation pruning strategy, the number of Gaussians only increases by 1x while the space-time resolution increases 15x.
24

---

(AEA Trainood
Naive
Stage 1 Ablations
EggoExo4
Out of Memory
Stage 2 Ablations
Stage 1 w/o D&P
Stage w/o D&P
w/ D&P
w/o Mixed Dataset Training
w/ Mixed Dataset Training
Figure 5: Ablation study proposed components.
B Additional Results
B.1
Qualitative Results of the Ablation Study
In fig. we show the ablation study results for the first and second stage training, respectively. As shown in the table and figure, our proposed loss and representation effectively model the dynamic regions and improve the reconstruction quality. Moreover, the proposed density control scheme effectively regularized the number of Gaussians with increased input count and input resolution. greatly improving details and avoiding using too much memory. Adding larger-scale datasets [12 71 helps generalization for both in-domain and out-of-domain datasets.
B.2
Results and Discussion on Pruning Pattern Selection
In eq.
7) after computing H once following the first-stage training, the top S entries are selected to define a shared pruning pattern for the second-stage stage training. This approach effectively shares the same pattern across all patches.
The motivation for this design is twofold:
Empirical activation consistency: Across the yd pixels within each patch, the model consistently favors similar pixels for activation and subsequent use in Gaussian rendering across all patches. This aggregation leads to clear shared activation pattern, visualized
25

---

in appendix A.2 (right),
(right),,, where the predicted opacity maps exhibit this pattern consistently. Quantitative results ( show that the shared pruning pattern achieves on-par or better performance compared to recalculating the pattern for each patch on-the-fly.
Implementation efficiency: Using a shared pruning pattern enables efficient implementation by discarding unused rows in the weight matrix of the final conneced layer of the decoder heads. This is considerably less essource-intensiin regarding both memory usage and computation time, compared dynamically sorting the opacity values for every patch during runtime.
Table 4: Comparison of pruning strategies on the ADT (34) dataset. "On-the-fly" computes unique pruning pattern for each patch. "Shared" (ours) uses single pattern for all patches.
Method
PSNR1
LPIPS
RMSE
Degreel
Speed Overhead
On-the-fly
28.36
0.241
0.78
25.95
Yes
Shared (Ours)
28.79
0.242
0.72
25.84
No
B.3
Supplementary Videos
We attach additional video results in the supplementary video material. The supplementary video is structured as follows:
00:00: 00:0::0000::0::1:000000000000000 Brief introduction to the input & output setting and goal of the paper.
:::00:1500::0000 Reconstruction and novel view rendering results for the depth, normal, optical flow, dynamic mask, and appearance inferred in rolling window fashion over long video.
00:00:45-00:01:000 More qualitative video results from other datasets.
:5:01:00000001:::0000 33: Comparison with baseline methods StaticLRM 69 L4GM and Shape-of-Motion [53].
00:01:35-00:02:000 Ablation study of the proposed components.
C Additional Disucssions
C.1 Additional Discussions on Expert Models
Aside from the expert normal and depth guidance, one natural way to improve the outputs' temporal consistency is to incorporate flow expert's guidance [49] It seems optical flow expert like RAFT [49] can easily be plugged into our pipelin,e but it's non-trivial in practice. There are two reasons we did not adopt such flow model for guidance:
In our preliminary experiments, we discovered that the estimated optical flow exhibits strong inconsistency, often reaching more than 10 pixels in cycle consistency errors. This in turn makes the training of the 4DGT model unstable and leads to NaN values the prediction.
A tracking expert model like TAPIR 66 would produce much more consistent results for guiding the prediction of dynamic Gaussians, as shown by Shape-of-Motinn [53]. However, the computation of such dense all-to-all tracking extremely time-consuming (a few hours for a video data).
clip), making it impractical for our large-scale training setup (1000 hours of
Due to these reasons, we leave the addition of the flow expert model's guidance to improve the temporal consistency future work.
C.2 Additional Discussions on the Choice of the Dynamic Gaussian Representation
The main purpose our choice of dynamic Gaussian representation is to enable seamless integration to feed-forward prediction pipeline, which can be used for self-supervised training on general
26

---

dynamic videos. Compared to per-frame 3DGS, static 3DGS, flow vector field or decomposed motion bases, we found the explicit modeling of the motion terms of dynamic Gaussians (adapted from (I.124, L.19)) to be better fit for this purpose.
FTGS 65 originally proposed in 4DGS 19
Compared a per-frame 3DGS 655 representation (denoted as the *per-frame* variant in the ablation studies of the paper), representation enables the integration of space-time information as a 4D Gaussian with a non-zero life-span, automatically encodes information across multiple frames, making it possible to train the 4DGT model in a self-supervised manner on monocular videos. In the most extreme case with infinite life-span, the repre- sentation is reduced to purely static 3DGS (denoted the * "Static-LRM" baseline in the paper) and would only work on static scenes. Compared to per- per-frame 3DGS and static 3DGS, our representation can freely encode the different levels of motion speed (from 0 to xo) of the dynamic scene. Comparison against the *StaticLRM* baseline and $per-frame 3DGS* variant be found in Table 2(a) of the main paper.
Compared to 3D flow vector field representation, like DynamiGGausiaas [65 our repre- sentation can be easily integrated into the pixel- -ixel-aligned feed-forward prediction pipeline for patch-based vision transformers. However, their flow vector field, which is typically encoded by an MLP, would be much harder to predict in a feed-forward manner. Similar problems exist for the rigid motion representation used in Shape-of-Motion [33, since it's extremely ill-posed to accurately predict their motion bases and coefficients without complicated initialization. In comparison,
representation does not require such careful initialization. In practice, we simply set all v, w to zero, to the timestamp of the corresponding frame, and I large value (50s).
(as used in Pred. 3D Compared to other implicit network methods like NeRF 30
Repr. [41), a Gaussian- based representation would enable much more efficient rendering and training.
C,3 Additional Discussions on Metric Scale Cameras
We empirically find the model works best when trained and inferred with metric-scale cameras due to the ambiguity in depth-scale estimation. Notably, the model doesn't It rely fully accurate scaling to perform well, as shown by experiments on the COP3D and EPIC-FIELDS datasets [46] 50], showing the ability to handle slight deviation from metric-scale calibrations. By introducing such non-metric datasets in training, we force the model to reason from the relative relation of the input cameras and the input images. However, the model would fail to predict coherent results when there exists onder-of-magniute scale error.
C.4 Additional Discussions on Limitations
Due to the ill-posed nature of monocular reconstruction (e.g., limited viewpoint coverage and low frame rates), our method, like other monocular approaches, can still exhibit some blurriness and artifacts, especially during sudden or very fast movements and when visualizing reconstructions from challenging viewpoints. These issues are largely inherent to current monocular reconstruction paradigms. Notably, however, as also pointed out by reviewers, our approach already surpasses prior state-of-the-as baselines in terms of sharpness and artifact reduction, and demonstrates results comparable to optimization-base datasets and introducing additional expert or supervisory signals to the 4DGT model, are promising directions for alleviating these remaining limitations.
methods. Further improvements, such scaling up the training
D Social Impact
As an early-stage research on 4D reconstruction, we do not foresee any immediate social impact from this work. However, it's worth noting that such feed-forward pipeline could be used to synthesize more convincing fake videos by introducing novel views.
27


Reviews:
[{"review_id": "4dJN1rOEGW", "reviewer": "Reviewer_koHi", "summary": "This paper introduces 4D Gaussian Transformer (4DGT), a feed-forward model for fast, high-quality 4D dynamic scene reconstruction from monocular videos. The key technical contribution is a novel density control strategy for training the model. This two-stage approach of pruning and densifying the predicted 4D Gaussians is presented as a crucial innovation, which the authors claim enables the 4DGT model to handle longer space-time input and remain efficient for rendering at runtime. Trained on real-world videos with pseudo-supervision, 4DGT achieves reconstruction quality competitive with slow optimization-based methods while being orders of magnitude faster.", "questions": "See the weakness section.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "###  **Strengths**\n\n* **Addresses a Valuable and Challenging Task:** The paper tackles a highly relevant and difficult problem in computer vision: fast 4D scene reconstruction from a monocular video. This is a valuable goal with strong potential for real-world applications like AR/VR and robotics.\n\n* **Strong Experimental Results:** The method demonstrates strong performance based on experimental evidence. It achieves rendering quality (PSNR, LPIPS) that is competitive with state-of-the-art optimization methods while being significantly faster. It clearly outperforms existing learning-based approaches on real-world datasets, validating the effectiveness of the proposed architecture and training strategy.\n\n* **Clarity and Presentation:** The paper is exceptionally well-written and clearly organized. The authors effectively motivate the problem, and the proposed 4DGT architecture and novel two-stage training pipeline are explained logically and intuitively, making the complex methodology accessible to readers.\n\n### **Weaknesses**\n\nWhile the paper presents a significant step forward, its claims could be further substantiated and its contributions clarified with additional analysis and experiments. My main suggestions for improvement are as follows:\n\n*   **Need for More Detailed Ablation Studies.** The current ablation study on the loss function (Table 2a) removes all regularization and guidance terms at once. The paper could be strengthened by a more granular analysis.\n\n*   **Quantifying Emergent Semantic Properties.** The paper makes a fascinating claim about the model learning \"emergent motion properties such as segmentation and flow\" (line 40). This is a potentially significant finding. To make this claim more robust, it would be highly beneficial to include quantitative and qualitative results.\n\n*   **Temporal Inconsistency.** Dynamic objects in the rendered videos exhibit noticeable flickering, highlighting a lack of temporal consistency. The paper's \"Expert guidance\" strategy successfully leverages pre-trained depth and normal models. A natural question arises: why not also incorporate guidance from a pre-trained optical flow expert? This would be a logical extension of the existing framework and a direct way to supervise the motion model to reduce these flickering artifacts.\n\n*   **Baseline Comparisons:** While the comparison to L4GM and SoM is strong and appropriate, the paper is missing comparisons to other relevant feed-forward methods likes:\n\n    [1] Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos[J]. arXiv preprint arXiv:2412.03526, 2024.\n\n    [2] Predicting 3D representations for Dynamic Scenes[J]. arXiv preprint arXiv:2501.16617, 2025.", "was_revised": true, "final_justification": "My initial review praised the paper's novel approach to 4D reconstruction but raised concerns about the substantiation of its claims, particularly regarding emergent properties, and the need for more thorough ablations and baseline comparisons. After carefully considering the author's rebuttal and the subsequent discussions, I have raised my score and recommend acceptance.", "rebuttal": "We thank the reviewer for their constructive feedback and their recognition of our work. We are especially encouraged that the reviewer finds our method intuitive and well-motivated, our evaluation strong, our results showing and our writing clear and accessible. In the following, we address each of the reviewer's comments and questions in detail.\n\n&nbsp;\n\n**More detailed ablation study.** We thank the reviewer for pointing this out. The regularization terms can be categorized into two groups:\n- $\\mathcal{L}_{\\mathbf{v},\\boldsymbol{\\omega},\\mathbf{l}}$ regularizes the motion of the dynamic Gaussian predictions. Without this term, although the quantitative results are not greatly affected, the model would fall into the trivial local minima of making every Gaussian transient and dynamic, not correctly modeling the scene's static or slow-moving parts. This would result in a purely-white motion mask. In **Table 1**, we evaluate the quality of the motion mask of our method without this term.\n- $\\mathcal{L}_{\\mathbf{N},\\mathbf{D}}$ provides expert guidance for the geometry of the dynamic Gaussian prediction. As shown in Figure 2 of the appendix, these terms can greatly improve the quality of the reconstructed geometry.\n\nWe will add these expanded ablation studies and discussion to the revised paper to make the exposition clearer.\n\n**Table 1**: Motion segmentation results on the ADT dataset [1].\n| Method                | *w/o* $\\mathcal{L}_{\\mathbf{v},\\boldsymbol{\\omega},\\mathbf{l}}$ | MegaSaM [2]     | **Ours**                 |\n| --------------------- | --------------------------------------------------------------- | --------------- | ------------------------ |\n| mIoU * 100 $\\uparrow$ | $9.4^{\\pm4.1}$                                                  | $77.4^{\\pm4.0}$ | $\\mathbf{81.2^{\\pm1.8}}$ |\n\n&nbsp;\n\n**Quantifying emergent properties.** To quantify and evaluate the emergent motion properties of the 4DGT model, we compare the quality of the estimated motion mask, using the ground truth dynamic object segmentation of the ADT dataset [1] in **Table 1**. Thanks to the explicit modeling of the motion parameters in our representation, our model can produce comparable motion segmentation against MegaSaM [2], which has explicit flow supervision, while being 200$\\times$ faster (1.5s v.s. 300s). We will include these quantitative and qualitative results in the revised paper.\n\n&nbsp;\n\n\n**Guidance from optical flow experts and consistency.** \nAlthough our model can handle moderately fast motion of the dynamic scene, it might produce some flickering during sudden movements of the dynamic part due to the ill-posed nature of monocular reconstruction (i.e. limited view and FPS).\nIt's worth noting that, as the reviewer pointed out, our method already produces much better results compared to the current state-of-the-art feed-forward methods and also achieves comparable results against optimization-based methods.\n\nIndeed, one natural way to improve its consistency is to incorporate a flow expert's guidance.\nIt seems an optical flow expert like RAFT [3] can easily be plugged into our pipeline but it's non-trivial in practice. There are two reasons we did not adopt such a flow model for guidance:\n- In our preliminary experiments, we discovered that the estimated optical flow exhibits strong inconsistency, often reaching more than 10 pixels in cycle consistency errors. This in turn makes the training of the 4DGT model unstable and leads to NaN values in the prediction.\n- A tracking expert model like TAPIR [4] would produce much more consistent results for guiding the prediction of dynamic Gaussians, as shown by Shape-of-Motion [5]. However, the computation of such dense all-to-all tracking is extremely time-consuming (a few hours for a 128-frame clip), making it impractical for our large-scale training setup (1000 hours of video data).\n\nDue to these reasons, we leave the addition of the flow expert model's guidance to improve the temporal consistency to future work. We will add these discussions to the revised paper and expand the limitation and future work section for this possible extension.\n\n&nbsp;\n\n**Baseline comparisons.** We thank the reviewer for pointing out these relevant feed-forward baselines. However, since both BTimer [6] and Pred. 3D Repr. [7] are not open-sourced, we do not make direct experimental comparisons with them. Compared to B-Timer [6], which adopts a per-frame Gaussian prediction pipeline, our method produces explicit dynamic Gaussians thus can model explicit motion, showing emergent capabilities like motion segmentation. Compared to Pred. 3D Repr. [7], which adopts a tri-plane based implicit representation, our Gaussian model enables fast rendering after the feed-forward reconstructrion. We will add these discussions to the revised paper. We will also add the missing reference to Pred. 3D Repr. [7] to the related work section, where we discuss the other feed-forward baselines (L95-L100).\n\n&nbsp;\n\n> [1] Aria digital twin: A new benchmark dataset for egocentric 3d machine perception, ICCV 2023  \n> [2] MegaSaM: Accurate, fast, and robust structure and motion from casual dynamic videos, CVPR 2025  \n> [3] RAFT: Recurrent all-pairs field transforms for optical flow, ECCV 2020  \n> [4] TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement, ICCV 2023  \n> [5] Shape of Motion: 4D reconstruction from a single video, arXiv 2024  \n> [6] Feed-forward bullet-time reconstruction of dynamic scenes from monocular videos, arXiv 2024  \n> [7] Predicting 3D representations for dynamic scenes, arXiv 2025"}, {"review_id": "ukBU3ziJCt", "reviewer": "Reviewer_aM6n", "summary": "This paper proposes a feed-forward model for dynamic Gaussian reconstruction from monocular videos. To model the dynamic Gaussians, this paper introduces additional four Gaussian parameters to represent the motion of Gaussians. The overall model is  a Transformer-based architecture to predict dynamic Gaussian parameters from posed images. To make the training more efficient and scalable, this paper proposes a two-stage training strategy and a density control method to remove redundant Gaussians. Experiments are conducted on several dynamic scene datasets and the proposed method achieves state-of-the-art results.", "questions": "The key questions are listed in the Weaknesses. In addition, \n\n- Could the authors report the exact numbers of Gaussians predicted by the model? For example, with 64 and 128 input frames. This would help understand the compactness of the learned 3D representation.\n\n- In L55-57, this paper mentions that \"Compared to prior methods that also train dynamic Gaussian prediction [41, 24], 4DGT generalizes better to complex real-world dynamics and also exhibits emergent motion properties such as motion segmentation.\" However, it seems that I couldn't find a comparison with [24] in the experiments. Could the authors elaborate more on how 4DGT generalizes better than [24]?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The overall idea is simple and the pipeline is clear, and the proposed method achieves state-of-the-art results compared to previous methods.\n- The proposed density control method effectively reduce the number of Gaussians, which also makes the proposed model more scalable.\n- Experiments are extensive, with comparisons to several different baselines and evaluations on several different datasets.\n\n### Weaknesses\n\n- The choice of the dynamic Gaussian representation (L122-137) doesn't seem to be fully justified. This paper introduces four additional Gaussian attributes to model the Gaussian's motion across time. However, it's not fully clear why this paper chooses to use such a representation. In particular, I wonder if this representation is firstly proposed in this paper, or it's based on previous methods (seems that I couldn't find the related references in the paper). I am also curious about how this representation would compare to 3D flow vector based motion representation or the rigid motion representation as proposed in Shape of Motion paper. I think currently the description of the dynamic Gaussians representation lacks a bit of intuition, it would be better if the authors could clarify why this paper chooses to use this representation. I am also not fully clear whether a careful initialization is needed for the four additional Gaussian attributes. \n- I think it would be better if the authors could show how the Gaussians evolve over time, this would also help readers better understand how the proposed motion representation works in practice. \n- It's not very clear if the model relies on metric-scale camera calibrations. According to the training datasets, this paper uses both metric-scale camera annotations and COLMAP camera parameters, which might not be metric scale. However, in the limitation, this paper mentions that \"We observe that the quality may degrade in videos recorded by an unseen type of device due to the inaccurate *metric* scale calibrations\". A deeper discussion on why this happens would be beneficial.\n- I am a bit struggling at understanding which component in the proposed method contributes most to the final performance. From Table 2(a), it seems the loss functions are pretty important. However, a breakdown of different loss terms would be beneficial to gain more insights on the proposed method.", "was_revised": true, "final_justification": "The rebuttal has adequately addressed my questions, and thus I vote for acceptance.", "rebuttal": "We thank the reviewer for their insightful feedback and their recognition of our paper. We are particularly encouraged that the reviewer finds our method effective and our evaluations thorough. In the following, we prepare detailed responses to each of the reviewer's comments and questions.\n\n&nbsp;\n\n\n**Choice of dynamic Gaussian representation.** \nWe thank the reviewer for pointing out this confusion. \nThe main purpose for our choice of dynamic Gaussian representation is to enable seamless integration to a feed-forward prediction pipeline, which can be used for self-supervised training on general dynamic videos. \nCompared to per-frame 3DGS, static 3DGS, flow vector field or decomposed motion bases, we found the explicit modeling of the motion terms of dynamic Gaussians (adapted from FTGS [2], originally proposed in 4DGS [1] (L124, L129)) to be a better fit for this purpose.\n\n- Compared to a per-frame 3DGS [5] representation (denoted as the *per-frame* variant in the ablation studies of the paper), our representation enables the integration of space-time information as a 4D Gaussian with a non-zero life-span automatically encodes information across multiple frames, making it possible to train the 4DGT model in a self-supervised manner on monocular videos. In the most extreme case with infinite life-span, the representation is reduced to a purely static 3DGS (denoted as the *Static-LRM* baseline in the paper) and would only work on static scenes.  \n  Compared to per-frame 3DGS and static 3DGS, our representation can freely encode the different levels of motion speed (from $0$ to $\\infty$) of the dynamic scene. Comparison against the *StaticLRM* baseline and *per-frame 3DGS* variant can be found in Table 2(a) of the main paper.\n- Compared to a 3D flow vector field representation, like DynamicGaussians [6], our representation can be easily integrated into the pixel-aligned feed-forward prediction pipeline for patch-based vision transformers. However, their flow vector field, which is typically encoded by an MLP, would be much harder to predict in a feed-forward manner.  \n  Similar problems exist for the rigid motion representation used in Shape-of-Motion [7], since it's extremely ill-posed to accurately predict their motion bases and coefficients without complicated initialization.  \n  In comparison, our representation does not require such careful initialization. In practice, we simply initialize all $\\mathbf{v}, \\boldsymbol{\\omega}$ to zero, $\\mathbf{t}$ to the timestamp of the corresponding frame and $\\mathbf{l}$ to a large value (50s).\n- Compared to other implicit network-based methods like NeRF [3] (as used in Pred. 3D Repr. [4]), a Gaussian-based representation would enable much more efficient rendering and training.\n\n\nWe will add these discussions to the revised paper to better motivate the design choices of our dynamic Gaussian representation. We will also revise the reference to make the exposition clearer. We will also add the details about initialization and add a visualization of the dynamic Gaussians' movement to show their evolution.\n\n&nbsp;\n\n**Metric-scale cameras.** We thank the reviewer for pointing out this confusion. We empirically find the model works best when trained and inferred with metric-scale cameras due to the ambiguity in depth-scale estimation. Notably, the model doesn't rely on fully accurate scaling to perform well, as shown by experiments on the COP3D and EPIC-FIELDS datasets [8,9], showing the ability to handle slight deviation from metric-scale calibrations. By introducing such non-metric datasets in training, we force the model to reason from the relative relation of the input cameras and the input images. However, the model would fail to predict coherent results when there exists an order-of-magnitude scale error. We will add these discussions to the limitation section and add more visualization on the COP3D and EPIC-FIELDS datasets to make the exposition clearer.\n\n&nbsp;\n\n**Breakdown of different loss terms.** We thank the reviewer for pointing this out. The regularization terms can be categorized into two groups:\n- $\\mathcal{L}_{\\mathbf{v},\\boldsymbol{\\omega},\\mathbf{l}}$ regularizes the motion of the dynamic Gaussian predictions. Without this term, although the quantitative results are not greatly affected, the model would fall into the trivial local minima of making every Gaussian transient and dynamic, not correctly modeling the scene's static or slow-moving parts. This would result in a purely-white motion mask. In **Table 1**, we evaluate the quality of the motion mask of our method without this term.\n- $\\mathcal{L}_{\\mathbf{N},\\mathbf{D}}$ provides expert guidance for the geometry of the dynamic Gaussian prediction. As shown in Figure 2 of the appendix and Table 2(a) of the paper, these terms can greatly improve the quality of the reconstructed geometry.\n\nWe will add these expanded ablation studies and discussion to the revised paper to make the exposition clearer.\n\n**Table 1**: Motion segmentation results on the ADT dataset [10].\n| Method                       | *w/o* $\\mathcal{L}_{\\mathbf{v},\\boldsymbol{\\omega},\\mathbf{l}}$ | **Ours**                 |\n| ---------------------------- | --------------------------------------------------------------- | ------------------------ |\n| mIoU $\\times$ 100 $\\uparrow$ | $9.4^{\\pm4.1}$                                                  | $\\mathbf{81.2^{\\pm1.8}}$ |\n\n&nbsp;\n\n**Exact number of Gaussians.** The number of dynamic Gaussians predicted by the first stage is pixel-aligned. Thus, it can be computed by (derived from Eq. (4) of the paper):\n$$\n  N_{\\mathbf{g}} = N \\times H \\times W.\n$$\nFor the resolution of 252 $\\times$ 252 and 16 images (half spatial resolution and 1/4$\\times$ temporal resolution of the second stage (L193 and L252 of the paper)), this corresponds to 508032 (0.5m) Gaussians.\nFor the second stage, the resolution is increased to 504 $\\times$ 504 and 64 images (L177-179, L195 and L252 of the paper). With the proposed patch-based pruning strategy, the number of Gaussians can be computed by (derived from Eq (4) and L168-169 of the paper):\n$$\n  N_{\\mathbf{g}} = N \\times H \\times W \\times \\frac{S}{p^2},\n$$\nresulting in a total of 829440 Gaussians. \nFinally, the proposed multi-level spatial attention mechanism introduces two additional downsampled outputs, with 1/4$\\times$ and 1/16$\\times$ the spatial resolution, respectively, leading to a final GS count of 829440 * (1 + 1/4 + 1/16) = 1088640 (1m) for the second stage.  \nThanks to our proposed selected activation pruning strategy, the number of Gaussians only increases by 1$\\times$ while the space-time resolution increases 15$\\times$.\nWe will add these analyses to the revised paper to illustrate the compactness of the proposed model.\n\n&nbsp;\n\n**Comparison with BTimer (cited as [24]).** We thank the reviewer for pointing out this confusion. Since BTimer [11] is not open-sourced, we do not make direct experimental comparisons with it. Compared to BTimer [11], which adopts a per-frame Gaussian prediction pipeline for the bullet-time frame only, our method produces explicit dynamic Gaussians thus can model explicit motion, showing emergent capabilities like motion segmentation. We will update the description in the introduction section (L55-57) to fix the exposition issue:\n- Revised description: \"compared to prior dynamic Gaussian methods that only train on synthetic object-level data [41], 4DGT generalizes better to complex real-world dynamics. Compared to the per-frame prediction pipeline [24], 4DGT also exhibits emergent motion properties such as motion segmentation.\"\n\n&nbsp;\n\n> [1] Real-time photorealistic dynamic scene representation and rendering with 4D gaussian splatting, ICLR 2024  \n> [2] FreeTimeGS: Free gaussian primitives at anytime and anywhere for dynamic scene reconstruction, CVPR 2025  \n> [3] Nerf: Representing scenes as neural radiance fields for view synthesis, ECCV 2020  \n> [4] Predicting 3D representations for dynamic scenes, arXiv 2025  \n> [5] 3D Gaussian splatting for real-time radiance field rendering, SIGGRAPH 2023  \n> [6] 4D gaussian splatting for real-time dynamic scene rendering, CVPR 2024  \n> [7] Shape of Motion: 4D reconstruction from a single video, arXiv 2024  \n> [8] Common pets in 3d: Dynamic new-view synthesis of real-life deformable categories, CVPR 2023  \n> [9] Epic Fields: Marrying 3d geometry and video understanding, NeurIPS 2023  \n> [10] Aria digital twin: A new benchmark dataset for egocentric 3d machine perception, ICCV 2023  \n> [11] Feed-forward bullet-time reconstruction of dynamic scenes from monocular videos, arXiv 2024"}, {"review_id": "olniAFEpha", "reviewer": "Reviewer_8Pg8", "summary": "This paper proposes a scalable 4D dynamic reconstruction model that leverages 4D Gaussian Splatting (4DGS) to represent and render dynamic scenes. The model can explicitly disentangle dynamic content from the static background and generate realistic optical flow. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art approaches across multiple datasets.", "questions": "- Although the method surpasses existing state-of-the-art techniques, there remains a gap between the presented results and practical deployment. So, could the outputs of this method serve as a strong initialization for optimization-based reconstruction approaches?\n- Some results exhibit noticeable over-smoothing, such as the third column in Figure 3 where the hand appears less detailed compared to SoM. Could the authors provide an explanation for this phenomenon?\n- Would it be possible to include a comparison with optimization-based methods? Such a comparison would help readers better understand the relative strengths and limitations of the proposed approach.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- The proposed feedforward framework enables fast and scalable dynamic scene reconstruction.\n- The method achieves superior performance compared to existing state-of-the-art techniques on various benchmark datasets.\n- The experimental results are comprehensive and compelling, providing strong empirical support for the method's effectiveness.\n\nWeaknesses:\n\n- The reconstruction of dynamic objects still exhibits certain artifacts; slight camera motion shows the geometric inconsistencies.\n- Some reconstructed results appear overly smoothed or blurred, potentially compromising fine details.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed the majority of my concerns. I will maintain my original score.", "rebuttal": "We thank the reviewer for their constructive feedback, insightful questions and their recognition of our contribution. We are deeply motivated that the reviewer finds our performance superior, our validation thorough and the proposed method effective. In the following, we address each of the reviewer's comments and questions in detail.\n\n&nbsp;\n\n**Initializing optimization-based method for improved quality.** We thank the reviewer for the suggestion. In **Table 1**, we present the results of using the output of the 4DGT model as an initialization for a optimization-based baseline (denoted as *$\\mathit{Ours}_{\\mathit{tune10s}}$*), i.e. finetuning our proposed dynamic Gaussian representation on the input video for only 10 seconds (100 iterations), armortizing to 150ms per frame. We can already achieve better results compared to the optimization-based SoM [1], which requires 30000 iterations of iterative update, a complex tracking expert model (TAPIR [4]) and sophisticated densification strategies (3DGS [2]). Even with the finetuned model, we still achieve a 350$\\times$ speedup against SoM [1]. Note that further finetuning will yield even better reconstruction results. We will add these experiments to the revised paper.\n\n**Table 1**: Quantitative results for initialization on the ADT dataset [3], compared to the optimization-based SoM [1]. *$\\mathbf{Ours}_{\\mathbf{tune10s}}$* denotes finetuning the output of the 4DGT model for 10 seconds (100 iterations).\n\n| Method                                 | PSNR $\\uparrow$              | LPIPS $\\downarrow$            | RMSE $\\downarrow$             | Degree $\\downarrow$          | Recon. Time $\\downarrow$     |\n| -------------------------------------- | ---------------------------- | ----------------------------- | ----------------------------- | ---------------------------- | ---------------------------- |\n| SoM [1]                                | $30.30 ^ {\\pm1.64}$          | $0.242 ^ {\\pm0.024}$          | $4.16 ^ {\\pm3.00}$            | $34.07 ^ {\\pm3.61}$          | $60000$ ms/f                 |\n| **$\\mathbf{Ours}_{\\mathbf{tune10s}}$** | $\\mathbf{31.98 ^ {\\pm1.01}}$ | $\\mathbf{0.220 ^ {\\pm0.012}}$ | $\\mathbf{0.850 ^ {\\pm0.452}}$ | $\\mathbf{19.25 ^ {\\pm2.65}}$ | **$\\mathbf{25 + 150}$ ms/f** |\n\n&nbsp;\n\n**Comparison with SoM [1] (comparison with optimization-based method).** SoM [1] is a state-of-the-art optimization-based method for dynamic scene reconstruction from monocular videos. It performs per-scene optimization of a Gaussian-based dynamic motion representation with human-labeled motion masks and costly expert models regularization, especially dense all-frame-to-all-frame tracking. Our method achieves comparable results against SoM [1] while being 2400 $\\times$ faster. Since SoM [1] performs a lengthy iterative optimization, it might produce sharper details on some areas of the video. Note that tuning the output of the 4DGT model for only 10s (100 iterations, 150 ms per frame) can already produce better results than SoM (30000 iterations), as shown in **Table 1**. We will expand the discussion in the revised paper to make the exposition clearer.\n\n&nbsp;\n\n**Quality.** Similar to other monocular reconstruction methods, we still observe certain artifacts when viewing the reconstruction results from certain angles (mentioned in the limitation section). We leave it as future work to incorporate larger-scale datasets and more expert guidance for the 4DGT model to further mitigate this. It's worth noting that, as the reviewer pointed out, our method already produces much better results compared to the current state-of-the-art and also achieves comparable results against optimization-based methods. We will add these discussions to the revised paper.\n\n&nbsp;\n\n> [1] Shape of Motion: 4D reconstruction from a single video, arXiv 2024  \n> [2] 3D Gaussian splatting for real-time radiance field rendering, SIGGRAPH 2023  \n> [3] Aria digital twin: A new benchmark dataset for egocentric 3d machine perception, ICCV 2023  \n> [4] TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement, ICCV 2023"}, {"review_id": "2NXuld0Uwp", "reviewer": "Reviewer_YKwU", "summary": "This paper introduces 4DGT, a large-scale 4D reconstruction model predicting 4D Gaussian from real-world monocular videos. Comparing to optimization-based method, this feed-forward model significantly reduces the inference time, while achieving great reconstruction quality. It proposes several techniques to facilitate training, such two stage training, pixel-level Gaussian pruning, multi-level attentions. It outperforms other feed-forward reconstruction models with a large margin.", "questions": "1. Eq(7) seems problematic, the subscript $i$ and $j$ still exist after summing up. It should be $h_k$.\n2. Continue from the previous question, it is unclear how $S$ is selected from $h_k$. And since $k \\in {1, 2,..., p^2}$ and  $h_k$ is patch-level independent, does it mean you prune with the same pattern for all patches? Authors should discuss the advantage and disadvantage of this design choice.\n3. How this model performs in outdoor scenes, especially with infinitely far-away sky?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The first feed-forward 4DGS reconstruction model learning from posed monocular videos. Since the input data is easy to acquire, It shows great scale-up ability.\n2. ​Density control strategy​ (pruning & densification) to improve efficiency and sampling rates. Comparing to GS-LRM, this model is more efficient.\n3. The results are strong, the rendered appearance, normal and depth are much better than the baseline.\n\nWeaknesses\n1. The model still have difficulty modeling fine-grained motion, e.g. the finger is blur in demo videos.\n2. SoM with 3DGS leads to bad geometry quality, especially normal. I would expect a more comprehensive upper bound (e.g. optimization-based dynamic 2DGS) with the same geometry supervision as 4DGT.", "was_revised": true, "final_justification": "Authors have addressed my concern. I am happy with accepting this paper.", "rebuttal": "We thank the reviewer for their insightful and valuable comments and their recognition of our work. We are greatly motivated that the reviewer finds our method performing well both in terms of quality and efficiency and acknowledges the scalability of our pipeline. We provide detailed responses to each of the reviewer's comments and questions in the following.\n\n&nbsp;\n\n**More comprehensive upper-bound.** In **Table 1**, we present the improved upper-bound optimization-based method. *SoM-2DGS-Geometry [1]* augments SoM [1] with 2DGS [2] and uses the same normal regularization as ours. With the stronger geometry supervision, we do observe an increase in the normal quality in this improved upper-bound method. It's worth noting that our feed-forward prediction method still achieves comparable results against *SoM-2DGS-Geometry [1]*, while being 3 orders of magnitude faster. After finetuning the feed-forward prediction for only 10 seconds (100 iterations, 150 ms per frame), we can achieve even better results. We will add these experiments to the revised paper.\n\n**Table 1**: Comparison with a more comprehensive upper-bound on the ADT [3] dataset. *SoM-2DGS-Geometry [1]* denotes switching to 2DGS [2] for SoM [1] and adding the same geometry regularization as 4DGT. *$\\mathit{Ours}_{\\mathit{tune10s}}$* denotes finetuning the output of the 4DGT model for 10 seconds (100 iterations).\n| Method                             | PSNR$\\uparrow$              | LPIPS$\\downarrow$            | RMSE$\\downarrow$           | Degree$\\downarrow$          | Recon. Time$\\downarrow$ |\n| ---------------------------------- | --------------------------- | ---------------------------- | -------------------------- | --------------------------- | ----------------------- |\n| SoM [1]                            | $30.30 ^ {\\pm1.64}$         | $0.242 ^ {\\pm0.024}$         | $4.16^{\\pm3.00}$           | $34.07 ^ {\\pm3.61}$         | $60000$ ms/f            |\n| SoM-2DGS-Geometry [1]              | $28.40^{\\pm 2.60}$          | $0.281^{\\pm 0.027}$          | $4.18^{\\pm 2.98}$          | $\\mathbf{18.46^{\\pm 1.76}}$ | $60000$ ms/f            |\n| **Ours**                           | $28.31^{\\pm 1.51}$          | $0.243^{\\pm 0.021}$          | $0.93^{\\pm 0.46}$          | $25.94^{\\pm 1.84}$          | **$\\mathbf{25}$ ms/f**  |\n| $\\mathbf{Ours}_{\\mathbf{tune10s}}$ | $\\mathbf{31.98^{\\pm 1.01}}$ | $\\mathbf{0.220^{\\pm 0.012}}$ | $\\mathbf{0.85^{\\pm 0.45}}$ | $19.25^{\\pm 2.65}$          | $25 + 150$ ms/f         |\n\n&nbsp;\n\n**Pruning pattern choice and Eq. (7).** We thank the reviewer for pointing out the issue in the subscripts of Eq. (7). Indeed, after performing summation across all patches, the subscript $i,j$ should be discarded. We provide a corrected version of Eq. (7) as follows:\n\n$$\n  \\mathbf{H} = \\sum_{i=1,j=1}^{N,HW/p^2} \\mathbf{M}_{i,j},\n$$\n\nwhere $\\mathbf{M}_{i,j} \\in \\mathbb{N}^{p^2}$ is the activation mask for patch $i,j$, $\\mathbf{H} \\in \\mathbb{N}^{p^2}$ is the histogram of all activation masks and $p$ is the patch size.\n\nAfter computing the histogram $\\mathbf{H} \\in \\mathbb{N}^{p^2}$ once after the first stage training, we select its top $S$ values to serve as the pruning pattern for the second stage training. This indeed indicates that the pruning pattern is shared across all patches. The reason behind this design choice is twofold:\n- We empirically find that in the $p^2$ pixels of a patch, the model tends to favor similar pixels to be activated and later used in the Gaussian rendering process across all patches, as shown in Figure 1 of the appendix (zoom in on the predicted opacity for a better visualization). After aggregating the histogram across all patches, a clear pattern of activation emerges, as shown on the right side of Figure 1 of the appendix. Moreover, in **Table 2**, we conduct quantitative experiments to compare against computing the pruning pattern for each patch separately on the fly and find that using a shared pattern achieves on-par performance.\n- With a shared pruning pattern, the pruning operator can be efficiently implemented as simply discarding the unused rows in the weight matrix of the last fully-connected layer of the decoder heads, which is much less demanding both in terms of memory usage and computation time than re-sorting the opacity values on the fly for each patch.\n\nWe will add these expanded discussions and results to the revised paper and fix the exposition issue of Eq. (7) to make this process clearer.\n\n**Table 2**: Quantitative comparison between on-the-fly re-computation of pruning pattern for each patch and our shared pattern approach on the ADT [3] dataset. *Shared (Ours: Table 2(b) +D&P)* denotes our shared pattern pruning approach, corresponding to the *+D&P* row of Table 2(b) of the main paper.\n| Method                             | PSNR$\\uparrow$   | LPIPS$\\downarrow$ | RMSE$\\downarrow$ | Degree$\\downarrow$ | Mem. & Speed Overhead $\\downarrow$ |\n| ---------------------------------- | ---------------- | ----------------- | ---------------- | ------------------ | ---------------------------------- |\n| On the fly Recalc.                 | $28.36$          | $\\mathbf{0.241}$  | $0.78$           | $25.95$            | $\\mathrm{Yes}$                     |\n| **Shared (Ours: Table 2(b) +D&P)** | $\\mathbf{28.79}$ | $0.242$           | $\\mathbf{0.72}$  | $\\mathbf{25.84}$   | $\\mathbf{No}$                      |\n\n&nbsp;\n\n**Ourdoor scenes.** Our method does not rely on the assumption of the scene being indoor since we train on such outdoor data from the Ego-Exo4D dataset [4]. That said, since we use the same depth activation (Sigmoid with near-far planes) as GS-LRM [5], it currently cannot handle infinite depth and we leave it as future work to incorporate a disparity-based depth prediction scheme for handling such cases. We will add these visualizations and discussions in the revised paper.\n\n&nbsp;\n\n**Quality.** Due to the ill-posed nature of the monocular reconstruction problem (i.e. limited view and FPS), we might still observe some blurriness for sudden and very fast movements, similar to other monocular reconstruction methods. We leave it as future work to incorporate larger-scale datasets and more expert guidance for the 4DGT model to further mitigate this. We will add these discussions to the revised paper. As the reviewer noted, our method already achieves much sharper results compared to the state-of-the-art baselines.\n\n&nbsp;\n\n> [1] Shape of Motion: 4D reconstruction from a single video, arXiv 2024  \n> [2] 2d gaussian splatting for geometrically accurate radiance fields, SIGGRAPH 2024  \n> [3] Aria digital twin: A new benchmark dataset for egocentric 3d machine perception, ICCV 2023  \n> [4] Ego-Exo4D: understanding skilled human activity from first- and third-person perspectives, CVPR 2024  \n> [5] Gs-lrm: Large reconstruction model for 3d gaussian splatting, ECCV 2024"}]

Author rebuttal:

Meta-review: This paper proposes a novel feedfoward approach for reconstructing 4D Gaussians. Four expert reviewers provided final ratings of 4 x accept. They appreciated the work for addressing a novel task, its strong results and for providing extensive and thorough experimental evaluation. The reviewers' questions were addressed during the author-reviewers discussion phase. The AC concurs with the reviewers' consensus and recommends acceptance. Congratulations! The authors should make the changes to the final manuscript that they have promised.
Decision: Accept (spotlight)