Title: 4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image-Based Rendering

Large Reconstruction Model

3DGS

4DGS

Self-Supervised Training

Monocular Reconstruction
Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We proposed a novel density control strategy in training, which enables our 4DGT to handle longer space-time input. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can outperform prior Gaussian-based networks significantly in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.


Peer reviews:
comments: SUMMARY:
This paper introduces 4DGT, a large-scale 4D reconstruction model predicting 4D Gaussian from real-world monocular videos. Comparing to optimization-based method, this feed-forward model significantly reduces the inference time, while achieving great reconstruction quality. It proposes several techniques to facilitate training, such two stage training, pixel-level Gaussian pruning, multi-level attentions. It outperforms other feed-forward reconstruction models with a large margin.

STRENGTHS AND WEAKNESSES:
Strengths
1. The first feed-forward 4DGS reconstruction model learning from posed monocular videos. Since the input data is easy to acquire, It shows great scale-up ability.
2. ​Density control strategy​ (pruning & densification) to improve efficiency and sampling rates. Comparing to GS-LRM, this model is more efficient.
3. The results are strong, the rendered appearance, normal and depth are much better than the baseline.

Weaknesses
1. The model still have difficulty modeling fine-grained motion, e.g. the finger is blur in demo videos.
2. SoM with 3DGS leads to bad geometry quality, especially normal. I would expect a more comprehensive upper bound (e.g. optimization-based dynamic 2DGS) with the same geometry supervision as 4DGT.

QUESTIONS:
1. Eq(7) seems problematic, the subscript $i$ and $j$ still exist after summing up. It should be $h_k$.
2. Continue from the previous question, it is unclear how $S$ is selected from $h_k$. And since $k \in {1, 2,..., p^2}$ and  $h_k$ is patch-level independent, does it mean you prune with the same pattern for all patches? Authors should discuss the advantage and disadvantage of this design choice.
3. How this model performs in outdoor scenes, especially with infinitely far-away sky?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-11

comments: SUMMARY:
This paper proposes a scalable 4D dynamic reconstruction model that leverages 4D Gaussian Splatting (4DGS) to represent and render dynamic scenes. The model can explicitly disentangle dynamic content from the static background and generate realistic optical flow. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art approaches across multiple datasets.

STRENGTHS AND WEAKNESSES:
Strengths:

- The proposed feedforward framework enables fast and scalable dynamic scene reconstruction.
- The method achieves superior performance compared to existing state-of-the-art techniques on various benchmark datasets.
- The experimental results are comprehensive and compelling, providing strong empirical support for the method's effectiveness.

Weaknesses:

- The reconstruction of dynamic objects still exhibits certain artifacts; slight camera motion shows the geometric inconsistencies.
- Some reconstructed results appear overly smoothed or blurred, potentially compromising fine details.

QUESTIONS:
- Although the method surpasses existing state-of-the-art techniques, there remains a gap between the presented results and practical deployment. So, could the outputs of this method serve as a strong initialization for optimization-based reconstruction approaches?
- Some results exhibit noticeable over-smoothing, such as the third column in Figure 3 where the hand appears less detailed compared to SoM. Could the authors provide an explanation for this phenomenon?
- Would it be possible to include a comparison with optimization-based methods? Such a comparison would help readers better understand the relative strengths and limitations of the proposed approach.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a feed-forward model for dynamic Gaussian reconstruction from monocular videos. To model the dynamic Gaussians, this paper introduces additional four Gaussian parameters to represent the motion of Gaussians. The overall model is  a Transformer-based architecture to predict dynamic Gaussian parameters from posed images. To make the training more efficient and scalable, this paper proposes a two-stage training strategy and a density control method to remove redundant Gaussians. Experiments are conducted on several dynamic scene datasets and the proposed method achieves state-of-the-art results.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The overall idea is simple and the pipeline is clear, and the proposed method achieves state-of-the-art results compared to previous methods.\n- The proposed density control method effectively reduce the number of Gaussians, which also makes the proposed model more scalable.\n- Experiments are extensive, with comparisons to several different baselines and evaluations on several different datasets.\n\n### Weaknesses\n\n- The choice of the dynamic Gaussian representation (L122-137) doesn't seem to be fully justified. This paper introduces four additional Gaussian attributes to model the Gaussian's motion across time. However, it's not fully clear why this paper chooses to use such a representation. In particular, I wonder if this representation is firstly proposed in this paper, or it's based on previous methods (seems that I couldn't find the related references in the paper). I am also curious about how this representation would compare to 3D flow vector based motion representation or the rigid motion representation as proposed in Shape of Motion paper. I think currently the description of the dynamic Gaussians representation lacks a bit of intuition, it would be better if the authors could clarify why this paper chooses to use this representation. I am also not fully clear whether a careful initialization is needed for the four additional Gaussian attributes. \n- I think it would be better if the authors could show how the Gaussians evolve over time, this would also help readers better understand how the proposed motion representation works in practice. \n- It's not very clear if the model relies on metric-scale camera calibrations. According to the training datasets, this paper uses both metric-scale camera annotations and COLMAP camera parameters, which might not be metric scale. However, in the limitation, this paper mentions that \"We observe that the quality may degrade in videos recorded by an unseen type of device due to the inaccurate *metric* scale calibrations\". A deeper discussion on why this happens would be beneficial.\n- I am a bit struggling at understanding which component in the proposed method contributes most to the final performance. From Table 2(a), it seems the loss functions are pretty important. However, a breakdown of different loss terms would be beneficial to gain more insights on the proposed method.\n\nQUESTIONS:\nThe key questions are listed in the Weaknesses. In addition, \n\n- Could the authors report the exact numbers of Gaussians predicted by the model? For example, with 64 and 128 input frames. This would help understand the compactness of the learned 3D representation.\n\n- In L55-57, this paper mentions that \"Compared to prior methods that also train dynamic Gaussian prediction [41, 24], 4DGT generalizes better to complex real-world dynamics and also exhibits emergent motion properties such as motion segmentation.\" However, it seems that I couldn't find a comparison with [24] in the experiments. Could the authors elaborate more on how 4DGT generalizes better than [24]?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces 4D Gaussian Transformer (4DGT), a feed-forward model for fast, high-quality 4D dynamic scene reconstruction from monocular videos. The key technical contribution is a novel density control strategy for training the model. This two-stage approach of pruning and densifying the predicted 4D Gaussians is presented as a crucial innovation, which the authors claim enables the 4DGT model to handle longer space-time input and remain efficient for rendering at runtime. Trained on real-world videos with pseudo-supervision, 4DGT achieves reconstruction quality competitive with slow optimization-based methods while being orders of magnitude faster.\n\nSTRENGTHS AND WEAKNESSES:\n###  **Strengths**\n\n* **Addresses a Valuable and Challenging Task:** The paper tackles a highly relevant and difficult problem in computer vision: fast 4D scene reconstruction from a monocular video. This is a valuable goal with strong potential for real-world applications like AR/VR and robotics.\n\n* **Strong Experimental Results:** The method demonstrates strong performance based on experimental evidence. It achieves rendering quality (PSNR, LPIPS) that is competitive with state-of-the-art optimization methods while being significantly faster. It clearly outperforms existing learning-based approaches on real-world datasets, validating the effectiveness of the proposed architecture and training strategy.\n\n* **Clarity and Presentation:** The paper is exceptionally well-written and clearly organized. The authors effectively motivate the problem, and the proposed 4DGT architecture and novel two-stage training pipeline are explained logically and intuitively, making the complex methodology accessible to readers.\n\n### **Weaknesses**\n\nWhile the paper presents a significant step forward, its claims could be further substantiated and its contributions clarified with additional analysis and experiments. My main suggestions for improvement are as follows:\n\n*   **Need for More Detailed Ablation Studies.** The current ablation study on the loss function (Table 2a) removes all regularization and guidance terms at once. The paper could be strengthened by a more granular analysis.\n\n*   **Quantifying Emergent Semantic Properties.** The paper makes a fascinating claim about the model learning \"emergent motion properties such as segmentation and flow\" (line 40). This is a potentially significant finding. To make this claim more robust, it would be highly beneficial to include quantitative and qualitative results.\n\n*   **Temporal Inconsistency.** Dynamic objects in the rendered videos exhibit noticeable flickering, highlighting a lack of temporal consistency. The paper's \"Expert guidance\" strategy successfully leverages pre-trained depth and normal models. A natural question arises: why not also incorporate guidance from a pre-trained optical flow expert? This would be a logical extension of the existing framework and a direct way to supervise the motion model to reduce these flickering artifacts.\n\n*   **Baseline Comparisons:** While the comparison to L4GM and SoM is strong and appropriate, the paper is missing comparisons to other relevant feed-forward methods likes:\n\n    [1] Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos[J]. arXiv preprint arXiv:2412.03526, 2024.\n\n    [2] Predicting 3D representations for Dynamic Scenes[J]. arXiv preprint arXiv:2501.16617, 2025.\n\nQUESTIONS:\nSee the weakness section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}