MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors

Authors: Qingming LIU, Yuan Liu, Jiepeng Wang, Xianqiang Lyu, Peng Wang, Wenping Wang, Junhui Hou
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Gaussian Splatting, Dynamic Novel-view Synthesis, Neural Rendering

Abstract:
In this paper, we propose MoDGS, a new pipeline to render novel-view images in dynamic scenes using only casually captured monocular videos. Previous monocular dynamic NeRF or Gaussian Splatting methods strongly rely on the rapid movement of input cameras to construct multiview consistency but fail to reconstruct dynamic scenes on casually captured input videos whose cameras are static or move slowly. To address this challenging task, MoDGS adopts recent single-view depth estimation methods to guide the learning of the dynamic scene. Then, a novel 3D-aware initialization method is proposed to learn a reasonable deformation field and a new robust depth loss is proposed to guide the learning of dynamic scene geometry. Comprehensive experiments demonstrate that MoDGS is able to render high-quality novel view images of dynamic scenes from just a casually captured monocular video, which outperforms baseline methods by a significant margin.  Project page: https://MoDGS.github.io

Decision: accept (poster)
Reviewer scores: 8, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1730356177688

thread_type:
official_review

<thread_object>
creator: reviewer_gzks

<content>
summary:
The paper introduces MoDGS, a method for dynamic view synthesis from monocular videos, leveraging a Gaussian-based splatting technique combined with deformation fields and an ordinal depth loss to reconstruct scenes. This framework integrates a 3D-aware initialization to align Gaussian representations in a canonical space, while the ordinal depth loss is used to improve scene geometry continuity. MoDGS is claimed to improve over previous dynamic NeRF approaches and related deformation methods, with results evaluated on the DyNeRF and Nvidia datasets.

strengths:
1. I think the 3D-aware initialization process is a strong point, as it specifically addresses a common issue in monocular reconstruction. By initializing Gaussians instead of relying on random initialization, this method seems to potentially add more consistency.  

2. The ordinal depth loss is, in my view, an interesting idea. It tries to tackle scale ambiguity in monocular depth estimation, which I think is particularly relevant in dynamic scenes. This loss formulation promotes depth consistency across frames, an essential factor when handling complex, moving scenes.

weaknesses:
1. I think the innovation is quite incremental for the reason that compared to closely related works like Deformable 3DGS and 4DGS, the methodological innovation appears incremental, mainly optimizing existing elements (depth consistency and deformation) rather than proposing a new structural approach.


2. Besides, the approach relies heavily on pre-trained depth models. MoDGS relies on single-view depth estimators like GeoWizard for depth initialization, which brings into question the independence of its results. The approach leverages external models as priors, potentially limiting its novelty and raising questions regarding knowledge distillation. The extent to which these pre-trained models influence the final performance is not rigorously analyzed.  
 

3. While MoDGS integrates external depth estimation for initialization, there is no formalized knowledge distillation to adaptively refine the model during training. This absence may reduce the adaptability of MoDGS across different dynamic scenes where pre-trained depth estimators may not perform equally well.

questions:
1. How does MoDGS handle scenarios where the pre-trained depth estimator provides inconsistent depth due to environmental variations? Has any analysis been conducted to measure performance stability when GeoWizard or other models are less reliable?  


2. Would MoDGS perform as well on datasets with higher motion complexity or less predictable scene geometry? Testing on a broader range of datasets, such as those with cluttered backgrounds or multiple moving objects, would better validate the method's generalization.   

3.  Considering MoDGS’s reliance on single-view depth priors, would a formalized knowledge distillation framework improve model autonomy by adapting these priors dynamically during training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730369270053

thread_type:
official_review

<thread_object>
creator: reviewer_fcsy

<content>
summary:
The paper presents MoDGS, a novel pipeline for synthesizing dynamic scenes from casually captured monocular videos. Unlike existing methods requiring large camera motions, MoDGS leverages single-view depth estimation for 3D reconstruction and introduces a 3D-aware initialization alongside an ordinal depth loss. These innovations enable robust, high-quality novel view synthesis, outperforming state-of-the-art methods in rendering casually captured videos.

strengths:
* A differentiable order-based loss function, the ordinal depth loss, is proposed, with detailed descriptions of its motivation and its distinctions from other depth loss functions.
* It demonstrates significant superiority over multi-view camera methods in reconstruction metrics and visual results, with ablation studies validating the importance of the "3D-aware initialization scheme" and "ordinal depth loss."
* The paper is well-written and easy to follow.

weaknesses:
* **The contributions and innovations are limited**. This work is based on the previous canonical space paradigm of 3D Gaussian Splatting (3DGS) combined with deformation fields, with the main contributions being a deformable 3DGS initialization method and a depth loss. The primary principle of the former relies on predicting per-pixel 3D flow using current state-of-the-art monocular depth estimation and optical flow estimation methods. However, the sole innovative aspect lies in converting 2D optical flow to 3D flow using the estimated depth map. As for the depth loss, although it is well-motivated and provides performance improvement, it essentially replaces the Pearson correlation loss with an order correlation loss.
* **The experimental comparisons lack fairness**. In most quantitative comparisons, this work is only compared against methods that require multi-view camera input. It is recommended to include quantitative and qualitative comparison results with methods under the same setting of "casually captured monocular video." It is also perplexing that the authors mention "RoDynRF, a method that adopts single-view depth estimation as supervision" in "Baseline methods", yet I only found comparative results for this method in Fig. 6.

questions:
Kindly refer to the [Weaknesses].
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660207927

thread_type:
official_review

<thread_object>
creator: reviewer_xth1

<content>
summary:
The paper proposes MoDGS, a novel pipeline to render high quality novel views of dynamic scenes from casually captured monocular videos. Unlike traditional dynamic scene reconstruction methods that rely on rapid camera motions to establish multiview consistency, MoDGS is designed for videos with static/slowly moving cameras, where such consistency is weaker. The core of their method involves using a single-view depth estimation technique to guide scene learning and introducing a 3D-aware initialization method to construct a realistic deformation field. MoDGS incorporates an innovative ordinal depth loss to address the challenge of depth inconsistency across frames, enhancing the coherence and quality of rendered views. Experiments on datasets such as DyNeRF, Nvidia, and a self-collected dataset demonstrate it's ability to outperform SOTA methods in novel view synthesis, achieving superior image quality even in challenging dynamic scenarios.

strengths:
MoDGS represents an original approach within  novel view synthesis  and dynamic scene modeling by specifically addressing the limitations of existing methods for casually captured monocular videos. The authors introduce a 3D-aware initialization mechanism and an ordinal depth loss, that offer a solution that successfully reduces the dependency on rapid camera motion. The novel use of ordinal depth loss to maintain depth order among frames, rather than relying solely on absolute values, represents an innovative perspective on addressing depth consistency issues, which has practical implications for improving depth coherence in dynamic scenes captured casually. I believe the paper is well-executed in terms of technical rigor, with comprehensive evaluations across three datasets: DyNeRF, Nvidia, and a newly created monocular casual video dataset. Each component of MoDGS is thoroughly tested and ablated to demonstrate its impact on the final results. This systematic experimentation supports the author’s claim that MoDGS significantly outperforms other approaches in the quality of novel-view rendering for dynamic scenes. The paper is structured logically, with clear explanations of each component of the MoDGS pipeline. The figures visually support the textual explanations, making complex concepts more understandable to a reader. The method  has significant implications for real-world applications that involve casually captured videos, such as mobile AR/VR, video editing, and 3D content creation. By enabling high-quality novel view synthesis from single-camera footage without multiview camera motions, MoDGS broadens the scope of dynamic scene reconstruction, making it accessible to a wider range of use cases. The method’s ability to handle both static and dynamic elements in monocular videos opens new avenues for monocular depth estimation and dynamic scene modeling, where single-camera approaches have been historically constrained by depth inconsistency issues.

weaknesses:
While the ordinal depth loss is a novel way to improve depth coherence, I believe the paper may benefit from more discussion on its limitations. Specifically, the ordinal depth loss assumes a consistent depth order among frames, which may not hold in scenes with complex occlusions or reflections. MoDGS assumes smooth transitions between frames for consistent depth ordering. However, the approach may face challenges in scenes with rapid or erratic movement where objects appear and disappear frequently. While it performs well on scenes with relatively smooth dynamics, addressing how the method might be adapted or optimized for highly dynamic environments would improve its versatility. The method relies heavily on single view depth estimators to guide the reconstruction process. Although the depth estimation technique used is SOTA, it still inherits the limitations of single view estimators, particularly in complex scenes with specular surfaces or low-lit conditions. Including a more detailed analysis on how the quality of the depth estimator impacts the proposed method’s performance, and potentially exploring integration with other depth supervision methods could potentially make the approach more adaptable across varying input qualities.

questions:
1. Can you elaborate on the choice of ordinal depth loss over other depth loss functions, such as perceptual depth consistency? How did the ordinal depth loss compare to other depth loss formulations in preliminary experiments, and what were the observed advantages or disadvantages?
2. How robust is MoDGS in scenarios with heavy occlusions or specular reflections? Would integrating additional priors or multi-scale depth estimations help in such cases?
3.   How does MoDGS compare with recent depth consistency techniques, particularly those used in self-supervised monocular depth estimation? Exploring this comparison could shed light on the effectiveness of the ordinal depth loss relative to existing methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665144317

thread_type:
official_review

<thread_object>
creator: reviewer_1rxu

<content>
summary:
The paper introduces MoDGS (Monocular Dynamic Gaussian Splatting), a novel approach for rendering dynamic 3D scenes from casually captured monocular videos, overcoming limitations faced by prior dynamic NeRF and Gaussian Splatting methods via depth estimation. These existing approaches require either extensive camera movement or synchronized multi-view setups to establish multiview consistency, which is lacking in casual, minimally moving videos.

To tackle this challenge, MoDGS incorporates recent advancements in single-view depth estimation to guide the learning of a deformation field that represents scene dynamics. The method introduces a novel ordinal depth loss to address the depth inconsistency in single-view depth maps, enhancing the robustness and continuity of 3D scene reconstruction.

Comprehensive experiments across multiple datasets (Nvidia, DyNeRF, DAVIS, and a self-collected casual video dataset) demonstrate that MoDGS produces high-quality novel views in dynamic scenes, outperforming state-of-the-art methods. The authors also plan to release their code and dataset to support future research in this area.

strengths:
For the novelty, this paper makes a distinct contribution to introducing depth supervision into the domain of dynamic Gaussian Splatting (DGS) for monocular dynamic input. This approach is novel yet intuitive, filling a key gap in the field for cases where the input consists of casually captured videos with minimal camera movement. Compared to the other papers in the field that mechanically put all fancy complicated input feature streams or loss functions together, the proposed solution is conceptually straightforward but impactful, pushing forward the capabilities of monocular dynamic scene reconstruction.

The experiments are well-designed and executed, rigorously testing the proposed method across various datasets, including Nvidia, DyNeRF, and DAVIS. Each experiment logically supports the methodology, demonstrating how the 3D-aware initialization and ordinal depth loss contribute to enhanced depth consistency and scene deformation modeling. The results clearly show MoDGS’s robustness and superiority over baseline methods, adding confidence in its effectiveness.

The paper is presented with clarity and precision, making even technical aspects of the method easy to follow. The figures and tables are well-constructed and informative, providing visual clarity to support the text and helping to reinforce the main findings. The logical flow, from problem statement to method explanation and results, enables readers to understand the method's motivation and benefits seamlessly.

weaknesses:
MoDGS is validated across several datasets, which demonstrates its robustness. However, the paper could discuss the potential limitations in generalizing this approach to different depth estimation models. It would demonstrate the robustness of the proposed method and its generalizability.

questions:
From the perspective of a peer, I suggest the authors address the concept of 'Depth supervised in dynamic GS' in the title. After all, a novel method would be more informative and important to the other researchers than a usage scenario like ''casually captured video".
</content>
</thread_object>

</thread>
<thread>
creation date:
1732209183262

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
### summary:

- **Acknowledgement.** 
We sincerely appreciate your valuable feedback on our work. We are delighted that you
that most of the reviewers have recognized our work’s strengths, including acknowledging the novelty and effectiveness of our 3D-Aware-init and ordinal depth loss (Reviewer 1rXU, xth1, Gzks), well-designed experiments(Reviewer 1rXU,xth1),  good experimental results (Reviewer 1rXU,xth1, FCsY), clear presentation(Reviewer 1rXU, FCsY). Thank you once again for your valuable feedback. In each response, we will thoroughly address your concerns, providing detailed explanations to clarify the points and answer your questions.



- **our revisions.**
To conclude: We added **five** new sections in the Appendix and added **four** limitations in Sec. 4.4.  
1) In the A.15 section, we discuss our MoDGS and other depth consistency methods and analyze them from a knowledge distillation perspective. 
2) In A.16, we present ablation studies about using perceptual loss to the rendered depth maps. 
3) In A.17, We provide results on scenes with relatively complex motions to demonstrate the robustness of MoDGS.
4) In A.18, We show results on scenes with relatively rapid motions to explore the potential of applying MoDGS in such challenging settings.
5) In A.19, We discuss how to handle scenes with heavy occlusions and specular objects.
methods completely fail.
6) In the limitation section, we outline potential limitations concerning extremely low-light conditions, rapidly moving objects, specular scenes, and our depth order assumption.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents MoDGS, a dynamic scene modeling algorithm using casually captured monocular videos. To overcome the rapid camera motion assumption in previous methods, it integrates single-view depth estimation, 3D-aware initialization, and robust ordinal depth loss, and it shows superior scene reconstruction and rendering performance.

The reviewers found the proposed components to be technically novel and interesting. The 3D-aware initialization improves the consistency of the initial Gaussians, and the ordinal depth loss can handle the problem of depth consistency across frames. The ablation study and experimental validations are provided at a sufficient level, and the paper is well written and clearly structured.

The reviewers also found that the proposed method builds incrementally on existing approaches such as deformable 3DGS by optimizing depth consistency and deformation, and its performance depends significantly on external single-view depth estimators. The influence of these models is not fully analyzed, especially in challenging conditions such as low illumination or complex scenes, and this may be the main limiting factor of the proposed algorithm, especially in complex scenes where monocular depth estimation becomes unreliable. Also, the comparison with multi-view methods in a monocular video setting may not be a fair comparison. 

During the rebuttal period, the authors provided answers to the reviewers' questions and performed additional experiments with different depth estimators.
All reviewers agree that the technical contribution of the proposed algorithm is significant and the paper is clearly written, and most reviewers agree to accept the paper.