MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos

## Abstract

Abstract In this paper, we propose MoDGS, a new pipeline to render novel views of dynamic scenes from a casually captured monocular video. Previous monocular dynamic NeRF or Gaussian Splatting methods strongly rely on the rapid movement of input cameras to construct multiview consistency but struggle to reconstruct dynamic scenes on casually captured input videos whose cameras are either static or move slowly. To address this challenging task, MoDGS adopts recent single-view depth estimation methods to guide the learning of the dynamic scene. Then, a novel 3D-aware initialization method is proposed to learn a reasonable deformation field and a new robust depth loss is proposed to guide the learning of dynamic scene geometry. Comprehensive experiments demonstrate that MoDGS is able to render high-quality novel view images of dynamic scenes from just a casually captured monocular video, which outperforms state-of-the-art methods by a significant margin. The code will be publicly available.
Project page: https://MoDGS.github.io

## 1 Introduction

Novel view synthesis (NVS) is an important task in computer graphics and computer vision, which greatly facilitates downstream applications such as augmented or virtual reality. In recent years, the novel-view-synthesis quality on static scenes has witnessed great improvements thanks to the recent development of techniques such as NeRF (Mildenhall et al., ), Instant-NGP (Müller et al., ), and Gaussian Splatting (Kerbl et al., ), especially when there are sufficient input images. However, novel view synthesis in a dynamic scene with only one monocular video still remains a challenging task.

Dynamic View Synthesis (DVS) has achieved impressive improvements along with the emerging neural representations (Mildenhall et al., ) and Gaussian splatting (Kerbl et al., ) techniques. Most of the existing DVS methods (Cao & Johnson, ; Yang et al., ) require multiview videos captured by dense synchronized cameras to achieve good rendering quality. Though some works can process a monocular video for DVS, as pointed out by DyCheck (Gao et al., ), these methods require the camera of the monocular video to have extremely large movements, which is called “Teleporting Camera Motion” on different viewpoints, so these methods can utilize the multiview consistency provided by this pseudo multiview video to reconstruct the 3D geometry of the dynamic scene. However, such large camera movements are rarely seen in casually captured videos because casual videos are usually produced by smoothly moving or even static cameras. When the camera moves slowly or is static, the multiview consistency constraint will be much weaker and all these existing DVS methods fail to produce high-quality novel-view images, as shown in Fig. 1.

In this paper, we present Monocular Dynamic Gaussian Splatting (MoDGS) to render novel-view images from casually captured monocular videos in a dynamic scene. MoDGS addresses the weak multiview constraint problem by adopting a monocular depth estimation method  (Fu et al., ), which provides prior depth information on the input video to help the 3D reconstruction.

However, we find that simply applying a single-view depth estimator in DVS to supervise rendered depth maps is not enough for high-quality novel view synthesis. First, the depth supervision only provides information for each frame but does not help to associate 3D points between two frames in time. Thus, we still have difficulty in learning an accurate time-dependent deformation field. Second, the estimated depth values are not consistent among different frames.

To learn a robust deformation field from a monocular video, we propose a 3D-aware initialization scheme for the deformation field. Existing methods (Katsumata et al., )
solely rely on supervision from 2D flow estimation, which produces deteriorated results without sufficient multiview consistency.
We find that directly initializing the deformation field in the 3D space greatly helps the subsequent learning of the 4D representations and improves the rendering quality as shown in Fig. 1.

To better utilize the estimated depth maps for supervision, we propose a novel depth loss to address the scale inconsistency of estimated depth values across different frames.

Previous methods (Li et al., ; Liu et al., ) supervise the rendered depth maps using a scale-invariant depth loss by minimizing the $L2$ distance of normalized rendered depth and depth priors, and the most recent method  (Zhu et al., ) proposed to supervise the rendered depth maps using a Pearson correlation loss to mitigate the scale ambiguity between the reconstructed scene and the estimated depth maps. However, the estimated depth maps of different frames are not even consistent after normalizing to the same scale. To address these challenges, we observe that despite the inconsistency in values, the orders of depth values of different pixels in different frames are stable, which motivates us to propose an ordinal depth loss. This novel ordinal depth loss enables us to fully utilize the estimated depth maps for high-quality novel view synthesis.

To demonstrate the effectiveness of MoDGS, we conduct experiments on three widely used datasets, the Nvdia (Yoon et al., ) dataset, the DyNeRF (Li et al., ) dataset, and the Davis (Pont-Tuset et al., ) dataset. We also present results on a self-collected dataset containing monocular in-the-wild videos from the Internet.
We adopt an exact monocular DVS evaluation setting that only uses the video of one camera as input while evaluating the video of another camera. Results show that our method outperforms previous DVS methods by a large margin and achieves high-quality NVS on casually captured monocular videos.

Figure: Figure 1: Given a casually captured monocular video of a dynamic scene, MoDGS is able to synthesize high-quality novel-view images in this scene. In the middle column, the baseline method (Yang et al., ) fails to correctly reconstruct the 3D dynamic scenes on this static monocular video. The white regions in cyan bounding boxes are not visible in the input video (red bounding boxes) so there are some artifacts for these invisible regions. In the rightmost column, the input estimated monocular depth is inconsistent (red bounding boxes); however, our proposed ordinal depth loss effectively ensures more consistent depth outputs. This loss enhances the accuracy and reliability of learning underlying geometry.
Refer to caption: x1.png

## 2 Related Work

In recent years, numerous works have focused on the task of novel view synthesis in both static and dynamic scenes. The main representatives are Neural Radiance Field (Mildenhall et al., ) and Gaussian Splatting (Kerbl et al., ), along with their variants.
In this paper, we primarily focus on view synthesis in dynamic scenes.

Figure: Figure 2: Overview. Given a casually captured monocular video of a dynamic scene, MoDGS represents the dynamic scene with a set of Gaussians in a canonical space and a deformation field represented by an MLP $\mathcal{T}$. To render an image at a specific timestamp $t$, we deform all the Gaussians by $\mathcal{T}_{t}$ and then use the splatting technique to render images and depth maps. While in training MoDGS, we use a single-view depth estimator GeoWizard (Fu et al., ) to estimate depth maps and compute the rendering loss and an ordinal depth loss for training.
Refer to caption: x2.png

Dynamic NeRF.
Recent dynamic NeRF methods can be roughly categorized into two groups.
1) Representing by time-varying neural radiance fields conditioned on time (Gao et al., ; Li et al., ; Park et al., ).
For example, Park et al. () proposes a simple spatiotemporal radiance field by interpolating the feature vectors indexed by time.
2) Representing by a canonical space NeRF and deformation field (Guo et al., ; Li et al., ; Park et al., ; ; Pumarola et al., ; Tretschk et al., ; Xian et al., ).
For example, NSFF (Li et al., ) models the dynamic components using forward and backward flow represented as 3D dense vector fields;
Nerfies (Park et al., ) and HyperNeRF (Park et al., ) model the scene dynamics as a deformation field mapping to a canonical space.
Recent advances in grid-based NeRFs (Müller et al., ; Sara Fridovich-Keil and Alex Yu et al., ; Chen et al., ) demonstrate that the training of static NeRFs can be significantly accelerated. Consequently, some dynamic NeRF works utilize these grid-based or hybrid representations for fast optimization (Guo et al., ; Cao & Johnson, ; Fang et al., ; Fridovich-Keil et al., ; Shao et al., ; Wang et al., ; ; Song et al., ; You & Hou, ).

Dynamic Gaussian Splatting.
The recent emergence of 3D Gaussian Splatting (3DGS) demonstrates its efficacy for super-fast real-time rendering attributed to its explicit point cloud representation.
Recent follow-ups extend 3DGS to model dynamic 3D scenes.
Luiten et al. () track dynamic 3D Gaussians by frame-by-frame training from synchronized multi-view videos.
Yang et al. () propose a deformable version of 3DGS by introducing a deformation MLP network to model the 3D flows.
Wu et al. () and Duisterhof et al. () also introduce a deformation field but using a more efficient Hexplane representation (Cao & Johnson, ).
Yang et al. () proposes a dynamic representation with a collection of 4D Gaussian primitives, where the time evolution can be encoded by 4D spherical harmonics.
Bae et al. () encodes motions with a per-Gaussian feature vector.
Some other works (Li et al., ; Lin et al., ; Liang et al., ) also study how to effectively encode the motions for Gaussians with different bases.
To effectively learn the motions of Gaussians, some works (Feng et al., ; Yu et al., ; Huang et al., ) resort to clustering the motions together for a compact representation.

DVS from Casual Monocular Videos.
As shown in DyCheck (Gao et al., ), many existing monocular dynamic view synthesis datasets used for benchmarking, like D-NeRF (Pumarola et al., ), HyperNeRF (Park et al., ), and Nerfies (Park et al., ), typically involve significant camera movements between frames but with small object dynamic motions. While this capture style helps with multi-view constraints and dynamic 3D modeling, it is not representative of casual everyday video captures. When using casual videos, the reconstruction results from these methods suffer from quality degradation.
Some works address dynamic 3D scene modeling using monocular casual videos. DynIBaR (Li et al., ) allows for long-sequence image-based rendering of dynamic scenes by aggregating features from nearby views, but its training cost is high for long per-scene optimization. Lee et al. () proposes a hybrid representation that combines static and dynamic elements, allowing for faster training and rendering, though it requires additional per-frame masks for dynamic components. RoDynRF (Liu et al., ) focuses on robust dynamic NeRF reconstruction by estimating NeRF and camera parameters together. DpDy (Wang et al., ) enhances quality by fine-tuning a diffusion model with SDS loss supervision (Poole et al., ), but it demands significant computational resources.
Concurrent works like DG-Marbles (Stearns et al., ) use Gaussian marbles and a hierarchical learning strategy to optimize representations. Shape-of-Motion (Wang et al., ) and Mosca (Lei et al., ) rely on explicit motion representation and initialize scene deformation with depth estimation and video tracking priors. However, our MoDGS method effectively uses only noisy inter-frame flow maps from RAFT (Teed & Deng, ) as input, performing well without the need for strong long-range pixel correspondence.

Ordinal Relation in Depth Maps.

The ordinal relation between pixels has been investigated in recent years, especially in the field of monocular depth estimation.  Zoran et al. () proposed to use a three-category classification network to predict the order relation of given pixel pairs. Then the depth can be extracted by optimizing a constrained quadratic optimization problem. Similarly,  Fu et al. () treats the depth prediction problem as a multi-class classification problem.

Chen et al. () further proposes a ranking loss to learn metric depth, which encourages a small difference between depths if the ground-truth relation is equal; otherwise it encourages a large difference. Then, Pavlakos et al. () extends this differentiable ranking loss to the human pose estimation task.
However, these works only utilize limited numbers of depth orders for training (one pair in  Chen et al. () and 17 pairs in  Pavlakos et al. ()), resulting in coarse supervision for depth maps.
The direct application of their ranking loss as depth supervision has yet to be explored.
Moreover, our ordinal depth takes the rendered metric depth maps as input, which are dense grids of float numbers.
Our task is different from previous depth estimation and pose estimation tasks and we present a comparison between our ordinal depth loss and depth ranking loss in Appendix A.10.

Figure: Figure 3: (a) Initialization of the deformation field. We first lift the depth maps and a 2D flow to a 3D flow and train the deformation field for initialization. (b) Initialization of Gaussians in the canonical space. We use the initialized deformation field to deform all the depth points to the canonical space and downsample these depth points to initialize Gaussians.
Refer to caption: x3.png

## 3 Proposed Method

Given a casually captured monocular video, we aim to synthesize novel view images from this video. We propose MoDGS, which achieves this by learning a set of Gaussians $\{G_{i}|i=1,~{}\cdots,~{}N\}$ in a canonical space and a deformation field $\mathcal{T}_{t}:\mathbb{R}^{3}\to\mathbb{R}^{3}$ to deform these Gaussians to a specific timestamp $t$. Then, for a timestamp $t$ and a camera pose, we use splatting to render an image.

##### Overview.

As shown in Fig. 2, to train MoDGS, we split the monocular video into a sequence of images $\{I_{t}|t=1,...,T\}$ with known camera poses. We denote our deformation field as a function $x_{t}=\mathcal{T}_{t}(x)$, which maps a 3D location $x\in\mathbb{R}^{3}$ in the canonical 3D space to a location $x_{t}\in\mathbb{R}^{3}$ in the 3D space on time $t$. For every image $I_{t}$, we utilize a single-view depth estimator (Fu et al., )
to estimate a depth map $D_{t}$ for every image and utilize a flow estimation method RAFT (Teed & Deng, ) to estimate a 2D optical flow $F_{t_{i}\to t_{j}}$ between $I_{t_{i}}$ and $I_{t_{j}}$ where $t_{i}$ and $t_{j}$ are two arbitrary timestamps. Then, we initialize our deformation field by a 3D-aware initialization scheme as introduced in Sec. 3.2. After initialization, we train our Gaussians and deformation field with a rendering loss and a new depth loss introduced in Sec. 3.3.
In the following, we first begin with the definition of the Gaussians and the rendering process in MoDGS.

### 3.1 Gaussians and Deformation Fields

##### Gaussians in the canonical space.

We define a set of Gaussians in the canonical space, we follow the original 3D GS (Kerbl et al., ) to define a 3D location, a scale vector, a rotation, and a color with spherical harmonics. Note this canonical space does not explictly correspond to any timestamp but is just a virtual space that contains the canonical locations of all Gaussians.

##### Deformation fields.

The deformation field $\mathcal{T}_{t}$ used in MoDGS follows the design of Omnimotion (Wang et al., ) and CaDeX (Lei & Daniilidis, ) which is an invertible MLP network (Dinh et al., ). This is an invertible MLP means that both $\mathcal{T}_{t}$ and $\mathcal{T}_{t}^{-1}$ can be directly computed from the MLP network. All $\mathcal{T}_{t}$ at different timestamps $t$ share the same MLP network and the time $t$ is normalized to $[0,1]$ as input to the MLP network.

##### Render with MoDGS.

After training both the Gaussians in canonical space and the deformation field, we will use the deformation field to deform the Gaussians in the canonical space to a specific time step $t$. Then, we follow exactly the splatting techniques in 3D GS (Kerbl et al., ) to render images from arbitrary viewpoints.

### 3.2 3D-aware Initialization

Original 3D Gaussian splatting (Kerbl et al., ) relies on the sparse points from Structure-from-Motion (SfM) to initialize all the locations of Gaussians. When we only have a casually captured monocular video, it is difficult to get an initial set of sparse points for initialization from SfM. Though it is possible to initialize all the Gaussians from the points of the estimated single-view depth of the first frame, we show that this leads to suboptimal results. At the same time, we need to initialize not only the Gaussians but also the deformation field. Thus, we propose a 3D-aware initialization scheme for MoDGS.

##### Initialization of depth scales.

Since the estimated depth maps on different timestamps would have different scales, we first estimate a coarse scale for every frame to unify the scales.
We achieve this by first segmenting out the static regions on the video and then computing the scale with a least square fitting (Chung et al., ).
The static regions can be determined by either thresholding on the 2D flow (Teed & Deng, ) or segmenting with a segmentation method like SAM2 (Ravi et al., ).

Then, on these static regions, we reproject the depth values at a specific timestamp to the first frame and minimize the difference between the projected depth and the depth of the first frame, which enables us to solve for a scale for every frame. We rectify all depth maps with the computed scales. In the following, we reuse $D_{t}$ to denote the rectified depth maps by default.

##### Initialization of the deformation field.

As shown in Fig. 3 (left), given two depth maps $D_{t_{i}}$ and $D_{t_{j}}$ along with the 2D flow $F_{t_{i}\to t_{j}}$, we lift them to a 3D flow $F_{t_{i}\to t_{j}}^{3D}$. This is achieved by first converting the depth maps into 3D points in the 3D space. Then, the estimated 2D flow $F_{t_{i}\to t_{j}}$ actually associate two sets of 3D points, which results in a 3D flow $F_{t_{i}\to t_{j}}^{3D}$. After getting this 3D flow, we then train our deformation field $\mathcal{T}$ with this 3D flow. Specifically, for a pixel in $I_{t_{i}}$ whose corresponding 3D point is $x_{t_{i}}$, we query $F^{3D}_{t_{i}\to t_{j}}$ to find its target point $x_{t_{j}}$ in the $t_{j}$ timestamp. Then, we minimize the difference by

$$ $\ell_{\text{init}}=\sum\|\mathcal{T}_{t_{j}}\circ\mathcal{T}^{-1}_{t_{i}}(x_{t _{i}})-x_{t_{j}}\|^{2}.$ (1) $$

We train the MLP in $\mathcal{T}$ for a fixed number of steps to initialize the deformation field.

##### Initialization of Gaussians.

After getting the initialized deformation field, we will initialize a set of 3D Gaussians in the canonical space as shown in Fig. 3 (right). We achieve this by first converting all the depth maps to get 3D points. Then, these 3D points are deformed backward to the canonical 3D space. This means that we transform all the depth points of all timestamps to the canonical space, which results in a large amount of points. We then evenly downsample these points with a predefined voxel size to reduce the point number and we initialize all our Gaussians with the locations of these downsampled 3D points in the canonical space.

### 3.3 Ordinal Depth Loss

##### Pearson correlation loss.

Existing dynamic Gaussian Splatting or NeRF methods also adopt a depth loss to supervise the learning of their 3D representations. One possible solution (Zhu et al., ; Li et al., ; Liu et al., ) is to maximize a Pearson correlation between the rendered depth and the estimated single-view depth

$\operatorname{Corr}\left(\hat{D}_{t},D_{t}\right)=\frac{\operatorname{Cov}
\left(\hat{D}_{t},D_{t}\right)}{\sqrt{\operatorname{Var}\left(\hat{D}_{t}
\right)\operatorname{Var}\left(D_{t}\right)}}$,

where $\operatorname{Cov}\left(\cdot,\cdot\right)$ and $\operatorname{Var}\left(\cdot,\cdot\right)$ means
the covariance and variance respectively,
$D_{t}$ and $\hat{D}_{t}$ are the estimated depth and the rendered depth respectively. Since the estimated single-view depth has ambiguity in scale, the Pearson correlation loss avoids the negative effects of the scale ambiguity. Note that in  Li et al. () and  Liu et al. (), the loss is called normalized depth loss, which is equivalent to Pearson correlation here as shown in the supplementary material.

Figure: Figure 4: We show the estimated single-view depth maps at two different timestamps $D_{t_{i}}$ and $D_{t_{j}}$ after normalization to the same scale. Since the single-view depth estimator is not accurate enough, the depth maps are not linear related so the scale normalization does not perfectly align them. However, the order of depth values on three corresponding pixels is stable for these two depth maps, which motivates us to propose an ordinal depth loss for supervision.
Refer to caption: x4.png

##### Limitations of Pearson correlation loss.

However, we find that this Pearson correlation depth loss is still suboptimal. As shown in Fig. 4, the estimated depth maps at two different timestamps are still not consistent with each other after normalization. Making two depth maps consistent after normalization actually requires these two depth maps to be related by a linear transformation, i.e. $D_{t+1}=aD_{t}+b$ with $a$ and $b$ two constants. However, the single-view depth estimation method is not accurate enough to guarantee the linear relationship between two estimated depth maps at different timestamps. In this case, the Pearson correlation loss still brings inconsistent supervision.

##### Ordinal depth loss.

To address this problem, our observation is that though we cannot guarantee depth consistency after normalization, as shown in Fig. 4, the order of depth value is consistent among two different frames. Thus, this motivates us to ensure the order of depth is correct by a new ordinal depth loss. We first define an order indicator function

$$ ${\mathcal{R}}(D_{t}(u_{1}),D_{t}(u_{2}))=\begin{cases}+1,&D_{t}(u_{1})>D_{t}(u _{2})\\ -1,&D_{t}(u_{1})<D_{t}(u_{2})\\ \end{cases},$ (2) $$

where $\mathcal{R}$ is the order indicator function on depth map $D_{t}$ which indicates the order between the depth values of pixels $u_{1}\in\mathbb{R}^{2}$ and $u_{2}\in\mathbb{R}^{2}$, and $D_{t}(u)$ means the depth value on the pixel $u$. Then, we define our ordinal depth loss based on the depth order by

$$ $\ell_{\text{ordinal}}=\|\text{tanh}\left(\alpha(\hat{D}_{t}(u_{1})-\hat{D}_{t} (u_{2}))\right)-\mathcal{R}\left(D_{t}(u_{1}),D_{t}(u_{2})\right)\|,$ (3) $$

where $\text{tanh}(x)=\frac{{e^{x}-e^{-x}}}{{e^{x}+e^{-x}}}$, $\hat{D}_{t}$ means the rendered depth map at timestamp $t$, $\hat{D}_{t}(u)$ is the depth value of this rendered depth map on the pixel $u$, $\alpha$ is a predefined constant. Eq. 3 means we transform the depth difference between $\hat{D}_{t}(u_{1})$ and $\hat{D}_{t}(u_{2})$ to 1 or -1 by tanh function. Then, we force the depth order of the rendered depth map $\hat{D}_{t}$ to be consistent with the order in the predicted depth map $D_{t}$. In the implementation, we randomly sample 100k pairs $(u_{1},u_{2})$ to compute the ordinal depth loss.

### 3.4 Training of MoDGS

After initializing the Gaussians and the deformation fields, we use MoDGS to render at a specific timestamp and compute the rendering loss $\ell_{\text{render}}$ and the ordinal depth loss $\ell_{\text{ordinal}}$. So the total training loss for MoDGS is

$$ $\ell=\lambda_{\text{ordinal}}\ell_{\text{ordinal}}+\lambda_{\text{render}}\ell _{\text{render}}.$ (4) $$

## 4 Experiments

Figure: Figure 5: Qualitative comparison on the novel-view renderings of the DyNeRF (Li et al., ) and Nvidia (Yoon et al., ) datasets. We compare MoDGS with SC-GS (Huang et al., ), Deformable-GS (Yang et al., ), and HexPlane (Cao & Johnson, ).
Refer to caption: x5.png

Figure: Figure 6: Qualitative comparison of DVS quality on the MCV dataset. We compare MoDGS with SC-GS (Huang et al., ), Deformable-GS (Yang et al., ), HexPlane (Cao & Johnson, ), and RoDynRF (Liu et al., ).
Refer to caption: x6.png

### 4.1 Evaluation Protocols

##### Datasets.

We conducted experiments on four datasets to demonstrate the effectiveness of our method. The first dataset is the DyNeRF (Li et al., ) dataset which consists of 6 scenes. On each scene, we have 18-20 synchronized cameras capturing 10-30 second videos. In these videos, there is mainly a man working on a desktop, like cutting beef or dumping water. We use camera0 for training and evaluate the results on camera5 and camera6.
The second dataset is the Nvidia (Yoon et al., ) dataset which contains more diverse dynamic subjects like jumping, playing with balloons, and opening an umbrella. The Nvidia dataset contains 8 scenes, which also has 12 synchronized cameras. We train all methods on camera4 and evaluate with camera3 and camera5. Besides, we also collect 6 online videos to construct an in-the-wild dataset, called the Monocular Casual Video (MCV) Dataset, to demonstrate our method can generalize to in-the-wild casual videos. The MCV dataset contains diverse subjects like skating, a dog eating food, YOGA, etc. The MCV dataset only contains a single video for each scene, so we cannot evaluate the quantitative results but only report the qualitative results on this dataset.
We also present results of the Davis dataset (Pont-Tuset et al., ) in Sec. A.6 of the appendix.

##### Evaluation setting.

Previous DVS methods (Cao & Johnson, ; Yang et al., ; Gao et al., ) all use different cameras to train the dynamic NeRF or Gaussian Splatting. Even though they only use one camera at a specific timestamp, they use different cameras at different timestamps so that a pseudo multiview video can be constructed to learn the 3D structures of the scene. Since our target is to conduct novel-view synthesis on the casually captured images, we do not adopt such ”teleporting camera motions” to construct training videos but just adopt one static camera to record training videos. Then, we render the images from the viewpoints of another camera for evaluation.

##### Metrics.

To evaluate the rendering quality, we have to render images from a new viewpoint and compare them with the ground-truth images. However, if the input video is almost static, the input video will contain insufficient 3D information and there would exist an ambiguity in scale. Thus, different DVS methods would choose different scales in reconstructing dynamic scenes so that the rendered images on the novel viewpoints are not aligned with the given ground-truth images. To address this problem, we manually label correspondences between the training images and ground-truth novel-view images. Then, we render a depth map on the training image using the reconstructed dynamic scene and optimize for a scale factor to scale the depth value to satisfy these labeled correspondences. After aligning the scale factors of different methods with the ground-truth images, we compute the SSIM, LPIPS, and PSNR between the rendered images and the ground-truth images.

**Table 1: Quantitative results on the DyNeRF (Li et al., ) and Nvidia (Yoon et al., ) datasets. We compare our method with SC-GS (Huang et al., ), Deformable GS (Yang et al., ) (D-GS) and HexPlane (Cao & Johnson, ) in PSNR$\uparrow$, SSIM$\uparrow$, and LPIPS$\downarrow$. For a per-scene breakdown of the metrics, please refer to Table 6 and Table 7 in A.9.**
|  | DyNeRF | Nvidia |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | PSNR$\uparrow$ | SSIM$\downarrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\downarrow$ | LPIPS$\downarrow$ |
| HexPlane | 15.33 | 0.5593 | 0.4514 | 17.17 | 0.3675 | 0.4756 |
| SC-GS | 18.77 | 0.7359 | 0.2310 | 17.59 | 0.4679 | 0.3348 |
| D-GS | 19.55 | 0.7446 | 0.2171 | 18.07 | 0.4650 | 0.3422 |
| Ours | 22.64 | 0.8042 | 0.1545 | 19.27 | 0.5235 | 0.2581 |

##### Baseline methods.

We compare MoDGS with 4 baseline methods to demonstrate the superior ability of MoDGS to synthesize novel-view images with casually captured monocular videos. These methods can be categorized into two classes. The first is the NeRF-based methods including HexPlane (Cao & Johnson, ) and RoDynRF (Liu et al., ). HexPlane represents the scene with six feature planes in both 3D space and time-space. We find that HexPlane does not produce reasonable results if only a monocular video from a single camera is given as input. Thus, other than the input monocular video, we use another video from a different viewpoint to train HexPlane for the DVS task. RoDynRF is a SoTA NeRF-based DVS method that also adopts single-view depth estimation as supervision for the 3D dynamic representations. We train it with the same single-view depth estimator GeoWizard (Fu et al., ) as ours. The second class is the Gaussian Splatting-based DVS methods including Deformable-GS (Yang et al., ) and SC-GS (Huang et al., ). Deformable-GS also associates a deformation field with a set of canonical Gaussians for DVS. SC-GS learns a set of keypoints and uses the deformation of these keypoints to blend the deformation of arbitrary 3D points.

### 4.2 Comparison with Baselines

The qualitative results on the DyNeRF and Nvidia datasets are shown in Fig. 5. Other qualitative results on our MCV dataset are shown in Fig. 6. The quantitative results on the Nvidia and DyNeRF datasets are shown in Table 1.
Supplementary videos contain more comparison results.

Synthesizing novel views from a casually captured monocular video is a challenging task. As shown in Fig. 5, though baseline methods achieve impressive results on these benchmarks with “teleporting camera motions”, these methods fail to correctly reconstruct the 3D geometry of the dynamic scenes and produce obvious artifacts on both dynamic foreground and static background. The main reason is that the monocular camera is almost static and does not provide enough multiview consistency to reconstruct high-quality 3D geometry for novel view synthesis. In the third row of Fig. 5, SC-GS (Huang et al., ) fails to reconstruct the dynamic foreground subject because SC-GS has an initialization process that treats the whole scene as a static scene and trains on the scene for a number of steps. When the foreground subject is moving with a large motion (like skating from left to right), it would be ignored by the static scene initialization and then we fail to reconstruct in the subsequent steps.

In comparison, our method relies on a 3D-aware initialization which provides a strong basis for the subsequent optimization. Meanwhile, our ordinal depth loss enables the 3D prior from the single-view depth estimator for an accurate reconstruction of the dynamic scenes. The quantitative results in Table 1 also show that our method achieves the best performances in all metrics on both datasets. Note that there are still some artifacts on occlusion boundaries because the input monocular camera is almost static and these regions are not visible in our input videos.

### 4.3 Ablation studies

We conduct ablation studies with our initialization and depth loss on the DyNeRF (Li et al., ) dataset to demonstrate their effectiveness. The qualitative results are shown in Fig. 7 and Fig. 8 while the quantitative results are shown in Table 2.
In the appendix, we also provide a comparison with depth warping, robustness to depth noises, a discussion on video depth, a comparison with other depth ranking losses, and a discussion about alpha values.

Figure: Figure 7: Visualization of rendering results using and without using our 3D-aware initialization.
Refer to caption: x7.png

#### 4.3.1 3D-aware initialization

To show the effectiveness of our 3D-aware initialization, we adopt a random initialization for the deformation field. Based on the random initialization, we still deform all the depth points backward to the canonical space and downsample these points to initialize the Gaussians. Then, we follow the exact same training procedure to train the randomly initialized baseline method. The final results of this random initialization are shown in Fig. 7. In comparison with our 3D-aware initialization, this random initialization produces more artifacts on the dynamic foreground human, which demonstrates that our 3D-aware initialization provides a good initial point for subsequent 3D dynamic scene reconstruction.

#### 4.3.2 Ordinal depth loss

To demonstrate the effectiveness of our proposed ordinal depth, we experiment with two baseline settings, removing the depth loss and utilizing the Pearson correlation as the depth loss.
As shown in Fig. 8, when there is no depth loss, the reconstructed 3D geometry contains much noise and obvious artifacts exist in the scene. Since we perform the 3D-aware initialization on this experiment to ensure a fair comparison, the results without depth loss still show reasonable rendered depth maps.
Adopting the Pearson depth loss improves the quality by linear correlating the rendered depth maps and input depth maps but still produces noisy depth inside a region. In comparison, our ordinal depth loss enables a smooth reconstruction of the depth map in the interior while maintaining sharp edges at boundaries. Thus, the proposed ordinal depth loss enables a more robust reconstruction of the dynamic scene.

Figure: Figure 8: Visualization of the rendered depth and RGB images using our ordinal depth loss and the Pearson correlation loss.
Refer to caption: x8.png

**Table 2: Ablation studies with the 3D-aware initialization (“3D-aware Init”) and Depth Loss on the DyNeRF (Li et al., ) dataset. “Ordinal” means the ordinal depth loss while “Pearson” means the Pearson correlation loss.**
| 3D-aware Init | Loss | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| --- | --- | --- | --- | --- |
| $\times$ | Ordinal | 21.27 | 0.7655 | 0.1984 |
| ✓ | Pearson | 21.77 | 0.7938 | 0.1680 |
| ✓ | Ordinal | 22.96 | 0.8103 | 0.1518 |

### 4.4 Limitations

Though our method can conduct dynamic view synthesis from casually captured monocular videos, the task is still extremely challenging.
One limitation is that our method can only reconstruct the visible 3D parts but cannot imagine the unseen parts, which leads to artifacts when rendering novel view videos on these unseen parts. Incorporating recent 3D-related diffusion generative models (Chung et al., ; Liu et al., ; Long et al., ) could be a promising direction to solve this problem, which we leave for future works. Another limitation is that the current training time is comparable with existing DVS methods, which could take several hours for a single scene. How to efficiently reconstruct the dynamic field would be an interesting and promising future research topic. Meanwhile, when the camera is completely static, our method strongly relies on the single-view depth estimator to estimate the 3D depth maps. Though existing single-view depth estimators (Fu et al., ; Ke et al., ; Yang et al., ; Bhat et al., ) are trained on large-scale datasets and predict reasonable depth maps for most cases, these depth estimators may fail to capture some details which degenerate the quality. Meanwhile, specular reflection can not be robustly and well handled, which is also an extremely challenging problem in the monocular setting, which we leave for future works.

## 5 Conclusion

In this paper, we have presented a novel dynamic view synthesis paradigm called MoDGS. In comparison with existing DVS methods which requires “teleporting camera motions”, MoDGS is designed to render novel view images from a casually captured monocular video. MoDGS introduces two new designs to finish this challenging task. First, a new 3D-aware initialization scheme is proposed, which directly initializes the deformation field to provide a reasonable starting point for subsequent optimization. We further analyze the problem of depth loss and propose a new ordinal depth loss to supervise the learning of the scene geometry. Extensive experiments on three datasets demonstrate superior performances of our method on in-the-wild monocualr videos over baseline methods.

## Appendix A Appendix

### A.1 Implementation Details

We implement our MoDGS with PyTorch. To initialize the deformation field, we train it with 20k steps as stated in Sec. 3.2. Subsequently, we jointly train the 3D Gaussians and the deformation field with the rendering loss and the ordinal depth loss for another 20k steps.
In Sec. 3.2, the flow is computed in evenly sampled key frames($1/5$). And the downsampling voxel size for Gaussian initialization is $0.004^{3}$ (scenes are normalized to $[-1,1]^{3}$ ).
For the outer optimization loop and rendering loss, we exactly follow the original 3DGS. And we use Gaussian centers to render depth  (Yang et al., ).
We adopt an Adam optimizer for optimization. The learning rate for 3D Gaussians exactly follows the official implementation of 3D GS (Kerbl et al., ), while the learning rate of the deformation network undergoes exponential decay from 1e-3 to 1e-4 in initialization and from 1e-4 to 1e-6 in the subsequent optimization. We set $\alpha=100$ for $\ell_{\text{ordinal}}$.
The weight of our depth order loss is 0.1. When computing depth ordinal loss, we first normalize the depth range to [0,1] and we only consider the depth pair with a difference larger than 0.02 for loss computation. The whole training takes around 3.5 hours to converge (2 hours for the initialization and 1.5 hours for the subsequent optimization) on an NVIDIA RTX A6000 GPU, which uses about 14G memory. The rendering speed of MoDGS is about 75 FPS.

### A.2 Real Depth Recovery

The depth prediction of GeoWizard (Fu et al., ) is normalized depth values in [0,1] and we have to transform them into real depth values. We follow their official implementation to estimate a scaling factor and an offset value on the normalized normal maps by minimizing the normal maps estimated from the transformed real depth values and the estimated normal maps from GeoWizard.
The optimization process takes just several seconds. Note that even after this normalization, the depth maps of different timestamps still differ from each other in scale.

Figure: Figure 9: We provide a pair of images on the iPhone (Gao et al., ) dataset. We draw two correspondences in the same color and their corresponding epipolar lines that are computed from the provided camera poses. The epipolar lines deviate far from the correspondences, which demonstrate that the camera poses are not accurate enough.
Refer to caption: x9.png

### A.3 Pose Errors in the iPhone dataset

DyCheck (Gao et al., ) adopts the iPhone dataset as the evaluation dataset for the DVS on casually captured videos. We do not adopt this dataset because we find that the poses on this dataset are not very accurate. An example is shown in Fig. 9, where we draw two correspondences and their corresponding epipolar line in the same color. The epipolar line is computed from the provided camera poses. As we can see, the epipolar line does not pass through the correspondence, which means that the provided poses are not accurate enough. We have tried to rerun COLMAP on the iPhone dataset but cannot get reasonable results.

### A.4 Difference from Depth Map Warping

Figure: Figure 10: Comparison of our renderings and depth warping. Our method accumulates information among different timestamps and thus is able to render more completed images.
Refer to caption: x10.png

MoDGS learns a set of 3D Gaussians in a canonical space and a deformation field to transform it to an arbitrary timestamp. This means that MoDGS is able to accumulate information among different timestamps to reconstruct a more completed scene than just using a single-view depth estimation. We show the difference between our renderings and just warping the training view using the estimated single-view depth map in Fig. 10. As we can see, MoDGS produces more completed reconstruction on contents that are not visible on this timestamp. Meanwhile, MoDGS rectifies the single-view depth maps to be more accurate so the rendering quality is much better.

### A.5 Pearson Depth Loss and Scale-shift Invariant Depth Loss

In this section, we will prove that the Pearson Depth loss is equivalent to Scale-Shift Invariant loss.
Previous works (Li et al., ; Liu et al., ) use scale and shift invariant depth loss by minimizing the $L2$ distance of normalized ground truth depth map $\text{Norm}(D_{t})$ and the normalized rendered depth map $\text{Norm}(\hat{D_{t}})$

$$ $\ell_{\text{depth}}=\|\text{Norm}(D_{t})-\text{Norm}(\hat{D}_{t})\|$ (5) $$

where $\text{Norm}(D_{t})$ denotes $\cfrac{D_{t}-u_{D_{t}}}{\sigma_{D_{t}}}$.

$$ $\displaystyle\left(\cfrac{D_{t}-u_{D_{t}}}{\sigma_{D_{t}}}-\cfrac{D_{t}-u_{ \hat{D_{t}}}}{\sigma_{\hat{D_{t}}}}\right)^{2}$ (6) $\displaystyle=-\cfrac{2(D_{t}-u_{D_{t}})(D_{t}-u_{\hat{D_{t}}})}{\sigma_{D_{t} }\sigma_{\hat{D_{t}}}}+\left(\cfrac{D_{t}-u_{D_{t}}}{\sigma_{D_{t}}}\right)^{2 }+\left(\cfrac{\hat{D_{t}}-u_{\hat{D_{t}}}}{\sigma_{\hat{D_{t}}}}\right)^{2}.$ $$

where $D_{t}$ is the predicted depth prior, $\sigma_{D_{t}}$ and $u_{D_{t}}$ denotes the standard deviation and means respectively. Since it is based on the input, The second term is a constant.
For the third term, we substitute the $\sigma_{\hat{D_{t}}}$ using its definition:
$\sigma_{\hat{D_{t}}}=\sqrt{\frac{\sum_{i=1}^{N}\left(\hat{D_{t}^{i}}-u_{\hat{D
_{t}}}\right)^{2}}{N}}$.
The sum of the third term over all pixels in a depth map is:
$\Sigma_{i}^{N}(\left(\cfrac{\hat{D_{t}^{i}}-u_{\hat{D_{t}}}}{\sigma_{\hat{D_{t
}}}}\right)^{2})=N$, Where $N$ is the total number of pixels.

We can see that both the second term and third term are constant. The first term is a
simple transformation of the Pearson correlation coefficient. Thus, minimizing the $L2$ distance is equivalent to maximizing the coefficient.

### A.6 Results on the Davis dataset

#### A.6.1 Preprocessing Procedure

On the Davis dataset, we adopt the preprocessing procedure outlined in the concurrent method, Shape-of-Motion (Wang et al., ), to obtain these camera poses. Specifically, we first run UniDepth (Piccinelli et al., ) to acquire camera intrinsics and depth maps. Then, we utilize a SLAM method, Driod-SLAM (Teed & Deng, ), with the depth maps from UniDepth as input to solve for the camera poses.

#### A.6.2 Comparison with Shape-of-Motion

Figure: Figure 11: We show a visual comparison of rendered novel view images and depth maps between Shape-of-Motion(Wang et al., ) and MoDGS. The rendering quality is comparable but our method shows more details in some cases.
Refer to caption: x11.png

We provide an experiment to compare Shape-of-Motion with our method.
In our experiments, we train shape-of-motion on the Davis dataset following their default training setting and use the same preprocessing strategy like depth estimation to train MoDGS.
The visual comparison is shown in Fig. 11, which demonstrates that our method shows comparable results as Shape-of-Motion but renders more details in some cases.
We also present the quantitative comparison results on the Nvidia dataset  (Yoon et al., ) in Table 3.

**Table 3: Comparison with Shape-of-Motion(SoM) on Nvidia dataset**
|  | Balloon2-2 | Jumping | Truck-2 | Skatting2 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| SoM | 19.52 | 0.5096 | 0.2790 | 20.32 | 0.6771 | 0.2560 | 22.53 | 0.7414 | 0.1853 | 23.51 | 0.7861 | 0.1658 |
| Ours | 20.47 | 0.5275 | 0.2408 | 22.18 | 0.7075 | 0.2125 | 23.69 | 0.7455 | 0.1551 | 25.64 | 0.7996 | 0.1518 |

### A.7 Discussion about alpha value in ordinal depth loss

After roughly aligning the depth maps, we normalize the scene to $[-1,1]^{3}$ and empirically set alpha to 100. We present further ablations in Table 4, where decreasing alpha leads to worse results while increasing alpha yields comparable results. The reason is that large alpha values compel the loss to mainly concentrate on the ordinality rather than the differences in depth values.

**Table 4: Quantitative results for different alpha values.**
| Setting | PSNR | SSIM | LPIPS |
| --- | --- | --- | --- |
| alpha=10 | 21.05 | 0.7575 | 0.2370 |
| alpha=40 | 21.39 | 0.7813 | 0.1916 |
| alpha=100(Ours) | 23.23 | 0.8083 | 0.1592 |
| alpha=1000 | 23.16 | 0.8058 | 0.1715 |

### A.8 Discussion about the Robustness to the Depth noises

To show the robustness of our Ordinal Depth Loss to depth noise, we manually add Gaussian noises to depth maps and use the noisy depth maps as supervision signals. The metric results of them are shown in Table  5. (on the “flame_steak” scene), which shows our method is robust to small depth noises.

**Table 5: Quantitative results with different levels of added Gaussian noise $Norm(\mu,\sigma)$**
| Setting | PSNR | SSIM | LPIPS |
| --- | --- | --- | --- |
| $Norm(0,0.20)$ | 23.18 | 0.8090 | 0.1667 |
| $Norm(0,0.05)$ | 23.35 | 0.8100 | 0.1633 |
| Ours | 23.23 | 0.8083 | 0.1592 |

### A.9 Per-Scene breakdown results on the DyNeRF (Li et al., 2022 ) dataset and the Nvidia (Yoon et al., 2020 ) dataset

In Tables 6 and Table 7, we provide a breakdown of the metrics for different scenes in the DyNeRF and Nvidia datasets, respectively.

**Table 6: Per-scene results on the DyNeRF dataset.**
| Method | cut_roasted_beef | sear_steak | coffee_martini |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| HexPlane | 16.76 | 0.5382 | 0.5054 | 16.89 | 0.5897 | 0.5049 | 13.26 | 0.4049 | 0.5835 |
| SC-GS | 20.69 | 0.7414 | 0.2625 | 21.23 | 0.7870 | 0.2188 | 19.02 | 0.7124 | 0.2151 |
| D-GS | 22.20 | 0.7808 | 0.1931 | 23.56 | 0.8101 | 0.1773 | 19.23 | 0.7013 | 0.2270 |
| Ours | 23.98 | 0.8221 | 0.1438 | 23.53 | 0.8126 | 0.1642 | 21.37 | 0.7962 | 0.1473 |

**Table 7: Per-scene results on the Nvidia dataset.**
| Method | Playground | DynamicFace-2 | Balloon1-2 | Umbrella |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| HexPlane | 13.30 | 0.1641 | 0.5240 | 9.92 | 0.1873 | 0.6091 | 15.48 | 0.2793 | 0.5342 | 18.70 | 0.2238 | 0.4427 |
| SC-GS | 12.60 | 0.2059 | 0.5063 | 9.75 | 0.3312 | 0.4307 | 16.63 | 0.4412 | 0.3656 | 18.71 | 0.3501 | 0.3663 |
| D-GS | 12.62 | 0.2009 | 0.5116 | 10.36 | 0.3277 | 0.4563 | 10.36 | 0.3277 | 0.4563 | 19.16 | 0.3381 | 0.3300 |
| Ours | 13.38 | 0.2343 | 0.4542 | 12.40 | 0.4118 | 0.2442 | 16.93 | 0.4371 | 0.3059 | 19.47 | 0.3246 | 0.3004 |

### A.10 Comparison with depth ranking Loss

We provide a quantitative comparison between our ordinal depth loss and other depth ranking losses (Chen et al., ; Pavlakos et al., ) in Table 8. In this experiment, we replace our ordinal depth loss with their depth ranking loss while keeping all other configurations fixed.
The results demonstrate that our ordinal depth loss significantly outperforms their approach across all scenes. This highlights the effectiveness of our method in capturing depth information more accurately than the existing depth ranking losses.

In Fig. 12, we present a visual comparison of rendered depth maps using depth ranking loss and ordinal depth loss. Our depth values are more smooth and well-distributed while depth values learned from rank depth loss show high contrast and more noise. The reason is that depth ranking loss encourages large contrast to satisfy the given depth order and is not robust to the noises in the input depth.

Figure: Figure 12: A visual comparison of rendered depth maps between using depth ranking loss (Wang et al., ) and our ordinal depth loss.
Refer to caption: x12.png

**Table 8: Comparison with depth ranking loss (Chen et al., ; Pavlakos et al., ),denoted as $DRL$ for simplification.**
|  | sear_steak | cut_beef | coffee_martini |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| $DRL$ | 18.22 | 0.7390 | 0.2228 | 17.07 | 0.7162 | 0.2283 | 14.09 | 0.7465 | 0.1902 |
| Ours | 23.53 | 0.8126 | 0.1642 | 23.98 | 0.8221 | 0.1438 | 21.37 | 0.7962 | 0.1473 |

### A.11 ablation using depth from Video-depth estimation methods

To show that our method is also compatible with recent video depth methods, we conduct an ablation experiment using the depth maps predicted by the video depth estimator  (Shao et al., ) and different types of depth loss as supervision. As shown in Table 9, the video depth estimator improves the results, and our ordinal depth loss still consistently improves the results.

**Table 9: Video depth ablation. $VD+P$ refers to supervised using video depth maps with pearson depth loss, $VD+O$ refers to supervised using video depth maps with ordinal depth loss.**
|  | sear_steak | cut_beef | coffee_martini |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| $VD+P$ | 23.43 | 0.7983 | 0.1732 | 23.19 | 0.8134 | 0.1472 | 20.01 | 0.7664 | 0.1647 |
| $VD+O$ | 23.48 | 0.8091 | 0.1711 | 23.61 | 0.8103 | 0.1532 | 21.55 | 0.8001 | 0.1488 |

### A.12 Results of NeRF-based methods using depth supervision

We present the results of HexPlane (Cao & Johnson, ) when supervised with depth loss in Table 10. Using depth maps for supervision only leads to a slight improvement for HexPlane and MoDGS still outperforms HexPlane by a large margin.

**Table 10: Metric results when HexPlane (Cao & Johnson, ) is incorporated with depth supervision, denoted as “HexPlane+D”.**
|  | flame_steak | cut_beef | coffee_martini |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |
| HexPlane | 16.97 | 0.7528 | 0.2543 | 16.76 | 0.5382 | 0.5054 | 13.26 | 0.4049 | 0.5835 |
| HexPlane+D | 17.21 | 0.6156 | 0.4646 | 18.82 | 0.6344 | 0.414 | 15.78 | 0.4708 | 0.5106 |
| Ours | 23.23 | 0.8083 | 0.1592 | 23.98 | 0.8221 | 0.1438 | 21.37 | 0.7962 | 0.1473 |

### A.13 Different random initialization methods

In the ablation study of our paper, we use the Kaiming initialization as the random initialization. We additionally provide ablations using zero deformation initialization of the deformation field in Table 11(flame_steak scene of DyNeRF). In this experiment, we still initialize the canonical Gaussian spaces but adopt a different initialization strategy for the deformation field MLP.

**Table 11: Quantitative comparison results for different initialization methods**
| Method | PSNR | SSIM | Lpips |
| --- | --- | --- | --- |
| ZeroDeformationInit | 22.34 | 0.7906 | 0.2151 |
| KaimingInit | 22.24 | 0.7744 | 0.2036 |
| 3D-AwareInit (Ours) | 23.23 | 0.8083 | 0.1592 |

### A.14 Length of videos that MoDGS can handle

Currently, we can handle videos up to 10 seconds. Scaling to longer videos($\sim$30 seconds) is possible if we use a larger deformation MLP network and train for more iterations.

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: From the perspective of a peer, I suggest the authors address the concept of 'Depth supervised in dynamic GS' in the title. After all, a novel method would be more informative and important to the other researchers than a usage scenario like ''casually captured video".
soundness: 4
strengths: For the novelty, this paper makes a distinct contribution to introducing depth supervision into the domain of dynamic Gaussian Splatting (DGS) for monocular dynamic input. This approach is novel yet intuitive, filling a key gap in the field for cases where the input consists of casually captured videos with minimal camera movement. Compared to the other papers in the field that mechanically put all fancy complicated input feature streams or loss functions together, the proposed solution is conceptually straightforward but impactful, pushing forward the capabilities of monocular dynamic scene reconstruction.

The experiments are well-designed and executed, rigorously testing the proposed method across various datasets, including Nvidia, DyNeRF, and DAVIS. Each experiment logically supports the methodology, demonstrating how the 3D-aware initialization and ordinal depth loss contribute to enhanced depth consistency and scene deformation modeling. The results clearly show MoDGS’s robustness and superiority over baseline methods, adding confidence in its effectiveness.

The paper is presented with clarity and precision, making even technical aspects of the method easy to follow. The figures and tables are well-constructed and informative, providing visual clarity to support the text and helping to reinforce the main findings. The logical flow, from problem statement to method explanation and results, enables readers to understand the method's motivation and benefits seamlessly.
summary: The paper introduces MoDGS (Monocular Dynamic Gaussian Splatting), a novel approach for rendering dynamic 3D scenes from casually captured monocular videos, overcoming limitations faced by prior dynamic NeRF and Gaussian Splatting methods via depth estimation. These existing approaches require either extensive camera movement or synchronized multi-view setups to establish multiview consistency, which is lacking in casual, minimally moving videos.

To tackle this challenge, MoDGS incorporates recent advancements in single-view depth estimation to guide the learning of a deformation field that represents scene dynamics. The method introduces a novel ordinal depth loss to address the depth inconsistency in single-view depth maps, enhancing the robustness and continuity of 3D scene reconstruction.

Comprehensive experiments across multiple datasets (Nvidia, DyNeRF, DAVIS, and a self-collected casual video dataset) demonstrate that MoDGS produces high-quality novel views in dynamic scenes, outperforming state-of-the-art methods. The authors also plan to release their code and dataset to support future research in this area.
weaknesses: MoDGS is validated across several datasets, which demonstrates its robustness. However, the paper could discuss the potential limitations in generalizing this approach to different depth estimation models. It would demonstrate the robustness of the proposed method and its generalizability.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. Can you elaborate on the choice of ordinal depth loss over other depth loss functions, such as perceptual depth consistency? How did the ordinal depth loss compare to other depth loss formulations in preliminary experiments, and what were the observed advantages or disadvantages?
2. How robust is MoDGS in scenarios with heavy occlusions or specular reflections? Would integrating additional priors or multi-scale depth estimations help in such cases?
3.   How does MoDGS compare with recent depth consistency techniques, particularly those used in self-supervised monocular depth estimation? Exploring this comparison could shed light on the effectiveness of the ordinal depth loss relative to existing methods.
soundness: 4
strengths: MoDGS represents an original approach within  novel view synthesis  and dynamic scene modeling by specifically addressing the limitations of existing methods for casually captured monocular videos. The authors introduce a 3D-aware initialization mechanism and an ordinal depth loss, that offer a solution that successfully reduces the dependency on rapid camera motion. The novel use of ordinal depth loss to maintain depth order among frames, rather than relying solely on absolute values, represents an innovative perspective on addressing depth consistency issues, which has practical implications for improving depth coherence in dynamic scenes captured casually. I believe the paper is well-executed in terms of technical rigor, with comprehensive evaluations across three datasets: DyNeRF, Nvidia, and a newly created monocular casual video dataset. Each component of MoDGS is thoroughly tested and ablated to demonstrate its impact on the final results. This systematic experimentation supports the author’s claim that MoDGS significantly outperforms other approaches in the quality of novel-view rendering for dynamic scenes. The paper is structured logically, with clear explanations of each component of the MoDGS pipeline. The figures visually support the textual explanations, making complex concepts more understandable to a reader. The method  has significant implications for real-world applications that involve casually captured videos, such as mobile AR/VR, video editing, and 3D content creation. By enabling high-quality novel view synthesis from single-camera footage without multiview camera motions, MoDGS broadens the scope of dynamic scene reconstruction, making it accessible to a wider range of use cases. The method’s ability to handle both static and dynamic elements in monocular videos opens new avenues for monocular depth estimation and dynamic scene modeling, where single-camera approaches have been historically constrained by depth inconsistency issues.
summary: The paper proposes MoDGS, a novel pipeline to render high quality novel views of dynamic scenes from casually captured monocular videos. Unlike traditional dynamic scene reconstruction methods that rely on rapid camera motions to establish multiview consistency, MoDGS is designed for videos with static/slowly moving cameras, where such consistency is weaker. The core of their method involves using a single-view depth estimation technique to guide scene learning and introducing a 3D-aware initialization method to construct a realistic deformation field. MoDGS incorporates an innovative ordinal depth loss to address the challenge of depth inconsistency across frames, enhancing the coherence and quality of rendered views. Experiments on datasets such as DyNeRF, Nvidia, and a self-collected dataset demonstrate it's ability to outperform SOTA methods in novel view synthesis, achieving superior image quality even in challenging dynamic scenarios.
weaknesses: While the ordinal depth loss is a novel way to improve depth coherence, I believe the paper may benefit from more discussion on its limitations. Specifically, the ordinal depth loss assumes a consistent depth order among frames, which may not hold in scenes with complex occlusions or reflections. MoDGS assumes smooth transitions between frames for consistent depth ordering. However, the approach may face challenges in scenes with rapid or erratic movement where objects appear and disappear frequently. While it performs well on scenes with relatively smooth dynamics, addressing how the method might be adapted or optimized for highly dynamic environments would improve its versatility. The method relies heavily on single view depth estimators to guide the reconstruction process. Although the depth estimation technique used is SOTA, it still inherits the limitations of single view estimators, particularly in complex scenes with specular surfaces or low-lit conditions. Including a more detailed analysis on how the quality of the depth estimator impacts the proposed method’s performance, and potentially exploring integration with other depth supervision methods could potentially make the approach more adaptable across varying input qualities.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Kindly refer to the [Weaknesses].
soundness: 2
strengths: * A differentiable order-based loss function, the ordinal depth loss, is proposed, with detailed descriptions of its motivation and its distinctions from other depth loss functions.
* It demonstrates significant superiority over multi-view camera methods in reconstruction metrics and visual results, with ablation studies validating the importance of the "3D-aware initialization scheme" and "ordinal depth loss."
* The paper is well-written and easy to follow.
summary: The paper presents MoDGS, a novel pipeline for synthesizing dynamic scenes from casually captured monocular videos. Unlike existing methods requiring large camera motions, MoDGS leverages single-view depth estimation for 3D reconstruction and introduces a 3D-aware initialization alongside an ordinal depth loss. These innovations enable robust, high-quality novel view synthesis, outperforming state-of-the-art methods in rendering casually captured videos.
weaknesses: * **The contributions and innovations are limited**. This work is based on the previous canonical space paradigm of 3D Gaussian Splatting (3DGS) combined with deformation fields, with the main contributions being a deformable 3DGS initialization method and a depth loss. The primary principle of the former relies on predicting per-pixel 3D flow using current state-of-the-art monocular depth estimation and optical flow estimation methods. However, the sole innovative aspect lies in converting 2D optical flow to 3D flow using the estimated depth map. As for the depth loss, although it is well-motivated and provides performance improvement, it essentially replaces the Pearson correlation loss with an order correlation loss.
* **The experimental comparisons lack fairness**. In most quantitative comparisons, this work is only compared against methods that require multi-view camera input. It is recommended to include quantitative and qualitative comparison results with methods under the same setting of "casually captured monocular video." It is also perplexing that the authors mention "RoDynRF, a method that adopts single-view depth estimation as supervision" in "Baseline methods", yet I only found comparative results for this method in Fig. 6.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. How does MoDGS handle scenarios where the pre-trained depth estimator provides inconsistent depth due to environmental variations? Has any analysis been conducted to measure performance stability when GeoWizard or other models are less reliable?  


2. Would MoDGS perform as well on datasets with higher motion complexity or less predictable scene geometry? Testing on a broader range of datasets, such as those with cluttered backgrounds or multiple moving objects, would better validate the method's generalization.   

3.  Considering MoDGS’s reliance on single-view depth priors, would a formalized knowledge distillation framework improve model autonomy by adapting these priors dynamically during training?
soundness: 2
strengths: 1. I think the 3D-aware initialization process is a strong point, as it specifically addresses a common issue in monocular reconstruction. By initializing Gaussians instead of relying on random initialization, this method seems to potentially add more consistency.  

2. The ordinal depth loss is, in my view, an interesting idea. It tries to tackle scale ambiguity in monocular depth estimation, which I think is particularly relevant in dynamic scenes. This loss formulation promotes depth consistency across frames, an essential factor when handling complex, moving scenes.
summary: The paper introduces MoDGS, a method for dynamic view synthesis from monocular videos, leveraging a Gaussian-based splatting technique combined with deformation fields and an ordinal depth loss to reconstruct scenes. This framework integrates a 3D-aware initialization to align Gaussian representations in a canonical space, while the ordinal depth loss is used to improve scene geometry continuity. MoDGS is claimed to improve over previous dynamic NeRF approaches and related deformation methods, with results evaluated on the DyNeRF and Nvidia datasets.
weaknesses: 1. I think the innovation is quite incremental for the reason that compared to closely related works like Deformable 3DGS and 4DGS, the methodological innovation appears incremental, mainly optimizing existing elements (depth consistency and deformation) rather than proposing a new structural approach.


2. Besides, the approach relies heavily on pre-trained depth models. MoDGS relies on single-view depth estimators like GeoWizard for depth initialization, which brings into question the independence of its results. The approach leverages external models as priors, potentially limiting its novelty and raising questions regarding knowledge distillation. The extent to which these pre-trained models influence the final performance is not rigorously analyzed.  
 

3. While MoDGS integrates external depth estimation for initialization, there is no formalized knowledge distillation to adaptively refine the model during training. This absence may reduce the adaptability of MoDGS across different dynamic scenes where pre-trained depth estimators may not perform equally well.

Meta-review: This paper presents MoDGS, a dynamic scene modeling algorithm using casually captured monocular videos. To overcome the rapid camera motion assumption in previous methods, it integrates single-view depth estimation, 3D-aware initialization, and robust ordinal depth loss, and it shows superior scene reconstruction and rendering performance.

The reviewers found the proposed components to be technically novel and interesting. The 3D-aware initialization improves the consistency of the initial Gaussians, and the ordinal depth loss can handle the problem of depth consistency across frames. The ablation study and experimental validations are provided at a sufficient level, and the paper is well written and clearly structured.

The reviewers also found that the proposed method builds incrementally on existing approaches such as deformable 3DGS by optimizing depth consistency and deformation, and its performance depends significantly on external single-view depth estimators. The influence of these models is not fully analyzed, especially in challenging conditions such as low illumination or complex scenes, and this may be the main limiting factor of the proposed algorithm, especially in complex scenes where monocular depth estimation becomes unreliable. Also, the comparison with multi-view methods in a monocular video setting may not be a fair comparison. 

During the rebuttal period, the authors provided answers to the reviewers' questions and performed additional experiments with different depth estimators.
All reviewers agree that the technical contribution of the proposed algorithm is significant and the paper is clearly written, and most reviewers agree to accept the paper.

Decision: Accept (Poster)