MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors
Authors: Qingming LIU
Yuan Liu
Jiepeng Wang
Xianqiang Lyu
Peng Wang
Wenping Wang
Junhui Hou
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Gaussian Splatting
Dynamic Novel-view Synthesis
Neural Rendering
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In this paper, we propose MoDGS, a new pipeline to render novel-view images in dynamic scenes using only casually captured monocular videos. Previous monocular dynamic NeRF or Gaussian Splatting methods strongly rely on the rapid movement of input cameras to construct multiview consistency but fail to reconstruct dynamic scenes on casually captured input videos whose cameras are static or move slowly. To address this challenging task, MoDGS adopts recent single-view depth estimation methods to guide the learning of the dynamic scene. Then, a novel 3D-aware initialization method is proposed to learn a reasonable deformation field and a new robust depth loss is proposed to guide the learning of dynamic scene geometry. Comprehensive experiments demonstrate that MoDGS is able to render high-quality novel view images of dynamic scenes from just a casually captured monocular video, which outperforms baseline methods by a significant margin.  Project page: https://MoDGS.github.io

Published as a conference paper at ICLR 2025
MODGS: DYNAMIC GAUSSIAN SPLATTING FROM CASUALLY-CAPTURED MONOCULAR VIDEOS WITH DEPTH PRIORS
Qingming Liu Yuan Liu
Jiepeng Wang'
Xianqiang Lyv!
Peng Wang*
Wenping Wang Junhui Hou't
City University of HongKong qingmingl 1ogggiiu u @ f foxmail. com
HKUST HKKU "TAMU
SCUHK(SZ)
yuanlyeust.he
ABSTRACT
In this paper, we propose MoDGS, a new pipeline to render novel views of dy- namic scenes from a casually captured monocular video. Previous monocular dynamic NeRF or Gaussian Splatting methods strongly rely on the rapid move- ment of input cameras to construct multiview consistency but struggle to recon- struct dynamic scenes casually captured input videos whose cameras are either static or move slowly. To address this challenging task, MoDGS adopts recent single-view depth estimation methods to guide the learning of the dynamic scene. Then, a novel 3Dawware initialization method is proposed to learn a reasonable deformation field and new robust depth loss is proposed to guide the learning of dynamic scene geometry. Comprehensive experiments demonstrate that MoDGS is able to render high-quality novel view images of dynamic scenes from just a casually captured monocular video, which outperforms state-of-the-ed methods by a significant margin. Project page: http://MoDGGSgiihuub.o
INTRODUCTION
Novel view synthesis (NVS) is an important task in computer graphics and computer vision, which greatly facilitates downstream applications such augmented virtual reality. In recent years, the novel-view-synthest quality on static scenes has witnessed great improvements thanks to the recent development of techniques such as NeRF (Mildenhall et al., 2020), Instant-NGP (Müller et al., 2022), and Gaussian Splatting (Kerbl et al., 2023), especially when there are sufficient input images. However, novel view synthesis in a dynamic scene with only one monocular video still remains challenging task.
Dynamic View Synthesis (DVS) has achieved impressive improvements along with the emerging neural representations (Mildenhall et al., 2020) and Gaussian splatting (Kerbl et al., 2023) tech- niques. Most of the existing DVS methods (Cao Johnson, 2023; Yang et al., 2023) require mul- tiview videos captured by dense synchronized cameras to achieve good rendering quality. Though some works can process a monocular video for DVS, as pointed out by DyCheck (Gao al., 2022), these methods require the camera of the monocular video to have extremely large movements, which is called "Teleporting Camera Motion" on different viewpoints, so these methods can utilize the mul- tiview consistency provided by this pscudo multiview video reconstruct the 3D geometry of the dynamic scene. However, such large camera movements are rarely seen in casually captured videos because casual videos usually produced by smoothly moving camera moves slowly or is static, the multiview consistency constraint will be much weaker and all these existing DVS methods fail to produce high-quality novel-view images, as shown in Fig. 1.
static cameras. When the
In this paper, we present Monocular Dynamic Gaussian Splatting (MoDGS) to render novel-view images from casually captured monocular videos in a dynamic scene. MoDGS addresses the weak multiview constraint problem by adopting monocular depth estimation method (Fu et al., 2024), which provides prior depth information on the input video help the 3D reconstruction. However, we find that simply applying a single-view depth estimator in DVS to supervise rendered depth
Equul contribution. This project was primarily completed while Qingming was at CHtyUHK. Coorespponing Authors. Email: wt.hou@cityu.ede edk. This project was supported in part by the NSFC Excellent Young Scientists Fund 624221 18, and in part by the Hong Kong Research Grants Council under Grant 11219422 and Grant 11219324.

---

Published as a conference paper at ICLR 2025
Gaussians in
Inconsistent depth inputs
canonical space
Monocular Video
Deform field
3D-Aware init
3D-Aware init
w/O
Ordinal depth loss
Baseline
Ours
Consistent depth renderings
Figure I: Given casually captured monocular video of dynamic scene, MoDGS is able to synthe- size high-quality novel-view images in this scene. the middle column, the baseline method (Yang et al., 2023) fails to correctly reconstruct the 3D dynamic scenes this static monocular video. The white regions in cyan bounding boxes are not visible in the input video (red bounding boxes) there some artifacts for these invisible regions. In the rightmost column, the input estimated monocular depth is inconsistent (red bounding boxes); however, our proposed ordinal depth loss effectively ensures more consistent depth outputs. This loss enhances the accuracy and reliability of learning underlying geometry.
maps is not enough for high-quality novel view synthesis. First, the depth supervision only provides information for each frame but does not help to associate 3D points between two frames time. Thus, we still have difficulty in learning an accurate time-dependent deformation field. Second, the estimated depth values are not consistent among different frames.
To learn a robust deformation field from a monocular video, we propose a 3D-awar initialization scheme for the deformation field. Existing methods (Katsumata et al., 2024) solely rely on super- vision from 2D flow estimation, which produces deteriorated results without sufficient multiview consistency. We find that directly initializing the deformation field in the space greatly helps the subsequent learning of the 4D representations and improves the rendering quality as shown in Fig. 1.
To better utilize the estimated depth maps for supervision, we propose a novel depth loss to address the scale inconsistency of estimated depth values across different frames. Previous methods (Li et al., 2023b; Liu et al., 2023a) supervise the rendered depth maps using scale-invariant depth loss by minimizing the L2 distance of normalized rendered depth and depth priors, and the most recent method (Zhu et al., 2023c) proposed to supervise the rendered depth maps using a Pearson correlation loss mitigate the scale ambiguity between the reconstructed scene and the estimated depth maps. However, the estimated depth maps of different frames are not even consistent after normalizing to the same scale. To address these challenges, we observe that despite the inconsistency in values, the orders of depth values of different pixels in different frames are stable, which motivates us propose an ordinal depth loss. This novel ordinal depth loss enables us fully utilize the estimated depth maps for high-quality novel view synthesis.
To demonstrate the effectiveness of MoDGS, we conduct experiments on three widely used datasets, the Nvdia (Yoon et al., 2020) dataset, the DyNeRF (Li et al., 2022) dataset, and the Davis (Pont- Tuset al., 2017) dataset. We also present results on self-collected dataset containing monocular on-the-wild videos from the Internet. We adopt exact monocular DVS evaluation setting that only uses the video of one camera as input while evaluating the video of another camera. Results show that method outperforms previous DVS methods by large margin and achieves high-quality NVS on casually captured monocular videos.
RELATED WORK
In recent years, numerous works have focused on the task of novel view synthesis both static and dynamic scenes. The main representatives are Neural Radiance Field (Mildenhall al., 2020) and
2

---

Published as a conference paper at ICLR 2025
Gaussians in canonical space
Ordinal depth loss
Input video
Geo Wizard
time
Deform field
Tt
Rendered depth
Estimated depth
Splatting
Rendering loss
Rendered image
Figure 2: Overview. Given a casually captured monocular video of a dynamic scene, MoDGS represents the dynamic scene with a set Gaussians in a canonical space and deformation field represented by an MLPT. To render an image at a specific timestamp t, we deform all the Gaussians by Tr and then use the splatting technique to render images and depth maps. While in training MoDGS, we use single-view depth estimator GeoWizard (Fu et al., 2024) to estimate depth maps and compute the rendering loss and an ordinal depth loss for training.
Gaussian Splatting (Kerbl et al., 2023), along with their variants. In this paper, we primarily focus on view synthesis in dynamic scenes.
Dynamic NeRF Recent dynamic NeRF methods can be roughly categorized into two groups. 1) Representing by varying neural radiance fields conditioned on time (Gao et al., 2021; Li al., 2022: Park et al., 2023). For example, Park et al. (2023) proposes simple spatiotemporal radi- ance field by interpolating the feature vectors indexed by time. 2) Representing by a canonical space NeRF and deformation field (Guo et al., 2023; Li et al., 2021; Park et al., 2021a;;; Pumarola et al., 2021; Tretschk et al., 2021; Xian et al., 2021). For example, NSFF (Li et al., 2021) models the dynamic components using forward and backward flow represented as 3D dense vector fields; Nerfies (Park et al., 2021a) and HyperNeRF (Park et al., 2021b) model the scene dynamics as de- formation field mapping to a canonical space. Recent advances in grid-based NeRFs (Müller al., 2022: Sara Fridovich- Keil and Alex Yu et al., 2022; Chen et al., 2022) demonstrate that the training of static NeRFs can be significantly accelerated. Consequently, some dynamic NeRF works utilize these grid-based or hybrid representations for fast optimization (Guo et al., 2023; Cao & Johnson, 2023; Fang et al., 2022; Fridovich-Ken et al., 2023; Shao al., 2023; Wang al., 2023a;b; Song et al., 2023; You & Hou, 2023).
Dynamic Gaussian Splatting. The recent emergence of 3D Gaussian Splatting (3DGS) demon- strates its efficacy for super-fast real-time rendering attributed to its explicit point cloud represen- tation. Recent follow-ups extend 3DGS to model dynamic 3D scenes. Luiten et (2023) track dynamic 3D Gaussians by Crame-by-frame training from synchronized multi-view videos. Yang et al. (2023) propose a deformable version of 3DGS by introducing a deformation MLP network model the 3D flows. Wu et al. (2024) and Duisterhof et al. (2023) also introduce deforma- tion field but using a more efficient Hexplane representation (Cao Johnson, 2023). Yang et al. (2024c) proposes dynamic representation with collection of 4D Gaussian primitives, where the time evolution can be encoded by 4D spherical harmonics. Bae et al. (2024) encodes motions with per-Gaussian feature vector. Some other works (Li et al., 2023a; Lin et al., 2024; Liang et al., 2023) also study how to effectively encode the motions for Gaussians with different bases. To effectively learn the motions of Gaussians, some works (Feng et al., 2024; Yu al., 2023; Huang et al., 2024) resort to clustering the motions together for a compact representation.
DVS from Casual Monocular Videos. As shown in DyCheck (Gao al., 2022), many existing monocular dynamic view synthesis datasets used for benchmarking, like D-NeRF (Pumarola et al., 2021), HyperNeRF (Park et al., 2021b), and Nerfies (Park et al., 2021a), typically involve signifi- cant camera movements between frames but with small object dynamic motions. While this capture style helps with vuew constraints and dynamic 3D modeling. it is not representative of casual everyday video captures. When using casual videos, the reconstruction results from these methods suffer from quality degradation. Some works address dynamic 3D scene modeling using monocu- lar casual videos. DynlBaR (Li et al., 2023b) allows for long-sequence image-based rendering of
3

---

Published as a conference paper at ICLR 2025
Depth points
Canonical space
Time
Depth points &j
Donse Points
Deform Hold
Deform field
RAFT
Downsample
Tune 5g
Depth points e
Deform field
Defomm field
Spurse points
Xxy Lnex
Init Gaussians
Depth points Ey
(a) Initialization of deformation field
(b) Initialization of Gaussians
Figure 3: (a) Initialization of the deformation field. We first lift the depth maps and a 2D flow to 3D flow and train the deformation field for initialization. (b) Initialization of Gaussians in the canonical space. We use the initialized deformation field to deform all the depth points to the canonical space and downsample these depth points to initialize Gaussians.
dynamic scenes by aggregating features from nearby views, but its training cost is high for long per-scene optimization. Lee et al. (2023) proposes a hybrid representation that combines static and dynamic elements, allowing for faster training and rendering, though it requires additional per-frame masks for dynamic components. RoDynRF (Liu et al., 2023a) focuses on robust dynamic NeRF reconstruction by estimating NeRF and camera parameters together. DpDy (Wang et al., 2024a) enhances quality by fine-tuning diffusion model with SDS loss supervision (Poole et al., 2022), but it demands significant computational resources. Concurrent works like DG-Marbles (Stearns et al., 2024) use Gaussian marbles and a hierarchical learning strategy to optimize representations. Shape-of-Motte -oatoonnoooiio (Wang et al., 2024b) and Mosca (Lei et al., 2024) rely on explicit motion represen- tation and initialize scene deformation with depth estimation and video tracking priors. However, our MoDGS method effectively uses only noisy inter-frame flow maps from RAFT (Teed Deng, 2020) as input, performing well without the need for strong long-range pixel correspondence.
Ordinal Relation in Depth Maps. The ordinal relation between pixels has been investigated in recent years, especially in the field of monocular depth estimation. Zoran et al. (2015) proposed to use a chree-category classification network to predict the order relation of given pixel pairs. Then the depth can be extracted by optimizing constrained quadratic optimization problem. Similarly, Fu et al. (2018) treats the depth prediction problem as multi-class classification problem. Chen et al. (2016) further proposes a ranking loss to learn metric depth, which encourages small difference between depths if the ground- -tuuh relation is equal; otherwise encourages a large difference. Then, Pavlakos al. (2018) extends this differentiable ranking loss to the human pose estimation task. However, these works only utilize limited numbers of depth orders for training (one pair in Chen al. (2016) and 17 pairs in Pavlakos et al. (2018)), resulting in coarse supervision for depth maps. The direct application of their ranking loss as depth supervision has yet to be explored. Moreover, our ordinal depth takes the rendered metric depth maps as input, which are dense grids of float numbers. Our task is different from previous depth estimation and pose estimation tasks and we present a comparison between our ordinal depth loss and depth ranking loss Appendix A.
3 PROPOSED METHOD
Given a casually captured monocular video, we aim to synthesize novel view images from this video. We propose MoDGS, which achieves this by learning a canonical space and a deformation field T R + R² to deform these Gaussians to a specific timestamp Then, for a timestamp and a camera pose, we use splatting to render an image.
of Gaussians {G,li 1, N) in
Overview. As shown in Fig. 2, to train MoDGS, we split the monocular video into sequence of images [T = 1.... T} with known camera poses. We denote our deformation field as a function Jr = Tx(x), which maps a 3D location x R in the canonical 3D space to a location H, E R in the 3D space time t. For every image 4, we utilize a single-view depth estimator (Fu et al., 2024) to estimate a depth map D, for every image and utilize a flow estimation method RAFT (Teed & Deng, 2020) to estimate 2D optical flow Fi,tie between 4t, and I,, where t; and ty two arbitrary timestamps. Then, we initialize our deformation field by a 3D-aware initialization scheme as introduced in Sec. 3.2. After initialization, train Gaussians and deformation field with
4

---

Published as a conference paper at ICLR 2025
rendering loss and new depth loss introduced in Sec. 3.3. In the following, first begin with the definition of the Gaussians and the rendering process in MoDGS.
3.1 GAUSSTANS AND DEFORMATION FIELDS
Gaussians in the canonical space. We define a set of Gaussians in the canonical space, we follow the original 3D GS (Kerbl et al., 2023) to define 3D location, scale vector, a rotation, and color with spherical harmonics. Note this canonical space does not explictly correspond to any timestamp but is just virtual space that contains the canonical locations of all Gaussians.
Deformation fields. The deformation field Te used in MoDGS follows the design of Omnimo- tion (Wang et al., 2023c) and CaDeX (Lei & Danilidis, 2022) which is an invertible MLP net- work (Dinh et al., 2016). This is invertible MLP means that both Tj and T can be directly computed from the MLP network. All Tr at different timestamps share the same MLP network and the time is normalized to [0, 1] as input to the MLP network.
Render with MoDGS. After training both the Gaussians in canonical space and the deformation field, we will use the deformation field to deform the Gaussians in the canonical space to a specific time step t. Then, we follow exactly the splatting techniques in 3D GS (Kerbl et al., 2023) to render images from arbitrary viewpoints.
3.2 3D-AWARE INITIALIZATION
Original 3D Gaussian splatting (Kerbl et al., 2023) relies on the sparse points from Structure-from Motion (SfM) to initialize all the locations of Gaussians. When we only have a casually captured monocular video, it is difficult to get an initial set of sparse points for initialization from SIM. Though it is possible to initialize all the Gaussians from the points of the estimated single-view depth of the first frame, we show that this leads to suboptimal results. At the same time, we need to initialize not only the Gaussians but also the deformation field. Thus, we propose a 3D-aware initialization scheme for MoDGS.
Initialization of depth scales. Since the estimated depth maps on different timestamps would have different scales, we first estimate coarse scale for every frame to unify the scales. We achieve this by first segmenting out the static regions on the video and then computing the scale with a least square fitting (Chung et al., 2023b). The static regions on the 2D flow (Teed & Deng, 2020) or segmenting with a segmentation method like SAM2 (Ravi et al., 2024). Then, on these static regions, we reproject the depth values specific timestamp to the first frame and minimize the difference between the projected depth and the depth of the first frame, which enables us to solve for scale for every frame. We rectify all depth maps with the computed scales. In the following, we reuse D, to denote the rectified depth maps by default.
be determined by either thresholding
Initialization of the deformation field. As shown in Fig. 3 (left), given two depth maps D., and Dr along with the 2D flow Fe.n,ty we lift them to a 3D flow r&b -t, This is achieved by first converting the depth maps into 3D points in the 3D) space. Then, the estimated 2D flow Fi,-it, actually associate two sets of 3D points, which results in 3D flow F After getting this 3D flow, we then train our deformation field T with this 3D flow. Specifically, for pixel 4t, whose corresponding 3D point is zt,, we query t to find its target point 2t, in the ty timestamp. Then, we minimize the difference by
iini EIIT, e TT, (I.)) - -
We train the MLP in T for a fixed number of steps to initialize the deformation field.
Initialization of Gaussians. After getting the initialized deformation field, we will initialize a set of 3D Gaussians in the canonical space as shown in Fig. 3 (right). We achieve this by first converting all the depth maps to get 3D points. Then, these 3D points are deformed backward to the canonical 3D space. This means that we transform all the depth points of all timestamps to the canonical space, which results in a large amount of points. We then evenly downsample these points with predefined voxel size to reduce the point number and we initialize all our Gaussians with the locations of these downsampled 3D points in the canonical space. Here, more advanced learning- based adaptive fusion strategies (You et al., 2023) could be adopted to downsample the points, potentially improving the representation.
5

---

Published as a conference paper at ICLR 2025
3.3 ORDINAL DEPTH LOss
Pearson correlation loss. Existing dynamic Gaussian Splatting or NeRF methods also adopt depth loss to supervise the learning of their 3D representations. One possible solution (Zhu et al., 2023c; Li et al., 2021; Liu al., 2023a) is maximize Pearson correlation between the rendered depth and the estimated single-view depth
Corr (.,,
Coo( Dc) VVrr(Dy (D)) Var(D,)
where Cov (t--) and Var (+:-) means the covariance and variance respectively, Dr and Dr are the estimated depth and the rendered depth respectively. Since the estimated single-view depth has ambiguity in scale, the Pearson correlation loss avoids the negative effects of the scale ambiguity. Note that in Li et al. (2021) and Liu et al. (2023a), the loss is called normalized depth loss, which is equivalent to Pearson correlation here as shown in the supplementary material.
Limitations of Pearson corre- lation loss. However, we find that this Pearson correlation depth loss is still suboptimal. As shown in Fig. 4, the esti- mated depth maps at two differ- ent timestamps are still not con- sistent with each other after nor- malization. Making two depth maps consistent after normal- ization actually requires these two depth maps be related by a linear transformation, i.e. aD; and b two constants. How- ever, the single-view depth es- timation method is not accu- rate enough guarantee the lin- ear relationship between two es- timated depth maps at differ- ent timestamps. In this case, the Pearson correlation loss still brings inconsistent supervision.
Depth map Du
Depth map Dey
U/
Mg
24g
14c
Diy(
Dr (na)
Dr Dr,(u)
D++1
with
Dr, (ug)
Dr (Mt)
No linear consistent De, (u (mn) Dr, (ua)) (Ha (u,)) (un) De, Dr,(Ma)- (ay,)aaDD, (u,(
Order consistent D, (u,)) Dr, (u.) > Dr, (ug) Dr, (ug)) (e (u6)> Dr, (ws)
(ne)
Dr((u)
Figure 4: We show the estimated single-view depth maps at two different timestamps Dr. and Di, after normalization to the same scale. Since the single-view depth estimator is not accurate enough, the depth maps are not linear related so the scale nor- malization does not perfectly align them. However, the order of depth values three corresponding pixels is stable for these two depth maps, which motivates us to propose an ordinal depth loss for supervision.
Ordinal depth loss. To ad- obser-
dress this problem, vation is that though we cannot guarantee depth consistency af- ter normalization, as shown in Fig. 4, the order of depth value is consistent among two different frames. Thus, this motivates us to ensure the order of depth is correct by new ordinal depth loss. We first define an order indicator function
R(De(ui), De(wz)) = { +1, D.(un) > Di(u2) De(uz) < (1(
(2)
where R is the order indicator function on depth map D; which indicates the order between the depth values of pixels #1 E RR² and u2 E R.,, and (u(u) means the depth value on the pixel u. Then, we define our ordinal depth loss based on the depth order by
Lodtiall = |[tanh (<(D((w))-D(())))) R (D((())) ll.
(3)
where tanh(x) ---e== D. means the rendered depth map timestampt t, (v) is the depth value c*
of this rendered depth map the pixel u, is predefined constant. Eq. 3 means transform the depth difference between D((u) and (u to or by tanh function. Then, force the depth order of the rendered depth map Ú, to be consistent with the order in the predicted depth map D.. In the implementation, we randomly sample 100k pairs (u1, u2) to compute the ordinal depth loss.

---

Published as a conference paper at ICLR 2025
Input View
GT- -m-lmage
SC-GS
Deformable-GS
HexPlane
Ours
Figure 5: Qualitative comparison the novel-view renderings of the DyNeRF (Li et al., 2022) and Nvidia (Yoon et al., 2020) datasets. We compare MoDGS with SC-GS (Huang et al., 2024), Deformable-Gs (Yang al., 2023), and HexPlane (Cao & Johnson, 2023).
Input View
SC-GS
DeformableGG8
HexPlane
RoDynRF
Ours
Figure 6: Qualitative comparison of DVS quality on the MCV dataset. We compare MoDGS with SC-GS (Huang et al., 2024), Deformable-Gi (Yang al., 2023), HexPlane (Cao & Johnson, 2023), and RoDynRF (Liu al., 2023a).
3.4 TRAINING OF MoDGS
After initializing the Gaussians and the deformation fields, MoDGS to render at specific timestamp and compute the rendering loss frender and the ordinal depth loss fontial- So the total training loss for MoDGS is
È = Fordinal&ordin + Avender Egender-
(4)
EXPERIMENTS
4.1 EVALUATION PROTOCOLS
Datasets. We conducted experiments four datasets to demonstrate the effectiveness of our method. The first dataset is the DyNeRF (Li et al., 2022) dataset which consists 6 scenes. On each scene, we have 18-20 synchronized cameras capturing 10-30 second videos. In these videos, there is

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative results on the DyNeRF (Li et al., 2022) and Nvidia (Yoon et al., 2020) datasets. We compare our method with SC-GS (Huang et al., 2024), Deformable GS (Yang et al., 2023) (D-GS) and HexPlane (Cao & Johnson, 2023) in PSNR SSIM1, and LPIPS For a per-scene breakdown of the metrics, please refer to Table 6 and Table 7 in A.9.
DyNeRF
Nvidia
Methods
PSNR
SSIM
LPIPS|
PSNRT
SSIM
LPIPS|
HexPlane
15.33
0.5593
0,4514
17.17
0.3675
0,4756
SC-GS
18.77
0.7359
0,2310
17.59
0.4679
0.3348
D-GS
19.55
0.7446
0,2171
18.07
0,4650
0.3422
Ours
22.64
0.8042
0.1545
19.27
0.5235
0.2581
mainly a working on desktop, like cutting beef dumping water. We use camera0 for train- ing and evaluate the results on camera5 and camera6. The second dataset is the Nvidia (Yoon et al., 2020) dataset which contains more diverse dynamic subjects like jumping, playing with balloons, and opening an umbrella. The Nvidia dataset contains 8 scenes, which also has 12 synchronized cameras. We train all methods on camera4 and evaluate with camera3 and camera5. Besides, we also collect 6 online videos to construct an in-the-wild dataset, called the Monocular Casual Video (MCV) Dataset, to demonstrate our method can generalize to in-the-wild casual videos. The MCV dataset contains diverse subjects like skating, a dog eating food, YOGA, etc. The MCV dataset only contains a single video for each scene, so we cannot evaluate the quantitative results but only report the qualitative results on this dataset. We also present results of the Davis dataset -ost-Tusstt al., 2017) in Sec. A.6 of the appendix.
Evaluation setting. Previous DVS methods (Cao & Johnson, 2023; Yang et al., 2023; Gao et al., 2021) all use different cameras train the dynamic NeRF or Gaussian Splatting. Even though they only use one camera at a specific timestamp, they use different cameras at different timestamps so that a pseudo multiview video can be constructed learn the 3D structures of the scene. Since our target is to conduct novel-view synthesis on the casually captured images, we do not adopt such "teleporting camera motions" to construct training videos but just adopt one static camera to record training videos. Then, we render the images from the viewpoints of another camera for evaluation.
Metrics. To evaluate the rendering quality, we have render images from a new viewpoint and compare them with the ground-truth images. However, if the input video is almost static, the input video will contain insufficient 3D) information and there would exist ambiguity in scale. Thus, different DVS methods would choose different scales in reconstructing dynamic scenes so that the rendered images on the novel viewpoints are not aligned with the given ground-truth images. To address this problem, we manually label correspondences between the training images and ground- truth novel-view images. Then, we render depth map on the training image using the reconstructed dynamic scene and optimize for a scale factor to scale the depth value to satisfy these labeled corre- spondences. After aligning the scale factors of different methods with the ground-truth images, we compute the SSIM, LPIPS, and PSNR between the rendered images and the ground-true -ruth images.
Baseline methods. We compare MoDGS with 4 baseline methods demonstrate the superior ability of MoDGS to synthesize novel-view images with casually captured monocular videos. These methods can be categorized into two classes. The first is the NeRe-based methods including Hex- Plane (Cao & Johnson, 2023) and RoDynRF (Liu et al., 2023a). HexPlane represents the scene with six feature planes in both 3D space and time-space. We find that HexPlane does not produce rea- sonable results if only a monocular video from a single camera is given as input. Thus, other than the input monocular video, we use another video from different viewpoint to train HexPlane for the DVS task. RoDynRF is SoTA beRe-based DVS method that also adopts single-view depth estimation as supervision for the 3D dynamic representations. We train it with the same single-view depth estimator Geo Wizard (Fu et al., 2024) as ours. The second class is the Gaussian Splatting- based DVS methods including Deformable-eeeee GS (Yang et al., 2023) and SC-GS (Huang et al., 2024). Deformable-GS also associates deformation field with set of canonical Gaussians for DVS. SC- GS learns a set of keypoints and uses the deformation of these keypoints to blend the deformation of arbitrary 3D points.
8

---

Published as a conference paper at ICLR 2025
4.2 COMPARISON WITH BASELINES
The qualitative results on the DyNeRF and Nvidia datasets are shown in Fig. 5. Other qualitative results on our MCV dataset are shown in Fig. 6. The quantitative results on the Nvidia and DyNeRF datasets are shown in Table 1. Supplementary videos contain more comparison results.
Synthesizing novel views from casually captured monocular video is a challenging task. As shown in Fig. 5, though baseline methods achieve impressive results on these benchmarks with "teleporting camera motions", these methods fail to correctly reconstruct the 3D geometry of the dynamic scenes and produce obvious artifacts on both dynamic foreground and static background. The main reason is that the monocular camera is almost static and does not provide enough multiview consistency to reconstruct high-quality 3D geometry for novel view synthesis. In the third row of Fig. 5, SC- GS (Huang et al., 2024) fails to reconstruct the dynamic foreground subject because SC-GS has an initialization process that treats the whole scene static scene and trains the scene for number of steps. When the foreground subject is moving with a large motion (like skating from left to right), it would be ignored by the static scene initialization and then fail reconstruct the subsequent steps.
In comparison, our method relies on a 3D-aware initialization which provides a strong basis for the subsequent optimization. Meanwhile, our ordinal depth loss enables the 3D prior from the single- view depth estimator for an accurate reconstruction of the dynamic scenes. The quantitative results in Table also show that our method achieves the best performances in all metrics both datasets. Note that there are still some artifacts on occlusion boundaries because the input monocular camera is almost static and these regions are not visible in our input videos.
4.3 ABLATION STUDIES
We conduct ablation studies with initialization and depth loss on the DyNeRF (Li et al., 2022) dataset to demonstrate their effectiveness. The qualitative results are shown in Fig. 7 and Fig. while the quantitative results are shown in Table 2. In the appendix, with depth warping, robustness to depth noises, a discussion on video depth, a comparison with other depth ranking losses, and discussion about alpha values.
also provide comparison
4.3.1 3D-AWARE INITIALIZATION
To show the effectiveness of our 3D-aware initial- ization, we adopt ran- dom initialization for the deformation field. Based on the random initializa- tion, we still deform all the depth points backward to the canonical space and downsample these points to initialize the Gaussians. Then, we follow the exact same training procedure to train the randomly initial- ized baseline method. The final results of this random initialization are shown in Fig. 7. In comparison with our 3D-aware initialization, this random ini- tialization produces more artifacts on the dynamic foreground human, which demonstrates that our 3D-aware initialization provides a good initial point for subsequent 3D dynamic scene reconstruc- tion.
3D Awwre Init
Random Init
Figure 7: Visualization of rendering results using and without using
our 3D-aware initialization.
4.3.2 ORDINAL DEPTH LOSS
To demonstrate the effectiveness our proposed ordinal depth, we experiment with two baseline settings, removing the depth loss and utilizing the Pearson correlation as the depth loss. As shown in Fig. 8, when there is no depth loss, the reconstructed 3D geometry contains much noise and obvious artifacts exist in the scene. Since we perform the 3D-aware initialization ensure a fair comparison, the results without depth loss still show reasonable rendered depth maps. Adopting the Pearson depth loss improves the quality by linear correlating the rendered depth maps
this experiment to

---

Published as a conference paper at ICLR 2025
and input depth maps but still produces noisy depth inside region. In comparison, our ordinal depth loss enables a smooth reconstruction of the depth map in the interior while maintaining sharp edges at boundaries. Thus, the proposed ordinal depth loss enables more robust reconstruction of the dynamic scene.
Input Truth Figure 8: Visualization of the rendered depth and RGB images using our ordinal depth loss and the Pearson correlation loss.
No Depth Loss
Pearson Correlation Loss Ordinal Depth Loss
4.4 LIMITATIONS
Though method can conduct dy- namic view synthesis from casually captured monocular videos, the task is still extremely challenging. One limitation is that our method can only reconstruct the visible 3D parts but cannot imagine the unseen parts, which leads to artifacts when render- ing novel view videos on these un-
Table 2: Ablation studies with the 3D-aware initialization
("3D-aware Init") and Depth Loss on the DyNeRF (Li et al., 2022) dataset. "Ordinal" means the ordinal depth loss while "Pearson" means the Pearson correlation loss.
3D-aware Init
Loss
PSNR
SSIM
LPIPS,
X
Ordinal
21.27
0.7655
0.1984
Pearson
21.77
0.7938
0. 1680
Ordinal
22.96
0.8103
0.1518
seen parts. Incorporating recent 3D-related diffusion generative models (Chung et al., 2023a; Liu et al., 2023b; Long et al., 2023) could be a promising direction to solve this problem, which we leave for future works. Another limitation is that the current training time is comparable with exist- ing DVS methods, which could take several hours for a single scene. How to efficiently reconstruct the dynamic field would be an interesting and promising future research topic. Meanwhile, when the camera is completely static, our method strongly relies on the single-view depth estimator to estimate the 3D depth maps. Though existing single-view depth estimators (Fu et al., 2024; Ke et al., 2023; Yang et al., 2024a; Bhat et al., 2023) are trained on large-scale datasets and predict reasonable depth maps for most cases, these depth estimators may fail to capture some details which degenerate the quality. Another limitation is that we assume consistent depth order, though this as- sumption is satisfied by most methods (Yang et al., 2024b; Bochkovskii et al., 2024; Ke al., 2023; Fu et al., 2024). Additionally, challenge arises in rapidly moving scenes. Such rapid motions may result in inaccurate camera pose estimation, which is still challenging for MoDGS and all existing dynamic Gaussian reconstruction methods. Mcanwhile, scenes featuring heavy specular reflections and low-tight conditions cannot be well handled, which is also an extremely challenging problem in the monocular setting, which we leave for future works.
CONCLUSION
In this paper, we have presented novel dynamic view synthesis paradigm called MoDGS. In com- parison with existing DVS methods which requires "teleporting camera motions". MoDGS is de- signed to render novel view images from a casually captured monocular video. MoDGS introduces two new designs to finish this challenging task. First, new 3D-aware initialization scheme is proposed, which directly initializes the deformation field to provide a reasonable starting point for subsequent optimization. We further analyze the problem of depth loss and propose new ordinal depth loss supervise the learning of the scene geometry. Extensive experiments three datasets demonstrate superior performances of our method on in-the-wild monocular videos over baseline methods.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jeongmin Bae, Seoha Kim, Youngsik Yun, Hahyun Lee, Gun Bang, and Youngjung Uh. Per- gaussian embedding- deformation for deformable 3d gaussian splatting. In European Con- ference Computer Vision (ECCV), 2024.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 2023.
Aleksei Bochkovskii, Amael Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than second. arXiv, 2024. URL https: : http///avvvvooo www.oogg///ooo.oooooooo....o.os/ /at.////..... 02073.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 130-141, 2023.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, pp. 333-350. Springer, 2022.
Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild.
Advances in neural information processing systems, 29, 2016.
Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprini arXiv:2311.13388 2023a.
Jaeyoung Chung. Jeongtaek Oh, and Kyoung Mu Lee. Depth-regularized optimization for 3d gaus- sian splatting in few-shot images. arXiv preprint arXiv:2311.13398 2023b.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv::6055 160.008833, 2016.
Bardienus P Duisterhof, Zhao Mandi, Yunchao Yao, Jia-Wei Liu, Mike Zheng Shou, Shuran Song, and Jeffrey Ichnowski. Md-splatting: Learning metric deformation from 4d gaussians in highly deformable scenes. arXiv preprint arXiv:2312.00583 2023.
Cheng-De Fan, Chen Then-Wei Chang, Yi-Ruei Liu, Jie- Ying Lee, Jiun-Long Huang, Yu-Chee Tseng, and Yu-Lun Liu. Spectromotion: Dynamic 3d reconstruction of specular scenes. arXiv preprint 1142:994...17249 2024.
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nicßner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022.
Qi- Yuan Feng, Hao-Xiang Chen, Qun-Ce Xu, and Tai-Jiang Mu. Sls4d: Sparse latent space for 4d novel view synthesis. IEEE Transactions Visualization and Computer Graphics, pp. 1-14, 2024. doi: 10. 10.10997VCC..2224332994111
Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12479-12488, 2023.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 2002-2011, 2018.
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In ECCV, 2024.
Chen Gao, Ayush Saraf. Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5712-5721,, 2021.
11

---

Published as a conference paper at ICLR 2025
Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dy- namic view synthesis: A reality check. Advances in Neural Information Processing Systems, 35: 13768-337801 2022.
Xiang Guo, Jiadai Sun, Yuchao Dai, Guanying Chen, Xiaoqing Ye, Xiao Tan, Errui Ding, Yu- meng Zhang, and Jingdong Wang. Forward flow for novel view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16022-16033, 2023.
Yi-Hua Huang, Yian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc- gs: Sparse-controlled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 4220 4230, 2024.
Kai Katsumata, Duc Minh Vo, and Hideki Nakayama. A compact dynamic 3d gaussian representa- tion for real-time dynamic view synthesis. ECCV, 2024.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- based image generators for monocular depth estimation. arXiv preprint arXiv:2312.02215, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphies, 42(4), 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Yao-Chih Lee, Zhoutong Zhang, Kevin Blackhurn-Matzen, Simon Niklaus, Jianming Zhang, Jia-Bin Huang, and Feng Liu. Fast view synthesis of casual videos. arXiv preprint arXiv:2312.02135, 2023.
Jiahui Lei and Kostas Daniilidis. Cadex: Learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6624-6634, 2022.
Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Danilidiis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. arXiv preprint arXiv:2405.17421, 2024.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5521-5523 2022. 2022.
Zhan Li, Zhang Chen, Zhong Ii, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. arXiv preprint arXiv:2312 16812, 2023a.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neu- ral dynamic image- based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4273-4284, 2023b.
Yiqing Liang, Numair Khan, Zhengqin Li, Thu Nguyen- Phuuo,, Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. arXiv preprint urXiv:2312.1144 1445,, 2023.
Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian- flow: 4d reconstruction with dynamic 3d gaussian particle. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pp. 21 1145, 2024.
12

---

Published as a conference paper at ICLR 2025
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng. Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia- Baa-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 13-23 2023a.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint N3Nv:23309.3353,, 2023b.
Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingje Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, and Wenping Wang. Wonder3d: Single image to 3d using cross-domain diffusion, 2023.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv22308.09713 2023.
Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation. ACM Transactions on Graphics (ToG), 39(4))11-1, 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pp. 405-421. Springer, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1-15, 2022.
Jinshan Pan, Haoran Bai, and Jinhui Tang. Cascaded deep video deblurring using temporal sharpness prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3043-3051, 2020.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Scitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865-58744 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla and Steven Seitz. Hypernerf: A highee-ddimennsionall representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106. 13228, 2021b.
Sungheon Park, Minjung Son, Seokhwan Jang, Young Chun Ahn, Ji-Yeon Kim, and Nahyup Kang. Temporal interpolation is all you need for dynamic neural radiance fields. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp. 4212-42211 2023.
Georgios Pavlakos, Xiaowei Zhou, and Kostas Danilidiss Ordinal depth supervision for 3d human pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 7307-7316, 2018
Luigi Piccinelli, Yung-Hsu Yang. Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. UniDepth: Universal monocular metrie depth estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and
Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv: /70.00/70000075. 2017.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209, 14988, 2022.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEFF/CVF Conference on Computer Vision and Paltern Recognition, pp. 10318-1037,, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint 2024. URL https: / rxxiv org/ abs 2400 00714.
13

---

Published as a conference paper at ICLR 2025
Sara Fridovich-Kei and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi
Liao. Learning temporally consistent video depth from video diffusion priors, 2024.
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Ten- sor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16632-16642, 2023.
Liangehen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and An- dreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Gruphics, 29(5):2332-2742, 2023.
Colton Stearns, Adam W Harley, Mikaela Uy. Florian Dubost, Federico Tombari, Gordon Wetzstein, and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402-419 Springer, 2020.
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 16558-16569, Cur- ran Associates, Inc., 2021. URL http::: 8 htt.aa//e////0000//0/0000 /tt.://///.../...000000666662222 Patpeaddc.....e.eeeee........
neurips cctccc///aae sec/paper_fille
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Con-
ference Computer Vision, pp. 12959-12970, 2021.
Chaoyang Wang, Peiye Zhuang, Aliaksandr Siarohin, Junli Cao, Guocheng Qian, Hsin- Hsin-Ying Lee, and Sergey Tulyakov. Diffusion priors for dynamic view synthesis from monocular videos. arXiv preprint arXiv:24001 105583, 2024a.
Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed neural vox- els for fast multi-view video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19706 19706-19716, 2023a.
Liao Wang, Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, and Minye Wu. Neural residual radiance fields for streamably free-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 76-87, 2023b.
Qianqian Wang, Yen-Yu Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski. and Noah Snavely. Tracking everything everywhere all once. In Proceedings of the IEEE/CVF 2023c.
International Conference on Computer Vision, pp.
Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. 2024b.
Yiran Wang, Min Shi, Jiaqi Li, Zihao Huang, Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. Neural video depth stabilizer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9466-9476, October 2023d.
Yiran Wang, Min Shi, Jiaqi Li, Chaoyi Hong, Zihao Huang, Juewen Peng, Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. Nvds Towards efficient and versatile neural stabilizer for video depth estimation. IEEË Transactions on Pattern Analysis and Machine Intelligence, pp- 1-18, 2024c. doi: (0.1100//AAMM.2044444500
14

---

Published as a conference paper at ICLR 2025
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20310- 20320, June 2024.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9421-9431, 2021.
Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, and Marc Pollefeys. Depthsplat: Connecting gaussian splatting and depth. arXiv preprint arXiv:2410.13862. 2024.
Lihe Yang, Bingyi Kang. Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024a.
Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.094 09414, 2024b.
Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene rep- resentation and rendering with 4d gaussian splatting. In International Conference on Learning Representations (ICLR), 2024c.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.132 13101, 2023.
Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from monocular camera. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp- 5336-5345, 2020.
Meng You and Junhui Hou. Decoupling dynamic monocular videos for dynamic view synthesis. arXiv preprint arXiv:2304.01716 2023.
Meng You, Mantang Guo, Xianqiang Lyu. Hui Liu, and Junhui Hou. Learning a locally unified 3d point cloud for view synthesis. JEEE Transactions on Image Processing, 32:5610-5622, 2023. doi: 10.1 10.11099/1.22223333111555
Heng Yu, Joel Julin, Zoltán Á Milacski, Koichiro Niiumma, and László A Jeni. Cogs: Controllable gaussian splatting. arXiv preprint arXiv:2312.05666 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Qi Zhu, Naishan Zheng, Jie Huang, Man Zhou, Jinghao Zhang, and Feng Zhao. Learning spatio- temporal sharpness map for video deblurring. IEEE Transactions on Circuits and Systems for Video Technology, 2023a.
Qi 7hu, Man Zhou, Naishan Zheng, Chongyi Li, Jie Huang, and Feng Zhao. Exploring temporal frequency spectrum in deep video deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12428-12437, 2023b.
Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. Fsgs: Real-time few-shot view synthe- sis using gaussian splatting. arXiv preprint trXiv:23/2.0045.. 2023c.
Daniel Zoran, Phillip Isola, Dilip Krishnan, and William T Freeman. Learning ordinal relationships for mid-level vision. In Proceedings of the IEEE international conference on computer vision, pp. 388-396, 2015.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLEMENTATION DETAILS
We implement MoDGS with PyTorrc. To initialize the deformation field, train it with 20k steps as stated in Sec. 3.2. Subsequently, we jointly train the 3D Gaussians and the deformation field with the rendering loss and the ordinal depth loss for another 20k steps. In Sec. 3.2, the flow is computed in evenly sampled key frames(e.g. 1/5). And the downsampling voxel size for Gaussian initialization is 0.004 (scenes are normalized loop and rendering loss, we exactly follow the original 3DGS. And we use Gaussian centers to render depth (Yang et al., 2023). We adopt an Adam optimizer for optimization. The learning rate for 3D Gaussians exactly follows the official implementation of 3D GS (Kerbl et al., 2023), while the learning rate of the deformation network undergoes exponential decay from le-3 le-4 in initialization and from le-4 to lc-6 in the subsequent optimization. We set
[-1, 1]3 ). For the outer optimization
100 for Continal- The weight of our depth order loss is 0.1. When computing depth ordinal loss, we first normalize the depth range to [0,1] and only consider the depth pair with difference larger than 0.02 for loss computation. The whole training takes around 3.5 hours to converge (2 hours for the initialization and 1.5 hours for the subsequent optimization) on an NVIDIA RTX A6000 GPU, which uses about 14G memory. The rendering speed of MoDGS is about 75 FPS.
A.2 REAL DEPTII RECOVERY
The depth prediction of GeoWizard (Fu al., 2024) normalized depth values in [0.1] and we have to transform them into real depth values. We follow their official implementation estimate scaling factor and an offset value on the normalized normal maps by minimizing the normal maps estimated from the transformed real depth values and the estimated normal maps from Geo Rizard.... The optimization process takes just several seconds. Note that even after this normalization, the depth maps of different timestamps still differ from each other in scale.
Figure 9: We provide a pair of images on the iPhone (Gao et al., 2022) dataset. We draw two correspondences in the same color and their corresponding epipolar lines that computed from the provided camera poses. The epipolar lines deviate far from the correspondences, which demonstrate that the camera poses are not accurate enough.
A.3 POSE ERRORS IN THE IPHONE DATASET
DyCheck (Gao et al., 2022) adopts the iPhone dataset as the evaluation dataset for the DVS on casually captured videos. We do not adopt this dataset because find that the poses this dataset
16

---

Published as a conference paper at ICLR 2025
POAL
Depth Warping
Ours
Figure 10: Comparison of our renderings and depth warping. Our method accumulates information
among different timestamps and thus is able to render more completed images.
are not very accurate. An example is shown in Fig. where we draw two correspondences and their corresponding epipolar line in the same color. The epipolar line is computed from the provided camera poses. As we can see, the epipolar line does not pass through the correspondence, which means that the provided poses are not accurate enough. We have tried rerun COLMAP the iPhone dataset but cannot get reasonable results.
A.4 DIFFERENCE FROM DEPTH MAP WARPING
MoDGS learns a set of 3D Gaussians in a canonical space and deformation field to transform it arbitrary timestamp. This means that MoDGS is able to accumulate information among different timestamps to reconstruct more completed scene than just using a single-view depth estimation. We show the difference between our renderings and just warping the training view using the estimated single-view depth map in Fig. 10. As we can see, MoDGS produces more completed reconstruction on contents that are not visible on this timestamp. Meanwhile, MoDGS rectifies the single-view depth maps to be more accurate so the rendering quality is much better.
A.5 PEARSON DEPTH LOSS AND SCALE-SHIFT INVARIANT DEPTH LOSS
In this section, we will prove that the Pearson Depth loss is equivalent to Scale-Shift Invariant loss. Previous works (Li et al., 2021; Liu et al., 2023a) use scale and shift invariant depth loss by minimizing the L2 distance of normalized ground truth depth map Norm and the normalized rendered depth map Norm (D.)
Edeph | (D:) Norm(D,)
(5)
17

---

Published as a conference paper at ICLR 2025
Dj UD.
where No) denotes
JD:
D. UD,
Dj uD
OD:
D
(6)
2(D, - uDD (D.))DD uD
D; u (Dr- De-uDi + (D. D.
Dr
2
OD
DD.
where Dr is the predicted depth prior, dD, and uD, denotes the standard deviation and means re- spectively. Since it is based the input, The second term is constant. For the third term,
Σ
D " N
substitute the o, using its definition: oB
The sum of the third term over
Di up - TD,
all pixels in depth map is: EN(
N. Where N is the total number of pixels.
We can see that both the second term and third term are constant. The first term is a simple trans- formation of the Pearson correlation coefficient. Thus, minimizing the L2 distance is equivalent to maximizing the coefficient.
A.6 RESULTS ON THE DAVIS DATASET
A.6.1
PREPROCESSING PROCEDURE
On the Davis dataset, we adopt the preprocessing procedure outlined in the concurrent method, Shape-of-Motion (Wang et al., 2024b), to obtain these camera poses. Specifically, we first run UniDepth (Piccinelli al., 2024) to acquire camera intrinsies and depth maps. Then, we utilize SLAM method, Driod-SLAM (Teed & Deng, 2021), with the depth maps from UniDepth input to solve for the camera poses.
A.6.2 COMPARISON WITH SHAPE-OF-MOTION
We provide an experiment to compare Shape-of-Motion with our method. our experiments, train shape-of-motion on the Davis dataset following their default training setting and use the same preprocessing strategy like depth estimation train MoDGS. The visual comparison shown in Fig. 11, which demonstrates that our method shows comparable results as Shape-of-Motion but renders more details in some cases. also present the quantitative comparison results on Nvidia dataset (Yoon et al., 2020) in Table 3.
Table 3: Comparison with Shape-of ooappe-ff-Mannn oftoooofooooooMMMMMM on Nvidia dataset
Balloon2-2
Jumping
Truck-2
Skatting2
Method
PSNR
SSIM
PSNR
SSIM
LPIPS
SSIM
LPIPS
PSNR
SSIM
LPIPS
0.5096
0.2790
20.32
0.6771
0.2560
22.53
0.7414
0.1853
23.51
0.7861
0.1658
Ours
20.47
0,5275
0.2408
22.18
0.7075
0.2125
23.69
0.7455
0.1551
25.64
0.7996
0.1518
A.7 DISCUSSION ABOUT ALPHA VALUE IN ORDINAL DEPTH LOSS
After roughly aligning the depth maps, we normalize the scene to [-1, 1] and empirically set alpha to 100. We present further ablations in Table 4, where decreasing alpha leads to worse results while increasing alpha yields comparable results. The reason that large alpha values compel the loss to mainly concentrate the ordinality rather than the differences in depth values.
A.8 DISCUSSION ABOUT THE ROBUSTNESS TO THE DEPTH NOISES
To show the robustness of our ordinal depth loss to depth noise, we manually add Gaussian noises to depth maps and use the noisy depth maps as supervision signals. The metric results of them
18

---

Published as a conference paper at ICLR 2025
Input View
Shape-of-Motion
Ours
Figure 1I: We show a visual comparison rendered novel view images and depth maps between Shape-of MotooofffooooooWaaaeeeeae al., 2024b) and MoDGS. The rendering quality is comparable but our method shows more details some cases.
Table Quantitative results for different alpha values.
Setting PSNR
LPIPS SSIM
alpha=10
21.05
0.7575
0,2370
alpha=40
21.39
0.7813
0.1916
alpha=100(Ours)
23.23
0.8083
0.1592
alpha=1000
23.16
0.8058
0.1715
shown in Table 5 (on the "flame_steak" scenee) which shows our method is robust to small depth noises.
Table 5: Quantitative results with different levels of added Gaussian noise Norm(H, o)
Setting
PSNR
SSIM
LPIPS
Norm (0, 0.20)
23.18
0.8090
0.1667
Norm(O, 0.05)
23.35
0.8100
0,1633
Ours
23.23
0.8083
0.1592
A.9 PER-SCENE BREAKDOWN RESULTS THE DYNERF (LI ET AL., 2022) DATASET AND THE NVIDIA (YOoN ET AL., 2020) DATASET
In Tables 6 and Table 7. we provide a breakdown of the metrics for different scenes in the DyNeRF and Nvidia datasets, respectively.
19

---

Published as a conference paper at ICLR 2025
Table 6:
Per-scene results the DyNeRF dataset.
Method
cut_roasted beef
sear bear_steak
coffee_martin
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
HexPlane
16.76
0.5382
0.5054
16.89
0.5897
0.5049
13.26
0,4049
0.5835
SC-GS
20.69
0.7414
0.2625
21.23
0.7870
0.2188
19.02
0.7124
0.2151
D-GS
22.20
0.7808
0.1931
23.56
0.8101
0.1773
19.23
0.7013
0.2270
Ours
23.98
0.8221
0.1438
23.53
0.8126
0.1642
21.37
0.7962
0.1473
Method
cook-spinach
flame steak
Hame-salmon.1
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
HexPlane
16.95
0.7286
0.2223
16.97
0.7528
0.2543
11.16
0.3417
0.6382
SC-GS
16.70
0.7377
0.2117
17.31
0.7532
0.2527
17.65
0,6834
0.2253
D-GS
17.20
0.7195
0.2329
16.62
0.7523
0.2559
18.48
0.7038
0,2166
Ours
22.40
0.7823
0.1728
23.23
0.8083
0.1592
21.33
0.8038
0.1399
Table 7: Per-scene results on the Nvidia dataset.
Method
Playground
DynamicFace-2
Balloon1-2
Umbrella
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
HexPlane
13.30
0.1641
0.5240
9.92
0.1873
0.6091
15.48
0.2793
0.5342
18.70
0.2238
0.4427
SC-GS
12.60
0.2059
0.5063
0.3312
16.63
0.3656
18.71
0.3501
0.3663
D-GS
12.62
0.2009
0.5116
10.36
0.3277
0.4563
10.36
0.3277
0.4563
19.16
0.3381
0.3300
Ours
13.38
0.2343
0.4542
12.40
0.4118
0.2442
16.93
0.4371
0.3059
19.47
0,3246
0,3004
Method
Balloon2-2
Skatting2
Truck-2
Jumping
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
HexPlane
18.42
0.3159
0.4881
21.39
0.6403
0.3983
20.85
0.5687
0.3653
19.28
0.5605
0.4431
SC-GS
18.28
0.3893
0.3671
23.07
0.7186
0.2002
21.48
0.6374
0.2078
20.17
0.6698
0.2343
D-GS
18.57
0.3941
0.3768
0.7582
21.38
0.2142
0.6604
0.2700
Ours
20.47
0,5275
0.2408
25.64
0.7996
0.1518
23.69
0.7455
0.1551
22.18
0.7075
0.2125
A.10 COMPARISON WITH DEPTH RANKING Loss
We provide a quantitative comparison between our ordinal depth loss and other depth ranking losses (Chen et al.. 2016; Pavlakos et al., 2018) Table 8. this experiment, we replace our ordi- nal depth loss with their depth ranking loss while keeping all other configurations fixed. The results demonstrate that our ordinal depth loss significantly outperforms their approach across scenes. This highlights the effectiveness of our method in capturing depth information more accurately than the existing depth ranking losses.
In Fig. 12, present visual comparison of rendered depth maps using depth ranking loss and or- dinal depth loss. Our depth values are more smooth and well-distributed while depth values learned from rank depth loss show high contrast and more noise. The reason is that depth ranking loss en- courages large contrast to satisfy the given depth order and is not robust to the noises the input depth.
Table 8: Comparison with depth ranking loss (Chen et al., 2016; Pavlakos et al., 2018), denoted as DRL for simplification.
sear steak
cut beef
coffee martini
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
DRL
18.22
0.7390
0.2228
17.07
0.7162 0.8221
0.2283
14.09
0.7465 0.7962
0.1902 0.1473
Ours
23.53
0.8126
0.1642
23.98
0.1438
21.37
20

---

Published as a conference paper at ICLR 2025
Input Depth
Depth Ranking Loss
Ours
Figure 12: A visual comparison of rendered depth maps between using depth ranking loss (Wang al., 2024b) and our ordinal depth loss.
A.11 ABLATION USING DEPTH FROM VIDEO-DEPTH ESTIMATION METHODS
To show that method is also compatible with recent video depth methods, we conduct an ablation experiment using the depth maps predicted by the video depth estimator (Shao et al., 2024) and different types of depth loss supervision. As shown in Table 9, the video depth estimator improves the results, and our ordinal depth loss still consistently improves the results.
Table 9: Video depth ablation. VD + P refers to supervised using video depth maps with pearson depth loss, VD O refers to supervised using video depth maps with ordinal depth loss.
sear steak
cut beef
coffee martini
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
VD P
23.43
0.7983
0.1732
23.19
0,8134
0.1472
20.01
0.7664
0.1647
VD +0
23.48
0.8091
0.1711
23.61
0.8103
0.1532
21.55
0.8001
0.1488
A.12 RESULTS OF NERF-BASED METHODS USING DEPTH SUPERVISION
We present the results HexPlane (Cao Johnson, 2023) when supervised with depth loss in Table 10. Using depth maps for supervision only leads to slight improvement for HexPlane and MoDGS still outperforms HexPlane by large margin.
Table Metrie results when HexPlane (Cao & Johnson, 2023) incorporated with depth super- vision, denoted as "HexPlane+D"..
flame steak
cut beef
coffee _martini
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
HexPlane
16.97
0.7528
0.2543
16.76
0.5382
0.5054
13.26
0.4049
0.5835
D
17.21
0.6156
0.4646
18.82
0.6344
0.414
15.78
0.4708
0.5106
Ours
23.23
0.8083
0.1592
23.98
0.8221
0.1438
21.37
0.7962
0.1473
21

---

Published as a conference paper at ICLR 2025
A.13 DIFFERENT RANDOM INITIALIZATION METHODS
In the ablation study of our paper, we use the Kaiming initialization as the random initialization. We additionally provide ablations using zero deformation initialization of the deformation field in Ta- ble (ffamme steak scene of DyNeRF). In this experiment, we still initialize the canonical Gaussian spaces but adopt a different initialization strategy for the deformation field MLP.
Table 11: Quantitative comparison results for different initialization methods
Method
PSNR
SSIM
Lpips
ZeroDefoormationlnit
22.34
0.7906
0.2151
KaimingInit
22.24
0.7744
0.2036
3D-AwareInit (Ours)
23.23
0.8083
0.1592
A.14 LENGTH OF VIDEOS THAT MoDGS CAN HANDLE
Currently, we can handle videos up to 10 seconds. Scaling to longer videos(~<30 seconds) is possible we use larger deformation MLP network and train for more iterations.
A.15 DISCUSSION ABOUT DEPTH CONSISTENCY METHODS
From the knowledge distillation perspective, method be regarded distillation frame- work to get consistent video depth. In our framework, we adopt the 3D Gaussian field as the 3D) representation and the ordinal depth loss as the supervision distill the monocular depth estima- tion. Meanwhile, we adopt the rendering loss to further regularize the 3D representation, which enables distill temporally consistent monocular video depth from the inconsistent estimated depth maps. The inconsistent input depth and the distilled consistent video depth can be visualized in the supplementary video. This depth distillation mechanism is also adopted by self-supervised video depth estimation methods (Wang et al., 2023d; 2024c; Xu et al., 2024; Luo et al., 2020).
Although our method primarily targets novel-view synthesis dynamic scenes, we also conducted an experiment to compare the depth map quality of MoDGG with rencent video depth stabilization method (Wang et al., 2023d). Fig. 16, can observed that the depth maps from MoDGS contain more details and are more temporally stable. We sincerely recommend you to watch our supplementary videos.
A.16 COMPARISON WITH PERCEPTUAL DEPTH LOSS
In this experiment, we substitute our ordinal depth loss with a perceptual depth loss, specifically the learned perceptual image patch similarity loss (LPIPS) (Zhang et al., 2018) while maintaining all other experimental conditions unchanged. For the LPIPS loss, we utilize the AlexNet (Krizhevsky et al., 2012) in computing the LPIPS loss. We present quantitative comparison of our ordinal depth loss against perceptual depth losses in Table 12. The results indicate that our ordinal depth loss outperforms the alternative LPIPS depth loss. Furthermore, the rendered depth maps using perceptual losses in Fig. 13. Compared to the maps produced with the LPIPS loss, our depth maps smoother and contain less noise.
provide visual comparison of
Table 12: Comparison between perceptual depth loss ("LPIPS") and our original depth loss ("Ordi- nal").
sear steak
cut beef
coffee martini
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
LPIPS
22.43
0.8049
0.1782
23.36
0.8102
0.1591 0.1438
20.97
0.7820 0.7962
0.1575 0.1473
Ordinal
23.53
0.8126
0.1642
23.98
0.8221
21.37
22

---

Published as a conference paper at ICLR 2025
Input Depth
LPIPS
Ours
Figure 13: A visual comparison of rendered depth maps using the perceptual depth loss and our ordinal depth loss.
A.17 RESULTS ON SCENES WITH HIGHLY COMPLEX MOTIONS
Reconstructing 4D dynamic fields in complex scenes using casually captured monocular videos remains challenging task. In playground and balloon2 scenes, which both contain moving objects (the human and balloon) and complex motion, MoDGS handles the situation relatively well and generates reasonable results. We present visual results in Fig. 15. As shown in the figure, the appearances of both objects can be well reconstructed. For more complex motions, MoDGS may fail due the difficulty in predicting robust depth maps, which leave for future works.
A.18 RESULTS ON SCENES WITH RAPID MOTION
Rapidly moving scenes pose significant challenges for dynamic Gaussian field reconstruction. Such rapid motions can lead to inaccurate camera pose estimations, which remain problematic for MoDGS and all existing dynamic Gaussian reconstruction methods. We conducted experiment on high-speed motion scene(rollerblade from the Davis dataset) containing motion blur (in the regions of hands, feet, and hair) to evaluate our MoDGS performance under challenging conditions. As shown in Fig. 15, MoDGS produces reasonable results in the severely blurred regions. The results demonstrate that our approach maintains robust functionality even in rapidly changing envi- ronments. To better address this issue, we may combine deblur methods (Zhu et al., 2023b;a; Pan al., 2020) with motion priors future works.
A.19 DISCUSSION ON HOW TO HANDLE SCENARIOS WITH HEAVY OCCLUSIONS OR SPECULAR.
How to handle scenarios with heavy occlusions. If the occluded regions are observed at some timestamps, MoDGS is able to accumulate information among different timestamps to complete these occluded regions on some timesteps as shown in Appendix A.4. However, MoDGS cannot generate new contents for the occluded regions, which indeed degenerate the rendering quality on occluded regions. Incorporating some generative priors such as diffusion models could alleviate this problem.
Possible solutions dealing with specular scenarios. MoDGS can reconstruct some specular ob- jects but show some artifacts because MoDGS does not involve any special design for specular objects. As shown by recent work (Fan et al., 2024), potential solution for this is to introduce inverse rendering in dynamic scenes, which enables accurate modeling of specular surfaces in dy- namic scenes.
23

---

Published as a conference paper at ICLR 2025
A.20 DISCUSSION ABOUT THE ROBUSTNESS TO DIFFERENT DEPTH ESTIMATORS
To further demonstrate the robustness of our ordinal depth loss across different depth estimation methods, compare the performance of MoDGS using various depth estimators (Yang et al., 2024a; Ke et al., 2023; Shao et al., 2024) while keeping all other configurations unchanged. The quantitative results obtained using these methods are presented in Table 13. When different depth maps are utilized, the performance of our method exhibits minor variation, which demonstrates that our method robust various depth estimation approaches and does not rely on a specific depth prior distribution, such as Geowizard (Fu et al., 2024).
In Fig. 17, We also provide a visual comparison of depth maps generated by these methods and ren- dered using our MoDGS. Although some flickering and inconsistencies can be observed in the input depth maps, MoDGS still produces stable and consistent depth renderings. We sincerely recommend you watch our supplementary videos.
Table 13: Quantitative results with different depth estimation methods, such as Marigold (Ke et al., 2023), ChronoDepth (Shao et al., 2024), DepthAnything (Yang et al., 2024a) and Geo Wizard (Fu et al., 2024).
Depth estimator
PSNR
SSIM
LPIPS
Marigold (Ke et al., 2023)
23.12
0.8057
0.1712
ChronoDepth (Shao al., 2024)
23.44
0.8105
0.1687
DepthAnything (Yang et al., 2024a)
23.35
0.8097
0.1621
GeoWizard (Fu al., 2024)
23.23
0.8083
0.1592
Input View
Rendered View
Figure 14: Input views and novel view synthesis results from MoDGS for scenes with complex motions in the NVIDIA dataset.
24

---

Published as a conference paper at ICLR 2025
mow
Input View
Rendered View
Figure 15: Input views and novel view synthesis results produced by MoDGS on the rollerblade scene from the Davis dataset, which features rapid motions and motion blur.
25

---

Published as a conference paper at ICLR 2025
Input View
NVDS
Ours
Input View
NVDS
Ours
Figure 16: A visual comparison of depth maps estimated by NVDS (Wang et al., 2023d) and rendered by MoDGS on Camel and Bear scene from the Davis dataset.
26

---

Published as a conference paper at ICLR 2025
Input (epth((Detthnyyhiingg
Rendered Depth
Input (eath(Marigold)
Rendcred Depth
Input Certhh(hhro)))pphh)
Rendered Depth
Input Depth(Geowizard)
Rendered Depth
visual comparison of rendered depth maps from our MoDGS and input depth maps estimated by different depth estimation methods, such as Marigold (Ke et al., 2023), ChronoDepth (Shao et al., 2024), DepthAnything (Yang et al., 2024a) and GeoWizard (Fu et al., 2024).
Figure 17:
27


Reviews:
[{"review_id": "jEYfHT7GvX", "reviewer": "Reviewer_1rXU", "summary": "The paper introduces MoDGS (Monocular Dynamic Gaussian Splatting), a novel approach for rendering dynamic 3D scenes from casually captured monocular videos, overcoming limitations faced by prior dynamic NeRF and Gaussian Splatting methods via depth estimation. These existing approaches require either extensive camera movement or synchronized multi-view setups to establish multiview consistency, which is lacking in casual, minimally moving videos.\n\nTo tackle this challenge, MoDGS incorporates recent advancements in single-view depth estimation to guide the learning of a deformation field that represents scene dynamics. The method introduces a novel ordinal depth loss to address the depth inconsistency in single-view depth maps, enhancing the robustness and continuity of 3D scene reconstruction.\n\nComprehensive experiments across multiple datasets (Nvidia, DyNeRF, DAVIS, and a self-collected casual video dataset) demonstrate that MoDGS produces high-quality novel views in dynamic scenes, outperforming state-of-the-art methods. The authors also plan to release their code and dataset to support future research in this area.", "questions": "From the perspective of a peer, I suggest the authors address the concept of 'Depth supervised in dynamic GS' in the title. After all, a novel method would be more informative and important to the other researchers than a usage scenario like ''casually captured video\".", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "For the novelty, this paper makes a distinct contribution to introducing depth supervision into the domain of dynamic Gaussian Splatting (DGS) for monocular dynamic input. This approach is novel yet intuitive, filling a key gap in the field for cases where the input consists of casually captured videos with minimal camera movement. Compared to the other papers in the field that mechanically put all fancy complicated input feature streams or loss functions together, the proposed solution is conceptually straightforward but impactful, pushing forward the capabilities of monocular dynamic scene reconstruction.\n\nThe experiments are well-designed and executed, rigorously testing the proposed method across various datasets, including Nvidia, DyNeRF, and DAVIS. Each experiment logically supports the methodology, demonstrating how the 3D-aware initialization and ordinal depth loss contribute to enhanced depth consistency and scene deformation modeling. The results clearly show MoDGS’s robustness and superiority over baseline methods, adding confidence in its effectiveness.\n\nThe paper is presented with clarity and precision, making even technical aspects of the method easy to follow. The figures and tables are well-constructed and informative, providing visual clarity to support the text and helping to reinforce the main findings. The logical flow, from problem statement to method explanation and results, enables readers to understand the method's motivation and benefits seamlessly.", "weaknesses": "MoDGS is validated across several datasets, which demonstrates its robustness. However, the paper could discuss the potential limitations in generalizing this approach to different depth estimation models. It would demonstrate the robustness of the proposed method and its generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M7Fumw0rOc", "reviewer": "Reviewer_xth1", "summary": "The paper proposes MoDGS, a novel pipeline to render high quality novel views of dynamic scenes from casually captured monocular videos. Unlike traditional dynamic scene reconstruction methods that rely on rapid camera motions to establish multiview consistency, MoDGS is designed for videos with static/slowly moving cameras, where such consistency is weaker. The core of their method involves using a single-view depth estimation technique to guide scene learning and introducing a 3D-aware initialization method to construct a realistic deformation field. MoDGS incorporates an innovative ordinal depth loss to address the challenge of depth inconsistency across frames, enhancing the coherence and quality of rendered views. Experiments on datasets such as DyNeRF, Nvidia, and a self-collected dataset demonstrate it's ability to outperform SOTA methods in novel view synthesis, achieving superior image quality even in challenging dynamic scenarios.", "questions": "1. Can you elaborate on the choice of ordinal depth loss over other depth loss functions, such as perceptual depth consistency? How did the ordinal depth loss compare to other depth loss formulations in preliminary experiments, and what were the observed advantages or disadvantages?\n2. How robust is MoDGS in scenarios with heavy occlusions or specular reflections? Would integrating additional priors or multi-scale depth estimations help in such cases?\n3.   How does MoDGS compare with recent depth consistency techniques, particularly those used in self-supervised monocular depth estimation? Exploring this comparison could shed light on the effectiveness of the ordinal depth loss relative to existing methods.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "MoDGS represents an original approach within  novel view synthesis  and dynamic scene modeling by specifically addressing the limitations of existing methods for casually captured monocular videos. The authors introduce a 3D-aware initialization mechanism and an ordinal depth loss, that offer a solution that successfully reduces the dependency on rapid camera motion. The novel use of ordinal depth loss to maintain depth order among frames, rather than relying solely on absolute values, represents an innovative perspective on addressing depth consistency issues, which has practical implications for improving depth coherence in dynamic scenes captured casually. I believe the paper is well-executed in terms of technical rigor, with comprehensive evaluations across three datasets: DyNeRF, Nvidia, and a newly created monocular casual video dataset. Each component of MoDGS is thoroughly tested and ablated to demonstrate its impact on the final results. This systematic experimentation supports the author’s claim that MoDGS significantly outperforms other approaches in the quality of novel-view rendering for dynamic scenes. The paper is structured logically, with clear explanations of each component of the MoDGS pipeline. The figures visually support the textual explanations, making complex concepts more understandable to a reader. The method  has significant implications for real-world applications that involve casually captured videos, such as mobile AR/VR, video editing, and 3D content creation. By enabling high-quality novel view synthesis from single-camera footage without multiview camera motions, MoDGS broadens the scope of dynamic scene reconstruction, making it accessible to a wider range of use cases. The method’s ability to handle both static and dynamic elements in monocular videos opens new avenues for monocular depth estimation and dynamic scene modeling, where single-camera approaches have been historically constrained by depth inconsistency issues.", "weaknesses": "While the ordinal depth loss is a novel way to improve depth coherence, I believe the paper may benefit from more discussion on its limitations. Specifically, the ordinal depth loss assumes a consistent depth order among frames, which may not hold in scenes with complex occlusions or reflections. MoDGS assumes smooth transitions between frames for consistent depth ordering. However, the approach may face challenges in scenes with rapid or erratic movement where objects appear and disappear frequently. While it performs well on scenes with relatively smooth dynamics, addressing how the method might be adapted or optimized for highly dynamic environments would improve its versatility. The method relies heavily on single view depth estimators to guide the reconstruction process. Although the depth estimation technique used is SOTA, it still inherits the limitations of single view estimators, particularly in complex scenes with specular surfaces or low-lit conditions. Including a more detailed analysis on how the quality of the depth estimator impacts the proposed method’s performance, and potentially exploring integration with other depth supervision methods could potentially make the approach more adaptable across varying input qualities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yb0Vc9Is4b", "reviewer": "Reviewer_FCsY", "summary": "The paper presents MoDGS, a novel pipeline for synthesizing dynamic scenes from casually captured monocular videos. Unlike existing methods requiring large camera motions, MoDGS leverages single-view depth estimation for 3D reconstruction and introduces a 3D-aware initialization alongside an ordinal depth loss. These innovations enable robust, high-quality novel view synthesis, outperforming state-of-the-art methods in rendering casually captured videos.", "questions": "Kindly refer to the [Weaknesses].", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* A differentiable order-based loss function, the ordinal depth loss, is proposed, with detailed descriptions of its motivation and its distinctions from other depth loss functions.\n* It demonstrates significant superiority over multi-view camera methods in reconstruction metrics and visual results, with ablation studies validating the importance of the \"3D-aware initialization scheme\" and \"ordinal depth loss.\"\n* The paper is well-written and easy to follow.", "weaknesses": "* **The contributions and innovations are limited**. This work is based on the previous canonical space paradigm of 3D Gaussian Splatting (3DGS) combined with deformation fields, with the main contributions being a deformable 3DGS initialization method and a depth loss. The primary principle of the former relies on predicting per-pixel 3D flow using current state-of-the-art monocular depth estimation and optical flow estimation methods. However, the sole innovative aspect lies in converting 2D optical flow to 3D flow using the estimated depth map. As for the depth loss, although it is well-motivated and provides performance improvement, it essentially replaces the Pearson correlation loss with an order correlation loss.\n* **The experimental comparisons lack fairness**. In most quantitative comparisons, this work is only compared against methods that require multi-view camera input. It is recommended to include quantitative and qualitative comparison results with methods under the same setting of \"casually captured monocular video.\" It is also perplexing that the authors mention \"RoDynRF, a method that adopts single-view depth estimation as supervision\" in \"Baseline methods\", yet I only found comparative results for this method in Fig. 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JriR1rRBLO", "reviewer": "Reviewer_Gzks", "summary": "The paper introduces MoDGS, a method for dynamic view synthesis from monocular videos, leveraging a Gaussian-based splatting technique combined with deformation fields and an ordinal depth loss to reconstruct scenes. This framework integrates a 3D-aware initialization to align Gaussian representations in a canonical space, while the ordinal depth loss is used to improve scene geometry continuity. MoDGS is claimed to improve over previous dynamic NeRF approaches and related deformation methods, with results evaluated on the DyNeRF and Nvidia datasets.", "questions": "1. How does MoDGS handle scenarios where the pre-trained depth estimator provides inconsistent depth due to environmental variations? Has any analysis been conducted to measure performance stability when GeoWizard or other models are less reliable?  \n\n\n2. Would MoDGS perform as well on datasets with higher motion complexity or less predictable scene geometry? Testing on a broader range of datasets, such as those with cluttered backgrounds or multiple moving objects, would better validate the method's generalization.   \n\n3.  Considering MoDGS’s reliance on single-view depth priors, would a formalized knowledge distillation framework improve model autonomy by adapting these priors dynamically during training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. I think the 3D-aware initialization process is a strong point, as it specifically addresses a common issue in monocular reconstruction. By initializing Gaussians instead of relying on random initialization, this method seems to potentially add more consistency.  \n\n2. The ordinal depth loss is, in my view, an interesting idea. It tries to tackle scale ambiguity in monocular depth estimation, which I think is particularly relevant in dynamic scenes. This loss formulation promotes depth consistency across frames, an essential factor when handling complex, moving scenes.", "weaknesses": "1. I think the innovation is quite incremental for the reason that compared to closely related works like Deformable 3DGS and 4DGS, the methodological innovation appears incremental, mainly optimizing existing elements (depth consistency and deformation) rather than proposing a new structural approach.\n\n\n2. Besides, the approach relies heavily on pre-trained depth models. MoDGS relies on single-view depth estimators like GeoWizard for depth initialization, which brings into question the independence of its results. The approach leverages external models as priors, potentially limiting its novelty and raising questions regarding knowledge distillation. The extent to which these pre-trained models influence the final performance is not rigorously analyzed.  \n \n\n3. While MoDGS integrates external depth estimation for initialization, there is no formalized knowledge distillation to adaptively refine the model during training. This absence may reduce the adaptability of MoDGS across different dynamic scenes where pre-trained depth estimators may not perform equally well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)