MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting
Authors: Ruijie Zhu
Yanzhe Liang
Hanzhi Chang
Jiacheng Deng
Jiahao Lu
Wenfei Yang
Tianzhu Zhang
Yongdong Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Gaussian
Novel View Synthesis
Optical flow
Dynamic scene
Primary area: machine_vision

Abstract: Dynamic scene reconstruction is a long-term challenge in the field of 3D vision. Recently, the emergence of 3D Gaussian Splatting has provided new insights into this problem. Although subsequent efforts rapidly extend static 3D Gaussian to dynamic scenes, they often lack explicit constraints on object motion, leading to optimization difficulties and performance degradation. To address the above issues, we propose a novel deformable 3D Gaussian splatting framework called MotionGS, which explores explicit motion priors to guide the deformation of 3D Gaussians. Specifically, we first introduce an optical flow decoupling module that decouples optical flow into camera flow and motion flow, corresponding to camera movement and object motion respectively. Then the motion flow can effectively constrain the deformation of 3D Gaussians, thus simulating the motion of dynamic objects. Additionally, a camera pose refinement module is proposed to alternately optimize 3D Gaussians and camera poses, mitigating the impact of inaccurate camera poses. Extensive experiments in the monocular dynamic scenes validate that MotionGS surpasses state-of-the-art methods and exhibits significant superiority in both qualitative and quantitative results. Project page: https://ruijiezhu94.github.io/MotionGS_page.

MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting
Ruijie Zhu"
Yanzhe Liang"
Hanzhi Chang
Jiacheng Deng Yongdong Zhang
Jiahao Lu
Wenfei Yang
Tianzhu Zhang
University of Science and Technology of China Deep Space Exploration Lab
(ruijiezhu, yzliang, changhz, dengjc, lujiahao] ustc. edu. en, [yanguf, tzzhang, 2hydr3) edu. cn
Abstract
Dynamic scene reconstruction is a long-term challenge in the field 3D vision. Recently, the emergence of 3D Gaussian Splatting has provided new insights into this problem. Although subsequent efforts rapidly extend static 3D Gaussian to dynamic scenes, they often lack explicit constraints on object motion, leading to optimization difficulties and performance degradation. To address the above issues, we propose novel deformable 3D Gaussian splatting framework called MotionGS, which explores explicit motion priors to guide the deformation of 3D Gaussians. Specifically, we first introduce an optical flow decoupling module that decouples optical flow into camera flow and motion flow, corresponding to camera movement and object motion respectively. Then the motion flow can effectively constrain the deformation of 3D Gaussians, thus simulating the motion of dynamic objects. Additionally, camera pose refinement module is proposed to alternately optimize 3D Gaussians and camera poses, mitigating the impact inaccurate camera poses. Extensive experiments in the monocular dynamic scenes validate that MotionGS surpasses state-of-the-art methods and exhibits significant superiority in both qualitative and quantitative results. Project page: attps://ruujiezhuu4, github. io/MotionoS...... _page/
1 Introduction
Dynamic scene reconstruction aims to model the 3D structure and appearance of time-evolving scenes, enabling novel-view synthesis at arbitrary timestamps. computer vision, attracting widespread attention from the research community and finding important applications in areas such as virtual/augmented reality and 3D content production. In comparison to static scene reconstruction, dynamic scene reconstruction remains longstanding open challenge due to the difficulties arising from motion complexity and topology changes.
a crucial task the field of 3D>
In recent years, plethora of dynamic scene reconstruction methods [22 5 6 g have been proposed based on Neural Radiance Fields (NeRF) [9] driving rapid advancements this field. While these methods exhibit impressive visual quality, their substantial computational overhead impedes their applications in real-time scenarios. Recently, a novel approach called 3D Gaussian Splatting (3DGS) [10), has garnered widespread attention in the research community. By introducing explicit 3D Gaussian representation and efficient CUDA-based rasterizer, 3DGS has achieved unprecedented high-quality novel-view synthesis with real-time rendering, Subsequent methods [1]
[2
13
[4
"Equal contribution
Corresponndig author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Object Motion
Mation
G
Flow
Optical
Flow
C
Camara
Gaussian flow under optical flow supeeviston motion flow supeevision
Gaussian flow under
Camera Motion
Flow
Image txput
(b)
(a) Gaussian flow under different supervision. We model Gaussian flow under the supervision of optical flow and motion flow respectively. The latter produce more direct description of object motion, thereby effectively guiding the deformation of 3D Gaussians. (b) The decoupling of optical flow. We decouple the optical flow into motion flow which is only related to object motion and camera flow which is only related to camera motion.
Figure 1:
15 16 T
rapidly extend 3DGS dynamic scenes, also named 4D scenes. Initially, D-3DGS proposes to iteratively reconstruct the scene frame by frame, but it incurs significant memory overhead. The more straightforward approaches [[3] [7] utilize deformation field simulate the motion of objects by moving the 3D Gaussians to their corresponding positions at different time steps. Besides, some methods [12 [8] do not independently model motion but treat space-time as a whole to optimize. While these methods effectively extend 3DGS to dynamic scenes, they rely solely on appearance to supervise dynamic scene reconstruction, lacking explicit motion guidance on Gaussian deformation. When object motion is irregular (e.g., sudden movements), the model may encounter optimization difficulties and fall into local optima.
Based on the above discussions, we argue that explicit motion guidance is indispensable for the deformation of 3D Gaussians. Benefiting from the advancements in optical flow estimation [19 20
natural solution is to utilize an off-the-shee optical flow network to provide 2D motion priors 21 22
However, the formation of optical flow is affected by both camera motion and object motion, which is not conducive to explicit modeling of object motion. Therefore, it is necessary to separate the optical flow related only to the moving object (i.e., motion flow) to guide Gaussian deformation more efficiently. As shown in Figure [a), directly using optical flow (column 2) to supervise the Gaussian deformation will inevitably include the contribution of static objects to the optical flow, while using motion flow as supervision (column 3) can easily avoid this. Besides, the estimated camera pose in dynamic scenes is not always accurate. Due to the lack of geometric consistency between adjacent frames for moving objects, using point correspondences on dynamic objects to calculate the camera pose can lead to erroneous offsets, thereby affecting the optimization of 3DGS.
To address the above issues, we propose a novel deformable 3D Gaussian Splatting framework called MotionGS, which explicitly constrains the deformation of 3D Gaussians by extracting the motion priors from optical flow. Our method includes an optical flow decoupling module and camera pose refinement module. In the optical flow decoupling module, we decouple the 2D optical flow into camera flow and motion flow, as shown in Figure [[).. The camera flow comes from the camera ego-motion, while the motion flow comes from the motion of dynamic objects. We use the motion flow to directly constrain the deformation of 3D Gaussians (i.e., Gaussian flow). Since the calculation of Gaussian flow is directly implemented in the CUDA-based rasterizer, this process is differentiable and efficient. In the camera pose refinement module, we first fix the 3D Gaussians and then utilize photometric consistency loss to backpropagate gradients to camera poses, thereby alternately optimizing 3D Gaussians and camera poses to further enhance the rendering quality.
To sum up, our main contributions are as follows:
We propose a novel deformable 3D Gaussian framework called MotionGS, which provides explicit motion guidance for deformable 3DGS and achieves high-quality dynamic scene reconstruction with real-time rendering.
The proposed optical flow decoupling module effectively separates the flow caused solely by object motion, thereby efficiently supervising the deformation of 3D Gaussians. The

---

proposed pose refinement module alternately optimizes 3DGS and camera poses, reducing reliance on accurate camera poses and further boosting rendering quality.
Extensive experiments have demonstrated the effectiveness of the proposed method. Results on the NeRF-DS and HyperNeRF datasets validate the state-of-the tott-offthee-rt performance of our approach in dynamic scene reconstruction.
2 Related Work
2.1
Novel-View Synthesis (NVS)
Novel view synthesis has been a hot research topic in the field of computer vision and graphics in
recent years. NeRF (9), which represents 3D scene by neural radiance fields, first achieves high-
resolution photorealistic results in this field. Despite many subsequent works [23] 24 25 266 (27
28
have been proposed to improve its efficiency and quality, NeRF-based methods still struggle to render high-quality images with real-time rendering speed. Recently, by modeling 3D
29 30
scenes using a set anisotropic 3D Gaussian with an efficient rasterizer, 3D Gaussian Splatting (3DGS) [I0] has shown remarkable performance with real-time rendering. Compared to NeRF, 3DGS
an explicit 3D scene representation method with better scalability and editability. Therefore, has been rapidly extended to other 3D vision tasks, including sparse- view reconstruction 332
33 35
3D generation 36 37
38
39 40
scene editing [4] 42 43
and SLAM 44
46
2.2
Dynamic Scene Reconstruction
In recent years, various dynamic scene reconstruction approaches have been proposed, which can be broadly categorized into NeRF-based and BDGS-based methods. NeRF-based works 488
||
49 2
4
50
51 5
usually map dynamic scenes to a canonical space and render images based on this 3D
canonical space. This kind of 4D scene representation intuitive but requires a well- reconstruucd
54 these NeRF-based methods require frequent point sampling MLP queries, suffering from long training and rendering time. With the proposal of 3DGS, many works [T6
canonical space. Other works propose to use time-varying NeRFs 6
or explicit 8 57 58 59 to represent and render dynamic scenes. However, all
3
52 53
representations [55] 56
15 [3 17 [2] use 3DGS as the fundamental model for 4D scene representation. For instance, D-3DGS I1] models dynamic scenes by allowing the positions and rotation matrixes of 3DGS to change over time. Deformable 3DGS [17] uses MLP to model a deformation field based on time and the canonical Gaussian space. SC-GS [5] bounds dense 3DGS with sparse control points, calculating the movement of Gaussians in coarse-to-fine manner. Despite they have performed impressive rendering quality in some dynamic scenes, they lack explicit motion guidance to constrain the movement of Gaussian, resulting in degraded performance in more complex dynamic scenes. Recent works 21 (2) compose the movement of 3D points through their corresponding Gaussians, using 2D flow priors supervise the deformation of 3DGS. Inspired by them, we decompose the optical flow to obtain more direct motion supervision, thus achieving higher rendering quality.
21
60
14
61
62 63
2.3 NVS with Pose Optimization
68 69 70 have noticed that it is difficult to derive precise camera poses of input images in the real world, so they address novel view synthesis together with camera pose optimization. i-NeRF (64) initially estimates camera poses by matching the input images. Other methods such as NeRFmm (65] and Nope-NeRI (69 use monocular depth priors as guidance to do the joint optimization of NeRF and camera poses. Recently, CF-3DGS (70] proposes progressive reconstruction and leverages photometric loss to learn the affine transformation of Gaussians to mostly effective only for static scenes and
Several NVS works 64 65| 66 67
optimize the camera pose. However, these methods lack support for dynamic scenes. Motivated by these methods, we aim to extend 3DGS to dynamic scenes with pose optimization, thus boosting the rendering quality and robustness.
3 Preliminary
In this section, we briefly introduce the modeling and rendering of 3DGS Section and the deformable extension of 3DGS towards dynamic scene reconstruction in Section 3.2
3

---

3.1 3D Gaussian Splatting
As an explicit 3D representation similar to point clouds, 3DGS models the scene with a set of 3D Gaussians. However, different from point clouds, each 3D Gaussian in the scene has its own opacity E [0. 1], center position μ E R3x1, and covariance matrix Σ E R3x3 These properties determine the contribution and influence range of 3D Gaussians on rendering. For position a E [3x1 in 3D) space, the corresponding contribution of a 3D Gaussian on can be formulated as:
G(z) o "<0-=)==~" (#-)(=-)),
(I)
For differentiable optimization, the covariance matrix Σ can be decomposed into a scaling matrix and rotation matrix R: Σ = RSS'R where scaling matrix S = diag(1sa, $y, sz]) and rotation matrix R can be transformed from a quaternion [we, Tas Ty. Ta.. Then the 3D Gaussians can splatted to 2D camera plane through differential gaussian splatting. Specially, given a viewing transform matrix W and the Jacobian matrix J of the affine approximation
the projective transformation, we obtain the 2D covariance matrix Σzd through: ΣnD JWEJ WWE.T.J J Similarly, we can obtain the 2D center position μ2D of 3D Gaussians in camera plane. Therefore, given a 2D pixel p, the rendering contribution of 3D Gaussian the viewpoint W can be obtained through 2D version of ( To model the appearance of 3D Gaussians, spherical harmonics (SH) are introduced to define the color Finally, for each pixel, the rendering results of 3DGS can be derived by calculating the color contribution of all the related Gaussians. This process is known as a-blending:
N Ga j=1
{-1
C=
Σ
(1,))
(2)
where Ci On represent the color and density computed from the i-th 3D Gaussian.
3.2 Deformable 3D Gaussian Splatting
To extend 3DGS to dynamic scenes, intuitive approach is to utilize a learnable deformation field to fit the movement of objects in the real world through Gaussian deformation. This idea originates from F-RF-b methods such as D-NeRF [I] and has been effectively applied to 3DGS in recent works 17 13 In these deformable 3DGS methods, a deformation network D is typically used to model the movement of the center position of 3D Gaussians. Additionally, due to the inherent properties of 3D Gaussians, the deformation network D may also consider the rotation and scaling factors of 3D Gaussians they vary over time. Therefore, the deformation of 3D Gaussians can be formulated as:
(μ + Aurr + Ar, As) D(,,,,,,,t s,t,,,
where t is the timestamp, μ4, r, are the center position, rotation quaternion and scaling factors of 3D Gaussians, and Au,, Ar, As are their residuals, respectively. Due to the various implementations of deformable 3DGS, in this paper we focus solely on the deformation aspect without discussing the other designs and specific differences in these works. We select method [17] our baseline, leveraging explicit motion guidance and camera pose refinement to further enhance the rendering quality and the robustness in dynamic scenes.
Methodology
In this section, we first introduce the overall architecture of our approach in Section 4.1 Then the optical flow decoupling module is introduced to derive motion guidance for Gaussian deformation in (ection[422) The camera pose refinement module is introduced alternately optimize 3D Gaussians and camera poses in Section 4.3 Finally, the overall loss function is introduced in Section 4.4
4.1 Overall Architecture
The overall architecture of our method is illustrated in Figure 2 Our method primarily focuses on the reconstruction of monocular dynamic scenes. Firstly, following 3DGS [10, we initialize camera poses and 3D Gaussians using COLMAP (71). Given two adjacent frames I, and Ie+1, we compute forward optical flow Fr using an off-the-shelf flow estimation network. Meanwhile, we can obtain the rendered depth map D, of frame I, at time f through the rasterizer. By feeding the depth

---

Decoupling
FlowNet
O ptical Flow
Pose
Motion Flow
Ce
Motion Guidance
Depth
Camera Flow
C++1
Pose
Depth Splatting
Au Ar As
T AT
Splatting
Gausslan Deformation
Gaussian
Transformation
G.
G
G
The overall architecture of MotionGS. It can be viewed as two data streams: (1) The 2D data stream utilizes the optical flow decoupling module to obtain the motion flow the 2D motion prior; (2) The 3D data stream involves the deformation and transformation of Gaussians to render the image for the next frame. During training. alternately optimize 3DGS and camera poses through the camera pose refinement module.
Figure 2:
map 1t camera poses C₁ C(+1. and optical flow priors Fi-e+1 into the optical flow decoupling module, we can calculate the motion flow F solely related to object movement. After predicting the deformation of Gaussians through the deformation network D, obtain the state of 3D Gaussians at time t + and render the Gaussian flow FC F from time t to + 1 under the assumption of stationary camera viewpoint for the frame I,- The motion flow should be consistent with the Gaussian flow, thus providing explicit motion guidance to Gaussian deformation. Additionally, since the initialized camera poses may be inaccurate, we add small residual AT to the relative camera pose T. Leveraging the proposed camera pose refinement module, we cleverly backpropagate gradients to the camera poses, achieving refinement of the camera poses. During training, alternately optimize 3D Gaussian and camera poses to enhance the rendering quality and robustness in dynamic scenes.
4.2 Optical Flow Decoupling Module
To provide explicit motion guidance for the deformation of Gaussians, we first utilize an off-the-shelf optical flow network to predict 2D motion priors. Since optical flow influenced by both camera movement and object motion, we decompose it into camera flow and motion flow as illustrated the Figure bb). Camera flow represents the optical flow caused solely by camera movement, assuming the objects in the scene remain stationary. In contrast, motion flow considers the camera as stationary, capturing only the movement of the objects. Essentially, optical flow can be viewed as the effectively isolate object motion,
vector sum of these components. By decoupling them, providing precise guidance for Gaussian deformation.
Camera flow and motion flow. We use schematic diagram Figure 3 to illustrate the detailed calculation process. Camera flow can be directly computed from the camera poses and the depth of the current frame. Specifically, at the timestamp t, we obtain the depth map D, corresponding to frame I, directly from Gaussians through the rasterizer. Given the intrinsics K, and extrinsics T, of camera C,, we can reproject point p. from frame I, to 3D space using its depth Dr:
T K, Drpr,
(4)
where ÿt is the homogeneous coordinate of pr. Assuming & does not move, we can obtain the projection pi ++1 2, on frame 4+1:
p (
(5)
where Ke+1 and T++1 are the intrinsies and extrinsics of camera C++1, proj() projects the 3D coordinates to 2D image planes by dividing the last dimension (depth). Then the camera flow can be
5

---

Gr41
Xe+1
D,
Gis
AT
G
Gu1
T
Cerl
Training Gaussians (Camera Pose Fixed)
Finetuning Camera Pose
(Gaussians Fixed)
Figure 3: Flow calculation.
Figure 4: Pose refinement on iterative training.
defined
F = Pt.
(6)
which indicates the flow caused solely by camera movement. As the point 2t moves over time, we denote its updated position 24+1- This new point Z++1 is then projected onto frame T1+1 as Pr+1- Thus, the optical flow Fi-t+1 between two adjacent frame is defined as Pt+1 - Pr- Finally, the motion flow 1-1441 is derived by subtracting the camera flow from the optical flow:
FM Fi Fi-1+1 F = P++1
which also corresponds the optical flow caused object movement at fixed viewpoint.
Gaussion flow. To establish a correspondence between Gaussian deformation and motion flow, we need to splat the Gaussian deformation onto the 2D image plane, which is not implemented in the original 3DGS framework. Inspired by recent work [21). we introduce the concept of Gaussian flow, denoted as F. . to describe the 2D projection of Gaussian deformation, and implement in the CUDA-based rasterizer. The core idea is to model the contribution of Gaussians to the optical flow by first transforming 3D Gaussians canonical Gaussian space and then transforming them back to the state at the next time step. Please refer to Appendix A. for the specific derivation and modeling process of Gaussian flow. Gao al. (21 computes the deformation of 3D Gaussians from time to t ++1 under the transformation of the camera viewpoint from C₁ to Cz+1. corresponding to optical flow. Different from our Gaussian flow is designed match the motion flow, representing the deformation of 3D Gaussians from time to + 1 fixed under the camera viewpoint Ct+1-
Flow loss. To effective constrain the Gaussian deformation, we use a L1 loss between motion flow and Gaussian flow for simplicity:
C
|| (-)4+)))
(8)
where sg() means stop gradient. Note that we also stop the gradients of all variables at time / in the calculation of Gaussian flow for more efficient training.
Discussion. The benefits of decoupling the optical flow are evident. Since motion flow is only related to object motion, it can directly provide motion guidance. More importantly, in some previous works 4 54, 6) off-the-shelf segmentation network is often used to segment out the dynamic objects in the scene (such as humans, animals, cars, etc.). However, such masks are only used in their photometric loss mask out dynamic regions. In contrast, dynamic masks more directly. By masking static objects with these masks, we can obtain a clear motion flow for supervising Gaussian deformation. If optical flow is used as motion guidance, this advantage will no longer exist because static objects can also contribute to the optical flow.
motion flow benefits from these
4.3 Camera Pose Refinement Module
In monocular dynamic scenes, due to the complexity of motion and sparsity of observations, even widely used methods like COLMAP [71] cannot accurately estimate camera poses. Since optimization of 3DGS requires precise camera poses input, it often performs poorly in complex

---

dynamic scenes. Existing EDGS-based dynamic scene reconstruction methods rarely take this into account. Inspired by pose-free optimization methods for static scene reconstruction [72] 70], we design the camera pose refinement module. By alternately optimizing 3D Gaussian primitives and camera poses during training, we improve the rendering quality of 3DGS and its robustness in
dynamic scenes.
Iterative training. Since the supervision of 3DGS primarily relies on photometric consistency loss, simultaneously optimizing camera parameters and 3DGS can be considered chicken-and-en problem. Therefore, similar to Bundle Adjustment, we adopt an alternating optimization strategy to train the model. Specifically, assuming G, is the Gaussian at time I, we first predict the deformation of the Gaussian using the deformation field D. We denote the deformed Gaussian as Gt+1 Since the observation viewpoint changes from time i to ( + 1, Gt needs to be transformed once again under the camera C++1 to render frame 1++1- We denote the transformed Gaussian G++1- This transformation process actually corresponds to camera motion. To achieve differentiable optimization, we introduce small residual AT into the relative pose T from camera viewpoint C₁ to C++1. treating it as a Iearnable SE(3) transformation. With this small change, we enable gradients to backpropagate to the camera poses. During the optimization of camera poses, we freeze all attributes of 3D Gaussians to improve training stability and robustness. Then we update the camera poses initialized by COLMAP with the optimized relative camera poses, achieving global pose refinmeent.
Discussion. While several methods have been proposed for pose-free optimization in static scenes, dynamic scenes present greater challenges due to their inherently under-constrained nature. As result, to ensure stable and robust optimization, our approach still leverages camera poses computed by COLMAP as an initialization step. This also necessitates the presence of sufficient static features in the scene. Fortunately, static features are commonly found in most real-world environments, particularly in background regions.
4.4 Optimization
Thanks to the integration of optical flow rendering and camera pose gradient computation in our rasterization process, the overall training pipeline of our method is end-to-end differentiable. The overall training loss is given by: C= Cruselime + XC
where Lrassllnee is the photometric loss used in our baseline [17], À is the weight of our flow loss.
5 Experiment
5.1
Experimental Setup
To highlight the abilities of our method in handling complex dynamic scenes, select two repre- sentative monocular dynamic scene datasets for evaluation: NeRF-DS [73] and HypenNeRF [49]. Our implementation is mainly based on Py Poroh.. We use simple Adam 774 optimizer to adjust the rotation increment and translation increment the camera, and the learning rates of the two are set to 3e-3 and le-1, respectively. The entire training process requires 20,000 iterations. We set to 0.5 for NeRF-DS and 0.1 for HyperNeRR scene. The rest of the settings are consistent with the baseline method [17]. All experiments are performed single Nvidia RTX 3090 GPU. For more implementation details, please refer to Appendix A.2
5.2 Results
Following previous methods, we use metrics PSNR, SSIM, and LPIPS for evaluation. For more visualizations, please refer to Appendix A.4
Results on the NeRF-DS dataset. Table I shows the performance comparison results with the state-of-the-art methods on the NeRF-DS dataset. dynamic monocular scenes, especially those with rapid movements and high complexity, our method significantly outperforms the baseline method. For example (see Figures 5 and 12), the plate scene, our method accurately renders the reflections and sharp edges of the moving plate while significantly reducing visual distortions such floating
7

---

Table 1: Quantitative comparison on NeRF-DS dataset per-scene. We highlight the best and the
second best results in each scene. NeRF-DS and HyperNeRF employ MS-SSIM and LPIPS with while other methods and ours use SSIM and LPIPS with the VGG [76] network.
the AlexNet 75
Method
Sieve
Plate
Hell
Press
PSNR1
SSM
LPIPS]
PSNRI
SSIM)
LPIPS.
PSNR1
SSIM
LPIPS_
PSNR
SSIM
LPIPS]
313-3
10
0.8003
0.2347
16.14
0.4093
0,2503
22.89
0.8663
0.2904
Vax
21.49
0.3176
20.58
0.3317
23.08
0.2568
34.47
0.8613
0.3001
HyperNoRF
49
25.43
0.8798
0.1645
18.93
0.7709
0.2940
23.06
0.8097
0,2052
26.15
0.8897
0.1959
NeRF-DS
25.78
0.8900
0.1472
20.54
0.8012
0.1996
23.19
0.8212
0.1867
25.72
0.8618
0.2047
Deformable-33
0.1502
18.82
0.3551
25.41
0,8614
Ours
26.17
0.8834
0.1502
21:01
0.8213
0.1967
26.33
0.8688
0.1490
26.63
0.8965
0.1955
Method
Cap
As
Basin
Meam
SSIMT
LPIPS]
SSIMT
LPIPS]
PSNR1
LPIPS
PSNRT
LPIPS
3D-GS
10
21.71
0.8304
0.2548
22.69
0.8017
0.2994
18.42
0.7170
0.3153
20.29
0.7816
0.2920
TiNevVox
19
19.71
0.8109
0.3643
21.26
(18289
0.3967
20.66
08145
0.2690)
21.61
(18234
0.2766
HyperNeRF
0.1777
0.1911
23.45
0.1990
NeRF-DS
24.91
0.871
0.1737
25:13
0.87788
0.1741
19.96
0.8166
0.1855
23.60
0.8494
0.1816
Deformable-
3DGS
h
24.76
0.5576
0.1544
26.08
0,8827
0.1832
19.61
0.7888
0.1871
23.61
0.8391
0.1970
Ours
24.97
0.8916
0.1556
26.56
0.8902
0.1757
20.11
0,8126
0.1865
24.54
0.8656
0.1719
Table 2: Quantitative comparison on HyperNeRF's vrig dataset per-scene.
Printer
Chicken
Mean
Method
PSNR1
SSIM
PSNR
SSIM
PSNR1
SSIM
PSNR1
SSIM
PSNR1
SSIM
HyperNeRI 49
20.0
0.63
27.4
0.63
19.5
0.21
22.1
0.72
22.3
0.55
TiNeuVox 59
22.8
0.73
28.2
0.79
21.3
0.31
24.4
0.64
24.2
0.62
Deformable 3DGS
Fp
20.5
0.64
22.8
0.61
20.5
0.35
26.0
0.83
22.5
0.61
Ours
21.8
0.71
26.8
0.79
22.3
0.38
28.2
0.86
24.8
0.69
Table 3: Ablations on the key components of our proposed framework.
Setting
PSNR t
SSIM
LPIPS |
Baseline
23.61
0.8394
0.1970
optical flow guidance
23.37
0.8333
0,2112
motion flow guidance
24.12
0.8609
0.1763
motion flow guidance + camera pose refinement
24.54
0.8656
0.1719
artifacts. Similarly, in the basin scene, our method effectively models the smooth surface of the basin, in contrast to other methods that result in bumpy basin bottom. This is mainly because our proposed framework can provide accurate and effective motion guidance for Gaussian deformation.
Results on the HyperNeRF dataset. For scenes captured in the wild using smartphones, Table summarizes the relevant performance comparison results. Our method also achieves consistent performance improvements in these scenarios. Qualitatively, our approach excels at accurately recon- structing scene geometry and appearance, even under irregular camera movements and inaccurate camera poses. For instance (see F and [ [3). the chicken scene, our method captures the subtle bumps the red shell, while in the broom scene, it accurately renders the details of the broom in fast movement. This is mainly attributed to the motion guidance and camera pose refinement proposed by our method, both of which enhance the rendering performance of the baseline method.
and
5.3 Ablation Study
In this section, we conduct ablations on the NeRF-DS dataset to validate the effectiveness of the key components of our method, as shown in Table 3 For more ablations, please refer to Appendix A.3
Effectiveness of the optical flow decoupling module. To illustrate the necessity of optical flow decoupling module, we conduct ablations using direct optical flow supervision instead of decoupled motion flow constraints. As shown in the row Table 3 directly using optical flow to supervise Gaussian motion even results in a performance decline compared to the baseline. This performance drop likely due to the inherent ambiguity created by the mixed camera and object movements. When camera and object movements not separated, the supervision signal becomes noisy and less
8

---

Ground Truth
O
Deformable 3DGS
NeRF-DS
3DGS
Figure 5: Qualitative comparison NeRF-DS dataset. Refer to Figure 12 for more scenes.
Chicken
Ground Truth
Ours
Deformable 3DGS
3DGS
Ground Truth
Ours
Defarmable 3DGS
3DGS
Figure 6: Qualitative comparison on HyperNeRF dataset. Refer to Figure for more scenes.
Current frame
Next frame
Rendered Image
Rendered deoth
Optical flow
Camera flow
Motion flow
Gaussian flow
Current frame
Next frame
Rendered image
Rendered depth
Optical flow
Camera flow
Motion flow
Gaussian flow
Figure Visualization of all data flows. Each example corresponds to two rows.
effective. This ambiguity hampers the optimization process of 3DGS, thereby reducing the motion modeling capabilities of the deformation field. In contrast, we use motion flow as supervision, which effectively provides explicit motion guidance for Gaussian deformation, thereby better modeling complex dynamic scenes. As shown in Figures and [4] only the motion flow can clearly highlights movement information in dynamic regions. This explicit motion information efficiently constrains the Gaussian flow, ensuring that the motion guidance remains consistent and effective.
Effectiveness of the camera pose refinement module. Leveraging the alternating optimization of 3DGS and camera poses, our approach adaptively corrects potential errors in camera poses. Further- more, the updated camera poses contribute to more accurate camera flow, thus improving the accuracy of motion guidance. In row 4 of Table 3 our camera pose refinement module, built upon motion guidance, yields substantial performance gains for the model. This iterative optimization process enhances the robustness of our model in complex dynamic scenes. For instance, in the HyperNeRR

---

Ours (finetuned) COLMAP
Ours (finetuned) COLMAP
Ours (finetuned)
Ours (finetuned)
Ours (finetuned)
COLMAP
COLMAP
COLMAP
Figure 8: Visualization of the camera trajectories optimized by our method and COLMAP.
dataset, our method reconstructs more plausible results compared to the baseline approach. Unlike static scene datasets (e.g., Tanks & Temples) that use COLMAP to obtain the ground truth of camera poses, we assume that COLMAP may not provide accurate poses for dynamic scene datasets. In this setting, we lack ground truth for a direct quantitative evaluation for refined camera pose. Therefore, we provide visualizations of the pose refinement process in the Figure X as qualitative comparison.
Conclusion
In this paper, we propose MotionGS, novel deformable 3D Gaussian Splatting framework for explicitly modeling and constraining object motion in dynamic scene reconstruction. The proposed framework includes two key modules: the optical flow decoupling module and the camera poserefine- ment module. The optical flow decoupling module decouples the motion flow related solely to object motion from the optical flow priors, providing explicit supervision for Gaussian deformation. The camera pose refinement module alternately optimizes 3DGS and camera poses, further enhancing the rendering quality and robustness of our model in dynamic scenes. Quantitative and qualitative results on the NeRF-DS and HyperNeRF datasets strongly demonstrate the contributions and effectiveness of our proposed method. More importantly, the proposed improvements are agnostic to specific network designs, which can be applied to similar deformation-base 3DGS methods. In future work, we aim develop a 3DGS method that does not rely on camera pose inputs, thereby achieving robust high-quality reconstruction in dynamic scenes.
Acknowledgements
This work was supported by the Anhui Provincial Natural Science Foundation (Grant No. 2308085QF222), the National Natural Science Foundation of China (Grant No. 12150007), the De- fense Science and Technology Foundation Strengthening Program (Grant No. 2023-JCIQ-IJ-00199, and the Youth Innovation Promotion Association.
References
[1] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguee D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1031 10327, 2021.
[2] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865 5874, 2021.
[3] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498-6500 2021.
[4] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance ficlds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13-23, 2023.
[5] Hang Gao, Ruilong Li. Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Advances in Neural Information Processing Systems, 01.33768-337888 2022.
10

---

[6] Zhengqi Li. Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4273-4284, 2023.
[7] Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes. In Proceedings of the IFEF/CVF Conference on Computer Vision and Pattern Recognition, pages 130-141, 2023.
[8] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang. and Yebin Liu. Tensortd: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16632-166444 16662- 2023.
[9] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[10] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküthlrr and George Drettakis. 3d gaussian splatting for real tile radiance field rendering. ACM Transactions on Graphics, 42(4): 14, 2023.
[11] Jonathon Laiten, Georgios Kopanas, Bastian Leihe, and Deva Ramanan, Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308. 09713, 2023.
[12] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. arXiv preprint arXiv:2312.16812, 2023.
[13] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
[14] Zeyu Yang, Hongye Yang. Zijie Pan, and Li Zhang, Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. International Conference on Learning Representations (ICLR), 2024.
[15] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao,, and Xiaojuan Qi. Se-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. arXiv preprint 1xXXx:33122149377 2023.
[16] Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-foow 4d reconstruction with dynamic 3d gaussian particle. arXiv preprint arXiv:3312.03431 2023.
[17] Ziyi Yang, Xinyu Gao. Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for figh-ficelity monocular dynamic seene reconstruction. arXiv preprint arXiv:2309.13100 2023.
[18] Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang Hc, Wenzheng Chen, and Baoquan Chen. 4d gaussian splatting: Towards efficient novel view synthesis for dynamic scenes, arXiv preprint arXiv:2402.03307 2024.
[19] Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Flowformer: A transformer architecture for optical flow. In European conference on computer vision, pages 668-685. Springer, 2022.
[20] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. In Proceedings of the ILEÉ/CVF Conference on Computer Vision and Patter Recognition, pages 8121-8130, 2022.
[21] Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, and Ulrich Neumann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint arXiv:2403. 12366, 2024.
[22] Zhiyang Guo, Wengang Zhou, Li Li, Min Wang, and Hougiang Li. Motion-aware 3d gaussian splatting for efficient dynamic scene reconstruction. arXiv preprint arXiv:2403. 2024.
[23] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Bartin-Brualla,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[24] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti- aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470- 5479, 2022.

---

[25] Cheng Sun, Min Sun, and Hwann-Tzoen Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[26] Qiangeng Xu, Zexiang Xu. Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: foint-based neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 5438-544 5348-548,222 2022.
[27] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualle Noah Snavely, and Thomas Funkhouser. Ibrntt Learning multi- view image- rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 4690-4699, 2021.
[28] Thomas Müller, Alex Evans. Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1 2022.
[29] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752- 5761, 2021.
[30] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf+++ Analyzing and improving neural radiance fields, arXiv preprint arXiv:2010.07492, 2020.
[31] Ruijie Zhu, Jiahao Chang. Ziyang Song, Jiahuan Yu, and Tianzhu Zhang. Tiface: Improving facial reconstruction through tensorial radiance fields and implicit surfaces. arXiv preprint arXiv:2312.09527. 2023.
[32] Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Opti- mizing sparse-view 3d gaussian radiance fields with global-local depth normalization. arXiv preprint vnxiv:2403.06912 2024.
[33] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang. Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mysplat: Efficient 3d gaussian splatting from sparse view images. arXiv preprint 2403: 14627, 2024.
[34] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction, 2024.
[35] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[36] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint XXXv:2309911653 2023.
[37] Andreas Blattmann, Robin Rombach, Huan Ling. Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.
[38] Jiaxiang Tang. Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large nie gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:4402.05054, 2024.
[39] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 16213, 2023.
[40] Xu Yinghao, Shi Zifan, Yifan Wang, Chen Hansheng, Yang Ceyuan, Peng Sida, Shen Yujun, and Wetzstein Gordon. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation, 2024.
[41] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang. Yikai Wang. Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussiancditor: Swift and controllable 3d editing with gaussian splatting, 2023.
[42] Jiemin Fang, Junjie Wang. Xiaopeng Zhang, Lingxi Xie. and Qi Tian. Gaussianeditor: Editing 3d gaussians delicately with text instructions. In CVPR, 2024.
[43] Shijie Zhou, Haoran Chang. Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. arXiv preprint arXi::2312.03203, 2023.
12

---

[44] Chi Yan. Delin Qu. Dan Xu, Bin Zhao. Zhigang Wang. Dong Wang, and Xuelong Li. Gs-slam: Dense visual slam with 3d gaussian splatting. In CVPR, 2024.
[45] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[46] Hidenobu Matsuki, Riku Murai, Paul H. J. Kelly, and Andrew J. Davison. Gaussian Splatting SLAM. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[47] Huajian Huang, Longwei Li, Cheng Hui, and Sai-Kit Yeung. Photo-slam: Real-time simultancous localization and photorealistic mapping for monocular, stereo, and rgb-d cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
[48] Yilun Du, Yinan Zhang. Hong- Hlong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14304-1433 14314. IEEE Computer Society, 2021.
[49] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A niigher-dimensioall representation for topologi- cally varying neural radiance fields. arXiv preprint arXiv:2106. 1010006..2.. 13228, 2021.
[50] Qianqian Wang, Yen-Yo Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Alcksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19795-19806, 2023.
[51] Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2863-2873, 2022.
[52] Wenqi Xian, Jia Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9421-9431, 2021.
[53] Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis, arXiv preprint arXiv:2/05.05994, 2021.
[54] Chen Gao. Ayush Saraf. Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5712-5721, 2021.
[55] Kaichen Zhou, Jia-Xing Zhong. Sangyun Shin, Kai Lu, Yiyuan Yang, Andrew Markham, and Niki Trigoni. Dynpoint: Dynamic neural point for view synthesis. Advances in Neural Information Processing Systems, 36, 2023.
[56] Sara Fridovich Keil,, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488, 2023.
[57] Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou. 4k4d: Real-time 4d view synthesis at 4k resolution. arXiv preprint arXiv:2310. 1448, 2023.
[58] Feng Wang. Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed neural voxels for fast multi-view video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19706-19715 2023.
[59] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[60] Kai Katsumata, Duc Minh Vo, and Hideki Nakayama. An efficient 3d gaussian representation for monocularmmulti-view dynamic scenes. arXiv preprint arXiv:2311.12892 2023.
[61] Zhicheng Lu, Xiang Guo, Le Hui, Tianrui Chen, Ming Yang, Xiao Tang. Feng Zhu, and Yuchao Dai.
3d geometry-awane deformable gaussian splatting for dynamic view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2024.
[62] Devikalyan Das, Christopher Wewer, Raza Yunus, Eddy llg, and Jan Eric Lenssen. Neural parametric gaussians for monocular non-rigid object reconstruction. arXiv preprint arXiv:3312.01196, 2023.
13

---

[63] Jiakai Sun, Han Jiao, Guangyuan Li. Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. arXiv preprint arXiv:2403.00
2024.
[64] Lin Yen-Chen Pete Florence. Jonathan T Barron, Alberto Rodriguez. Phillip Isola. and Tsung- Yi Lin. inerf: Inverting neural radiance fields for pose estimation. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1323-1330. IEEE, 2021.
[65] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 2021.
[66] Yitong Xia, Hao Tang, Radu Timofte. Lue Gool. Sinerf: Sinusoidal neural radiance fields for joint pose estimation and scene reconstruction. arXiv preprint (xXiv:22100045555 2022.
[67] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5741-5751 2021.
[68] Shin-Fang Chng., Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey. Garf: Gaussian activated radiance fields for high fidelity reconstruction and pose estimation. arXiv e-prints, pages arXiv-2200 2022.
[69] Wenjing Bian, Zirui Wang. Kejie Li, Jia-Wang Wang Bian, and Victor Adrian Prisacariu. nere-nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4160-4169, 2023.
[70] Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A Efros,, and Xiaolong Wang. Colmap-free 3d gaussian splatting. arXiv preprint arXiv:2312.07504, 2023.
[71] Johannes Lutz Schönberger and Mincail Frahm. fructure-from motion Computer Vision and Pattern Recognition (CVPR), 2016.
from-motoonn revisited. In Conference
[72] Hidenobu Matsuki. Riku Murai, Paul H. J. Kelly, and Andrew J. Davison. Gaussian Splatting SLAM. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[73] Zhiwen Yan, Chen Li, and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 8285 8295, 2023.
[74] Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.660 2014.
[75] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[76] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1409.14000000 2014.
[77] René Ranftl, Katrin Lasinger. David Hafner, Konrad Schindler. and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(3). 2022.
[78] Lingtong Kong and Jie Yang. Mdflow: Unsupervised optical flow learning by reliable mutual knowledge distillation. IEEE Transactions on Circuits and Systems for Video Technology. 33(2):677 2022.
[79] Ruijie Zhu, Chuxin Wang, Ziyang Song, Liu, Tianzhu Zhang, and Yongdong Zhang. Scaledepth: Decomposing metric depth estimation into scale prediction and relative depth estimation. arXiv preprint arXiv:2407008187, 2024.
[80] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove. Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5521-5531, 2022.
14

---

A Appendix
A.1 Formulation of Gaussian Flow
I-th Gaussian at time
Canonical Gaussian Space
(-th Gaussian time e
L
pi,t
(0,0)
p1,++
FG 1,++
= Xi,++ Xt
Figure 9; The formulation of Gaussian flow. We first project the point d corresponding to the i-th Gaussian at time t into the canonical Gaussian space, and then reproject this point from the canonical Gaussian space the i-th Gaussian time 1.
Motivated by [21, we formulate the Gaussian flow f-+++1 to simulate the motion of dynamic object the scene, as shown in Figure Specifically, Gaussian flow corresponds to the deformation 3D Gaussians from time t to + in the camera viewpoint Cz+1. For point we first transform it to canonical space corresponding to the Gaussian at time t:
+ Eg (x+ - -4,,),
(10)
where pi,t and Σx, are the center position and covariance matrix of i-th Gaussian at the timestamp t. Then we transform 2, back to the Gaussian at the next time step t + 1:
01,++1 -i,--i + Hi,r
(11)
where 14,4+1 and Ei,+1 the center position and covariance matrix of i-th Gaussian at the timestamp t + 1. Therefore, the flow contribution from i-th Gaussian to this point can be defined as:
FC P Ri,411 Tt-
(12)
Finally, all Gaussian flow contributions to the point be accumulated in similar way a- blendering:
=
Σ
Wi
FG
i1-+0+1
(13)
1=1
wi(Fi,411- xz), -xx),
(14)
where w; is the weight of a--blendering. Note that since the computation of forward optical flow is referenced frame 4, the Gaussian flow should be consistently rendered under the camera viewpoint Cr, corresponding to the decoupled motion flow. It represents the 2D splatting of the Gaussian deformation field from time & when the camera viewpoint remains unchanged.
A.2 More Implementation Details
The NeRF-DS dataset consists eight stereo camera video sequences of daily scenes. These scenes contain high-speed moving high-gloss surface objects and changing camera poses, which pose challenges for dynamic scene modeling. The HyperNeRF dataset includes additional complications such topological changes and inaccurate camera poses. For the NeRF-DS dataset, we use the default resolution x80x270 for all scenes for training and testing. We train the model using images from the left camera and test it on the right camera. For the HyperNeRF dataset, select four sets of scenes in the vrig subset (3D Printer, Chicken, Broom, and Banana) for training and testing, with x downsampling resolution of 536 x 960.
Datasets.
'The canonical space mentioned here should not be confused with the canonical space in the Gaussian deformation field. It represents the standard Gaussian distribution space.
15

---

Table 4: Training time comparison across different models.
Training Time
As
Basin
Bell
Cup
Plate
Press
Sieve
Baseline
Ih
Ih 1mm
Ih 42m
3m
Oh 51m
Oh 57m
Ours (w/o pose refinement)
1h 8m
1h 15m
1h 53m
Ih 13m
1h 6m
Oh 58m
1h 3m
Ours
1h 33m
1h 46m
2h 4m
1h 34m
1h 30m
1h 17m
1h 25m
NeRF-DS
43m
6h 48m
6h 49m
50m
53m
48m
6h 47m
Table Max GPU memory usage comparison across different models.
Max GPU Memory (GB)
As
Basin
Bell
Cup
Plate
Press
Sieve
Baseline
15.67
13.61
15.97
15.29
9.66
10.65
12.17
Ours
16.61
14.52
17.73
15.70
10.62
11.62
12.97
Table 6: FPS, number 3D Gaussians and storage on the NeRF-DS dataset per scene.
Scene
FPS
Nums
Storage
AS
50
178K
49M
Basin
29
250K
70M
Bell
19
379K
97M
Cup
35
200K
54M
Plate
32
220K
58M
Press
45
197K
53M
Sieve
35
200K
54M
Implementation details. To achieve the differentiable Gaussian flow and differentiable camera pose, integrate the forward and backward processes into rasterizer. To provide reliable optical flow, we choose GMFlow [20] as the default optical flow network. In order to stabilize the initialization process of the scene, introduce motion constraints only after the Gaussian starts to deform and move. In the NeRF-DS dataset, we set the weight the flow loss À to 0.5; while in the HyperNeRF scene, it is set to 0.1. Camera pose optimization is also only activated during the Gaussian deformation stage.
Data sampling mechanism. We adopt the same data sampling strategy as the baseline method, i.., reading image sequences in randomly shuffled order. For an N-frame video, the frames are shuffled and then read sequentially. In each iteration, we read two frames and calculate the optical flow between them. To enhance efficiency, the second image from the last iteration is used as the first image the current iteration. Thus, except for the first iteration, only one new image is read in each subsequent iteration. Consequently, there are N iterations per epoch, with optical flow computed once in each iteration. This strategy balances the introduction of accurate motion priors with maintaining training efficiency. During the first epoch of training, calculate the optical flow for all adjacent frame pairs, resulting in a total of N optical flow maps. In subsequent epochs, do reshuffle the image sequence, allowing the first epoch. This effectively eliminates the need to recompute optical flow maps in each epoch, significantly reducing computational overhead.
to reuse the optical flow maps calculated in
Training time and GPU memory. For model training, list the training time per scene and peak memory usage on the NeRF-DS dataset as shown in Tables 4 and providing a comprehensive assessment of resource usage during training. Compared increased training time and peak memory usage. This is primarily due to the additional rendering of Gaussian flow and the refinement of camera poses, which necessary for our method.
baseline, our approach incurs
FPS, number of 3D Gaussians and storage. We provide statistics of FPS and number of Gaussians on NeRF-DS dataset, as shown in Table In most scenes of NeRF-DS, our method MotionGS achieves real-time rendering (FPS>33)..
16

---

Table 7: Ablations other choices of our proposed framework. For fair comparison, we do not activate the proposed camera pose refinement module during training.
Row
Setting
PSNR
SSIM t
LPIPS
Baseline
23.61
0.8394
0.1970
w/o Motion mask
23.13
0.8242
0.2249
Different depth choice (Midas 77
23.58
0.8384
0.1969
4
Different optical flow network Former [19]
23.97
0.8525
0.1893
Different optical flow network (MDFlow (78)
23.25
0,8308
0.2137
Self-supervised flow supervision loss
23.76
0.8474
O. 1807
Lower flow loss weight (A 0.2)
23.46
0.8343
0,2042
8
Higher flow loss weight (A 0.8)
23.75
0.8470
0.1819
Ours (w/o camera posc refinement)
24.12
0.8609
0.1763
A.3 More Ablations
We summarize the ablations other choices of proposed framework in Table For fair comparison, we do not activate the proposed camera pose refinement module during training, since also influences the flow calculation. Our interpretation and analysis of the ablations
follows.
Effectiveness of motion mask (row 2). Introducing a motion mask allows the motion flow to focus on the motion of dynamic objects, thereby reducing interference from static areas. When the motion mask is removed, the performance declines. We attribute this degradation to inaccurate optical flow in the background areas, which introduces errors in the motion guidance and subsequently leads to incorrect Gaussian deformations.
Different depth choice (row 3). Estimating the accurate depth maps for depth warping a critical issue when calculating camera flow. We find that using depth prediction from offline estimator Midas (77) yields suboptimal results. This approach degrades the quality of subsequent motion flow, reducing the accuracy of motion constraints and ultimately impacting the reconstruction quality. We attribute this degradation to the inherent scale ambiguity [79] the depth estimator, as shown in Figure I0] In contrast, using rendered depth by 3DGS ensures scale and geometric consistency and provides superior detail.
Owrs
MiDas
Figure 10: Rendered depth from 3D Gaussian splatting (ours) and off-the-shelf monocular depth estimator (MiDas). Our rendered depth has richer details and is scale-aligned with the scene. MiDas rendered depth is usually more smooth and suffers from scale ambiguity.
Different optical flow network (row 4-5). Our method relies on existing 2D optical flow estimators to provide motion guidance for the 3D Gaussian fields. The choice of optical flow prior can lead to performance differences. When we replace GMFlow (20) with another supervised method FlowFormer [19], the performance deteriorates. This is mainly due to the fact that FlowFormer performs inadequately the "plate" scene, resulting in an overall performance decrease. Additionally, when replace GMFlow (20] with self-supervised method MDFlow (78), the performance is
17

---

even worse. This phenomenon may also illustrates the importance of accurate motion priors, while erroneous or noisy motion constraints may even have negative effect on the optimization.
Self-supervised flow supervision loss (row 6). Inspired by self-supervised optical flow estimation methods, we attempt to provide motion priors in a self-supervised manner. Specifically, we estimate the Gaussian flow corresponding to the optical flow and use it to warp the L, frame. We then compute the photometric loss with the 4+1 frame. As shown the table, this method outperforms our baseline but is less effective compared to proposed method. We hypothesize that the discrepancy arises because the self-supervised loss may not provide accurate supervision in areas with similar colors. Nevertheless, it is evident that employing self- supervise optical flow loss reduce dependence off-the-shelf optical flow estimation. When an optical flow estimation network either unavailable inaccurate, this approach can as valuable alternative improve rendering quality.
Different flow loss weights (row 7-8). We compare the rendering performance under different flow loss weights. The results indicate that the selected weight (A 0.5) achieves the best rendering quality. We speculate that excessively large loss weights may disrupt the original optimization process based on rendering losses, while too small weights may result in insufficient motion guidance.
A.4 More Visualizations
Please refer Figures 12
1 and 14
A.5 Limitation
Ground Truth
Ours Motion flow supavisee
Ours Optical flow supervised
Failure case in DyNeRF dataset. Since the viewpoints are fixed and sparse, neither motion flow nor optical flow can help method avoid floating artifacts.
Figure 11:
During our experiments, we identify several unresolved issues. Specifically, when applying method to the DyNeRF dataset [80), we encounter significant challenges as illustrated in Figure[T] Upon further analysis, find that the fixed and sparse camera viewpoints in the DyNeRF dataset hinder accurate depth rendering, affecting subsequent camera flow calculations and leading to artifacts. The inaccuracies in motion flow primarily comes from the inaccuracy of the camera flow, rather than a failure of the optical flow estimation itself. It is also important to clarify that the DyNeRF dataset not continuous monocular video but rather dynamic scenes with sparse viewpoints, which posed challenges to the canonical 3D Gaussian initialization. Moving forward, our focus will be on addressing these issues to further improve the robustness of our model in dynamic scene reconstruction. We aim to develop more stable and reliable motion priors and adapt our approach to handle scenarios with minimal object movement more effectively. By doing so, we hope to extend the applicability and reliability of our method across wider range of dynamic scenes.
A.6 Broader Impacts
To the best of our knowledge, the proposed method will not have significant negative social impact. The proposed dynamic reconstruction method can be used to reconstruct and render some daily dynamic scenes. Users can use the video shot by their mobile phones input to obtain an explicit 3D asset represented by Gaussian and deformation field. This 3D asset can be used for subsequent editing, development, secondary creation for entertainment.
18

---

A.7 Data Availability
The datasets that support the findings of this study are available the following repositories: NeRF DS [73] at https://gt thub. com
wataa////oooaaaeeaaa//ooo.o 1ttp://oo.-prer-eeeeee under Apache- 2.0 license, HyperNeRF 49 at https://githuu com google/ h
hypernert/ releases/ taap:/tt///v vo. under 2pa license. The code of our baseline [7 is available at
https://g thub http:://g.......ooooo/gggaaa/aa/ wtt.ggrrraaae/eoooaa/aaaaaaaaa 3D- Gauss iinssiissss under MIT license.
Ground Truth
Our
Deformable 3DGS
NeRF-DS
3DGS
besin
bell
cup
plaee
Slave
Figure 12: Qualitative comparison on NeRF-DS dataset per-scene. Compared with the state-of- the-art methods, our method can render more reasonable details, especially on dynamic objects.
19

---

Ground Truth
Ours
Deformable 3DGS
HyperNeRF
3DGS
3D Printer
Broom
Chicken
Peel Banana
Figure 13: Qualitative comparison on HyperNeRF dataset per-scene. Compared with the state- of-the-art methods, our method is more robust reconstructing dynamic scenes. Even if the input camera pose is not accurate on HyperNeRF dataset, method can adaptively optimize the camera poses and produce reasonable rendering results.
20

---

Figure 14: Visualization of all data flows. In order: ground truth of 7,, ground truth of Te41, rendered image of Tt, rendered depth of frame Jt optical flow, camera flow, motion flow, Gaussian flow.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state the main contributions of this paper and the differences from previous methods in both the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Appendix A.5 for the discussion of limitations and failure cases.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
We provide the necessary formulas and related explanations in the main text. Please see refer to Appendix A. for the derivation of Gaussian flow.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We identify our baseline method and explain the implementation details where we differ from the baseline method in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The datasets and baseline methods this paper are publicly available. Our source code will be openly accessible after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppllligggddpeeeeeeeeeee Cooee///ddeeebmmmiia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided necessary implementation details in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the experimental settings of previous works to conduct experiments. Therefore, the comparison with previous works is fair.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the information our computer resources in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
We ensure that our experiments comply with the NeurIPS Code of Ethics in all respects.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to our discussion on broader impacts in Appendix A.6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks for misuse of released data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have appropriately cited related work and abide by their licenses and terms. Please refer to Appendix A.7
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "4pjmpjQphP", "reviewer": "Reviewer_rwSL", "summary": "This paper presents a novel approach to dynamic scene reconstruction by incorporating explicit motion priors into 3D Gaussian Splatting (3DGS). The proposed framework, MotionGS, introduces an optical flow decoupling module that separates camera flow and motion flow, which respectively correspond to camera movement and object motion. This separation allows for more precise motion guidance during the deformation of 3D Gaussians. Additionally, a camera pose refinement module is implemented to alternately optimize 3D Gaussians and camera poses, addressing inaccuracies in camera pose estimation. Extensive experiments demonstrate that MotionGS outperforms state-of-the-art methods on datasets such as NeRF-DS and HyperNeRF, achieving significant improvements in both qualitative and quantitative results for dynamic scene reconstruction.", "questions": "If no flow net is adopted, is it possible to calculate the loss as follows?\n1. Estimate optical flow as the sum of camera flow and motion flow\n2. Using optical flow to warp I_t+1, and calculated the loss with I_t.", "limitations": "No negative societal impact", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is written clearly: The authors present a well-structured flow from the problem definition, through the intuition behind the approach, to the implementation and analysis. Each section is closely connected to the main point, making the paper easy to follow.\n\n2. The proposed method is novel, intuitive, and simple to implement: MotionGS introduces a unique approach by decoupling optical flow into camera flow and motion flow, providing explicit motion guidance for 3D Gaussian Splatting. This method is innovative and straightforward, making it easy to adopt and integrate into existing systems.\n\n3. Experiments validate the results well, with extensive visualizations and ablation studies: The paper robustly validates the proposed method with extensive experiments on datasets like NeRF-DS and HyperNeRF. Numerous visualizations and detailed ablation studies clearly demonstrate the effectiveness of each component of MotionGS.", "weaknesses": "1. The time, memory, and storage costs are not revealed: While the proposed method is illustrated as a simple solution, it is crucial to analyze the additional computational burden it imposes. The paper does not provide an analysis of the time, memory, and storage requirements, which are important factors in evaluating the practicality of the method.\n\n2. The data sampling mechanism is missing: The paper lacks details on the data sampling mechanism used for training. It is unclear how image pairs are sampled—whether t and t+1 can be any two frames in the video or they must be adjacent to each other. If the former is true, it would imply that N(N−1) optical flow maps need to be calculated, which could significantly impact storage and time requirements. Providing these details is essential to understand the overall efficiency and feasibility of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Section 1 Response to Reviewer rwSL\nWe thank the reviewer for the constructive assessment of our work. In the subsequent sections, we respond to each concern in detail. Please feel free to use the discussion period if you have any additional questions.\n## 1.1 Weaknesses\n### 1.1.1 Time, memory, and storage costs\nThank you for your question. In Table 4 of Appendix A.2, we have provided a detailed breakdown of the rendering speed and storage requirements of our method on the NeRF-DS dataset. Our approach aims to introduce explicit motion constraints and pose refinement during training without bringing additional burden to the original Deformable-3DGS during inference. Thus, the rendering speed and inference time of our method are consistent with our baseline.\n\nFor model training, we list the training time per scene and peak memory usage on the NeRF-DS dataset as below, providing a comprehensive assessment of resource usage during training. Compared to our baseline, our approach incurs increased training time and peak memory usage. This is primarily due to the additional rendering of Gaussian flow and the refinement of camera poses, which are necessary for our method.\n## Training Time on the NeRF-DS Dataset\n| Training Time                         | As     | Basin   | Bell   | Cup    | Plate   | Press   | Sieve   |\n|:-------------------------|:-------|:--------|:-------|:-------|:--------|:--------|:--------|\n| Baseline         | 1h 1m  | 1h 11m  | 1h 42m | 1h 3m  | 1h 0m   | 0h 51m  | 0h 57m  |\n| Ours(w/o pose refinement) | 1h 8m  | 1h 15m  | 1h 53m | 1h 13m | 1h 6m   | 0h 58m  | 1h 3m   |\n| Ours                     | 1h 33m | 1h 46m  | 2h 34m  | 1h 34m | 1h 30m  | 1h 17m  | 1h 25m  |\n| NeRF-DS                  | 6h 43m | 6h 48m  | 6h 49m | 6h 50m | 6h 53m  | 6h 48m  | 6h 47m  |\n## Max GPU Memory Usage on the NeRF-DS Dataset\n| Max GPU Memory (GB)         |    As |   Basin |   Bell |   Cup |   Plate |   Press |   Sieve |\n|:---------|------:|--------:|-------:|------:|--------:|--------:|--------:|\n| Baseline | 15.67 |   13.61 |  15.97 | 15.29 |    9.66 |   10.65 |   12.17 |\n| Ours     | 16.61 |   14.52 |  17.73 | 15.7  |   10.62 |   11.62 |   12.97 |\n### 1.1.2 Data sampling mechanism\nThanks for your question. Here is a more detailed explanation of our data sampling mechanism:  \n-  **Data Sampling Strategy**: We adopt the same data sampling strategy as the baseline method, i.e., reading image sequences in a randomly shuffled order. For an N-frame video, the frames are shuffled and then read sequentially. In each iteration, we read two frames and caluate the optical flow between them. To enhance efficiency, the second image from the last iteration is used as the first image in the current iteration. Thus, except for the first iteration, only one new image is read in each subsequent iteration. Consequently, there are (N-1) iterations per epoch, with optical flow computed once in each iteration. This strategy balances the introduction of accurate motion priors with maintaining training efficiency.\n\n- **Optical Flow Calculation and Storage Strategy**: During the first epoch of training, we calculate the optical flow for all adjacent frame pairs, resulting in a total of (N-1) optical flow maps. In subsequent epochs, we do not reshuffle the image sequence, allowing us to reuse the optical flow maps calculated in the first epoch. This effectively eliminates the need to recompute optical flow maps in each epoch, significantly reducing computational overhead. Taking the \"as\" scene (846 frames) in the NeRF-DS dataset as an example, the optical flow calculation for each pair takes approximately 22 ms, resulting in a total computation time of 18.7 seconds. Each optical flow map requires around 0.99 MB of storage, resulting in a total storage requirement of 835.13 MB.\n\n## 1.2 Questions\n### 1.2.1 Self-supervised flow loss\nWe appreciate the insightful suggestion provided by the reviewer. Following the reviewer's suggestion, we conduct experiments to evaluate this approach and have confirmed its effectiveness. Specifically, we estimate the Gaussian flow corresponding to the optical flow and use it to warp the $I_t$ frame. We then compute the photometric loss with the $I_{t+1}$ frame. In experiments on the NeRF-DS dataset shown as below, this method outperforms our baseline but is less effective compared to our proposed method. \n## Novel View Synthesis Results of NeRF-DS Dataset\n| Method        |   SSIM↑  |   PSNR↑  |    LPIPS↓   |\n|:--------------|-------:|-------:|-----------:|\n| baseline      | 0.8394 | 23.61| 0.1970     |\n| ours(w/o pose refinement)          | **0.8609** | **24.12**| **0.1763**    |\n| Self-supervised flow loss       | 0.8474 | 23.76| 0.1807     |\n\nWe hypothesize that the discrepancy arises because the self-supervised loss may not provide accurate supervision in areas with similar colors. Nevertheless, it is evident that employing self-supervised optical flow loss can reduce dependence on off-the-shelf optical flow estimation. We provide qualitative experimental results in Figure A of the PDF attachment, which show that this idea can effectively provide motion constraints. When an optical flow estimation network is either unavailable or inaccurate, this approach can serve as a valuable alternative to improve rendering quality. \n\nWe sincerely appreciate your valuable idea, which has greatly enhanced our understanding to the problem. We are excited to incorporate this perspective into our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GnZdwzSN6Q", "reviewer": "Reviewer_4hTv", "summary": "The paper proposed MotionGS a novel deformable 3D Gaussian splatting approach. The approach initializes camera poses and 3D Gaussians based on an analytic structure-from-motion method as 3DGS. In addition an optical flow network is used to compute optical flow between neighboring frames. Given the initial depth of the 3D Gaussians, the paper proposes to use an optical flow decoupling module separating the camera and object motion. This process is optimized to obtain better camera poses and better views through 3D Gaussian splatting and then iterated. The paper uses Gaussian flow which has been very recently proposed by Gao et al., 2024, however only matching the flow due to object motion. The iterative improvement of the camera poses is based on the estimation of small residuals between consecutive views. The paper reports experiments on the NeRF-DS and the HyperNeRF dataset and achieving high construction quality and improving of Deformable-3DGS. An ablation study is presented comparing object motion flow with optical flow without separation and the effects of camera pose refinement. The supplemental material contains more details on Gaussian flow, more implementation details, visualizations and discussions. The appendix also contains a further ablation study quantifying the influence of the depth, optical flow etc.", "questions": "The key for MotionGS appears to be the separation of optical flow into object flow and camera flow. The paper does not really review work in this area, would there other ways to attribute the flow to object and camera motion?\n\nIf Deformable 3DGS fails to find a reasonable canonical representation at initialization, is there a fallback to start the iterative optimization?\n\nOn which data was the optical flow network trained on, and has the optical flow network been finetuned on the data? \n\nWould self-supervised optical flow methods which are known to generalize better, improve the flow estimation?", "limitations": "The authors provide a limitation statement in Appendix A.5 but focused only on optical flow failures. The claim of optical flow failure for static cameras is surprising, maybe a shortcoming of the particular flow estimation. It would be more interesting to investigate the robustness to large motions which likely needs to severe flow failures. In addition the limitation of structure-from-motion and depth estimation are not discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Explicit motion guidance for 3D Gaussian reconstruction is a logical next step for deformable 3D Gaussians. \n\nThe paper combines and extends some very recent parallel work in optimization of camera pose estimation by Fu et al., 2024 and optical flow based Gaussian flow by Gao et al., 2024. \n\nThe paper demonstrates the successful application of the proposed MotionGS framework to non-rigid scenes recorded by video streams.\n\nThe reported reconstruction quality of MotionGS is competitive and often state-of-the-art for videos in the chosen datasets.", "weaknesses": "The paper has a large range of limiting assumptions which are not always made clear. COLMAP must find an initial solution for camera pose estimation which requires enough static features in the scene. A video input stream is needed to ensure that neighboring viewpoints are very close for optical flow estimation to succeed. The 3DGS canonical representation must be found in order for depth estimation to work.\n\nThe datasets used in evaluating the proposed method can be processed with deformable 3DGS and hence the proposed MotionGS method is limited to improving reconstruction quality. No reconstruction beyond the limitation of prior work is demonstrated and the above limiting assumptions makes this likely challenging.\n\nThe quality of camera pose estimation is not well evaluated. There is only a visualization in the appendix which is very hard to interpret. (See Fu et al. 2024 for a more meaningful evaluation).\n\nThe proposed method appears to be quite brittle as can see from the ablation study in Table 5 in appendix A.3. The method falls below the baseline by leaving out the motion mask, switching to single image depth estimation and even by switching to another SOTA flow estimation network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Section 2 Response to Reviewer 4hTv\nWe thank the reviewer for the constructive assessment of our work. In the subsequent sections, we respond to each concern in detail. Please feel free to use the discussion period if you have any additional questions.\n## 2.1 Weaknesses\n### 2.1.1 Limiting assumptions\nPlease see Section 0.1 for details.\n### 2.1.2 Beyond the limitation of prior work\nWe appreciate the reviewer's careful examination. Indeed, in some challenging scenarios, our baseline method is confined to reconstructing static backgrounds and exhibits limited performance on dynamic objects. As shown in Figure 6, the baseline method fails to capture even the basic structure of the broom. The baseline method also acknowledges this limitation and attributes it to inaccuracies in camera poses. Therefore, we propose camera pose refinement module to iteratively optimize the 3D Gaussians and camera poses. As a result, MotionGS is more capable of handling scenes with complex motions and inaccurate camera poses than our baseline method.\n### 2.1.3 The quality of camera pose estimation\nPlease see Section 0.3 for details.\n### 2.1.4 Ablation study in Table 5\nWe appreciate the reviewer's detailed analysis of our work. Here, we provide a detailed explanation and clarification regarding the ablation study presented in Table 5: \n1. **The Role of the Motion Mask:**\n   - The motion mask plays a critical role in our pipeline. As discussed in Section 4.2 of the main paper, our method benefits more from the motion mask than previous approaches[7]. Specifically, our method effectively utilizes the motion mask to filter out unreasonable motion flow in static areas. In contrast, when only optical flow supervision is used, the motion mask cannot be directly applied.\n2. **Limitations of Monocular Depth Estimator:**\n   - Our ablation experiments show that using an off-the-shelf monocular depth estimator introduces a performance drop. We argue that this is due to the scale ambiguity of monocular depth estimators, which results in inaccurate camera flow and motion guidance. In contrast, depth maps rendered by our method provide accurate scales and better details, as shown in the Figure B of the PDF attachment.\n3. **Impact of Optical Flow Networks:**\n   -   The performance drop observed when switching optical flow networks is indeed unexpected.  With further analysis, we find that FlowFormer performed inadequately in some challenging scenes, especially for the \"plate\" scene, resulting in an overall decrease in performance.  Since GMFlow is trained on more datasets, it may have stronger zero-shot generalization capability than Flowformer.  \n## 2.2 Questions\n### 2.2.1 Related work on optical flow decoupling\nPlease see Section 0.2 for details.\n### 2.2.2 Fallback to start the iterative optimization\nThank you for this important question. Like our baseline method, we freeze the deformation field during the Gaussian initialization to obtain a reliable canonical representation. In most tested datasets, this canonical representation can be initialized well. Even with inaccurate camera poses, our method can render satisfactory results through iterative optimization. Currently, we have not yet explored extreme cases (e.g. COLMAP failure) or developed alternative initialization strategies to replace COLMAP.  \n### 2.2.3 Optical flow network\nThanks for your question. The GMFlow model[4], is trained on a range of datasets (KITTI, HD1K, FlyingThings3D, and Sintel). We use the pre-trained model from the original paper without further fine-tuning on our datasets.\n### 2.2.4 Self-supervised optical flow methods \nThank you for the valuable suggestion. We used the self-supervised optical flow estimation algorithm MDFlow[5] in our ablation experiments. The results show that the motion constraints provided by the self-supervised optical flow estimation network cannot bring effective improvement to the network. \n\n## Novel View Synthesis Results of NeRF-DS Dataset\n| Method        |   SSIM↑  |   PSNR↑  |    LPIPS↓   |\n|:--------------|-------:|-------:|-----------:|\n| baseline      | 0.8394 | 23.61| 0.1970     |\n| ours(w/o pose refinement)          | **0.8609** | **24.12**| **0.1763**   |\n| Self-supervised flow estimator      | 0.8308 | 23.25| 0.2137     |\n\nWe speculate that this is because self-supervised methods cannot provide sufficiently accurate optical flow priors. With the increasing amount of annotated data and the development of foundational models in recent years, the generalization ability of fully-supervised optical flow estimation methods has significantly improved, surpassing traditional self-supervised methods. Similarly, works like MonoNeRF[6], Dynpoint[3], and DynIBaR[7] also use fully-supervised optical flow networks for optical flow priors.This phenomenon may also illustrates the importance of accurate motion priors, while erroneous or noisy motion constraints may even have a negative effect on the optimization.\n\n## 2.3 Limitations\nWe appreciate the reviewer's insightful feedback. We recognize that the term \"instability in optical flow computation\" mentioned in our paper might have introduced some confusion. Upon further analysis, we find that the fixed and sparse camera viewpoints in the DyNeRF dataset hinder accurate depth rendering, affecting subsequent camera flow calculations and leading to artifacts as shown in Figure B of the PDF attachment. The inaccuracies in motion flow primarily comes from the inaccuracy of the camera flow, rather than a failure of the optical flow estimation itself.  It is also important to clarify that the DyNeRF dataset is not continuous monocular video but rather dynamic scenes with sparse viewpoints, which posed challenges to the canonical 3D Gaussian initialization. This poor initialization prevents our method from rendering accurate depth and performing subsequent optimizations, and in future work we consider combining sparse-view 3DGS methods to improve the robustness of our MotionGS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "snGLhor8km", "reviewer": "Reviewer_pL6e", "summary": "This paper proposes using off-the-shelf 2D optical flow to supervise the deformation field for 3D Gaussian Splatting (3DGS) in dynamic scenes. The optical flow is decomposed into camera flow and motion flow. The 3DGS flow is projected into 2D to match the estimated flow. Camera pose and Gaussian parameters are optimized iteratively. Experiments demonstrate the effectiveness of the proposed method.", "questions": "1.Without explicitly modeling reflections, how does it perform well on the NeRF-DS dataset, even outperforming NeRF-DS with additional modeling?\n2.With additional optimizations, how much longer would the training time be compared to the baseline and other methods?", "limitations": "Yes, other limitations are listed in the weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The formulation and utilization of optical flow are reasonable, and the experiments demonstrate its effectiveness.", "weaknesses": "1.The idea of using optical flow to guide dynamic 3DGS modeling is not particularly novel; it seems more like an engineering effort.\n2.With the additional 2D optical flow supervision, the improvement shown in the experiments is subtle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Section 3 Response to Reviewer pL6e\nWe thank the reviewer for the constructive assessment of our work. In the subsequent sections, we respond to each concern in detail. Please feel free to use the discussion period if you have any additional questions.\n## 3.1 Weaknesses\n### 3.1.1 Novelty\nWe appreciate your detailed review of our work. We speculate that there might be some misunderstandings about the novelty and contributions of our work, and we'd like to make further clarification.\n\nOur primary contribution is the separation of camera motion and object motion in optical flow, which provides clear motion guidance for dynamic Gaussians. Specifically, our method begins by acquiring an optical flow prior through an off-the-shelf optical flow estimation network. It then calculates the optical flow caused solely by camera motion by integrating depth and camera pose. Finally, it isolates the optical flow attributed to object motion to constrain the deformation of 3D Gaussians. This approach effectively creates a precise correspondence between the 2D motion prior and the 3D Gaussian motion.\n\nTherefore, it is not entirely accurate to characterize our work as merely applying optical flow supervision to Gaussian motion. As shown in Table 3 of our main paper, directly constraining the Gaussians and deformation fields with optical flow even results in performance degradation. We believe this is because the camera motion incorrectly influence the deformation of dynamic Gaussians, while our method effectively addresses this issue.\n\nPrevious research on motion in NeRF and 3DGS fields, such as [7] and [8], have not addressed the interplay between camera and object motion from the perspective of optical flow formation. In contrast, our method first proposes to decouple optical flow for motion guidance, presenting a novel solution for dynamic scene rendering.\n### 3.1.2 Performance gains\nWe appreciate the reviewer's examination of our work. We would like to provide a detailed clarification and further explanation regarding our performance contributions.\nPrevious dynamic Gaussian methods often struggle to achieve accurate reconstruction in dynamic scenes with complex motion and imprecise poses, sometimes even failing to recover the structure of dynamic objects, as illustrated in Figure 6 of our main paper. By incorporating both motion guidance and camera pose optimization, our approach achieves more accurate reconstruction in complex dynamic scenes, as demonstrated in Table 1 and 2 of our main paper (with a 0.93dB mean PSNR increase in NeRF-DS and 2.3dB mean PSNR in HyperNeRF). The ablation study also indicates that our method significantly improves the rendering quality in scenarios with complex motion and inaccurate poses. \n## 3.2 Questions\n### 3.2.1 Modeling reflections\nWe appreciate the reviewer's questions. Indeed, we do not explicitly model reflections like NeRF-DS does. However, our approach has two advantages over NeRF-DS:\n- Effectiveness of the deformable 3DGS Framework: \n\nThe 3DGS technique represents the scene as a set of anisotropic 3D Gaussians and employs an efficient differentiable rasterizer for rendering, achieving high-quality and real-time results [9]. On multiple datasets, 3DGS has matched or surpassed the performance of previous state-of-the-art NeRF methods. Our baseline method extends the rendering quality advantages of 3DGS to dynamic scenes by incorporating deformation fields into the 3DGS framework. As shown in Table 1, even our baseline method achieves rendering performance comparable to NeRF-DS methods without explicit modeling of reflections. The anisotropic 3D Gaussian spheres and the deformation field can model complex geometric details over time, enabling the baseline method to deliver high-quality rendering in dynamic scenes.\n- Effectiveness of Motion Guidance: \n\nWhile the baseline method delivers satisfactory rendering in most cases, there are still limitations in some challenging dynamic regions (e.g., the plate in Figure 5 of our main paper). Our method enhances performance in dynamic scenes by providing reliable motion constraints to the Gaussian deformation. By accurately separating and constraining different motion components in dynamic scenes, our method performs exceptionally well even in foreground areas with complex textures and reflections.\n### 3.2.2 Training time\nWe appreciate the reviewer's question. We recognize that training time is a crucial factor in assessing the practicality of a method. Below, we provide detailed data of training times for our method on the NeRF-DS dataset:\n## Training Time on the NeRF-DS Dataset\n| Training Time                         | As     | Basin   | Bell   | Cup    | Plate   | Press   | Sieve   |\n|:-------------------------|:-------|:--------|:-------|:-------|:--------|:--------|:--------|\n| Baseline         | 1h 1m  | 1h 11m  | 1h 42m | 1h 3m  | 1h 0m   | 0h 51m  | 0h 57m  |\n| Ours-w/o pose refinement | 1h 8m  | 1h 15m  | 1h 53m | 1h 13m | 1h 6m   | 0h 58m  | 1h 3m   |\n| Ours                     | 1h 33m | 1h 46m  | 2h 34m  | 1h 34m | 1h 30m  | 1h 17m  | 1h 25m  |\n| NeRF-DS                  | 6h 43m | 6h 48m  | 6h 49m | 6h 50m | 6h 53m  | 6h 48m  | 6h 47m  |\n\nOur method exhibits a slight increase in training time compared to the baseline method. This is primarily due to the inclusion of a differentiable Gaussian flow rasterizer[8] and the process of camera pose refinement. Notably, since the proposed improvements only influence the training process, our method maintains same real-time rendering speeds with our baseline during inference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vjKO3l91ib", "reviewer": "Reviewer_2xXk", "summary": "The proposed method extends the idea of GaussianFlow to explicitly decouple camera motion and object motion from optical flow on input monocular video. Specifically, the iterative camera pose refinement further boosts the rendering quality and performance on various datasets. Extensive qualitative and quantitative results are shown in the paper to valid the proposed contributions.", "questions": "1. Since the paper showcased the failure case in Fig.13, I would like to know whether the results could be improved if the camera refinement step was removed (leaving that optical flow equals to motion flow) considering camera poses in DyNeRF dataset are fixed? (Original GaussianFlow paper minimized the difference between optical flow and Gaussian flow, where the proposed method minimized the difference between Gaussian flow and motion flow, which is optical flow - camera flow given camera flow in DyNeRF is 0).", "limitations": "Limitations have already been included and acknowledged in the paper. The proposed method seems to be less effective on cases with less camera motions.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper extends the idea of GaussianFlow to explicitly decouple the camera motion and object motion from optical flow, which is reasonable and the formulation of the proposed method also makes sense.\n2. Explicitly modeling camera motions in an iterative way during training is useful for monocular NVS because the rendering error partially brought by incorrect camera pose estimation could be further decreased. Though the camera pose refinement is not originally introduced by the paper, the decoupling scheme is highligh.\n3. Ablation study (both the main paper and appendix) is detailed and inspiring, validating the main contributions claimed in the paper. \n4. Both qualitative and quantitative results look good.", "weaknesses": "No prominent weakness is witnessed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Section 4 Response to Reviewer 2xXk\nWe thank the reviewer for the constructive assessment of our work. In the subsequent sections, we respond to each concern in detail. Please feel free to use the discussion period if you have any additional questions.\n## 4.1 Questions\n### 4.1.1 Failure case in DyNeRF dataset\nThank you to the reviewer for the constructive feedback. As discussed in our response to reviewer 4hTV, the failure on the DyNeRF dataset primarily stems from incorrect computation of the camera flow. More specifically, this issue arises from inaccurate depth estimation. As illustrated in Figure D of the PDF attachment, our method renders incorrect depth, leading to noticeable artifacts. Theoretically, removing the camera refinement step and directly deriving the Gaussian flow from the optical flow can avoid introducing erroneous depth. Unfortunately, after conducting experiments as suggested by the reviewer, the rendered scenes still exhibit certain artifacts, which typically appear only in the test views and not in the training views. We speculate that this issue arises because of the sparse viewpoint setup in the DyNeRF dataset, which is composed of videos captured by several fixed-position cameras rather than a single monocular video. These sparse viewpoints may lead to suboptimal Gaussian initialization and cause the model to overfit to the training views during subsequent optimization.\n\nNevertheless, we sincerely thank the reviewer for this insightful ideas, which inspired us to think more deeply about the possible limitations of our method. For scenes with stationary cameras, using reliable depth priors to provide regularized constraints may be a potential solution to alleviate this ill-posed problem, such as the recent work MoDGS[10]. In future work, we aim to combine sparse-view 3DGS methods to enhance the quality of dynamic scene reconstruction under sparse viewpoint conditions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Section 0: Response to all reviewers
We would like to extend our heartfelt gratitude to all the reviewers for their thorough evaluation and constructive feedback on our work. Their insights have been invaluable in refining and enhancing the quality of our research. Below, we provide additional explanations and clarifications, along with a list of references cited during the rebuttal period and an attached PDF containing supplementary figures.

## Section 0.1: Clarifications on Assumptions and Preconditions
To provide a clearer understanding of our approach, we would like to further explain the assumptions and preconditions underlying our method:

1. **Use of COLMAP:**
   - Like many 3DGS-based methods, our approach relies on COLMAP for initial camera pose estimation, which requires the presence of sufficient static features in the scene. Typically, such static features are widely present in most real-world scenarios, especially in background areas. Moreover, even if the initial poses provided by COLMAP are not perfectly accurate (e.g., HyperNeRF dataset), our camera pose refinement module can adaptively adjust these poses, ensuring high-quality reconstruction of dynamic scenes.

2. **Use of Optical Flow Estimation Network:**
   - We recognize that computing optical flow requires sufficient overlap between images. In all the tested scenes, this overlap condition is consistently met between any two frames (see data sampling mechanism in Section 1.1.2).
   - For cases where there might be significant viewpoint differences in long videos, we recommend segmenting the video and shuffling the frames within each segment to ensure sufficient overlap between adjacent frames in the shuffled sequence.

3. **Canonical Representation of 3DGS and Depth Estimation:**
   - Similar to our baseline, we first initialize a canonical 3DGS without deformation fields. This is crucial for obtaining scale-consistent depth necessary for subsequent camera flow computation. Poor Gaussian initialization might be a challenge of our method, as seen in the failure case in DyNeRF scenes.

## Section 0.2: Related Work on Optical Flow Decoupling
To offer a more comprehensive understanding of our approach, we further discuss related work on optical flow decoupling. Utilizing optical flow to provide motion priors for dynamic scenes has been explored in previous works. For instance, Dynamo-Depth[1] synthesizes optical flow from both camera motion and object motion for self-supervised depth estimation. However, methods for dynamic scene reconstruction typically do not explicitly decompose optical flow into camera and object motions. Instead, they focus on identifying pixel correspondences in 3D space, known as scene flow. For example, NSFF[2] combines the predicted scene flow with camera poses to project points onto adjacent frames to compute optical flow, and then uses optical flow priors to supervise the scene flow. Similarly, Dynpoint[3] calculates scene flow priors by estimating depth and optical flow, which are then used to constrain correspondences between adjacent frames in dynamic scenes. Unlike these NeRF-based methods, the correspondence between Gaussian and pixels is a complex many-to-many mapping, making it unable to directly use scene flow for constraints. To address this issue, we propose decoupling 2D motion flow from optical flow to constrain the deformation field.
 
## Section 0.3: Evaluation of Camera Pose Quality
We'd like to further discuss and elaborate on the challenge of evaluating camera pose quality in dynamic scenes. Unlike static scene datasets (e.g., Tanks & Temples) that use COLMAP to obtain the ground truth of camera poses, we assume that COLMAP may not provide accurate poses for dynamic scene datasets. Therefore, we refine the camera poses initialized by COLMAP. In this setting, we lack ground truth for a direct quantitative comparison. Despite this, the Table 5 of our main paper shows that the camera poses refined by our method further boost the rendering quality, demonstrates the accuracy of our refined poses. For further clarity, additional visualizations of pose trajectories are provided in Figure C and its video version are included in our comments to the AC.

## Section 0.4: References
Here are references cited during the rebuttal:

[1] Dynamo-depth: fixing unsupervised depth estimation for dynamical scenes, NeurIPS 2023

[2] Neural scene flow fields for space-time view synthesis of dynamic scenes, CVPR 2021

[3] Dynpoint: Dynamic neural point for view synthesis, NeurIPS 2023

[4] Gmflow: Learning optical flow via global matching, CVPR 2022

[5] MDFlow: Unsupervised optical flow learning by reliable mutual knowledge distillation, TCSVT 2022

[6] MonoNeRF: learning generalizable NeRFs from monocular videos without camera poses, ICCV 2023

[7] Dynibar: Neural dynamic image-based rendering, CVPR 2023

[8] Gaussianflow: Splatting gaussian dynamics for 4d content creation, arXiv 2024

[9] 3D Gaussian Splatting for Real-Time Radiance Field Rendering, SIGGRAPH 2023

[10] MoDGS: Dynamic Gaussian Splatting from Causually-captured Monocular Videos, arXiv 2024

We appreciate the reviewers’ suggestions, which have guided us in further improving the robustness and clarity of our work. We look forward to any additional feedback and are committed to advancing this research in light of the insightful comments provided.

Thank you once again for your time and effort in reviewing our submission. We have also included an attached PDF document with additional details and supporting information for your reference.

Meta-review: After the initial stage, reviewers raised some concerns regarding the novelty, robustness, and the lack of critical performance analysis (speed, memory).

All of the aforementioned issues were successfully cleared by the authors which lead reviewers to rate the paper unanimously above the acceptance bar. 

The AC strongly incites the authors to include the additional experiments conducted for the rebuttal in the final version of the paper. Most importantly, please include the additional memory & speed evaluation. Furthermore, please make sure to include results of the discussion with reviewer 4hTv whose contribution was very fruitful.
Decision: Accept (poster)