Gaussian Splatting Lucas-Kanade
Authors: Liuyue Xie
Joel Julin
Koichiro Niinuma
Laszlo Attila Jeni
Venue: ICLR 2025 Poster (2025)
Keywords: Gaussian Splatting
regularization
novel view synthesis
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Gaussian Splatting and its dynamic extensions are effective for reconstructing 3D scenes from 2D images when there is significant camera movement to facilitate motion parallax and when scene objects remain relatively static. However, in many real-world scenarios, these conditions are not met. As a consequence, data-driven semantic and geometric priors have been favored as regularizers, despite their bias toward training data and their neglect of broader movement dynamics.

Departing from this practice, we propose a novel analytical approach that adapts the classical Lucas-Kanade method to dynamic Gaussian splatting. By leveraging the intrinsic properties of the forward warp field network, we derive an analytical velocity field that, through time integration, facilitates accurate scene flow computation. This enables the precise enforcement of motion constraints on warp fields, thus constraining both 2D motion and 3D positions of the Gaussians. Our method excels in reconstructing highly dynamic scenes with minimal camera movement, as demonstrated through experiments on both synthetic and real-world scenes.

Published as a conference paper at ICLR 2025
GAUSSIAN SPLATTING LUCAS- -ANADE
Liuyue X Joel Julin', Koichiro Niinuma", László A. Jeni CCamegie Mellon University, FFujitsu Research of America (liuyuex, ajulin]@andres cmu edu, laszlo jeni kni inuma@ fujii wwt.jiss..ommm com
ABSTRACT
Gaussian Splatting and its dynamic extensions are effective for reconstructing 3D scenes from 2D images when there is significant camera movement to facilitate motion parallax and when scene objects remain relatively static. However, in many real-world scenarios, these conditions semantic and geometric priors have been favored as regularizers, despite their bias toward training data and their neglect of broader movement dynamics.
met. As consequence, data-driven
Departing from this practice, we propose a novel analytical approach that adapts the classical Kanade method to dynamic Gaussian splatting. By leveraging the intrinsic properties of the forward warp field network, we derive an analytical velocity field that, through time integration, facilitates accurate scene flow compu- tation. This enables the precise enforcement of motion constraints on warp fields, thus constraining both 2D motion and 3D positions of the Gaussians. Our method excels in reconstructing highly dynamic scenes with minimal camera movement, demonstrated through experiments both synthetic and real-world scenes. |
ace
sin aa
w/ flow
w/ seene flow
w/ depth
Figure 1: Data-driven depth and optical flow supervisions produce inaccurate geometries. Instead, we derive the analytical warp field of Gaussians to refine geometries and motions.
INTRODUCTION
The task of reconstructing dynamic 3D scenes from monocular video presents significant challenge. particularly when constrained by limited camera movement. This limitation results in a lack of parallax and epipolar constraints, which are crucial for accurately estimating scene structures Gao cct al. (022);; Schönberger & Frahm (2016). Implicit neural representation methods like NeRF have shown promise in mitigating this issue by modeling dynamic scenes using implicit radiance field Mildenhall et al. (2021a): Chen et al. (2023): y: [i et al. (2021d). Explicit approaches like Gaussian splatting Kerbl et (2023). ), while more efficient and often improve rendering quality, face additional hurdles. Gaussian splatting relies on an explicit representation of the scene using volumetric Gaussian functions. This necessitates high degree of accuracy geometric understanding ensure proper scene depiction. Consequently, despite advancements in dynamic Gaussian splatting techniques, the requirement for diverse viewing angles effectively constrain the placement of these Gaussians remains limiting factor. This inherent limitation restricts their effectiveness for capturing scenes from static viewpoints or dealing with objects exhibiting rapid and complex movements.
Project page: https://gs-Ik.gitile githuu..o

---

Published as a conference paper at ICLR 2025
Recent Gaussian splatting frameworks for modeling dynamic scenes commonly utilize canonical Gaussian space as their foundation. This space is typically initialized using Structure from Motion (SfM) Micheletti et al. (2013) point clouds, with which each Gaussian is assigned attributes that describe its position, orientation, and lighting-dependent color. This canonical space acts as a reference point from which deformations are applied to represent the dynamic scene at different time steps. The process of deforming the canonical space to depict the scene at a particular time step achieved through a warp field, often referred to as a forward warp field due to its function of warping the canonical representation forward to specific point in time. This warp field is typically modeled using learnable Multi Luyer Perceptrons (MLPs) or offset values, as seen in frameworks like DeformableGS Yang et al. (2023) and Dynamic-GS Luiten et al. (2024). These approaches have laid the groundwork for dynamic Gaussian scene modeling.
Structural cues such as optical flow and monocular depth have been explored to regulate the temporal transitions of Gaussians. However, imposing structural constraints on a generic forward warp field, as seen in DeformableGS Yang et al. (002)), is challenging. Although many studies leverage depth and flow priors, they often fail to regu laize motion through the warp field, resulting in sub-optimal trajectories and insufficient motion regularization. Additionally, employing time integration an auxiliary network to predict the velocity field can introduce computational overhead and lack theoretical constraints. In contrast this data-driven approach, demonstrate that it is possible to derive velocity fields directly from a generic warp field by adapting the Jucas-Kanade method with a small motion constraint for Gaussian Splatting. This allows for effective time integration for structural comparisons. By regulating Gaussian motions through analytical velocity fields, we address the limitations of data-driven structural regularizations, enabling continuous learning of the warp field in a time domain without the need for additional learnable parameters.
In real-world captures, our approach outperforms state-of of-te---the-ar methods, including other dynamic Gaussian techniques. We demonstrate that our warp field regularization accommodates deformations in complex scenes with minimal camera movement, achieving results that are competitive with NeRF frameworks and enabling Gaussian splatting to model highly dynamic scenes from static cameras. To summarize, our approach offers two key advantages:
Reduced Bias: Methods relying solely on data-driven learning of warp fields can be biased towards the visible time steps, neglecting the overall movement of Gaussians. Our method, applicable to forward warp field techniques, ensures accurate Gaussian trajectories throughout the sequence. Improved Tractability: The derived analytical solution for warp field velocities provides greater tractability compared to directly supervising the flow field the difference in estimated deformations between consecutive frames.
2 RELATED WORKS
Neural Radiance Fields (NeRFs), NeRFs Mildenhall et al. (2021b) have demonstrated exceptional capabilities in synthesizing novel views of static scenes. Their approach uses a fully connected deep network to represent scene's 's radiance and density. Since its introduction, NeRF has been extended
to dynamic scenarios by several works Gafni et al. (2021) )); Park et al. (202 a b): Pumarola et al. (202I): Tretschk al. ()02Ib); Yu et al. (2023). ). Some of these methods leverage rigid body motion fields Park et al. (2021a la b) while others incorporate translational deformation fields with temporal positional encoding Pumarola al, (202T); Tretschk al. (2021b). Despite the advantages of NeRF and its dynamic extensions, they often impose high computational demands for both training and
rendering.
Gaussian Splatting. The computational complexity of Neural Radiance Fields (NeRF) has driven the development of alternative 3D scene representation methods. 3D Gaussian Splatting (3DGS) Kerbl et al. (2023) has emerged as promising solution, leveraging 3D Gaussians to model scenes and offering significant advantages over NeRF in terms of rendering speed and training efficiency.
While initially focused static scenes, recent works Yang et al. (203)); y Wu et al. (2024) have extended 3DGS to the dynamic domain. These approaches introduce a forward warp field that maps canonical Gaussians to their corresponding spacetime locations, enabling the representation of dynamic scene content. DeformableGS Yang et al.] (2023) employs an MLP to learn positional, rotational, and scaling offsets for each Gaussian, creating an over-parameterized warp field that
2

---

Published as a conference paper at ICLR 2025
captures complex spacetime relationships. 4DGaussians Wu et al. (2024) further refine this approach by leveraging hexplane encoding Fang et al. (0022); Cao & Johnson (2023) to connect adjacent
Gaussians. DynamicGS Luiten et al. (2024) incrementally deforms Gaussians along the tracked
time frame. Despite these advancements, existing dynamic 3DGS methods struggle with highly dynamic scenes and near-static camera viewpoints. To overcome these limitations, we propose novel approach that grounds the warp field directly on approximated scene flow.
Flow supervision. Optical flow supervision has been widely adopted for novel view synthesis and 3D reconstruction. Dynamic NeRFs Li et al. (2021d; 2023 2021a); Tretschk et al. (2021a); Wang et al. (2023); Chen et al. (2023) have explored implicit scene flow representation with an Invertible Neural Network, semi-explici representation from time integrating learned velocity field, and explicit analytical derivation from the deformable warp field. Flow-sup NeRF Wang al. (2023) pioneered the use of analytical flow supervision in NeRF, laying the foundation for more precise and physically grounded modeling of dynamic scenes.
Several Gaussian Splatting works have begun exploring flow supervision. Motion-aware GS Guo et al. (2024) applies flow supervision on the cross-dimensionally matched Gaussians from adjacent frames, without explicitly accounting for the flow contributions of Gaussians to each queried pixel. Gao et Gao et al. (2024) on avatar rendering apply flow supervision for adjacent frames after rendering the optical flow map with a-blending Wallace (981). However, while these approaches introduce data-driven flow supervision, they do not effectively regularize the motion through the warp field, leading to suboptimal trajectories and insufficient motion regularization. This limitation makes them suitable only for enforcing short-term geometric consistency, while their performance deteriorates for out-of-distribution time steps Aoki et al. (2019); Li al. (202Ib). In the context of dynamic Gaussian Splatting, we propose to regu larize the deformable warp field through analytical time integration to ensure consistent geometric relationships across time steps.
PRELIMINARY
Dynamic Gaussian Splatting. Dynamic 3D Gaussian Splatting frameworks follow similar opti- mization pipeline as static Gaussian Splatting. Each Gaussian is defined by parameters for its mean μ., covariance Σ. and spherical harmonics SH color coefficients. To project Gaussians from 3D to 2D, we calculate the view space covariance matrix as follows:
E' = JVEV"
where J is the Jacobian of the projective transformation and V is the viewing transform. To represent a scene's radiance, the covariance Σ is decomposed into scaling matrices S and rotation matrices R RSS R² This decomposition supports differential optimization for dynamic Gaussian Splatting. The color C' of pixel u on the image plane is derived using point-based volume rendering equation by taking samples of density o, transmittance T, and color along the ray with interval 6; as:
N C(u) Σ (1- exp(-o,8,)) ea with T. exp
Eo,o
Σ 010
(2)
Adaptive density control manages the density of 3D Gaussians in the optimization, allowing dynamic variation over iterations. For more details, refer to Kerbl et al. (023).
Dynamic Gaussian Splatting frameworks utilize a canonical space and learn warp fields, denoted as F, which transform canonical Gaussians into their deformed states at time t. The offset in Gaussian parameters due to this transformation is expressed as:
8G ().
Typically, these deformed geometries are mapped through a neural network Yang et al. (2023) Luiten ct al. (2024); Wu et al. (2024), with optional color mapping applied as part of the transformation process. Following deformation, a novel-view image i rendered using differential rasterization with view matrix V and target time t. While MLP-based deformation mapping often suffers from overfitting to training views, leading to degraded novel-view reconstruction, we address this by enforeing analytical constraints the warp field. This ensures Gaussian motions adhere to
3

---

Published as a conference paper at ICLR 2025
Photometric loss
Space
Forwoed
Deformation Field
Dillinatee Rostrriztiinn
tnitillzzd
Sceze flow loss
points
v(Gi 3
0t
0( A0) 000
(0(()) Cutt
Figure 2: Analytical scene flow from warp field. With canonical Gaussians Ges we transform them forward in time to Ge, then perform time integration from warp field velocities v(G;t) to derive I:4A1. The Gaussian offsets G44.r G, are compared reference scene flow.
the expected scene flow field, mitigating the issues of overfitting and SIM-initialized point cloud limitations.
Warp field expression with twist increments. Warp fields, previously defined for point cloud registration tasks Aoki et al. (2019), can be used to model rigid transformations between sets of Gaussian means. Let μ4, N and μts TR Na33 represent the target and source Gaussian means, respectively, where N₃ and N₂ denote the number of Gaussians in each set. The warp field, which defines the rigid transformation aligning the source Gaussian set to the target set in the special
Euclidean group (3),,, can be expressed as W(E) exp
Here, E R the twist parameters, and P are the generator matrices of the group. The warp field is optimized by minimizing the difference between the transformed source Gaussian means and the target means:
argmin ((o(W(E)) p.)) - 112
where á Rxxx + RKK is an encoding function that either explicitly extracts geometric features such as edges and normals or implicitly encodes the Gaussian means into feature vectors. The notation (.) represents the application of the warp field transformation.
This optimization can be solved iteratively given the twist parameter Jacobian matrix J, denoting the changes in the warp field concerning twist parameters, that can be further decomposed into product of warp field gradient and the encoding functional gradient as:
(W- (E) .T) d0(W (E) -17) OET
J
(5)
a(W-1(E) ,
The optimization process iteratively learns the twist increment Ag that best aligns the encoded source Gaussian means with the target Gaussian means.
GAUSSIAN SPLATTING WARP FIELD REGULARIZATION BY SCENE FLOW
We investigate the problem of fitting deformable Gaussian Splatting from monocular video sequence. methods to set the canonical Gaussians
To initialize the process, we employ cencture-from-meen and camera parameters. For a given set of 3D Gaussians time t, the warp field deforms these Gaussians from their canonical positions to their dynamic locations at time t. Our goal is to find the optimal network parameters for the forward warp field WG, t). This optimization aims to ensure that both the derived scene flow T and the rasterized image I, closely match the expected flow field T and the reference image I,:
= Lootor + L Lmotion = I. Izl + T - T
(6)
The scene flow component can be further divided into optical flow and depth estimations, corre- sponding in-plane and out-of-plane motions. Consider Gaussian mean μ (() moving in the
4

---

Published as a conference paper at ICLR 2025
view
Deformable GS
Deformable GS
leust
most
motion
motion
(a) motion mups
(b) trujectories
Figure 3: (a) Visualization Gaussians' travel distance || Moll2- In both the scenes, the humans are stationary with dinosaur balloons moving around. Our result correctly identifies the dynamic regions, whereas the baseline model forms motions in the background and on supposedly stationary humans compensate for photometric correctness. (b) 3D visualization of the motion trajectories. Our result shows clean trajectories from the waving balloon.
scene. Its instantaneous scene flow If the corresponding pixel location on the image plane is u, = (f), the optical flow can be expressed as the projection of the scene flow to the image plane, 0y with du, as the instantaneous camera projective relationship:
du: du: du dt
(7)
du dt
Deriving an approximation to the scene flow needs the expression reversed. We can represent the relationship between and u, with dependencies on time as μ (tu;(tt;t) Vedula et al. (005).. By differentiating this mapping with respect to time, we obtain:
du Op du; Ou; di
djp
+ dl u.
(8)
This equation describes the motion of point in 3D space while decomposing the motion of Gaussian in the world into two components. The in in-plane component du: represents the instantaneous optical flow, while the out-of-plan term reflects the motion of μ along the ray corresponding to u, This decoupling allows us to regularize the optical flow field and the depth field separately from the estimated scene flow.
di
The scene flow regularization can thus be re-expressed combination of optical flow loss applied over the interval of + At) and depth loss of Gaussians (t + At), where the next step Gaussians analytically derived:
motion (t) = | - T- - aLL flow + +885L depth
Decoupling the motions alleviates the constraints on the supervision dataset and adds flexibility to the optimization process. A visualization of the regularization pipeline is provided in Fig. 2 Further implementation details of the losses are available Appendix A.
4.1 FORWARD WARP FIELD VELOCITY
To address the challenge of transforming canonical Gaussians G over time, our goal is to derive the velocity field from the forward warp field. Specifically, we seek to identify a warp field W that advances the canonical Gaussians in time, with underlying velocity field v(G; that quantifies the changes in Gaussian parameters within the world coordinate system at a specific time t. This derivation is conceptually similar to that of twist Jacobians Aoki al. (2019 Li et al. (b021b) and the analytical warp regularization in NeRF Wang et al. (2023). ) effectively capturing continuous deformation over time.
In the context of Gaussian Splatting, the motion of Gaussians between two adjacent frames is bijective because all Gaussians transformed from the same canonical space, creating one-to-one
5

---

Published as a conference paper at ICLR 2025
correspondence among the Gaussians. Consequently, can compute the velocity field directly from the forward warp field (9, t). To analytically derive the velocity field v(G;), t). we must represent the gradient of the warp field in relation to time and warp parameters. Assuming that the Gaussian expression. By applying a Taylor series expansion to the warp field, we decompose the velocity into essential components: the feature gradient of the warp field with respect to the corresponding Gaussian parameter and the analytical warp Jacobian Wang et al. (2023): AAoki al. (2019). This relationship expressed as:
motions are small, we can utilize a modified
t) dW
(9.1) = Jg (W:t
(10)
dt
In this equation, the term @W,, indicates how each output Gaussian parameter varies with respect to the input canonical Gaussian parameters. Meanwhile, .g describes how modifications in the warp field output influence the transformation of the Gaussians. In practical scenarios, the warp Jacobian can face numerical instabilities when det(Jg) < c, especially if the canonical Gaussian scene has not yet stabilized. Discarding these unstable motions can negatively impact subsequent densification and pruning processes. To mitigate these instabilities, we substitute the inverse Jacobian Jg" with the Penoose pseudoinverse Jg.
The derived expression is valid for any Gaussian parameter input into the deformation network, provided the input and output dimensions are consistent. However, given that the scaling terms and spherical harmonics of the Gaussians show minimal changes between adjacent frames, we choose to omit these from the calculations for improved efficiency. Summarizing, we compute the velocity fields corresponding to the displacement and rotation of Gaussians as follows:
OW. (Wo+(1;)) v( () = J (g
v(p 1) 4
(11)
Ot
where u(1z; : R3 + RR3 and v(q:t) Rd x IR R describe the positional and rotational
velocity of Gaussian in the world coordinate, respectively.
4.2 SCENE FIELD FROM TIME INTEGRATION
Time integration on the velocity field. Given the velocity field from equation 11 and 3D Gaussians (()) observed at time step we can apply time integration using Runge-Kutta Batschelet 19522) numerical solver on the velocity field to obtain the offset parameters for Gaussians at time (f + At) Specifically, we employ Monte Carlo integration with a sampling size of 10 points over each domain, which provides an efficient and well-balanced alternative to deterministic approaches. The integration formulated as:
G + At) = C(t) + S. I+
(G(())
(12)
We then train the warp field network, identical the one outlined Yang et al. (2023), to minimize the difference between the analytically derived scene flow, and the ground truth scene flow decomposed into optical flow and depth describing the in-plane and out-of-plane Gaussian motions.
Optical flow and depth rendering from Gaussians. For each Gaussian at time i, we compute each of the corresponding positions and rotations at (t + At). For rendering the optical flow and depth map, consider the full freedom of each Gaussian motion.
For rendering the optical flow map O want to calculate the motion's influence each pixel us from all Gaussians in the world coordinate. In the original 3D Gaussian Splatting, a pixel's color is the weighted sum of the projected 2D Gaussian's 's radiance contribution. Analogous to this formulation, the optical flow can be rendered by taking the weighted sum of the 2D Gaussians' contribution to pixel shifts, as derived in previous literature flow regularization Gao et al. (2024):
Guo et al. (2024):
i-1 II - a.) Ø=1
O EiNOO
- (E E (ui, (s,,,) + P1,t2 un
(13)
1=1
where On denotes the optical flow of Gaussian G. over the time interval, and w; the weighing factor of each Gaussian from a-blending.
Te denotes E.T.a

---

Published as a conference paper at ICLR 2025
Similarly, the frame's z-depth estimates Di+A/ can be rendered from the discrete volume rendering approximation accounting for the per- Ger-Gausiian contributions:
i-1
DiA = Lind,, ai
II (1-a)))
(14)
j=1
where di,t, denotes the z-depth value analytically derived Gaussian G3 from the viewing space at (l At). We normalize the optical flow and depth estimates for numerical stability. The optical flow and depth maps are rasterized simultaneously in the same forward pass with color. A performance comparison with various flow and depth priors can be found in A.3
EXPERIMENTS
In this section, first provide the implementation details of the proposed warp field regulation and then validate our proposed method on five dynamic scene datasets captured with different levels of camera movements. Our method outperforms baseline approaches in both static and dynamic camera settings, achieving state-of-the-art results in quantitative and qualitative evaluations.
novel view
depth
ground truth
depth
noval view
Deformable flow
flow
Figure 4:
Qualitative comparisons on the Dynamic Scenes dataset. Compared the baseline method, our approach can achieve superior rendering quality on real datasets with lower EMFs.
5.1 IMPLEMENTATION DETAILS
We implement the warp field using PyTorch Paszke al. 2019). leveraging its Autodi f library for
gradient and Jacobian computations. The framework is optimized with Adam Kingma & Ba (2015) as with 3DGS Kerbl et al. (2023) on a NVIDIA A100 Tensor Core Nvidia Corporation (2024) We use the Torchdi f ffeg Chen (2018) library for numerical integration. Modified from Deformable GS, the inputs to the warp field include 3D Gaussian positions (І4), time (f), and 3D Gaussian rotations (q) in quaternions. Velocity derivations enabled by removing the stop gradient operation from the network inputs. Positional encoding Mildenhall et al. (202a)) is applied to extend inputs' frequency band Zheng al. (2022). We refer readers to Appendix A.2 for more implementation details.
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative evaluation of novel view synthesis on the Dynamic Scenes dataset. See Sec. 5.2 for descriptions of the basclines.
mathod
Playground SSIM 1
Balloon] SSIM 1
Balloon2 SSIM
Umbrella SSIM
PSNR 1
LPMPS
PSNR 1
LPIPS
PSNR
LPIPS
PSNR
LPIPS
NSFF
24.69
D.889
D,085
24.36
0.891
0.061
30.59
0.95
0.030
24.40
0.847
0.088
NR-NeRF
14.16
0.337
0.363
15.98
0.444
0.277
20.49
0.731
0.358
20.20
0.526
0.315
Nerties
22.18
0.802
0.133
23.36
0.852
0.102
24.91
0.864
0.089
24.29
0.803
0.169
flow)
0.109
25.82
0.899
0.813
0.123
Flow-sup. NeRF
16.70
0.597
0.168
19.53
0.654
0.375
20.13
0.719
0.113
18.00
0.597
0.148
Deformable GS
24.82
0.616
0.343
22.40
0.833
0.137
21.19
0.818
0.153
22.35
0.711
0.186
4DGaussians
21.39
24.73
0.801
21.29
0.560
0.352
Ours
26.34
0.756
0.184
26.35
0.848
25.89
0.911
0.151
23.02
0.746
0.176
Table 2: Quantitative results on the DyCheck dataset.
Table 3: Quantitative results for the Deformable GS method.
Method
Apple
Spin
Block
Toddy
Paper Windmill
NSEF
17.54
18.38
16.61
13:65
17.34
Method
PSNR
SSIM
LPIPS
Marbles
17.57
15.49
16.83
13.57
18.67
Deformable 68 w/
20.97
4D Goussians
15:41
14.41
11:28
12.36
15,60
Defonmable depth
21.17
16.71
17.41
Deformable GS w/ both
23.94
0.202
0.28
Ours pose refined
22.61
24.09
23:27
17.78
19.9
Deformable (IS
24.82
0.645
0.34
5.2 EVALUATION
Here analyze the performance of our method quantitatively and qualitatively. We aim to study the analytical scene flow regularization on the warp field helps disambiguate the dynamic scene geometry and promote the reconstruction of low EMF scenes.
Benchmarked datasets. We evaluate our method using real-world monocular datasets with varying camera motion. These include DyCheck dataset Gao et al. (2022), captured with handheld monocular cameras; Dynamic Scene Yoon et al. al.] (2020), captured by a stationary multi-view camera rig with significant scene motion; Plenoptic Video Li et al. (2021c), captured using static rig with 21 GoPro cameras GoPro (003);: Hypernerf Park et al. (2021b), which captures objects with moving topologies from moving camera, suitable for quasi- static reconstruction; and sequences from DAVIS 2017 Perazzi et al. (2016) dataset containing near-static monocular videos.
Quantitative baseline comparisons. We evaluate our method's novel-view synthesis capabilities on Plenoptic Videos, Hypernerf, Dynamic Scenes, and DyCheck, ordered by decreasing Effective Multi-view Factor (EMF) Gao et al. (2022). These datasets present increasing challenges, with DyCheck posing the greatest difficulty due to its low EMF and unreliable object motion. We first assess our method on Hypernerf and Plenoptics with higher EMFs against recent methods 3DGS
Kerbl et al. (2023), Deformable GS Yang et al. (2023). 4DGaussians Wu et al. (2024), and MA-GS
Guo et al. (2024). We report the standard metrics LPIPS Zhang et al. (2018), SSIM Wang et al. (2004), and PSNR Welstead (2009) in Table [ and 4 We see that our method consistently outperforms these
Gaussian baselines on scenes with higher EMFs.
To further assess method's performance monocular sequences with lower EMFs, evaluate its performance against recent methods Marbles [Stearns et al.) (2024), NSFF [i et al. (2021d), NR-NeRF Tretschk Tretschk
al.] (202Ib). Nerfies Park al. (2021a). and Flow-supervised NeRF ?, see [and]] 2 These comparisons highlight the effectiveness of our analytical scene flow approach compared other flow-guided rendering frameworks. Notably, our method outperforms NSFF, which uses similar supervision style but a different scene flow derivation. The comparison with Flow-supervised NeRF, which also uses analytical scene flow derived from an implicit radiance field, serves an ablation study, demonstrating the effectiveness of our derived scene flow an explicit scene representation. A.3
Table and
Comparisons with direct depth and flow Supervision. We note that it is essential to understand the performance improvements over regularizing the deformable network directly with depth and flow priors. We include ablation experiments with Deformable GS supervised with depth and flow losses, see Table The results suggest that directly applying depth and flow regularizations encourages the scene overfit to training viewpoints, leading to degraded performance. With depth regularizations, the scene geometries become more prominent, with a sacrifice in motion smoothness. Regularizing offsets between frames can enforce accurate piecewise motion but still cannot generalize well to
8

---

Published as a conference paper at ICLR 2025
Table Quantitative evaluation of novel view synthesis on the Plenoptic Videos dataset. See See 5.2 an analysis of the performance.
spinach
PSNR
SSIM
LPMPS 4
PSNR
SSIM
LPUPS 4
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
Deformable C8
32.97
013447
01.887
30.72
0.941
0.090
33.68
0.955
0.079
32.46
0.948
0.085
4DGaussians
31.98
0.938
0.055
31.56
0.939
0.062
31.20
0.949
0.945
31.58
0.942
0JIS5
MA-GS
0.060
0,059
0.084
0.057
0.068
Table 5: Quantitative evaluation on the Hyper- nerf dataset. See Sec. for for detailed explana- tions.
Table 6: Ablation study on scene flow de-
composition on "Playground" scene from Dy- namic Scenes dataset. See Sec. 5.2 for detailed descriptions of each ablated design.
method
PSNR
SSIM
3DG3
20.84
0.70
0.45
SSIM
Deformalbee
0.70
4TXiusssinns
26.98
0.711
0.374
Ours
27.38
0.81
0.36
Sup.
Ours
26.34
0.756
0.184
SIM point cloud
(4) TTnstable recosstuctionn when recorstruct
camera poses degruded tenderingg
dynamic region.
Figure 5: Our method struggles with insufficient point cloud initialization due to reliance on non-rigid warping of scene geometry. In the "skating" scene, this results in unstable geometry at certain angles. Inaccurate camera calibrations also degrade our method, as shown in the "toby-sit" sequence, where miscalibration distorts the scene geometry.
the discontinued unseen viewpoints. In Table we ablate the analytical regularization and report the outcomes of the scene. As shown, each of the scene flow components helps in achieving high-quality synthesis. These ablation experiments reaffirm the validity of the proposed warp field regularization.
Trajectories from scene flow fields. We visualize the scene flow trajectories from Dynamic Scenes Dataset sequences to assess the quality and smoothness of the derived flow field. The Gaussians are subsampled from the dynamic regions of the canonical Gaussian space, and then time integrated to produce their respective displacements across time, visualized colored trajectories in Fig. We note that the trajectories are smooth and follow the expected motions of the dynamic objects. The visualized trajectories of the sequences demonstrate the framework's potential to be extended for tracking in dynamic 3D Gaussian scenes. More trajectories on the DAVIS dataset will be shown in Appendix Fig. A7
Geometric consistency. Fig. 4 compares our method with Deformable GS, which learns Gaussians' time dependency without scene flow regularization. While Deformable GS renders visually accurate novel views, the underlying scene structures are inaccurate, leading to blurry dynamic objects and artifacts in depth maps. Our method, leveraging analytical scene flow regularization, achieves more accurate geometries, reflected in both visual and quantitative results (Table L This also enables accurate Gaussian motion tracking (Fig. B). To explore alternative design choices, we compare with the direct flow and depth supervision on Deformable GS (Fig. [T.. These comparisons indicate that our approach, which enforces regularization on the expected scene flow over a continuous time frame, results in more plausible scene structures compared to discrete time step supervision. We additionally conduct experiments on the DyCheck benchmark dataset, see Table 2 The additional visual comparison is shown in Appendix Fig. A2 with comparisons to the deformable baseline. Our results show that the analytical regularization significantly improves the render quality of the DyCheck dataset, especially when accurate camera poses are provided Wang al. (2024).

---

Published as a conference paper at ICLR 2025
Static and dynamic motion separation. Our approach formulates deformations within canonical Gaussian space. To assess the fidelity of our method in separating static and dynamic regions, we visualize the travel distance of each Gaussian from its canonical projection to queried viewing camera in Fig- [ The plots are color-coded by the absolute travel distance, with yellow indicating larger distances and purple indicating smaller distances.
In the Playground scene featuring human-object interactions, the visualized travel distance is greatest the far-side boundaries of the moving object as it recedes from the human. The scene's background is correctly rendered as static with minimal motion. This visualization demonstrates the effectiveness of our method accurately identifying dynamic regions. Moreover, these results suggest promising future directions for optimizing dynamic scene rendering by filtering out static regions from the optimization process, potentially reducing computational costs.
LIMITATIONS
Sensitivity to SIM Initialization: Our approach is sensitive to the quality of the Structure-fromt Motion (SfM) Gaussian space initialization. Accurate initialization crucial for 3D Gaussian Splatting, particularly without other scene priors. As shown in Fig. 5 reconstruction quality
deteriorates when dynamic regions have insufficient initialized points.
Camera Parameter Sensitivity: Our method is sensitive to inaccuracies in bundle-adjusted cam- era intrinsic and extrinsic parameters. Errors in these parameters lead to inaccurate rasterization projections, causing noisy geometry or failed reconstruction, shown in Fig. bb.
Computational Complexity: The pseudo- pseudo-inverse inverse operation for velocity field computation requires (N x d) square matrix decompositions, where N is the number of Gaussians, and d is the number of Gaussian parameters. Potential mitigations include optimized implementations or pre- pre-fittring to remove static Gaussians.
7 CONCLUSION
This paper presents a method for incorporating scene flow regularization into deformable Gaussian Splatting. We derive analytical scene flow representation, drawing upon the theoretical foundations of rigid transformation warp fields in point cloud registration. Our approach significantly enhances the structural fidelity of the underlying dynamic Gaussian geometry, enabling reconstructions of scenes with rapid motions.
Comparison with other Deformable Gaussian Splatting variants demonstrates that regularizing on the warp-field derived scene flow produces more accurate dynamic object geometries and improved motion separation due to its continuous-time regularization. Additionally, comparison with flow- supervised NeRFs reveals the advantages of using Gaussians' explicit scene representations.
While our method exhibits limitations as discussed in the previous section, the accurate geometries learned from dynamic scenes open up promising possibilities for future applications, including 3D tracking from casual hand-held device captures and in-the-wild dynamic scene rendering.
ACKNOWLEDGHMENT
This work was supported by Fujitsu.
REFERENCES
Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey. Pointnetlk: Robust & efficient point cloud registration using pointnet. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2019-June, 2019. doi: 10.1109,
10

---

Published as a conference paper at ICLR 2025
Eduard Batschelet. Über die numerische Auflösung Randwertproblemen bei elliptischen partiellen Differentialgleichugeen. ZAMP Zeitschrift für angewandte Mathematik und Physik, 3(3), 1952. ISSN 00442275. doi: 10. 100000//0020008244.
Reiner Birkl, Diana Wofk, and Matthias Müller. Midas v3.1 - a model zoo for robust monocular relative depth estimation. arXiv preprint (xXtv:2307.14460) 2023.
Ang Cao and Justin Johnson. HexPlane: Fast Representation for Dynamic Scenes. 2023. doi: 10.1 1099evpp52729..022222 200.7999.023000211
Ricky T. Q. Chen. torchdi ffeg
torchdiffeq, 2018.
URL htttp:://gthhu..omm/rees
Shuo Chen, Binbin Yan, Xinzhu Sang, Duo Chen, Peng Wang, Xiao Guo, Chongli Zhong, and Huam- ing Wan. Bidirectional Optical Flow NeRF: High Accuracy and High Quality under Fewer Views. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023, volume 37, 2023. doi: 10. 10.1609/aaii. w3t.v337.1221..
Qiaole Dong and Yanwei Fu. Memflow: Optical flow estimation and prediction with memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14304-14314, IEEE Computer Society, 2021.
Inc. Epic Games. Reality Capture, 2024.
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. SIGGRAPH Asia 2022 Conference Papers, 2022.
Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias NieBner. Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction. In Proc. IEEE/CVF CVPR, pp. 8649-8658, 2021.
Hang Gao, Ruilong Li. Shubham Tulsiani. Bryan Russell, and Angjoo Kanazawa. Monoc- ular dynamic view synthesis: A reality check. In S. Koyejo, S. Mohamed, Cho, and Oh (eds.), Advances in Neural Infor- mation Processing Systems, volume 35, pp. 33768-33788 Curran Associates, Inc., 2022. URL https:://proceedpinnggooo/ www.nnee.ruc..ooo wttppppp.cpppppp.p.pepepppppppee httpppppprr.alleepppppp/22//222 httpa//eeeeeeaerroee//e//eeeaeaa File/ httlec/aaazzffffeeeee///eeeeesa2
A. Agarwal, D. Belgrave,
Confereeen pddf
Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Chen, Danhang Tang, and Ulrich Neumann. GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation. 3 2024.
Inc. GoPro. Go Pro Hero 7, 2023.
Guangcong, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Zhiyang Guo, Wengang Zhou, Li, Min Wang, and Houqiang Li. Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction. 3 2024.
E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), Jul 2017. URL http: //w.///m./...///////// wnf.onntiiii..... uni freiburg.gg.. /e/
/Publicatioons/0117 1
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing tiffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
11

---

Published as a conference paper at ICLR 2025
Bernhard Kerbl, Georgios Kopanas, Thomas Leimküihler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4):1-11, 2023.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, 2015.
Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Neural Scene Flow Prior. Advances in Neural Information Processing Systems, volume 10, 2021a.
Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. PointNetlk revisited. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2021b. doi: 10.1 NLL11099CVPR44437.22000 201.R4444477.221.01222
Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Pointnetlk revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12763-12772. June 2021c.
Xueqian Li, Jianqiao Zheng, Francesco Ferroni, Jhony Kaesemodel Pontes, and Simon Lucey. Fast Neural Scene Flow. Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 9878-9890, 2023.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural Scene Flow Fields for Space- Time View Synthesis of Dynamic Scenes. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2021d. doi: 10. LLL109CCVRR4633.222..006666
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians:
Tracking by persistent dynamic view synthesis. In 3DV, 2024.
Natan Micheletti, Jim Chandler, and Stuart N Lane. Structure from Motion (SfM) Photogrammetry. British Society for Geomorphology Geomorphological Techniques, 2(2):1-12, 2015. ISSN 2047- 0371. doi: 10.5 194isppssaacciesss 9444pppartiisssX--5-53331... XL-55-4-37-20155
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65 (1):99- 106, dec 2021a. ISSN 0001-0782. doi: 1051345/3003250. htttsss/:/dooi.oo org/ 10.1145/3503250
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, et al. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Commun. ACM, 65(1): 99-106, 2021b.
Nvidia Corporation. NVIDIA A100 Tensor Core, 2024.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, et al. Nerfies: Deformable Neural Radiance Fields. In Proc. IEEE/CVF ICCV, pp. 5865-5874, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. HyperNeRF: Higher-Dimensions Representation for Topologically Varying Neural Radiance Fields. ACM Transactions Graphics, 40(6), lb. ISSN 15577368, doi: 10. 00.114//477851334484453
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito. Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, bigh-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019.
F. Perazzi, Pont- Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine Haatie--oounng... A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-December, 2016. doi: (0.1109/CVP..20166 25.
12

---

Published as a conference paper at ICLR 2025
Albert Pumarola, Enrie Corona, Gerard Pons-Moll and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proc. IEEE/CVF CVPR. pp. 10318-10327, 2021.
Johannes Lutz Schönberger and Jan-Michael Frahm. Bructure-from--oonn Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Colton Stearns, Adam W. Harley, Mikacla Uy, Florian Dubost, Federico Tombari, Gordon Wetzstein, and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos. In ArXiv, 2024.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference Computer Vision, 2020, URL httpps///pppl..mmmm...cooooooaa.
CorpusID:21466788
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video. In Proceedings of the IEEE International Conference on Computer Vision, 2021a. doi: 10. 11000CCCV489222221.000777
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, al. Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video. Proc. IEEE/CVF ICCV, pp. 12959-12970, 2021b.
Vedula, P. Rander, R. Collins, and T. Kanade. Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3) 2005. doi: 10.1109/PPAM..2055.655
Bruce A. Wallace. Merging and transformation of raster images for cartoon animation. In Proceedings of the 8th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1981, 1981. doi: 10.1145/8002224868811
Chaoyang Wang, Lachlan Ewen MacDonald, László A. Jeni, and Simon Lucey, Flow supervision for deformable nerf. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21128 11128-21137, June 2023.
Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. 2024.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Trans. Image Process., 13(4):600-612, 2004.
Stephen Welstead. Fractal and Wavelet Image Compression Techniques. 2009, doi: 10.1117/3. 353798.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.0941 094144 2024.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction. 9 2023.
Jac Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020. doi: 10. (LL.1///CVYR2220000000000.50.
Heng Yu, Joel Julin, Zoltán A. Milacski, Koichiro Niinuma, and Lászłó A. Jeni. Dylin: Making light field networks dynamic. In 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12397-12406, 2023. doi: 10. 10.11099CVPPS52222222..11111
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
13

---

Published as a conference paper at ICLR 2025
Jianqiao Zheng, Sameera Ramasinghe, Xueqian Li, and Simon Lucey. Trading positional complexity vs. deepness in coordinate networks. Proceedings of the European Conference on Computer Vision (ECCV), 2022.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX / SUPPLEMENTAL MATERIAL
A.1 SCENE FLOW REGULARIZATION FORMULATION
Recall that in addition the photometric regularization detailed in the original 3D Gaussian Splatting Kerbl et al. (023), motion is used as the structural regularization for the derived scene flow consisting of weighted depth and optical flow terms Lmotion (t) aLL flow + depth
The optical flow term minimizes the absolute error between the rendered optical flow map as computed from accounting for all Gaussians' 2D projected motion contributions to each rendered pixel, and the reference flow from RAFT Teed & Deng (2020). In particular, the optical flow term is expressed as:
Llow
10 Oc-2+1l.
(15)
As noted in CrrrseeeRFGGuangcooon et al. 2023). monocular depth maps are scale-ambiguous thus directly using monocular depth priors with p-norm regularizations hinders the spatial coherency. We apply local depth ranking loss on the rendered depth maps. We transfer the depth ranking knowledge from monocular depth dmono to rendered d, with:
Leepth
Σ Cimono Kddmooo
max(d -, + m,
(16)
where m is a tunable small margin defining the permissible depth ranking errors, and ka 1/2 are the indices of the randomly queried pixels. If the depth ranking of the monocular depth pixels and
the rendered pixels are not consistent, then it entails inaccuracies in the warped Gaussians at time (t At). penalizing the learned warp field.
The coefficients a and 3 reweight the magnitude of loss terms to compensate for regularizing with monocular depth cues. practice, we notice that setting the two coefficients with similar magnitude yields satisfactory results for scenes initialized with dense point clouds. Whereas with insufficient SfM initialization, the motion coefficient needs to be weighed down in earlier iterations to allow the canonical scene to stabilize.
A.2 ADDITIONAL IMPLEMENTATION DETAILS
Scene initialization. We define the initialization stage as the first N iterations without deploying scene flow regularizations. To warm up the optimization, during the initialization stage, we only compute the photometric losses with the warp field network unfrozen. Note that we discarded the static scene initialization stage typically adopted by other dynamic Gaussian splatting frameworks as it provided incremental benefits.
The length of the warm-up period is defined based on the quality of the SfM scene. For scenes with rich point clouds to start with, we employ 3000 iterations for initialization to refine the canonical structures. Some highly dynamic scenes contain only few points or no points in the dynamic regions. Since our method relies on analytical warping from the canonical scene and needs Gaussians at the dynamic regions start with, we extend the warm-up period 5000-10, 000 to recover as many Gaussians as possible at the dynamic regions.
Motion masking. The initialized Gaussians generally carry inaccurate motions to compensate for photometric accuracies, regardless of whether the Gaussians are located at the presumably static or dynamic regions. Thus we first randomly sample pixels from the rendering to correct motions of all Gaussians equally across the scene. Since static regions are usually larger than dynamic regions and are easier to converge, this allows us to disambiguate the static and dynamic regions.
After ensuring that most Gaussians in the static regions have been sufficiently constrained, apply large motion mask to reconstruct small and fast-moving objects. The motion mask defined similarly as prior work [i et al. (2021d) as a binary segmentation mask pixels with normalized motions larger than 0.1. This additional motion masking step allows us to focus on rapid motions when most of the scene motions have converged.
Training details. The hyperparameters for learning the Gaussian parameters are kept consistent as the original 3D Gaussian Splatting Kerbi al. (003).. For comparable analysis on the analytical
15

---

Published as a conference paper at ICLR 2025
warp field regularization, follow setups in the baselines Yang et al. (2023) Wu al. (2024). The learning rate of the warp field network is empirically set to decay from 8 to 1.6e-6, 6. and Adam's 's range is set to (0.9, 0.999). As with data preparation, (2016) and Reality Capture Epic Games (2024) to estimate camera intrinsics and extrinsics, which kept fixed during optimization. We use Marigold Ke et al. (024) and RAFT Teed Deng (020) for depth and flow maps.
used COLMAP Schönberger & Frahm
A.3 FURTHER DISCUSSIONS ON MOTION LEARNING
Geometry enhances novel view reconstruction quality. Fig. A1 presents novel view synthesis results and learned geometries from our method alongside two baseline models. This comparison highlights the impact of accurate underlying geometries on synthesized novel views. All three models converge on the training views, so we focus on the results for unseen viewpoints.
4DGaussians
Leamnd Velocity
Ours
Figure Al: Visualization of Gaussian trajectories and comparison of novel view synthesis results with learned geometries.
rgb
depth
rgb
depth
rgb
depth
Deformable 65
Ours
+HI
Figure A2: Qualitative comparison of results on scenes from the DyCheck dataset.
4DGaussians Wu et al. (024) learns the warp field without incorporating motion cues. This approach leads noisy underlying geometry with artifacts around specular areas, likely introduced to compensate for photometric inaccuracies. Consequently, its synthesized dinosaur exhibits blurriness around the edges. Another baseline predicts velocities directly instead of relying on analytical derivation as in Du et al. Du et al. (2021). We adopt the velocity field model from Du et al. and perform time integration to derive scene flow. However, this data-driven velocity field tends to produce an "averaged" geometry, resulting in blurriness around the dinosaur's 's head area.
In contrast, our method generates the most crisp depth map and the highest quality synthesized result. This improvement stems from our ability to accurately learn dynamic geometries. The left-hand side of the figure demonstrates that even with monocular sequences, our method effectively separates the foreground, background, and carried object. Our method serves general tool for enhancing dynamic Gaussian splatting frameworks without requiring additional network modules. This is further exemplified by the performance DyCheck dataset, as shown Fig. A2
16

---

Published as a conference paper at ICLR 2025
presents a comparison of motion thresholds at different travel distances and our learned trajectories. The first row visualizes the 3D trajectories of Gaussians, color-coded by z-depth magnitude and in learned color. This visualization clearly demonstrates the accurate motions of human waving balloon. The second row displays the motion plot generated by our method, while the third row showcases the results from the DeformableGS baseline model Yang et al. (2023). The baseline model exhibits a significant error, moving the stationary human instead of the balloon. Notably, the side of the balloon furthest from the human experiences the least movement, while the human's torso exhibits the most displacement. This outcome directly contradicts expectations. contrast, our method accurately captures the motions, correctly ranking their magnitudes. This highlights the effectiveness of our approach in distinguishing and representing the dynamic elements within the scene.
Motion thresholding visualizations. Fig. A3
moticn threshold
Figure A3: 3D trajectories and thresholded motion plots. The first row motion plots are results from method and the second results from baseline model DeformableGS.
Sensitivities to SfM initialization continued. As discussed in the limitations section, our method relies analytical warping from canonical space, making it sensitive to Structure Structure-from- from-Motion (SfM) initialization, particularly when dynamic regions lack sufficient point correspondences. Fig. A4 illustrates this sensitivity using the "train" scene from the DAVIS dataset Perazzi al. (2016). The left side of the figure shows the SfM-initialized mesh, where the toy train traveling on the circular track is not captured. The top right figure displays the learned trajectories from the baseline model, exhibiting irregular Gaussian movements within the dynamic area.
The bottom right figure presents the learned trajectories from our method. While our method captures the expected trajectory of the moving train, it struggles to reconstruct dense Gaussians for the train due to an insufficient number of initialized Gaussians in this region. Additionally, the Gaussians are not accurately placed at the correct depth, as the monocular depth ranking loss only enforces the correct depth ranking order. Employing alternative depth regularization techniques could improve the learned depth in absolute scale.
Sensitivity to the choice depth and flow supervision Understanding the sensitivity of the choice of depth and flow supervision is crucial for accurately demonstrating the merits of our work. To explore this further, we performed experiments on the "Playground" scene in which different methods for depth and flow supervision were used. Visual comparisons of the ablated priors shown in Fig, A5 For depth supervision, we tested Depth Anything V2 Yang et al. (2024) and Midas Birkl al. 2003) achieving PSNR scores of 26.96 and 26.78, respectively. For flow supervision,
17

---

Published as a conference paper at ICLR 2025
imput images
(b) initialized
trajectorics
Figure A4: SIM initialized mesh and the reconstructed toy train trajectories. The right top subfigure shows trajectories from the baseline model, and the right bottom subfigure shows trajectories from our model.
evaluated FlowNet Olg al. (2017) and MemFlow Dong (0224 achieving PSNR scores of 26.38 and 27.04. As reference, our original setup achieved PSNR of 26.34. The results indicate that Midas yields the poorest performance, producing blurry depth maps. In contrast, Depth Anything V2 provides sharper details but struggles with background accuracy in outdoor scenes. Our chosen method, Marigold Ke et al. (2024), gives more visually accurate depth maps, capturing distant background details. However, performs slightly worse quantitatively compared the others. Nevertheless, the variation in depth priors appears to have minimal impact on our method's final results, most recent methods exhibit similar performance levels. Regarding flow estimation, Flow Net produced less accurate flow maps, failing to capture scene movement effectively. Conversely, MemFlow, the current state-of-the-art optical flow estimator, achieved the best results with sharp details and accurate flow. Our selected method, RAFT Teed & Deng (2020), also provides accurate flow maps but
sharp as MemFlow, which likely responsible for the improved PSNR.
MiDaS
Depth Anything V2
Marigold
depth
flow
FlowNet
MemFlow
RAFT
Figure A5: Qualitative comparison of various depth and flow methods applied to the Playground scene. Methods discussed in the paper are shown in the right column.
Network Jacobians uncertainty measure. Deformation networks exhibit distinct patterns in regions that are easier to reconstruct compared to those that are more challenging. To explore this, we analyze the Jacobian of these deformations with respect to time, enabling us to quantify uncertainty directly from these patterns. Here in Fig. A6 we present preliminary results on a controlled synthetic scene, where the red and green balls have randomized positions in the training dataset, while the blue ball accurately corresponds to the correct time step. Brighter colors indicate higher amounts of uncertainty.
More results on real scenes. More qualitative results on dynamic scenes can found in Fig.
18

---

Published as a conference paper at ICLR 2025
Figure A6: The uncertainties of the Gaussians are visualized in the plot, with the brighter regions higher in magnitude.
19

---

Published as a conference paper at ICLR 2025
greun Iraii
DONO Viso
depti
Truck
ground Iruth
view
depth
(b) Skating
Figure A7: Qualitative comparisons on Dynamic Scenes Dataset Yoon et al. 2020
20


Reviews:
[{"review_id": "tDCcuNmP8z", "reviewer": "Reviewer_GbSS", "summary": "This is a paper about the fundamental question of how to derive the velocity of Gaussian means in Gaussian splatting from the finite forward warp transformation. \nIt assumes that these transformations are small and that one can live with a first-order Taylor expansion. \n\nThis involves the derivative of the Gaussian parameters after the motion with respect to the Gaussians in the canonical frame and the warp Jacobian.\nThen, assuming that scaling and directional reflectivity do not change, one can derive the Jacobian with respect to the mean and the Jacobian with respect to the orientation.", "questions": "Q1: Given the analytical form of the derivation, it would be interesting to study classic optical flow problems for Gaussian flow: What is the aperture problem in the Gaussian case. What happens when the Gaussian covariance is rank deficient? What scene structures would cause an ambiguous flow? \n\nQ2: What would happen when the spherical harmonic coefficients would change? \n\nQ3: What happens with a local deformation like scale change or shear?\n\nQ4: What exactly is the difficulty of including second-order terms?\n\nQ5: How is the method's sensitivity to the choice of canonical frame? DGS methods have been proposed that do not require a choice of canonical frame.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "+ Deriving analytical solutions is elegant and makes an approach interpretable. \n\n+ The analytical derivation enables a sound application of the flow term in the Dynamic Gaussian case. \n\n+ The paper is a pleasure to read.\n\n+ A direct flow regularization makes a network overfit on viewpoints.\n\n+ The idea of flow supervision in Gaussian splatting increases performance.", "weaknesses": "While it is nice for a paper to be self-contained, everything \nin this paper until eq. (10) is identical to the \"Flow supervision for Deformable NeRF\" paper. Eq. (12) is the same as Eq. (7) in the latter. Optical flow rendering is same as in GaussianFlow. The reader is wondering, what exactly is the contribution of this paper when the main ingredients are in the Flow Supervision and the GaussianFlow paper. It really hurts to say that because any reader would enjoy such elegant analytical derivations. Please explain in your response.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TXtlwDuki8", "reviewer": "Reviewer_GTCV", "summary": "The paper addressed the problem of Gaussian Splatting estimation in dynamics scene and with small movements of the camera or the objects. The authors propose an analytical solution to the deformable warp field used to regularize the estimation instead of a learnable  field or offset-values as applied in the state of the art approaches. \nThe key contribution is to provide a detailed derivation and stable estimation, basically a a Gaussian Splatting warp field regularization by scene flow. The approach is then evaluated on various datasets and show improvements compared to the existing solutions.", "questions": "Three limitations are mentioned in the paper. \nI would like to add:\n- \"Sensivity to noise\". What is the behavior under noise? is that possible to further reduce the effect of noise?\n- Computational complexity and related aspects are not mentioned at all. What is the advantage of the approach? (compared to pure learning approaches) \n- The paper mentions a motion masking step. How is the threshold defined? Do theses parameters generalize?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The strength of the paper is to propose a direct analytical solution to the problem of the estimation of the warp field. The solution is mathematically transparent and provides superior results compared to a purely data-driven approach. \nIt is a fully new solution which is very relevant for dynamic gaussian estimation and related applications. It is strongly inspired by Lucas-Kanade point tracking and has the same level of generality.", "weaknesses": "- A reference implementation should be provided since the approach is very relevant to the community and may include many small details, which have their importance and not detailed in the paper.\n- The method relies on external algorithms to provide depth and optical flow information, which are used to supervise the scene flow regularization. The appendix discusses experiments with different depth and flow estimation methods. However, this introduces an additional source of potential error and highlights the importance of using high-quality depth and flow estimation techniques.\n- EMF is not explained when it was mentioned but later in the text.\n- One reference is missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HmCYiyV23B", "reviewer": "Reviewer_uwP8", "summary": "This paper proposes a novel dynamic Gaussian splatting approach based on an analytical velocity field formulation. Similar to Lucas-Kanade, the method linearizes the warp function at specific time intervals and approximates it with a first-order expansion. This approach effectively regularizes scene flow, resulting in improved dynamic reconstruction performance on benchmark datasets.", "questions": "- What exactly is the network architecture of the warp field?\n- Compared to fully MLP-based Deformable 3DGS, grid-based warp fields such as 4DGS should struggle less with incorrect motion correlations triggered by a single global MLP representation. How much improvement does the proposed method offer for 4DGS-type warp fields?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The analytical formulation of the vector field is interesting.\n- More tractable than fully MLP-based Deformable 3DGS or other 2D prior-based supervision approaches.\n- Competitive benchmark performance.\n- The paper is clearly written and easy to follow.", "weaknesses": "- Limited evaluation dataset. Since the method builds on Deformable GS, it would be beneficial to evaluate on the same benchmark datasets, such as DNeRF, HyperNeRF, and NeRF-DS.\n- While the method claims tractability as an advantage due to the analytical formulation, reliance on a black-box MLP weakens this claim.\n- The video supplementary does not show particularly impressive results; there is noticeable unnatural blurring in the GSLK_network video", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tM8huiAPll", "reviewer": "Reviewer_D8fp", "summary": "This paper tackles the challenges posed by dynamics and the lack of motion parallax that Gaussian Splatting methods face in real-world settings. The author proposes an analytical solution that adapts Lucas-Kanade method to dynamic Gaussian splitting. The proposed method leverages the forward warp field and derives an analytical velocity field for accurate scene flow computation. The experiment shows that “GAUSSIAN SPLATTING LUCAS-KANADE” outperforms state-of-the-art methods on both synthetic and real-world scenes, and demonstrates accurate reconstructing capability in complex scenes with minimal camera movement.", "questions": "1. The author should provide more details about the experimental setup, e.g., time step ∆t  in Eq.12. Please clarify how is it determined? does the method still satisfy the small motions assumption stated in Line 273 when ∆t is large?\n\n2. The authors highlight in Section 6 that their method is sensitive to SfM initialization. Nevertheless, in my experience, 3DGS can still produce high-quality renderings even when initialized with random point clouds. I find it puzzling that the authors emphasize this aspect, and I would appreciate it if they could provide a more in-depth explanation for this claim.\n\n3. It would be beneficial if the authors could furnish a detailed analysis of the time complexity for each component of the algorithm, as well as the memory requirements during the training process. This information is essential for assessing the algorithm's efficiency and scalability.\n\nCurrently, I would rate this paper as borderline. However, I am willing to reconsider my evaluation if the authors can provide additional clarifications and engage in a more in-depth discussion to address the concerns I have raised.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method analytically derives the expression of warp field velocities and provides better tractability compared to directly supervising the flow field as the difference in estimated deformations between consecutive frames. Besides, the method is applicable to forward warp field techniques, and ensures accurate Gaussian trajectories throughout the sequence.\nIn real-world application, the author demonstrates that the warp field regularization accommodates deformations in complex scenes with minimal camera movement, achieving results.\nThe paper is well-written, concise, and has excellent formatting of figures and formulas.", "weaknesses": "Despite the authors' efforts to derive an analytical velocity field and incorporate flow and depth supervision to regularize the Gaussian optimization, I have identified several questions that require attention.\n\n1. The author suggests that the velocity field can be derived from the forward warp field by utilizing a modified Lucas-Kanade expression, and computing the fields corresponding to the displacement and rotation of Gaussians. Nevertheless, I argue that this claim is not entirely accurate. A crucial oversight in the authors' derivation is that optical flow or warp fields are inherently tied to both camera pose and Gaussian motion. Yet, the analysis seems to only consider the scenario flow resulting from the motion of the Gaussian alone, as evident in equations[8][11]. The motivation behind the authors' decision is unclear, but it is likely that incorporating camera pose variations into the Jacobian expression would lead to improved outcomes.\n\n1. The authors should provide a more comprehensive explanation of their implementation, as the description of time integration on the velocity field in Section 4.2 remains unclear. Specifically, what is the value of the time step ∆t employed by the authors, and how is it determined? Furthermore, does the method still satisfy the small motions assumption stated in Line 273 when ∆t is large?\n\n1. The proposed algorithm's reliance on computationally intensive operations, such as the (N × d) pseudo-inverse and time integration, raises concerns about its efficiency. To facilitate a more comprehensive assessment of the method's practical viability, it would be beneficial for the authors to provide a detailed analysis of the time complexity for each algorithmic component, as well as the overall memory requirements. This is essential for evaluating the algorithm's real-world applicability.\n\n1. In my opinion as a reader, the paper's formulas are somewhat verbose, as the Autodiff library is leveraged for efficient gradient and Jacobian computations, obviating the need for manual CUDA implementation. Consequently, it may not be essential to include the detailed derivation process in the main body of the paper.\n\n1. Although the paper's experimental design is thorough, it still exhibits some limitations. The experimental setup is somewhat brief and insufficient. I believe that relevant methods, such as [1][2], should be included as baselines to more thoroughly validate the effectiveness of this method. Additionally, I highly recommend that the authors incorporate a comparison with PhotoSLAM to further bolster the experimental setup.\n\n[1] Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction\n\n[2] DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)