Gaussian Splatting Lucas-Kanade

Authors: Liuyue Xie, Joel Julin, Koichiro Niinuma, Laszlo Attila Jeni
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Gaussian Splatting, regularization, novel view synthesis

Abstract:
Gaussian Splatting and its dynamic extensions are effective for reconstructing 3D scenes from 2D images when there is significant camera movement to facilitate motion parallax and when scene objects remain relatively static. However, in many real-world scenarios, these conditions are not met. As a consequence, data-driven semantic and geometric priors have been favored as regularizers, despite their bias toward training data and their neglect of broader movement dynamics.

Departing from this practice, we propose a novel analytical approach that adapts the classical Lucas-Kanade method to dynamic Gaussian splatting. By leveraging the intrinsic properties of the forward warp field network, we derive an analytical velocity field that, through time integration, facilitates accurate scene flow computation. This enables the precise enforcement of motion constraints on warp fields, thus constraining both 2D motion and 3D positions of the Gaussians. Our method excels in reconstructing highly dynamic scenes with minimal camera movement, as demonstrated through experiments on both synthetic and real-world scenes.

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1729257602518

thread_type:
official_review

<thread_object>
creator: reviewer_d8fp

<content>
summary:
This paper tackles the challenges posed by dynamics and the lack of motion parallax that Gaussian Splatting methods face in real-world settings. The author proposes an analytical solution that adapts Lucas-Kanade method to dynamic Gaussian splitting. The proposed method leverages the forward warp field and derives an analytical velocity field for accurate scene flow computation. The experiment shows that “GAUSSIAN SPLATTING LUCAS-KANADE” outperforms state-of-the-art methods on both synthetic and real-world scenes, and demonstrates accurate reconstructing capability in complex scenes with minimal camera movement.

strengths:
The proposed method analytically derives the expression of warp field velocities and provides better tractability compared to directly supervising the flow field as the difference in estimated deformations between consecutive frames. Besides, the method is applicable to forward warp field techniques, and ensures accurate Gaussian trajectories throughout the sequence.
In real-world application, the author demonstrates that the warp field regularization accommodates deformations in complex scenes with minimal camera movement, achieving results.
The paper is well-written, concise, and has excellent formatting of figures and formulas.

weaknesses:
Despite the authors' efforts to derive an analytical velocity field and incorporate flow and depth supervision to regularize the Gaussian optimization, I have identified several questions that require attention.

1. The author suggests that the velocity field can be derived from the forward warp field by utilizing a modified Lucas-Kanade expression, and computing the fields corresponding to the displacement and rotation of Gaussians. Nevertheless, I argue that this claim is not entirely accurate. A crucial oversight in the authors' derivation is that optical flow or warp fields are inherently tied to both camera pose and Gaussian motion. Yet, the analysis seems to only consider the scenario flow resulting from the motion of the Gaussian alone, as evident in equations[8][11]. The motivation behind the authors' decision is unclear, but it is likely that incorporating camera pose variations into the Jacobian expression would lead to improved outcomes.

1. The authors should provide a more comprehensive explanation of their implementation, as the description of time integration on the velocity field in Section 4.2 remains unclear. Specifically, what is the value of the time step ∆t employed by the authors, and how is it determined? Furthermore, does the method still satisfy the small motions assumption stated in Line 273 when ∆t is large?

1. The proposed algorithm's reliance on computationally intensive operations, such as the (N × d) pseudo-inverse and time integration, raises concerns about its efficiency. To facilitate a more comprehensive assessment of the method's practical viability, it would be beneficial for the authors to provide a detailed analysis of the time complexity for each algorithmic component, as well as the overall memory requirements. This is essential for evaluating the algorithm's real-world applicability.

1. In my opinion as a reader, the paper's formulas are somewhat verbose, as the Autodiff library is leveraged for efficient gradient and Jacobian computations, obviating the need for manual CUDA implementation. Consequently, it may not be essential to include the detailed derivation process in the main body of the paper.

1. Although the paper's experimental design is thorough, it still exhibits some limitations. The experimental setup is somewhat brief and insufficient. I believe that relevant methods, such as [1][2], should be included as baselines to more thoroughly validate the effectiveness of this method. Additionally, I highly recommend that the authors incorporate a comparison with PhotoSLAM to further bolster the experimental setup.

[1] Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction

[2] DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting

questions:
1. The author should provide more details about the experimental setup, e.g., time step ∆t  in Eq.12. Please clarify how is it determined? does the method still satisfy the small motions assumption stated in Line 273 when ∆t is large?

2. The authors highlight in Section 6 that their method is sensitive to SfM initialization. Nevertheless, in my experience, 3DGS can still produce high-quality renderings even when initialized with random point clouds. I find it puzzling that the authors emphasize this aspect, and I would appreciate it if they could provide a more in-depth explanation for this claim.

3. It would be beneficial if the authors could furnish a detailed analysis of the time complexity for each component of the algorithm, as well as the memory requirements during the training process. This information is essential for assessing the algorithm's efficiency and scalability.

Currently, I would rate this paper as borderline. However, I am willing to reconsider my evaluation if the authors can provide additional clarifications and engage in a more in-depth discussion to address the concerns I have raised.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691568722

thread_type:
official_review

<thread_object>
creator: reviewer_uwp8

<content>
summary:
This paper proposes a novel dynamic Gaussian splatting approach based on an analytical velocity field formulation. Similar to Lucas-Kanade, the method linearizes the warp function at specific time intervals and approximates it with a first-order expansion. This approach effectively regularizes scene flow, resulting in improved dynamic reconstruction performance on benchmark datasets.

strengths:
- The analytical formulation of the vector field is interesting.
- More tractable than fully MLP-based Deformable 3DGS or other 2D prior-based supervision approaches.
- Competitive benchmark performance.
- The paper is clearly written and easy to follow.

weaknesses:
- Limited evaluation dataset. Since the method builds on Deformable GS, it would be beneficial to evaluate on the same benchmark datasets, such as DNeRF, HyperNeRF, and NeRF-DS.
- While the method claims tractability as an advantage due to the analytical formulation, reliance on a black-box MLP weakens this claim.
- The video supplementary does not show particularly impressive results; there is noticeable unnatural blurring in the GSLK_network video

questions:
- What exactly is the network architecture of the warp field?
- Compared to fully MLP-based Deformable 3DGS, grid-based warp fields such as 4DGS should struggle less with incorrect motion correlations triggered by a single global MLP representation. How much improvement does the proposed method offer for 4DGS-type warp fields?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730731789298

thread_type:
official_review

<thread_object>
creator: reviewer_gtcv

<content>
summary:
The paper addressed the problem of Gaussian Splatting estimation in dynamics scene and with small movements of the camera or the objects. The authors propose an analytical solution to the deformable warp field used to regularize the estimation instead of a learnable  field or offset-values as applied in the state of the art approaches. 
The key contribution is to provide a detailed derivation and stable estimation, basically a a Gaussian Splatting warp field regularization by scene flow. The approach is then evaluated on various datasets and show improvements compared to the existing solutions.

strengths:
The strength of the paper is to propose a direct analytical solution to the problem of the estimation of the warp field. The solution is mathematically transparent and provides superior results compared to a purely data-driven approach. 
It is a fully new solution which is very relevant for dynamic gaussian estimation and related applications. It is strongly inspired by Lucas-Kanade point tracking and has the same level of generality.

weaknesses:
- A reference implementation should be provided since the approach is very relevant to the community and may include many small details, which have their importance and not detailed in the paper.
- The method relies on external algorithms to provide depth and optical flow information, which are used to supervise the scene flow regularization. The appendix discusses experiments with different depth and flow estimation methods. However, this introduces an additional source of potential error and highlights the importance of using high-quality depth and flow estimation techniques.
- EMF is not explained when it was mentioned but later in the text.
- One reference is missing

questions:
Three limitations are mentioned in the paper. 
I would like to add:
- "Sensivity to noise". What is the behavior under noise? is that possible to further reduce the effect of noise?
- Computational complexity and related aspects are not mentioned at all. What is the advantage of the approach? (compared to pure learning approaches) 
- The paper mentions a motion masking step. How is the threshold defined? Do theses parameters generalize?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731300563661

thread_type:
official_review

<thread_object>
creator: reviewer_gbss

<content>
summary:
This is a paper about the fundamental question of how to derive the velocity of Gaussian means in Gaussian splatting from the finite forward warp transformation. 
It assumes that these transformations are small and that one can live with a first-order Taylor expansion. 

This involves the derivative of the Gaussian parameters after the motion with respect to the Gaussians in the canonical frame and the warp Jacobian.
Then, assuming that scaling and directional reflectivity do not change, one can derive the Jacobian with respect to the mean and the Jacobian with respect to the orientation.

strengths:
+ Deriving analytical solutions is elegant and makes an approach interpretable. 

+ The analytical derivation enables a sound application of the flow term in the Dynamic Gaussian case. 

+ The paper is a pleasure to read.

+ A direct flow regularization makes a network overfit on viewpoints.

+ The idea of flow supervision in Gaussian splatting increases performance.

weaknesses:
While it is nice for a paper to be self-contained, everything 
in this paper until eq. (10) is identical to the "Flow supervision for Deformable NeRF" paper. Eq. (12) is the same as Eq. (7) in the latter. Optical flow rendering is same as in GaussianFlow. The reader is wondering, what exactly is the contribution of this paper when the main ingredients are in the Flow Supervision and the GaussianFlow paper. It really hurts to say that because any reader would enjoy such elegant analytical derivations. Please explain in your response.

questions:
Q1: Given the analytical form of the derivation, it would be interesting to study classic optical flow problems for Gaussian flow: What is the aperture problem in the Gaussian case. What happens when the Gaussian covariance is rank deficient? What scene structures would cause an ambiguous flow? 

Q2: What would happen when the spherical harmonic coefficients would change? 

Q3: What happens with a local deformation like scale change or shear?

Q4: What exactly is the difficulty of including second-order terms?

Q5: How is the method's sensitivity to the choice of canonical frame? DGS methods have been proposed that do not require a choice of canonical frame.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732312335892

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
**To all Reviewers:** We value all reviewers' recognition of our method’s motivation for addressing monocular dynamic Gaussian Splatting, and its strong performance. Thank you very much for the thorough reviews and feedback. We appreciate the recommended experiments and comments on the paper presentation to help refine our paper’s presentation. After carefully studying the recommendations, we included additional details about our method and supporting experiments in each discussion thread. 

To summarize our approach, we present a method for regularizing dynamic Gaussian Splatting by capturing the underlying warp field dynamics. Unlike other works, our method is agnostic to the choice of deformation network and can be trained with images captured with minimal camera motions. Beyond the standard multi-view sequences, we showcase compelling performance on monocular sequences, including the NVIDIA Dynamic Scenes dataset and DAVIS dataset.

We want to thank all reviewers again and hope our additional results and rebuttal responses can clarify any doubts. We are more than happy to provide any further explanations and look forward to discussing more!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an analytical approach to regularize dynamic Gaussian splatting using a modified Lucas-Kanade framework to compute velocity fields of Gaussian motion. It aims to improve the tractability and accuracy of dynamic scene reconstruction by using analytical formulations for scene flow instead of data-driven approaches. The proposed method achieves competitive results on monocular sequences and datasets with minimal camera motion and complex object dynamics.

Strengths:
- Analytical formulation: a novel analytical solution on velocity fields for dynamic Gaussian splatting for better tractability and interpretability.
- Practical settings: designed for realistic scenarios with minimal camera motion, where current algorithms have difficulties.
- Strong experimental results on monocular datasets and dynamic scenes.

Weaknesses:
- Unclear technical novelty: significant overlap in mathematical formulation with existing work such as "Flow Supervision for Deformable NeRF" and "GaussianFlow", raising concerns about originality.
- Limited experimental scope: evaluation on standard benchmarks such as DNeRF and NeRF-DS would better validate the generalizability of the method.
- Computational complexity: computationally intensive operations like pseudo-inverse and time integration may limit practical applications.

The paper provides a clear analytical contribution that improves the understanding and application of Gaussian splatting in dynamic scenarios. It demonstrates competitive experimental results in real-world applications with minimal camera motion. 
The theoretical originality of the paper is undermined by its significant overlap with previous work, but the rebuttal partially addresses the issue.