SplineGS: Learning Smooth Trajectories in Gaussian Splatting for Dynamic Scene Reconstruction

Authors: Jihwan Yoon, Sangbeom Han, Jaeseok Oh, Minsik Lee
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 4D Gaussian splatting, Dynamic scene reconstruction, Novel-view synthesis for dynamic scenes

Abstract:
Reconstructing complex scenes with deforming objects for novel view synthesis is a challenging task. Recent works have addressed this with 3D Gaussian Splatting, which effectively reconstructs static scenes with high quality in short training time, by adding specialized modules for the deformations of Gaussian blobs. However, designing an effective deformation module that incorporates appropriate spatiotemporal inductive biases still remains unresolved. To address this issue, we propose SplineGS in this paper, which utilizes non-uniform rational B-splines (NURBS), an extension of B-spline, to represent temporally smooth deformation. A set of representative trajectories are learned based on NURBS, and the individual trajectories of Gaussian blobs are represented as linear combinations of these trajectories for spatial smoothness. The weights of the combinations are trained based on a multi-resolution hash table and an MLP, with the positions of the Gaussian blobs as the keys. Thanks to this design, the proposed method does not need any regularizers for trajectories, which enables efficient training. Experiments demonstrate that the proposed method provides competitive performance over the existing methods with much shorter training time.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730121979618

thread_type:
official_review

<thread_object>
creator: reviewer_j4yf

<content>
summary:
This paper proposes SplineGS, a model that enables Gaussian Splatting to handle trajectories of Gaussian smoothly. SplineGS first predict position and rotation using an MLP over time and employ non-uniform rational B-splines.

strengths:
1. SplineGS enables smooth representation of dynamic scenes without regulerizations.

2. SplineGS provides competitive performance compared to the existing methods.

weaknesses:
1. Lacks significant advantages over other models. For example, SC-GS shows better rendering quality on the D-NeRF dataset, and on the Neu3D dataset, SplineGS’ PSNR is only 0.03 dB higher than STG. It seems like SplineGS' main contribution is creating smooth trajectories for Gaussians, but only the visualization in Figure 1 supports this. I suggest adding visualizations that shows the Gaussians being learned in a smooth trajectory on various scenes.

2. In line 85, the B-spline functions as a smooth regularizer, but according to line 231, rendering requires $p + M + 1$ additional points. Using the smooth regularizer should allow rendering with just one point, which seems inefficient. Therefore, a comparison of the trade-off between rendering speed and rendering quality is necessary, especially in contrast to using the regularizer.

3. The appearance or disappearance of objects cannot be handled, as mentioned in the conclusion. However, in real-world scenes like Neu3D, I believe that such cases also exist, but there is a lack of analysis on this aspect. In Fig. 6, the flame streak scene in Neural 3D Video shows flames appearing and disappearing, suggesting that the method may handle disappearing objects. However, it remains unclear how SplineGS manages this. I recommend an analysis, such as visualizing the trajectories of the Gaussians.

4. SplineGS performs video view synthesis, but no video results are provided.

questions:
1. I believe that for the Flame Salmon scene, experiments can be conducted using only the first 300 frames, as done in 4DGS, STG, and Gaussian-Flow. Could you provide results of the scene?

2. Are there trajectory results on the Neu3D dataset?

3. Can you provide LPIPS results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730378044702

thread_type:
official_review

<thread_object>
creator: reviewer_x185

<content>
summary:
This paper proposes SplineGS, which reconstructs dynamic scenes using the non-uniform rational B-splines (NURBS).
SplineGS can learn temporally smooth trajectories based on NURBS,  and learn weights of the combinations by a multi-resolution hash table and an MLP.
SplineGS leads to high-fidelity rendering and fast training times.

strengths:
- SplineGS can obtain smooth trajectories utilizing NURBS.
- The paper is well written, and the idea is easy to understand.

weaknesses:
- The novelty is limited. 
   1. Although SplineGS does not need any regularizers for trajectories, NURBS also can be treated as a regularizer. More importantly, this paper does not demonstrate the usefulness of the learned smooth trajectories.
   2. To obtain the weights, SplineGS employs a multi-resolution hash table and an MLP, which can be treated as learning the weights of LBS through the neural network. Compared to other methods of obtaining LBS weights (such as Gaussian kernel RBF), what are the advantages of using neural networks?
  3. In my opinion, the shorter training time and higher FPS are more due to L1 regularization of the opacity value in Eq. (15), which reduces the number of Gaussians.
- Although this paper achievies state-of-the-art or at least competitive performance, hower the performance is not consistent. For example, compared with 4DGS (Yang et al., 2024a), SplineGS achieves higher PSNR and FPS on the D-NeRF dataset, while achieving similar PSNR and lower FPS on the Neu3D dataset. Experiments on more real datasets are needed to evaluate the effectiveness of SplineGS.

questions:
See `Weaknesses`. More,
1. The number of Gaussians per scene, the number of representative trajectories, the training time, and the FPS of SplineGS for the D-NeRF dataset and Neu3D dataset should be provided.
2. `LPIPS` metric also should be reported. As shown in D-3DGS, results on the “Lego” scene of the D-NeRF dataset should not be reported, nor should performance on the validation set be reported.
3. I would like to know the performance of replacing the multi-resolution hash table and MLP with the Gaussian kernel RBF adopted by SC-GS.
4. It would be better to provide some videos to intuitively demonstrate the performance of SplineGS.
5. Are there any failure cases? For example, how SplineGS performs on the “vrig-broom” scene on the Hyper-NeRF dataset.
6. Laking the results of 4DGS on the terms of `SSIM` and `Training time` for the Neu3D dataset.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730479067437

thread_type:
official_review

<thread_object>
creator: reviewer_axdz

<content>
summary:
This paper proposed a new gaussian splatting method to reconstruct the dynamic scene. This paper investigate the usage of non-uniform rational B-splines (NURBS) in temporally smooth deformation representation. Based on the NURBS, they learned a set of trajectories and than linearly combine these trajectories to get the individual trajectories. They can achieve the competitive performance over the existing methods with much shorter training time.

strengths:
1. This paper combined a new representation, called non-uniform rational B-splines (NURBS), with gaussian splatting to reconstruct the dynamic scene, which is declared to be effective in representing smooth deformation.
2. The results show that this method can achieve higher or competitive performance with existing SOTA methods in both the monocular dynamic and multi-view dynamic benchmarks.

weaknesses:
1. To some content, this method still relies on the canonical space (the initial gaussians), but using different strategy to model the deformation. So how about the performance in the non-rigid scenes (scenes like HyperNeRF-data[1])? 
2. Actually, there are some similar strategies or representations can achieve the same effect, like Fourier (Gaussion-flow [2] adopted this presentation). The crucial point is finding a more robust representation to model the deformation or trajectory. So what's the difference or advantage of the used NURBS over other representations like Fourier?

[1] HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields.
[2] Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle.

questions:
Compared with 4DGS, why this method has a significant advantage on D-NeRF dataset but has a relative small advantage on Neu3D dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714983209

thread_type:
official_review

<thread_object>
creator: reviewer_ceb3

<content>
summary:
This paper proposes SplineGS, a novel method for reconstructing dynamic scenes using 3D Gaussian splatting. The authors propose to utilize Non-Uniform Rational B-Splines (NURBS) to represent the trajectories of Gaussian blobs, aiming to achieve temporally smooth deformations. This approach is a departure from existing methods that rely on neural representations, which often require explicit regularization or specialized training mechanisms to ensure smooth trajectories. The paper demonstrates the effectiveness of SplineGS on two datasets, D-NeRF (synthetic) and Neu3D (real-world), showing competitive or superior performance compared to some existing methods.

strengths:
The paper presents a novel approach by employing NURBS to represent Gaussian blob trajectories in dynamic scene reconstruction, a significant contribution to the field of 3D Gaussian splatting.

By utilizing NURBS, SplineGS provides smooth deformations without the need for explicit regularizers or complex training mechanisms, leading to faster and more efficient training. This efficiency translates to competitive or state-of-the-art performance on both synthetic (D-NeRF) and real-world (Neu3D) datasets, demonstrating its effectiveness in reconstructing dynamic scenes.

weaknesses:
1. While the authors claim that NURBS inherently provide temporal smoothness, the paper lacks direct visual evidence to support this claim. The authors should include visualizations of the Gaussian trajectories generated by SplineGS, compared with trajectories from other methods. For instance, plotting the trajectories over time for a specific Gaussian blob in SplineGS and other methods like 4D-GS or D-3DGS would visually illustrate the differences in smoothness.

2. The paper focuses on modeling the deformation of Gaussian centers and rotations but omits incorporating scaling into its deformation model. This omission could limit the expressiveness and applicability of SplineGS, especially when dealing with objects that undergo significant changes in size or shape during motion. The authors should address this limitation and discuss its potential impact.

3. While the paper claims novelty in its use of NURBS for representing Gaussian blob trajectories, this claim requires further examination. The Spline-NeRF paper (Knodt, 2022) explores the use of Bezier splines for trajectory representation in dynamic scene reconstruction, a conceptually similar approach albeit within a NeRF framework. The authors should cite and acknowledge this related work and elaborate on the specific novelties of their NURBS-based method within the context of 3D Gaussian splatting.

4. The evaluation primarily focuses on datasets with limited camera motion. To comprehensively assess SplineGS's capabilities, the evaluation should be broadened to include datasets with more pronounced camera motion, such as HyperNeRF dataset. This would provide valuable insights into SplineGS's performance under challenging conditions and enable a more direct and thorough comparison with baseline methods like 4D-GS and Deformable 3D Gaussians, which are evaluated on HyperNeRF.

questions:
See the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732109462057

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common response to all reviewers

comment:
We would like to express our sincere gratitude for the insightful comments provided by the reviewers. We have revised the paper considering the comments, mostly clarifying some possible misconceptions/misleading phrases and providing more results. Here are some general responses to frequently asked questions, and we will also provide more detailed answers to the individual reviewers.

[supplementary material link](https://anonymous.4open.science/r/SplineGS-0AF1/)

### ● A new visualization video.
We provide a video [(anonymous link)](https://anonymous.4open.science/r/SplineGS-0AF1/SplineGS_video.mp4) comparing the qualitative results of SplineGS and two of the existing methods, which can also be found in the new supplementary material. The video contains the visualizations for the D-NeRF and Neu3D datasets, which demonstrates that SplineGS can provide high-quality results with natural, realistic trajectories based on NURBS. Notably, SplineGS produces smooth trajectories even when increasing the rendering frame rate.


### ● Quantitative comparison on the HyperNeRF dataset.
We provide the results on the HyperNeRF dataset as requested by many reviewers (which has also been added to Table 9 in the appendix). Here, SplineGS achieves comparable performance to the other methods. A critical problem with the HyperNeRF dataset is that the provided camera pose information is not accurate, which is why many recent works omit this dataset. An additional camera pose optimization is required to handle this problem, which is another set of problems left as future work.

| Method   | 3D Printer | Chicken | Broom |
|----------|------------|---------|-------|
| D-3DGS   | 20.28      | 22.76   | 20.47 |
| 4D-GS    | 22.10      | 28.70   | 22.00 |
| SplineGS | 21.97      | 29.13   | 20.56 |

### ● Relatively smaller performance gap on Neu3D.
The Neu3D dataset, consisting of real data, involves changes in color and opacity. Some models, such as 4DGS and STG, incorporate modeling for these. On the other hand, SplineGS concentrates only on the changes in blob positions and rotations, showing a relatively small performance gap for this dataset compared to that in the D-NeRF dataset. Nevertheless, SplineGS achieves higher performance despite having fewer deformation components, suggesting that the proposed representation is quite effective for trajectory modeling. This discussion was already present in Section 4. Considering color and density changes is a separate research direction that requires careful consideration, which is not the main focus of this paper.

### ● LPIPS results.
We provide the LPIPS results as requested by many reviewers (which have also been added to Table 7 in the appendix). SplineGS achieves the best performance in most cases.

| Method   | Trex   | Jumping Jacks | Hell Warrior | Stand up | Bouncing Balls | Mutant | Hook   | Lego   | Mean   |
|----------|--------|---------------|--------------|----------|----------------|--------|--------|--------|--------|
| 4D-GS   | 0.0131 | 0.0128        | 0.0369       | 0.0074   | 0.0155         | 0.0167 | 0.0272 | 0.0382 | 0.0210 |
| D-3DGS  | 0.0098 | 0.0126        | 0.0234       | 0.0063   | 0.0093         | 0.0052 | 0.0144 | 0.0183 | 0.0124 |
| SC-GS    | 0.0119 | 0.0115        | 0.0280       | 0.0072   | 0.0216         | 0.0069 | 0.0139 | 0.0499 | 0.0119 |
| SplineGS | 0.0047 | 0.0065        | 0.0091       | 0.0027   | 0.0025         | 0.0019 | 0.0065 | 0.0298 | 0.0080 |

### ● A new scatter plot comparing efficiency and performance.
We provide a scatter plot [(anonymous link)](https://anonymous.4open.science/r/SplineGS-0AF1/Figure5.png) illustrating the processing time and the number of blobs for various methods, which has also been added to Figure 5 in the appendix. In the plot, we can confirm that SplineGS has a high average FPS even though it uses large numbers of Gaussian blobs compared to the other methods. This demonstrates the inherent efficiency of the proposed representation. In the plot, we can also confirm that SplineGS provides high performance with moderate training time compared to the existing methods.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method for reconstructing dynamic scenes using 3D Gaussian splatting where Non-Uniform Rational B-Splines (NURBS) are utilized to represent the trajectories of Gaussian blobs, aiming to achieve temporally smooth deformations. Based on the NURBS, the proposed method learns a set of trajectories and then linearly combines these trajectories to get the individual trajectories. Employing NURBS to represent Gaussian blob trajectories in dynamic scene reconstruction should be highly appreciated. By utilizing NURBS, the method provides smooth deformations without the need for explicit regularizers or complex training mechanisms, leading to faster and more efficient training.  On the other hand, the reviewers raised concerns regarding unclear novelty against prior works, insufficient analysis of the experimental results, limitation of the proposed.  Advantages against 4D-GS, D-3GS, Fourier representation should be clearly argued with evidence. The performances are not consistent over evaluated datasets: D-NeRF and Neu3D, meaning significant improvement on D-NeRF while small one on Neu3D compared with 4DGS.  Evaluation on HyperNeRF which is with more pronounced camera motion is needed.  Not handling scale deformation need in-depth discussion.  The authors provided additional experiments and in-depth analysis on the results, and argued advantages/contributions of the paper compared some similar works. The authors’ rebuttal and following discussion between the authors and the reviewers have resolved most of the raised concerns, leading to positive consensus for the paper among the reviewers.  This paper should be accepted, accordingly.  AC suggests the authors to deeply argue the potential extension of the proposed method.  Since the deformation considered in the paper is limited to position and rotation changes, in-depth discussion to handle scale, color, and/or density changes will be beneficial to the community.