KEYPOINT-GUIDED 4D GAUSSIAN SPLATTING WITH DECOUPLED SPATIO-TEMPORAL FLOW REFINEMENT

Authors: Jusheng Zhang, Jinzhou Tang, Zhuojie Yang, Sidi Liu, Kaiyu Wu, Mingyan Li, Jian Wang, Keze Wang, Yufeng Yang
Primary area: generative models
Keywords: Keypoint, 4D Gaussian Splatting

Abstract:
We propose KG4D, a novel method for generating time-aware 4D representations
from a single static image or video. Previous methods largely rely on weak su-
pervision signals, failing to introduce fine-grained supervision necessary for cap-
turing detailed spatio-temporal dynamics. In contrast, our approach employs Har-
monic Spatio-temporal Encoding (HSE) to achieve efficient spatio-temporal sep-
aration during training, allowing the model to represent dynamic scene changes
more accurately. Furthermore, Keypoint Feature Calibration (KFC) ensures pre-
cise pose consistency, and Wasserstein Gradient Flow (WGF) enhances motion
coherence, effectively reducing artifacts. Comprehensive evaluation and ablations
demonstrate that our proposed KG4D outperforms existing state-of-the-art meth-
ods on various benchmarks in dynamic 4D generation and novel viewpoint syn-
thesis, validating its effectiveness and superior generation capability.

Decision: reject
Reviewer scores: 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1729471176504

thread_type:
official_review

<thread_object>
creator: reviewer_aqwd

<content>
summary:
The paper presents a pipeline to generate 4D scene with a single image as input. A multi-view diffusion model is used to first generate a static 3D scene, and a video generation model is used to generate the motion as the forth dimension. Key point feature is used in both stages to enhance the consistency during the reconstruction.

strengths:
1. The proposed idea is reasonable. Generating the spatial and temporal dimensions separately and using key points to avoid inconsistencies seems helpful.

weaknesses:
The paper seems to have been written in a hurry and is not yet finished. The idea is sound, so I hope the author better prepare the paper before submission next time. For instance:

1. There are no citation referred in the first two sub-section of the related work (lines 91-107).
2. The abstraction mentioned "evaluation on various benchmarks" (line 32), but only on dataset is evaluated.
3. The experiment section mentioned the comparisons are made on various SOTA methods (lines 403-404, 420-421). However, only a few of them are evaluated.
4. The experiment is not convincing. The dataset is not clearly claimed in the Table title. The numbers are not match between the "ours" row in Table 1 and 2 (probably they are for different datasets, but I didn't see where is mentioned).
5. The experiment section 5.4 contains no result, all the Tables and image results are not referred and discussed.
6. The equations are hard to follow especially for the section 4.2. The letter "z" in the equation in line 243 is not defined. $K^{ref},P^{ref},I^{ref}$ in line 262 are not defined. A citation reference is missing in line 260.

questions:
If the author would like to rebuttal, I do have several questions regarding to the proposed method.

1. As I know, the original DG4D dataset dose not contain key point ground truth, but the paper mentioned the ground truth keypoints $P^{sup}$ in line 215. Is this keypoint manually labelled by the author? Or how the ground truth keypoints are selected.
2. Is there a reason why different sets of keypoint are used for the spatial and temporal generation stages.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730028301073

thread_type:
official_review

<thread_object>
creator: reviewer_kf4q

<content>
summary:
This paper proposes a novel method for generating time-aware 4D representations from a single static image or video. The proposed HSE  more effectively captures dynamic scene changes by decoupling spatial and temporal information. The KFC provides additional supervision of local patterns in spatial dimensions to ensure accurate keypoint alignment in 3D Gaussian Splatting. The WGF  is implemented to enhance motion consistency and stability, effectively reducing artifacts and improving visual coherence. Through comprehensive evaluations, this paper establishes a new benchmark for 4D Gaussian splatting-based methods, demonstrating KG4D's state-of-the-art performance in generating realistic, temporally consistent dynamic scenes.

strengths:
1. The emphasis on decoupling spatial and temporal dimensions, along with the use of keypoint guidance, is a significant contribution.  
2. The visualization results are impressive and effectively demonstrate the method's capabilities.

weaknesses:
1. The writing requires improvement, particularly in the Method section.  
   (a) The citation in line 147 is invalid.  
   (b) The equation in line 257 is difficult to understand.  
   (c) What does C refer to in line 259? Does it represent J?  
   (d) There is a missing citation in line 260.  
   (e) The description in section 4.3 is vague, especially from lines 324 to 341.

2. Regarding contributions:  
   (a) How is the spatial and temporal components decoupled? A detailed description is lacking.  
   (b) Will the design of keypoint feature calibration introduce a large number of parameters ($R^{N,J} $)? Could a self-supervised approach, similar to that used in SCGS[1], be better?  
   (c) The section on motion capturing via wasserstein gradient flow needs more explanation to clarify the meaning of optimal transport.

3. Regarding experiments: The experimental section is somewhat lacking, with insufficient ablation studies. For example, the specific role of spatial-temporal decoupling and the visualization of keypoint feature calibration need to be better explained.
4. The authors did not provide video results.

[1] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730468440994

thread_type:
official_review

<thread_object>
creator: reviewer_zqc9

<content>
summary:
This paper claims that they introduce a novel 4D generation framework, KG4D, which is featured by HSE, KFC, and WGF and outperforms existing state-of-the-art methods on various benchmarks in dynamic 4D generation and novel viewpoint synthesis.

strengths:
The idea of endowing Gaussian with keypoint features and incorporating some regularization for them into generation is interesting. However, the solutions and results presented in this manuscripts are highly suspicious as they completely disregard basic academic standards obviously lacking the basic knowledge for this area.

weaknesses:
The reviewer believes this manuscript should not deserve serious consideration as it is full filled with nonsense sounds totally generated by an AI. Both the motivation and the solutions lack precise and clear meaning, making it incompetent for an academic paper. Moreover, there are numerous errors throughout the whole paper, e.g. they even have not provide correct reference for 3D Gaussian Splatting.

questions:
None
</content>
</thread_object>

</thread>


Metareview: