H3D-DGS: Exploring Heterogeneous 3D Motion Representation for Deformable 3D Gaussian Splatting
Authors: Bing He
Yunuo Chen
Guo Lu
Qi Wang
Qunshan Gu
Rong Xie
Li Song
Wenjun Zhang
Venue: NeurIPS 2025 poster ()
Keywords: 3DGS
3D motion representation
dynamic scene reconstruction
streaming reconstruction
Primary area: applications
TL;DR: Streaming dynamic scene reconstruction with a novel 3d motion representation, accurate, fast, compact.

Abstract: Dynamic scene reconstruction poses a persistent challenge in 3D vision. Deformable 3D Gaussian Splatting has emerged as an effective method for this task, offering real-time rendering and high visual fidelity.
This approach decomposes a dynamic scene into a static representation in a canonical space and time-varying scene motion.
Scene motion is defined as the collective movement of all Gaussian points, and for compactness, existing approaches commonly adopt implicit neural fields or sparse control points. 
However, these methods predominantly rely on gradient-based optimization for all motion information. Due to the high degree of freedom, they struggle to converge on real-world datasets exhibiting complex motion.
To preserve the compactness of motion representation and address convergence challenges, this paper proposes heterogeneous 3D control points, termed \textbf{H3D control points}, whose attributes are obtained using a hybrid strategy combining optical flow back-projection and gradient-based methods. 
This design decouples directly observable motion components from those that are geometrically occluded.
Specifically, components of 3D motion that project onto the image plane are directly acquired via optical flow back projection, while unobservable portions are refined through gradient-based optimization.
Experiments on the Neu3DV and CMU-Panoptic datasets demonstrate that our method achieves superior performance over state-of-the-art deformable 3D Gaussian splatting techniques. Remarkably, our method converges within just 100 iterations and achieves a per-frame processing speed of 2 seconds on a single NVIDIA RTX 4070 GPU.

H3D-DGS: Exploring Heterogeneous 3D Motion Representation for Deformable 3D Gaussian Splatting
Bing He 1 '& Yunuo Chen '& Guo Lu & Qi Wang" & Qunshan Gu
& Rong Xie & Li Song' && Wenjun Zhang Shanghai Jiao Tong University, 2 Alibaba Group
(sandwick ch, theorem) (hanowich_thhereem) jsj ju..tu.ee muu.edu.com cn fcyril-chenyn tu. edu.
Other DGS Mathods
Our H3D-DGS Method
Performance
000
30/00
20:08
04124
No Reg.
GS 3D Motion
Multi view Images Extra Info. Complex Reg.
Feed-Formee Forward
GS 3D Motion
Multi viaw Images
SG.M
Optical Flow Reg.
Figure 1: H3D-DGS: Previous deformable 3DGS methods heavily depend on gradient-basee methods, often introducing complex loss regularizations recover 3D motion. We challenge this convention by observing that, the observability of 3D motion varies across spatial directions. Specifically, the motion components observable in camera's image plane can be directly estimated using optical flow. In contrast, motion components orthogonal to the image plane must be inferred from additional viewpoints. We propose directly incorporate these observable components into fixed, feed- forward 3D motion representation, allowing the neural network to focus solely on learning the unobservable motion. To this end, introduce heterogeneous 3D motion representation- H3D control points which decouple observable and learnable motion components. This heterogeneous structure serves an effective inductive bias, promoting physically plausible 3D motion estimation. our H3D-DGS achieves both superior performance and faster convergence compared to existing methods.
Abstract
Dynamic scene reconstruction poses a persistent challenge in 3D vision. De- formable 3D Gaussian Splatting has emerged as an effective method for this task, offering real-time rendering and high visual fidelity. This approach decomposes a dynamic scene into a static representation in canonical space and time-varying scene motion. Scene motion is defined as the collective movement of all Gaus- sian points, and for compactness, existing approaches commonly adopt implicit neural fields or sparse control points. However, these methods predominantly rely gradient-based based optimization for all motion information. Due to the high degree of freedom, they struggle to converge on real-worke datasets exhibiting complex motion. To preserve the compactness of motion representation and ad- dress convergence challenges, this paper proposes heterogeneous 3D control points, termed H3D control points, whose attributes are obtained using hybrid strategy combining optical flow projeprooeotion and gradicnt- based methods. This design decouples directly observable motion components from those that geomet- rically occluded. Specifically, components of 3D motion that project onto the
"Authors contributed equally to this work. Corresponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

image plane are directly acquired via optical flow back projection, while unob- servable portions are refined through gradient- optimization. Experiments on the Neu3DV and CMU- CMU-Panoptic datasets demonstrate that our method achieves superior performance over state-of-the-art deformable 3D Gaussian splatting tech- niques. Remarkably, our method converges within just 100 iterations and achieves a per-frame processing speed of 2 seconds on a single NVIDIA RTX 4070 GPU.
Introduction
Reconstructing real-world scenes is a long-standing challenge in the field of 3D vision. Recently, 3D Gaussian Splatting (3D-GS) Kerbl et al. (2023) has demonstrated remarkable success in produc- ing high-quality reconstructions for static scenes, This technique utilizes Gaussians to model the scene, assigning them with physically meaningful properties, and renders image by "splatting" these Gaussians onto the image plane.
Why control points? Compared to NeRF Mildenhall et al. (202I). a key advantage of 3D-GS lies in its discrete structure. This structure ensures that scene representation-vye Gaussians -ss concentrated at occupied regions within the scene, avoiding the inefficiencies of a global field that allocates unnecessary representational capacity to empty space. Similarly, the global implicit neural field used for deformation faces a similar issue: only a small part of the scene is dynamic, while the majority remains static. Therefore, discrete, localized motion representation holds promise, as offers precise and flexible modeling of 3D motions at a local level.
In our approach, we adopt control points the motion representation because, despite being compact, they can efficiently represent local motion. We further distinguish between the background and the moving objects in the 3D scene, such that our control points are applied only to the latter.
Why heterogeneous? Realizing accurate 3D motion representation requires reconstructing motion unobservable from a single camera. In a multi-view system, traditional graphics methods Vedula et al. 1999) struggle to align complex 3D points. While gradient-based methods Luiten et al. (2023)
circumvent the 3D alignment problem, they suffer from poor convergence due to their high degrees of freedom (DoF) and require extensive regularization, making them time-consuming.
Our heterogeneous approach differs by combining graphics- based techniques with gradient-based methods. Recognizing that optical flow is the 2D projection of scene flow, we introduce local decoupling strategy, dividing local 3D motion into observable and unobservable components. Com- ponents of 3D motion that project onto the camera plane are directly acquired via optical flow back projection. In contrast, the unobservable portion is refined through gradient-badd based methods. Since we have incorporated regularization into the structural design of the control point, backpropagation in method focuses only on the complex portion of the 3D motion. Our method mitigates convergence issues and achieves fast optimization. This new form of heterogeneous 3D motion representation is referred to as the "H3D control points" approach.
Streaming framework. Building on the introduced 3D motion representation, we present novel generalized streaming framework for dynamic 3D real-world reconstruction with multiple cam- era setups. Beginning with an initial 3D reconstruction, our workflow decomposes the dynamic reconstruction process into distinct submodules: 3D segmentation, H3D control point generation, object-wise motion manipulation, and residual compensation. This structured approach minimizes accumulated errors and ensures compact and robust representation. Our key contributions are follows:
We propose a strategy to split 3D motion into observable and unobservable components. Observable motion (projected to the camera plane) is estimated directly via optical flow back- projection, while unobservable components are optimized using gradients backpropagation.
We introduce H3D control points, a heterogeneous motion representation that significantly improves convergence and accuracy. Our method converges within 100 iterations and achieves processing speed of 2 seconds per frame on single NVIDIA 4070 GPU.
We develop a streaming framework for real-world dynamic scene reconstruction, setting a new benchmark on the Neu3DV and CMU-Panoptic datasets.
2
Related Work
2.1
Dynamic Scene Reconstruction
Neural Radiance Field (NeRF) Mildenhall et al. (2021) have demonstrated strong performance in novel view synthesis by modeling scenes with global continuous implicit functions. Numerous

---

extensions have adapted NeRF to dynamic scenes, Mainstream methods Xian et al. (2021 Wang
itt al. (202T): Park al.
2222aa b); Pumarola et (2021; Du et al. (2021); et a.. (202] 3 Liu
et al. y022g; Fang et al.
2022); Shao et al. 2023 Cao and Johnson (2023) Fridovich-Kei et al.
(2023)
: Liu et al. |ll] (2023): : Song et (2023) modeled dynamic scenes by learning a deformation field
that warps a canonical 3D representation over time. Other research Wang et al. (2023b): [Li et al.
(2023); Lin et al. (2022] 2023) improved reconstruction quality by incorporating camera pose priors.
Additional supervision information, such as depth Attal et al. (2021 1) and optical flow Wang et al. (2023a), were also employed to guide training. NeRFPlayer SSng et al. (2023), used self-supervised learning to segment dynamic scenes into static, deforming, and newly appearing regions, applying tailored strategies to each.
Recently, 3D-GS Kerbl et al. (2023) introduced elegant point-based rendering approach with efficient CUDA implementations. Many 4D Gaussian methods parallel NeRF-based approaches
by incorporating temporal dynamics into spatial representations. For instance, Yang al.
(2023b)
incorporated time-variant attributes into Gaussians, while Dynamic-Gi. Luiten et al. (023) learned
dense Gaussian motion directly. Subsequent works Guo al. (2024); Zhu et al. (2024) incorpo-
rated optical flow to enhance motion accuracy. 3DGStream [Snn et al. (2024) proposed a Neural
Transformation Cache to model per-frame motion. Gaussian-Flow Lin et al. (2024) introduced
Dual-Domain Deformation Model for point-wise motion representation. Spacetime Gaussian Li et al. (2024) proposed a feature splatting and rendering approach. Several studies Wu al. (2023 Yang et al. (2023a); Huang et al. (2023); [in et al. (2024); Das et al. (2024); Yu et al. (2024) deformed canonical Gaussians using global implicit fields to capture 4D dynamics.
2.2 3D Control Points
Real-world scenes typically consist of large static background and a smaller dynamic foreground. While global neural fields are compact in representation, they often lack the flexibility to selectively model only the dynamic regions, and require architectural redesigns adapt across different scenes. In contrast, 3D control point methods are promising due to their ability to flexibly capture localized motion and scale effectively. Traditional graphics approaches (Sorkine (2005); Yu et al. (2004) have long offered flexible deformation techniques that preserve geometric details. Among Tiese,/Sumne et al. (207)) introduced Embedded Deformation (ED) graphs, which sparse control points to represent the motion of dense surfaces, achieving a balance between compactness and flexibility. HiFi4G Jiang et al. (2023) directly adopted EDDggrapss, but its surface reconstruction requires dense camera coverage and is computationally expensive. Moreover, unlike meshes, the volumetric radiance representation of 3D Gaussians not well-suited for the thin nature of surfaces HHangg et al. (2024). Recent gradient-based methods have explored the use of control points for compact motion representation in 3D. SP-GS Wan et al. (2024) clustered Gaussians into superpoints, primitive motion groups, while SC-GS Huang et al. (2023) also adopted the concept of control points, although its direct optimization is challenged by high degrees of freedom (DoFs).
Table Method Comparison. Rep. and Manip. are abbreviations for representa- tion and manipulation, respectively.
Table 2: Average reconstruction results for the Neu3DV dataset. Training time is reported in hours.
Large Move
Fast Train
Compact Rep.
Motion Manip.
Training Time.
Metrics
PSNRT
SSIM
LPIPS1
Dy-GS
/
x
x
X
Dy-GS
27.65
0.9232
0,1313
57.35
MA-GS
x
x
4D-GS
x
/
/
MA-G5
28.76
0.9299
0.1146
45.38
SP-GS
x
x
4D-GS
30.49
0.9401
0.0998
6.88
SC.GS
x
SP-GS
27.03
0.9109
0.1225
40.32
Ours
30.91
0.0941
Our method introduces four key distinctions from existing gradient-based approaches. First, our method robustly handles real-world datasets with large-scale motion Joo et al. (2017). where prior methods Guo et al.] (2024); [Huang et al.] (2023); WWan et al.] (2024); y; Wu et al. (2023) struggle. Second, we introduce structural regularization that improves training efficiency, enabling our model to train as fast as seconds per frame. Third, our H3D control points offer a compact motion representation, requiring only 0.2% as many control points as Gaussians. In comparison, methods like Dynamic- GaSiiiee-SStitenn et al. (2023) and GS-Flow Gao et al. (2024) use redundant motion representations, applying residuals to every Gaussian. Fourth, our control points influence nearby Gaussians spatially,
3

---

Projected
3D Unobserved
small of parallel rays
point
of
VS.
V
Translation O''
00
z
X
Of
Ray Coord.
Cam. Coord.
Image
Figure 2: H3D Control Points. To predict dense 3D motion in sparse manner, we propose H3D control points containing local translation and rotation information. Unlike previous works which learn all motion information with gradient- craseent-based method, we exploit 2D motion priors derived from the optical flow. Both translation and rotation are divided into projected observable part and learnable unobservable part. We model the localized light as parallel rays and make a detailed derivation.
inheriting the desirable properties of traditional control points and supporting user-driven motion manipulation Huang al. 2023).
Method
In this section, we introduce our method for dynamic scene reconstruction with Gaussians. First, we present the notations and symbols used throughout the section facilitate understanding. We then begin with the key idea of our approach-motion decoupling-aan and provide its detailed formulation. Next, we introduce the control point action pattern, where motion information constructed using the proposed motion decoupling strategy. Based on the resulting H3D control points, we describe the full pipeline for dynamic scene reconstruction. Finally, we present the associated loss functions used to optimize the model.
3.1 Mathematical Notation
The symbols used in method are defined follows:
Scalar values are denoted standard font, while vectors and matrices are bolded. The superscripts indicate reference systems, and the subscripts denote point-specific information. Fuclidean distance is represented by || |l. and learnable parameters are marked with hat symbol Positions x; and optical flow values u. are measured in pixels, while 3D positions X,Y, and velocities V are measured in meters. The focal length of the camera is denoted as f. in pixels in Eq. in meters Eqs. 3 (Ce, Cy) indicates the center of projection in pixels, K represents the intrinsic matrix of the camera, R is the rotation matrix, q denotes the unit rotation quaternion, and t represents translation.
For simplicity, linear velocity V and translation t, as well as angular velocity and angular variation represented by the unit quaternion q, are treated analogously our calculations, both represent changes in motion attributes between two neighboring frames.
3.2 Local 3D Motion Decoupling and Heterogeneous 3D Control Points Generation
To effectively represent local translation and rotation, the attributes of a 3D control point encompass its 3D position, local spatial translation, and rotation.
Typically, the 3D position of point is obtained by back projection. Given point located at (xfj, 26) on the image plane with depth 2 its 3D position in the camera coordinate system defined as:
Ze X
f (as Cz. vo cy. T.
Now we focus translation and rotation. At any given viewpoint, portion of the 3D rigid motion projected onto the image plane. This motion component can therefore extracted directly from optical flow and projected into space with appropriate design.

---

Image Center
H3D Control Point
B
V
Surface
Ng's Neigliborhood
Focal
Ideal Control Point
Fg
Image
Dizsed Surface
Center of Projection
Center Projection
Biased Control Point
(b)
(c)
(d)
Figure 3: Auxiliary Diagrams for Local Motion Mapping. (a) An illustration of angles, points, and rays within the neighborhood of Xg. (b) A quantitative depiction of motion projection. (c) A comparison between Gaussians distributed
the ground truth surface and control points located on a biased surface. (d) Illustration for grid sampling. Grid sampling is performed independently for each camera provide 2D motion prior for H3D control points.
To facilitate this, we define a "ray coordinate system," " where the z-axis aligns with the H3D control point, extending from the camera toward the scene. This motion decoupling process is illustrated in The components v y : are derived from optical flow, while vr " w remain learnable. To extend the motion representation the local space rather than just a single point, introduce a local ray approximation assumption. This allows any point I; in the neighborhood N around zo (see Fig- ) share the same local coordinate system. The derivation, given in the Appendix, demonstrates that parallel light retains only motion components perpendicular to the ray direction. The relationship between image plane optical flow and true motion information thus projective:
Fig. 2
Σ dr
dy
d dt
=== N
fft N [Rr..2.22 K-1
Σ N;EN
(n, U.)T
(2)
dt
:2.22
X;EN
where (ut, U:) denotes the optical flow of points x,, The inverse intrinsic matrix K- converts the optical flow to meters, and rotation matrix Rer transforms the system from the camera to the ray coordinate system. The notation F1.2.2 selects the matrix's first two rows and columns, and N the number of pixels in A. To better interpret scene dynamics Vy. we use the focal length f adjust translation from normalized to physical space.
As illustrated Fig. 3(b) the formulas for translational and rotational information can expressed respectively as:
dx dy
Z dt dt jT f
V
(3)
w 1 N
(x] - x() v | - x6l²
(4)
X.EN
Thus, in the ray coordinate system, 3D position X 2D translation V", and ID rotation w/ of H3D control point are determined, while the learnable parts V Wr og complete the motion information, represented as:
t" = (V" Vy VT),
(5)
q"' Fuler2Quat/ (aaoaaat((((() ((f, di, wf)),
(6)
The function Euler2Quat converts Fuler angles quaternions. Fuler angles are intuitive and interpretable for decoupling and controlling rotation components, while quaternions are used in the underlying 3D Gaussian Splatting framework operates represent rotation.
At this stage, all attributes of the H3D control point are derived, completing its construction from the optical flow. For each camera. H3D control points are independently sampled using uniform grid pattern, as illustrated in 3dd) The sampling interval determines the density of H3D control points, and the radius of the circular sampling area represents the range of motion cues. The H3D control points from all cameras are aggregated to represent the motion across the entire scene. An intuitive schematic also provided in Appendix Fig. [bb
We also experimented with the incorporation of local macro-rotation following the ED-graph method Sumner et al. (2007), but observed a decline in performance. We attribute this to struc- tural differences between Gaussians and meshes: while meshes are densely distributed along object
5

---

surfaces, Gaussians exhibit a looser spatial distribution due to their higher degrees of freedom. Con- sequently, we omit the component concerning local macro rotation to minimize biases introduced by rough depth estimations, as illustrated in Fig. 3(c) Although this adjustment compromises some sparsity, optimizes our motion representation approach for volumetric radiance representations like Gaussians.
3.3 Motion Manipulation
The translation t and rotation of each Gaussian G are computed by interpolating from its K- nearest[Cover and Hart (1967) control points C, within its spatial neighborhood No. The interpolation weights wi are inversely proportional to the Euclidean distances to the control points.
(t,
Σ C,ENe
wi (t;, q:),
1/ Xc;
Wi ECEN, 1/ [X Xx.
(8)
The position X, and rotation qe of the Gaussian at time are given by:
X, --tt+
(9)
9: 99
(10)
In practice, we set K = 3, meaning that each moving Gaussian derives its motion from the three nearest control points.
3.4
Streaming workflow
133 Control Points
All Franes Update
Key Brates Update
Operation Flow
Conzol Pours
GS Residaa
radient Flow
Optical Flows
Generation
Leamable
Partial- Lemmalle Param.
Objects
Motion Manipulation
3D Gaussians GLi
Residual Compensation
Segmeetattin
Diff Caussian
Bockground
Rasterizalion
2D Masks
3D Gaussians G.
Image
Figure Streaming workflow. The workflow starts by segmenting the scene into static background and moving objects using 3D segmentation algorithm. Optical flow is then applied to generate H3D control points. Motion-related attributes of the Gaussians are manipulated on object-wise basis. To prevent reconstruction failures from error accumulation, Gaussian attributes are periodically updated keyframe manner, capturing additional scene information as attribute residuals of the Gaussians.
The streaming workflow is depicted in Fig. 4 The proposed method comprises four independent modules: 3D segmentation, H3D control point generation, object-wiss wise motion manipulation, and residual compensation. Inputs include 3D Gaussians, 2D masks, and optical flow. 3D Gaussians are obtained either from the initial static scene reconstruction or the previous frame. The 2D object masks generated via the SAM-track method[Cheng et al. (2023). Optical flows are derived from the DIS method Kroeger et al. (2016).
3D Segmentation. The purpose of 3D segmentation is label each Gaussian either part of moving object or the static background. The control points act on their spatially proximate Gaussians. To ensure they only influence moving objects, it is essential to accurately define the spatial regions where each local representation applies. To achieve this, we utilize multiview masks and employ a Gaussian category voting algorithm segment the scene into dynamic objects and static background regions, following an approach similar to SA-GS Hu et al. (2024). Specifically, each Gaussian is projected onto the image planes of all training viewpoints. By tallying the category labels across views, the Gaussian is assigned the category with the most votes. H3D control points, which also retain category labels when projected back into 3D, only manipulate Gaussians the same category. This framework inherently supports topological transformations between objects with different categories but similar spatial locations.
Residual Compensation. This module is designed to mitigate error accumulation and maintain stable long-term reconstruction. Inspired by video coding techniques like residual coding Wiegand
et al. 2003): Sze et al. (2014), integrate keyframe-based update mechanism. The system

---

performs full optimization of both Gaussians and H3D control points at keyframes, whereas during non-keyframe intervals, Gaussian attributes remain fixed and only control point parameters are updated. To formalize this structure, we introduce the concept of a Group of Scenes (GOS), where GOS-N denotes a sequence with one keyframe followed by N-1 non-keyframes. This design achieves a balance between computational efficiency and temporal reconstruction stability.
3.5 Loss Function
We adopt the same single- eigge-image reconstruction loss used in the original 3D Gaussian Splatting framework, consisting of an LI loss combined with a D-SSIM term, which are computed between the rendered image I render and the ground truth image gt:
L= (1 - X) (Frender, [ge) + ALD-SSIM LLD-ssIM(Trenderr Igt);
(11)
with À set to 0.2. Our motion modeling and streaming pipeline are designed for both computa- tional efficiency and robustness, enabling effective convergence without the need for additional loss constraints.
4 Experiment
SP-05
OF
PSND
U amm
Sysari
Das
Figure 5: Left: Frame 20 of the "coffee martini" sequence from the Neu3DV dataset. Right: Frame 74 of the "softball" sequence from the CMU-Panoptic dataset.
4.1 Datasets and Implementation Details
Neu3DV Dataset Li et al. (202T). The Neural 3D Video Synthesis Dataset includes six sequences, originally captured at resolution of 2704 x 2028, which were downsampled to 1352 x 1014 for training. The sequence 'flame salmon_1 contains 1200 frames, while the remaining five sequences consist of 300 frames each. All sequences were recorded using 15 to 20 static cameras, all placed to form a fanned-out arrangement front of the scene.
CMU-Panoptic
Dataset Joo et al. (2017). The CMU-Panoptic Dataset comprises three sequences featuring complex, dynamic object motions. Each sequence is recorded at resolution of 640 x 360 and consists of 150 frames. The data was collected using static cameras evenly distributed in spherical arrangement around the scene, with 27 used for training and 4 reserved for testing,
Implementation Details. All experiments were conducted on NVIDIA RTX 4070 GPUs. To evaluate the effectiveness of H3D control point method for motion representation, we compared with SP-GS Wan et al. (2024), SC-GS Huang et al. (2023), Dynamic-GS Luiten et al. (002)). MA-GS al. (2024) and 4D-GS Wu al. (2023), all maintaining constant number of Gaussian points. Using the official implementations provided by each baseline, we ensured identical 3I) point initialization. In accordance with the 4D-GS setup, we employed COLMAP Schönberger and Frahm (2016); Schönberger et al. (2016) to initialize 3D points from the first training frames and followed the 10k-iteration reconstruction strategy of Dynamic-GS. For subsequent frames, we trained keyframes with 500 iterations and non-keeframes with 100 iterations, using a single Adam optimizer with fixed learning rates as in Dynamic-GS. Since our method requires both object masks and optical flow as inputs, we specify the sources for these data. For the Neu3DV Dataset, we obtained the object masks using the SAM-track method Cheng et al. (2023). For the CMU Dataset, we reused the foreground masks provided by Dynamic-GS Luiten et al. (2023). Unless otherwise specified, we used DIS Kroeger et al. (2016) as the default optical flow model. The H3D control point sampling configurations were tailored to each dataset based on resolution and motion complexity. We set the
7

---

Table 3: Per-scene results for the Neu3DV dataset. Each cell is color-coded denote performance best for the top performance, second for the second best, and third for the third best.
ranking:
Scene
sear panr_sink
cook spiiach
cut mustted heef
Metrics
PSNRT
SSIMT
LPIPS
PSNRT
SSIMT
LPIPS
PSNRT
SSIMT
LPIPS
Dynamic-GS
P7
31.38
0.9469
0.1119
29.98
0.9388
0.1179
29.64
0,9360
0.1248
MA-GS
30.36
0,9508
31.15
0.9378
0,1053
31.17
0.9401
0.1157
4D-GS 48
31.62
0.9569
0,0808
32.79
0,9522
0.0926
32.13
0.9467
0.0959
SP-GS 43
30.75
0.9474
0.0931
31.32
0.9445
0.0914
30.44
0.9457
0.0942
SC-GS
31.60
0,9510
0.1345
Ours-GoS1
33.23
0,9654
0.0719
33.20
0.9586
0.0796
33.00
0.9609
0.0795
Ours-Go85
33.72
0.9661
0.0704
32,91
0.9579
0.0819
33.23
0.9592
0.0835
Ours-GoS10
33.64
0.9655
0.0716
32.65
0.9553
0.0861
32:47
0.9555
0.0890
Scene
fame_steek
Name _ame___amoo___
coffee__marting
Metrics
PSNRT
SSIMT
LPIPS
PSNRT
SSIM1
LPIPS
PSNRT
SSIM1
LPIPS
Dynamic- GS
30.41
0.9429
0,1121
20.19
0.8875
0.1583
24.29
0,8870
0.1630
MA-GS 1
29.14
0.9456
0.1008
25.05
0.9075
0,1274
25.72
0.8979
0.1533
4D-GS 48
29.28
0.9545
0.0836
28.27
0.9106
0,1289
28.87
0.9198
0.1168
SP-GS
25.59
0,8934
0,1248
25.13
0.9057
0.1320
19.26
0.8291
0.1996
SC GS 14
24.82
0.8972
0.2239
Ours- GoSI
32.84
0.9645
0.0723
28.00
0.9173
0.1083
26.90
0.9140
0.1170
Ours-Go85
0.9155
0.1127
26.71
0.9119
0.1242
Ours-GoS10
32.94
0.9631
0.0733
27.17
0.9127
0.1165
26.51
0.9100
0.1283
Table 4: Per-scene results for the CMU-Panoptic dataset.
Scene
seftball SSIMT
bonzs
banketball SSIMT
Metrics
PSNR1
LPIPS
PSNR1
SSIM1
LPIPS
PSNR1
LPIPS
Dymmic--GS 271
26.93
0.9076
0.1804
27.79
0.9069
0.1769
28.54
0,9032
0.1812
Go82
27.88
0.9227
27.72
0.9203
grid interval to 64 pixels with a 16-pixel radius for the Neu3DV dataset, and 32 pixels with an 8-pixel radius for the CMU dataset.
4.2 Reconstruction Results
Quantitative Results. We first report average quantitative results including PSNR, SSIM, and LPIPS in Tab. 2 Our method consistently outperformed SP-GS, MA-GS, Dynamic-GS, 4D-GS across all metrics and demonstrated significant advantage in training time. While MA-GS also leveraged optical flow, did so via a gradient-basee
method with motion represented as an implicit neural field. Our approach achieved superior visual quality and more efficient training. Per-scene quantitative results are provided in Tab. 3 and Tab, 4 Our approach achieved of (SOTA) PSNR performance in most sequences and yields even better SSIM and LPIPS scores across more scenes, indicating improved perceptual reconstruction quality. In the Neu3DV dataset, SC-GS demonstrates vulnerability when dealing with real-world dynamic scenarios, failing converge efficiently across multiple sequences. In the CMU-Panoptic dataset, MA-GS, SP-GS, SC-GS, 4D-GS all encounter difficulties handling dynamic motion. For instance, 4D-GS fails under rapid scene movement, resulting in object disappearance as depicted in Fig.
Subjective Assessment. In the Neu3DV dataset, the only case where our method did not surpass 4D-GS in objective metrics is shown in Fig. 5 This is primarily due to suboptimal static scene initialization, which decoupled from the subsequent motion modeling in our pipeline. The resulting background artifacts lowered the overall score. In dynamic regions, however, our method still delivered superior detail fidelity and accurate scene reconstruction. from global optimization of the static background, boosting its overall metric but producing inferior quality motion areas. Notably, 4D-GS reconstructed non-existent coffee liquid. SP-GS amplified background artifacts by using unreliable clustering centers as motion control points. SC-GS suffered
contrast, 4D-GS benefited

---

Scar Steak's First Frames
Table 5: Comparison between Gaussians and control points.
.4.0
33.5
33.0
Points
Points
Attr. Dim.
Param.
Category
Num.
Num.
32.5
Scene GS
100k
13
1000k
32.0
Obj.
10k
2 13
>.
Obj. Ctrl
0. 0.2k
2.5k
1.8k
22.5k
31.5
31.0
No Control
Table 6: Comparison between different opti- cal flow methods.
30.5
Puniid
Full Control
0
10
1.5
20
25
30
O.F. Model
2D MSFL
PSNR
SSIM
I.PISS..
Timostop
Figure 6: An example illustrating reconstruc- tion quality degradation across frames under three different settings of 3D control points.
PWC
3V
4.553e-5
33.39
0.9644
0.0737
Net 32
1.509e-5
33.55
0.9649
0.0725
DIS 18
1.230e-5
33.64
0.9655
0.0716
from blurry dynamic regions, as its high-degree-of-freedd control points are challenging train via gradient descent. Dynamic-GS produced blurry results because their complex loss functions lack generalization to dynamic scenarios.
In the CMU-Panoptic dataset, Dynamic-GS reconstructed noisy backgrounds and blurred feet, while 4D-GS struggled with large motion due limitations in its global deformation field. Our method, by contrast, preserved fine textures and structural details without over-smoothing or object disappearance. Subjective comparisons clearly show our method excels in handling challenging motion. Further qualitative results are provided in the supplementary video.
Covergence Speed. Our non-keyframes converged within 2 seconds per frame, and keyframes took approximately 10 seconds each. We anticipate further reductions in processing time as we continue refining the implementation.
4.3 Ablation Study
we compare 3D points across various categories to highlight the compactness of our control point method. Each Gaussian is characterized by 13 attributes. The number of parameters increases with higher harmonic degrees. contrast, the number of 3D control points is significantly smaller, and their attributes remain fixed, indicating much more compact representation and greater potential for streaming transmission.
Points Parameter Comparison. In Tab. 5
The rendering quality of three methods was evaluated under the GoS- 10 configuration: No Control, which lacks motion modeling and simply duplicates keyframes to subsequent non-keyframes; Partial Control, which manipulates the scene using only the projected motion attributes in H3D control points; and Full Control, which manipulates the scene using the complete set of motion attributes in H3D control points.
Effectiveness of H3D Control Points.
The PSNR values for the first 30 frames of the sear steak sequence are presented in Fig. 6 The No Control method experienced rapid quality degradation over time, while Partial Control showed moderate improvement by modeling partial motion. In contrast, Full Control consistently maintained higher quality throughout the sequence.
By examining the gap between eurves over each segment (e.g., steps 0-9, 10-19, 20-29) one can observe the progressive contribution of each component. Moreover, focusing solely on the Full Control setting, which corresponds the GoS-10 setting in our main experiment, one can evaluate how reconstruction quality evolves over time in a streaming setup. As small errors accumulate frame- by-frame (e.g., 0-9, 10-19, 20-29), the residual compensation module applied keyframes (e.g.. 9 9-10, 20) effectively corrects these accumulated errors and restores high-quality reconstruction.
More Advanced Optical Flow Method Leads to Better Reconstruction Result. We further analyzed the impact of different optical flow algorithms Kroeger et al.) (2016); Ranjann and Black (2017); (Sun et al. (2018) on reconstruction performance. Among them, DIS DiSekre et al. (2016) provided the most accurate 2D flow estimates on the Neu3DV dataset. Optical flow accuracy was assessed via the average MSE between each frame and its warped predecessor, where lower 2D alignment error correlated strongly with improved 3D reconstruction quality. This highlights the potential for enhancing the pipeline by adopting more advanced optical flow methods. Furthermore,

---

the observed positive correlation between optical flow accuracy and reconstruction quality validates the effectiveness of our 3D motion modeling approach.
5 Conclusion and Discussion
We propose a novel heterogeneous 3D motion representation framework to address the challenges of dynamic scene reconstruction. By integrating discrete local motion modeling and leveraging H3D control points, our approach effectively decouples observable and learnable components of 31) motion, enabling precise and flexible representation. The framework further establishes streaming pipeline that incorporates key innovations in 3D segmentation, motion manipulation, and residual compensation, ensuring robust and efficient reconstruction.
Our method surpasses existing state-of- ofate-of-the-ar 4D Gaussian splatting approaches on real-world datasets. However, it still has limitations. The quality of 4D reconstruction influenced by the initial static reconstruction, which remains an underexplored area with room for improvement. Additionally, the current method requires multi-view inputs from initial frames and does not support monocular video, a limitation we aim to address in future work.
Another limitation arises from background artifacts such as jittering, which are particularly noticeable in static regions (e.g., windows in some video sequences). These artifacts primarily stem from the incremental nature of our streaming reconstruction framework. Unlike global optimization methods that jointly optimize across all frames, our method updates each frame sequentially based on the previous reconstruction and the current input. While this design improves flexibility and enables low- latency streaming reconstruction, it also introduces susceptibility cumulative noise and temporal drift, leading to observable floaters and jitter in the background. Addressing these artifacts presents an important direction for future work, for example by incorporating lightweight temporal smoothing or consistency mechanisms, while maintaining the efficiency of online reconstruction.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under Grant Nos. 62431015 and 62471290.
References
David Arthur, Sergei Vassilvitskii, et al. k-means++: The advantages of careful seeding. In Soda pages 1027-1035, 2007.
Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O'Toole. Törf: Time-of-fight radiance fields dynamic scene view synthesis. Advances neurul information processing syxtems, 34:26289-26333 2021.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300114,2200 2023.
Yangming Cheng, Liulei Li, Yuanyou Xu. Xiaodi Li, Zongxin Yang, Wenguan Wang. and Yi Yang. Segment and track anything. arXiv preprint 235iv:2305.06658. 2023.
Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions information theory, 13(1):21 27, 1967.
Devikalyan Das, Christopher Wewer, Raza Yunus, Eddy Ilg, and Jan Eric Lenssen. Neural parametric gaussians for monocular non-rigid object reconstruction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1071 10725-0 2024.
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 202/ IEEE/CVF International Conference Computer Vision (ICCV), pages 143144 IEEE Computer Society, 2021.
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
Sara Fridovich- Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K- K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12479-1 12488, 2023.
10

---

Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang. and Ulrich Neu- mann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint urXiv:2403.12360 2024.
Zhiyang Guo, Wengang Zhou, Li Li, Min Wang, and Houqiang Li. Motion-aware 3d gaussian splatting for efficient dynamic scene reconstruction. IEEE Transactions Circuity and Systems for Video Technology. 2024.
Xu Hu. Yuxi Wang, Lue Fan, Junsong Fan, Junran Peng, Zhen Lei. Qing Li, and Zhaoxiang Zhang, Semantic anything in 3d gaussians. arXiv preprint arXiv:2401. 17857 2024.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometri- cally accurate radiance fields. arXiv preprint arXiv:2403. 17888, 2024.
Yi-Hlua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Se-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312 14937, 2023.
Yuheng Jiang. Zhchao Shen, Penghan Wang, Zhuo Su, Yu Hong, Yingliang Zhang, Jingyi Yu, and Lan Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting, arXiv preprint arXiv:2312.03m 2023.
Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Scott Godisart, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social interaction capture. IEEE Transactions on Puttern Analysis and Machine Intelligence. 2017.
Bernhard Kerbl. Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4): 2023.
Till Kroeger, Radu Timofte, Dengxin Dai, and Luc Van Gool. Fast optical flow using dense inverse search. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 14, 2016, Proceedings, Part IV 14, pages 471 488. Springer, 2016.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498 6508, 2021.
Zhengqi Li. Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4273 4284, 2023.
Zhan Li. Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8508-854 8820,,5222222 2024.
Haotong Lin. Sida Peng. Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-viewpoint video. SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
and Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21136 21145, 2024.
Haotong Lin, Sida Peng, Zhen Xu, Tao Xie, Xingyi He, Hujun Bao, and Xiaowei Zhou. Im4d: real-time novel view synthesis for dynamic scenes. arXiv preprint arXiv:2310.08585, 2023.
Jia-Wei Liu, Yan-Pei Cao. Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Devrf: Fast deformable voxel radiance fields for dynamic scenes. Advances in Neural Information Processing Systems, 35:36762 36775, 2022.
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf. Changil Kim. Yung- Yu Chuang. Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13-23, 2023.
Jonathon Luiten, Georgios Kopanas. Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint XrXiv: 2.308.99133, 2023.
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 1091006 2021.
11

---

Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin- Martin-Brualla, and Steven Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv: 106. 1.3288, 2021b.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer, D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10318- 10321, 2021.
Anurag Ranjan and Michael Black. Optical flow estimation using spatial pyramid network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4161-4170, 2017.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-frommmottion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Johannes Lutz Schönberger. Enliang Zheng. Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16632-16642, 2023.
Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen. Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions Visualization and Computer Graphics, (9(5):2732-27444 2023.
Olga Sorkine. Laplacian mesh processing. Furographics (State of the Art Reports), 4(4):1, 2005.
Robert W Sumner, Johannes Schmid, and Mark Pauly. Embedded deformation for shape manipulation. In ACM xiggruph 2007 pupers. pages 80-es. 2007.
Deqing Sun, Xiaodong Yang, Ming-Ya Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934-8943, 2018.
Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20675-20685, 2024.
Vivienne Sze, Madhukar Budagavi, and Gary Sullivan. High efficiency video coding (heve). In Integrated circuit and systems, algorithums and architectures, page 40. Springer, 2014.
Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade. Three-dimensional scene flow. In Proceedings of the Seventh IEEE International Conference on Computer Vision, pages 722-729. IEEE, 1999.
Diwen Wan, Ruijie Lu, and Gang Zeng. Superpoint gaussian splatting for real-time high-fidelity dynamic scene reconstruction. arXiv preprint arXiv:2406. 03697, 2024.
Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2 105,09944 2021.
Chaoyang Wang, Lachlan Ewen MacDonald, Laszlo A Jeni, and Simon Lucey. Flow supervision for deformable nerf. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 21128-21137, 2023a.
Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed neural voxels for fast multi-view video synthesis. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 19706-19716, 2023b.
Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Trunsactions on circuits and systems for video technology, 13(7):560-576 2003.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-tim time dynamic scene rendering. arXiy preprint arXiv:231000858., 2023.
12

---

Wenqi Xian, Jia-Bin Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free- viewpoint video. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition. pages 9421-9431, 2021.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101, 2023a.
Zeyu Yang. Hongye Yang. Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310. 2023b.
Heng Yu, Joel Julin, Zoltán Á Milacski, Koichiro Ninumaa, and László Jeni. Cogs: Controllable gaussian splatting. In Proceedings of the IEFFJCVF Conference on Computer Vision and Pattern Recognition, pages 21624-21633, 2024.
Yizhou Yu, Kun Zhou, Dong Xu, Xiaohan Shi, Hujun Bao, Baining Guo, and Heung- Shum. Mesh editing with poisson-based gradient field manipulation. In ACM SIGGRAPH 2004 Papers, pages 644-651. 2004.
Ruijic Zhu, Yanzhe Liang, Hanzhi Chang, Jiacheng Deng, Jiahao Lu, Wenfei Yang, Tianzhu Zhang, and Yongdong Zhang. Motiongs: Exploring explicit motion guidance for deformable 3d gaussian splatting. Advances Neural Information Processing Systems, 37:101790-01187,, 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how answer these questions. For each question in the checklist:
You should answer [Yes]. [No] or [NA]
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found. IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and Introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Conclusion
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
14

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Method 3.1.3.2 Appendix A.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
All assumptions should be clearly stated or referenced
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Method, Experiment
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
15

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Code will be released after the paper published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
Saaiic//uuidess/ooeeSbbmmsssss sos for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/
gwwss CCodeSoodieeeeeee ssi mollicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
16

---

Answer: [Yes]
Justification: Experiment 4.1 Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Not common in the field
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma eгrоr bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Abstract, Experiment 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
17

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Shanghghppee//arrrr...o/o//////t
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: no socictal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
18

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
19

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.MM policy hatt:p/://eeaarrppe.e/////oorrrp for what should or should not be described.
20

---

A Appendix
A.1 Near-Paralle Light Hypothesis
Image Center
Optical Flos Model
Neighborhood
Focal
f
Forrel
Morken Ealo
R
No
Center of Projection
HSO Control Phants
(a)
(b)
Figure 7: Detailed diagrams for near-parallel light hypothesis proof and H3D control points sampling: (a) Illustration for angles, points, rays Xg 's neighborhood. (b) Workflow for acquiring 2D motion prior.
In the image plane, we consider small region centered at with radius R. Within this region, the rays from the camera projection center to each pixel can be approximated as nearly parallel. detailed illustration is provided in Fig. 7(a) For an arbitrary point x; in the neighborhood of Xo. we aim to show that the angle between the reference ray ro (passing through Xo) and the neighboring ray is first-order small quantity.
To begin, we drop a perpendicular (plumb line) from Xe the ray r; representing the shortest distance between the point and the ray. The length of this perpendicular segment can be expressed as:
|x;
X cos ax,
(12)
where |-| denotes the Euclidean distance, and is the angle between the plumb line and the line connecting Xy to x;. Next, applying the cosine theorem, we represent the distance from the projection center to
1/ cos B,
(13)
where / is the camera focal length, and 3 is the angle between the ray ro and the principal optical axis. Then, using the sine theorem, can express the angle 0 between rays rg and ry:
0: arcsin x;
Xo
COS a
(14)
1/ cos
Since ||x; - Xoll is always less than R, and the trigonometric terms involved are bounded by 1, the angle remains small as long the focal length significantly larger than the neighborhood radius R.
Therefore, we conclude the proof of local ray near- pear-parallelism. practical applications, using normalized focal length greater than 1000 pixels, it is sufficient to restrict the local region to radius of pixels to ensure the validity of this approximation.
A.2 2D Motion Prior Acquisition
We propose an intuitive workflow for acquiring 3D motion priors, as illustrated Fig. 7(b) The inputs are two consecutive frames and object-wise masks.
First, an optical flow network estimates dense motion between the two frames. we generate sampling grid for each object within the view. This grid then passed the local motion calculator- abstracted version of the method described in Sec. 3.2 which processes the grid and associates the resulting 2D motion priors with the corresponding 3D control points.
It is worth noting that the 3D control points are acquired independently from all training viewpoints. denser and more redundant distribution of control points
This multi-view acquisition results compared to single-view methods, thereby reducing the effective influence range of each point. To compensate for this, we adopt a larger sampling interval when selecting control points and use smaller neighborhood around each point for computing motion priors.
21

---

Prunc: Human/Dog 1958/312 PSNR/SSIM
Prune Human/Dog:
Prune 90%:
Human/Dog- : 212/31
PSNR/SSIM: 33.44800.644
PSNR/SSIM 33.44200.6600
Figure 8: Schematic comparison of Gaussians and control points for humans and dogs: We visualized the topology using red and green line segments- red lines connect Gaussian points, while green lines connect control points. Additionally, first frame for both human and dog subjects at various pruning rates, along with the corresponding reconstruction quality over the full sequence.
report the number of control points in the
A.3
H3D Control Points Sparsification
H3D control point pruning is an optional step that further demonstrates the advantages of discrete motion representation. As shown in Fig 8 visualize the correspondence between Gaussians and control points under varying pruning rates. By sparsifying the control points, we achieve substantial gains in representation compactness while maintaining comparable reconstruction quality. For stable clustering initialization eettallittion-espcccillllll when using large number of cluster centers-ve we recommend adopting the k-means++ strategy et al. (2007). This method offers good trade-off between computational efficiency and reconstruction accuracy, making it well-suited for scenarios requiring densely distributed control points.
A.4
3D Motion Visualization.
Figure 9: 3D Motion Visualization: Visualization of Gaussian 3D motion the "sear_steak" and "boxes" sequences.
We visualized the motion of Gaussians in Fig. 9 The motion trajectories accurately depict the subject turning steaks and lifting boxes.
A.5 Significance of Initial 3D Scene.
The suboptimal static scene reconstruction in the first frame of the coffee_mattin martini sequence noticeably impacted the quality the subsequent 4D reconstruction. To further analyze this dependency, we evaluated PSNR, SSIM, and LPIPS-VGG metrics across sequences in the Neu3DV dataset. As shown in Fig. TO the scatter plots exhibit a strong correlation berween the quality of the initial 3D reconstruction and the final 4D results, with points closely following the
line. This alignment suggests that the initial static reconstruction defines an upper bound on the achievable dynamic reconstruction quality. Importantly, our method is designed fully exploit this potential: it builds on
22

---

612-
238
Coll Lobicc
OLL meaee
Bane
Elon sdiae.
ca
0.0
450
Figure 10: Reconstruction quality correlation between the initial frame and the entire video.
the static scene without introducing new Gaussians during the sequence, making efficient use of the available representation regardless of its initial fidelity.
A.6 More Detailed Settings for Fair Comparison
SP-GS Wan et al. (2024) and SC-GS Setting. used the official codebases and initialized Gaussians from the same point cloud to ensure consistency.
MA-GS Guo et al. (2024) Setting. We followed the official implementation and used the same initial point cloud. We selected the deformation-bant pipeline, which employs implicit neural network for motion representation and incorporates optical flow via gradient- bassed optimization.
Dynamic-GS Luiten
(2023) Setting. This approach requires 2D foreground masks and segmentation labels for the initial 3D points. For fair comparison, we merged our objects' 2D masks and applied the labeling strategy described in Sec. (c.433.jjr to assign semantic categories to the initial 3D points. Additionally, we reduced the training iterations from 2k to 0.5k per frame when evaluating on CMU-Panoptic dataset.
"flame_salmon_ 1" sequence, four times longer than the other sequences, we expanded the training iterations from 17k to 68k to ensure fair comparison.
4D-GS Wu al. (202)) Setting. For the "flame_see salmon_
A.7 Additional Subjective Results at Novel Viewpoints
We present additional qualitative results from various sequences both image and video formats to support intuitive evaluation of our method. The images organized in two-row groups, arranged from left to right and top to bottom in the following order: GT, Dynamic-GS, 4D-GS, and Ours. Corresponding video results provided in the accompanying MP4 files.
23

---

Figure 11: More subjective outputs.
24

---

25

---

26


Reviews:
[{"review_id": "TCmJX7MApe", "reviewer": "Reviewer_a1ri", "summary": "The author proposed a new method to model Gaussian dynamics. Unlike previous gradient based method, the author tries to derive the motions of Gaussian control points in a hybrid style: for the full 6 DoF Vx, Vy, Vz and Wx, Wy, Wz, half of them are estimated from optical flow in a closed form style, while the other half are optimized with backpropagation. Finally, the full set of Gaussians are deformed in accordance to its K-nearest control points. Other modules including residual compensation and 3D segments are also introduced to further enhance the robustness of the proposed method.", "questions": "- confusing per-frame optimization cost: seems that you have keyframe 40s and non-keyframe 2s as optimization cost. Considering the method is evaluated in the GoS-10 setup, then how it comes in Figure 1 that your average per frame optimization cost is less than 4s?\n- it is still uncertain how you select control points from optical flows. From my understand, optical flow is a dense map, did you make any subsampling in the pixel space?\n- is the depth information Z required when trying to estimate the control points' motion. If it is, how do you acquire it.", "limitations": "Null", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Good:\n- AFAIK, it's the first work that tries to directly infer the (partial of) 3D motion of control point in a closed form, compared with most other works leveraging optical flow as a differentiable loss or regularization.\n- The proposed hybrid motion estimation method **greatly reduce the optimization space of Gaussians' motion**, hence achieve a fast per-frame optimization speed. This could be promising for real time dynamic reconstruction for the future works. \n\nWeakness:\n- Inappropriate selection of baseline methods. SP-GS is a very weak baseline and SC-GS/DynamicGS are rather earlier works. There are plenty other works that also tries to model gaussian dynamics, including but not limited to: shape-of-motion, Gaussian-Flow.\n- Inappropriate selection of evaluation task. since it is rather a motion modeling instead of 3D dynamic reconstruction, the proposed method is expected to be evaluated on dynamic point tracking task instead of merely dynamic NVS.\n- the ablation study is somewhat limited. Though the hybrid motion estimation reduce the optimization space, the introduced residual compensation is somewhat \"cheating\" by introducing extra optimization space. To validate the effectiveness of proposed method, ablation study of such residual block is necessary.", "was_revised": true, "final_justification": "Overall the paper shows an innovative method that directly infer Gaussian motion attributes from 2d trajectories, indicating a clear difference from previous optimization-dominant methods. Though im still not satisfied with its evaluation sections with absence of point tracking tasks like dynamic3DGS yet i understand its hard for the author to implement this during rebuttal period. **Yet i will highly recommend the author add such task evaluations in their cam ready version. **", "rebuttal": "**W1: Inappropriate selection of baseline methods.**  \n> *SP-GS is a very weak baseline and SC-GS/DynamicGS are rather earlier works. There are plenty other works that also tries to model gaussian dynamics, including but not limited to: shape-of-motion, Gaussian-Flow.*\n\n**A1:**  \nWe appreciate the reviewer’s suggestion. We are confident that our selection of baseline methods is both adequate and fair given the design goals of our approach, and we have conducted further analysis of the two suggested methods — *Shape-of-Motion* and *Gaussian-Flow* — to better clarify our positioning.\n\nOur response is structured as follows:\n\n---\n\n**1. Adequacy and fairness of the selected baselines**  \nOur method is designed around a compact, control-point-based motion representation in a streaming paradigm. Based on these principles, we selected the following baseline methods:\n\n- **SC-GS** and **SP-GS**, both of which adopt control point clustering for Gaussian management and serve as representative approaches in this design space.\n- **DynamicGS**, which supports online reconstruction and dynamic updates in a streaming setting.\n- **MA-GS**, which incorporates optical flow as a prior for estimating Gaussian motion.\n\nThese methods align closely with the structure, input setting, and objectives of our work, enabling a focused and fair comparison.\n\n---\n\n**2. Additional analysis of Shape-of-Motion and Gaussian-Flow**  \nWhile we agree that recent methods such as *Shape-of-Motion* and *Gaussian-Flow* are valuable to discuss, they present limitations that hinder fair empirical comparisons:\n\n- **Shape-of-Motion** addresses a different task — reconstructing dynamic scenes from a single monocular video. Our method, in contrast, is designed for multi-view dynamic reconstruction. Furthermore, its official implementation does not support multi-view settings, making direct comparison infeasible under our task formulation.\n\n- **Gaussian-Flow** adopts a dense modeling strategy that differs significantly from ours. It fits a motion field using a mixture of trigonometric functions and polynomials applied to a large number of Gaussians. Our method instead uses sparse H3D control points, resulting in a more compact, interpretable, and scalable design.\n\n  *Detailed comparison:* In Figure 4 of the Gaussian-Flow paper, sequences such as *coffee_martini* exhibit noticeable blurring in hand regions. Our Figure 5 shows clearer reconstructions on the same sequence, indicating improved fidelity. In terms of efficiency, Gaussian-Flow reconstructs only the first 300 frames with 30k iterations, achieving 30.5 PSNR. Our method reconstructs the full 1200-frame sequences and achieves 30.9 PSNR, demonstrating stronger performance with fewer Gaussians and faster convergence.\n\n---\n\n**Summary**  \nIn summary, while we acknowledge the significance of newer works, our chosen baselines remain the most fair and informative given the task alignment and technical feasibility. We also provide further comparison and analysis to clarify distinctions and support our methodological decisions.\n\n**W2: Inappropriate selection of evaluation task.**  \n> *Since it is rather a motion modeling instead of 3D dynamic reconstruction, the proposed method is expected to be evaluated on dynamic point tracking task instead of merely dynamic NVS.*\n\n**A2:**  \nWe appreciate the reviewer’s insight. While we agree that dynamic point tracking could provide a direct validation of motion estimation, our method is designed and evaluated primarily as a dynamic 3D scene reconstruction system. Accordingly, we believe dynamic novel view synthesis (NVS) remains the most appropriate and fair evaluation protocol for our objective, consistent with prior work such as 4D-GS and MA-GS.\n\n---\n\n**1. Task alignment**  \nOur method focuses on reconstructing dynamic 3D scenes from multi-view videos, and the quality of novel views directly reflects the fidelity of the learned motion and geometry. This makes NVS a natural evaluation target and aligns with the standard benchmarks used by closely related methods.\n\n---\n\n**2. Motion modeling insight**  \nTo further reveal the behavior of our motion representation, we include schematic visualizations of the inferred 3D motion field in Appendix Figure 9. These qualitative results demonstrate that our hybrid motion estimation produces temporally coherent and spatially structured motion fields, offering interpretability beyond raw NVS performance.\n\n---\n\n**3. Broader applicability**  \nWe view our motion representation as a generalizable building block that can serve various downstream tasks. Although our current focus is on improving reconstruction performance, the representation is expressive enough to potentially support point tracking or motion segmentation with minor adaptation.\n\n---\n\n**Summary**  \nOur evaluation strategy centers on dynamic 3D reconstruction, for which dynamic NVS is the most appropriate benchmark. We agree that evaluating on point tracking is meaningful, and we believe future extensions of our method could naturally enable such tasks.\n\n**W3: The ablation study is somewhat limited.**  \n> *Though the hybrid motion estimation reduce the optimization space, the introduced residual compensation is somewhat \"cheating\" by introducing extra optimization space. To validate the effectiveness of proposed method, ablation study of such residual block is necessary.*\n\n**A3:**  \nWe thank the reviewer for this valuable suggestion. We agree that the residual compensation module introduces additional flexibility, and we have conducted an ablation study to examine its contribution in isolation.\n\n---\n\n**1. Effectiveness of residual compensation**  \nAs shown in Figure 6, we provide an ablation across different timesteps to assess the residual compensation module. Without this module, PSNR drops progressively over time (see the red curve across timestep 0-9, 10-19, 20-29), indicating error accumulation due to drift from imperfect motion estimation or appearance changing. When residual compensation is enabled, the scene fidelity remains consistently high (timestep 9-10, 19-20), demonstrating the module’s effectiveness in maintaining reconstruction quality.\n\n---\n\n**2. Role and justification**  \nWhile it does expand the representation space, residual compensation is not a standalone deformation model — it refines the output of hybrid motion estimation rather than replacing it. We designed it to address small misalignments and accumulated errors that arise naturally in long sequences. Its role is analogous to a corrective layer that stabilizes motion over time.\n\n---\n\n**Summary**  \nWe agree that ablation of the residual compensation module is important and have already included such a study (Fig. 6). The results show that it meaningfully improves temporal stability without undermining the goal of compact motion modeling.\n\n**Q1: Confusing per-frame optimization cost**  \n> *seems that you have keyframe 40s and non-keyframe 2s as optimization cost. Considering the method is evaluated in the GoS-10 setup, then how it comes in Figure 1 that your average per frame optimization cost is less than 4s?*\n\n**A4:**  \nWe appreciate the reviewer for highlighting this inconsistency. There was indeed a typographical error in the originally stated optimization times. The correct optimization cost is approximately **10 seconds for keyframes** and **1.5 seconds for non-keyframes**, as also reported in Table 2.\n\n---\n\n**1. Linearity in per-frame optimization cost**  \nDespite the alternating update scheme between keyframes and non-keyframes, the optimization time per frame is roughly proportional to the training iteration across the sequence. This results in a near-linear(5:1) trend in per-frame cost when averaged over the entire sequence.\n\n---\n\n**2. Correction in reporting**  \nWe will correct the stated values in the camera-ready version to avoid future confusion and ensure consistency across all sections of the paper.\n\n\n**Q2: Optical flow subsampling**  \n> *From my understand, optical flow is a dense map, did you make any subsampling in the pixel space?*\n\n**A5:**  \nYes, we perform subsampling as part of our control point selection strategy.\n\n---\n\n**1. Subsampling strategy**  \nAs described in the main paper (L237–240) and the appendix (L471–482), we sample control points in a regular grid pattern over the image space to reduce redundancy and computational overhead.\n\n---\n\n**2. Per-view control**  \nEach camera view independently samples its own set of control points from the corresponding optical flow field. These control points are used to extract motion cues in the local image space.\n\n---\n\n**3. Aggregation and usage**  \nThe sampled control points across views are aggregated and jointly used to guide the motion of the Gaussians via our control-based deformation framework.\n\n\n**Q3: Use of depth information $Z$**  \n> *Is the depth information Z required when trying to estimate the control points' motion. If it is, how do you acquire it.*\n\n**A6:**  \nYes, depth information from the previous frame(kept fixed in the optimization process of the current frame) is required to estimate the 3D positions and motion of control points.\n\n---\n\n**1. Purpose of depth**  \nControl points are distributed on the object surface in the previous frame and are used to warp the corresponding regions to the next frame. To compute their 3D motion, we must first recover their 3D positions, which requires depth information. Moreover, motion information of control points also requires depth information.\n\n---\n\n**2. Source of depth**  \nWe obtain this depth using the `diff-gaussian-rasterization-w-depth` module provided by Dynamic 3D Gaussian. This module renders both the RGB image and the depth map for each frame. The depth values are computed via alpha-blending over Gaussians, consistent with volumetric rendering formulations. This approach ensures smooth and physically meaningful depth estimation from the current Gaussian representation."}, {"review_id": "KL2CsEkAaB", "reviewer": "Reviewer_9PN5", "summary": "This paper proposes H3D control points, a heterogeneous 3D motion representation that separates observable motion (estimated via optical flow) from unobservable motion (learned via gradients). The method achieves faster convergence and higher reconstruction quality on dynamic scenes compared to prior 3D Gaussian Splatting approaches.", "questions": "1. Could you provide any evidence for the decoupling between the observable and unobservable components?\n2. In Figure 5, we can find that the proposed method is better than 4D-GS, while the numbers of the method are worse. Do you have any explanations?\n3. Please explain the results in detail in Tab.3. especially why the proposed method on coffee_martini is significantly worse, while the proposed approach is significantly better.\n\nI think this paper should be compared with recent motion-based approaches with external optical flow. e.g., Motion-GS, at least provide a discussion with it. Threrefore, In the pre-rebuttal phrase, I give a current rating, though I think the idea makes sense and the performance is fine.", "limitations": "yes", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n1. Decoupling the motion into two parts seems reasonable, and the paper presents a good analysis on this in the introduction.\n2. The proposed approach achieves clear improvement compared to the baseline methods.\n\nCons:\n\n1. The paper seems to lack experimental analysis on the unobservable and observable parts. For example, could you find the corresponding observable parts contributed by optical flow, and how the proposed method contributes to the unobservable parts?\n2. You should include the cites in Table 3.\n3. Since you have utilized the third-party optical flow estimation method, could you provide the corresponding inference time for reconstruction? I think it is better to provide a comparison with other methods for the content in L270-L272.\n4. Importantly, to some extent, this paper focuses on leveraging motion flow to facilitate deformable 3D Gaussian Reconstruction. However, this paper does not discuss previous similar approaches, neither in the introduction nor related work. From L70-L92, this paper merely elaborates previous works, rather than discussing the relations between this submission and previous approaches. e.g., MotionGS, Gaussian-Flow, and so on. For the performance comparison, this paper also did not compare with those approaches. Could you provide any explanations?", "was_revised": true, "final_justification": "I think the response has partly addressed my concerns. For the comparison with previous similar work, the response claims that MotionGS is for single-view, while this paper is for multiple-view reconstruction. I did not think this was an efficient explanation. Meanwhile, there are only explanations, no quantative comparisons. I thus maintain my rating.", "rebuttal": "**W1&Q1: Decoupling observable and unobservable motion**  \n> *The paper seems to lack experimental analysis on the unobservable and observable parts. For example, could you find the corresponding observable parts contributed by optical flow, and how the proposed method contributes to the unobservable parts? Could you provide any evidence for the decoupling between the observable and unobservable components?*\n\n**A1:**  \nWe appreciate the reviewer’s thoughtful question. The decoupling between observable and unobservable motion is not only a conceptual design but is explicitly enforced in the architecture of the proposed H3D control points.\n\nThe **observable component** is derived directly from optical flow and remains fixed throughout the optimization process. In contrast, the **unobservable component** is learned through gradient-based optimization and is responsible for compensating motion not recoverable from 2D flow cues alone — such as depth ambiguity or occluded motion.\n\nWe provide two pieces of evidence to support the effectiveness of this decoupling strategy:\n\n---\n\n**1. Structural argument**  \nSince the observable part is fixed and derived solely from optical flow, any failure in the decoupling mechanism would directly impair 3D motion estimation. This would manifest as degraded reconstruction quality both qualitatively and quantitatively. However, our method achieves strong performance across all benchmarks (Tables 1–3), indicating that the unobservable component complements the flow-based motion effectively and validates the robustness of the decoupling design.\n\n---\n\n**2. Ablation study (Figure 6)**  \nWe conduct an ablation analysis to isolate the contributions of the observable and unobservable components in our decoupled motion representation. The **blue curve** represents the setting without any motion modeling, leading to the fastest degradation in reconstruction quality. The **orange curve** reflects the effect of using only the fixed, flow-derived observable motion, which significantly slows error accumulation. The **red curve** corresponds to the full model that additionally incorporates the learned unobservable component, yielding the best performance. This progression clearly demonstrates that both components are essential and complementary for achieving stable and accurate dynamic reconstruction.\n\n---\n\n**Summary**  \nThe decoupling is structurally enforced and empirically validated through benchmark performance and targeted ablation. Our design ensures both interpretability and effectiveness in capturing complex 3D motion.\n\n**W2: Missing citations in Table 3**  \n> *You should include the cites in Table 3.*\n\n**A2:**  \nThank you for pointing this out. We acknowledge the omission and will include proper citations for all baseline methods in Table 3 in the camera-ready version to ensure clarity and credit to prior work.\n\n**W3: Runtime of optical flow estimation**  \n> *Since you have utilized the third-party optical flow estimation method, could you provide the corresponding inference time for reconstruction? I think it is better to provide a comparison with other methods for the content in L270-L272.*\n\n**A3:**  \nThank you for the suggestion. The inference time of the optical flow estimation module is as follows:\n\n- **0.03 seconds** for DIS on CPU  \n- **0.03 seconds** for SpyNet on an RTX 4070 GPU\n- **0.04 seconds** for PWC-Net on an RTX 4070 GPU\n\nWe will include this information in the revised version for clarity and fairness in runtime comparisons.\n\nImportantly, the optical flow is computed only once per frame and does not incur any iterative optimization overhead, making it efficient to integrate into our pipeline. In addition, we will expand our discussion in Lines 270–272 to provide a more complete comparison of overall runtime performance with baseline methods.\n\n\n\n**W4: Missing discussion of related approaches**  \n> *Importantly, to some extent, this paper focuses on leveraging motion flow to facilitate deformable 3D Gaussian reconstruction. However, this paper does not discuss previous similar approaches, neither in the introduction nor related work. From L70–L92, this paper merely elaborates previous works, rather than discussing the relations between this submission and previous approaches. e.g., MotionGS, Gaussian-Flow, and so on. For the performance comparison, this paper also did not compare with those approaches. Could you provide any explanations?*\n\n**A4:**  \nWe appreciate the reviewer’s feedback and agree that a thorough comparison with related work is essential. Our selection of baseline methods was guided by architectural and methodological alignment to ensure fair and meaningful evaluation:\n\n- **Control point–based motion:** Our method leverages sparse control points to represent motion. Accordingly, we compare against SP-GS and SC-GS, which similarly rely on control point clustering for deformable Gaussian control.\n\n- **Streaming reconstruction:** Our pipeline operates in an online, streaming setting. Therefore, we include Dynamic3DGS, which is designed for streaming-based Gaussian reconstruction.\n\n- **Optical flow as prior:** We utilize external optical flow to guide motion inference, akin to MA-GS. Though our integration strategy differs significantly, MA-GS serves as the most appropriate baseline for fair comparison under this design paradigm.\n\n---\n\nWe also acknowledge the relevance of more recent approaches and will clarify these relationships in the revised manuscript:\n\n- **MotionGS:** This method shares conceptual similarities with MA-GS in its use of optical flow as supervision. However, its official implementation is limited to monocular input, whereas our pipeline is designed for multi-view dynamic reconstruction. Given this, we believe that comparison with MA-GS offers a more appropriate and fair evaluation of the effectiveness of our approach.\n\n- **Gaussian-Flow:** This recent method adopts a dense modeling approach, using a mixture of trigonometric and polynomial functions to represent both motion and appearance in the Gaussian field. In contrast, our proposed H3D control points offer a sparse, interpretable, and significantly more compact representation for dynamic motion.\n\n---\n\n**Qualitative differences**  \nAs illustrated in Figure 4 of the Gaussian-Flow paper, sequences like *coffee_martini* still exhibit visible motion artifacts such as blurred hands. Our Figure 5, which visualizes the same sequence, demonstrates clearer structural fidelity, validating the effectiveness of our motion representation.\n\n---\n\n**Efficiency and scalability**  \nGaussian-Flow reconstructs only the first 300 frames of each sequence, requiring 30k iterations to achieve 30.5 PSNR. By contrast, our model reconstructs more challenging sequences up to 1200 frames with a higher PSNR of 30.9, using fewer parameters and faster convergence. This highlights both the scalability and the compactness of our design.\n\n---\n\nWe appreciate the reviewer’s suggestion and will revise the **Related Work** and **Experimental** sections to explicitly address these comparisons.\n\n**Q2: Discrepancy between visual quality and metrics in Figure 5**  \n> *In Figure 5, we can find that the proposed method is better than 4D-GS, while the numbers of the method are worse. Do you have any explanations?*\n\n**A5:**  \nYes — our method achieves higher perceptual quality in dynamic regions, despite reporting lower global metrics. This discrepancy arises from limitations in the scene and the evaluation protocol.\n\nAs discussed in Lines 255–260, the *coffee_martini* sequence suffers from limited camera coverage, especially over static regions such as the background. These areas are poorly reconstructed and introduce floating artifacts and noise, which negatively affect global metrics like PSNR and SSIM.\n\nHowever, our proposed heterogeneous motion representation enables high-fidelity reconstruction of the dynamic foreground (e.g., hands), which is clearly superior to that of 4D-GS in Figure 5. Unfortunately, standard metrics weigh all pixels equally, so background artifacts dominate the score — even when foreground quality improves significantly.\n\nThis case highlights a known mismatch between human perceptual quality and pixel-level error metrics, especially in unbalanced scenes. We will clarify this issue more explicitly in the revised paper.\n\n\n**Q3: Clarification of results in Table 3**  \n> *Please explain the results in detail in Table 3, especially why the proposed method on coffee_martini is significantly worse, while the proposed approach is significantly better.*\n\n**A6:**  \nThe lower metrics on *coffee_martini* are caused by poor initial static reconstruction, while our method still achieves strong dynamic reconstruction quality.\n\nOur method follows a streaming paradigm, where the final performance depends heavily on the initial reconstruction quality. As shown in Appendix Figure 10 (Lines 495–504), suboptimal geometry early in the sequence limits the effectiveness of later motion modeling. In the *coffee_martini* sequence, limited camera viewpoints and scene complexity yield a weaker initial reconstruction, degrading the overall PSNR and SSIM metrics.\n\nDespite this, our heterogeneous motion representation enables accurate reconstruction of dynamic regions — particularly visible in the motion of the hands. This highlights the strength of our approach in modeling deformable motion, even when the static base is suboptimal.\n\nWe believe this observation underscores a promising direction for future work: improving the robustness of static initialization to further enhance dynamic scene performance."}, {"review_id": "R4Y3i0tjBD", "reviewer": "Reviewer_CMCX", "summary": "This paper proposes a method to improve 4D Gaussian reconstruction. Moving objects are segmented from the background, and their motion is modeled using control points. Optical flow is incorporated to enhance motion reconstruction accuracy.", "questions": "- Some methods are only compared in Tables 3 and 4, but lack corresponding qualitative comparisons in the video.\n- Important baselines are missing, such as Spacetime Gaussian, which is mentioned in the Related Work section.\n- Jittering artifacts are noticeable in the background (e.g., windows in the video from 0:00–0:08 and 0:45–1:20). Is this caused by inaccurate segmentation or reflectance-related issues?\n- Tables 3 and 4 should include citations for the compared methods.\n- The organization of Section 3 could be improved by adding an overview paragraph before introducing symbols.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "4D reconstruction is a key task in computer vision. The proposed approach is technically sound and demonstrates good efficiency, as shown in the experiments. However, the results still contain some artifacts in the background, and the experimental evaluation could be further strengthened.\nIn terms of methodology, while some of the components, such as foreground-background segmentation and control point modeling, are inspired by prior work, the implementations with integration of optical flow appear to improve the performance. Therefore, I would consider the paper to offer moderate novelty. Besides, the paper is easy to follow, and the code release of the proposed method would also be valuable to the community.", "was_revised": true, "final_justification": "Although the paper has some limitations, the proposed method outperforms state-of-the-art approaches, and the authors have committed to improving the paper. Therefore, I maintain a generally positive view of the paper.", "rebuttal": "**Q1: Qualitative comparisons missing for some methods in Tables 3 and 4**  \n> *Some methods are only compared in Tables 3 and 4, but lack corresponding qualitative comparisons in the video.*\n\n**A1:**  \nThank you for the suggestion. While the supplemental video includes representative sequences to illustrate key differences, it does not currently cover all evaluated methods from Tables 3 and 4 due to space constraints.\n\nWe agree that a complete qualitative comparison is valuable. In the final version, we will include additional qualitative results covering all methods to provide a fair and comprehensive visualization of reconstruction quality across the full benchmark suite.\n\n**Q2: Missing baseline**\n> *Important baselines are missing, such as Spacetime Gaussian, which is mentioned in the Related Work section.*\n\n**A2:**  \nWe appreciate the reviewer’s comment. We carefully selected baselines that align with the core architectural and functional design of our method. Specifically:\n\n- **Control-point-based motion representation:** We compare against SC-GS and SP-GS, which also use control points to guide Gaussian motion.\n- **Streaming reconstruction:** We include DynamicGS as a baseline due to its online, frame-wise reconstruction pipeline.\n- **Optical flow as a motion prior:** We include MA-GS, which incorporates optical flow similarly to our method.\n\nRegarding **Spacetime Gaussian (STG)**, we expand on this comparison in Lines 227–228. While STG is a strong recent method, several key differences make a direct comparison challenging:\n\n- **Variable vs. fixed primitives:** STG uses temporally varying opacity to allow Gaussians to appear and disappear over time. This increases flexibility but results in a much larger effective number of Gaussians for longer sequences. Our method maintains a fixed and compact set of H3D control points, enabling more efficient and interpretable motion modeling within a constrained representation space.\n\n- **Training efficiency:** STG reports 40–60 minutes of training for a 50-frame sequence on a single A6000 GPU. Extrapolating this to the full Neural 3DV dataset (2700 frames) implies more than 45 hours of training time. In contrast, our method reconstructs the same dataset in just 1.9 hours.\n\nWe acknowledge the significance of STG and will clarify its relation to our work in the final version. However, given the substantial differences in assumptions, runtime complexity, and representation scale, we believe our current selection of baselines provides the most meaningful and fair evaluation of our method's contributions.\n\n**Q3: Jittering artifacts in the background**  \n> *Jittering artifacts are noticeable in the background (e.g., windows in the video from 0:00–0:08 and 0:45–1:20). Is this caused by inaccurate segmentation or reflectance-related issues?*\n\n**A3:**  \nThank you for the observation. The jittering artifacts in the background primarily arise from the characteristics of our *streaming reconstruction* framework. Unlike global optimization methods that jointly optimize across all frames, our method updates each frame incrementally based on the previous frame and the current input.\n\nWhile this design significantly improves flexibility for real-time or streaming scenarios—allowing for low-latency processing and online reconstruction—it also introduces susceptibility to cumulative noise and temporal drift. These factors contribute to the observed background artifacts such as floaters and jittering.\n\nAddressing these limitations is a valuable direction for future work, potentially by integrating lightweight temporal smoothing or consistency mechanisms without compromising streaming efficiency.\n\n**Q4: Missing citations in Tables 3 and 4**  \n> *Tables 3 and 4 should include citations for the compared methods.*\n\n**A4:**  \nThank you for the suggestion. We acknowledge this oversight and will include proper citations for all baseline and compared methods in Tables 3 and 4 in the camera-ready version to ensure clarity and proper attribution.\n\n**Q5: Section 3 organization improvement**  \n> *The organization of Section 3 could be improved by adding an overview paragraph before introducing symbols.*\n\n**A5:**  \nThank you for your suggestion. We plan to add the following overview paragraph at the beginning of Section 3 to improve clarity and guide the reader:\n\n*“In this section, we introduce our method for dynamic scene reconstruction with Gaussians. First, we present the notations and symbols used throughout the section to facilitate understanding. We then begin with the key idea of our approach—motion decoupling—and provide its detailed formulation. Next, we introduce the control point action pattern, where motion information is constructed using the proposed motion decoupling strategy. Based on the resulting H3D control points, we describe the full pipeline for dynamic scene reconstruction. Finally, we present the associated loss functions used to optimize the model.”*\n\nWe believe this will improve the readability and logical flow of the section."}, {"review_id": "pBrwKk5BA5", "reviewer": "Reviewer_J5Ab", "summary": "This paper proposes H³D-DGS, a novel framework for dynamic 3D scene reconstruction using Deformable 3D Gaussian Splatting. The key idea is to decouple the observable and unobservable components of 3D motion by leveraging optical flow to directly compute motion projected onto the image plane and learning the residual (unobservable) motion via gradient-based methods. This results in Heterogeneous 3D (H3D) control points, which serve as a more efficient and physically grounded representation of 3D motion.", "questions": "- The paper introduces several key components—motion decoupling, control point interpolation, object-wise manipulation, and residual compensation—but no ablation study is conducted to isolate the contribution of each module.\n\n- The loss function (Eq. 11) adopts a standard L_1_D-SSIM formulation, but there is no analysis of its necessity or comparison with other regularization terms (e.g., temporal smoothness, deformation consistency, or sparsity) commonly used in dynamic 3D reconstruction.\n\n- How does the method perform in cluttered or partially labeled 3D scenes, where object-wise segmentation may be noisy or incomplete?\n\n---Update after rebuttal---\nThe author rebuttal has addressed most of my concerns, so I will increase my score to borderline accept.", "limitations": "See the weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths：The paper introduces a novel combination of scene graphs and object-level latent vectors, along with a deformable guidance strategy that balances realism and generalizability. Extensive experiments across four datasets, including ablations, support its effectiveness. The method is clearly presented with helpful figures like Fig. 2 and 3.\n\nWeakness:\n- Equation (6): Euler2Quat is not defined or referenced, leaving readers unclear on its role.\n- Equation (11): The loss function is incomplete—no variables or inputs are specified in the expression.\n- Table 1 highlights advantages like “Large Move,” “Fast Train,” “Compact Rep,” “Motion Manip,” but the experiments do not systematically correspond to these capabilities.\n- In Tables 3 and 4, the meaning of GoS1, GoS5, GoS10, GoS2 is not explained, which obstructs full understanding of results.", "was_revised": true, "final_justification": "Thank you for providing a clear and comprehensive response. Your rebuttal has addressed most of my concerns, so I will increase my score to borderline accept.", "rebuttal": "**W1: Equation (6) — Undefined use of Euler2Quat**  \n> *Euler2Quat is not defined or referenced, leaving readers unclear on its role.*\n\n**A1:**  \nThank you for pointing this out. The function `Euler2Quat` converts Euler angles to quaternions. Euler angles are more intuitive and interpretable for decoupling and controlling rotation components, which is why we adopt them during motion decomposition. However, the underlying 3D Gaussian Splatting framework operates using quaternions to represent rotation. Thus, we convert the decoupled Euler angle outputs into quaternions before feeding them into the Gaussian representation.\n\nWe will add a brief explanation and reference in the revised version to clarify this conversion step.\n\n\n**W2: Equation (11) — Incomplete loss function**  \n> *The loss function is incomplete—no variables or inputs are specified in the expression.*\n\n**A2:**  \nThank you for your suggestion. The loss function in Equation (11) is computed between the rendered image and the ground truth image. We agree that the variables and inputs should be made explicit for clarity.\n\nIn the camera-ready version, we will revise the equation to include these input terms and ensure the notation clearly reflects the role of each variable in the loss computation.\n\n\n**W3: Table 1 claims vs. experimental validation**  \n> *Table 1 highlights advantages like \"Large Move,\" \"Fast Train,\" \"Compact Rep,\" \"Motion Manip,\" but the experiments do not systematically correspond to these capabilities.*\n\n**A3:**  \nThank you for pointing this out. We clarify that each of the capabilities listed in Table 1 is supported by specific experimental evidence:\n\n- **Large Move:** The CMU-Panoptic dataset contains large-scale motions that challenge many existing dynamic 3DGS methods. As described in Lines 220–221 and 251–253, our method successfully reconstructs scenes in this setting, demonstrating robustness to large motion.\n\n- **Fast Train:** We discuss training efficiency in Table 2 and Lines 270–272. Our method converges faster than prior approaches, validating the \"Fast Train\" claim.\n\n- **Compact Rep:** Table 5 shows that our H3D control points form a compact representation of motion. Unlike dense per-frame residuals, our control points use fewer attributes and less than 2% of the parameter count — achieving efficient motion encoding.\n\n- **Motion Manip:** The mechanism by which our H3D control points influence Gaussians is described in Lines 174–180. Similar to SC-GS, our control points are designed to support object-level manipulation. As the motion is modeled explicitly through these control points, they naturally enable motion editing and manipulation.\n\n**W4: Unclear meaning of GoS1, GoS5, GoS10, GoS2 in Tables 3 and 4**  \n> *In Tables 3 and 4, the meaning of GoS1, GoS5, GoS10, GoS2 is not explained, which obstructs full understanding of results.*\n\n**A4:**  \nThank you for raising this point. We provide a detailed explanation of the GoS settings in Lines 199–206 of the paper. Briefly, **GoS** (Group of Splatting) refers to the frequency of residual compensation used during dynamic reconstruction.\n\nA GoS-*N* setting means that for every group of *N* frames, there is one key frame and *N–1* non-key frames. The key frame is fully optimized, while the non-key frames rely more heavily on motion propagation from the key frame using our H3D control point mechanism.\n\nA smaller *N* (e.g., GoS1) corresponds to more frequent optimization and higher reconstruction fidelity, while a larger *N* (e.g., GoS10) increases the challenge by reducing supervision frequency, thereby placing greater emphasis on the robustness of the motion representation.\n\n\n**Q1: Missing ablation for key components**  \n> *The paper introduces several key components—motion decoupling, control point interpolation, object-wise manipulation, and residual compensation—but no ablation study is conducted to isolate the contribution of each module.*\n\n**A5:**  \nThank you for the detailed observation.\n\nThe term **“control point interpolation”** appears to be a typo and likely refers to the process of generating and placing H3D control points, which form the core of our proposed motion representation. This novel mechanism is central to reconstruction quality. We evaluate the impact of control point density in Appendix Figure 8 (Lines 483–491), which supports our choice of a compact yet expressive control structure.\n\n---\n\nWe also isolate the effect of **motion decoupling** and **residual compensation** in our ablation study (Figure 6). These experiments demonstrate that both components are critical for handling complex motion: the decoupled motion provides a strong prior from optical flow, while the residual term corrects the unobservable dynamics, leading to significant improvements in temporal consistency and reconstruction accuracy.\n\n**1. Effectiveness of motion decoupling**  \nWe conduct an ablation analysis to isolate the contributions of the observable and unobservable components in our decoupled motion representation. The **blue curve** represents the setting without any motion modeling, leading to the fastest degradation in reconstruction quality. The **orange curve** reflects the effect of using only the fixed, flow-derived observable motion, which significantly slows error accumulation. The **red curve** corresponds to the full model that additionally incorporates the learned unobservable component, yielding the best performance. This progression clearly demonstrates that both components are essential and complementary for achieving stable and accurate dynamic reconstruction.\n\n\n\n**2. Effectiveness of residual compensation**  \nAs shown in Figure 6, we provide an ablation across different timesteps to assess the residual compensation module. Without this module, PSNR drops progressively over time (see the red curve across timestep 0-9, 10-19, 20-29), indicating error accumulation due to drift from imperfect motion estimation or appearance changing. When residual compensation is enabled, the scene fidelity remains consistently high (timestep 9-10, 19-20), demonstrating the module’s effectiveness in maintaining reconstruction quality.\n\n---\n\nRegarding **object-wise manipulation**, this feature serves primarily as a qualitative validation of our motion representation. It emerges naturally from the H3D control point mechanism (Lines 174–180), enabling semantically meaningful and interpretable manipulations. While it is difficult to isolate quantitatively, its effectiveness is reflected in the structured spatial allocation of control points and their semantic correspondence across the scene.\n\n\n\n**Q2: Lack of analysis on loss function design**  \n> *The loss function (Eq. 11) adopts a standard L₁-DSSIM formulation, but there is no analysis of its necessity or comparison with other regularization terms (e.g., temporal smoothness, deformation consistency, or sparsity) commonly used in dynamic 3D reconstruction.*\n\n**A6:**  \nThank you for raising this point. In the current version, we deliberately adopt a streamlined loss function—based on L₁ and DSSIM—to emphasize the core contributions of our method: motion representation and structural regularization. Despite its simplicity, this formulation achieves robust performance across a range of datasets and scenarios, as demonstrated in our experimental results.\n\nWe agree that integrating additional regularization terms, such as temporal smoothness, deformation consistency, or sparsity, could further improve the stability and fidelity of dynamic reconstruction, particularly under extreme motion or occlusion. We view this as a valuable direction for future work and are encouraged by the potential for the community to build upon our framework with more sophisticated loss formulations.\n\n**Q3: Performance in cluttered or partially labeled 3D scenes**  \n> *How does the method perform in cluttered or partially labeled 3D scenes, where object-wise segmentation may be noisy or incomplete?*\n\n**A7:**  \nIn cluttered or partially labeled 3D scenes, our residual compensation module helps mitigate the effects of noisy or incomplete segmentation by refining motion estimation beyond what is captured by initial segmentation labels. This contributes to improved robustness in moderately challenging conditions.\n\nHowever, the overall performance may still degrade under severely erroneous segmentation, as inaccurate object boundaries can lead to improper control point grouping and motion estimation, resulting in suboptimal reconstruction.\n\nFortunately, modern 2D segmentation techniques exhibit strong generalization and are effective in many practical scenarios, which helps provide reliable segmentation masks. We also recognize the importance of advancing 3D semantic understanding and view this as a promising direction for future work—one that would directly enhance the effectiveness of methods like ours."}]

Author rebuttal:

Meta-review: This paper proposes a new fast dynamic Gaussian splatting method based on heterogeneous 3D control points. The main idea is to reduce the optimization space utilizing optical-flow back-projection. All reviewers agreed that this new hybrid representation is novel and innovative. All reviewers also agreed that the efficiency of the representation is convincing. Besides the "easy" questions that were quickly resolved in the rebuttal (e.g., requests for more ablation studies, which were already present in the appendix, and some minor technical questions), there were two main issues discussed during the discussion phase:

- Inappropriate comparison: After some discussion, it was shown that some of the recent methods are under single-view settings, which are not directly comparable to multi-view settings like the proposed method. Similarly, other methods also had some issues for direct comparisons (e.g., based on sparse vs. dense control points or different working regions). After the discussion, the comparison issues were mostly resolved.

- Background artifacts: The authors argued that this is mainly due to the streaming setting of the proposed method. The reviewer accepted this explanation.

AC agrees that the hybrid representation is novel and the efficiency is convincing, and also agrees on the conclusions of the above two issues. Accordingly, an accept decision is recommended. One of the reviewers suggested including evaluations on motion modeling itself, which AC also believes to be valuable if it is included in the final version.
Decision: Accept (poster)