SCas4D: Structural Cascaded Optimization for Boosting Persistent 4D Novel View Synthesis
Authors: Jipeng Lyu
Jiahua Dong
Yu-Xiong Wang
Venue: Accepted by TMLR ()

Abstract: Persistent dynamic scene modeling for tracking and novel-view synthesis remains challenging, particularly due to the complexity of capturing accurate deformations while maintaining computational efficiency. In this paper, we present SCas4D, a novel cascaded optimization framework that leverages inherent structural patterns in 3D Gaussian Splatting (3DGS) for dynamic scenes. Our key insight is that real-world deformations often exhibit hierarchical patterns, where groups of Gaussians undergo similar transformations. By employing a structural cascaded optimization approach that progressively refines deformations from coarse part-level to fine point-level adjustments, SCas4D achieves convergence within 100 iterations per time frame while maintaining competitive quality to the state-of-the-art method with only 1/20th of the training iterations. We further demonstrate our method's effectiveness in self-supervised articulated object segmentation, establishing a natural capability from our representation. Extensive experiments demonstrate our method's effectiveness in novel view synthesis and dense point tracking tasks. Please find our project page at https://github-tree-0.github.io/SCas4D-project-page/.

Jiahua Dong Published in Transactions on Machine Learning Research (06/2025)
SCas4D: Structural Cascaded Optimization for Boosting Persistent 4D Novel View Synthesis
Jipeng Lyu University of Mlinois Urbena- Chanopng
kyjpenglu@@mail.comm com
jiahuad24hilinoois.com edu
University of Blinois Urbana- Chana--n
Yu-Xiong Wang
ymu@ilkkooiscoo edu
University of Mlinois Urbona- Chann-Cn
Reviewed on OpenReview: https: // openreview. met/ forum? id- Yeyc
Frame 1
Frame t
Efficient Training
Point-Tracking
Articulated Objects Segmentation
Ours
Build Gaussians
Optimize
DynamiciDGS
Structural Cascade
Figure 1: Our method achieves satisfying rendering results with 100 training iterations frame. Leveraging learned deformation information, we also demonstrate successful articulated object segmentation.
Abstract
Persistent dynamic scene modeling for tracking and novel-view synthesis remains challeng- ing, particularly due to the complexity of capturing accurate deformations while maintaining computational efficiency. In this paper, present SCas4D, novel cascaded optimization framework that leverages inherent structural patterns in 3D Gaussian Splatting (3DGS) for dynamic scenes. Our key insight is that real-world deformations often exhibit hicrarchical patterns, where groups of Gaussians undergo similar transformations. By employing a struc- tural cascaded optimization approach that progressively refines deformations from coarse part-level fine point-level adjustments, SCas4D achieves convergence within 100 itera- tions per time frame while maintaining competitive quality to the mate-of-theeart method with only 1/20th of the training iterations. We further demonstrate our method's effec- tiveness in self-supervised articulated object segmentation, establishing a natural capability from our representation. Extensive experiments demonstrate our method's effectiveness in novel view synthesis and dense point tracking tasks. Please find our project page at https: htgp::///gthhbb tetccbbbrrre-.... w.w.-ubg..ttt..t.... Beo/SccsDD-project-paee

---

Published in Transactions on Machine Learning Research (06/2025)
1 Introduction
Dynamic novel-view synthesis provides powerful framework for modeling dynamic 3D scenes, with ap- plications in fields such AR/VR, robolics, and autonomous driving. The ability to learn and render dynamic scenes can enable immersive, interactive experiences. Recent advances (Chen et al., 2022; 2023a; Fridovich-Keil et al., 2022; Hu et al., 2023; Müller et al., 2022; Chen et al., 2023b; Garbin et al., 2021; Hedman et al., 2021; Reiser et al., 2023; Wizadwongsa et al., 2021; Chen et al., 2021; Niemeyer et al., 2022; Wynn & Turmukhambetov, 2023; Yu al., 2021), inspired by Neural Radiance Field (NeRF) (Milden- hall al., 2020), have leveraged radiance fields for 3D scene modeling. However, the inherent limitations of NeRF, including its high computational demand for network queries and volume rendering. Moreover, its implicit representation restricts some downstream applications, such precise tracking and articulated object segmentation.
The recent development. of 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) has notably improved efficiency in rendering static scenes by representing the 3D world with a collection of Gaussians and employing efficient rasterization. This insight has led to explorations of dynamic scene rendoring with 3DGS(Wu al., 2024; Duan et al., 2024; Sun et al., 2024; Luiten et al., 2024), with some methods (Luiten et al., 2024; Abou- Chakra et al., 2024; Zhang et al., 2022) focusing on learning transformations (e.g., position and rotation changes) for each Gaussian between frames. These methods achieve realistic deformation tracking over time. However, the structural information of 3DGS is underexplored. Specifically, the motion in 3D scenes often follows different structural patterns, such as rigid parts, non-rigid parts, and static backgrounds. While SCGS Huang et al. (2024) trics to apply sparse control points to represent motion, they fail to capture accurate and dense point-level tracking. Their requirements for correctly distributed control points further limit the application.
Inspired by these observations, we would like to answer the question that "Can 4D scenes directly benefit from the structural information of vanilla 3DGS. Since objects in real-world scenes often consist of multiple parts that exhibit similar deformations, we propose a structural cascaded optimization approach that orga- nizes the Gaussians in a top-down manner. In the coarse-level, we optimize the 3D parts to approximate their deformation in the new time frame. Following this, the fine-level optimization will further improve the deformation of each Gaussian. To ensure appropriate scale change between different levels, we adopt optimization with three levels that balance efficiency with the ability capture fine-grained motion.
Without significant modifications to 3DGS, our methods show the great potential of the underlying struc- tural information of 3DGS. By the structural cascaded optimization, we achieve 20x speedup over Dy- namic3DGS (Luiten et al., 2024) in training time. In the meanwhile, we maintain the ability to deliver comparable tracking performance for dense points and further provide the capability articulated object segmentation, as shown in Figure 1. Extensive experiments demonstrate the effectiveness our method in different tasks.
In summary, our contributions can be concluded as:
We explore the possibility of utilizing internal structural information from 3DGS for dynamic scenes, significantly accelerating the convergence speed.
We introduce cascaded structural optimization strategy with a multi-level deformation function that captures rotation, translation, and scaling. An articulated object segmentation method is proposed for 3DGS.
We achieve highly competitive performance in novel view rendering and point-tracking. Our method also shows the ability of high-quality articulated object segmentation.
2 Related Works
Static Niwe Synthesis has become popular in 3D vision in recent years. Specifically, given a set of images from different camera poses, high-fidelity rendered images on novel views are expected. The potential of achieving photorealistic results on this task is revealed by Neural Radiance Field (NeRF) (Mildenhall
2

---

Published in Transactions on Machine Learning Research (06/2025)
Inputs
Cluster
Cluster
Cluster
Cluster
K Layers:
Frame F. --
Image
Frame t 1
D₁
D2
D3
1
11
11
D D1 D2
D3
DK
Ground Truth Image
Rendered Image
Perform Deformation
Frame
Gradient Flow
Figure Our method first utilizes the Gaussians from the previous frame I and the new inputs for frame to learn the deformation D between these two frames. These Gaussians are organized into cascaded clusters with K layers. For each cluster layer, we learn a deformation function. Finally, the deformation D of each Gaussian is obtained by nesting these deformation functions.
et al., 2020), which encodes the scene as a fully connected deep network. Following this, a series of works are proposed improve the efficiency, rendering quality, storage consumption, and other aspects of NeRF (Chen et al., 2022; 2023a; Fridovich-Keil et al., 2022; Hu et al., 2023; Müller et al., 2022; Chen al., 2023b; Garbin et al., 2021; Hedman et al., 2021; Reiser et al., 2023; Wizadwongsa et al., 2021; Chen et al., 2021; Niemeyer al., 2022; Wynn & Turmukhanbetor, 2023; Yu et al., 2021). However, the design of costly volume rendering and neural networks makes the improvements very challenging, especially in balancing the time efficiency and rendering quality. Recently, 3D Gaussian Splatting (Kerbl et al., 2023) is proposed to elegantly solve this problem by explicit 3D Gaussian representation and differentiable rasterization.
Our work is highly inspired by this but extends from static scenes to dynamic scenes. In particular, we start from static 3D Gaussians and optimize towards the dynamic scene. The natural representation of 3D Gaussians allows for explicit modeling of deformation and high efficiency for both training and inference.
Dynamic Novel-View Synthesis is a more challenging task in dynamic scenes. Inspired by the success of NoRF (Mildenhall et al., 2020), various attempts have been made to model the dynamics (Attal et al., 2023; Cao & Johnson, 2023; Fang et al., 2022; Li et al., 2022b;e; 2021; 2023; Park et al., 2021a;b; Pumarola et al., 2021; Fridovich-Keil et al., 2023; Yang et al., 2022; Weng et al., 2022). These works solve the dynamic problem by different routes. Specifically, some works (Li et al., 2022b; Weng et al., 2022; Yang et al., 2022: Zhao et al., 2022) focus certain scenarios like human motion and leverage prior knowledge, such as human skeletons, to facilitate the synthesis. While achieving impressive results, the modeling strategy cannot be applied to general cases. Deformation-base based methods (Attal et al., 2023; Park et al., 2021a;b; Pumarola et al., 2021) build a canonical stage and warp the other frames to this stage. This approach can be applied to more general scenes but can't work well on complex scenes with high variations. Impressed by the high rendering speed of 3DGS (Kerbl et al., 2023), many recent works focus on dynamic scenes with the idea of 3DGS (Wu et al., 2024; Luiten et al., 2024; Yang et al., 2024; Duan et al., 2024; Sun et al., 2024). Dynamic3DGS (Luiten et al., 2024) optimize the attributes of existing Gaussians to deal with new frames and perform tracking. 4DGS (Wu et al., 2024) build multi-resolution voxel planet to compute voxel feature with timesteps. Realtime4DGS (Yang et al., 2024) build a 4D Gaussian structure and condition it to 3D Gaussian with a given timestep. 3DGStream (Sun el al., 2024) focuses on online training and builds a transformation cache for optimization. However, despite being an online method, 3DGStream continuously pruncs Gaussians during training, making impossible to perform 3D point tracking across all time frames. While all these methods benefit from the efficiency of differentiable rasterization, they fail leverage the internal structural information of the real world and still suffer from notable training time. SC-GS (Huang
3

---

Published in Transactions on Machine Learning Research (06/2025)
et al., 2024) utilizes control points to compress the motion information of Gaussians, but struggles to achieve accurate per-point tracking and highly relies on the distribution ion of control points.
Our method is mainly inspired by Dynamic3DGS (Luiten et al., 2024) and focuses on the online dynamic scenes (Sun et al., 2024; Li et al., 2022a; Wang et al., 2023; Song et al., 2023), where the method must continually deal with new incoming frames. To make online training much more efficient, we propose a multi-level structure for 3D Gaussians with a new deformation optimization strategy. In addition, our explicit deformation format allows for broad applications like part segmentation.
Compared to Dynamic3DGS (Luiten et al., 2024), our method introduces a key change in deformation modeling: we replace Gausiian updates with a coarse-to-fine, multi-layer deformation structure based on clustering. This structural design brings two main advantages. First, by grouping Gaussians with similar motion patterns, the optimization can move larger structures jointly, leading to significantly faster convergence. Second, the multi-layer hicrarchy enables refinement at different resolutions: the coarsest layers optimize group-level transformations, while the finest layer relains full per- Gaussinn parameter updates. As result, method maintains the same level of gramularity as Dynamic3DGS (Luiten et al., 2024), without sacrificing the ability to represent details.
Dynamic Novel- View Synthesis Datasets for online methods must provide multi-view inputs for each frame. As opposed to offline methods, online methods can only reconstruct one timestep of the scene at a time, with each timestep being initialized using the previous timestep's representation. Therefore, datasets commonly used in offiin dynamic synthesis, such as Pumarola et al. (2021) and Park et al. (2021b), cannot be applied in our case. Moreover, our multi-layer, coarse-to-fine design offers more efficient way to model dynamic Gaussians. It significantly accelerates the convergence during training while preserving the ability to model detailed deformations. Dalasets such as Li et al. (2022c) and Broxton et al. (2020), although appearing complex, involve only small-scale movements, As result, they are not suitable for evaluating our method's capability to model Gaussian dynamics. In the end, we selected accelerated versions of datasets Abou-Chakra et al. (2024) and Luiten et al. (2024) for testing, which meet the aforementioned requirements. For further details, please refer to Sec 4.1.
3 Method
Overview. In this section, we present the implementation details of our proposed structural cascaded optimization approach. As outlined in the Introduction, our study aims to answer the question: "Can 4D scenes directly benefit from the structural information inherent in vanilla 3DGS?" and to provide insights that could inspire online applications. Offline methods, while effective for high-quality reconstructions, usually do not have straightforward modeling of explicit point-level information. They also lack the potential for online roconstructions. Therefore, we build upon the online method Dynamic3DGS (Luiten et al., 2024) as our codebase and integrate our cascaded optimization approach where each frame's Gaussian outpuls are generated solely based on the state of Gaussians from the previous frame and the 2D image inputs from the current frame.
For the complete Dynamic Gaussian Splatting training task, we first perform static scene reconstruction based on the initial frame observations, following the standard 3D Gaussian Splatting (Kerbl et al., 2023) procedure. Given multi-view observations of static scene (Jo.1, (o,,,, Jo.2,..., !o.v) and their corresponding camera poses (C1, Cg.... .,, CN); we train a module O, that represents the parameters of all Gaussians. This module allows us to generate predicted image f for any input camera pose C, such that i = Oo(C).
Based on this, we can proceed with subsequent online dynamic scene reconstruction. To be more specific, we use Sp, Si,..., Sr to represent the dynamic scene from time frame 0 to time frame T. For each time frame f, we have sequence of images Ir,1, I..,..., Ju, from the cameras. Our goal is to train a representation @ that can fit the scenes Sy, S1....,Sy Given an arbitrary camera C at time frame i, we can predict the image as f = (C))
Throughout this process, we introduce a structural cascaded optimization approach that organizes the Gaus- sians in a coarso-to-fine manner, significantly reducing the number of training iterations required between consecutive frames. In the following sections, we provide a detailed explanation of each step in this approach.
4

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 3: Illustration of the deformation function parameters: Rotation (second), translation (third), and scaling (fourth) applied cluster. Initial state (first).
3.1 Preliminary
In this section, we outline the key concepts and steps involved learning the dynamic scene representation @ for sequence of dynamic scenes So, .1, Sr. For online dynamic scene reconstruction, our met focuses predicting the deformation between two consecutive frames using the reconstruction results from the previous frame and the current frame's input observations.
Assuming that the Gaussians of frame I - have been reconstructed, we need to predict the deformation for frame and obtain the scene representation θ for it. To be concrete, we want to predict the deformation D, that satisfies the following equation:
θ De(Or-1).
(1)
Thus, for any given t and C, we have
=) Di
(2)
Consider the changes of single Gaussian 9i in the scene at time frame t during deformation. Recall that its representation is defined as
( = Px.
-x.- ET. PP.)..).
(3)
This is a probability density function of the position in which Pt,: the centroid position, and En, the covariance matrix. In the deformation process D,, we assume that the corresponding deformation function of position is b which satisfies
4 4. Ptti
(4)
According the derivation Xie et al. (2024), the elormed centroid position P, and the covariance matrix E, as follows:
Pt, (P E,, = Vp....
(5)
This means that if we can learn the deformation function 2, of the scene, we can use Eq. (5) directly to update the parameters all the Gaussians. Thus, our task is transformed into learning d,, which will be discussed in the following sections.
3.2 Single-layer Deformation Function
The deformation function & can be a complicated non-linear one for the entire scene, making hard for us directly learn it. An intuitive idca is that we can cluster points that are close in space and make an approximation that all the Gaussians within one cluster follow the same deformation function, then the difficulty of learning the deformation function we can make the learning process more efficient than learning it for each Gaussian independently. Which will be revealed in the experiment results. Furthermore, the deformation function within one cluster can be constructed using deformations such as rolation, translation, and scaling, making it possible to parameterize &, in an explicit form.
whole will be reduced. Also, with this clustering structure,
The intuition that one small chunk of the object is nearly rigid, thus its movement can be represented by transformation and rotation around its centroid. Also, to increase the flexibility, add scaling
5

---

Published in Transactions on Machine Learning Research (06/2025)
factor. The deformation function D within one cluster j can be represented as
Zd = (R pJ) + tz) (tanh (e. (a - pÿ) + 83) + +1,
(6)
where a is the position of the point, Ia is the corresponding position after deformation, Pj is the centroid of the cluster, R₁ is the rotation matrix (stored as a quaternion to ensure that it represents a rotation), ty is the translation vector, and (tanh (c, (x - pÿ) + 53) + 1) as a whole is the scaling factor. The inner term cf (z= - pf) + s, represents a simple linear mapping from the local coordinate to a scalar, with ey E R controlling the direction and sensitivity of the scaling, and 8y E R acting as a bias. This design follows the standard form of a lincar transformation and introduces no special assumptions, while allowing cach cluster to flexibly learn a position-dependent scaling. The tanh design ensures the scaling factor remains in the range (0, preventing potential NaN problems during training. Additionally, the scaling factor is flexible, trainable linear function of (æ pÿ), with Cj and 8j as its parameters, allowing for adaptable scaling within single cluster. Fig. 3 illustrates the specific meaning of each parameter. In summary, to represent the deformation function 4 within cluster we need to learn trainable parameters Rj, fj. Cj and 85.
3.3 Cascaded Structural Optimization Strategy
The previously discussed content addresses the deformation formulation problem within a single-layer cluster. However, there is trade-off: if clusters are too small, they provide limited accolcration, while if they are too large, they group too many Gaussians, reducing the ability to capture detailed deformations. To address this, we use coarse-to-fne multi-layer cluster structure, starting with K-means clustering based centroids. Subsequently, by merging neighboring clusters, acquire a coarser layer of clusters. This process is iteratively repeated until we obtain the coarsest layer of clusters. Specifically, suppose a point P:-1 belongs to clusters J1, J2,..,,js jx at each layer respectively (K is the number of layers), and the deformation function within cluster j& is Φk.j.- Then, for the point Pi-1 at the t 1-th frame
Pt ΦK.jK (... (02.s (1h (P:-1)).....)
(7)
In our implementation, K ==3 The cascaded optimization framework shown in Fig. 2.
At the coarsest level, clusters are expected to make broad approximations of the scene's deformation. While this coarse clustering might not always align perfectly with the underlying rigid parts, the purpose is to rapidly bring the Gaussians closer to an optimal solution. Fine-level clusters, operating at higher resolutions, can then start optimization from an improved baseline, requiring fewer iterations to refine the deformation. This hierarchical approach reduces training cost while retaining the ability to express detailed motion.
To further enhance the fine-tuning capability of cach Gaussian, we introduce throe additional parameters for each Gaussian, which are Ap, ARt and As, corresponding to delta in centroids positions, rotations, and scalings. These delta values are applied to the Gaussians after they have been deformed by the deformation function.
3.4 Optimization Process and Training Strategies
Optimization Pipeline. Based on the previously discussed deformation process, we present our complete optimization pipeline for learning deformations, which consists of two main stages. In the initialization stage, we first train Gaussians on the static scene using observations from the initial frame, followed by a coarse-to-fine clustering of Gaussian centroids. This clustering generally only needs to be done once but can be updated mid-training if there are significant scene changes. In the training stage, for each subsequent frame, we refine the deformation parameters by combining the current input images with the Gaussians from the previous frame. Through backpropagation of 2D loss, we iteratively update the deformation parameters, using the previous frame's parameters as a starting point to effectively capture gradual changes over time.
Loss Functions for Deformation. In addition to the 2D image losses used in most Gaussian Splatting methods, following Luiten et al. (2024), we also use local-rigidity loss, isometry loss, and rotation loss restrict the movement of Gaussians in large regions of the same color. Furthermore, we add scale loss T.scale
6

---

Published in Transactions on Machine Learning Research (06/2025)
Metcia
Metlaol
Robor
Spting
Wheel
FustPritick
l'ecchlumss
Closh
Backethal
Boses
Puncotic
Jasgic
Softball
Trunis
Octiaaco
20.46
30.25
27.05
30.6
27.67
31.68
50:25
29.46
30:43
31.12
31,022
30.21
PSNR
Dy sn (Lniton et al, 2034)
21.26
23:00
26.14
21.96
25.41
21.44
20.45
70.20
50.06
30.56
30.64
20.77
MynamiidD)Ss5 (Laiitm al., 2224)
50.25
$0.88
26.59
$1.25
29.50
32.91
30.01
29.29
30.4
31.04
30.88
30.11
0.97
SSTMP
DDGr un (Tniton al, 2021)
0.90
0:98
0.89
0.94
0.92
0.92
0.92
0:85
0.58
0.94
0.96
0.91
DymamieDDSS So (Laitom el ai., 2024)
0.97
0.97
0.94
0.97
0.97
0.98
0.92
0.93
0.95
0.94
0.94
0.94
Outree
0.09
0.04
0.07
0.06
0.10
0.06
0.21
0.20
0.20
0.20
0.20
0.19
LPISS
Dynamie GSp (Luiten DruomicaDSSaee (Lesten 2024)
0.08
0123
0.31
0.04
0.09
0.05
0.22
Table 1: Online methods rendering results on the FastParticle and Panoptic datascts. Values represent mcan
metrics across all lesting views. Top-2 methods are bolded.
Metrics
Method
PastiParticle Peacitlunss
Puncpli
Robor
Spting
Wiasll
Robot 1994
Closa
BoskeTban
Doaes
Football
Toggl
Scotialll
Tenmi
30.28
27.315
$11.60
27.07
31.08
30.225
25.46
50.47
31.12
31.02
30.21
DymamicDDSS (Lailoon et ul., 2024)
21.28
23.06
26.24
24.08
21.44
22.2
29.77
PSNRT
Bea/TimneaDee (Yong et al., 2024)
IDGS (Wa es al., 2024;
25.86
25.07
22.54
21.90
20.56
26.56
27:35
25.25
28.00
24.72
27.50
22.36
25.51
20.25
28.02
27.50
27.08
25.48
25.10
27.63
20.73
25,01
27.
27.00
C550n-pretraniing (Tuong
2004)
15.76
17 08
17.90
10:42
14.58
10.72
21.43
23.66
20.87
21.05
21.10
SCC-CSipretssiinng [Husng
0.96
0.39
0.35
0.97
0.38
DymamiciDG (Luiloon et nl., 2024)
0.901
0.91
11.89
0.554
11.92
0.92
11.92
0.93
0.55
0.94
0.94
0.94
88111
Boa.TimesDDS (Yang et al., 2024)
4DGS (Wa 2024;
0.93
0.58
0.50
0.91
11,89
0.91
0.50
0.95
1.53
0.91
0.95
0.87
11.99
0.92
0.91
0.52
0,91
11.92
SC-G5/ne pretrninnet)) [Monng
2004)
9C-CS8ppetssiinng (Howng et 2024)
0.90
0.95
0.87
0.95
0.94
0.91
0.68
0.00
0.7)
0.71
0.70
Chax
0.09
0.04
0.07
000
0.10
0.000
0.21
0.20
0.33
0.20
0.20
0.19
DynamicIDGS (Lailen et sl. 2324)
0.15
0.08
11.11
0.001
1..38
0.31
11.22
6.21
01:21
0.20
0.21
11.21
LPIPS]
BoaTiimeDDDS (Yang
0.11
0.33
01.35
al., 2024;
0.08
0.00
0.52
SC-G5(ne pretreiming) (Tuang es al., 2024)
0.27
0.17
0.15
0.21
0.20
0.25
0.64
0.64
0.43
0.42
0.41
0.42
SC-CSiiretsiiiing [Huong et al., 2024;
0.12
0.04
0.12
0.04
0.07
0.07
0.45
0:45
0.44
0.41
0.41
0.42
Table 2: General dynamic methods rendering results on the FastiPrt and Panoptic datasets. Values
represent mcan metrics across all testing views. The best method is bolded.
Mutrics
Metheel
Robat
Spring
Wheel
Fast Pastic.e
Poncholums
Robat-Tuk
Cloch
Badbetball
Fonthel
Panoptic
Juggle
Softiel
Tennis
20 MTRE
De (Latten ct al., 2024)
Oumine
0.80%
7.84%
0.17%
2.30%
14.38%
18.50%
0.50%
S518
0.82%
4,425
0.24%
2:35%
0.37%
15.80%
0.22%
4.98%
7.64%
9.20%
8.15%
12.42%
0.39%
16.45%
25.10%
1.72%
Table 3: 2D bracking results on the FastParticle and Panoptic datasets. Values represent mean metrics across all testing trajectories. The best method is bolded.
to prevent the generation of Gaussians that are excessively large or elongated, helping to reduce artifacts during the deformation process. The explicit form of the scale loss is
1 N N
I seale
Σ max(00,caalc, max_ scc_ee
(8)
where scale,, is the scaling vector of Gaussian i at time 1, and max scale is hyper-parameter. We use fixed empirical weights of [0.19, 0.10, 0.19, 0.48, 0.05] for rigidity, isometry, rotation, scale, and RGB losses, respectively.
Training Strategy and Appearance Modeling. After the first round of the static stage, we fix the opacity and background logit of the Gaussians. For better rendering results, we make the color trainable, allowing it to better adapt to different lighting conditions. Specifically, in terms of appearance modeling, we follow the approach of Dynamic3DGS (Luiten et al., 2024), assigning each Gaussian trainable 3D RGB vector instcad of using spherical harmonics (SH). Additionally, we add soft RGB loss to constrain the changes in color. Regarding the coarse-lo-fin clustering, in our experiments, we use a structure with K = 3, where the clusters in the coarser layer are obtained by merging clusters from the finer layer. The finest layer clusters are obtained using KMcans of Gaussian centroid positions. The merging method involves calculating the average centroid position of Gaussians in each cluster, and then performing Agglomerative Clustering based this. The final numbers of clusters at each layer are 320, 1280.
7

---

Published in Transactions on Machine Learning Research (06/2025)
Dysami3DDS
Uur
Ground Truth
Clush
Rodoot
Footboll
Figure 4: Visual comparison of rendering results Partice after 100 iterations per frame training.
Figure 5: Articulated objects segmenta- tion results.
4 Experiments
4.1 Dataset Preparation
We conduct our experiments on two datasets: the Panoptic dataset Joo et al. (2015; 2019), which includes six real-world dynamic scenes (Basketball, Boxes, Football, Juggle, Softball, and Tennis). and the syn- thetic PartPcele dataset (Abou-Chakra et al., 2024), containing six highly dynamic scenes (Robol, Spring, Wheel, Pendulums, Robot- Task, and Cloth). As mentioned in Sec. 2, we deliberately chose these datasets with challenging motion patterns to evaluate our method's ability to quickly converge Gaussians a very short training period. The large motion between frames in these datasets increases the difficulty of rapid convergence, making them ideal for testing the robust of our approach. To further amplify this challenge, accelerated the motion in the FastParticle dataset. Additional details are available in the appendix.
4.2 Comparisons
In this section, demonstrate the effectiveness of our proposed cascaded optimization approach in signifi- cantly reducing the nunber of training ilerations required. This is achieved by comparing our method with four state-of-the-art dynamic Gaussian Splatting methods on view-syuthesis tasks. These methods include Dynamic3DGS (Luiten et al., 2024), RealTime4DGS (Yang et al., 2024), 4DGS (Wu et al., 2024), and SC- GS (Huang et al., 2024). Notably, Dynamic3DGS (Luiten al., 2024), which serves as our codebase, follows the same online dynamic scene reconstruction approach as ours, while the other three are offline methods.
Evaluation Metrics. We use the PSNR, SSIM, and LPIPS (Wang et al., 2004; Zhang et al., 2018). In the experiments, since training speed is greatly influenced by implementation and hardware, for a fair comparison, it is most reasonable to compare the rendering results at the same iteration number. To eliminate concerns about runtime speed. On our single NVIDIA A40 GPU, training 100 iterations takes 1-3 seconds.
Comparison with Online Methods. We conducted experiments comparing our method with Dy- namic3DGS (Luiten et al., 2024) on both the FastParticle and Panoptic datasets. The results are shown in Table 1. Since both methods follow the paradigm of first training static scene and then performing Gaussian Splatting training frame by frame, we fixed the number of training iterations between every two frames to 100 and 2000 for comparison. Here, provide the same static checkpoints for both methods for fairness. As mentioned carlier, our method significantly reduces the number of itcrations required to achieve satisfactory rendering results. From the results in the tables, it can be seen that our method achieves results comparable to Dynamic3DGS (Luiten et al., 2024) at 2000 iterations with only 100 iterations training between frames, and far surpasses Dynamic3DGS (Luiten et al., 2024) at 100 iterations. Fig. shows the rendering results of both methods 100 iterations, qualitatively demonstrating that our method can converge and achieve satisfactory visual results after being trained for only 100 iterations per time frame.
8

---

Published in Transactions on Machine Learning Research (06/2025)
weo Entangle
w Enkangle
Ground Truth
Febbtt
Figure Left: Comparing our tracking result (blue) to the ground truth (red). Right: Visualization of our tracking results.
Figure Ablation study of the entangled covariance matrix.
Pendulums
Spring
Wobot
20-
26-
26-
22-
20-
Duz
20
Oure
18-
Oun
300500
Zacatiou
Figure 8: Convergence speed comparison between our method and Dynamic3DGS (Luiten al., 2024) on the FastParticle dataset. The x-axis shows the number of training itcrations per frame, and the y-axis represents the mean PSNR across all testing views.
Comparison with Online and Offline Methods. We compared our method with the othe four methods on both datasets. For fairness, the two online methods trained 100 iterations per frame, while for the offline methods, we set their total iterations to 100 multiplied by the total number of frames. Similarly, we provided all methods with the same static scene checkpoints for fair comparisons. SC-GS (Huang et al., 2024) is a special case because control points, and the second stage begins the actual rendering training. Therefore, we provide pretraining refers to the scenario where SC-GS undergoes 10,000 iterations to establish control points before continuing with the same number of iterations pretraining refers to the case where we skip the additional 10,000 iterations and directly start the rendering training. The results are shown in Table 2. As can be seen from the table, our method achieved the best results across both datasets.
involves two stages: the first stage requires 10,000 iterations solely to establish
metrics:
method, effectively adding 10,000 extra iterations. No-
Point-Trackinn Additionally, we evaluated our mothod's point-tracking capability. Due to the challenge of obtaining 3D ground-truth tracking labels, manually annotated keypoints for all frames from selected camera view in each scene, using these as ground-truth data. Details of the annotation process are provided in the appendix. For tracking, projected all Gaussian centroids in each frame onto the camera plane to obtain predicted 2D points. We then selected candidate points within 10 pixels of the ground-truth 2D
Partiele
Metrics
Method
Robot
Spring
Wheel
Penduluus
Robot-Task
Clotb
PSNR1
Ours, K-3
29.46
30.28
27.95
30.60
27.67
31.68
Outs, K=1
24.56
26.16
25. 12
25.94
24.73
27.77
Ours, K-3
0.96
0.97
0.94
0.97
0.95
0.97
SSIM1
Ours, K=1
0.94
0.95
0.91
0.95
0.94
0.96
LPIPSI
Outs, K=3
0.09
0.04
0.07
0.06
0.10
0.06
Ours, K=]
0.12
0.07
0.10
0.08
0.112
0.07
Table 4: Ablation study for the number of cluster layers.
9

---

Published in Transactions on Machine Learning Research (06/2025)
keypoint from the first frame, choosing the one with the highest metric value as the final tracked point. This step was necessary because the candidates corresponded to different depths, and the 2D ground-truth coordinates alone were insufficient for determining which point to track. The candidate with the best metric was considered the 3D-consistently aligned point. We used the 2D Median Trajectory Error (MTE) as the metric, following Dynamic3DGS (Luiten et al., 2024). In Table 3, we report the normalized MTE, which is the pixel error normalized by the image diagonal length, along with visualizations of the tracking results in Fig. 6. We compare our method against Dynamic3DGS (Luiten et al., 2024), selected for its superior rendering performance and as the only baseline aligning with our settings. Our tracking outcomes significantly outperform the baseline across all scenes with the same number of training iterations. Notably, the "Wheel" scene exhibits a high 2D MTE due to the object's ' strong symmetry, leading to ambiguity in its rotational trajectory (see the appendix for the scene image).
4.3 Visualization of Part Segmentation
Next, we demonstrate another application of the learned deformation information: performing segmentation on articulated objects without any semantic knowledge. Many objects in daily life, although not rigid as a whole, are composed of many rigid parts. As humans, we can distinguish these parts by watching a dynamic video and observing their motions. Similarly, our study shows how we can segment different parts of an object based solely on their deformation information, without requiring any semantic knowledge.
After training, we can obtain the centroid positions and rotations of Gaussians at each time frame. We then use KMcans clustering to group Gaussians into different clusters based on this information. Specifically, for a given Gaussian i, we use the notations Pt., and RS, to represent its position and rotation matrix at time frame 1, respectively. The KMeans feature for each Gaussian is a tensor of shape [T, 15]. where T is the total number of time frames. This tensor is the concatenation of Pi.:: flattened Ra,,, and P:o repeated T times. Additionally, we multiply three hyperparameters: Ap, AR. and Xp, to these three parts before concatenation, respectively, balance their importance.
The intuition behind the KMeans design is that, (1) Gaussians belong to the same part of the object should be close to each other at all times, and (2) the rotations of Gaussians within the same rigid part should be the same.
As illustrated in Fig. 5, we present our segmentation results on the Panoptic and FastParticle datasets. To enhance visualization, we assign different colors to Gaussians belonging to distinct categories before rendering the final outcomes. Notably, our simple K-means algorithm yields highly intuitive segmentation results, regardless of whether the scene comprises synthetic objects (left) intricate real-world environments (right). This observation serves indirect evidence that the deformation information captured by our learned Gaussians closely aligns with the actual motion of objects in dynamic scenes.
4.4 Ablation Study
In this section, we conduct ablation studies to analyze the effectiveness of our method. We first analyze the convergence speed of our method and compare it with Dynamic3DGS (Luiten et al., 2024). Then, we study the inffuence of the number of clustering layers on the rendering results. Finally, we analyze the effect of the entangled covariance matrix on the rendering results.
Analyse of training iterations To compare the convergence speed of our method and Dy- namic3DGS (Luiten et al., 2024), we trained both methods for different iterations and evaluated their rendering results at these iterations. We trained both methods on the FastParticle dataset. We show the results in Fig. 8, where the x-axis represents the number of training iterations between every two time frames, and the y-axis represents the mean PSNR among all of the lesting views. It can be observed that our method converges much faster than Dynamic3DGS (Luiten et al., 2024), consistently outperforming Dynamic3DGS (Luiten et al., 2024) at the same number of iterations. After 2000 itcrations, both methods converge at the same point, which also confirms that our method is very close to convergence after training for just 100 iterations.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Number of Cluster Layers In our cascaded optimization framework design, we choose the mumber of layers K to be 3. Here, we conduct experiments to analyze the influence of K on the rendering results, and also to validate the effectiveness of our multi-layer clustering design. We conduct our experiments on the FastParticle datasct, and the results are shown in Table 4. It can be seen that the results of our method with =3 are much better than those with K = 1 across all metrics and scenes, revealing that. the coarse-to-fine structure can significantly reduce the number of training iterations, validating the intuition that moving large clusters of Gaussians at once can more quickly find suitable positions, thereby reducing unnecessary adjustments of Gaussian positions. We also test larger values of K (e.g., 4 and 5), and observe further improvements. In fact, increasing K introduces more parameters to optimize, which may slow convergence. Therefore, K = 3 strikes a good balance between efficiency and effectiveness. Detailed results are provided in the appendix.
Entangled Covariance Matrix As shown in Eq. (5), our method, our Gaussians' covariance matrices are not separately learned. Instead, they are coupled with the deformation of centroid positions, following the strategy introduced by PhysGS Xie et al. (2024). This makes it easier for our method to learn the correct rotations and scaling of the Gaussians. In Fig. 7, we show comparison between learning the covariance matrix separately from positions and our full implementation.
Using the wheel as an example, it should rotate around its own center. As shown in the left, without coupling, although the positions of the Gaussians are learned correctly, the Gaussians themselves do not rotate accordingly with the wheel, resulting in suboptimal final rendering. long as the deformation function is learned correctly, the rotations and scalings of the Gaussians are naturally adjusted accordingly, preventing artifacts where the Gaussians are incorrectly oriented.
our full implementation as
5 Conclusion
In this paper, we presented SCas4D a cascaded optimization framework for dynamic scene modcling, focusing on efficient tracking and novel-view synthesis. By leveraging hierarchical deformation patterns, our method refines adjustments from part-level to point-level, achieving convergence within 100 iterations while maintaining competitive quality with only 1/20th of the training iterations. We also demonstrated its effectiveness in self-supervised articulated object segmentation. Extensive experiments highlight its strong performance in novel view synthesis and dense point tracking, with potential for further improvement through other 3D Gaussian Splatting variants.
Acknowledgments
We thank Pavel Tokmakov for valuable discussions. This work was supported in part by NSF Grant 2106825, NIFA Award 2020-67021-32799, the Toyota Research Institute, Amazon, the IBM-Ilinnis Discovery Accel- crator Institute, and Snap Inc. This work used computational resources, including the NCSA Delta and DeltaAI supercomputers through allocations CIS230012 and CIS240370 from the Advanced Cyberinfras- tructure Coordination Ecosystem: Services & Support (ACCESS) program, as well the TACC Frontera supercomputer and Amazon Web Services (AWS) through the National Artificial Intelligence Research Re- source (NAIRR) Pilot.
References
Jad Abou-Chakra, Feras Dayoub, and Niko Sünderhauf. ParticleNerf: A particle-basod encoding for online neural radiance fields. In WACV, 2024.
Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, and Changil Kim. HyperReel: High-fidelity 6-dof video with ray-conditioned sampling. In CVPR, 2023.
Michael Broxton, John Flynn, Ryan Overbeek, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. Immersive light field video with a layered mesh representation. In SIGGRAPH, 2020.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In CVPR, 2023.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields.
ECCV, 2022.
Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, and Andreas Geiger. Dictionary fields: Learning a neural basis decomposition. ACM Transactions on Graphics (TOG), 42(4:11-12, 2023a.
Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. MobileNeRF: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In CVPR, 2023b.
Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, and Baoquan Chen. 4D Gaussian Splatting: Towards efficient novel view synthesis for dynamic scenes. arXiv preprint Xiu:2402.03307,, 2024.
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia, 2022.
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks In CVPR, 2022.
Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
Stephan Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. FastNeRF: High-fidelity neural rendering at 200fps. In ICCV, 2021.
Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In 1CCV, 2021.
Wenbo HU, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. TriMippR:: Tri-Mip representation for efficient anti-aliasing neural radiance fields. In ICCV, 2023.
Yi-Hua Huang, Yang-Tian Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. SC-GS: Sparse- controlled Gaussian Splatting for editable dynamic scenes. In CVPR, 2024.
Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015.
Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjoe, Timothy Godisart, Bart Nabbe, lain Mauthews, et al. Panoptic studio: A massively multiview system for social interaction capture. IEFE Transactions on Pattern Analysis & Machine Intelligence, (1(0)):190-200 204, 2019.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimküüler, and George Dreitakis. 3D Gaussian Splatting for real-time radiance field rendering. ACM Transactions on Graphies, 42(4):1-14, 2023.
Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Ping Tan. Streaming radiance fields for 3D video synthesis. Advances in Neural Information Processing Systems, 35:13485-13498, 2022a.
Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free animatable volumetric actors. In ECCV, 2022b.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3D vidco synthesis from multi-view video. In CVPR, 2022c.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely, Dynibar: Neural dynamic image-based rendering. In CVPR, 2023.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva. Ramanan. Dynamic 3D Gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multircsolution hash encoding. ACM transactions on graphics (TOG), 41(4):1-15, 2022.
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Brrall. Nerfies: Deformable neural radiance fields. In ICCV, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Sleven M Seitz. HyperNeRF: A higher-dimensional representation for topo- logically varying neural radiance fields. arXiv preprint arXin:210. 13228, 2021b.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF; Neural radiance fields for dynamic scenes. In CVPR, 2021.
Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. McRF: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics (TOG), 42(4):1-12 2023.
Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. NeRFplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 29(5):2732-277 2742, 2023.
Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3DGStream: On-the-fty training of 3D gaussians for efficient streaming of reatl-realisti free-viewpoint videos. orXiv preprint 0rXiv:240300144 2024.
Liao Wang, Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, and Minye Wu. Neural residual radiance fields for streamably viee-viewpoint videos. In CVPR, 2023.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4): (3(4):600 612, 2004.
Chung- Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan Barron, and Ira Kemnelmacher-Shiieeeee Shlizerman. HumanNeRF: Freo-viewpoint rendering of moving people from monocular vidco. In CVPR, 2022.
Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. Nex: Real-time view synthesis with neural basis expansion. CVPR, 2021.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4D Gaussian Splatting for real-time dynamic scene rendering. In CVPR, 2024.
Jamie Wynn and Daniyar Turmukhambetor, Deff Regularizing neural radiance fields with denois- ing diffusion models. In CVPR, 2023.
Tianyi Xic, Zeshum Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. PhysGaussian: integraed 3D Gaussians for generative dynamics. In CVPR, 2024.
Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3D neural models from many casual videos. In CVPR, 2022.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4D Gaussian Splatting. In TCLR, 2024.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. PixeINeRF: Neural radiance fields from one few images. In CVPR, 2021.
Qiang Zhang, Seung-Hwan Baek, Szymon Rusinkiewicz, and Felix Heide. Differentiable point-based radiance fields for efficient view synthesis. In SIGGRAPH Asia, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. HumanNeRI: Efficiently generated human radiance field from sparse inputs. In CVPR, 2022.
14

---

Published in Transactions on Machine Learning Research (06/2025)
A
Appendix
In this appendix, we offer additional details regarding our PastParticle and Panoptic datasets, which provide the necessary context for experiments. We delve into our method for articulated objects segmentation, presenting full qualitative results that demonstrate its effectiveness across various scenarios. Additionally, we clarify our rationale for maintaining the same number of itcrations in our comparisons and present a comparison under equal wall-clock time, showing that our method still oulperforms Dynamic3GGS Luiten et al. (2024). We also include visualizations illustrating the multi-layer clustering structure we employ, as well as the manually annotated tracking labels usod for evaluating 2D tracking results. Furthermore, we discuss our approach to learning the deformation, emphasizing the two-phase training strategy. Finally, we reflect on limitations, identifying potential areas for future improvement.
A.1 FastParticle and Panoptic Datasets
In this section, we introduce the PastParticle and Panoptic datasets used in our experiments in details. The real-world Panoptic dataset includes six scenes: Basketball, Boxes, Football, Juggle, Softball, and Tennis. Each frame in these scenes comes with segmentation provided by the original authors. Following Luiten et al. (2024), we distinguish between foreground and background in these scones and utilize background loss and floor loss accordingly. Each scene in this dataset contains 150 frames captured by a total of 31 cameras, with 27 cameras used for training and for testing.
The synthetic FastParticle dataset, which we have accelerated, contains six dynamic scenes: Robot, Spring, Wheel, Pendulums, Robot-Task, and Cloth. After acceleration, these scenes respectively have 35, 18, 38, 24, 35, and 35 frames. As illustrated in Fig. 9, we show the dynamic evolution of some scenes, highlighting the significant changes between frames. This dataset includes 40 cameras in total, from which we randomly select as testing cameras and the remaining 36 as training cameras.
For all experiments, we provide the same static chockpoints to all basclines. For the 12 scenes across the two datasets, we train for 20,000 iterations to obtain the checkpoints. Due to the varying complexity of the static scenes, 3,000 iterations are sufficient for most FastParticle scenes.
A.2 Articulated Objects Segmentation
As mentioned in Sec. 5.1. The intuition behind the KMeans design is that, (1) Gaussians belong to the same part of the object should be close to each other at all time, and (2) the rotations of Gaussians within the same rigid part should be the same. The first one can be trivial, here we provide more explanations about the second point. As shown in Fig. 10, suppose we have a rigid body with its controid denoted as Co. This rigid body can be considered as a combination of two smaller rigid bodies, with their centroids denoted as C1 and C2, respectively. After rotation, C1 and C2 move to Cf and Cg. Taking Co as the origin of the coordinate system, the movement of the rigid body can only be a rotation I around Co, and the two smaller rigid bodies move accordingly. When considering the left smaller rigid body alone, its motion should consist of translation of its centroid Cj and a rotation Ri around C1. We aim to prove that Rí R. Therefore, consider a point P on the left rigid body, which moves to point Pr after the movement. From the perspective of Co. we have
c.P' C.P-= - R R.P.
(9)
Also, from the perspective of C1, we can have
CoP' = R.P.P Cock
(10)
= RCCP - R CoC,
Therefore, we have
R RCP Rí C.P.
(11)

---

Published in Transactions on Machine Learning Research (06/2025)
Robot
Cloth
Robot- Task
Figure 9: This figure shows the evolution of three scenes from the FastParticle dataset, demonstrating the high dynamic characteristics of the accelerated dataset.
R
P
P'
Cz
C
Co
Figure 10: Illustration of a rigid body rotating R around its centroid. When considering the rigid body composed of two smaller rigid bodies, can be shown that the rotation of each smaller rigid body around its own centroid is the same with R.
Since the choice of P is arbitrary, can conclude that Rı = R. Similarly, we can prove that the rotation of the smaller rigid body on the right also R. The above demonstrates the case where the rigid body is divided into two parts. This conclusion can be generalized to any case of multiple divisions, meaning that all parts of the same rigid body have the same rotation. Returning to our problem, since the rotation of Gaussians is around their centroids, the Gaussians belonging the same rigid body should have the same rotation.
2

---

Published in Transactions on Machine Learning Research (06/2025)
Dynamic3DGS
Ours
Ground Truth
Cloth
Robot
Robot-Tase Task
Spring
Pendulums
Wheel
Figure 11: Qualitative results on FastParticle
3

---

Published in Transactions on Machine Learning Research (06/2025)
Dynamic3DGS DGS
Ours
Ground Truth
Football
Basketball
Juggle
Boxes
Softball
Tennis
Figure 12: Qualitative results on Panoptic

---

Published in Transactions on Machine Learning Research (06/2025)
Pasticil
Metrics
Method
Robot
Spring
Wheel
Pendulums
Robot- Task
Cloth
Ours100
29.46
30.28
27.95
30.60
27.67
31.68
PSNRÍ
Dynamic3DGS 300 Luiten et al. (2024)
27.66
27.16
26.67
29.57
26.79
30.41
Ours100
0.96
0.97
0,94
0.97
0.95
0.97
SSIMT
Dynamic3DGSSgm Luiten et al. (2024)
0.95
0.95
0.93
0.96
0.95
0.97
Ours100
0.09
0.04
0.07
0.06
0.10
0.06
LPIPS)
Dyuamic3DGSgnn Luiten et al. (2021)
0.10
0.06
0.08
0.06
0.10
0.07
Table 5: Comparison of our method trained with 100 iterations per time frame against Dynamic3DGS.
Particle
Method
Robot
Spring
Wheel
Pendulums
Robot-Task
Cloth
2
29.48
30.40
27.86
30.54
27.35
31.51
K 44
29.39
30.09
27.87
30.38
27.55
31.48
5
29.17
30.00
27.79
30.36
27.60
31.43
Table 6: More ablation study results for the number of cluster layers. The reported metric is PSNR.
A.3
Full Qualitative Results
In this section, we provide qualitative results on all 12 scenes from the two datasets. As shown in Fig. 11 and Fig. 12, both our method and Luiten et al. (2024) are trained 100 iterations between two consecutive frames.
A.4 Same Wall-clock Time Comparisons
In our experiments, we use the same number of iterations across different methods for consistency. While wall-clock time may vary depending on the specific implementation (e.g,, whether CUDA acceleration is employed), the number of iterations reflects the convergence speed of the algorithms. A lower number of iterations indicates faster convergence, showing that the optimization problem is easier to solve. This practice is commonly used in the evaluation of online methods, as demonstrated in the Dynamic3DGS Luiten et al. (2024) comparison (see Table their paper), where different met are also compared using the same number of iterations.
Even when comparing with equivalent wall time, our method remains superior. To further illustrate this, provide comparison of method trained for 100 iterations per frame versus Dynamic3DGS Luiten et al. (2024) trained for 300 iterations per frame on the FastParticle dataset. The results show that our method has average training speed per iteration approximately twice as fast as Dynanic3DGS Luiten et al. (2024). As seen in Table 5, despite the difference in iteration count, our method still outperforms Dynamic3DGS Luiten et al. (2024) in terms of both efficiency and final performance.
A.5 Ablation Study Number of Cluster Layers
We test layer numbers from 1 5 on the FastParticle dataset. As shown Table 6, = performs best. We find that increasing the number of cluster layers from 3 does not bring additional performance gains. On the contrary, it introduces more parameters to optimize, which may increase the required number of iterations and lead to diminishing returns. Therefore, we conclude that K = 3 offers the best trade-off between efficiency and performance in our framework.
=
A.6 Illustration of the Lulti-Layer Structure
In Fig. 13, we show the coarso-to-fine multi-layer clustering structures for two objects in the FastParticle dataset. Different colors in the figure represent different clusters, and for clarification, the same color in different layers does not indicate any correlation between the clusters.
5

---

Published in Transactions on Machine Learning Research (06/2025)
Layer_0
Layer 1
Layer 2
Cloth
Robot
Figure 13: Coarse-to- fine multi-layer clustering structures for two objects the FastPPrticle dataset.
A.7 Tracking Labels
Here, as shown in Fig. 14, we present all manually annotated 2D tracking ground truths. Since the human eye can only track points with distinct features across multiple frames, we only selected such points for annotation.
A.8 Learning the Deformation
Algorithm 1 summarizes our training process. Initially, we train our Gaussians on the static scene using ob- servations from the first frame. Subsequently, perform multilevel coarse-to-fine clustering for the centroids of the Gaussians. For each input in every time frame, loss and subsequently update our deformation functions.
optimization approach to backpropagate
Algorithm 1: Deformation based Dynamic Scene Reconstruction Algorithm
Input: Images from all frames
θ Oprev + Initialization stage (Static Ganssian Splatting);
Do Clustering:
for t in time_ fram__frasessee do
Initialize the Deformation D; for iter in maz iters do
Ocurr D(0ppev)) Images +- Render(Qeuur).
loss Loss(gt Images, Images); Backpropagate(loss)
end
end

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 14: Illustration of our manually annotated tracking ground truths.

---

Published in Transactions on Machine Learning Research (06/2025)
Particle
Method
Robot
Spring
Wheel
Pendulums
Robot-Tast
Cloth
N=16 16
29.76
30.43
27.97
30.84
27.69
31.56
N=
29.63
30.43
27.91
30.77
27.87
31.62
48
29.49
30.36
28.00
30.70
27.62
31.61
Table 7: PSNR results when varying the number of clusters in the first. layer.
For polential negative impacls, since LayeredGS can learn deformation information and be used for creating new motions or inserting objects, such applications can be used for fake news convince people by multi-view renderings. More censorship needs to be established in such cases.
A.9 Robustness the number of clusters in the first layer
To further examine the robustness of our method to the initial clustering configuration, we conducted an additional experiment by varying the number of clusters in the first layer. The default number of clusters used in the main experiments N 64. Reducing the number of clusters leads coarscr groupings, potentially introducing more clustering errors at the coarse level. Despite this, our model achieves comparable performance across different settings, as summarized in Table These results indicate that structural cascaded optimization remains robust even when the initial clustering is coarser, thanks to the refinement provided by the multi-layer hierarchy and per- Gau-Gansiian updates at the finest level.
A.10 Effect of the Max Scale Constraint
To further examine the role of the max scale constraint in our method, we conducted an experiment by varying the value of max_scale. The scale loss is designed to prevent Gaussians from becoming excessively large during deformation, which may otherwise introduce rendering artifacts. We found that setting reasonable threshold (in our main experiments, max_scale = 0.02) is sufficient to maintain rendering quality.
Our experiment shows that disabling this constraint (i.c., setting max_scale to a very large value) leads to significant PSNR degradation, as summarized in Table These results demonstrate the importance of the max scale constraint in preserving stability during optimization.
Particle
Method
Robot
Spring
Wheel
Pendulums
Robot-Task
Cloth
max scale=0.02
29.46
30.28
27.95
30.60
27.67
31.68
max scale=2.0 2.0
15.86
19.86
22.42
18.86
18.15
21.53
Table 8: PSNR results under different values of the max scale constraint.
A.11 Additional Experiment on Per-Gaussian Deformation for Segmentation
We provide an additional experiment to examine the difference between our method and Dy- namic3DGS Luiten et al. (2024) for segmentation purposes. Specifically, we trained Dynamic3DGS Luiten et al. (2024) for 2000 iterations and extracted per-Gaussian deformation (without any layered structure) perform segmentation, similar our approach. While this method produces visually reasonable motion, we found that the Gau-Gassiian rotations significantly noisier, leading to worse segmentation quality.

---

Published in Transactions on Machine Learning Research (06/2025)
Ours
Dynamic3DGS
Figure 15: Visual comparison of segmentation results using per-Gaussian deformation (Dynamic3DGS Luiten et al. (2024)) and our multi-layer deformation structure.
In contrast, our multi-layer structure encourages Gaussians to move coherently with their parent clusters, resulting in more stable and interpretable rotations. This structural regularization improves segmentation accuracy by reducing noise in the estimated motion. Please refer to Figure 15 the updated appendix for visual comparisons.
A.12 Limitations
While our method significantly reduces training iterations to 100 per frame, achieving real-time training and rendering remains challenge. Additionally, the learned deformation information is not fully utilized, and the presented articulated object segmentation results are not well refined. Future work will focus addressing these limitations by exploring real-time training approaches, refining deformation utilization techniques, and developing more sophisticated segmentation mothods.


Reviews:
[{"review_id": "DLD3MKgk1O", "reviewer": "Reviewer_JCjr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1:**The optimization framework is efficient.** The proposed cascaded optimization strategy significantly reduces the training iterations required per frame (only 100 iterations) compared to existing methods (Dynamic3DGS, which requires around 2000 iterations), achieving comparable or superior rendering quality. \n\n2: **Utilization of Structural Information**. SCas4D leverages inherent structural patterns within dynamic scenes by hierarchically organizing 3D Gaussians into clusters. This coarse-to-fine approach effectively captures both large-scale and fine-grained deformations, resulting in improved tracking and segmentation accuracy, particularly valuable for articulated object segmentation tasks.\n\n3: **The proposed method can be applied to many tasks**. Beyond novel-view synthesis, the method demonstrates versatility through applications in dense point tracking and self-supervised articulated object segmentation.\n\nWeaknesses:\n\n1: **Dependence on Initial Clustering.** The method heavily relies on an initial K-means clustering to structure the cascaded optimization. Poor initial clustering can significantly impact subsequent deformation refinement, potentially limiting the method's performance, especially in scenarios involving complex or ambiguous motions.\n\n2: **More in-depth analysis of hyperparameters**. The approach involves multiple hyperparameters, such as the number of clustering layers (the authors only test K=1 and K=3), the maximum scale constraints, and balancing weights in loss functions.\n\n3: **Potential extension of current pipeline**. Despite achieving impressive efficiency gains, the proposed method still cannot achieve full real-time training and rendering, which remains an essential barrier for broader applications in interactive or live systems. \n\nOverall, I think this paper is solid with comprehensive experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces SCas4D, a structural cascaded optimization framework that enhances 4D novel-view synthesis and dynamic scene modeling efficiency. SCas4D leverages internal structural information in 3D Gaussian Splatting (3DGS) to significantly speed up convergence rates. It employs a cascaded, multi-level optimization strategy that hierarchically captures deformation from coarse-level to fine-grained adjustments, and introduces a self-supervised articulated object segmentation method. Experimental results demonstrate SCas4D's ability to drastically reduce training iterations while maintaining high rendering quality and superior performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More analysis of clustering on the effect of the performance and other hyperparameters.", "broader_impact_concerns": "I don't have broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2R3UuPxUDl", "reviewer": "Reviewer_WWuz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**[Strengths]**\n- **The paper is well organized, and the writing is clear.** The main contents (figures, tables, writing) were clear and self-inclusive. This reviewer could find almost all the necessary information from the manuscript. Also, most visual presentations support the claims of the paper.  \n- **Benefits of structural optimization.** The proposed structural cascaded optimization for dynamic 3DGS optimization is technically sound, and the numbers and figures clearly support it. Also, it is interesting to see that this framework can inherently help intuitive articulated object segmentation.\n- **Well-designed experiments.** The authors clearly stated the differences between the online and offline methods for the task. Accordingly, the experiments are clearly designed to make fair comparisons, especially between the proposed method and offline methods. Specifically, the comparison with SC-GS (no-pretraining), SC-GS (pretraining) is clearly explained and makes sense.\n- **Included details** This reviewer could find almost all the details and rationale for the design choices from the manuscript. As the proposed method is built on top of the previous work Dynamic3DGS, which is already open-sourced, this reviewer thinks that the proposed technique can be easily followed up by the readers.\n\n\n**[Weaknesses]**\n\nThis reviewer could not find significant weaknesses or errors in the paper. Below are some questions and minor concerns:\n\n- **Question about articulated segmentation.** If the articulated object segmentation is based on the K-means clustering of the final dynamic 3DGS parameters, this reviewer thinks it could be also applied to the competing methods. How will it work with other methods, e.g., Dynamic3DGS? Is this task possible because of the proposed structured optimization technique? It would be interesting to report the differences with and without the proposed structured optimization.\n- **Question about Fig. 8: Convergence speed comparison.** While the initial scene states are identical for both Dynamic3DGS and Ours, the PSNR plot shows a significant gap between both methods. This reviewer is aware that the plot shows from iter 100 to 2000. However, this reviewer thinks that showing the plot from iter. 0 would be more compelling as it can highlight the benefit of the proposed method. How do the PSNR plots for iter 0-100 look? Plus, on which temporal frame have the PSNRs computed? This reviewer postulates that the gap should be small for the early frames, as both methods start from the same initial scene.\n- **Question about entangled covariance matrix ablation study.** This reviewer cannot see significant visual discrepancies in Fig. 7 w/ and w/o entangle. Also, it seems like the technique to entangle the Gaussian's covariance matrix is from PhysGaussian[1], not this paper's contribution. Could the authors clarify this matter?\n- **Detailed explanation for Eq. (6).** As far as this reviewer understood, the scale part, i.e., $(\\texttt{tanh}(\\mathbf{c}^{\\top}\\_{j}(\\mathbf{x}-\\mathbf{p}\\_{j}^{c})+s_{j}) + 1)$, of the Eq. (6) allows non-linear deformation with a Gaussian cluster. However, the detailed explanation about this term is missing. Specifically, the definitions for the trainable parameters $\\mathbf{c}\\_{j}$ and $s\\_{j}$ are missing.\n- **Wrong citation.** The authors missed to cite the original reference for the Panoptic Studio dataset. It should be citing [2] and [3].\n\n\n[1] Xie et al., \"PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics,\" CVPR 2024.\n\n[2] Joo et al., \"Panoptic Studio: A Massively Multiview System for Social Motion Capture,\" ICCV 2015.\n\n[3] Joo et al., \"Panoptic Studio: A Massively Multiview System for Social Interaction Capture,\" TPAMI 2017.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a multi-layer clustering-based structured optimization for representing dynamic scenes with 3D Gaussians. The authors claim the need to leverage the assumption of locally rigid transformation for dynamic scenes. Given a 3DGS reconstruction for the initial static state (scene) and the subsequent frames' multi-view observations, the proposed system optimizes learnable 3DGS deformation parameters, including rotation, translation, and position-aware scale parameters for 3D Gaussians. Instead of modeling a dynamic scene into a single coarse rigid component, the proposed system models dynamics in a coarse-to-fine manner, where they perform K-means clustering based on the centroids and merge them into bigger clusters (3 levels in practice). \n\nIn experiments, the proposed method achieves competitive reconstruction quality compared to the baseline method (Dynamic3DGS), even with 1/20th of the training iterations. Furthermore, the authors built reasonable experimental setups to compare with some offline dynamic reconstruction methods and achieved superior reconstruction quality. Interestingly, when K-means clustering for 3DGS parameters is performed on the final dynamic scene, it can natively segment articulable parts of the dynamic objects.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "For the revised manuscript, this reviewer expects changes as follows (per the weakness comments):\n- **Discussions about the effect of the proposed structured optimization strategy on the articulated object segmentation.** Is the structured optimization a crucial requirement for achieving such articulated object segmentation? If not, how does it compare to the articulated object segmentation results that one could get from the baseline method (Dynamic3DGS)?\n- **Providing more details about the convergence speed comparison.** How does the PSNR plot look in early iterations or early time frames?\n- **Clarification about entangled covariance matrix experiment.** Provide more explicit examples to show the difference and clarify if the covariance matrix entanglement is a proposed contribution or not.\n- **Detailed explanation for Eq. (6).** Please provide more details and definitions.\n- **Fixing wrong citation.** Please add the right citation for the dataset.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lmvP60gYkE", "reviewer": "Reviewer_QwXg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n- This paper offers a novel and strong inductive bias that the movements of all points in a rigid body can be describled as a single-layer deformation function.\n\n- This paper proposes a simple yet effective approach to leverage the insights to greatly enhance the training efficiency while maintaining quality.\n\n- The empirical evaluations significatly outperform baselines, maintaining onpar performance while reducing the optimizing steps for online rendering.\n\n- This paper is well-written, clearly presenting its motivation, contribution, and significance.\n\n## Weaknesses\n\nI'am not that falimilar with this area, even not falimilar with the rendering community. So, maybe I cannot offer very valuable comments regarding the weaknesses.\n\n- In my view, one potential weakness is the introduced inductive bias only applys for rigid body, not for deformable objects. So, it would be interesting to see the following work to decrease the deformable objects rendering cost.\n\n- I notice that this paper builds on top of Dynamic3DGS, but dose not introduce the details about it. It would be good if the authors could discuss more about the differences compared to Dynamic3DGS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to balance the efficiency and quality for 4D view rendering. The core contribution is the insight that every point in a rigid body should be the same. This powerful inductive bias can greatly enhance the rendering efficiency while preseving the quality.\n\nFollowing this insight, this paper proposed a coarse-to-fine rendering method. Specifically, the coarse level rendering optimizes the whole rigid object using the same transformation. Although this can be very coarse and suboptimal, this procedure can efficiently move all optimzed gaussians to near optimal places. After this, fine level clusters opterate in higher resolutions to refine the overall rendering quality.\n\nThe evaluations in terms of rendering quality and efficiency demonstrate the effectiveness of the proposed method, showing a good balance among these two important criterias.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please discuss more about the differences to Dynamic3DGS.\n\n- Why do the K=1 results in Table 4 surpass Dynamic3DGS in Table 1? K=1 can lost most fine-grained information and the quality may degenerate severely.\n\n- What will happen if K is further increased? Will it lead to a performance degeneration? Maybe more ablation studies can be performed.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The idea of this work is interesting. Also, the contributions touch on key topics—efficient optimization, dynamic scene modeling, and self-supervision—that are squarely in TMLR’s scope.
The paper presents a solid contribution to 4D novel view synthesis with comprehensive experiments. The AE believes there are many potential applications in the presented paper.

This submission satisfies all the TMLR criteria on novelty, contribution, findings, and broad interests with high quality.
Decision: Accept (poster)