Template-free Articulated Gaussian Splatting for Real-time Reposable Dynamic View Synthesis
Authors: Diwen Wan
Yuxiang Wang
Ruijie Lu
Gang Zeng
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian Splatting
View Synthesis
Skeleton Discovery
Primary area: machine_vision
TL;DR: We propose template-free method based on 3D-GS and superpoints  to discover the skeleton model of an articulated object and achieve real-time rendering.

Abstract: While novel view synthesis for dynamic scenes has made significant progress, capturing skeleton models of objects and re-posing them remains a challenging task. To tackle this problem, in this paper, we propose a novel approach to automatically discover the associated skeleton model for dynamic objects from videos without the need for object-specific templates. Our approach utilizes 3D Gaussian Splatting and superpoints to reconstruct dynamic objects. Treating superpoints as rigid parts, we can discover the underlying skeleton model through intuitive cues and optimize it using the kinematic model.  Besides, an adaptive control strategy is applied to avoid the emergence of redundant superpoints.  Extensive experiments demonstrate the effectiveness and efficiency of our method in obtaining re-posable 3D objects. Not only can our approach achieve excellent visual fidelity, but it also allows for the real-time rendering of high-resolution images.

Template-free Articulated Gaussian Splatting for Real-time Reposable Dynamic View Synthesis
Diwen Wan'
Yuxiang Wang' Ruijie Lu' Gang Zeng'
National Key Laboratory of General Artificial Intelligence.
School of IST, Peking University, China
(wan, yuannniziang1333 pku. edu. cn
edu,
Abstract
While novel view synthesis for dynamic scenes has made significant progress, capturing skeleton models of objects and re-posing them remains challenging task. To tackle this problem, in this paper, we propose novel approach to automatically discover the associated skeleton model for dynamic objects from videos without the need for object-specific templates. Our approach utilizes 3D Gaussian Splatting and superpoints to reconstruct dynamic objects. Treating superpoints as rigid parts, we can discover the underlying skeleton model through intuitive cues and optimize it using the kinematic model. Besides, an adaptive control strategy is applied to avoid the emergence of redundant superpoints. Extensive experiments demonstrate the effectiveness and efficiency of our method in obtaining re- re-posable 3D objects. Not only our approach achieve excellent visual fidelity, but it also allows for the real-time rendering of high-resolution images. Please visit our project page for more results: /ectss///dyvi github......................... LL/SK_GS/
Introduction
Novel view synthesis for 3D scenes is important for many domains including virtua//augmentcc/miieee reality, game or movie productions. In recent years, Neural Radiance Fields (NeRF) [1] have witnessed significant advances in both static and dynamic scenes. Among them, 3D Gaussian Splatting (3D-GS) (2) proposed a novel point-basse representation, and is capable of real-time rendering while ensuring the quality of generated images, bringing insights more complex task scenarios.
Although visually compelling results and fast rendering speed have been achieved in reconstructing dynamic scene, current methods mainly focus on replaying the motion in the video, which means just renders novel view images within the given time range, making it hard to explicitly repose control the movement individual objects in the scene. For some specific categories such as the human body human head, one main approach is to leverage the category prior knowledge such as templates to support the manipulation of reconstructed objects. However, it is hard for these methods generalize to large-scale in-the-wild scenes or human made articulated objects.
Some template-fres free methods attempt to address these challenges by building reposable models from videos. Watch-It-Move (WIM) 3 leverages ellipsoids, explicit representation, to coarsely model objects, and then estimate the residual by a neural network. The underlying intuition that more ellipsoids multi-view videos, WIM can learn both the appearance and structure of articulated objects. However, the reconstruction results of WIM are of low visual quality and the training and rendering speed is slow. Apart from WIM, Articulated Point NeRF (AP-NeRF) [4] samples feature point cloud from pre-trained dynamic NeRF model Vox (5)) and initializes the skeleton tree using the medial axis transform algorithm. By combining linear blend skinning (LBS) and point-
represent functional part. By observing the motion of parts from
point-based based rendering (6),
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

AP-NeRF jointly optimizes dynamic NeRF and skeletal model from videos. Compared to WIM, AP-NeRF achieves higher visual fidelity while significantly reducing the training time. However, AP-NeRF cannot achieve real-time rendering, which is still far from practical application.
In this paper, we target class-agnostic novel view synthesis of reposable models without the need for a template or pose annotations, while achieving real-time rendering. To enable fast rendering speed, we opt to represent the 3D object as 3D Gaussian Splatting. To be specific, we first reconstruct the 3D dynamic model using 3D Gaussians and superpoints, where each superpoint binds Gaussians with similar motions together. These superpoints will later be treated as the parts of an object. Afterward, skeleton model is discovered leveraging some intuitive cues under the guidance of superpoint motions from the video. Finally, we jointly optimize the skeleton model and pose parameters to match the motions of the training videos. During the optimization process of object reconstruction, we will inevitably generate a lot of redundant superpoints to fit the complex motion. To simplify the skeleton model and avoid overfitting, we employ an adaptive control strategy and regularization losses to reduce the number of superpoints. Our contributions can be summarized as follows:
We propose a novel method based on 3D Gaussians and superpoints for modeling appearance, skeleton model, and motion of articulated dynamic objects from videos. Our approach can automatically discover the skeleton model without any category-speciife prior knowledge.
We effectively learn and control superpoints by employing an adaptive control strategy and regularization losses.
We demonstrate excellent novel view synthesis quality while achieving real-time rendering various datasets.
2 Related Works
2.1 Static and Dynamic Neural Radiance Fields
In recent years, have witnessed significant progress in the field of novel view synthesis empowered by Neural Radiance Fields. While vanilla NeRF [1] manages to synthesize photorealistic images for any viewpoint using MIPs, subsequent works have explored various representations such 41) tensors (7). hash encodings (8). or other well-designed data structures [0] to improve rendering quality and speed. More recently, novel framework 3D Gaussian Splatting (2) has received widespread attention for its ability to synthesize high-fidelity images for complex scenes in real-time.
Meanwhile, many research works challenge the hypothesis of static scene in NeRFs and attempt to synthesize novel-view images of a dynamic scene at an arbitrary time from a 2D video, which is a more challenging task since the correspondence between different frames is non-trivial One line of research works [3) directly represents the dynamic scene with an additional time dimension or a dinendependent interpolation in latent space. Another line of work [14 represents the dynamic scene as a static canonical 3D scene along with its deformation fields. While one main bottleneck of synthesizing dynamic scene is speed, some works (1) propose to extend 3D Gaussian Splatting into 4D to mitigate the problem. Though being able to recover a high-fidelity scene, this method cannot directly support editing and reposing objects within it. In this work, we leverage 3D Gaussian Splatting as the representation for faster rendering speed.
2.2 Object Reposing
It's impractical to directly repose the deformation fields of dynamic NeRFs due to the complexity of high-dimension. Therefore, utilizing parametric templates based on object priors to represent deformation is adopted in many research works. The classes of parametric templates range from
human faces [3 24]. and bodies [25] 26 to non-human objects like animals [27]. With the help of skeleton-basee baseonn--aadddeeee LBS and 3D or 2D annotations, these parametric templates capable of representing
articulate human heads (3 and bodies [34 41). Though these template- based reposing methods
synthesize high-fidelin images, they are restricted to certain object classes and mainly deal with
rigid motions, not to mention the time-consuming process of annotations.
To alleviate the excessive reliance on domain-specift skeletal models, methods based on retrieval from database (42) or adaptation from generic graph (43) 44) are adopted. However, these methods
still of relatively low flexibility and diversity. Another line of work attempts to learn a more

---

Dynamic stage
Kinematic stage
time@ &
MLP +
R
Kinematic Model
R
MLP y
time
ja
Superpoints
joints
LBS weights
Adaptiva
0 h4
superpoints
Learmeble
h:
Diff Gaussian Rasterization
Parameters
Gaussians
operations
Diecovery
Images
Skeleton Model
middle output
Figure 1: The pipeline of proposed approach. Our approach follows two-stage training strategy. In the first stage (i.e. dynamic stage), we learn the 3D Gaussians and superpoints to reconstruct the appearance. Each superpoint is associated with rigid part, and the adaptive control strategy is used to control the count. After finishing the training of dynamic stage, we can discover the skeleton model based superpoints. After finish the second stage (i.e., kinematic stage), obtain articulated model based on the kinematic model.
general template-free object representation by 3D shape recovery 45). WIM proposes to jointly learn surface representation and LBS model for articulation without any supervision prior knowledge the structure. However, the reposing images are of low visual quality and the required training time is considerably long. AP-NeRF [4) achieves much faster training speed by leveraging a baint-based NeRF representation, but cannot support real-time rendering as well.
3
Methods
Our goal is to reconstruct reposable articulated object with real-time rendering speed from videos. The pipeline of proposed method illustrated in Fig. We represent the appearance of the articulated object 3D Gaussians in the canonical space while aggregating 3D Gaussians with similar motion into superpoints, which can be treated rigid parts. It is noteworthy that we apply a time-variant 6 DoF transformation matrix to model the motion of the object. Based these superpoints, leverage several intuitive observations to guide the discovery of the skeleton model, which includes both joints and skeletons. Since the observations hold for most objects, our method does not require category-specifte template or pose annotations. To reduce the redundant superpoint, we propose an adaptive control strategy to densify, prune, and merge superpoints during the training process.
3.1 Preliminaries: 3D Gaussian Splatting
3D Gaussian Splatting (3D-GS) 2 use a set of 3D Gaussians to represent a 3D scene. Each Gaussian Gy: {14; qi, St, oi, h;) is associated with position μ4; rotation matrix R, which is parameterized by a quaternion opacity σ; and spherical harmonics (SH) coefficients h, Therefore, the anisotropic 3D covariance matrix of Gaussian G; is defined as Σ; R,S.S, R,T. which is positive semi-definite matrix.
a scaling matrix S, which is parameterized by 3D vector 8,,
To render images, 3D-GS employs EWA Splatting algorithm [46] to project 3D Gaussian with center μ4 and covariance Σ, to 2D image space, and the projection can be approximated as a 2D Gaussian with center μ³, and covariance E, Let Q. K be the viewing transformation and projection matrix, μ and Σ are computed as
: K(Qu:)/(u)),.
= =QE,Q JT.
(I)
3

---

where is the Jacobian of the projective transformation. Therefore, the final opacity of a 3D Gaussian pixel coordinate æ is
0₂ o; expl
2
14) T 1 --4)
(2)
2
After sorting Gaussians by depth, the color at a can be computed by volume rendering:
N
i-1
Σ
c
II (1-0)))
(3)
j=1
where RGB color C; is evaluated by SH with coefficients h, and view direction.
Given multi-view images with known camera poses, 3D-GS optimizes static 3DD scene by minimizing the following loss function:
= (1-
+ (I,
where À = 0.2, and lgt is the ground truth. Besides, 3D-GS is initialized from from random point cloud or SfM sparse point cloud, and an adaptive density adjustment strategy applied to control the number of Gaussians.
3.2 Dynamic Stage
To reconstruct articulated object, we build the deformation based on superpoints and the linear blend skinning (LBS) 477. while the canonical model is represented by 3D-GS.
The superpoints P (p, E RR) are associated with set of 3D Gaussians, and can be used to represent the object's 's rigid parts. For timestamp 1, we directly use deformable field & learn time-variant 6 DoF transformation RR) E SE(3) of superpoint pj
(p).0
(5)
where R E SO(3) is the local rotation matrix and of E RR is the translation vector. Then, LBS employed derive the motion of each Gaussian by interpolating the transformations for their neighboring superpoints:
Σ JEN;
why (RjH + oz),, di = wirr)) & q.
(6)
JEN:
where r/ E I is the quaternion representation for matrix Rʃ, and ( is the production of quaternions. N2 denotes the K-nearest superpoints of Gaussian G,, wiy is the LBS weights between Gaussian G and superpoint Pý. which can be computed as:
exp W
Wii EREN, exp(
where W e NXM is a learnable parameter.
While keeping other attributes (i.e., 8;, 8; ha) of Gaussians the same as canonical space, we can render the image at timestamp following Eq.
3.3 Discovery of Skeleton Model
Treating each superpoint as a rigid part of the articulated object, we can discover the skeleton model (i.e., the 3D joints and the connection between joints) based on the motion of superpoints. Similar to WIM 3 there are some observations to help us discover the underlying skeleton. First, if there is joint between superpoints pa and Ps. the position of pa is more likely close the position of Ps- Second, when the relative pose between two parts changes, the joint between the two parts is relatively unchanged. Lastly, two connected parts be merged if they maintain the same relative pose throughout the whole sequence.
Let jos E 2 be the position of underlying joint between superpoints Pa and Ps, and R SO(33 the relative rotation matrix between two superpoints at time t. The relative transform between Pa and ps can be either represented by the global transform or the rotation of the joint, that is:
R
R of
R'
R'.
jab
R R.jjab
0
4
1
=
0
1
0
1
0
ab
1
(8)

---

where R = (R R R E (O(3) and t E IR' are the relative rotation matrix and translation vector between two superpoints respectively. Considering two joints jab and Jno. between Pa and pa. should be the same, we compute following distance das for every superpoint pair (a, b):
dab Σ tr, (jab a + Xalljab joallé
(9)
where X is the hyper-parameter, To prevent the distance from changing too quickly, we smooth it across training iterations:
da + 1) = (1 (T) + c. dar(T),
(10)
where 0.1 is the momentum, and T is the training iteration.
Similar to WIM 3), we discover the structure I' of joints based on the distance dub by the minimum spanning tree algorithm. We first select all pairs (a, b) if superpoint ps Ke-nearest neighborhood for superpoint Pa and sort the list of dab for those pairs in ascending order. We initialize as an empty set. We pick pair (a. from the lowest distance to the highest distance, and add this pair to while there is no path between a and b. After finishing the procedure, we obtain the final object structure r, which is acyclic graph, i.e., tree. We choose the node whose length of the longest path from itself to any other node is the shortest as the root node. If there is more than one candidate node randomly choose one the root.
3.4 Kinematic Stage
After discovering the skeleton model, we optimize the skeleton model and fine-tune 3D Gaussians by using the kinematic model. Specifically, we first predict time-variant rotations R, E SO(3) for each joint je by using deformable field V/:
(Jk, t) - R2
(11)
Then we forward-warp the superpoint Pj from the canonical space to the observation space of timestamp Ê via the kinematic model. The local transformation matrix T SE(3) of each joint is defined by a rotation R around its parent joint jk- Consequently, the final transformation of each superpoint Py can be computed as a linear combination of bone transformation:
T
R 0
of
i
T root
II REC,
T k. where T R
jk
R
(12)
where C, the list of ancestor of superpoint in skeleton model. T Tha is global transformation of root. Same as Sec. 3.2 we use LBS to derive the motion of each Gaussian and render images.
3.5 Adaptive Control of Superpoints
We use the farthest point sampling algorithm sample M Gaussians to initialize the superpoints Simply making superpoints learnable is not enough to model complex motion patterns. More importantly, wish to simplify the skeleton after training number of superpoints. Following 3D-GS [2] and SC-GS (88. we develop an adaptive control strategy prune, densify, and merge superpoints.
pose editing by reducing the
Prune: To determine whether a superpoint Py should be pruned, we calculate its overall impact W; = ΣieN, Wij, where N₁ = {i j E N.. is the set of Gaussians whose K nearest neighbors include superpoints Pj. When W, < 6pranes we prune this superpoint as it is of little contribution to the motion of 3D Gaussians.
Two aspects determining whether a superpoint should be split into two superpoints. On one hand, we clone superpoint when its impact W, is greater than threshold Setones indicating there is great amount of Gaussians associated with this superpoint, and cloning such superpoints helps model fine motion. On the other hand, caleulate the weighted Gaussians gradient norm of superpoint as:
Densify:
91 Σ iEN,
Wij
ac
12
(13)
ΣnEN, Wkj 3pp:
5

---

Table 1: Quality comparison of novel view synthesis for the D-NeRF dataset.
Method
Skeleton
PSNR1
SSIM
LPIPS}
FPS1
resolution
Opt. Time
D-NeRF [4
No
30.48
0,9683
0.0450
<1
400 400
20.0 hours
TiNeuVox: B 5
No
32.60
0,9783
0.0383
0.82
400 x 400
28.0 mins
Hexplane 49
No
29.81
0,9683
0.0400
1.37
400
400
11.5 mins
K- K-Plane hy hydd
50
No
31.02
0.9717
0.0495
0.52
400 x 400
52.0 mins
4D-GS 20
No
34.39
0,9830
0.0190
141.37
800
800
20.0 mins
SP-GS 22
No
37.55
0,9884
0.0137
234.83
800
800
52.3 mins
D-3D-GS
2
No
40.11
0.9918
0.0120
42.10
800
800
66.0 mins
SC-GS 48
No
42.98
0.9955
0.0028
123.04
400 x 400
53.3 mins
WIM 3
Yes
25.21
0.9383
0.0700
0.16
400
400
11 hours
AP-NeRF
Yes
30.91
0,9700
0.0350
1.33
400
400
150. mins
Ours
Yes
38.80
0.9870
0.0095
103.98
800 x 800
90.6 mins
Ours
Yes
39.23
0.9890
0.0070
110.90
400
400
92.5 mins
Table 2: Quantitative comparison of novel view synthesis on the Robots dataset.
Method
PSNRT
SSIM
LPIPS./
FPSt
resolution
WIM
29.11
0.9664
0.0350
0.11
512 x 512
AP-NeRF
32.45
0,9784
0.0202
0.89
512 X 512
Ours
34.34
0.9809
0.0187
137.76
512 x 512
where C is the loss function, which demonstrated in Appendix A We clone the superpoint Pj if gj is greater than the threshold gred
Merge: We merge the superpoints that should belong to the same rigid part. To determine which superpoints should be merged, first calculate the transformations Tý se3 for all superpoints at all training timestamps. Then, for each pair (a, b) of superpoints, we calculate the average relative transformations:
Da,b
Σ Log LTg Tbl..
(14)
N₂
where N, is the number of train timestamps, log() denotes the operation of converting rigid transformation matrix Lie algebra. small Da,b indicates two superpoints have similar motion patterns. Therefore, we merge two superpoints Pa and when Da.b < 8merge and ps is the K' Ke-nearest superpoints of Po.
Experiments
In this section, we present the evaluation our approach, which achieves excellent view-synthesis quality and real-time rendering speed. We also evaluate the contribution of each component through an ablation study. Additionally, we demonstrate the class-agnostic reposing capability. Please refer to our project /tttps:/dddvttt
rituub... io/SK/ /S/_SSKS)SS GS/ for video visualization.
4.1 Datasets and Evaluation Metrics
To ensure fair comparison with previous work, we choose the same datasets and configurations as 4P-NeR Specifically, choose three multi-view video datasets. First, the D-NeRF [14] dataset a sparse multi-view synthesis dataset, which includes 5 humanoids, 2 other articulated objects, and multi-compones scene. Each scene contains 50-200 frames. We choose of 8 scenes The second dataset, Robots (3] contains 7 topologically varied robots with multi-view synthetic video. We use 18
views for training and 2 views for evaluation. The third dataset, ZJU-MoCap [34]. commonly used
'ee exclude the Lego and Bouncing Balls scene due to errors in the Lego test split and the colpi-comppnent nature of Bouncing Balls

---

Jumping ackss
Mulant
Hook
Standup
Trex
WIM
AP -NRF
ground truth
Figure 2: Qualitative comparison on D-NeRF datasets.
for dynamic human reconstruction. Following WIM 3 and AP-NeRF 4 we evaluate 5 sequences
with 6 training views for each sequence. We use three metrics evaluate the image quality of the novel view, iee., peak signal-to-noise ratio (PSNR), structural similarity (SSIM) (5T), and learned perceptual image patch similarity (LPIPS) (52..
4.2 Implementation Details
We implement our framework using PyTorch. The number of superpoints is initialized as 512. For both deformable field & and V, we adopt the architecture NeRF(T). i.e., 8-layers MLP where each layer employs 256-dimensional hidden fully connected layer and ReLU activation function. We also employ positional encoding for the input coordinates and time. For optimization, we employ the Adam optimizer and use the different learning rate decay schedules for each component: the same as 3D-GS, while learning rate other components
learning rate about 3D Gaussians undergoes exponential decay, ranging from le-3 to le-5. We conducted all experiments on single NVIDIA Tesla V100 (32GB). More implementation details are shown Appendix A
4.3 Baselines
We mainly compare our method to state-of-the-an template-free articulated methods for view synthe- sis, i.e. WIM 3 and AP-NeRF [4). Besides, we also compare our method with NeRF-based and 3D-GS-based non-articulated methods. D-NeRF [14) extends NeRF dynamic scenes by warping static NeRF, Vox [ improves the visual quality and training speed by using voxel grids.

---

Atas
Baxter
Spot
WIM
AP-NeRF
ours
ground truth
Figure 3: Qualitative comparison for the Robots 3
dataset.
GT
OuTS
AP-NeRF
WIM
GT
ours
AP-NeRF
WIM
Figure 4: Qualitative comparison for the 7JU-MoCap
34 dataset.
HexPlane [49] and K-Planes 50 accelerate NeRF by decomposing the space-time volume into several planes. Similar D-NeRF. Deformable-3D-GG (21) extends static 3D-GS to the temporal domain. 4D-GS (20) accelerate Deformable33D-GS by decomposing neural voxel encoding algorithm inspired by HexPlane. Similar ours, SP-GS [22 and SC-GS [48] employ the superpoints/control points to reconstruct dynamic scenes. However, SP-GS and SC-GS can not extract skeleton from reconstructed model.
4.4 Comparisons on Synthetic Dataset
In our experiments, we benchmarked our method against several baselines using the D-NeRF dataset and Robots datasets. The quantitative comparison results, presented in Tab.[] demonstrate the superior performance of our approach terms of both rendering speed and visual quality. Specifically, our method significantly outperforms WIM and AP-NeRF not only in visual quality but also in rendering speed. Compared to 3D-GS based dynamic scenes reconstruction models, our method not only have similar performance, but also discover the skeleton and can repose the object to generate novel pose. Specifically, the rendering quality ours is higher than 4D-GS (20) and SP-GS [2]. and lower than D-3D-GS 21] and SC-GS [48] Our method also achieve real-time rendering 100 FPS), which near to the rendering speed of SC-GS [88] Fig. provides the qualitative comparisons of D-NeRF dataset, which demonstrates the advantages of method
related methods. We also provide results for Robots datasets, quantitatively Tab. [naad qualitatively Fig. 3 It is also clear that our approach is capable of producing high-fidelity novel views with real-time rendering speed. Per-scene results are shown in Appendix B
4.5 Comparison on Real-world Dataset
In Tab. and Fig. we compare our method to WIM and AP-NeRF in the ZJU-MoCap dataset. We observe that both methods can recover the 3D shape and skeleton models. However, imperfections in the camera calibrations (see Supplement F of (53) lead to lower visual quality in our results compared to WIM and AP-NeRF. With respect to rendering speed, our approach achieves up to 198.23 FPS. In stark contrast, the rendering speed of WIM and AP-NeRF is extremely slow.

---

Table Quantitative comparison for the ZJU-MoCap
34 dataset.
Method
PSNR
SSIM
LPIPS
FPSt
resolution
3IM
31.08
0,963
0.053
0.12
512 512
AP-NeRF[4
29.60
0,958
0.063
1.31
512 x 512
ours
29.11
0.961
0.063
198.23
512
512
Figure 5: Reposing using skeleton. Interpolation from canonical novel pose.
(a) Full (#sp: 97)
(b) wo Control (#sp: 512) (c) wo Merge (#sp: 608) (d) wo Larop (#sp: 453)
Figure 6: We visualize the rendering results of (a) full method, (b) method without adaptive control, (c) our method without merge superpoints, (d) our method without aarp (see Appendix A).
#sp denotes the number of superpoints. The blue points denotes superpoints.
4.6 Reposing
In Fig. interpolating the skeletons posing between user-defined poses. Video examples can found on our project webpage.
show model allows free changes poses and generates animating video by smoothly
4.7 Ablation Study
We use superpoints model the parts and motion of the object. The adaptive control of superpoints is the key to reducing the number of superpoints. Fig. (a). (b) and (c) intuitively illustrate the impact of this control strategy. Without the control strategy, the distribution of superpoints becomes uneven, with sparse representation in the arm region, which negatively impacts motion modeling. The merge process in the control strategy significantly reduces number of superpoints (97 vs 608), while superpoints are more distributed in the motion area. Besides, illustrated in Fig. (d), Larap (in Appendix A) also plays an important role controlling the density of superpoints. See Appendix C for more ablation studies.

---

Discuss
5.1 Limitations
We have demonstrated that our approach can achieve real-time rendering, state-of-the-ar visual quality, and straightforward reposing capability by skeleton and kinematic models. However, there are some limitations to our approach. Firstly, similar to WIM and AP-NeRF, the learned skeleton model of our approach is restricted to the kinematic motion space exhibited in the input video. Therefore, the skeleton model may have significant differences from the actual one, and extrapolation generate arbitrary unseen poses may cause errors. Secondly, our approach has similar limitations as other 3D-GS based methods for dynamic scenes. Specifically, the datasets with inaccurate camera poses will lead to reconstruction failures, and large motion long-term sequences can also result in failures. Lastly, the paper focuses on building the kinematic model for one articulated object. Exporting build kinematic models for multi-componen objects or complex scenes that contain multiple objects remains an opportunity for future research. Additionally, extending this approach to motion capture an interesting research direction.
5.2 Broader Impacts
Although our approach is universal, it also suitable for rending novel views and poses for humans. Therefore, we acknowledge that our approach potentially be used to generate fake images or videos. We firmly oppose the use of our research for disseminating false information or damaging reputations.
5.3 Conclusion
We have developed new method for real-time rendering of articulated models for high-quality novel view synthesis. Without any template or annotations, our approach can reconstruct a kinematic model from multi-view videos. With state-of-the-att visual quality and real-time rendering speed, our work represents a significant step towards the development of low-cost animatable 3D objects for use in movies, games, and education.
Acknowledgements
This work is supported by the Sichuan Science and Technology Program (2023YFYY0008), China Tower-Peking University Joint Laboratory of Intelligent Society and Space Governance, National Natural Science Foundation China (61632003, 61375022, 61403005). Grant SCITLAB-3000) of Intelligent Terminal Key Laboratory of SiChuan Province, Beijing Advanced Innovation Center for Intelligent Robots and Systems (2018IRS11) I), and PEKSenseTime Joint Laboratory of Machine Vision.
References
[1] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, "Nerf: Representing scenes as neural radiance fields for view synthesis," " in ECCV, 2020, pp. 405-421
[2] Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, "3d gaussian splatting for real-time radiance field rendering." ACM TOG, vol. 42, no. 4, Jul. 2023.
[3] A. Noguchi, U. Iqbal, J. Tremblay, T. Harada, and O. Gallo, "Watch it move: Unsupervised discovery of 3d joints for re- re-posing of articulated objects," in CVPR, 2022, pp. 3667-3677.
[4] L. Uzolas, E. Eisemann, and Kellnhofer, "Template-free articulated neural point clouds for reposable view synthesis," in NeurlPS, 2023.
[5] J. Fang al., "Fast dynamic radiance fields with time-aware neural voxels," SIGGRAPII Asia 2022 Conference Pupers, 2022.
[6] Q. Xu et al., "Point-nerf: Point- based neural radiance fields," in CVPR, 2022, pp. 5428-5438.
[7] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, "Tensorf: Tensorial radiance fields," in ECCV,
2022. pp. 333-350.
[8] Müller, A. Evans, C. Schied, and A. Keller, "Instant neural graphics primitives with multiresolution hash encoding," ACM TOG, vol. 41, no. 4, 102:1- Jul. 2022.
10

---

[9] T. Hu, S. Liu, Y. Chen, T. Shen, and J. Jia, "Efficientnerf efficient neural radiance fields," in CVPR. 2022. pp. 12 922-11211.
[10] C. Sun, M. Sun, and H.-T. Chen, "Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction," in CVPR, 2022, pp. 5459-5469.
[11] W. Xian, J.-B. Huang. J. Kopf, and C. Kim, "Space-time neural irradiance fields for free- viewpoint video," in CVPR, 2021, pp. 9421-9431.
[12] C. Gao, A. Saraf, J. Kopf, and J.-B. Huang, "Dynamic view synthesis from dynamic monocular video," in ICCV, 2021, pp. 5712-5721.
[13] Z. Li, S. Niklaus, N. Snavely, and O. Wang. "Neural scene flow fields for space-time view synthesis of dynamic scenes," in CVPR, 2021, pp. 6498-6508.
[14] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, "D-nerf: Neural radiance fields for dynamic scenes," in CVPR, 2021, pp. (0313-10332 322.
[15] K. Park et al., "Hypernerf: A higher-dimennsooall representation for topologically varying neural radiance fields," " ACM TOG, vol. 40, no. 6, Dec. 2021.
[16] Du, Y. Zhang, H.-X. Yu, B. Tenenbaum, and J. Wu, "Neural radiance flow for 4d view synthesis and video processing." in ICCV, 2021, pp. 14 1434-1443.
[17] Z. Li, Q. Wang, F. Cole, R. Tucker, and N. Snavely, "Dynibar: Neural dynamic image-based rendering," in CVPR, 2023, pp. 4273
[18] E. Tretschk. A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassner, and C. Theobalt, "Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video." in ICCV, 2021, pp. 12959-12970,
[19] J. Luiten, G. Kopanas, B. Leibe, and D. Ramanan, "Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis," in 3DV, 2024.
[20] G. Wu et al., "4D gaussian splatting for real-time dynamic scene rendering," in CVPR, 2024,
pp. 300-100033
[21] Z. Yang, X. Gao, W. Zhou, S. Jiao, Y. Zhang, and X. Jin, "Deformable 3d gaussians for high- figh-fideiity monocular dynamic scene reconstruction," in CVPR, 2024, pp. 20 301-2002033 341. [22] D. Wan, R. Lu, and G. Zeng, "Supeeppoitt gaussian splatting for real-time high-fidelity dynamic scene reconstruction," ICML, 2024, pp. 49 997--499972.
[23] V. Blanz and T. Vetter, "A morphable model for the synthesis 3d faces," in ACM SIGGRAPH. 1999, pp. 187-194.
[24] T. Li, T. Bolkart, M. Black, H. Li, and J. Romero, "Learning a model of facial shape and expression from 4d scans.," ACM TOG, vol. 36, no. 6, pp. 2017.
[25] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, "Smpl: A skinned multi-person linear model," ACM TOG, vol. 34, no. 6, 248:1-248:11 2015.
[26] G. Pavlakos et al., "Expressive body capture: 3d hands, face, and body from a single image," " in CVPR, 2019, pp. 10 10975-10988
[27] S. Zufli, A. Kanazawa, D. W. Jacobs, and M. J. Black, "3d menagerie: Modeling the 3d shape and pose of animals," in CVPR, 2017, pp. 6365-6373.
[28] Y. Zheng, W. Yifan, G. Wetzstein, M. J. Black, and O. Hilliges, "Pointavatar: Deformable point-based head avatars from videos," " in CVPR, 2023, pp. 21 057-21 067.
[29] G. Gafni, J. Thies, M. Zollhofer, and M. Nießner, "Dynamic neural radiance fields for monoc- ular 4d facial avatar reconstruction," in CVPR, 2021, pp. 8649-8658.
[30] Z. Wang et al., "Learning compositional radiance fields of dynamic human heads," in CVPR, 2021, pp. 5704-5713.
[31] Y. Guo, K. Chen, S. Liang, Y.-J. Liu, H. Bao, and J. Zhang, "Ad-nerf: Audio driven neural radiance fields for talking head synthesis," " ICCV, 2021, pp. 7794-5794.
[32] Y. Zheng, V. F. Abrevaya, M. C. Bühler, X. Chen, M. Black, and O. Hilliges, "Im avatar: Implicit morphable head avatars from videos," in CVPR, 2022, pp. 13 545-13 555.
[33] Y. Zhuang, H. Zhu, X. Sun, and X. Cao, "Mofanerf: Morphable facial neural radiance field,"
in ECCV, 2022, pp. 268-285.
[34] S. Peng et al., "Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans," in CVPR, 2021, pp. 9054-9063.
[35] S.-Y. Su, Yu, M. Zollhöfer, and H. Rhodin, "A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose," in NeurIPS, 2021, pp. 12 172-112....
11

---

[36] S. Peng et al., "Animatable neural radiance fields for modeling dynamic human bodies," in ICCV, 2021, pp. 34 323.
[37] H. Xu, T. Alldieck, and C. Sminchisescu, "H-nerf:
Neural radiance fields for rendering and temporal reconstruction of humans in motion," in NeurIPS, 2021, pp. 1495--10 14 966.
[38] Y. Kwon, D. Kim, D. Ceylan, and H. Fuchs, "Neural human performer: Learning generalizable radiance fields for human performance rendering," in NeurIPS, 2021, pp. 7411244552
[39] T. Hu, T. Yu, Z. Zheng, H. Zhang, Y. Liu, and M. Zwicker, "Hvtr: Hybrid volumetric--exturaal rendering for human avatars," in 3DV, 2022, pp. 197-208.
[40] L. Liu, M. Habermann, V. Rudnev, K. Sarkar, J. Gu, and C. Theobalt, "Neural actor: Neural free-view synthesis of human actors with pose control," ACM TOG, vol. 40, pp. 1-16, 2021.
[41] T. Xu, Y. Fujita, and E. Matsumoto, SSurface-aligned neural radiance fields for controllable 3d human synthesis," in CVPR, 2022, pp. 15 883-15 892.
[42] Y. Wu, Z. Chen, Liu, Z. Ren, and S. Wang. "Casa: Category-agnostic skeletal animal reconstruction," in NeurlPS, 2022, pp. 28 559-2599 259-28555 574.
[43] C.-H. Yao, W.-C Hung, Y. Li, M. Rubinstein, M.-H. Yang, and V. Jampani, "Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery," in NeurlPS, 2022,
pp. 15 296-15 308.
[44]
Wu, R. Li, T. Jakab, C. Rupprecht, and A. Vedaldi, "Magicpony: Learning articulated 3d animals in the wild," in CVPR, 2023, pp. 8792-8802.
[45] C.-H. Yao, W.-C. Hung, Y. Li, M. Rubinstein, M.-H. Yang, and V. Jampani, "Hi-lassie: High- fidelity articulated shape and skeleton discovery from sparse image ensemble," in CVPR, 2023, pp. 4853-4862.
[46] M. Zwicker, Pfister, J. van Baar, and M. H. Gross, "Ewa volume splatting," Proceedings Visualization, 2001. VIS 'O!., pp. 29- 39-538, 2001.
[47] R. W. Sumner, J. Schmid, and M. Pauly, "Embedded deformation for shape manipulation," ACM TOG, vol. 26, no. 3, 80 es, Jul. 2007.
[48] Y.-H. Huang, Y.-T. Sun, Z. Yang. X. Lyu, Y.-P. Cao, and X. Qi, "Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes," in CVPR, 2024, pp. 4220-4230.
[49] Cao and J. Johnson, "Hexplane: fast representation for dynamic scenes," in CVPR, 2023,
pp. 130-141.
[50] S. Fridovich-Keil, G. Meanti, F. R. Warburg, B. Recht, and A. Kanazawa, "K-planes: Explicit radiance fields in space, time, and appearance," in CVPR, 2023, pp. 12479-12 488.
[51] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error visibility structural similarity." " IEEE TIP, vol. 13, pp. 600-612, 2004.
[52] Z. Li, S. Niklaus, Snavely, and O. Wang, "Neural scene flow fields for space-time view synthesis of dynamic scenes," in CVPR, 2021, pp. 6494-6504.
[53] R. Li et al., "Tava: Template-free animatable volumetric actors," in ECCV, 2022, pp. 419 436.
12

---

A
More Implementation Details
The whole optimization progress be divided into three stages: dynamic joint, discovery stage and kinematic stage.
A.1 Dynamic Stage
In this stage, we aim to reconstruct the appearance of the object, and find the position of underlying joints. Therefore, we employ the Ljoints whose formula is:
M
Ljoint
Σ de
(15)
M
M M-1
(t,))e
Besides, to obtain good segmentation of object parts by using superpoints, we employ some regularization losses. To reduce the total number of superpoints, encourage nearby superpoints to possess same motion patterns via griiiddas-possiiee regularization Larap.
Larap Σ
log (R) RENT
Σ
R1012 + llo] 1
(16)
where 7T is the set of K'-nearest neighbor superpoints of superpoint j. For the blend skinning weight, we employ Ssmooth to encourage smoothness by penalizing divergence of skinning weight:
Ssmooh
lw; Wil-
(17)
i=1 JEN
Besides, we apply Leparse encourage sparsity that one Gaussian more likely associated with only one superpoint:
N
Epparse
wey log(wer) (1 Way) log(1 wiy)
(18)
total, our training loss dynamic stage is: >yGrgb Co + X2Curap + Azhsmooth + A4
(19)
where A = 11, 10 ,,.1,00,0000011 in our experiments.
In this stage, we conducted training for a total of 40k iterations. Similar to SC-GS scheme is as follows:
48 our training
1. In 0~~2k iterations, we fix deformable field
2. In 2k~10k iterations, train deformable field T which using the position of Gaussians rather than superpoints as inputs. At 7500 iteration, we sample M Gaussians by using futherest sampling algorithm. At 10k iteration, initialize superpoints by traind M Gaussians, and re-initialize the Gaussians in canonical space.
3. In 10 13k iterations, we fix deformable field t
4. In 13k 40k iterations, all parameters optimized.
During the first 20k iterations, We do not discover joints and skeleton, i.e., A 0. During 20k 40k iterations, we update the structure of skeleton every 100 iterations. Note that we stopped the gradient between joints and other parts so that the learning of joints has no effect on the learning of Gaussians, superpoints and deformation field.
During training, adopt the same adaptive control strategy for Gaussians as 3D-GS In details, in iterations I~~5500 and 10k~ 35k, we densify and prune Gaussians every 100 iterations, while the densify grad threshold is set to 0.0002 and the opacity threshold for prune is set to 0.005. We reset the opacity of Gaussians every 3k steps.
We densify and prune superpoints every 1000 steps between iterations 20k and 30k, and the hyper- parameter 8 5yrad 0.0002 and &prane 0.001. We merge superpoints every 1000 steps between iterations 30k and 40k, while threshold amerge 0.0005.
13

---

A.2 Skeleton Discovery Stage
In this stage, we aim to initialize deformable field V the using the learned deformable field in dynamic stage. Firstly, fix skeleton structure r, and initialize the joints jk. k 1..... .M 1 by underlying joints Job according to I. We then cache the outputs of deformable field & for all timestamps in the training dataset. Next, we cached motions of superpoints optimize the parameters of deformable field V, the positon of the joints jk by employing Adam optimizer and following loss functions:
M
Ldiscovery
Aa(llu - pi112 + Rog (R)) + Ao Aoll(R u²) (Ríp + xL) (20)
M
1=1
where fi; and RC is translations and rotations calculated by deformable field Φ, while μeī and R! is translations and rotations calculated by deformable field v and kinematic model. À = 1 controls the weights of the transform matrix difference, and Å = 0.1 adjusts the weights of relative offset of the superpoints position at timestamp
For this stage, we train for 10k iterations with a fixed learning rate of 10-3
A.3 Kinematic Stage
In the kinematic stage, we optimize the kinematic model, joints and Gaussians, while keeping the number of Gaussians, the number of superpoints and the structure of skeleton fixed, We also conduct training for total of 40k iterations by only using Legh.
A.4 Others
For the deformable field 4 and V, we adopt the following positional encoding for the input coordinates a I Nx3 and time t [0, 1]:
{pp) (sin(2* p), cos(2" p))
(21)
where L 10 for æ and for L.
The initial number of superpoints is set to 512, while K = K' = 5.
B Per Scene Results
B.1 Per-Scene Results for D-NeRF dataset
Tab. shows the per-scene quantitative results for the D-NeRF dataset.
B.2 Per-scene Results for Robots dataset
Tab. shows the per-scene quantitative results for Robots dataset.
C More Results
C.1 Required Resources
As shown in Tab. 6 present detailed information the optimization time and the required resources.
C.2 Ablation study for the number of initial superpoint
The ablation study on the number of initial superpoint is shown in Tab. 7
As shown in the Fig. 7 our method struggles to accurately reconstruct the endpoints of objects.
14

---

PSNR
Jumpinglacks
Mutant
Hook
T-Rex
StandUp
HellWarrior
Average
D-NeRF
14
32.80
31.29
29.25
31.75
32.79
25.02
30,48
TiNeu
34.23
33.61
31,45
32.7
35.43
28. 28.17
32.60
Hexplane
49
31.65
33.79
28.71
30.67
34..6
24.24
29.81
K-P K-Plane hybrid
50
32.64
33.79
28.5
31.79
33.72
25.7
31.02
D-3D-GS
21
37.59
42.61
37.09
37.67
44.30
41.41
40.11
4D-GS
20
35.44
37.43
33.01
33.61
38.11
28.77
34.39
SP-GS
22
35.56
39.43
35.36
32.69
42.07
40.19
37.55
SC-GS 48
41.62
45.08
39.81
40.70
47.81
42.88
42.98
WIM 3
29.77
25.80
25.33
26.19
27.46
16.71
25.21
AP-NeRF
34.50
28,56
30,24
32.85
31.93
27.53
30.94
Ours (800 x 800)
36.95
40.95
36.64
35.10
43.52
39.65
38,80
Ours (400 x 400)
36.70
41.96
36.77
36.14
43.84
39.95
39,23
SSIM
JumpingJacks
Mutant
Hook
T-Rex
StandUp
HellWarrior
Average
D-NeRF
0.98
0.97
0.96
0.97
0.98
0.95
0,9683
TiNeu B
0.98
0.98
0.97
0.98
0.99
0.97
0,9783
Hexplane
49
0.97
0.98
0.96
0.98
0,98
0.94
0,9683
K-Plane hybrid
50
0.977
0.983
0.954
0.981
0.983
0.952
0.9717
D-3D-GS
0.9929
0,987
0,9858
0.995
0.9947
0.9953
0.9918
4D-GS 20
0.9857
0,988
0.9760
0.985
0.9898
0.9733
0,9830
SP-GS 22
0.9950
0.9868
0.9804
0.9861
0.9926
0.9894
0.9884
SC-GS 48
0.9957
0.9977
0.9934
0,9972
0.9981
0,9908
0,9955
WIM
0.97
0.95
0.94
0.94
0.96
0.87
0.9383
AP-NeRF
0.98
0.96
0.97
0.98
0.97
0.96
0,9700
Ours (800 x 800)
0.9883
0,9921
0.9847
0,9877
0,9933
0.9761
0,9870
Ours (400 x 400)
0.9889
0.9951
0,9869
0.9934
0.9950
0.9749
0,9890
LPIPS
JumpingJacks
Mutant
Hook
T-Rex
StandUp
HellWarrior
Average
D-NeRF 14
0.03
0.02
0.11
0.03
0.02
0.06
0.0450
TiNeuVox
0.03
0.03
0.05
0.03
0.02
0.07
0.0383
Hexplane
0.04
0.03
0.05
0.02
0.07
0.0400
K-Plane hybrd
0.0468
0.0362
0.0662
0.0343
0.031
0.0824
0.0495
D-3D-GS 21
0.0126
0.0052
0.0144
0.0098
0.0063
0.0234
0.0120
4D-GS 20
0.0128
0.0167
0.0272
0.0131
0.0074
0.0369
0.0190
SP-GS 22
0.0069
0.0164
0,0187
0.0243
0.0096
0.0066
0.0137
SC-GS 48
0.0030
0.0011
0.0037
0.0014
0.0008
0.0068
0.0028
WIM
0.04
0.06
0.06
0.08
0.04
0.14
0.0700
AP-NeRF
0.03
0.03
0.05
0.02
0.02
0.06
0,0350
Ours (800 x 800)
0.0086
0.0038
0.0089
0.0105
0.0045
0.0203
0.0095
Ours (400 400)
0.0090
0.0022
0.0073
0.0054
0.0026
0.0155
0.0070
FPSt
JumpingJacks
Mutant
Hook
T-Rex
StandUp
Hellwarrior
Average
D-3D-GS
21
16.35
102.51
32.3
20.59
49.14
31.68
42.10
4D-GS
20
112.89
129.59
147.38
144.46
152.46
161.41
141.37
SP-GS 22
271.27
210.42
230.35
186.65
260.34
249.93
234.83
SC-GS 48
128.05
122.01
129.81
105.24
132.91
120.24
123.04
WIM 3
0.19
0.17
0.13
0.16
0.18
0.13
0.16
AP-NeRF
1.57
1.42
1.11
1.34
1.48
1.06
1.33
Ours (800 x 800)
106.81
101.04
103.60
98.38
102.02
112.03
103.98
Ours (400 x 400)
109.61
104.22
109.11
109.15
110.3
123.28
110.90
Table 4: Quantitative Results Per-Scene in D-NeRF dataset.
15

---

PSNR
Atlas
Baxter
Cassie
liwa
Nao
Pandas
Spot
Average
WIM 3
25.01
23.36
30.08
32.77
28.68
35.93
27.92
29.11
AP-NeRF(4 (4
32.56
30.74
31.44
35.78
30.64
32.44
33.57
32.45
ours
34.89
31.96
32.08
39.34
33.24
32.02
36.85
34.34
SSIMI
Atlas
Baxter
Cassic
liwa
Nao
Pandas
Spot
Average
WIM 3)
0.9405
0.9538
0.9712
0.9866
0.9546
0.9878
0.9704
0.9664
AP-NeRF(4
0.9833
0.9759
0.9775
0,9884
0.9615
0,9785
0.9837
0.9784
ours
0.9891
0.9561
0.9792
0.9928
0.9741
0.9896
0.9856
0.9809
LPIPS4
Atlas
Baxter
Cassie
liwa
Nao
Pandas
Spot
Average
WIM
0.0648
0.0455
0.0359
0.0160
0.0368
0.0177
0.0283
0.0350
AP-NeRF(4
0.0170
0.0196
0.0281
0.0129
0.0323
0.0203
0.0110
0.0202
ours
0.0092
0.0430
0.0249
0.0062
0.0187
0.0172
0.0114
0.0187
FPSt
Atlas
Baxter
Cassie
liwa
Nao
Pandas
Spot
Average
WIM
0.08
0.08
0.10
0.17
0.07
0.10
0.08
0.10
AP-NeRF
0.66
1.26
0.33
0.35
1.02
0.39
1.73
0.82
ours
123.32
149.43
139.08
122.59
141.58
146.73
141.62
137.76
Table 5: Quantitative Results Per-Scene in Robots dataset.
Table 6: Optimization time and required resources for each scene in the D-NeRF dataset.
scene
hellwarrior
hook
jumpingjacks
mutant
standup
average
Training Time (h)
1.17
1.52
1.50
1.55
1.40
1.92
1.51
GPU VRAM(GB)
1.23
3.63
2.31
3.33
2.19
4.32
2.83
num. of Gaussians (x10*)
0.60
1.54
1.11
1.44
0.97
1.92
1.28
num. of superpoints
188
184
112
51
134
42
118.5
(b)
(c)
Figure 8: Compare rendered images between canonical space and the warp space of timestamp 0. (a) canonical space in Dynamic stage, (b) time Dynamic stage, (c) canonical space in Kinematic stage, (d) at time in Kinematic stage, LB)LBS at time 0 in Kinematic stage (f)rrundd truth at time 0.
(a) AP-NeRF
(b) ours
(c) ground truth (d) AP-NeRF
(c)
(f) ground truth
Figure 7: Compare to AP-NeRF, ours method are more robust for complex motion.
16

---

Table 7: Ablation study for the number of initial superpoint M on the 'hellwarrior' scene of D-NeRF dataset.
128
256
384
512
640
768
896
1024
PSNR 39.61
39.70
39.83
39.69
39.89
39.56
39.72
40.43
SSIM
0.9765
0.9766
0.9779
0.9773
0.9781
0.9766
0.9767
0.9800
LPIPS
0.0191
0.0179
0.0173
0.0183
0.0169
0.0183
0.0186
0.0155
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions, which match experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this paper in Sec. 5.1
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide disclose all information in Sec. 3 and Appendix A needed to reproduce the experimental results of paper. Besides, we will release the source code once the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We released the source code GitHub, i.e., http:/tttt////gtt....///////aaaa
SK GS
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
cublicc/quiees/Codestbbaas casssiionooyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppllgguuuiieeeeae
wtweeee//ooSSSeisssssee.e www.ssssiicggooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Sec. Sec. 3 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do report error bars other information about statistical significance as this is not common procedure in the field and does not contribute to understanding our evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources in the paper and code.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://neurrii cc http://lcc//tthhccsiiii.iooo
Answer:
[Yes]
Justification:
We conducted this research in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts of this paper in Sec. 5.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "ZAxkyjGwky", "reviewer": "Reviewer_xZeD", "summary": "The paper proposes a novel approach for reconstructing reposable dynamic 3D objects from RGB videos using Gaussian Splatting, without requiring any template as input.\n\nTo achieve this, the paper suggests grouping Gaussians around superpoints, which are intended to represent rigid parts of the scene. By optimizing and analyzing these superpoints, a full skeleton model of an articulated object in the input video can be built, refined, and used for reposing purposes.\n\n**Details**\n\nThe approach consists of two main stages:\n\n1. *Dynamic Stage*. After optimizing a canonical 3D Gaussian Splatting (3DGS) representation for a few iterations, a set of superpoints is initialized in the scene. A deformable field mapping each superpoint to a time-variant 6DoF transformation is optimized. These transformations are used to derive the motion of Gaussians by interpolating transformations with neighboring superpoints through linear blend skinning (LBS). The paper also proposes a gradient-based strategy to control (prune, merge, or densify) the number of superpoints in the scene. Toward the end of the dynamic stage, a skeleton structure with joints is enforced and discovered in the scene by analyzing the distance between and configuration of superpoints.\n\n2. *Kinematic Stage*. After discovering the skeleton model of the scene, the number of Gaussians and superpoints is fixed and optimized along with a new MLP mapping skeleton joints to time-variant rotation matrices. These matrices are used to compute the motion of each Gaussian along the kinematic chains using LBS. After full optimization, the skeleton can be used for reposing and editing the reconstructed object.\n\nThe paper presents extensive experiments and demonstrates higher rendering performance and speed compared to concurrent reposable dynamic Radiance Field methods.", "questions": "1. What are the optimization time and required resources (e.g., VRAM) for the proposed approach? How does it compare to state-of-the-art methods?\n\n2. Do the authors have insights on how the method would perform in the context of a monocular video with a moving camera, which is a more realistic setting than multi-view videos?\n\n3. In Figure 5, the skeleton appears to be quite accurate, but some bones are located outside the geometry. Would it be possible to enforce the skeleton to be located “inside” the geometry, perhaps by applying a penalty that encourages superpoints to be close to the centroid of their associated Gaussians?", "limitations": "As I already mentioned it, the limitations are only discussed in the supplementary material, which is problematic in my opinion. I encourage the authors to try to move the limitations to the main paper in the final version.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The task addressed by the paper is challenging but crucial for many applications in both graphics and robotics. I appreciate the proposed strategy, which successfully retrieves kinematic chains from RGB videos.\n\n3. The quantitative evaluation presented in the paper is convincing and clearly demonstrates the superiority of the approach over concurrent methods.", "weaknesses": "1. The paper may lack sufficient skeleton examples to effectively demonstrate that the proposed approach can recover meaningful structures from RGB videos. Indeed, the primary goal is to recover skeleton structures and enable reposing capabilities, but only a single skeleton example is provided (Figure 5). Including more qualitative examples would likely make the paper more convincing.\n\n2. The paper does not provide details on the optimization time and required resources (e.g., VRAM) for the proposed approach. It appears that a large number of training iterations is needed; a comparison with previous state-of-the-art models would be valuable.\n\n3. The limitations of the approach are interesting but are only discussed in the supplementary material, which is problematic in my opinion. These limitations are crucial for further research and should be included in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. Below are our responses to the specific points raised.\n\n- **Q1/W2**: Optimization time and required resources\n\n  **Answer**: Similar to 3D-GS, the optimization time and required resources are dependent on the number of Gaussians. For the D-NeRF dataset, the average optimization time is 6.3 hours, and the average peak GPU memory is 4.08 GB. Please refer to Tables 1 and 3 in the attached `PDF` for details.\n\n- **Q2**: Performance on monocular video with a moving camera\n\n  **Answer**: The camera setup in the D-NeRF dataset can be considered as using a camera to capture 360-degree surround images centered on the target object. Therefore, using a monocular video with a moving camera as input, our method can achieve good performance. However, using forward-facing monocular videos as input may result in reconstruction failures or inaccurate skeletons.\n\n- **Q3**: Enforce the skeleton to be located \"inside\" the geometry\n\n  **Answer**: We acknowledge the suggestion to add an extra regularizer term to encourage the skeleton to be \"inside\" the geometry. We will explore this idea in our future work.\n\n- **W1**: More qualitative examples \n\n  **Answer**: We have provided additional qualitative examples in the supplementary video material.\n\n- **W3**: Limitations in the supplementary material\n\n  **Answer**: Thanks for your suggestion. We will move it into the main text in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GRc6wCTu66", "reviewer": "Reviewer_57cM", "summary": "The paper introduces a method combining 3D Gaussian Splatting and superpoints for dynamic object modeling, achieving real-time rendering and high visual fidelity. Empirical results show that the proposed method achieves state-of-the-art results on several benchmarks.", "questions": "1. The number of $M$ is not given in the paper, the author should do an ablation study on it.\n2. The proposed method and results should be discussed with the latest relevant methods as referenced in [1].\n\n[1] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes.", "limitations": "Please refer to the Weaknesses and Questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. The main contribution and methodology are well illustrated.\n2. The use of an adaptive control strategy to manage superpoints is innovative and helps in optimizing the model, avoiding redundancy, and maintaining efficiency.", "weaknesses": "1. Although this paper achieves real-time rendering compared to AP-NERF, I find it somewhat incremental and lacking in innovation since most parts of the method are existing concepts.\n2. This paper emphasizes the concept of \"Reposable,\" but the related experiments are very limited. A thorough analysis of this aspect could effectively distinguish this paper from AP-NERF.\n3. This method compares fewer baselines, and the quantitative results do not show significant improvements in rendering effects and speed compared to the baselines, as shown in Table 3, Table 4, and Table 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and efforts. \n\n- **W1**: Lacking in innovation\n\n  **Answer**: While our work builds upon previous works, to the best of our knowledge, it is the first work to discover the skeleton of articulated objects represented by 3D Gaussian Splatting.\n\n- **W2**: Distinguish this paper from AP-NERF by a thorough analysis\n\n  **Answer**: Both our method and AP-NeRF are proposed for learning articulated representations of dynamic objects. However, there are two main differences between our method and AP-NeRF:\n   1. While AP-NeRF is based on point-based NeRF representations, our method is based on 3D Gaussian Splatting, which is key to achieving real-time rendering.\n   2. While AP-NeRF extracts the skeleton using the Medial Axis Transform (MAT), our method discovers the skeleton based on the motion of superpoints.\n\n- **W3**: This method compares fewer baselines, and the quantitative results do not show significant improvements\n\n  **Answer**: \n  - Unlike other methods focused on reconstructing dynamic scenes (e.g., D-NeRF, HyperNeRF, Deformable-3D-GS, 4D-GS), our work, alongside with WIM and AP-NeRF, uniquely addresses explicit skeleton extraction from videos without any priors.\n  -  As evidenced in Tables 3, 4, and 5, our approach achieves real-time rendering (>100 FPS), surpassing WIM and AP-NeRF (<1 FPS). Furthermore, our method exhibits superior rendering quality on the D-NeRF and Robots datasets.\n- **Q1**:  Ablation study for the number of $M$\n\n  **Answer**: $M$ is the number of superpoints, which is initialized as 512 as shown in L212. We conduct the ablation study for $M$. The results are shown in Table 2 in the attached `PDF`.\n\n- **Q2**: Comparison with SC-GS\n  \n  **Answer**: Quantitative comparison between our method and SC-GS is provided in Table 1 in the attached `PDF`. Besides,\n  - **Difference**: Our method extracts the explicit skeleton model and refines it during the *Kinematic* stage, while SC-GS does not.\n  - **Similarity**: Both our method and SC-GS utilize the sparse points (i.e., superpoints in ours, control points in SC-GS) with LBS.  Notably, the Dynamic stage in our approach can be replaced with SC-GS.\n  - **Difference**: While SC-GS employs the Gaussian-kernel RBF to compute the weights of LBS, our method directly learns it (i.e., $\\mathbf{W}$ in Eq 7).\n  - **Difference**: SC-GS can edit motion by minimizing APAR energy. Compared with SC-GS, our method directly manipulates the skeleton, which is more efficient and simpler.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s3tLvBcG15", "reviewer": "Reviewer_m4oH", "summary": "This paper presents a novel approach for learning articulated objects, alongside the skeletons/kinematic trees directly from input videos, eliminating the need for pre-defined meshes or hand-crafted skeleton priors.\n\nSpecifically, the paper introduces a hierarchical 3D Gaussian representation, where a set of superpoints are used as guidance for deforming the sub-points using linear blend skinning (LBS). The skeletal structure is further derived from the superpoints, providing articulated structures that enable explicit pose control. Rendering is done by Gaussian Splatting, enabling real-time performance.\n\nOverall, the paper tackles a very challenging and useful problem for 3D modeling, the manuscript is easy to follow, and the approach is interesting.", "questions": "Some comments regarding the evaluation sections:\n- Tab 1, 2: does the performance gain come mainly from using a higher resolution, or does it come from “capturing better articulated structures”? While Gaussian splatting enables us to use higher resolution due to its rendering speed, it would be great if the paper could also include results with resolutions comparable to other setting (400x400 in Tab 1, and 800x800 in Tab 2). \n- Is there a way to properly evaluate how good the learned skeleton structure is? E.g., training a skeleton-based 3D articulated model using the skeleton from AP-NeRF v.s. WIM v.s. the proposed method. \n\nAlso, one small issue:\n- L147: should be (R^t_b)^-1 instead of R^t-1_b", "limitations": "The paper discussed some of their limitations, but it would be great if the paper could include more analysis/visual results for the issues mentioned above in the Weaknesses section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper lie in how it brilliantly leverages 3D Gaussians for capturing the underlying articulated structures in a video, without the need for 3D annotations or pre-defined structure priors.\n\n The design of superpoints naturally models prominent candidates that can serve as control points. Furthermore, as the control points are “learned” automatically, it can potentially arrive at a representation that better suits the possible motion of the articulated subject.\n\nOverall, the approach presented in the paper is pretty neat, and the experiments show pretty promising qualitative and quantitative results.", "weaknesses": "The approach does have some room for improvement. Specifically,\n- Limited reposability. As mentioned in L372-373, the approach is limited to the motion space in the input video. It would be great if the papers could include visual results for these failure cases. It will be interesting to see how good the learned LBS weights are.\n- Evaluated on datasets with limited motions: the videos used in the paper mostly contain repetitive motion sequences, and/or with small motions. It will be interesting to see how the proposed method performs on videos with complex/diverse/large motions (e.g., AIST datasets). Also, it is similarly unclear how the method can perform on in-the-wild videos with uncontrolled lighting, or with only a single view. \n\nOverall, these weaknesses are very common among template-free approaches, not specifically to the proposed method itself. Nevertheless, it would be great if the paper could include more figures, visual results, and analysis regarding these cases.\n\nThere are also some issues regarding the experiments, which I detailed in the Questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the useful suggestions.\n\n- **Q1**: Higher resolution\n  \n  **Answer**: We provide the results at the same resolution as WIM and AP-NeRF in the attached `PDF`. We believe the improved performance mainly comes from the powerful representation ability of 3D-GS and the better capture of articulated structures.\n\n- **Q2**: Evaluate skeleton model\n  \n  **Answer**: The biggest challenge in evaluating skeleton models is that the learned skeleton is uncertain in terms of the number of joints and the connections between joints.\n\n  In our view, there are three imperfect methods that might be used to evaluate the quality of the learned skeleton structure.\n  1. As shown in WIM, we can first learn a mapping between the learned skeleton model and a template (e.g., SMPL for humans), and then evaluate the mapped template. However, this way requires a template and the mapping process will introduce more errors.\n  2. Given test images with camera poses, all parameters except the rotation of joints are fixed, and the learned model is then optimized to fit the test images. After test-time optimization, the difference between the rendered image and test images shows how good the learned skeleton structure is. However, test-time optimization may fail or be suboptimal.\n  3. The repose process can be treated as an image generating process (e.g., GAN, Diffusion model). Therefore, we evaluate the learned skeleton structure by using the metrics (e.g., FID) that are common in the generative model. However, obtaining the rotation range of each joint and test camera pose is challenging.\n\n  Overall, rigorously evaluating skeleton models is a challenging problem, and we will try to address in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fLj6ps6UTo", "reviewer": "Reviewer_XcTH", "summary": "The authors propose to combine a reposable 4D reconstruction from multi-view video based on a skeletal LBS model with 3D Gaussian splatting. To this goal they introduce a novel strategy for estimation of the skeletal model from a superpoint clustering. The results demonstrate a superior image quality and, thanks to the representation, also fast rendering.", "questions": "Q1) The Eq. 13 could use more discussion. Does the formulation avoid over-segmentation of large but correctly rigid parts to multiple sub-parts? It would be useful to see the LBS visualized for the final shapes.\n\nQ2) Why are the skeletons so complex and noisy? Is that perhaps related to my other question about over-segmentation in Eq. 13? Are such skeletons practical for re-animation? How was re-animation done in the video? How many parameters / joint motions had to be defined? \n\nQ3) Why are there different resolutions reported for each method (Table 2 & 3). Does it mean each method was validated against a different reference image resolution? Or does it mean the methods were all tested against a full resolution reference but some were only trained using low-resolution data? That could potentially introduce considerable bias.\n\nQ4) The authors also do not provide any failure case examples/images. Does the method work perfectly for all tested scenes? What about the robots from the limitation Figure 6 of AP-NeRF? Does the new method handle these cases better?", "limitations": "The limitations and broader impacts are only discussed in the Appendix which I do not see as a responsible practice. It suggests that the authors do not give downsides of the method the same importance as to the upsides.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1) Implementation details provided for reproducibility.\n\nS2) The claims are validated on common datasets.\n\nS3) The result quality is visibly better than the prior work.\n\nS4) The skeleton construction is novel, technically sound and produces reposable models in variety of scenes.\n\nS5) The quality of exposition is good.", "weaknesses": "W1) The skeletons are over-segmented and unnatural and likely would not be very friendly for a human animator. They may still be suitable for data fitting but do not provide nearly as much regularization as an \"optimal\" (ground truth) skeleton would.\n\nW2) The limitations and broader impacts are only discussed in the Appendix which I do not see as a responsible practice. It suggests that the authors do not give downsides of the method the same importance as to the upsides.\n\nW3) The authors claim to report Statistical Significance without further comments (checklist item #7), but I cannot see any such features in the paper.\n\nW4) It may be a good idea to consider a higher quality captured dataset than ZJU-Mocap. It does not seem to allow for a useful comparison between the methods.\n\n\nOther minor issues and suggestions:\n\n- Figure 1: Each superpoints -> superpoint\n\n- L145: related rotation matrix -> relative?\n\n- $\\mathbf{W}$ is overloaded in Eq. 1 and Eq. 6 for two distinct things which is not ideal.\n\n- Eq. 13: $\\mathcal{L}$ without suffix is not defined.\n\n\n------------------------\n**Justification of recommendation**\nA solid paper with its incremental but non-trivial contribution stemming mainly from the novel skeleton construction. The experimental results are convincing and the main downside is the clutter and complexity of the recovered skeleton. Despite this, I am currently comfortable recommending acceptance under the assumption that the exposition issues are addressed (especially limitations). My final decision might change based on the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments.\n\n- **Limitations/W2**: Limitations and broader impacts \n  \n  **Answer**: We will move the limitations and broader impacts to the main paper in the final version. \n\n- **Q1**: More discussion about  Eq. 13.\n  \n  **Answer**: Based $g_i$ calculated by Eq. 13, we can densify more superpoints to accommodate complex motions. Therefore, in theory, this formula will not split a large rigid part into multiple sub-parts. Even if more sub-parts are generated, since the motion of these sub-parts is the same, the *merging process* will merge the sub-parts into one again.\n\n- **Q2**: Why are the skeletons so complex and noisy?\n  \n  **Answer**: To re-animate (/re-pose) the $k$-th joint of the object, we simply apply an additional rotation matrix $\\Delta \\mathbf{R}_k \\in \\mathrm{SO}(3)$ on the rotation $\\hat{\\mathbf{R}}_k$ predicted by deformable filed $\\Psi$ (Eq. 11), i.e., $\\hat{\\mathbf{R}}_k' = \\Delta \\mathbf{R}_k \\hat{\\mathbf{R}}_k$. Compared with 6 DoF of superpoints in the Dynamic stage, each joint in the Kinematic Stage has only 3 DoF i.e., only can rotate but not translate. Therefore, the extra DoF of superpoints is the main reason are the main reason why the discovered skeletons are so complex and noisy. As shown in Fig. 1 in the attached `PDF`, there are some inconsistencies in the 3D model in canonical space of Dynamic stage. However, the rendered image in the given timestamp matches the ground truth. The price of achieving all this is that the skeleton is complex and noisy. \n\n  Another reason is that reconstucted motion of superpoint is imperfect. The number of superpoints is therefore greater than the number of physically rigid parts. As a result, the skeleton discovered by the motion of superpoints  becomes complex and noisy.\n\n- **Q3**: Why are there different resolutions reported for each method?\n  \n  **Answer**: To solve this concern, we present the results of our method with the same resolutions in Table 1 in the attached `PDF`. The training and testing resolutions are the same. For example, when AP-NeRF is trained on $400\\times 400$ images, it is also tested on $400 \\times 400$ images.\n\n- **A4**: Any failure case examples/images\n  \n  **Answer**: For scenes with complex motions (e.g., combining long chains of rotations with texture cue ambiguity), our method can achieve better performance than AP-NeRF. Figure 2 in the attached `PDF` compares our method with AP-NeRF for the failure case of AP-NeRF. As shown in the figure, our method may not be able to reconstruct the endpoints of objects well.\n\n- **W3**: Statistical Significance \n  \n  **Answer**: The answer of Statistical Significance is No. And the Justification is: We do not report error bars or other information about statistical significance as this is not a common procedure in the field and does not contribute to understanding our evaluation.\n\n- **W4**: Higher quality captured dataset.\n  \n  **Answer**: Thanks for your suggestion. We follow the experiment setting of AP-NeRF. We will evaluate our method on higher-quality captured datasets in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their positive and constructive feedbacks.

The attached `PDF` contains 3 tables and 2 figures.

Meta-review: The paper is about reconstructing a 3D skeleton model from dynamic images with 3DGS and superpoints. The reviews were initially split, with key weaknesses pointed out. An active authors-reviewers discussion clarified all points and an acceptance consensus was reached.
Decision: Accept (poster)