Template-free Articulated Gaussian Splatting for Real-time Reposable Dynamic View Synthesis

Authors: diwen  wan
yuxiang  wang
ruijie  lu
gang  zeng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 13:51:18

Abstract: While novel view synthesis for dynamic scenes has made significant progress, capturing skeleton models of objects and re-posing them remains a challenging task. To tackle this problem, in this paper, we propose a novel approach to automatically discover the associated skeleton model for dynamic objects from videos without the need for object-specific templates. Our approach utilizes 3D Gaussian Splatting and superpoints to reconstruct dynamic objects. Treating superpoints as rigid parts, we can discover the underlying skeleton model through intuitive cues and optimize it using the kinematic model.  Besides, an adaptive control strategy is applied to avoid the emergence of redundant superpoints.  Extensive experiments demonstrate the effectiveness and efficiency of our method in obtaining re-posable 3D objects. Not only can our approach achieve excellent visual fidelity, but it also allows for the real-time rendering of high-resolution images.

Main paper: 
 introduction: 
 Novel view synthesis for 3D scenes is important for many domains including virtual/augmented/mixed reality, game or movie productions. In recent years, Neural Radiance Fields (NeRF) [1] have witnessed significant advances in both static and dynamic scenes. Among them, 3D Gaussian Splatting (3D-GS) [2] proposed a novel point-based representation, and is capable of real-time rendering while ensuring the quality of generated images, bringing new insights to more complex task scenarios.Although visually compelling results and fast rendering speed have been achieved in reconstructing a dynamic scene, current methods mainly focus on replaying the motion in the video, which means it just renders novel view images within the given time range, making it hard to explicitly repose or control the movement of individual objects in the scene. For some specific categories such as the human body or human head, one main approach is to leverage the category-specific prior knowledge such as templates to support the manipulation of reconstructed objects. However, it is hard for these methods to generalize to large-scale in-the-wild scenes or human-made articulated objects. Some template-free methods attempt to address these challenges by building reposable models from videos. Watch-It-Move (WIM) [3] leverages ellipsoids, an explicit representation, to coarsely model 3D objects, and then estimate the residual by a neural network. The underlying intuition is that one or more ellipsoids can represent a functional part. By observing the motion of parts from multi-view videos, WIM can learn both the appearance and structure of articulated objects. However, the reconstruction results of WIM are of low visual quality and the training and rendering speed is slow. Apart from WIM, Articulated Point NeRF (AP-NeRF) [4] samples feature point cloud from a pre-trained dynamic NeRF model (TiNeuVox [5] ) and initializes the skeleton tree using the medial axis transform algorithm. By combining linear blend skinning (LBS) and point-based rendering [6] , AP-NeRF jointly optimizes dynamic NeRF and skeletal model from videos. Compared to WIM, AP-NeRF achieves higher visual fidelity while significantly reducing the training time. However, AP-NeRF cannot achieve real-time rendering, which is still far from practical application.In this paper, we target class-agnostic novel view synthesis of reposable models without the need for a template or pose annotations, while achieving real-time rendering. To enable fast rendering speed, we opt to represent the 3D object as 3D Gaussian Splatting. To be specific, we first reconstruct the 3D dynamic model using 3D Gaussians and superpoints, where each superpoint binds Gaussians with similar motions together. These superpoints will later be treated as the parts of an object. Afterward, a skeleton model is discovered leveraging some intuitive cues under the guidance of superpoint motions from the video. Finally, we jointly optimize the skeleton model and pose parameters to match the motions of the training videos. During the optimization process of object reconstruction, we will inevitably generate a lot of redundant superpoints to fit the complex motion. To simplify the skeleton model and avoid overfitting, we employ an adaptive control strategy and regularization losses to reduce the number of superpoints. Our contributions can be summarized as follows:• We propose a novel method based on 3D Gaussians and superpoints for modeling appearance, skeleton model, and motion of articulated dynamic objects from videos. Our approach can automatically discover the skeleton model without any category-specific prior knowledge. • We effectively learn and control superpoints by employing an adaptive control strategy and regularization losses. • We demonstrate excellent novel view synthesis quality while achieving real-time rendering on various datasets.2 Related Works 
static and dynamic neural radiance fields: 
 In recent years, we have witnessed significant progress in the field of novel view synthesis empowered by Neural Radiance Fields. While vanilla NeRF [1] manages to synthesize photorealistic images for any viewpoint using MLPs, subsequent works have explored various representations such as 4D tensors [7] , hash encodings [8] , or other well-designed data structures [9, 10] to improve rendering quality and speed. More recently, a novel framework 3D Gaussian Splatting [2] has received widespread attention for its ability to synthesize high-fidelity images for complex scenes in real-time.Meanwhile, many research works challenge the hypothesis of a static scene in NeRFs and attempt to synthesize novel-view images of a dynamic scene at an arbitrary time from a 2D video, which is a more challenging task since the correspondence between different frames is non-trivial. One line of research works [11] [12] [13] directly represents the dynamic scene with an additional time dimension or a time-dependent interpolation in a latent space. Another line of work [14] [15] [16] [17] [18] represents the dynamic scene as a static canonical 3D scene along with its deformation fields. While one main bottleneck of synthesizing a dynamic scene is speed, some works [19] [20] [21] [22] propose to extend 3D Gaussian Splatting into 4D to mitigate the problem. Though being able to recover a high-fidelity scene, this method cannot directly support editing and reposing objects within it. In this work, we leverage 3D Gaussian Splatting as the representation for faster rendering speed.It's impractical to directly repose the deformation fields of dynamic NeRFs due to the complexity of high-dimension. Therefore, utilizing parametric templates based on object priors to represent deformation is adopted in many research works. The classes of parametric templates range from human faces [23, 24] , and bodies [25, 26] to non-human objects like animals [27] . With the help of skeleton-based LBS and 3D or 2D annotations, these parametric templates are capable of representing articulate human heads [28] [29] [30] [31] [32] [33] and bodies [34] [35] [36] [37] [38] [39] [40] [41] . Though these template-based reposing methods can synthesize high-fidelity images, they are restricted to certain object classes and mainly deal with rigid motions, not to mention the time-consuming process of annotations.To alleviate the excessive reliance on domain-specific skeletal models, methods based on retrieval from database [42] or adaptation from a generic graph [43, 44] are adopted. However, these methods are still of relatively low flexibility and diversity. Another line of work attempts to learn a more The pipeline of proposed approach. Our approach follows a two-stage training strategy. In the first stage (i.e. dynamic stage), we learn the 3D Gaussians and superpoints to reconstruct the appearance. Each superpoint is associated with a rigid part, and the adaptive control strategy is used to control the count. After finishing the training of dynamic stage, we can discover the skeleton model based on superpoints. After we finish the second stage (i.e., kinematic stage), we can obtain an articulated model based on the kinematic model.general template-free object representation by 3D shape recovery [3, 4, 45] . WIM [3] proposes to jointly learn a surface representation and LBS model for articulation without any supervision or prior knowledge of the structure. However, the reposing images are of low visual quality and the required training time is considerably long. AP-NeRF [4] achieves a much faster training speed by leveraging a point-based NeRF representation, but cannot support real-time rendering as well.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of this paper in Sec. 5.1. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methods: 
 Our goal is to reconstruct a reposable articulated object with real-time rendering speed from videos. The pipeline of proposed method is illustrated in Fig. 1 . We represent the appearance of the articulated object as 3D Gaussians in the canonical space while aggregating 3D Gaussians with similar motion into superpoints, which can be treated as rigid parts. It is noteworthy that we apply a time-variant 6 DoF transformation matrix to model the motion of the object. Based on these superpoints, we leverage several intuitive observations to guide the discovery of the skeleton model, which includes both joints and skeletons. Since the observations hold for most objects, our method does not require a category-specific template or pose annotations. To reduce the redundant superpoint, we propose an adaptive control strategy to densify, prune, and merge superpoints during the training process.3D Gaussian Splatting (3D-GS) [2] use a set of 3D Gaussians to represent a 3D scene. Each Gaussian G i : {µ i , q i , s i , σ i , h i } is associated with a position µ i , a rotation matrix R i which is parameterized by a quaternion q i , a scaling matrix S i which is parameterized by a 3D vector s i , opacity σ i and spherical harmonics (SH) coefficients h i . Therefore, the anisotropic 3D covariance matrix of Gaussian G i is defined asΣ i = R i S i S ⊤ i R ⊤ i, which is positive semi-definite matrix. To render images, 3D-GS employs EWA Splatting algorithm [46] to project a 3D Gaussian with center µ i and covariance Σ i to 2D image space, and the projection can be approximated as a 2D Gaussian with center µ ′ i and covariance Σ ′ i . Let Q, K be the viewing transformation and projection matrix, µ ′ i and Σ ′ i are computed asEQUATION)where J is the Jacobian of the projective transformation. Therefore, the final opacity of a 3D Gaussian at pixel coordinate x isEQUATIONAfter sorting Gaussians by depth, the color at x can be computed by volume rendering:EQUATIONwhere RGB color c i is evaluated by SH with coefficients h i and view direction.Given multi-view images with known camera poses, 3D-GS optimizes a static 3D scene by minimizing the following loss function:EQUATION) where λ = 0.2, and I gt is the ground truth. Besides, 3D-GS is initialized from from random point cloud or SfM sparse point cloud, and an adaptive density adjustment strategy is applied to control the number of Gaussians.To reconstruct an articulated object, we build the deformation based on superpoints and the linear blend skinning (LBS) [47] , while the canonical model is represented by 3D-GS.The superpoints P = {p j ∈ R 3 } M j=1 are associated with a set of 3D Gaussians, and can be used to represent the object's rigid parts. For timestamp t, we directly use deformable field Φ to learn time-variant 6 DoF transformation [R t j , o t j ] ∈ SE(3) of superpoint p j as:EQUATIONwhere R t j ∈ SO(3) is the local rotation matrix and o t j ∈ R 3 is the translation vector. Then, LBS is employed to derive the motion of each Gaussian by interpolating the transformations for their neighboring superpoints:EQUATIONwhere r t j ∈ R 4 is the quaternion representation for matrix R t j , and ⊗ is the production of quaternions. N i denotes the K-nearest superpoints of Gaussian G i . w ij is the LBS weights between Gaussian G i and superpoint p j , which can be computed as:EQUATIONwhere W ∈ R N ×M is a learnable parameter.While keeping other attributes (i.e., s i , σ i , h i ) of Gaussians the same as canonical space, we can render the image at timestamp t following Eq. 3.Treating each superpoint as a rigid part of the articulated object, we can discover the skeleton model (i.e., the 3D joints and the connection between joints) based on the motion of superpoints. Similar to WIM [3] , there are some observations to help us discover the underlying skeleton. First, if there is a joint between two superpoints p a and p b , the position of p a is more likely close to the position of p b . Second, when the relative pose between two parts changes, the joint between the two parts is relatively unchanged. Lastly, two connected parts can be merged if they maintain the same relative pose throughout the whole sequence.Let j ab ∈ R 3 be the position of underlying joint between superpoints p a and p b , and R t ab ∈ SO(3) is the relative rotation matrix between two superpoints at time t. The relative transform between p a and p b can be either represented by the global transform or the rotation of the joint, that is:EQUATIONwhere R t r = (R t b ) -1 R t a = R t ab ∈ SO(3) and t t r ∈ R 3 are the relative rotation matrix and translation vector between two superpoints respectively. Considering two joints j ab and j ba between p a and p b should be the same, we compute following distance d ab for every superpoint pair (a, b):EQUATIONwhere λ d = 1 is the hyper-parameter. To prevent the distance from changing too quickly, we smooth it across training iterations:EQUATIONwhere ϵ = 0.1 is the momentum, and τ is the training iteration.Similar to WIM [3] , we discover the structure Γ of joints based on the distance dab by the minimum spanning tree algorithm. We first select all pairs (a, b) if superpoint p b is K ′ -nearest neighborhood for superpoint p a and sort the list of dab for those pairs in ascending order. We initialize Γ as an empty set. We pick pair (a, b) from the lowest distance to the highest distance, and add this pair to Γ while there is no path between a and b. After finishing the procedure, we obtain the final object structure Γ, which is an acyclic graph, i.e., a tree. We choose the node whose length of the longest path from itself to any other node is the shortest as the root node. If there is more than one candidate node, we randomly choose one as the root.After discovering the skeleton model, we optimize the skeleton model and fine-tune 3D Gaussians by using the kinematic model. Specifically, we first predict time-variant rotations Rk ∈ SO(3) for each joint j k by using an deformable field Ψ:EQUATIONThen we forward-warp the superpoint p j from the canonical space to the observation space of timestamp t via the kinematic model. The local transformation matrix Tt k ∈ SE(3) of each joint k is defined by a rotation R t k around its parent joint j k . Consequently, the final transformation of each superpoint p j can be computed as a linear combination of bone transformation:EQUATIONwhere C j is the list of ancestor of superpoint j in skeleton model. T t root is global transformation of root. Same as Sec. 3.2, we use LBS to derive the motion of each Gaussian and render images.We use the farthest point sampling algorithm to sample M Gaussians to initialize the superpoints. Simply making superpoints learnable is not enough to model complex motion patterns. More importantly, we wish to simplify the skeleton after training to ease pose editing by reducing the number of superpoints. Following 3D-GS [2] and SC-GS [48] , we develop an adaptive control strategy to prune, densify, and merge superpoints.Prune: To determine whether a superpoint p j should be pruned, we calculate its overall impact W j = i∈ Ñj w ij , where Ñj = {i | j ∈ N i } is the set of Gaussians whose K nearest neighbors include superpoints p j . When W j < δ prune , we prune this superpoint as it is of little contribution to the motion of 3D Gaussians.Densify: Two aspects determining whether a superpoint should be split into two superpoints. On one hand, we clone a superpoint when its impact W j is greater than a threshold δ clone , indicating there is a great amount of Gaussians associated with this superpoint, and cloning such superpoints helps model fine motion. On the other hand, we calculate the weighted Gaussians gradient norm of superpoint j as: where L is the loss function, which demonstrated in Appendix A. We clone the superpoint p j if g j is greater than the threshold δ grad .EQUATIONMerge: We merge the superpoints that should belong to the same rigid part. To determine which superpoints should be merged, we first calculate the transformations T t j ∈ se3 for all superpoints at all training timestamps. Then, for each pair (a, b) of superpoints, we calculate the average relative transformations:EQUATIONwhere N t is the number of train timestamps, log() denotes the operation of converting a rigid transformation matrix to a Lie algebra. A small D a,b indicates two superpoints have similar motion patterns. Therefore, we merge two superpoints p a and p b when D a,b < δ merge and p b is the K ′ -nearest superpoints of p a .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We released the source code on GitHub, i.e., https://github.com/dnvtmf/ SK_GS.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we present the evaluation of our approach, which achieves excellent view-synthesis quality and real-time rendering speed. We also evaluate the contribution of each component through an ablation study. Additionally, we demonstrate the class-agnostic reposing capability. Please refer to our project https://dnvtmf.github.io/SK_GS/ for video visualization.To ensure fair comparison with previous work, we choose the same datasets and configurations as AP-NeRF [4] . Specifically, we choose three multi-view video datasets. First, the D-NeRF [14] dataset is a sparse multi-view synthesis dataset, which includes 5 humanoids, 2 other articulated objects, and a multi-component scene. Each scene contains 50-200 frames. We choose 6 of 8 scenes 1 The second dataset, Robots [3] ,contains 7 topologically varied robots with multi-view synthetic video. We use 18 views for training and 2 views for evaluation. The third dataset, ZJU-MoCap [34] , is commonly used for dynamic human reconstruction. Following WIM [3] and AP-NeRF [4] , we evaluate 5 sequences with 6 training views for each sequence. We use three metrics to evaluate the image quality of the novel view, i.e., peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [51] , and learned perceptual image patch similarity (LPIPS) [52] .We implement our framework using PyTorch. The number of superpoints is initialized as 512. For both deformable field Φ and Ψ, we adopt the architecture of NeRF [1] , i.e., 8-layers MLP where each layer employs 256-dimensional hidden fully connected layer and ReLU activation function. We also employ positional encoding for the input coordinates and time. For optimization, we employ the Adam optimizer and use the different learning rate decay schedules for each component: the learning rate about 3D Gaussians is the same as 3D-GS, while the learning rate of other components undergoes exponential decay, ranging from 1e-3 to 1e-5. We conducted all experiments on a single NVIDIA Tesla V100 (32GB). More implementation details are shown in Appendix A.We mainly compare our method to state-of-the-art template-free articulated methods for view synthesis, i.e. WIM [3] and AP-NeRF [4] . Besides, we also compare our method with NeRF-based and 3D-GS-based non-articulated methods. D-NeRF [14] extends NeRF to dynamic scenes by warping a static NeRF. TiNeuVox [5] improves the visual quality and training speed by using voxel grids. HexPlane [49] and K-Planes [50] accelerate NeRF by decomposing the space-time volume into several planes. Similar to D-NeRF, Deformable-3D-GS [21] extends static 3D-GS to the temporal domain. 4D-GS [20] accelerate Deformable-3D-GS by decomposing neural voxel encoding algorithm inspired by HexPlane. Similar to ours, SP-GS [22] and SC-GS [48] employ the superpoints/control points to reconstruct dynamic scenes. However, SP-GS and SC-GS can not extract skeleton from reconstructed model.In our experiments, we benchmarked our method against several baselines using the D-NeRF dataset and Robots datasets. The quantitative comparison results, presented in Tab. 1, demonstrate the superior performance of our approach in terms of both rendering speed and visual quality. Specifically, our method significantly outperforms WIM and AP-NeRF not only in visual quality but also in rendering speed. Compared to 3D-GS based dynamic scenes reconstruction models, our method not only have similar performance, but also discover the skeleton and can repose the object to generate a novel pose. Specifically, the rendering quality of ours is higher than 4D-GS [20] and SP-GS [22] , and lower than D-3D-GS [21] and SC-GS [48] . Our method also can achieve real-time rendering (>100 FPS), which is near to the rendering speed of SC-GS [48] . Fig. 2 provides the qualitative comparisons of D-NeRF dataset, which demonstrates the advantages of our method over related methods. We also provide results for Robots datasets, quantitatively in Tab. 2 and qualitatively in Fig. 3 . It is also clear that our approach is capable of producing high-fidelity novel views with real-time rendering speed. Per-scene results are shown in Appendix B.In Tab. 3 and Fig. 4 , we compare our method to WIM and AP-NeRF in the ZJU-MoCap dataset. We observe that both methods can recover the 3D shape and skeleton models. However, imperfections in the camera calibrations (see Supplement F of [53] ), lead to lower visual quality in our results compared to WIM and AP-NeRF. With respect to rendering speed, our approach achieves up to 198.23 FPS. In stark contrast, the rendering speed of WIM and AP-NeRF is extremely slow. #sp denotes the number of superpoints. The blue points denotes superpoints.In Fig. 5 , we show our model allows free changes poses and generates animating video by smoothly interpolating the skeletons posing between user-defined poses. Video examples can be found on our project webpage.We use superpoints to model the parts and motion of the object. The adaptive control of superpoints is the key to reducing the number of superpoints. Fig. 6 (a), (b) and (c) intuitively illustrate the impact of this control strategy. Without the control strategy, the distribution of superpoints becomes uneven, with sparse representation in the arm region, which negatively impacts motion modeling. The merge process in the control strategy significantly reduces the number of superpoints (97 vs 608), while the superpoints are more distributed in the motion area. Besides, as illustrated in Fig. 6 (d), L arap (in Appendix A) also plays an important role in controlling the density of superpoints. See Appendix C for more ablation studies. 
limitations: 
 We have demonstrated that our approach can achieve real-time rendering, state-of-the-art visual quality, and straightforward reposing capability by skeleton and kinematic models. However, there are some limitations to our approach. Firstly, similar to WIM and AP-NeRF, the learned skeleton model of our approach is restricted to the kinematic motion space exhibited in the input video.Therefore, the skeleton model may have significant differences from the actual one, and extrapolation to generate arbitrary unseen poses may cause errors. Secondly, our approach has similar limitations as other 3D-GS based methods for dynamic scenes. Specifically, the datasets with inaccurate camera poses will lead to reconstruction failures, and large motion or long-term sequences can also result in failures. Lastly, the paper focuses on building the kinematic model for one articulated object.Exporting build kinematic models for multi-component objects or complex scenes that contain multiple objects remains an opportunity for future research. Additionally, extending this approach to motion capture is an interesting research direction.Although our approach is universal, it is also suitable for rending novel views and poses for humans. Therefore, we acknowledge that our approach can potentially be used to generate fake images or videos. We firmly oppose the use of our research for disseminating false information or damaging reputations.We have developed a new method for real-time rendering of articulated models for high-quality novel view synthesis. Without any template or annotations, our approach can reconstruct a kinematic model from multi-view videos. With state-of-the-art visual quality and real-time rendering speed, our work represents a significant step towards the development of low-cost animatable 3D objects for use in movies, games, and education.The whole optimization progress can be divided into three stages: dynamic stage, joints discovery stage and kinematic stage.In this stage, we aim to reconstruct the appearance of the object, and find the position of underlying joints. Therefore, we employ the L joint , whose formula is:EQUATIONBesides, to obtain a good segmentation of object parts by using superpoints, we employ some regularization losses. To reduce the total number of superpoints, we encourage nearby superpoints to possess same motion patterns via as-rigid-as-possible regularization L arap :EQUATION)where N sp j is the set of K ′ -nearest neighbor superpoints of superpoint j. For the blend skinning weight, we employ L smooth to encourage smoothness by penalizing divergence of skinning weight:EQUATIONBesides, we apply L sparse to encourage sparsity so that one Gaussian is more likely associated with only one superpoint:EQUATIONIn total, our training loss of dynamic stage is:EQUATIONwhere λ = {1, 1., 10 -3 , 0.1, 0.1} in our experiments.In this stage, we conducted training for a total of 40k iterations. Similar to SC-GS [48] , our training scheme is as follows:In this stage, we aim to initialize deformable field Ψ the using the learned deformable field Φ in dynamic stage. Firstly, we fix skeleton structure Γ, and initialize the joints j k , k = 1, . . . , M -1 by underlying joints j ab according to Γ. We then cache the outputs of deformable field Φ for all timestamps in the training dataset. Next, we use cached motions of superpoints to optimize the parameters of deformable field Ψ, the positon of the joints j k by employing Adam optimizer and following loss functions:L discovery = 1 M M i=1 λ 5 (∥µ t i -μt i ∥ 2 2 + ∥ log(R t i -1 Rt i )) + λ 6 ∥(R t i p + µ t ) -( Rt i p + μt )∥ (20)where μt i and Rt i is translations and rotations calculated by deformable field Φ, while µ t i and R t i is translations and rotations calculated by deformable field Ψ and kinematic model. λ 5 = 1 controls the weights of the transform matrix difference, and λ 6 = 0.1 adjusts the weights of relative offset of the superpoints position at timestamp t.For this stage, we train for 10k iterations with a fixed learning rate of 10 -3 .In the kinematic stage, we optimize the kinematic model, joints and Gaussians, while keeping the number of Gaussians, the number of superpoints and the structure of skeleton fixed. We also conduct training for a total of 40k iterations by only using L rgb .For the deformable field Φ and Ψ, we adopt the following positional encoding for the input coordinates x ∈ R N ×3 and time t ∈ [0, 1]:EQUATIONwhere L = 10 for x and L = 6 for t.The initial number of superpoints is set to 512, while K = K ′ = 5.Tab. 4 shows the per-scene quantitative results for the D-NeRF dataset.Tab. 5 shows the per-scene quantitative results for Robots dataset.As shown in Tab. 6, we present detailed information on the optimization time and the required resources.The ablation study on the number of initial superpoint M is shown in Tab. 7.As shown in the Fig. 7 , our method struggles to accurately reconstruct the endpoints of objects. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We specify all the training and test details in Sec. 4,Sec. 3 and Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We do not report error bars or other information about statistical significance as this is not a common procedure in the field and does not contribute to understanding our evaluation.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discuss both potential positive societal impacts and negative societal impacts of this paper in Sec. 5.2. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.We exclude the Lego and Bouncing Balls scene due to errors in the Lego test split and the multi-component nature of Bouncing Balls . 
 
 Appendix:  


Decision: The paper is about reconstructing a 3D skeleton model from dynamic images with 3DGS and superpoints. The reviews were initially split, with key weaknesses pointed out. An active authors-reviewers discussion clarified all points and an acceptance consensus was reached.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel approach for reconstructing reposable dynamic 3D objects from RGB videos using Gaussian Splatting, without requiring any template as input.

To achieve this, the paper suggests grouping Gaussians around superpoints, which are intended to represent rigid parts of the scene. By optimizing and analyzing these superpoints, a full skeleton model of an articulated object in the input video can be built, refined, and used for reposing purposes.

**Details**

The approach consists of two main stages:

1. *Dynamic Stage*. After optimizing a canonical 3D Gaussian Splatting (3DGS) representation for a few iterations, a set of superpoints is initialized in the scene. A deformable field mapping each superpoint to a time-variant 6DoF transformation is optimized. These transformations are used to derive the motion of Gaussians by interpolating transformations with neighboring superpoints through linear blend skinning (LBS). The paper also proposes a gradient-based strategy to control (prune, merge, or densify) the number of superpoints in the scene. Toward the end of the dynamic stage, a skeleton structure with joints is enforced and discovered in the scene by analyzing the distance between and configuration of superpoints.

2. *Kinematic Stage*. After discovering the skeleton model of the scene, the number of Gaussians and superpoints is fixed and optimized along with a new MLP mapping skeleton joints to time-variant rotation matrices. These matrices are used to compute the motion of each Gaussian along the kinematic chains using LBS. After full optimization, the skeleton can be used for reposing and editing the reconstructed object.

The paper presents extensive experiments and demonstrates higher rendering performance and speed compared to concurrent reposable dynamic Radiance Field methods.
strength_weakness: strength_weakness: 1. The paper may lack sufficient skeleton examples to effectively demonstrate that the proposed approach can recover meaningful structures from RGB videos. Indeed, the primary goal is to recover skeleton structures and enable reposing capabilities, but only a single skeleton example is provided (Figure 5). Including more qualitative examples would likely make the paper more convincing.

2. The paper does not provide details on the optimization time and required resources (e.g., VRAM) for the proposed approach. It appears that a large number of training iterations is needed; a comparison with previous state-of-the-art models would be valuable.

3. The limitations of the approach are interesting but are only discussed in the supplementary material, which is problematic in my opinion. These limitations are crucial for further research and should be included in the main text.
questions: questions: 1. What are the optimization time and required resources (e.g., VRAM) for the proposed approach? How does it compare to state-of-the-art methods?

2. Do the authors have insights on how the method would perform in the context of a monocular video with a moving camera, which is a more realistic setting than multi-view videos?

3. In Figure 5, the skeleton appears to be quite accurate, but some bones are located outside the geometry. Would it be possible to enforce the skeleton to be located “inside” the geometry, perhaps by applying a penalty that encourages superpoints to be close to the centroid of their associated Gaussians?
limitations: limitations: As I already mentioned it, the limitations are only discussed in the supplementary material, which is problematic in my opinion. I encourage the authors to try to move the limitations to the main paper in the final version.
paper_summary: paper_summary: The paper introduces a method combining 3D Gaussian Splatting and superpoints for dynamic object modeling, achieving real-time rendering and high visual fidelity. Empirical results show that the proposed method achieves state-of-the-art results on several benchmarks.
strength_weakness: strength_weakness: 1. Although this paper achieves real-time rendering compared to AP-NERF, I find it somewhat incremental and lacking in innovation since most parts of the method are existing concepts.
2. This paper emphasizes the concept of "Reposable," but the related experiments are very limited. A thorough analysis of this aspect could effectively distinguish this paper from AP-NERF.
3. This method compares fewer baselines, and the quantitative results do not show significant improvements in rendering effects and speed compared to the baselines, as shown in Table 3, Table 4, and Table 5.
questions: questions: 1. The number of $M$ is not given in the paper, the author should do an ablation study on it.
2. The proposed method and results should be discussed with the latest relevant methods as referenced in [1].

[1] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes.
limitations: limitations: Please refer to the Weaknesses and Questions.
paper_summary: paper_summary: This paper presents a novel approach for learning articulated objects, alongside the skeletons/kinematic trees directly from input videos, eliminating the need for pre-defined meshes or hand-crafted skeleton priors.

Specifically, the paper introduces a hierarchical 3D Gaussian representation, where a set of superpoints are used as guidance for deforming the sub-points using linear blend skinning (LBS). The skeletal structure is further derived from the superpoints, providing articulated structures that enable explicit pose control. Rendering is done by Gaussian Splatting, enabling real-time performance.

Overall, the paper tackles a very challenging and useful problem for 3D modeling, the manuscript is easy to follow, and the approach is interesting.
strength_weakness: strength_weakness: The approach does have some room for improvement. Specifically,
- Limited reposability. As mentioned in L372-373, the approach is limited to the motion space in the input video. It would be great if the papers could include visual results for these failure cases. It will be interesting to see how good the learned LBS weights are.
- Evaluated on datasets with limited motions: the videos used in the paper mostly contain repetitive motion sequences, and/or with small motions. It will be interesting to see how the proposed method performs on videos with complex/diverse/large motions (e.g., AIST datasets). Also, it is similarly unclear how the method can perform on in-the-wild videos with uncontrolled lighting, or with only a single view. 

Overall, these weaknesses are very common among template-free approaches, not specifically to the proposed method itself. Nevertheless, it would be great if the paper could include more figures, visual results, and analysis regarding these cases.

There are also some issues regarding the experiments, which I detailed in the Questions section below.
questions: questions: Some comments regarding the evaluation sections:
- Tab 1, 2: does the performance gain come mainly from using a higher resolution, or does it come from “capturing better articulated structures”? While Gaussian splatting enables us to use higher resolution due to its rendering speed, it would be great if the paper could also include results with resolutions comparable to other setting (400x400 in Tab 1, and 800x800 in Tab 2). 
- Is there a way to properly evaluate how good the learned skeleton structure is? E.g., training a skeleton-based 3D articulated model using the skeleton from AP-NeRF v.s. WIM v.s. the proposed method. 

Also, one small issue:
- L147: should be (R^t_b)^-1 instead of R^t-1_b
limitations: limitations: The paper discussed some of their limitations, but it would be great if the paper could include more analysis/visual results for the issues mentioned above in the Weaknesses section.
paper_summary: paper_summary: The authors propose to combine a reposable 4D reconstruction from multi-view video based on a skeletal LBS model with 3D Gaussian splatting. To this goal they introduce a novel strategy for estimation of the skeletal model from a superpoint clustering. The results demonstrate a superior image quality and, thanks to the representation, also fast rendering.
strength_weakness: strength_weakness: W1) The skeletons are over-segmented and unnatural and likely would not be very friendly for a human animator. They may still be suitable for data fitting but do not provide nearly as much regularization as an "optimal" (ground truth) skeleton would.

W2) The limitations and broader impacts are only discussed in the Appendix which I do not see as a responsible practice. It suggests that the authors do not give downsides of the method the same importance as to the upsides.

W3) The authors claim to report Statistical Significance without further comments (checklist item #7), but I cannot see any such features in the paper.

W4) It may be a good idea to consider a higher quality captured dataset than ZJU-Mocap. It does not seem to allow for a useful comparison between the methods.


Other minor issues and suggestions:

- Figure 1: Each superpoints -> superpoint

- L145: related rotation matrix -> relative?

- $\mathbf{W}$ is overloaded in Eq. 1 and Eq. 6 for two distinct things which is not ideal.

- Eq. 13: $\mathcal{L}$ without suffix is not defined.


------------------------
**Justification of recommendation**
A solid paper with its incremental but non-trivial contribution stemming mainly from the novel skeleton construction. The experimental results are convincing and the main downside is the clutter and complexity of the recovered skeleton. Despite this, I am currently comfortable recommending acceptance under the assumption that the exposition issues are addressed (especially limitations). My final decision might change based on the rebuttal.
questions: questions: Q1) The Eq. 13 could use more discussion. Does the formulation avoid over-segmentation of large but correctly rigid parts to multiple sub-parts? It would be useful to see the LBS visualized for the final shapes.

Q2) Why are the skeletons so complex and noisy? Is that perhaps related to my other question about over-segmentation in Eq. 13? Are such skeletons practical for re-animation? How was re-animation done in the video? How many parameters / joint motions had to be defined? 

Q3) Why are there different resolutions reported for each method (Table 2 & 3). Does it mean each method was validated against a different reference image resolution? Or does it mean the methods were all tested against a full resolution reference but some were only trained using low-resolution data? That could potentially introduce considerable bias.

Q4) The authors also do not provide any failure case examples/images. Does the method work perfectly for all tested scenes? What about the robots from the limitation Figure 6 of AP-NeRF? Does the new method handle these cases better?
limitations: limitations: The limitations and broader impacts are only discussed in the Appendix which I do not see as a responsible practice. It suggests that the authors do not give downsides of the method the same importance as to the upsides.

Comments: title: Performance Improvement
Thank you for your feedback
Thanks!
Thanks for the detailed responses
Thank you
comment: Thanks for the author's response and the experimental validation done in a short period of time, which demonstrates the practical value and potential of the method. I also agree that optimizing the skeleton structure of the object is meaningful. 

I recommend that the author review the aforementioned work on video-based skeleton optimization and consider follow-up work, as it would be meaningful for assessing the overall contribution of the paper. Additionally, having reviewed other reviewers' feedback, I decided to raise my score. However, I believe that the method does not introduce new approaches or achieve state-of-the-art performance in either dynamic Gaussian reconstruction or skeleton optimization, so I lean towards Borderline Accept.
After intensive work, we have partially succeeded in integrating SC-GS into our work. The performance of the proposed method is greatly improved. Please refer to the tables below:

**PSNR**: 

|  scene       | average |  hellwarrior |  hook  |  jumpingjacks |  mutant |  standup  | trex    |
| ----             |  ---------- | --------------  | -------- | ------------------ | ---------- | -----------  | -------  |
| ours (old)   | 34.31    |  37.59         | 33.73  |  32.09            | 36.70    |  35.91      | 29.82 |
| ours (new) | 39.07    | 39.69          | 36.35  | 37.38            | 41.27    | 43.12       | 36.58 |
| SC-GS       | 43.04    | 42.93           | 39.87  | 41.13            | 45.19    | 47.89       | 41.24 |

**SSIM**: 

|  scene       | average |  hellwarrior |  hook   |  jumpingjacks |  mutant |  standup  | trex     |
| ----             |  ---------- | --------------  | --------  | ------------------ | ---------- | -----------  | -------   |
| ours (old)   | 0.9751   | 0.9667       | 0.9742 |  0.9746           | 0.9837  | 0.9797   | 0.9714 |
| ours (new) |  0.9876  | 0.9767       | 0.9829 |  0.9885           | 0.9927  | 0.9934   | 0.9912 |
| SC-GS       | 0.9975   | 0.994          | 0.997    | 0.998             | 0.999    | 0.999     | 0.998   |

Due to time constraints, we were unable to perfectly integrate SC-GS into our work, and there is still a lot of room for improvement. But the results in the above table are sufficient to provide evidence for the assert we discussed above.
 
After the discussion period, we will continue working on improving the performance by incorporating SC-GS and update the 
improved results in the final revised paper.

Thank you for your time and feedback again.
Thanks for your feedback.

There are significant differences between SC-GS's motivation and ours:
- SC-GS is proposed to solve the novel view synthesis problem for dynamic scenes, which introduces sparse control points together with an MLP for modeling scene motion. The main motivation is that we can use a sparse set of base to represent motions within the dynamic scene. Therefore, SC-GS does not involve articulated information.
- Our approach focuses on the task of extracting explicit skeletons from images. According to the methods you listed, this task is important and challenging. And the motivation of ours is that we  can extract explicit skeleton from the learned motions of superpooints. 

Compared with the listed methods, our method **does not require any priors**, such as templates, 3D datasets, etc. Specifically,
1. To recover the skeletal shape of an animal from a monocular video, CASA[1] first retrieves the most relevant articulated shape from a 3D character assets bank.
2. To reconstruct, rig, and animate the 3D objects from single images, Object Wake-Up[2] trained on the dataset with rigged 3D characters. The used “ModelsResource-RigNetv1” dataset only focuses on limited categories, e.g., chair, sofa, desk, and table.
3. While RigNet [3] aims to produce animation rigs from input character models, it also requires the dataset of 3D articulated characters to train the model.

We are working to incorporate SC-GS into our work to provide further evidence.
Thanks for your response.  I understand that the rebuttal time is limited, and I also acknowledge the technical differences between this work and SC-GS. 

However, I believe the overall motivation remains the same, which is to incorporate articulated information into 3DGS to control dynamic generation. The potential issues with SC-GS mentioned by the authors, as well as the performance gap, are not supported by any experimental evidence, so they fail to convince me. 

Additionally, if the focus of the paper is on obtaining the object's skeleton, there are many methods, such as [1], [2], and [3], that can extract heterogeneous skeletons from videos or meshes.

Finally, I will maintain my score unless there is additional evidence that can prove the points raised by the authors.

[1] CASA: Category-agnostic Skeletal Animal Reconstruction, NIPS 2022.

[2] Object Wake-Up: 3D Object Rigging from a Single Image, ECCV 2022.

[3] RigNet: Neural Rigging for Articulated Characters, TOG 2020.
First of all, although SC-GS uses control points to implement motion editing, there are no constraints between the control points. Therefore, compared to re-pose with skeleton, the motion editing in SC-GS :
-  1) is easy to generate physically unrealistic motion, and thus render unrealistic images;
-  2) need an additional optimization process, which is more complex and time-consuming.

We believe that discovering the skeleton of an object will be helpful in certain fields, such as game production, video generation, physical simulation, etc.

Second, it must be pointed out that the design of LBS is **not** the contribution of our paper. Our main contribution is to discover the skeleton of an articulated object by utilizing the motion of superpoints.

Third, the main reasons for the performance gap are as follows:
- To simplify the skeleton, the proposed adaptive control strategy significantly reduces the number of superpoints. ($512\to \sim 100$). The number of superpoints has an impact on performance.
- According to the code of SC-GS, SC-GS uses more tricks than ours, such as longer training time (80k), an extra MLP for input time (called time-net), additional hyper-features to help k-nearest neighbor search, the initialization train for control points and so on.

We are incorporating SC-GS into our work to achieve even higher performance.

If you have any further questions or would like additional clarification, please do not hesitate to contact us. We would be more than happy to provide additional information or discuss any aspect of our work in greater detail. Your feedback is deeply appreciated, and we remain fully committed to addressing any concerns you may have.
Thank you for the detailed response. I have carefully read your reply but still have some concerns. 

Firstly, regarding the novelty of the work, while it may be the first to propose the concept of a skeleton in 3DGS, SC-GS has already introduced the use of control points in 3DGS, although it did not explicitly calculate the connections. This significantly impacts the novelty of the work. 

Secondly, the difference in LBS design mentioned by the authors as a distinction from SC-GS seems to be a relatively minor modification. 

Lastly, and more critically, in the PDF provided by the authors, there is a significant performance gap between the proposed method and SC-GS, which makes me question the rationale behind these design differences. 

In conclusion, I remain negative about the work's overall contribution.
I would like to thank the authors for the efforts they made during the rebuttal as well as the answers to my questions.

I think the paper tackles a very challenging task (recovering skeletons with a template-free optimization pipeline) and provides an interesting solution relying on 3DGS. 

Even though the output skeletons are still over-complicated for potential applications in animation, the proposed approach is certainly a step forward for the field.
For these reasons, I recommend acceptance and decide to maintain my initial rating.
After going through the author's responses, as well as the comments from fellow reviewers. The experimental results, along with the rebuttal responses, adequately support the claim in the paper, and I appreciate how the work neatly combines existing techniques/concepts to capture the 3D appearance and, most importantly, the motion structure. Therefore, I stand by my original rating -- Weak Accept.
Thank you for the rebuttal, I have no further questions at the moment.

The visualization of the LBS weights in the PDF is useful and while it rather shows limited physiological plausibility of the animation tree, it would be good to include it in the paper to visualize any such limitations.