Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting

Authors: Yu Liu, Baoxiong Jia, Ruijie Lu, Junfeng Ni, Song-Chun Zhu, Siyuan Huang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: articulated object modeling

Abstract:
Building interactable replicas of articulated objects is a key challenge in computer vision. Existing methods often fail to effectively integrate information across different object states, limiting the accuracy of part-mesh reconstruction and part dynamics modeling, particularly for complex multi-part articulated objects. We introduce ArtGS, a novel approach that leverages 3D Gaussians as a flexible and efficient representation to address these issues. Our method incorporates canonical Gaussians with coarse-to-fine initialization and updates for aligning articulated part information across different object states, and employs a skinning-inspired part dynamics modeling module to improve both part-mesh reconstruction and articulation learning. Extensive experiments on both synthetic and real-world datasets, including a new benchmark for complex multi-part objects, demonstrate that ArtGS achieves state-of-the-art performance in joint parameter estimation and part mesh reconstruction. Our approach significantly improves reconstruction quality and efficiency, especially for multi-part articulated objects. Additionally, we provide comprehensive analyses of our design choices, validating the effectiveness of each component to highlight potential areas for future improvement.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730026933158

thread_type:
official_review

<thread_object>
creator: reviewer_x9y2

<content>
summary:
This paper presents a novel method for interactable articulated object modeling. The method leverages the skinning-like motion priors between object parts and the correspondences between two-state objects to model part-aware articulated objects. Results show that the results outperform existing works in both well-used benchmarks and datasets created by the author containing complex multi-part objects.

strengths:
+ The writing of the paper is easy to follow. 
+ The motivation of the paper is reasonable. By combining explicit 3D Gaussian modeling with proper articulation modeling, the method enables better articulation parameter estimation and object reconstruction.
+ The design for initialization and part segmentation based on different states and object parts is reasonable. 
+ Extensive experiments are conducted to evaluate the effectiveness of the method presented.

weaknesses:
- The method is designed for two-state objects. It is uncertain how this can be applied to object captures with more states. Moreover, the number of parts is set manually as hyperparameters making it difficult to scale to different objects.
- It seems the method combines existing designs to achieve 3DGS-based articulated modeling. Though it is not super interesting, the overall pipeline makes sense. However, I am not sure what is the argument for using 3DGS compared to NeRF and MVS point clouds.  
- The authors only compare the method with DTA for visualization and with DTA and PARIS on DETA-Multi and ArtGS-Multi. Is it possible to compare with more methods? For example, some NeRF-based methods. Some related suggestions: 
 [1] Real2Code: Reconstruct Articulated Objects via Code Generation; [2] Articulated Object Neural Radiance Field; 
- More visualization results are essential for understanding the method to complement Fig. 4. For example, how well does the matching process and refinement work? I think these visualizations are vital for multi-stage methods. 
- I am not sure why perception-based metrics are not reported for evaluations, e.g., PNSR and LPIPS for 3DGS evaluation. 
- Though the paper is motivated by interactive articulation objects, I did not see related evaluation and applications for it. 
- I did not see discuss about failure cases and limitations.

questions:
- What is the comparison baseline for w/o Cano. init? Does it work if we just take one-state GS as initialization?  I think it should be ok as two-state is only w.r.t. some partial rigid transformations. 
- Can you explain the gap between obvious improvement in articulation parameters estimation and minor improvement in overall, geometry modeling?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730294193286

thread_type:
official_review

<thread_object>
creator: reviewer_ax3x

<content>
summary:
- This paper works on the task of articulated object reconstruction from multi-view RGB-D images.
- This work proposes to leverage 3D Gaussian as the underline representation to reconstruct articulated objects with multiple parts in a self-supervised way.
- Along with the representation, several strategies are proposed to improve the reconstruction accuracy, such as a coarse-to-fine initialization for part geometry and skinning-inspired dynamics modeling for articulation estimation.
- Experiments demonstrate the effectiveness and robustness of the proposed method.

strengths:
-	This work is contributing to an increasingly important research problem of building digital twins for arbitrary articulated objects with complex structure.
-	The experiments are showing promising results overall.
-	The observation and analysis of the experiment results are well discussed.
-	This work is well contextualized in the first two sections.

weaknesses:
-	**The writing of the Method section should be improved**. The quality of the writing overall is great, but the method section requires further improvement to articulate the details of the approach, especially for the logic of the presentation and usage of mathematical symbols. The current form hinders the understanding of the proposed method. Please see more details for clarification in the `Question` section.
-	**The experiment evaluation needs a bit more care**. 
	-	There are at least 17 highlighting of numbers in `Table 1` is misplaced, e.g., PARIS*[Axis Ang, Real Objects-Fridge] in the second row should be the second best.
	-	The “Axis Pos” reported in `Table 5` for synthetic objects is almost no difference given the limited significant figures. It would be more meaningful to show the metrics with more significant figures or report the numbers by timing 10 or even 100. 
-	**More qualitative results**. The presentation of the experiment would benefit from showing more qualitative results (in different complexity) that compares with the baseline methods, which can visually demonstrate the gap revealed in the metric evaluation.
-	**More discussion of the failure cases and limitations**. For the completeness of the paper, the failure cases and discussion of the limitation accordingly should be incorporated. There are assumptions that are not explicitly mentioned in the paper, e.g., the number of parts is given, the two states of the observation are pre-aligned in the world/object coordinate system. Maybe there are other assumptions that can be discussed, e.g., whether the articulated parts are required to be opened to expose visibility as much as possible.

questions:
The below are some details of the approach or implementation that would benefit from clarification.

-	In Section 4.1:
	-	Once $ \mathcal{G}_{init}^c$ is initialized, are the Gaussians labeled as static/dynamics in binary or part ids?
	-	Since the refinement of $ \mathcal{G}_{coarse}^c$ only involves the static part, would the region of the movable part be very sparse? Does this sparsity remain for the optimization step later? Would it affect the reconstruction quality in the end?
	-	How many Gaussians are initialized in $ \mathcal{G}_{init}^c$?
-	In Section 4.2:
	-	What are the learnable centers $C_k$ useful for by intuition? What is $\mathbf{R}_k$ in $C_k$? Is $\mathbf{R}_k$ related to the articulation for the specific part?
	-	In equation 6, the variables $X_i^k$,  $\mu_i^c$, and $\mathbf{W}_\Delta$ are not defined.
	-	In lines 256-257, it is unclear how the distance matrix is related to part assignment. It is also unclear why identifying the sharp boundaries is the unique challenge in this case.
	-	In lines 258-260, why introducing $\mathbf{W}_\Delta$ is helpful in tackle the challenge?
-	In Section 4.3:
	-	In equation 7, $M_{ik}$ is undefined.
	-	In equation 8, $\mathbf{r}_i^c$ is undefined.

The below are some missing references for citation.
- [1] Yan, Zihao, et al. "RPM-Net: recurrent prediction of motion and parts from point cloud." arXiv preprint arXiv:2006.14865 (2020).
- [2] Jain, Ajinkya, et al. "Screwnet: Category-independent articulation model estimation from depth images using screw theory." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
- [3] Lei, Jiahui, et al. "Nap: Neural 3d articulated object prior." Advances in Neural Information Processing Systems 36 (2023): 31878-31894.
- [4] Liu, Jiayi, Manolis Savva, and Ali Mahdavi-Amiri. "Survey on Modeling of Human-made Articulated Objects." arXiv preprint arXiv:2403.14937 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626771360

thread_type:
official_review

<thread_object>
creator: reviewer_hluc

<content>
summary:
This paper presents ArtGS, an innovative method for reconstructing multi-part interactive objects. ArtGS utilizes 3D Gaussian Splatting combined with a coarse-to-fine standard Gaussian initialization strategy to achieve high-precision reconstruction of dynamic scenes.

strengths:
The key highlight of the paper is the innovative introduction of 3D Gaussian Splatting (GS) and superpoints, along with a coarse-to-fine Gaussian initialization strategy. Additionally, the method achieves state-of-the-art results on existing datasets, particularly excelling on a newly introduced dataset.

weaknesses:
The final visual renderings demonstrated in the paper primarily focus on cabinet-like objects, lacking validation on other categories. To better assess the generalizability of the method, I suggest adding renderings for a broader range of objects, such as chairs, scissors, and staplers, which have distinct motion components and structural differences. This would provide a more comprehensive evaluation of the method's applicability and accuracy across diverse objects.

Moreover, I have some doubts regarding the main contributions and novelty of this work. The success of this method may largely derive from elements such as the coarse-to-fine training strategy, joint optimization of static shape and articulation, the use of Gaussians instead of NeRF, and super-part rigging—all of which have been explored in previous research but are integrated here for this specific task. I would appreciate if the authors could further clarify their unique contributions.

Lastly, there is a lack of video examples to visually demonstrate the interaction of dynamic parts. Including video results would significantly help in understanding the method's advantages in complex dynamic scenarios.

questions:
Please refer to the weaknesses section for further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701776544

thread_type:
official_review

<thread_object>
creator: reviewer_jgzh

<content>
summary:
This paper introduces a part-aware articulated object reconstruction method with 3D Gaussian splatting. The method reconstructs the shape and articulation from two states of multi-view RGB-D images with a coarse-to-fine Gaussian initialization. A skinning-related center-based part modeling is designed to automatically assign Gaussians to statics and movable parts. The articulation type is learned through self-guided parameter learning.

strengths:
1. The paper is well-organized and the overall writing is good.

2. The optimization speed is improved by using 3D Gaussian splitting.

3. Center-based part modeling and assignment allow the model to assign Gaussian to different movable and static parts. The use of part centers allows multi-movable parts compared to PARIS and does not require prior part segmentation knowledge compared to DTA. 

4. The self-guided motion priors with part dual-quaternions improve articulation learning with a further heuristic joint type prediction.

5. The experiments and ablation studies are extensive.

weaknesses:
1. Only state 0 and state 1 results are presented, the authors should present more intermediate states like Figure 8 in PARIS where they illustrated results on t=0.25, t=0.5,  and t=0.75.

questions:
1. Have you tried to present the results when different movable parts are in different states?  Since this method models multiple movable parts with distinct part centers and dual quaternions, I believe it is possible to do that. It would be interesting to see those results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732288737826

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to the reviewers

comment:
We sincerely thank reviewers JGzH, HLUC, AX3x, x9y2 for their valuable feedback, especially highlighting our contributions on reasonable motivation (x9y2), extensive experiments and analyses (JGzH, AX3x, x9y2), effective designs for canonical Gaussians and part assignment module (JGzH, x9y2), and innovative initialization strategies (HLUC, x9y2). We follow comments and suggestions from all reviewers and revise our manuscript (colored in blue) accordingly as follows:

1. **More Visualization Results**. (Reviewer JGzH, HLUC, AX3x, x9y2) 
- We have created a [anonymous project website](https://artgs-iclr.github.io/artgs-iclr/) with interactive demonstrations and additional results: 
    - Simulation video of reconstructed meshes in IsaacSim.
	- Interactable meshes of complex synthetic and real-world objects.
	- Rendering results of all objects at interpolated states.
	- Qualitative comparison of rendering results on various objects at interpolated states and novel views.
	- Please note that as requested by the reviewers, we have included numerous visualization results, which may require some loading time. 
- We will provide more visualizations of the initialized canonical Gaussians and optimized Gaussians in the appendix. (Reviewer x9y2)


2. **Additional Experiments and Analysis**.
- We provide the values of Axis Pos multiplied by 1000 in Tab.6 in the appendix. (Reviewer AX3x)
- We added perception-based metrics (PSNR, LPIPS, SSIM) in Tab.7 in the appendix. (Reviewer x9y2)

3. **Failure Cases**. (Reviewer AX3x, x9y2)

We will add a visualization of failure cases in the appendix soon and we describe them here.
- For real-world objects with multiple parts, some part centers derived from clustering may be inaccurate, and these incorrectly initialized centers may not be corrected during the optimization process, resulting in degraded performance for parts with misaligned centers. However, we found that if we manually correct the wrong part centers before training (i.e. modifying the position of the wrong part centers), it works well.
- Similar motion. If two parts move in the same way in two states, such as two drawers being pulled apart together, our method may fail.

4. **Contributions and Novelty Clarification**. (Reviewer HLUC) 
- Our key innovation lies in solving the challenging problem of simultaneous optimization of canonical Gaussians, part assignment, and articulation parameters. Our method's success stems from:
    - Canonical Gaussians that effectively bridge 2 object states.
    - A spatial-aware part assignment module that enables accurate unsupervised part-segmentation.
    - Novel initialization strategies for canonical Gaussians and part centers that prevent optimization collapse.
- As acknowledged by all reviewers, with all the innovations proposed in this paper, ArtGS achieves:
    - State-of-the-art performance across diverse object categories.
    - 3-4x faster training compared to the previous SOTA method.
    - No requirement for priors from pre-trained models.
    - handling complex multi-part objects (up to 7 parts).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a method for part-aware articulated object reconstruction using 3D Gaussian splatting, with coarse-to-fine Gaussian initialization and self-guided motion priors. The reviewers agree on the strengths of the method, including its novel approach of part modeling, and integration of articulation learning. Extensive experiments validate its effectiveness and robustness. The coarse-to-fine initialization also improves its reconstruction quality.  During the discussion period, concerns from the reviewers were resolved, such as more qualitative results. However, concerns still remain about limited intermediate state results, writing clarity in the methods section, and broader generalization to diverse objects. I'd recommend the authors to address them in the revision.