Title: MIDGArD: Modular Interpretable Diffusion over Graphs for Articulated Designs
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D articulated objects

diffusion models

generative models
Abstract: Providing functionality through articulation and interaction with objects is a key objective in 3D generation. We introduce MIDGArD (Modular Interpretable Diffusion over Graphs for Articulated Designs), a novel diffusion-based framework for articulated 3D asset generation. MIDGArD improves over foundational work in the field by enhancing quality, consistency, and controllability in the generation process. This is achieved through MIDGArD's modular approach that separates the problem into two primary components: structure generation and shape generation. The structure generation module of MIDGArD aims at producing coherent articulation features from noisy or incomplete inputs. It acts on the object's structural and kinematic attributes, represented as features of a graph that are being progressively denoised to issue coherent and interpretable articulation solutions. This denoised graph then serves as an advanced conditioning mechanism for the shape generation module, a 3D generative model that populates each link of the articulated structure with consistent 3D meshes. Experiments show the superiority of MIDGArD on the quality, consistency, and interpretability of the generated assets. Importantly, the generated models are fully simulatable, i.e., can be seamlessly integrated into standard physics engines such as MuJoCo, broadening MIDGArD's applicability to fields such as digital content creation, meta realities, and robotics.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a diffusion-based framework for generating articulated objects represented as part graphs. Different from prior work [35] that generates both the part shapes and graph structures simultaneously, this work introduces a two-stage strategy that first generates the structure with per-part conditions and then generates the part shapes. It also proposes a part latent encoding that enables multimodal conditions including text and images.\n\nSTRENGTHS:\n- The reparameterization of the Plucker coordinates is simple but more natural than that in [35], making the diffusion results better satisfying the constraints.\n- The designed node embedding with image and text features is not only a more detailed representation of part shapes, but also enables more flexible conditioned generation for different downstream applications.\n- The framework relaxes the constraints of training on canonical-posed shapes. This is an important relaxable for better usage of data.\n\nWEAKNESSES:\n- As also briefly discussed in the paper, generating the parts independently is a bit unnatural to this problem, especially given the importance of inter-part motions/relations in articulated objects. But overall I am OK with this point given all the node conditions in the framework.\n- Quantitatively, the improvements compared to [35] on unconditioned generation seem marginal.\n- The description \"simulation-ready\" sounds overclaimed to me. When talking about \"simulation\", I feel most people will expect physical viability and accuracy. But from the demonstration on the website, it seems that the parts are not well connected. There is also no explicit physics enforcement in the framework.\n\nMinor things:\n- I believe it would be better to swap the two sentences \"This image latent is derived by\" (line 148) and \"...the structure generator denoises a latent representation of an image...\" (line 150), so the readers know where the \"image latent\" in line 148 comes from.\n\nQUESTIONS:\nI am curious about some details:\n- Sec. 3.4 says \"During training, the model uses renderings of the part mesh from a frontal view\" (line 217). How is the frontal view chosen? For inference-time image-conditioned generation, do the images have to be in frontal view, or can they be of any view? Are the images in Suppl. Fig. 5 of \"frontal view\"?\n- I wonder how exactly are the bounding boxes represented and predicted. Because for articulated objects, the part poses are changing during part motions. [35] only uses the rest-state bounding boxes (together with the joint limits). In this work, is the rest-state bounding box viewed as having the identity pose matrix?\n\nLIMITATIONS:\nLimitations are well-discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the problem of generating articulated 3D assets that are animatible. The authors mention that their generated shapes are directly compatible with existing physics simulation tools, i.e. MuJoCo. To this end, they first propose a structure generator, which conditionally or unconditionally generates an articulation graph encoding structural object information such as the kinematics attributes, i.e. the kinematic tree and joint types. Next, the multi-modal shape generator synthesizes a shape that follows the kinematic tree and size of the object while also allowing conditioning on multi-modal inputs such as text and images. Results demonstrate some improvement over the SOTA (NAP).\n\nOverall, I like the setting that the authors propose as well as their technical contribution to solve it. There are a few minor concerns about the presentation in terms of writing, clarity, and qualitative visualizations (see comments above). However, I believe those are minor. Thus, I recommend acceptance.\n\nSTRENGTHS:\n- Interesting setting with a lot of potential for future work. It seems only very limited amount of works (NAP) have explored this setting\n- Evaluation is to the best of my knowledge complete.\n- Well-written related work section and references seem to be complete\n- The paper is technically sound\n\nWEAKNESSES:\n- Writing:\n    - I would limit the contribution bullets to the technical contribution of this work rather than focusing on results and open sourcing.\n- Clarity\n    - I would introduce proper notations for all steps discussed in section 3.3 as I feel it is hard to understand the setting just from the textual descriptions, e.g. what is input, what is output.\n    - I have a similar concern regarding section 3.4\n- Video\n    - I would have expected to also see some video results of the generated and animated objects. While this is not strictly required, it would have made the exposition much more complete.\n\nQUESTIONS:\n--\n\nLIMITATIONS:\nLimitations are sufficiently discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper proposes several interesting improvements over existing articulated object modeling and highlights higher-quality part generation.

Specifically, an articulated object is parameterized to a graph where parts are nodes and joints are edges. This paper proposes a multi-model part VAE for generating higher quality and controllable shapes and improving physical awareness through a bounding box alignment. It also improves the kinematic structure generation by considering parameterize on the plucker manifold. With these technical contributions, the results show significant improvement over baselines and the supplemental repo provides simulatable mujoco files, showing the practical value of this work.

I recommend a clear acceptance of this paper.

STRENGTHS:
- Good results, supp MuJoCo simulation: I really like the MuJoCo demo in the suppl repo, which is impressive.
- Several technical solid and important improvements including the shape representation, plucker manifold encoding, and bounding box alignment. etc
- Multimodel information and condition: i specifically like the text and language part of the part shape encoding, which opens a lot of opportunities for conditioned generation and connection to LLM.

WEAKNESSES:
- More conditioned generation results: while the part-level multimodel information is used in shape encoding, it would be better to show more conditioned generation results using this information.
- Open source the part text and bounding box alignment data: it's not clear whether the shape part encoding training data will be publicly released or not.

QUESTIONS:
Please see the weakness

LIMITATIONS:
Adequately discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the task of 3D asset generation for articulated objects.\n\nThis work aims to enhance the prior approach by achieving three main objectives: 1) increasing the interpretability and controllability of the generation process; 2) generating more natural joint motions and 3) reducing violations of physical constraints.\n\nFor the first goal, they propose a framework MIDGArD to decompose the generation into two sequential modules: 1) a structure generator that unconditionally generates a kinematic graph and attributes for each articulated part; 2) a shape generator to produce part SDF that allows conditioning on the graph from the first module, an image, and a text description for each part.\n\nFor the second goal, they introduce a representation for joint parameters that allows the diffusion process to operate on the Plucker manifold directly.\n\nFor the third goal, they replace the AABBs with OBBs to bound the part shape which is claimed to achieve better part alignment.\n\nSTRENGTHS:\n- This work contributes to an increasingly important area and identifies three valuable perspectives for articulated object generation.\n- The proposed framework that separates the generation of the articulation structure and part geometry by using images as an intermediate proxy allows users to control the part shape more explicitly, compared with the prior work NAP [35] which requires modifying the latent code instead. \n- Both quantitative and qualitative evaluations are provided to demonstrate improved quality of data distribution modeling in the unconditional generation setting.\n\nWEAKNESSES:\n- The technical contribution is limited: \n\t- The network architecture of the structure generator is the same as NAP [35]. The contribution for this part is only the representation alternation for joint parameters and bounding boxes. These are reasonable changes for slightly better distribution modeling, but the insight is not particularly significant.\n\t- The shape generator is also adapted from an existing work SDFusion [7], where the benefits of detailed geometry modeling with TSDF and the flexibility of multimodal conditioning are just inherited from the original work. The only modification is an additional conditioning from a graph. However, how this graph condition affects the part geometry output and whether it is necessary is unknown. Also, there is no strong/sufficient qualitative evidence to show a better geometry generation compared to NAP.\n- The experiments and evaluation are insufficient and unclear:\n\t- The side-by-side qualitative comparison with prior work (e.g. NAP [35], CAGE [43]) is missing to convince the improvement in the claimed aspects.\n\t- Unfair comparison with NAP in the “reconstruction” setting in Table 1. According to the description in lines 280-282, NAP is only provided with graph topology and motion attributes for each node. In this case, there is no way for NAP to reconstruct the object with no shape information (which should have been given the latent code extracted from the ground truth part geometry). In contrast, MIDGArD is additionally provided with shape features extracted from the ground truth node images.\n\t- Lack of quantitative and qualitative results to support the argument of more “natural joint motion” and “less physical constraint violation” and how these improvements are correlated with the specific designs. Other ablation studies discussed in the manuscript are also generally hard to follow.\n- The related work section does not contextualize this work very well.\n\nQUESTIONS:\n- The argument in lines 47-48 “leveraging categorical embeddings for improving consistency across the graph” is unclear. What does consistency mean exactly? The ablation experiment to support this argument is missing.\n- Would it be possible to compare with CAGE [43] on certain aspects of the generation? E.g., in terms of the physical plausibility of objects in both abstraction level (bounding boxes) and mesh output. \n- The argument in line 33 “both NAP [35] and CAGE [43] provide no control on the part level due to their opaque encoding as a latent” is a bit misleading. What specific control on part is missing from the prior work? Also, to correct the fact, the representation used in CAGE has no latent encoding for any attribute, while the representation used in NAP only makes the geometric feature encoded as latent and leaves others in an explicit form.\n- How robust and generalizable the image control is?\n\nLIMITATIONS:\n- Under the formulation of this work, the image control from users is not particularly practical:\n\t- The image of a single articulated part is not natural to find easily.\n\t- The control is more of a post-editing. The user has to wait to see what object is generated and then find compatible part images to feed into the corresponding node in the graph. Otherwise, it won’t be matched with the articulation parameters and the part arrangement being generated.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-16"}