MIDGArD: Modular Interpretable Diffusion over Graphs for Articulated Designs
Authors: Quentin Leboutet
Nina Wiedemann
zhipeng cai
Michael Paulitsch
Kai Yuan
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D articulated objects
diffusion models
generative models
Primary area: generative_models
TL;DR: We present MIDGArD, a new generative framework for creating 3D articulated objects, separating structure and shape generation.

Abstract: Providing functionality through articulation and interaction with objects is a key objective in 3D generation. We introduce MIDGArD (Modular Interpretable Diffusion over Graphs for Articulated Designs), a novel diffusion-based framework for articulated 3D asset generation. MIDGArD improves over foundational work in the field by enhancing quality, consistency, and controllability in the generation process. This is achieved through MIDGArD's modular approach that separates the problem into two primary components: structure generation and shape generation. The structure generation module of MIDGArD aims at producing coherent articulation features from noisy or incomplete inputs. It acts on the object's structural and kinematic attributes, represented as features of a graph that are being progressively denoised to issue coherent and interpretable articulation solutions. This denoised graph then serves as an advanced conditioning mechanism for the shape generation module, a 3D generative model that populates each link of the articulated structure with consistent 3D meshes. Experiments show the superiority of MIDGArD on the quality, consistency, and interpretability of the generated assets. Importantly, the generated models are fully simulatable, i.e., can be seamlessly integrated into standard physics engines such as MuJoCo, broadening MIDGArD's applicability to fields such as digital content creation, meta realities, and robotics.

MIDGArD: Modular Interpretable Diffusion over Graphs for Articulated Designs
Quentin Leboutet
Nina Wiedemann
Zhipeng Cai
Michael Paulitsch
Kai Yuan
Intel Labs - XRL - eXtended Reality Laboratory
(firstname lastaasttm Lastname)@intel.. com
Abstract
Providing functionality through articulation and interaction with objects is key objective 3D generation. We introduce MIDGArD (Modular Interpretable Diffu- sion over Graphs for Articulated Designs), novel diffusion-based framework for articulated 3D asset generation. MIDGArD improves over foundational work in the field by enhancing quality, consistency, and controllability in the generation process. This is achieved through MIDGArD's modular approach that separates the problem into two primary components: structure generation and shape generation. The structure generation module of MIDGArD aims at producing coherent articula- tion features from noisy or incomplete inputs. acts on the object's structural and kinematic attributes, represented as features of graph that are being progressively denoised issue coherent and interpretable articulation solutions. This denoised graph then serves as an advanced conditioning mechanism for the shape generation module, a 3D generative model that populates each link of the articulated structure with consistent 3D meshes. Experiments show the superiority of MIDGArD on the quality, consistency, and interpretability of the generated assets. Importantly, the generated models fully simulatable, i.e., be seamlessly integrated into standard physics engines such as MuJoCo, broadening MIDGArD's applicability to fields such as digital content creation, meta realities, and robotics.
VQVAR
Night Table Body
Drower
Sequetiaa
Articulated Assct
Drawer
Doot
Revolute
Priemnticc
Trooded Iusagos
VQVAB
Partial
Demoland
Information
GAT
(T-1yx Denoóm
StructureGemerator
Shape Generator
Figure The MIDGAID generative pipeline.
Introduction
Despite their significance applications in meta realities and embodied AI, the creation of 3D models of articulated objects remains a manual endeavor, and existing datasets [92, 97, 65, 102, 47, 63, 15] are often limited in both scope and scale. Human designers utilize strong prior knowledge of object geometry and kinematic structures, capability that automated systems have yet to fully replicate.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

While extensive research has been conducted on generative models for static 3D objects [78. 74, 50, 48, 35. 8, 39, 54, 2, 95, 103, 24, 68, 9, 88, 75, 93, 6, 85, 98, 77, 52] and scenes [38. 31, 17, 71, 84, 14, 72, 53, 7, 12, 107], integrating 3D part geometry with kinematic structures has received far less attention. Generating articulated objects that are both geometrically detailed and kinematically accurate presents unique challenges due to the complexity of modeling part interactions and motions. Recent research efforts, such as Neural 3D Articulation Prior (NAP) [37] or Controllable Articulation GEneration (CAGE) [46], started addressing this gap, pioneering direct end-to-end generation of complete articulated assets using a denoising diffusion process acting on articulation graphs. However, generating consistent objects with high-fidelity shapes remains challenge. Identified shortcomings of NAP [37] include 1) unnatural motion caused by unconstrained screw- joint parametrization, 2) limited controllability, 3) limited interpretability and 4) the generation of inconsistent or unidentifiable shapes. Additionally, both NAP and related approaches lack part-level control due to the opaque encoding as latent within the graph, highlighting the need for enhanced interpretability.
In this work, we introduce MIDGArD, a novel framework designed to generate interpretable and simulatable 3D articulated assets in a controllable manner. MIDGArD addresses the limitations of previous methods by offering: 1) improved consistency in joint motion, 2) enhanced controllability of the generation process, 3) increased interpretability and 4) better overall asset quality. Furthermore, the 3D assets generated by MIDGArD are fully simulatable and can be seamlessly integrated into virtual environments or physics engines such as MuJoCo [86], significantly enhancing its utility for digital content creation and robotics. MIDGArD employs a modular and sequential approach based on two diffusion models: the structure generator and the shape generator (see Figure 1). The structure generator denoises an articulation graph, unconditionally or from incomplete heterogenous inputs. This articulation graph acts as abstract, yet interpretable object representation, encoding the structural and semantic information of every link, as well as kinematic attributes.
MIDGArD overcomes the identified limitations of prior works by 1) leveraging categorical embed- dings for improving consistency across the articulation graph; 2) representing nodes (i.e. unique object parts) via latent image codes rather than direct mesh encoding; 3) diffusing on the Plücker manifold, thereby eliminating the need for discretizing the generated motion parameters, and 4) accounting for the orientation of the parts for more consistent generations. As additional benefit, these modifications yields an image representation, and asset-level, body-level, and joint-level categorical data, that are human interpretable. Since the generation process is sequential, these quantities can be adjusted by the user before generating proceeding with the part geometry generation. Crucially, this interpretable structure enables the use of a multi-modal 3D generation model as the shape generator. The shape generator, here represented by SDFusion [8]. is queried multiple times to produce high-quality geometries for each part of the object (see Figure 1). To enhance consistency, we propose a constrained generation approach that enforces the part geometry to fit a given bounding box, here provided by the structure generator.
We demonstrate MIDGArD's capabilities by showcasing improved structural quality, consistency, and part-level control compared to NAP. Furthermore, we show qualitative results of MIDGArD generating diverse articulated objects dependent on the partial graph input and shape conditioning. To further improve the consistency and quality of the generated shapes, we propose create oriented bounding boxes that have a tighter fit to the ground truth while being properly aligned. The reduces the overlapping volume by 17% on average. Our contributions can be summarized as:
1. MIDGArD: A novel diffusion based framework for generating high high-quality, articulated, and simulatable 3D assets with improved consistency between structure and shape.
2. Enhanced Interpretability, and Controllability: Providing -uman-interpretable interme- diate outputs and allowing user adjustments at multiple levels of the generation process.
3. Enhanced Quality and Scalability: Relying on state-of-the-ard 3D foundation models for shape generation.
4. Improved part aligument: Introducing a bounding-box constrained shape generation approach that improves alignment of kinematic links by 17% on average.
5. Simulatable Assets:
Offering a pipeline to create fully simulatable assets compatible with physics engines such as MuJoCo.
Code and models are available at https: http:::///quuuttnn -ebouute-eebbut....mm www.eet.ggttuu.... wwt.gg.i..ttttiteeoooo 1o/MIDGArD.
2

---

2 Related Works
Structural Characterization and State Estimation of Articulated Objects Designing articulated objects involves understanding both their geometry and kinematic properties. Traditional methods often require extensive manual labor or rely on limited datasets, hindering scalability and applica- bility [45]. Early work morphable templates. such as SMPL for human bodies [55] and SMAL for quadruped animals [1 [114], demonstrated the potential of matching existing articulation structures (i.e., skeletons) to wide variety of inputs such static meshes monocular videos, thereby substantially simplifying the rigging process. However, while effective within specific categories, these templates lack the flexibility generalize arbitrary articulated mechanisms. To address these limitations, recent approaches have enhanced templates by incorporating implicit representations to model geometric deformations and appearances [40]. Robust neural rigging methods [61, 70] leverage template priors while remaining flexible enough to handle mesh elements that deviate from the template, such as clothing or hair. On the other hand, template-free methods [100, 101] rely solely on input mesh geometry but are still primarily limited to humanoid characters due to the scarcity of comprehensive datasets. Emerging approaches [27] aim to rig arbitrary static assets, potentially paving the way for unified, template-free rigging pipeline. Our research aligns with this direction by learning articulation priors for arbitrary objects without depending on predefined templates.
Significant strides has also been made in estimating articulation joint states and parameters from sensory observations. Early contributions such as [83] demonstrated that it possible to learn compact kinematic models of entire articulated objects solely based on pose observations. Probabilistic models have enabled robots learn their own kinematic structures through self-observation and motor babbling [10, 81. 82]. Gaussian processes have been employed to model part connectivity and link articulation in objects [87]. Interactive perception techniques further enhance this learning process by allowing robots to interact with objects to obtain their kinematic model [69, 33, 64, 18, 66, 3, 22, 44, 25]. Deep learning techniques were also used to train dedicated architectures to directly predict articulation models from sensory inputs [23, 104, 1, 41, 49, 94, 106, 34, 29, 28, 51, 102, 13, 99, 105, 43, 20, 109]. Interactive systems [67, 13] learn to predict potential motions of articulated objects, aiding downstream motion planning and interaction, while reinforcement learning based approaches [96] train policies to explore diverse interaction trajectories, contributing to actionable visual representations. Our method diverges from traditional approaches by not relying on explicit estimation pipelines; instead, we learn articulation priors implicitly within our model.
Articulated Object Generation Generating articulated objects involves synthesizing models that accurately represent both geometric details and kinematic behaviors. Methods such as Self- Supervised Category-Level Articulated Object Pose Estimation [51] and PD (Pose-aware Part Decom- position) [34] address the generation of joint parameters and part poses, respectively, contributing significantly to the field. Few generative frameworks address both aspects concurrently. Notable sashes--uiing
among them is NAP [37], which generates complete articulated assets inceuuinn graph attention networks. CAGE [46], enhances controllability and interpretability, albeit with limitations in mesh generation. More recently, [62] proposed a framework capable of reconstructing articulated assets from image inputs by using a 3D shape completion model to generate the different parts and a large language model (LLM) to predict the joint parameters. Progress was also made in physics-aware shape generation; for instance, [58] introduces an improved loss mechanism to account for internal collisions within generated articulated assets. In this work, we build up NAP and CAGE and design an approach to enhance the controllability of the generative process, and leverage maulti-oddl 3D generative model for improving the quality of the shape geometry.
3D Shape Generation and Completion Recent progress in diffusion models and 2D-guided generation techniques have revolutionized 3DD shape generation and completion. Two main types of generative pipelines are currently being explored. The first one leverages 2D diffusion models to optimize NeRFs [73, 42, 57]. enabling high-quality 3D generation without the need for direct 3D supervision. The second employs Latent Difffusion Models (LDMs) for different 3D representations, including point clouds [59, 113, 112, 60, 88), voxels [111]. SDFs|8, 76, 111, 48], 2hape2 108, 1 110], and triplanes [16]. It should be emphasized that despite the predominant role of diffusion models, other methodologies such as autoregressive models [78], have demonstrated potential in producing high-fidelity 3D shapes from minimal inputs. Conditional diffusion models enable multi- modal inputs. SDFusion [8], for instance, can generate 3D objects from text descriptions, images, or partial 3D shapes. Building upon these capabilities, we have adapted SDFusion as the shape generator within our framework. However, due to MIDGArD's modular design, the shape generator can be updated to incorporate the latest advancements in static 3D generation.
3

---

3 Synthesis of 3D Articulated Mechanisms
3.1 Method Overview
Both components of MIDGArD, namely the structure and the shape generator, leverage a denoising diffusion model [79, 80, 21J. Diffusion models operate by progressively degrading training data through the systematic application of Gaussian noise, a method known as the forward process. Subsequently, these models are trained to restore the original data by methodically removing this noise in what is known as the reverse process. Please refer to Appendix A and [21], [79] for details. As shown in Figure 1, the full MIDGArD pipeline can be separated into the Structure Generator and the Shape Generator. The Structure Generator (subsection 3.2) takes graph-based representation of an articulated asset which may be incomplete or affected by noise and resolves this representation through diffusion. The denoised graph features, decoded into suitable human interpretable - text prompts, images and bounding box information are then fed into the Shape Generator (subsection 3.3). another catuusion-bsscd model yielding the final articulated 3D object.
3.2 Structure Generator
Articulated Asset Representation and Parametrization Building upon [37], we encode the structural and kinematic attributes of every articulated asset into the node and edge features of a complete graph referred to as Gv. Our structure generation module leverages Graph ATtention (GAT) [90, 4, 32] denoising network to generate coherent and interpretable articulation features from noisy incomplete inputs (sce Figure 1). To improve the quality, controllability, and interpretability compared to [37], we modify the asset parametrization. Drawing on insights from [46], we incor- porate set of categorical variables the nodes and edges features to enhance asset consistency, interpretability and provide a more intuitive control interface over the generation process. Each asset is represented as a complete graph GN {ær, e)yy y that embodies the object's links and joints in its N nodes, and in its ((N - 1)/2 edges respectively. Rather than directly denoising and then decoding a low-dimensional shape latent f. E Dr for each link into 3D mesh [37, 36], we propose a two-step approach that operates on more manageable image latent 9, E IR De of the link. This image latent is derived by training on various views of every rigid body in the dataset. In our method, the structure generator denoises a latent representation of an image for each individual component of the articulated asset. This denoised image, along with the categorical information extracted from the graph, facilitates the creation of human-interpretablee priors for each link, namely a text description and a front view image. These prompts can then be utilized to condition 3D shape generation model yielding consistent meshes. Additionally, the graph itself encodes highly relevant structural features that act as supplementary conditioning signals, further guiding the shape generation model.
Quatt--Quat Variational Auto Encoder (VQVAE) [89]
Each node æ; represents link of the articulated asset and Vi, [0, - - the edge eij establishes a connection between node æ;; and node x. The node feature vector æ; is comprised of multiple components, specifically, 2; = jo,, ai, b;, d, r;, ri, 9, E RRD,, and the edge feature vector esy is formulated as c.j- = [e. P.j. Lzj] E RE, where Dv and D refer to the node and edge feature dimensions. To ensure consistent graph size across various objects. N is set to the maximum count of parts anticipated, using o, E [0, 1] as a binary indicator to denote part existence. Similarly, the edges denote symbolic existence and chirality through the indicator variable Cij E [0, 1]s. Joint types represented through j. E [0, 1) one-hot encoding that in our design has dimensionality of three to signify the types- - prismatic, revolute, or screw. Similarly, a; E [0, 1] and b, E [0, 1) denote the one-hot encoding vectors of asset (resp. body) categories. Similar to [37], we use Plücker coordinates as joint parametrization as this provides a compact representation for joint types, such as revolute, prismatic, and screw joints. The Plücker joint parametrization, articulated as P:y E R provides a harmonious portrayal of the different joint types under consideration, and joint limits are entailed by Lij E Rr,,. Ultimately, the vectors d, E gº, T; E R and t, E E depict the bounding box dimensions, orientation and position of each part. Note that r; and t; are defined for every part relative to its parent reference frame.
Unlike previous approaches [37, 46), we do not assume that the parts of the dataset are provided in a canonical orientation. Assuming that objects are in canonical orientation and not explicitly considering the orientation leads to degrading performance due to misaligned orientations of the datasct (see Figure 5, as well as Appendix B-Figure 10 and Appendix B B-Figure 11). Instead, we

---

estimate this orientation for each body in the dataset by computing the corresponding Oriented Bounding Box (OBB), which then used position the body in a canonical pose. The process (see Appendix B) begins by generating convex hull from the mesh. This step simplifies the complex geometry into a more manageable form, reducing dependency on the internal structure of the object. Next, we voxelize the convex hull to collect volumetric samples. We then execute Principal Component Analysis (PCA) to determine the primary axes of the mesh providing an initial approximation of the object's orientation. Following the PCA, we perform gradient descent on the attitude quaternion of the OBB, using the PCA-based initial estimate as starting point. The objective of this optimization is to minimize the volume of the OBB. This approach allows for a more flexible and accurate alignment of dataset bodies into canonical poses, thereby enhancing the overall quality and consistency of the articulated objects generated by our framework (see Figure 5, Table 3).
Denoising Diffusion on the Plücker Manifold To maintain the interpretability of the denoised quantity as a Plücker vector Pij [uij Piy], our approach introduces a novel parametrization allowing the diffusion process to be directly executed on the Plücker manifold P.. In contrast, the approach proposed in [37], denoises within R. thereby requiring a projection operation R + P at every denoising step. We propose an alternative formulation ki = [mij, nayy E of the joint parameters where mij = [B.j. Payl IR² represent the spherical parametrization of the Plücker axis Wij. This formulation yields uaz = [sin(0;)) cos(yes) ), sin sin(O,)) sin(())), cos (0;))] E R with ||, = 1. The vector ni E IR is then defined such that Vij = mij X Uij. This alternative parametrization guarantees the normalization of u:y and enforces by construction the orthogonality between Uij and Vij. which are key features of the Plücker manifold [30]. Consequently, our approach inherently satisfies the manifold's constraints, thereby streamlining the computational workflow by eliminating the necessity for iterative projections.
3.3
Shape Generator
The shape generator of MIDGArD aims to create the mesh for each component of an articulated object using the denoised object graph, which contains both semantic (object category) and visual (image latents) data. We leverage multi-moda 3D generative model that is trained independently from the structure generator. Our modular setup is designed accommodate any generative model, allowing for seamless integration of the latest advances in 3D generation. In this work, the SDFusion model [8] is used, conditioned on image, text as well as graph features input, and enhanced with a novel approach for bounding-boo worttg--boxcoooeeoe nox-constrained generation. The training pipeline is shown in Figure 2.
Multimodal Guidance for Consistent Shape Generation The core of SDFusion is a latent diffusion model, see [8] for details. Object meshes are transformed into truncated signed distance functions (TSDFs) and encoded using a pretrained and frozen 3D VQ-VAE [89] model. The diffused latent representations, along with image and text condition signals, are denoised through a 3D U-Net model. The resulting output latent is then decoded into a TSDF and converted back into mesh using marching cube algorithm [56]. It is worth emphasizing that we train and apply the diffusion model on the individual parts of each objects. querying the model multiple times at inference time generate a full articulated object (see Figure 1). The model is trained on the PartNet Mobility dataset [97] with the same train-test split as the structure generator. We modified SDFusion's 's conditioning [8] to align it with the output of the structure generator. The model is conditioned through cross-attention on the following modalities, either independently or in combination.
Single view image: A pretrained ResNet-18 18 model [19] encodes images. During training, the model uses renderings of the part mesh from a frontal view. At inference, the model decodes the image from the node features of the articulation graph.
Text: We employ BERT [11] for text encoding (similar to [84, 81). The text is constructed using the object category and asset type from the PartNet dataset in the format "A <asset type> part of a <object category>", for instance, "A lid part of a trash can".
Graph: The output from the structure generator is encoded and concatenated with image- and text embeddings to provide information about the part's role within the object, such as its expected size and its relation to other parts, i.c. joint types.
This multi-modal conditioning approach ensures the generation of consistent, high-quality parts that can be seamlessly integrated to form complete articulated objects, enhancing the overall performance and applicability of our framework.
5

---

Original SDFusion pipeline
BERT
New Components
Trained
Latent
Pretrained
Image
TSDF
Res Net
Denoising
Object TSDF
U-Net
Rg
Noisy
Diffuse
TSDE
Concat
Dec
Object TSDF
GroOs. attention
Denoised
Enc
TSDF
Figure 2: Architecture of the Shape Generator.
Enforcing Bounding Constraints Generating object parts independently offers increased flexibility and quality but introduces challenges related to the size and orientation of the generated parts. Intuitively, the shape generator must ensure that parts conform r; and d; in the articulation graph. To achieve the correct orientation, we generate links in a canonical pose and subsequently apply post-processing rotation based on r., the 3D orientation vector derived during structure generation. To train the model accordingly, by estimating and adjusting the pose of parts to a canonical orientation, detailed in Appendix B.
To ensure that generated parts match the sizes specified within the graph node features, we introduce bounding-bon constraint to the generative model (see Figure 2). Instead of generating the object directly, the diffusion model trained on the difference relative to a predefined bounding box. To enable this schema for latent diffusion,
calculate the difference ΣA 2, 2b. where 2, the TSDF VQ- (Q-VAF-encoded latent of the object, and 2; is the encoding of a simplified bounding- (values inside the box set to -0.01, and values outside denoiser with the bounding box information by concatenating 2b to the noisy version of Za. Other conditioning inputs, such images, text, and graph features encodings, are incorporated through cross-attention mechanisms. The object's s TSDF is then computed by decoding 2a + =6 using the 3D VQ- VAE. This method effectively guides the model produce objects that are closely matching the target bounding boxes even after a few training steps.
0.01). During inference, we provide the
Experiments, Results and Discussion
4.1 Experiment Setup
Dataset All experiments were conducted using the PartNet Mobility dataset [97], which contains diverse set of articulated 3D objects with detailed geometric and kinematic annotations. Each mesh in the dataset underwent two-step preprocessing routine. First, we enforced the manifold property as described in [26]. Second, we performed orientation estimation to establish a canonical pose for each object and accurately compute the corresponding bounding boxes (see Appendix B for details).
Evaluation Metrics We adopt the evaluation framework from NAP [37], which introduces the Instantiation Distance (II) metric for comparing articulated objects. ID measures the average Chamfer distance over sampled joint states, accounting for both geometric and kinematic differences. is defined as:
1001,,0)
[[min min (d.0..0.02..))) 91 L2EEq
Σ
O2. m))
Σ ogE
E min (á(O1, q. (2, (2. (1) 9iER1
M
M
where Q is a set of uniformly sampled joint poses and d is the minimum Chamfer distance over all possible canonicalizations of the mesh. We ID in conjunction with standard metrics for unconditional generation, namely minimum matching distance (MMD), coverage (COV) and I l-nearest-neighbor accuracy (I-NNA).

---

Implementation Details We trained the structure generator and the image VQ-VAE IQ-VAE on an NVIDIA RTX 3090 GPU, while the shape generator was trained on an NVIDIA RTX 6000 GPU. Evalua- tion took place on a single NVIDIA RTX 3090 GPU. The image VQ-VA VAE, which encodes latent representations of objects in the shape generator, was trained on 256 x 256-pixel front renderings of the the PartNet Mobility object meshes. The denoising model used in the structure generator contains six graph attention blocks, with a latent embedding size of 512 and 32 attention heads. We set the maximum number of nodes in the graph to N = 8. Our training parameters closely follow those in NAP, with the key difference being the use of an implicit denoising diffusion pipeline [80] over 100 time steps, as opposed to DDPM with 1,000 time steps. This modification significantly accelerates inference speed. Our shape generator is adapted from SDFusion [8] and trained on the PartNet Mobility dataset. We used the same hyperparameters as the multimodal model in SDFusion and utilized their pre-trained VQ-VAE checkpoint. We excluded 10 categories from training due to their objects containing numerous equally-shaped parts (e.g., keyboards with over 30 keys).
4.2 Results and Discussion
Asset Generation To the best of our knowledge, NAP is the only ap- proach for generating articulated 3D objects without prior knowledge about their geometric structure. We first evaluate MIDGArD in an unconditional generation setting, comparing its performance to NAP. For a fair comparison, retrained NAP on our preprocessed version of the PartNet Mobility dataset. The results are presented in Table 1. MIDGArD outperforms NAP in terms of MMD and COV metries, indicating better diversity and coverage of the generated samples. Specifically, observe a 6.4% improvement in MMD and a 3.7% improvement in COV. The 1-NNA metric is comparable between both methods. We also perform ablation study on the effect of the Plücker manifold parameterization. As shown in Table I using the Plücker manifold improves the MMD and COV metrics, suggesting enhanced consistency and diversity in the generated assets.
Unconditional Articulated
Table I: Comparison to NAP in unconditional generation setting.
Unconditional ID
Generative Paradign/Method
MMD 4
COV t I-NNA 4
NAP
0.0282
0.4675
0.5831
Ours (Plücker manifold)
0.0264
0.4857
0,5831
Ours (No Plücker manifold) 0.0270
0.4779
0.6221
To further evaluate the physical plausibility of the generated assets, we analyze the distribution of joint types in the training data and compare it with those in samples generated by NAP and MIDGArD. We sampled 400 objects generated by each method and counted each joint type only once per object to minimize the impact of objects with multiple joints of the same type.
Table 2: Distribution of joint categories in real data vs generated data from NAP and approach.
Screw
Revolute
Prismatic
Training data
6%
62%
32%
NAP-generate MIDGArDD-eneratce 2%
95%
1%
4%
62 %
36 %
The results in Table 2 show that NAP predominantly produces screw joints, which are rare in the training data. In contrast, MIDGArD generates objects with a joint type distribution closely matching that of the training data, thereby enhancing the plausibility of the generated assets. We computed the Chi-Square statistic to quantif'y the deviation from the expected joint type distribution. NAP's generated data yielded a high x² value of 5618, indicating significant difference from the training data distribution. In contrast, MIDGArD's generated data resulted in a low x² value of 12.7, demonstrating close aligmment with the expected distribution.
Figure 3.B provides a side-by-side qualitative comparison between MIDGArD and NAP. MIDGArD generates objects with higher geometric quality and more realistic motion compared to NAP. For instance, the fan generated by MIDGArD exhibits detailed geometry, and the laptop displays realistic opening motion, whereas NAP's outputs often show unnatural joint motions and inconsistent shapes.
7

---

Conditional
Side by-------kkkeee comparison
Lipho
Gode
Derver
Artollent
"sastors"
Partriavel image conditioring
partinttes
(6) Typical balture caNAs
MA
Imape lmnge Imaee ebtect
Spuewny
Doves
"Cheee""
'strag Aumiure
Bounding
Digenser
trishan
Figure 3: A) PariToMotion constrained structural generation. B) Side-by-side comparison between our approach and NAP. C) Typical failure cases. D) Part-level image conditioning.
Additionally, as shown in Figure 3.C, the absence categorical information in NAP leads to the generation of objects composed of mismatched parts that do not cohesively fit together. In approach, while there may be occasional issues with geometry or kinematics, the object parts remain semantically consistent, preserving the integrity of the overall structure. Finally, illustrated in Figure 3.D, our conditional part-generation proves to be robust to images from different viewpoints, yielding enhanced flexibility and generalization capability of our method.
Conditional Generation and Controllability One of the key advantages of MIDGArD over existing approaches is its enhanced controllability, enabling users to guide the generation process using graph attributes such as articulated asset types and body types. We showcase this capability in (Image+Text) To-Object" setup (see Figure 3.A1), and a PPar-TToMotion setup (see Figure 3.A2), where the model is provided with part features only (i.e., joint data) and outputs consistent articulated assets. Notably, MIDGArD can also be controlled using only asset- level data; for instance, "fan" without needing to provide detailed information for each node (see Appendix C for additional qualitative results). Furthermore, the modular structure of our generation pipeline enables fine- grained, part-level control. Users can specify the desired appearance or attributes of individual parts by adjusting the human-interpretable articulation solutions synthesized by the structure generator accordingly. The resulting images and text description will then be used as conditioning signals for the shape generation module. As illustrated in Figure varying the image inputs while keeping other attributes fixed results in generated parts that adapt accordingly. demonstrating the flexibility and responsiveness of our approach. This level of control allows for precise customization of generated assets, facilitating applications that require specific design features or aesthetic qualities (see Appendix D for additional qualitative results).
query for specific categories such "storage furniture'
decoded Imagea
sampled images (1)
sampled Imagee
Figure 4: Image guidance of the shape generation process

---

bounding box tormat
Trained with axis-aligned bounding boxes
Trained with oriented bounding boxes
B
cases missing joint categonical data
Generated graph
mosh format
Figure Failure cases due to due to axis-aligned bounding boxes and parameter discretization. A) The use of Axis Alligned Bounding Boxes tends to confuse the 3D generation model. B) The use of categorical joint variable MIDGArD allows better joint motion resolution and helps filtering unrealistic solutions obtained with former pipelines.
Constrained Shape Generation We conducted ablation study to evaluate the effectiveness of our bounding-box constrained shape generation approach introduced in section 3.3. The experiments performed on part-samples from PartNet Mobility. We randomly sampled ten objects per category from the test data, including all their parts, resulting in 1044 samples across 32 categories. First, we measure whether the generated object fits within the bounding box. The model output y is a sampled TSDF of resolution 64, Y gen E R 666x644 where negative values indicate the object's interior. We compare YoeR with the TSDF of the bounding box yoh and compute the count of all cells outside of the bounding box (OBBeouns) and their sum of SDF values (OBB
BB (Ois, Ogen)=
Σ
0 YgEn i5k
Ygen <0],
(2)
8jk
OBB sum (Obs, Ogen)
Σ zjk
ygen 19k
1 igh
O
Vgen
0).
fjk
Secondly, we evaluate the generation quality after transforming the TSDF into mesh, computing the Chamfer distance (CD) between generated and real objects based on 5000 sampled points.
Table 3 compares our bb prior approach the original SDFusion model trained on PartNet Mobility. For encoding the part dimensions, we use an MLP with three hidden layers (size 16, 64 and 256 a GAT-encoding, see Appendix
respectively). For a comparison preprocessing the dataset to rectify objects into canonical pose, dubbed "original + bb MLP + oriented dataset", already improves the generation results significantly, leading to lower OBB O.Bont OB Baum and CD. The bb prior method, however, consistently ensures accurate proportions and sizes for the generated links, diminishing erroneous volume. Additionally, bb prior bb ML.P method secures the best Chamfer Distance 0.072. An alternative technique involves resizing generated objects to suit the bounding box dimensions. Such postprocessing of the model outputs reduces the average CD to 0,0061, close matching that of the bb prior method. Nonetheless, such resizing may lead to significant distortions in object proportions, thereby underscoring the superiority of the bb prior method.
Our first modification of
Analysis of Oriented Bounding Boxes As Table 3 shows, using oriented bounding boxes drastically improves the reconstruction performance. To quantify the change the dataset, we measured the volume reduction of the bounding boxes after applying our PCA-based rectification approach. On average, the bounding box volume decreased by 17.4%, with one-third of the samples experiencing
Table 3: Ablation of bounding-box constrained shape generation. The original SDFusion pipeline, only enhanced with bounding-box conditioning, is compared to our approach. Training on prepro- cessed dataset and post- post-processing the generated shapes afterwards already improves the performance significantly. The best results are achieved using our bounding box SDF prior with post-processing
OBB
OBB
CD (generated)
CD (scaled)
bb prior bb MLP (Ours)
1146
10.89
0.0072
0.0043
original MLP + oriented dataset
1490
37.15
0.0152
0.0061
original (SDFusion) bb MLP
3795
183.5
0.0307
0.008

---

volume decrease of more than 10% and 20% of the samples experiencing a volume decrease of more than 30%. Figure 5 illustrates failure cases where using axis-aligned bounding boxes leads to unrealistic part shapes, such as thick cabinet door.
5 Conclusions
In this work, we introduced MIDGArD, a novel framework for generating 3D articulated objects. MIDGArD employs modular approach, combining interpretable articulation graph generation with high-quality shape generation. By leveraging categorical parametrization, MIDGArD enhances the consistency and plausibility of articulated objects, producing high-quality meshes with accurate dimensions through a constrained shape generation mechanism. The human- interrrattee representa- tion of images and text within the articulation graph allows for part-level and multi-moda control over the generation process. The models produced by MIDGArD are fully simulatable, paving the way for applications in text-guided image-guided content creation.
MIDGArD addresses several issues identified Neural 3D Articulation Prior (NAP) by improving joint conditioning with categorical embeddings, enforcing physical constraints via bounding- box constrained generation, and providing part-level control using latent image codes. Our results demonstrate MIDGArD's superior performance in terms of structural quality, consistency, and interpretability compared to existing methods. The framework's ' ability to generate diverse articulated objects based on partial graph inputs and shape conditioning underscores its potential for broad applications in digital content creation and robotics.
Despite the advancements introduced by MIDGArD, there are several limitations and areas for future improvement: 1) We observe that the current metrics for articulated generation are still very limited and benchmarks as well evaluation frameworks must be developed for this field. For instance, Instantiation Distance still based on point clouds, which ignores cases of small parts of the object with unrealstic motion, such as cart wheel moving up to meter away from the cart. 2) Overcoming limitations of graph scalability posed by node number constraints remains an important challenge. 3) Furthermore, it would be interesting investigate the of mixed integer noise methods such as MiDi [91] in the articulation graph generation pipeline to process the continuous and discrete variables. 4) Enhanced conditioning of the structure generation process using natural language image data represents a highly promising avenue for future research. Foundational efforts in this area, such those by Cai al. [5], have laid the groundwork for further exploration. 5) Integrating specialized templates, such as human body poses, could significantly enhance MIDGArD's modelling versatility. Lastly, 6) testing alternative 3D generation models and adapting MIDGArD handle different geometric primitives could broaden its generalization capabilities.
10

---

References
[1] Ben Abbatematteo, Stefanie Tellex, and George Konidaris. Learning to generalize kinematic models novel objects. In Proceedings of the 3rd Conference on Robot Learning, 2019.
[2] Mohammad Samiul Arshad and William J Beksi. List: Learning implicitly from spatial transformers for single-view 3d reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9321-9330, 2023.
[3] Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav S Sukhatme. Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6):1273-122) 1291, 2017.
[4] Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105. 14491, 2021.
[5] Zhou Cai, Xiyuan Wang, and Muhan Zhang. Latent graph diffuusio: A unified framework for generation and prediction on graphs. arXiv preprint WxXiv:2402.025/6 2024.
[6] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.1387, 2023.
[7] Yamei Chen, Yan Di, Guangyao Zhai, Fabian Manhardt, Chenyangguang Zhang, Ruida Zhang, Federico Tombari, Nassir Navab, and Benjamin Busam. Secondpose: Se (3)-consistent dual- stream feature fusion for category-level pose estimation. arXiv preprint arXiv:231 1. 2023.
[8] Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465, 2023.
[9] Ruihang Chu, Enze Xie, Shentong Mo, Zhenguo Li, Matthias Nießner, Chi-Wing Fu, and Jiaya Jia. Diffcomplete: cerfusion-based generative 3d shape completion. Advances Neural Information Processing Systems, 36, 2024.
[10] Anthony Dearden and Yiannis Demiris. Learning forward models for robots. IJCAI, volume 5, page 1440, 2005.
[1I] Jacob Devlin, Ming-Wer Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
[12] Yan Di, Chenyangguang Zhang, Chaowei Wang, Ruida Zhang, Guangyao Zhai, Yanyan Li, Bowen Fu, Xiangyang Ji, and Shan Gao. Shapemaker: Self-supervised joint shape canonicalization, segmentation, retrieval and deformation. arXiv preprint urXiv:2311.11106, 2023.
[13] Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint urXiv:2205.0433.. 2022.
[14] Chuan Fang, Xiaotao Hu, Kunming Luo, and Ping Tan. Ctrl-room: Controllable text-to-3d room meshes generation with layout constraints. arXiv preprint arXiv:2310.03602, 2023.
[15] Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross- Cross-category domain-generalle generalizable object perception and manipulation via generalizable and actionable parts. Proceedings of the IEFE/CV Conference on Computer Vision and Pattern Recognition, pages 7081-7091, 2023.
[16] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint
2023.
11

---

[17] Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layoutransformer. Layout generation and completion with self-attention. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 1004- 1014, 2021.
[18] Karol Hausman, Scott Niekum, Sarah Osentoski, and Gaurav Sukhatme. Active articulation model estimation through interactive perception. In 2015 IEEE International Conference on Robotics and Aulomation (ICRA), pages 3305-3312. IEEE, 2015.
[19] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Nick Heppert, Muhammad Zubair Irshad. Sergey Zakharov, Katherine Liu, Rares Andrei Ambrus, Jeannette Bohg, Abhinav Valada, and Thomas Kollar. Carto: Category and joint agnostic reconstruction of articulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 166640-6651,,, 2020.
[22] Cheng-Chup Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. arXiv preprint arXiv:2302.0/295, 2023.
[23] Ruizhen Hu, Wenchao Li, Oliver Van Kaick, Ariel Shamir, Hao Zhang, and Hui Huang. Learning to predict part mobility from a single static snapshot. ACM Transactions on Graphics (TOG), 36(6):1-13, 2017.
[24] Yubin Hu, Sheng Ye, Wang Zhao, Matthieu Lin, Yuze He, Yu-Hui Wen, Ying He, and Yong- Jin Liu, O' 2-recon: Completing 3d reconstruction of occluded objects in the scene with a pre-trained 2d diffusion model. arXiv preprint arXiv:2308.0055)) 2023.
[25] Jiahui Huang, He Wang. Tolga Birdal, Minhyuk Sung, Federica Arrigoni, Shi-Min Hu, and Leonidas Guibas. Multibodysyne: Multi-body segmentation and motion estimation via 3d scan synchronization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7108 1188-7118, 118, 2021.
[26] Jingwei Huang, Hao Su, and Leonidas Guibas. Robust watertight manifold surface generation method for shapenet models. arXiv preprint arXiv: 8802006698 2018.
[27] Denys Iliash, Hanxiao Jiang, Yiming Zhang, Manolis Savva, and Angel X Chang. S2o: Static to openable enhancement for articulated 3d objects. arXiv preprint arXiv:2409. 2at.22499.8896 18896, 2024.
[28] Ajinkya Jain, Stephen Giguere, Rudolf Lioutikov, and Scott Niekum. Distributional depth- based estimation of object articulation models. In Conference on Robot Learning, pages 1611-1621. PMLR, 2022.
[29] Ajinkya Jain, Rudolf Lioutikov, Caleb Chuck, and Scott Niekum. Screwnet: Category- independent articulation model estimation from depth images using screw theory. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13670-13677. IEEE, 2021.
[30] Michael Joswig, Thorsten Theobald, Michael Joswig, and Thorsten Theobald. Plücker coor- dinates and lines in space. Polyhedral and Algebraic Methods Computational Geometry, pages 193 003-207,200 2013.
[31] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layout- vae: Stochastic scene layout generation from a label set. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9895-9904, 2019.
[32] Kamil Kamiński, Jan Ludwiczak, Maciej Jasiáski, Adriana Bukala, Rafal Madaj, Krzysztof S7c7epaniak, and Stanislaw Dunin-Horkawier Rossmann-toolbox: a deep learning-based pro- tocol for the prediction and design of cofactor specificity rossmann fold proteins. Briefings in bioinformatics, 23(1) 23())bbba3371, 2022.
12

---

[33] Dov Katz and Oliver Brock. Manipulating articulated objects with interactive perception. In 2008 IEEE International Conference on Roboties and Automation, pages 272-277. IEEE, 2008.
[34] Yuki Kawana, Yusuke Mukuta, and Tatsuya Harada. Unsupervised pose-aware part decompo- sition for man-made articulated objects. In European Conference on Computer Vision, pages 558-575. Springer, 2022.
[35] Juil Koo, Seungwoo Yoo, Minh Hieu Nguyen, and Minhyuk Sung. Salad: Part-level la- tent diffusion for 3d shape generation and manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14441-14451, 2023.
[36] Jiahui Lei, Congyue Deng, Karl Schmeckpeper, Leonidas Guibas, and Kostas Daniilidis. Efem: Equivariant neural field expectation maximization for 3d object segmentation without scene supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[37] Jiahui Lei, Congyue Deng, Bokui Shen, Leonidas Guibas, and Kostas Danilidiss Nap: Neural 3d articulated object prior. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[38] Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. arXiv preprint 1oXiv:9901.06700 2019.
[39] Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12642-12651, 2023.
[40] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4d scans. ACM Trans. Graph., 194-1, 2017.
[41] Xiaolong Li, He Wang. Li Yi, Leonidas Guibas, Lynn Abbott, and Shuran Song. Category- level articulated object pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3706-3715, 2020.
[42] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 300-309, 2023.
[43] Gengxin Liu, Qian Sun, Haibin Huang, Chongyang Ma, Yulan Guo, Li Yi, Hui Huang, and Ruizhen Hu. Semi-weakly supervised object kinematic motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21726-21735, 2023.
[44] Jiayi Liu, Ali Mahdavi-Amiri, and Manolis Savva. Paris: Part-level reconstruction and motion analysis for articulated objects. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 352-363, 2023.
[45] Jiayi Liu, Manolis Savva, and Ali Mahdavi-Amins Survey on modeling of articulated objects. arXiv preprint
2024.
[46] Jiayi Liu, Hou In Ivan Tam, Ali Mahdavi-Amiri, and Manolis Savva. Cage: Controllable articulation generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17880-17889) 2024.
[47] Liu Liu, Wenqiang Xu, Haoyuan Fu, Sucheng Qian, Qiaojun Yu, Yang Han, and Cewu Lu. Akb-48: A real-world articulated object knowledge base. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14809-14818, 2022.
[48] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng. Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent milti-view generation and 3d diffusion. arXiv preprint arXiv:231 231 1. 1.07885, 2023.
13

---

[49] Qihao Liu, Weichao Qiu, Weiyao Wang, Gregory D Hager, and Alan L Yuille. Nothing but geometric constraints: A model-free method for articulated object pose estimation. arXiv preprint arXiv:20/2.0008S 2020.
[50] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298-9999999 9309, 2023.
[51] Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Li Yi. Self-supervised category-level articulated object pose estimation with part-level se (3) equivariance. arXiv preprint arXiv:2302.. 14268, 2023.
[52] Zhen Liu, Yao Feng, Michael Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. arXiv preprint arXiv:2303.08113, 2023.
[53] Zhengzhe Liu, Jingyu Hu, Ka-Hei Hui, Xiaojuan Qi, Daniel Cohen- -r, and Chi-Wing Fu. Exim: A hybrid explicit-implicit representation for text-guided 3d shape generation. ACM Transactions on Graphics (TOG), 42(6): 2023.
[54] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma. Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross domsin diffusion. arXiv preprint arXiv:2310.15000 2023.
[55] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851-866, 2023.
[56] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. In SIGGRAPH, pages 163-169. ACM, 1987.
[57] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp. Tsung- Yi Lin, Ming- Yu Liu, Sanja Fidler, and James Lucas. Att3d: Amortized text-to- 3d object synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17946-17956. 2023.
[58] Rundong Luo, Haoran Geng, Congyue Deng. Puhao Li, Zan Wang, Baoxiong Jia, Leonidas Guíbas, and Siyuang Huang. Physpart: Physically plausible part completion for interactable objects. arXiv preprint arXiv:2408. 13724, 2024.
[59] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[60] Zhaoyang Lyu, Zhifeng Kong, Xudong Xu, Liang Pan, and Dahua Lin. A condi- tional point seffusiion-efinemeee paradigm for 3d point cloud completion. arXiv preprint arXiv:2112.03530, 2021.
[61] Jing Ma and Dongliang Zhang. Tarig: Adaptive template- aware neural rigging for humanoid characters. Computers & Graphics, 114:158-167, 2023.
[62] Zhao Mandi, Yijia Weng, Dominik Bauer, and Shuran Song. Real2code: Reconstruct articu- lated objects via code generation. arXiv preprint wwXiv:2406.08474 2024.
[63] Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel Chang, and Manolis Savva. Multiscan: Scalable rgbd scanning for 3d environments with articulated objects. Advances in Neural Information Processing Systems, 35:9058-9077 2022.
[64] Roberto Martin Martin and Oliver Brock. Online interactive perception of articulated ob- jects with multi-level recursive estimation based on task-specific priors. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2494 2501. IEEE, 2014.
[65] Roberto Martin-Martin, Clemens Eppner, and Oliver Brock. The rbo dataset of articulated objects and interactions. The International Journal of Robotics Research, 1013-19919 2019.
14

---

[66] Roberto Martin-Martin. Sebastian Höfer, and Oliver Brock. An integrated approach to visual perception of articulated objects. In 2016 IEEE international conference on roboties and automation (ICRA), pages 5091-5097. IEEE, 2016.
[67] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. WhereZact: From pixels to actions for articulated 3d objects. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 0823-6823, 2021.
[68] George Kiyohiro Nakayama, Mikaela Angelina Uy Jiahui Huang, Shi-Min Hu, Ke Li, and Leonidas Guibas. Difffacto: Controllable part-based 3d point cloud generation with cross diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14257-14267, 2023.
[69] Neil Nie, Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Structure from action: Learn- ing interactions for articulated object 3d structure discovery. arXiv preprint arXiv:2207.08997, 2022.
[70] Stefan Novaković and Vladimir Risojević. Learning localization of body and finger animation skeleton joints on three-dimensional models of human bodies. In 2024 Zooming Innovation in Consumer Technologies Conference (ZINC), pages 19-24. IEEE, 2024.
[71] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems, 14:12013-12026 2021.
[72] Akshay Gadi Patil, Supriya Gadi Patil, Manyi Li, Matthew Fisher, Manolis Savva, and Hao Zhang. Advances in data-driven analysis and synthesis of 3d indoor scenes. arXiv preprint arXiv:2304.03188 2023.
[73] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209, 14988, 2022.
[74] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023.
[75] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube )) Large-scale 3d generative modeling using sparse voxel hierarchies. arXiv preprint arXiv:23122 13.003806 2023.
[76] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4219, 2024.
[77] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusion for 3d generation. arXiv preprint wwXiv:2308.16522 2023.
[78] Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. Meshgpt: Generating triangle meshes with decoder-only transformers. arXiv preprint arX5v:2311.1547,, 2023.
[79] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using noncquilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[80] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[81] Jürgen Sturm, Christian Plagemann, and Wolfram Burgard. Adaptive body scheme models for robust robotic manipulation. In Robotics: Science and systems. Zurich, 2008.
15

---

[82] Jurgen Sturm. Christian Plagemann, and Wolfram Burgard. Unsupervised body scheme learning through self pelf-peeception. In 2008 IEEE International Conference on Robotics and Automation, pages
IEEE, 2008
[83] Jürgen Sturm, Cyrill Stachniss, and Wolfram Burgard. A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research, 41:477- 526, 2011.
[84] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303. /4207, 2023.
[85] Jiaxiang Tang. Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309 /6653, 2023.
[86] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 20/2 FEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[87] üirgnn Sturm, Vijay Pradeep, Cyrill Stachniss, Christian Plagemann, Kurt Konolige, and Wol- fram Burgard. Learning kinematic models for artieulated objects. In Twenty- Firesty-Fisstt International Joint Conference on Artificial Intelligence. Citeseer, 2009.
[88] Arash Vahdat, Francis Williams, Zan Gojeie, Or Litany, Sanja Fidler, Karsten Kreis, et al. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems,
2022.
[89] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances neural information processing systems, 30, 2017.
[90] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint /r1iv 10903, 2017.
[91] Clement Vignac, Nagham Osman, Laura Toni, and Pascal Frossard. Midi: Mixed graph and 3d denoising diffusion for molecule generation. arXiv preprint arXiv:2302.00004 002009048 2023.
[92] Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8876 8884, 2019.
[93] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: figh-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[94] Yijia Weng, He Wang, Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, and Leonidas Guibas. Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13209-13218, 2021.
[95] Qianyi Wu, Kaisiyuan Wang, Kejie Li, Jianmin Zheng, and Jianfei Cai. Objectsdf++: Improved object-compositional neural implicit surfaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21 21764-21774 2023.
[96] Ruihai WU, Yan Zhao, Kaichun Mo. Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. arXiv preprint arXiv:2106. 14440, 2021.
[97] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated past-based interactive environment. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 11097-11107, 2020.
16

---

[98] Jiale Xu, Xintao Wang, Weihao Cheng. Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: shoo text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. In Proceedings of the IEEF/CVE Conference on Computer Vision and Puttern Recognition, pages 20908-20918 2023.
[99] Xianghao Xu, Yifan Ruan, Srinath Sridhar, and Daniel Ritchie. Unsupervised kinematic motion detection for part-segmented 3d shape collections. In ACM SIGGRAPHI 2022 Conference Proceedings, pages 1-9, 2022.
[100] Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. Rignet: Neural rigging for articulated characters. arXiv preprint arXiv:2005.0055: 2020.
[101] Zhan Xu, Yang Zhou, Li Yi, and Evangelos Kalogerakis. Morig: Motion-aware rigging of character meshes from point clouds. In SIGGRAPII Asia 2022 conference papers, pages 1-9, 2022.
[102] Zihao Yan, Ruizhen Hu, Xingguang Yan, Luanmin Chen, Oliver Van Kaick, Hao Zhang, and Hui Huang. Rpm-net: recurrent prediction of motion and parts from point cloud. arXiv preprint arXiv:2006.14865, 2020.
[103] Jianglong Ye, Peng Wang, Kejie Li, Yichun Shi, and Heng Wang. Consistent-1-to-3: Con- sistent image to 3d view synthesis via geometry-awan diffusion models. arXiv preprint arXiv:2310.03022 2023.
[104] Li Yi, Haibin Huang. Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas. Deep part induction from articulated object pairs. arXiv preprint vrXiv:180.004417 2018.
[105] Qiaojun Yu, Junbo Wang, Wenhai Liu, Ce Hao, Liu Liu, Lin Shao, Weiming Wang, and Cewu Lu. Gamma: Generalizable articulation modeling and manipulation for articulated objects. arXiv preprint 2339: 2023.
[106) Vicky Zeng, Tabitha Edith Lee, Jacky Liang, and Oliver Kroemer. Visual identification of articulated object parts. In 2021 IEEE/RS International Conference on Intelligent Robots and Systems (IROS), pages 2443-2450 IEEE, 2021.
[107] Guangyao Zhai, Evin Pinar Örek, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Commonscenes: Generating commonsense 3d indoor scenes with scene graphs arXiv preprint 1.XX::2235..6283. 2023.
[108] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023.
[109] Li Zhang, Zean Han, Yan Zhong, Qiaojun Yu, Xingyu Wu, et al. Vocapter: Voting-based pose tracking for category-level articulated object via inter-frame priors. ACM Multimedia 2024, 2024.
[110] Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. arXiv preprint arXiv:2406. 13897, 2024.
[111] Xin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung-YYung Shum. Locally attentional sdf diffusion for controllable 3d shape generation. arXiv preprint arXiv:2305.04444 2023.
[112] Chenliang Zhou, Fangcheng Zhong, Param Hanji, Zhilin Guo, Kyle Fogarty, Alejandro Sztrajman, Hongyun Gao, and Cengiz Oztireli. Frepolad: Frequency-reetiffed point latent diffusion for point cloud generation. arXiv preprint axXiv:2311.12000, 2023.
[113] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 5826-5835, 2021.
[114] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 3d menagerie: Modeling the 3d shape and pose of animals. In Proceedings of the IEEE conference on 2017.
computer vision and pattern recognition, pages
17

---

A Details of the Training and Inference Process
A.1 Training and Inference of the Structure Generator
As highlighted in [37, 46]. the different characteristics of an articulated asset can represented as features of a complete graph Gv {ar, (ar,e), with N nodes, and N(N 1)/2 edges. We train image VQVAE [89] using different views of every single object of the dataset. The latent code of every object composing articulated asset is of dimension 8 8. This latent code is vectorized in an array f; E RR before being concatenated to the reference node feature vector 2; (see Figure 6).
"Night
Image VQVAE
"Night Table Body"
E
Dromes"
"Revolute" Priamalie"
Latons
Latent
"Night Table"
Articulation Graph
Reahope
Figure 6: Detail of the structure generation training pipeline.
The node and edge features of the obtained asset graph GN are then iteratively corrupted by Gaussian noise that Graph ATtention network (GAT) learns to predict (see Figure 7).
"Night
"Night Body"
Laceot
Encodet
Decodet
Image VQVAE
Drawne
"Revolate" Prismatie
x
1
(T-b) Denoise
Diffuse
Reference Articulation Graph
Noisy Graph
Denoised Articulation Craph
Reshape
Articulation
GAT
Figure 7: Detail the structure generation training pipeline.
The denoised graph GN obtained the output of the pipeline then be post-pree processed into human interpretable form, the node, joint and asset categorical information being converted into text while the denoised image latents f, of every node are decoded into corresponding images. This human interpretable content can then be passed as a condition signal to the shape generator (see Figure 8).
A.2 Denoising Diffusion Probabilistic Model (DDPM)
A.2.1 Forward Process
Given a sample Ty drawn from distribution g (2o), series of noisy samples I4, Vt E [1,... can be obtained by gradually corrupting with Gaussian noise ~ ~~ ~N(0,,1 following a variance schedule 3 < ... < Sr:
T
q (0)) =
II ( (.-1),, q (xx, (0 1)= = (V1-((VV/// 111-
(4)
18

---

Table
"Drawer
(T-1Xx Denoise
Incomplete Articulation Graph
Denoised Articulation Graph
GAT
Reshaps
Figure 8: Detail of the structure generation inference pipeline.
relevant property such a diffusion process that æ can be sampled from By through
q( VVVā ãt) I),
(5)
where == 1 Be and ā. = II Otg eventually yielding æ, = vor.00 + vI ōGE.
A.2.2 Reverse Process.
Initiating from standard Gaussian distribution, ær N0, I), denoising model pe parameterized by trainable weights 0, learns to approximate a series of Gaussian transitions pe (2 These transitions incrementally denoise the signal such that
T
Po (ZorT) = po (ZTT II Po (x+-1
(6)
[=1
Po (ar--1 = N (1o (It,,) t). Eg (at, t))
Following the approach from [21], previous work articulated asset generation [37] [46] define Eo (xc, and Eo(x,, t) = of? I yielding the following Langevin dynamics
Bs
Hg
v
VT
B. V1 at
eo (mi,1) o12, (0,1,
(8)
you
where en (Rt, 1) is learnable network approximating the per-step noise on Ct.
A.2.3
Loss Function
The variational lower bound is used to optimize the negative log- log-likelihood. Following the simplified training objective outlined in DDPM [21]. the training loss can simplified to the following expression
C == [|e:- -0 (Vors + vI - ore, 1) 11]
(9)
19

---

B Data Preprocessing
B.1 Mesh Preprocessing
High-quality meshes essential for applications such generating Truncated Signed Distance Functions (TSDFs), which are turn widely used within advanced 3D latent diffusion models such as SDFusion [8]. However, many meshes in popular datasets, including PartNettMobilit [97], contain inconsistencies and errors that render them unsuitable for such purposes. To address these issues, we incorporate preprocessing routine that includes manifoldization process [26] repair and convert flawed meshes into watertight, manifold counterparts (see Figure 9).
(a) Original mesh with inconsistencies
(b) Repaired watertight manifold mesh
Figure 9: Mesh repair pipeline: (a) The original mesh exhibits holes and non-manifold edges; (b) After preprocessing, the mesh is converted into watertight manifold suitable for TSDF generation.
B.2 Orientation Estimation and OBB Computation
Unlike previous approaches [37, (6), do not assume that the dataset meshes already provided in a canonical orientation. Our method hence involves an orientation estimation process for each individual mesh, performed through the calculation of its Oriented Bounding Box (OBB). This OBB is subsequently utilized to align each body into standard pose. The orientation estimation begins by
(a) Original mesh
(b) Mesh with computed OBBs
(c) Mesh with original AABBs
Figure 10: Comparison of bounding boxes: (a) The original mesh: (b) The mesh enclosed by Oriented Bounding Boxes (OBBs), providing tight and accurate fit; (c) The mesh enclosed by original Axis-Aligned Bounding Boxes (AABBs).
applying Principal Component Analysis (PCA) to the mesh data. To mitigate the influence of complex internal geometries, such as shelves within furniture models, we first compute the convex hull of mesh. This step simplifies the geometry while preserving the overall shape. We then voxelize the convex hull and uniformly sample 10,000 volumetrie points from it. Applying PCA these sampled points provides initial estimate of the mesh's 's principal axes and orientation. Building upon this
20

---

initial estimate, we refine the attitude quaternion of the OBB through gradient descent optimization. The objective minimize the volume of the OBB, achieving tighter around the mesh. During optimization, we iteratively adjust the orientation by recalculating the OBB volume along the oriented principal axes. To reliably navigate the solution space and avoid local minima, we incorporate heuristic that explores potential orientations within cone defined by an angle centered around the current best estimate. This approach allows for nuanced adjustments and aids converging to global minimum, thereby enhancing the precision of the orientation estimation. By combining PCA for initial alignment with heuristic-augmennte gradient descent for volume minimization, our method ensures accurate and robust alignment of dataset meshes into their canonical poses (see Figure 10). This preprocessing step significantly enhances the overall quality and consistency of the articulated objects generated by our framework.
(a) Oriented Bounding Boxes
(b) Generated asset with Oriented Bounding Boxes
(c) Generated asset
Figure 11: An articulated asset simulated in the MuJoCo[86] physics engine.
C Controlling Generation with Asset Labels
MIDGArD incorporates categorical embeddings asset types and object categories directly into the node features of the articulation graph. These embeddings enable the model to understand and generate objects that belong to specified category, capturing the common structural and kinematic features associated with that category. The model is trained on diverse dataset that includes various object categories with different articulation patterns. By learning statistical priors over these categories, MIDGArD can infer plausible structures and motions for new objects within the same category. The structure generator is designed to handle incomplete or partial inputs, allowing it to fill in missing details based on learned patterns. In the absence of detailed node-level information, the model can perform unconditional generation, producing coherent articulation graphs solely based on the provided asset category. The of categorical embeddings and latent codes that correspond to numan-uunderstandable concepts (like "storage furniture") makes easier for users to interact with and guide the generation process.
(a) Controlled generations with asset label "storage furniture"
(b) Controlled generations with asset label "fan"
Figure 12: Controlled Generation with Asset Labels.
21

---

D
Controlling Generation with Image Input
key advantage of MIDGArD's modular setup is the controllability of the generative process, not only graph level (by supplying partial graphs) but also on a part-level, by replacing the generated image or text information. In Figure 13, we provide examples how modifying the input image to the shape generator (SDFusion) changes the object appearance. Here, randomly choose images from PartNet Mobility, sampling among all parts with the same category. For example, we sample from from all meshes with the descriptions "A wheel part of a cart" to generate the wheel in the fifth row of Figure 13.
From decoded Images
sampled imgge (1)
Conditioned sarppled images (2)
Ag giaossss body part Eynglasses
Alcc as of deperser
NAP (closest sample in unconditional generation)
Figure 13: Controlling the shape appearance of object parts via image conditioning. Top: Image inputs allow to control the object design part-level. Bottom: NAP struggles to generate objects with many parts and yields unrealistic motion.
22

---

E Encoding object-level information with Graph Attention Networks
For capturing the relationships between parts, we experiment with graph encodings as input the conditional shape generation process. To generate the geometry for a specific part Ø, we construct a simplified graph from the articulation graph that provides information about the size of the in comparison to other parts, as well as the joints between & and other parts. Specifically, let Go aw Ex)) be a graph with 4, as the node feature vector for the i-th node and the Plücker coordinates Pij serving as edge features. We test two options mark Φ the part to be generated: (1) tree structure where Uphi is constructed by reducing the graph to the parts adjacent to d and (2) complete graph (Vs
where the node features augmented with binary indicator that equals I for the node representing o. These structures are processed with a Graph Attention Network (GAT), which is trained concurrently with the diffusion model. As shown in Table 4, the results do not match the ones of a simple MLP embedding, neither with nor without our prior method. We hypothesize that this is due concurrently training the GAT with the diffusion model, possibly converging slower than an MLP. Furthermore, many objects PartNet Mobility have few parts, leading to small graphs of less than 4 nodes. Future work could try pretrain GAT on graph-encoding improve the conditional generation with object-level information.
Table 4: Shape generation results contrasting an MLP-bbaed embedding to GAT encoder.
B
OBB
CD (generaled)
CD (scaled)
bb prior bb MLP prior tree GAT
1146
10.89
0.0072
0.0043
1246
10.97
0.0076
0.0044
bb prior graph GAT original + bb MLP oriented dataset
1226
13.12
0,0084
0.0046
1490
37.15
0.0152
0.0061
original + oriented dataset Iree GAT
1994
69.66
0.0203
0.0051
original + oriented dataset graph GAT
1710
49.42
0,0218
0.0052
original + bb MLP
3795
183.5
0.0307
0.008
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are support by detailed results, comparison State of the Art, open-source code upon acceptance, explicit mention of limitations, and a detailed method section, and thorough steps to reproduce our results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: please find our limitations of the work the Conelusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer:
[NA]
Justification:
This work das not include theoretical results, proofs, or theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: we rigorously outlined the dataset, training method, implementation detail in the Result section and will enable reproducibility of results through open-source code upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Data of the results is provided at https: htttps//qquentn leti--eebbut.ttte github.. io/MIDGArD/. Opensource code will be provided upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
casssioooolli imPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /ttpp.ccpppplliippppp cooollc/guidess
ceabbbmmsssonPPe ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we rigorously outlined the dataset, training method, implementation detail in the Result section and will enable reproducibility of results through open-source code upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: no experiments that produce error bars or similar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: yes, details are provided in the implementation details section of results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttear////////.././ee cc http/uublc//tttie wtw.:hhhhiciiideeeeeee
Answer: [Yes]
Justification: we follow the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: no societal impact of the work is apparent.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: produced all codes and original owner of assets. For third- parrt-party code, we explicitely cite and mark the dependency to those and credit the authors accordingly.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcee thcode. wwwccoo//a/oo/oo datasatssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provided detailed explanation in the paper, will have documentation in the open-source code, and provide a detailed Supplementary Materials pdf with additional info.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "jWdvoMR3P2", "reviewer": "Reviewer_T3Uj", "summary": "This work addresses the task of 3D asset generation for articulated objects.\n\nThis work aims to enhance the prior approach by achieving three main objectives: 1) increasing the interpretability and controllability of the generation process; 2) generating more natural joint motions and 3) reducing violations of physical constraints.\n\nFor the first goal, they propose a framework MIDGArD to decompose the generation into two sequential modules: 1) a structure generator that unconditionally generates a kinematic graph and attributes for each articulated part; 2) a shape generator to produce part SDF that allows conditioning on the graph from the first module, an image, and a text description for each part.\n\nFor the second goal, they introduce a representation for joint parameters that allows the diffusion process to operate on the Plucker manifold directly.\n\nFor the third goal, they replace the AABBs with OBBs to bound the part shape which is claimed to achieve better part alignment.", "questions": "- The argument in lines 47-48 “leveraging categorical embeddings for improving consistency across the graph” is unclear. What does consistency mean exactly? The ablation experiment to support this argument is missing.\n- Would it be possible to compare with CAGE [43] on certain aspects of the generation? E.g., in terms of the physical plausibility of objects in both abstraction level (bounding boxes) and mesh output. \n- The argument in line 33 “both NAP [35] and CAGE [43] provide no control on the part level due to their opaque encoding as a latent” is a bit misleading. What specific control on part is missing from the prior work? Also, to correct the fact, the representation used in CAGE has no latent encoding for any attribute, while the representation used in NAP only makes the geometric feature encoded as latent and leaves others in an explicit form.\n- How robust and generalizable the image control is?", "limitations": "- Under the formulation of this work, the image control from users is not particularly practical:\n\t- The image of a single articulated part is not natural to find easily.\n\t- The control is more of a post-editing. The user has to wait to see what object is generated and then find compatible part images to feed into the corresponding node in the graph. Otherwise, it won’t be matched with the articulation parameters and the part arrangement being generated.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "- This work contributes to an increasingly important area and identifies three valuable perspectives for articulated object generation.\n- The proposed framework that separates the generation of the articulation structure and part geometry by using images as an intermediate proxy allows users to control the part shape more explicitly, compared with the prior work NAP [35] which requires modifying the latent code instead. \n- Both quantitative and qualitative evaluations are provided to demonstrate improved quality of data distribution modeling in the unconditional generation setting.", "weaknesses": "- The technical contribution is limited: \n\t- The network architecture of the structure generator is the same as NAP [35]. The contribution for this part is only the representation alternation for joint parameters and bounding boxes. These are reasonable changes for slightly better distribution modeling, but the insight is not particularly significant.\n\t- The shape generator is also adapted from an existing work SDFusion [7], where the benefits of detailed geometry modeling with TSDF and the flexibility of multimodal conditioning are just inherited from the original work. The only modification is an additional conditioning from a graph. However, how this graph condition affects the part geometry output and whether it is necessary is unknown. Also, there is no strong/sufficient qualitative evidence to show a better geometry generation compared to NAP.\n- The experiments and evaluation are insufficient and unclear:\n\t- The side-by-side qualitative comparison with prior work (e.g. NAP [35], CAGE [43]) is missing to convince the improvement in the claimed aspects.\n\t- Unfair comparison with NAP in the “reconstruction” setting in Table 1. According to the description in lines 280-282, NAP is only provided with graph topology and motion attributes for each node. In this case, there is no way for NAP to reconstruct the object with no shape information (which should have been given the latent code extracted from the ground truth part geometry). In contrast, MIDGArD is additionally provided with shape features extracted from the ground truth node images.\n\t- Lack of quantitative and qualitative results to support the argument of more “natural joint motion” and “less physical constraint violation” and how these improvements are correlated with the specific designs. Other ablation studies discussed in the manuscript are also generally hard to follow.\n- The related work section does not contextualize this work very well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1.1** While our structure generator builds up on NAP \\cite{lei2023nap} and also applies a denoising diffusion process on the graph, the representation and generative process are fundamentally different:\n1.) Our approach *modularizes* articulated asset generation. This fundamentally diverges from the approach taken in NAP which aims to learn both shape and structure generation within one model. While both the shape generator and the structure generator build up on previous work, they are substantially modified to enable their interplay for generating articulated assets. Only this modular approach enables fine-grained control on the object level (Table 1 and Fig. A in the attached PDF) and part level (Fig. 3, Fig. 5 appendix). 2.) The graph representation is different to NAP's representation. In the node features, the geometry encoding is removed, the image encoding and categorical variables are added, and bounding boxes are replaced by oriented bounding boxes. Only the part-existence indicator is the same. For edge features, joint category is added and we directly denoise on the Plücker manifold. 3.) These changes enable more control and enhances intuitiveness for the use (see examples for conditional generation, Figure A in the attached PDF). \n\n**Weakness 1.2** We respectfully disagree. As outlined in Section 3.4, we do not only supply the graph as another conditioning input, but modify the SDFusion pipeline to generate only the *difference* to the bounding box (see Figure 2). Thus, the core innovation within the shape generation pipeline is the introduction of a bounding box constraint. To the best of our knowledge, this is novel. This constraint, combined with the use of oriented bounding boxes (OBBs), enables generation of geometries that are more closely aligned with the articulation solutions proposed by the structure generator, as demonstrated in Table 2. The synergy between conditional graph generation, and constrained 3D generation based on OBBs results in realistic articulated assets with appropriate dimensions, as shown in Fig. 4A.\n\n**Weakness 2.1** Per the reviewer's suggestions, we've added a side-by-side qualitative comparison with NAP (see general author rebuttal). Figure B in attached PDF shows that our model 1) improves the quality of the geometry (see fan), 2) improves the physical plausibility (see laptop motion), and 3) generates more consistent and realistic shapes (see both globes). \n\n**Weakness 2.2** We acknowledge the concern raised regarding the fairness of the comparison, and will clarify this in the revised manuscript. Our aim was to demonstrate the conditioning capabilities of our approach, which are not present in NAP. We do not see a feasible alternative that would ensure a fair comparison and are very open to a fairer comparison that can highlight MIDGArD's capability of conditioning (compared to the lack of any text- or image-based conditioning in NAP) as one of the novelties and contributions of our work. \n\nRegarding the suggestion that NAP should have been given the latent code extracted from the ground truth part geometry: This would create an unfair advantage over our method, which relies solely on single-view images. Additionally, the practical applications where part-geometry is readily available are quite limited, thus the utility of such a capability is uncertain. Leaving this drawback aside, for a fair qualitative comparison one can consider the Part2Motion experiment in NAP (\\cite{lei2023nap} Fig 5) as the counterpart to our results for text- and image-based conditional generation provided in the attached PDF Figure A.\n\n**Weakness 2.3** See general response for experiments and results.\n\n**Question 1** By \"consistency across the graph,\" we refer to the semantic coherence of part relationships within the articulated structure. For instance, a \"bottle\" is unlikely to contain a \"drawer\". We found that adding categorical information to the graph improves the consistency. In MIDGArD-generated data, we found that our model generates 100% valid examples (see general author rebuttal).\n\nQualitatively, this advantage of our approach is seen in the failure cases (attached PDF Fig. C). In NAP, the lack of categorical information leads to objects of mixed parts that do not fit together. In our approach, there may be issues with the geometry or kinematics, but the object parts remain consistent. \n\n**Question 2** \nA fair comparison with CAGE is not possible, because CAGE was designed for a very limited setting, i.e., (1) providing shape geometry via \\textit{part retrieval} instead of part generation. This method does not generalize and leads to failure cases where parts overlap and don't fit (see \\cite{liu2023cage} Fig 10), (2) assuming canonical poses of the objects and axis-aligned bounding boxes. Thus, the experiments in CAGE are restricted to 8 categories from PartnetMobility (3) requiring actionable parts, e.g., doorknobs and handles, which were added by the authors of CAGE and only for 8 categories.\n\n**Question 3** We have corrected and clarified the mentioned line. Indeed, for NAP we refer to the latent encoding the geometry. For CAGE, it is rather the part-retrieval procedure that prevents control on this level. \nOur framework allows users to specify part-level details through images and text, e.g., requiring a lamp to consist of a \"lamp stand\" and a \"lamp shade\" and guiding the style of the lamp shade with an image. This control mechanism is not available in prior work.\n\n**Question 4** It generalises. See general response for results.\n\n**Limitation 1** One could leverage existing image segmentation pipelines to automatically extract relevant images from a single picture. Also, one image oftentimes suffices (e.g. sunglasses), 3) we are planning to provide a library of part images to assist guidance.\n\n**Limitation 2** As shown in the attached PDF Figure A, the user can directly input images and text instead of post-editing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zUk6BamJN9", "reviewer": "Reviewer_jtcL", "summary": "This paper proposes several interesting improvements over existing articulated object modeling and highlights higher-quality part generation.\n\nSpecifically, an articulated object is parameterized to a graph where parts are nodes and joints are edges. This paper proposes a multi-model part VAE for generating higher quality and controllable shapes and improving physical awareness through a bounding box alignment. It also improves the kinematic structure generation by considering parameterize on the plucker manifold. With these technical contributions, the results show significant improvement over baselines and the supplemental repo provides simulatable mujoco files, showing the practical value of this work.\n\nI recommend a clear acceptance of this paper.", "questions": "Please see the weakness", "limitations": "Adequately discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good results, supp MuJoCo simulation: I really like the MuJoCo demo in the suppl repo, which is impressive.\n- Several technical solid and important improvements including the shape representation, plucker manifold encoding, and bounding box alignment. etc\n- Multimodel information and condition: i specifically like the text and language part of the part shape encoding, which opens a lot of opportunities for conditioned generation and connection to LLM.", "weaknesses": "- More conditioned generation results: while the part-level multimodel information is used in shape encoding, it would be better to show more conditioned generation results using this information.\n- Open source the part text and bounding box alignment data: it's not clear whether the shape part encoding training data will be publicly released or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlightning the strength of our approach. We are delighted that you find the MuJoCo simulations in the supplementary repository impressive and recognize the significance of our technical improvements, including shape representation, Plücker manifold encoding, and bounding box alignment. These were key motivations in our effort to advance the field of articulated object modeling. \n\n**Conditioned generation results: while the part-level multimodal information is used in shape encoding, it would be better to show more conditioned generation results using this information.**\n\nWe acknowledge your request and have included additional conditioned generation results in the attached PDF, specifically Figure A (\"Conditional generation\"). This figure showcases our model's capability to generate consistent articulated assets based on supplied part features in form of image and text. In addition to the experiment in Figure 5 of the original submission, this figure shows two settings: Generating an articulated object solely based on image and text input, and generating the articulated object based on image, text and bounding box input. The latter is comparable to the \"Part2Motion\" setup in NAP. However, in NAP one would have to provide the full geometry for each part, whereas we demonstrate the same capability just based on human-interpretable input features. \n\nFurthermore, we have extended our experiment on image-based conditioning (Figure 3 and 5 of the original manuscript) by another experiment using images from arbitrary viewpoints for conditioning (Figure D in the attached PDF). This shows the generalisation capability of our approach.\n\nWe hope these experiments match the reviewer's expectations and we welcome further suggestions to improve our empirical results.\n\n**Open source: it's not clear whether the shape part encoding training data will be publicly released or not.**\n\nThank you for your suggestions. We commit to open-source the data (text, bounding box etc) and source code upon the paper's acceptance facilitating reproducibility and further research in this area. This commitment aligns with our goal of promoting transparency and accessibility within the research community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UAtqHaVjuU", "reviewer": "Reviewer_2qcf", "summary": "This work tackles the problem of generating articulated 3D assets that are animatible. The authors mention that their generated shapes are directly compatible with existing physics simulation tools, i.e. MuJoCo. To this end, they first propose a structure generator, which conditionally or unconditionally generates an articulation graph encoding structural object information such as the kinematics attributes, i.e. the kinematic tree and joint types. Next, the multi-modal shape generator synthesizes a shape that follows the kinematic tree and size of the object while also allowing conditioning on multi-modal inputs such as text and images. Results demonstrate some improvement over the SOTA (NAP).\n\nOverall, I like the setting that the authors propose as well as their technical contribution to solve it. There are a few minor concerns about the presentation in terms of writing, clarity, and qualitative visualizations (see comments above). However, I believe those are minor. Thus, I recommend acceptance.", "questions": "--", "limitations": "Limitations are sufficiently discussed in the main paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Interesting setting with a lot of potential for future work. It seems only very limited amount of works (NAP) have explored this setting\n- Evaluation is to the best of my knowledge complete.\n- Well-written related work section and references seem to be complete\n- The paper is technically sound", "weaknesses": "- Writing:\n    - I would limit the contribution bullets to the technical contribution of this work rather than focusing on results and open sourcing.\n- Clarity\n    - I would introduce proper notations for all steps discussed in section 3.3 as I feel it is hard to understand the setting just from the textual descriptions, e.g. what is input, what is output.\n    - I have a similar concern regarding section 3.4\n- Video\n    - I would have expected to also see some video results of the generated and animated objects. While this is not strictly required, it would have made the exposition much more complete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback that help us to improve the clarity and quality of our paper.\n\n**Writing (comment: I would limit the contribution bullets to the technical contribution of this work rather than focusing on results and open sourcing)**:\n\nThank you for your suggestion. We fully agree and will replace bullet points 4-6 with the following technical contributions: \n* An approach for constrained shape generation within oriented bounding boxes that improves alignment of the kinematic links by 17\\% on average. \n* A pipeline to create fully simulatable assets with an interface to Mujoco \n\n**Notation and clarity (comment: I would introduce proper notations for all steps discussed in section 3.3 as I feel it is hard to understand the setting just from the textual descriptions, e.g. what is input, what is output. I have a similar concern regarding section 3.4)**\n\nThank you for this suggestion to improve the clarity of the paper. We will rewrite this part to improve the description of the setting. Specifically, we will add that we train the model to learn the distribution of articulated object graphs, by applying a denoising diffusion process: \n* Input: Noisy Graph $\\boldsymbol{\\tilde{G}}_N = \\left\\lbrace \\boldsymbol{\\tilde{x}}, \\boldsymbol{\\tilde{e}} \\right\\rbrace$ with node and edge features as introduced in section 3.3. For conditional generation, the input is a partially noisy graph, where certain node or edge features are masked.\n* Output: Denoised articulation graph $\\boldsymbol{G}_N = \\left\\lbrace \\boldsymbol{x}, \\boldsymbol{e}\\right\\rbrace_N$ \n \nSimilarly, for section 3.4, we will add the following description: \n\nThe aim of the structure generator is to learn the conditional probability distribution $P(z_i | a_i, b_i, d_i, r_i, t_i, g_i)$ for generating the latent representation $z_i $ of a part geometry's Signed Distance Function (SDF) using a diffusion model, conditioned on inputs $[a_i, b_i, d_i, r_i, t_i, g_i]$ as explained in section 3.3.\n\nThis is achieved 1) by transforming $a_i$ and $b_i$ into a text description and encoding with the BERT model, 2) by transforming the image (decoded $g_i$) via a ResNet and 3) by applying constrained generation with $r_i$ and $t_i$ as explained in section 3.4\n\n**Video (comment: I would have expected to also see some video results of the generated and animated objects. While this is not strictly required, it would have made the exposition much more complete.)** :\n\nWe have provided videos in a repository here: https://anonymous.4open.science/r/MIDGArD-E1DE/README.md (see folder “gifs”). We apologize if the reference to this repository was unclear in the paper. Upon publication, we will provide a proper website with these examples in addition to the open-source code.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AWzbLkmHV3", "reviewer": "Reviewer_czmP", "summary": "This paper presents a diffusion-based framework for generating articulated objects represented as part graphs. Different from prior work [35] that generates both the part shapes and graph structures simultaneously, this work introduces a two-stage strategy that first generates the structure with per-part conditions and then generates the part shapes. It also proposes a part latent encoding that enables multimodal conditions including text and images.", "questions": "I am curious about some details:\n- Sec. 3.4 says \"During training, the model uses renderings of the part mesh from a frontal view\" (line 217). How is the frontal view chosen? For inference-time image-conditioned generation, do the images have to be in frontal view, or can they be of any view? Are the images in Suppl. Fig. 5 of \"frontal view\"?\n- I wonder how exactly are the bounding boxes represented and predicted. Because for articulated objects, the part poses are changing during part motions. [35] only uses the rest-state bounding boxes (together with the joint limits). In this work, is the rest-state bounding box viewed as having the identity pose matrix?", "limitations": "Limitations are well-discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The reparameterization of the Plucker coordinates is simple but more natural than that in [35], making the diffusion results better satisfying the constraints.\n- The designed node embedding with image and text features is not only a more detailed representation of part shapes, but also enables more flexible conditioned generation for different downstream applications.\n- The framework relaxes the constraints of training on canonical-posed shapes. This is an important relaxable for better usage of data.", "weaknesses": "- As also briefly discussed in the paper, generating the parts independently is a bit unnatural to this problem, especially given the importance of inter-part motions/relations in articulated objects. But overall I am OK with this point given all the node conditions in the framework.\n- Quantitatively, the improvements compared to [35] on unconditioned generation seem marginal.\n- The description \"simulation-ready\" sounds overclaimed to me. When talking about \"simulation\", I feel most people will expect physical viability and accuracy. But from the demonstration on the website, it seems that the parts are not well connected. There is also no explicit physics enforcement in the framework.\n\nMinor things:\n- I believe it would be better to swap the two sentences \"This image latent is derived by\" (line 148) and \"...the structure generator denoises a latent representation of an image...\" (line 150), so the readers know where the \"image latent\" in line 148 comes from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Part-level generation**\n\nPart-level generation requires conditioning on the articulation graph, since the appearance of a part depends on its role within the graph. In future work, we aim to test further possibilities to condition the part-generation on already existing parts, e.g. by framing it as a shape completion problem. However, we chose this approach since it also has several benefits:\n\n* As already noted by the reviewer, it allows for part-level control via text or images.\n\n* learning to generate object parts is arguably simpler than learning to generate arbitrary objects. Generating a complex high-resolution object is still an open challenge in 3D generation, but generating basic components of objects such as wheels and boxes is feasible. \n\n**Unconditional generation**\n\nNAP is already a rather compact and powerful model but the quality measurement metric (Instantiation Distance or ID) has limitations, as it mixes shape geometry with structure and motion evaluations; two properties that should be separately measured. Since points are sampled randomly for the CD, this ignores unrealistic states of small parts, such as the lid of the bottle in Fig 4. This is an issue that was also raised in CAGE. However, for a fair comparison to NAP's results, we used their evaluation protocol. We provide a qualitative comparison between NAP and MIDGArD in the attached PDF, Figure (B). We also provide further quantitative results to show the improvement of our approach in terms of physical plausibility, we compute the distribution of joint types and compare the distribution among our generated objects with the ones from NAP and the real data. As shown in the main rebuttal section, our model learns to match the distribution well. Since we determine the joint ranges based on the type, this directly leads to an improved motion plausibility. Finally, it is worth noting that one of our contribution is the higher level of control achieved with our approach, offering novel ways to guide the generation of articulated objects. Please refer to the attached PDF, specifically Figure (A) in the main rebuttal section.\n\n**Simulation ready sounds overclaimed.**\n\nBy \"simulation-ready\", we refer to our contribution of providing the possibility to export the result to Mujoco, which is indeed a physically accurate simulation environment. Building a MuJoCo environment with multiple articulated assets will be straightforward with our codebase, which is why “simulation-ready” seemed to be a suitable term. Nevertheless, we do not want to neglect the huge effort involved with building full simulations, so we will weaken “simulation-ready” to “simulatable” in our revised manuscript.\n\n**It seems that the parts are not well connected**\n\nWe believe the reviewer refers to the fact that screws and other connectors are not shown in the simulated videos. The connectors are omitted since they are not part of the ground truth dataset. However, adding those would be straightforward based on the predicted edge features in the graph (joint type \\& Plucker coordinates).  The crucial part of the simulation in our point of view is the motion of the object parts, which is realistic in our videos. Please let us know if we have addressed your concerns or if any further clarifications would be useful.\n\n**There is no explicit physics enforcement.** \n\nWe agree that there is no strict enforcement of physics in the framework; however, this is a rather difficult endeavor as it usually requires the interaction of the learning pipeline with a physics engine. Meanwhile, several components of our framework ensure physical plausibility:\n\n* Our bounding-box constrained generation approach (with subsequent part scaling and orientation) ensures that the parts fit within the object in a physically plausible way. \n\n* We enforce realistic kinematics by specifying the joint ranges based on the joint category. As Fig 4B shows, this works much better than the approach taken in NAP, i.e., predicting the joint range directly.\n\n**Frontal-view images**\n\nWhile we use a specific perspective for training and testing in the initial submission, we provide additional results here showing that our model generalizes to image inputs from other perspectives (see Fig. D in the main rebuttal section). \nThe PartnetMobility dataset is already normalized such that all parts are shown from the front (if such asymmetry even exists). However, rendering with a camera angle of [0,0,0] is problematic since some objects are not recognizable from the front; e.g., a wheel may appear as a rectangle from the front. Therefore, our “frontal view” refers to a rendering from an axis angle [\\phi/6,  \\pi / 12, 0] which usually shows the relevant part geometry. We've added this clarification in the manuscript. \nDue to the variety of parts in our training dataset, we hypothesize that our model should also generalize to images from other views. We tested this by conditioning the model on images from a randomly sampled perspective.\n\n**Are the images in Suppl. Fig. 5 of \"frontal view\"?**\n\nYes, the images in Suppl Fig 5 are rendered with the same angle as the images used for training \n\n**Bounding boxes**\n\nIn the articulation graph, the part-bounding box is represented by three parts: 1) its center, 2) its size 3) its rotation. These three components are generated for each part in the structure generator. The shape generator, in turn, is conditioned on the desired size (2) and generates a centered object of the correct size. The rotation (3) is applied afterwards. Note that this approach allows to generate parts in arbitrary orientation, in contrast to previous approaches (CAGE), that assumes that all parts have axis-aligned bounding boxes when the full object is in resting state.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the editors and reviewers for the time they devoted to reviewing our paper and for their valuable feedback and constructive criticism. We have endeavored to address every suggestion and additional comment to the best of our abilities. Below, we provide a summary of our approach to the review:
*  **Additional Experiments and Comparative Analysis:** We provided additional experiment results, such as a (1) qualitative and quantitative side-by-side comparison to NAP [37], (2) additional conditional generation results, and (3) quantitative analysis of the plausibility and consistency of the generated assets (see below).
* **Clarifications and Revisions:** We have improved the clarity of our method and results thanks to thoughtful feedback of the reviewers and thoroughly incorporated all reviewer's feedback. Detailed descriptions of the data processing in the structure and shape generator are provided, along with empirical results highlighting conditional generation capability and qualitative improvements.
* **Related Work:**
    We have revised the related work section to better contextualize our contributions within the existing literature and emphasize the novelty and impact of our work.

We hope these revisions and clarifications address the reviewers' concerns and enhance the understanding of our contributions. We remain committed to further improving our manuscript and welcome any additional feedback.

Best regards,

Authors of the submitted paper

**------------------- Additional experiments------------------**

**1) Conditional generation capability**
We provide additional results demonstrating the conditional generation capability of our approach in a "PartToMotion" setup where the model is provided with part features only (i.e. no joint data) and outputs consistent articulated assets (see attached PDF - Figure "conditional generation"). In contrast to NAP, our model can be guided with image and text input instead of requiring full geometries.

**2) Side-by-side comparison**
Figure B in the attached PDF provides a side-by-side comparison between our approach and NAP. Since the graphs generated by NAP do not include categorical information, we manually go through the generated data of NAP and MIDGArD and selected pairs of assets with the most similar appearance. Figure B shows that our approach 1) improves the quality of the geometry (see the fan and holes in drawer-meshes), 2) improves the physical plausibility (see laptop motion), and 3) generates more consistent and realistic shapes (see both globes).

**3) Physical plausibility** 
We provide further results supporting the improvement in physical plausibility achieved with our method. In Figure 4, we have already shown qualitative examples where NAP yields unrealistic joint ranges. Our method alleviates such failure cases by introducing categorical joint labels in the articulation graph. Here, we support this observation with a quantitative analysis of the generated joint types. Specifically, we compare the distribution of joint types in the training data with those in samples generated by NAP and MIDGArD (400 samples each). To minimize the impact of objects having many joints of the same type, we count each joint type only once per object. 

|                     | Screw | Revolute | Prismatic |
|---------------------|-------|----------|-----------|
| Training data       | 0.06  | 0.62     | 0.32      |
| NAP-generated       | 0.95  | 0.01     | 0.04      |
| MIDGArD-generated   | 0.02  | 0.62     | 0.36      |

The results indicate that NAP predominantly produces screw joints, despite their low occurrence in the training data. Conversely, the objects generated by MIDGArD exhibit a joint type distribution similar to that of the training data, thereby enhancing the physical plausibility of the generated data. The Chi-Square statistic, which measures how much the observed counts deviate from the expected counts, confirms the large difference with $\chi^2(\text{NAP}) = 5618$ whereas $\chi^2(\text{MIDGArD})=12.7$. NAP's high $\chi^2$ value indicates that there is a large difference between the observed and expected counts, whereas MIDGArD's low $\chi^2$ indicates that the observed data are close to the expected data. More intuitively, the maximal misalignment of our method in the three joint categories is 4% compared to maximal mismatch of 89% for NAP. 

**4) Consistency of parts within the object**

By "consistency" we refer to the semantic coherence of part relationships within the articulated structure. For instance, a "bottle" asset is unlikely to contain a "drawer" body. We found that our approach of adding categorical information to the graph improves the consistency. Unfortunately, it is not possible to measure the consistency within NAP-generated objects due to the lack of categorical information. In MIDGArD-generated data, we can compute whether part-types (e.g., "leg") occur in conjunction with a fitting asset type (e.g., "table"), or whether they are mixed (e.g., a "leg" being part of a "laptop"). Assume that every part-asset-type combination occurring in the training dataset is valid, our model actually generates 100% valid examples. In other words, none of the generated graphs is a mixture of parts that typically belong to different asset types.

Qualitatively, this advantage of our approach is visible in the failure cases (see attached PDF - Figure C). In NAP, the lack of categorical information leads to objects of mixed parts that do not fit together. In our approach, there may be issues with the geometry or kinematics, but the object parts remain consistent. 

**5) Generalisation to images from various perspectives**: We show in the attached PDF - Figure D that our conditional part-generation generalizes to images from different viewpoints.

These results will be included in the revised manuscript.

Meta-review: This paper received divergent initial opinions with three reviewers in support of acceptance and one negative reviewer.  The main concerns expressed by the reviewers were: unclear comparisons or missing comparisons against the most relevant prior work (NAP and CAGE), questions about the fairness of the evaluation methodology, a somewhat contrived "single part in an image" conditioning setup, seemingly marginal quantitative improvements over the prior work baselines, and a lack of discussion about how the contributions of the paper relate to those of the most relevant prior work (NAP and CAGE).

The rebuttal responded to reviewer concerns with detailed clarifications and additional experiments.  The positive reviewers remained positive. The one initially negative reviewer raised their strong reject to a weak reject but remained unconvinced with the rebuttal clarifications on whether the comparisons and evaluation against prior work are complete and fair enough to substantiate the claimed contributions.  Due to the severity of these concerns, there was extensive discussion in the rebuttal and post-rebuttal period.  Taking into account the willingness of the authors to clarify and improve the paper, the AC and reviewers support acceptance, provided the following revisions are incorporated into the manuscript:
1. More precisely articulate the contributions (e.g., how image/text/graph control is specified by the users, and how the Plücker manifold enhances the representation).
1. Acknowledge aspects of the approach inspired by prior work and how their benefits align with those demonstrated in the prior work (e.g., conditioning on object categories to achieve semantically consistent parts and adding joint type as a categorical label to improve kinematic plausibility, as shown in CAGE).
1. Qualitatively demonstrate the diversity of outputs when using the same image/text input.
1. Be transparent about how the comparison with NAP is conducted and clarify why CAGE was not included as a comparison point.

Thus, the AC recommends acceptance and strongly encourages the authors to incorporate clarifications and revisions from the review cycle and especially targeting the above points, into the final manuscript.
Decision: Accept (poster)