MIDGArD: Modular Interpretable Diffusion over Graphs for Articulated Designs

Authors: quentin  leboutet
nina  wiedemann
zhipeng  cai
michael  paulitsch
kai  yuan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:02:32

Abstract: Providing functionality through articulation and interaction with objects is a key objective in 3D generation. We introduce MIDGArD (Modular Interpretable Diffusion over Graphs for Articulated Designs), a novel diffusion-based framework for articulated 3D asset generation. MIDGArD improves over foundational work in the field by enhancing quality, consistency, and controllability in the generation process. This is achieved through MIDGArD's modular approach that separates the problem into two primary components: structure generation and shape generation. The structure generation module of MIDGArD aims at producing coherent articulation features from noisy or incomplete inputs. It acts on the object's structural and kinematic attributes, represented as features of a graph that are being progressively denoised to issue coherent and interpretable articulation solutions. This denoised graph then serves as an advanced conditioning mechanism for the shape generation module, a 3D generative model that populates each link of the articulated structure with consistent 3D meshes. Experiments show the superiority of MIDGArD on the quality, consistency, and interpretability of the generated assets. Importantly, the generated models are fully simulatable, i.e., can be seamlessly integrated into standard physics engines such as MuJoCo, broadening MIDGArD's applicability to fields such as digital content creation, meta realities, and robotics.

Main paper: 
 introduction: 
 Despite their significance for applications in meta realities and embodied AI, the creation of 3D models of articulated objects remains a manual endeavor, and existing datasets [92, 97, 65, 102, 47, 63, 15] are often limited in both scope and scale. Human designers utilize strong prior knowledge of object geometry and kinematic structures, a capability that automated systems have yet to fully replicate.While extensive research has been conducted on generative models for static 3D objects [78, 74, 50, 48, 35, 8, 39, 54, 2, 95, 103, 24, 68, 9, 88, 75, 93, 6, 85, 98, 77, 52] and scenes [38, 31, 17, 71, 84, 14, 72, 53, 7, 12, 107] , integrating 3D part geometry with kinematic structures has received far less attention. Generating articulated objects that are both geometrically detailed and kinematically accurate presents unique challenges due to the complexity of modeling part interactions and motions.Recent research efforts, such as Neural 3D Articulation Prior (NAP) [37] or Controllable Articulation GEneration (CAGE) [46] , started addressing this gap, pioneering direct end-to-end generation of complete articulated assets using a denoising diffusion process acting on articulation graphs. However, generating consistent objects with high-fidelity shapes remains a challenge. Identified shortcomings of NAP [37] include 1) unnatural motion caused by unconstrained screw-like joint parametrization, 2) limited controllability, 3) limited interpretability and 4) the generation of inconsistent or unidentifiable shapes. Additionally, both NAP and related approaches lack part-level control due to the opaque encoding as a latent within the graph, highlighting the need for enhanced interpretability.In this work, we introduce MIDGArD, a novel framework designed to generate interpretable and simulatable 3D articulated assets in a controllable manner. MIDGArD addresses the limitations of previous methods by offering: 1) improved consistency in joint motion, 2) enhanced controllability of the generation process, 3) increased interpretability and 4) better overall asset quality. Furthermore, the 3D assets generated by MIDGArD are fully simulatable and can be seamlessly integrated into virtual environments or physics engines such as MuJoCo [86] , significantly enhancing its utility for digital content creation and robotics. MIDGArD employs a modular and sequential approach based on two diffusion models: the structure generator and the shape generator (see Figure 1 ). The structure generator denoises an articulation graph, unconditionally or from incomplete heterogenous inputs. This articulation graph acts as an abstract, yet interpretable object representation, encoding the structural and semantic information of every link, as well as kinematic attributes.MIDGArD overcomes the identified limitations of prior works by 1) leveraging categorical embeddings for improving consistency across the articulation graph; 2) representing nodes (i.e. unique object parts) via latent image codes rather than direct mesh encoding; 3) diffusing on the Plücker manifold, thereby eliminating the need for discretizing the generated motion parameters, and 4) accounting for the orientation of the parts for more consistent generations. As additional benefit, these modifications yields an image representation, and asset-level, body-level, and joint-level categorical data, that are human interpretable. Since the generation process is sequential, these quantities can be adjusted by the user before generating proceeding with the part geometry generation. Crucially, this interpretable structure enables the use of a multi-modal 3D generation model as the shape generator. The shape generator, here represented by SDFusion [8] , is queried multiple times to produce high-quality geometries for each part of the object (see Figure 1 ). To enhance consistency, we propose a constrained generation approach that enforces the part geometry to fit a given bounding box, here provided by the structure generator.We demonstrate MIDGArD's capabilities by showcasing improved structural quality, consistency, and part-level control compared to NAP. Furthermore, we show qualitative results of MIDGArD generating diverse articulated objects dependent on the partial graph input and shape conditioning. To further improve the consistency and quality of the generated shapes, we propose a to create oriented bounding boxes that have a tighter fit to the ground truth while being properly aligned. The reduces the overlapping volume by 17% on average. Our contributions can be summarized as:1. MIDGArD: A novel diffusion-based framework for generating high-quality, articulated, and simulatable 3D assets with improved consistency between structure and shape.2. Enhanced Interpretability, and Controllability: Providing human-interpretable intermediate outputs and allowing user adjustments at multiple levels of the generation process.3. Enhanced Quality and Scalability: Relying on state-of-the-art 3D foundation models for shape generation.4. Improved part alignment: Introducing a bounding-box constrained shape generation approach that improves alignment of kinematic links by 17% on average. 
related works: 
 Structural Characterization and State Estimation of Articulated Objects Designing articulated objects involves understanding both their geometry and kinematic properties. Traditional methods often require extensive manual labor or rely on limited datasets, hindering scalability and applicability [45] . Early work on morphable templates, such as SMPL for human bodies [55] and SMAL for quadruped animals [114] , demonstrated the potential of matching existing articulation structures (i.e., skeletons) to a wide variety of inputs such as static meshes or monocular videos, thereby substantially simplifying the rigging process. However, while effective within specific categories, these templates lack the flexibility to generalize to arbitrary articulated mechanisms. To address these limitations, recent approaches have enhanced templates by incorporating implicit representations to model geometric deformations and appearances [40] . Robust neural rigging methods [61, 70] leverage template priors while remaining flexible enough to handle mesh elements that deviate from the template, such as clothing or hair. On the other hand, template-free methods [100, 101] rely solely on input mesh geometry but are still primarily limited to humanoid characters due to the scarcity of comprehensive datasets. Emerging approaches [27] aim to rig arbitrary static assets, potentially paving the way for a unified, template-free rigging pipeline. Our research aligns with this direction by learning articulation priors for arbitrary objects without depending on predefined templates.Significant strides has also been made in estimating articulation joint states and parameters from sensory observations. Early contributions such as [83] demonstrated that it is possible to learn compact kinematic models of entire articulated objects solely based on pose observations. Probabilistic models have enabled robots to learn their own kinematic structures through self-observation and motor babbling [10, 81, 82] . Gaussian processes have been employed to model part connectivity and link articulation in objects [87] . Interactive perception techniques further enhance this learning process by allowing robots to interact with objects to obtain their kinematic model [69, 33, 64, 18, 66, 3, 22, 44, 25] . Deep learning techniques were also used to train dedicated architectures to directly predict articulation models from sensory inputs [23, 104, 1, 41, 49, 94, 106, 34, 29, 28, 51, 102, 13, 99, 105, 43, 20, 109] . Interactive systems [67, 13] learn to predict potential motions of articulated objects, aiding downstream motion planning and interaction, while reinforcement learning based approaches [96] train policies to explore diverse interaction trajectories, contributing to actionable visual representations. Our method diverges from traditional approaches by not relying on explicit estimation pipelines; instead, we learn articulation priors implicitly within our model. Articulated Object Generation Generating articulated objects involves synthesizing models that accurately represent both geometric details and kinematic behaviors. Methods such as Self-Supervised Category-Level Articulated Object Pose Estimation [51] and PD (Pose-aware Part Decomposition) [34] address the generation of joint parameters and part poses, respectively, contributing significantly to the field. Few generative frameworks address both aspects concurrently. Notable among them is NAP [37] , which generates complete articulated assets-including meshes-using graph attention networks. CAGE [46] , enhances controllability and interpretability, albeit with limitations in mesh generation. More recently, [62] proposed a framework capable of reconstructing articulated assets from image inputs by using a 3D shape completion model to generate the different parts and a large language model (LLM) to predict the joint parameters. Progress was also made in physics-aware shape generation; for instance, [58] introduces an improved loss mechanism to account for internal collisions within generated articulated assets. In this work, we build up on NAP and CAGE and design an approach to enhance the controllability of the generative process, and leverage a multi-modal 3D generative model for improving the quality of the shape geometry. 3D Shape Generation and Completion Recent progress in diffusion models and 2D-guided generation techniques have revolutionized 3D shape generation and completion. Two main types of generative pipelines are currently being explored. The first one leverages 2D diffusion models to optimize NeRFs [73, 42, 57] , enabling high-quality 3D generation without the need for direct 3D supervision. The second employs Latent Diffusion Models (LDMs) for different 3D representations, including point clouds [59, 113, 112, 60, 88] , voxels [111] , SDFs [8, 76, 111, 48] , shape2vecset [108, 110] , and triplanes [16] . It should be emphasized that despite the predominant role of diffusion models, other methodologies such as autoregressive models [78] , have demonstrated potential in producing high-fidelity 3D shapes from minimal inputs. Conditional diffusion models enable multimodal inputs. SDFusion [8] , for instance, can generate 3D objects from text descriptions, images, or partial 3D shapes. Building upon these capabilities, we have adapted SDFusion as the shape generator within our framework. However, due to MIDGArD's modular design, the shape generator can be updated to incorporate the latest advancements in static 3D generation.3 Synthesis of 3D Articulated MechanismsQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: please find our limitations of the work in the Conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method overview: 
 Both components of MIDGArD, namely the structure and the shape generator, leverage a denoising diffusion model [79, 80, 21] . Diffusion models operate by progressively degrading training data through the systematic application of Gaussian noise, a method known as the forward process. Subsequently, these models are trained to restore the original data by methodically removing this noise in what is known as the reverse process. Please refer to Appendix A and [21] , [79] for details. As shown in Figure 1 , the full MIDGArD pipeline can be separated into the Structure Generator and the Shape Generator. The Structure Generator (subsection 3.2) takes graph-based representation of an articulated asset -which may be incomplete or affected by noise -and resolves this representation through diffusion. The denoised graph features, decoded into suitable -human interpretable -text prompts, images and bounding box information are then fed into the Shape Generator (subsection 3.3), another diffusion-based model yielding the final articulated 3D object.Articulated Asset Representation and Parametrization Building upon [37] , we encode the structural and kinematic attributes of every articulated asset into the node and edge features of a complete graph referred to as G N . Our structure generation module leverages a Graph ATtention (GAT) [90, 4, 32] denoising network to generate coherent and interpretable articulation features from noisy or incomplete inputs (see Figure 1 ). To improve the quality, controllability, and interpretability compared to [37] , we modify the asset parametrization. Drawing on insights from [46] , we incorporate a set of categorical variables to the nodes and edges features to enhance asset consistency, interpretability and provide a more intuitive control interface over the generation process. Each asset is represented as a complete graph G N = {x, e} N that embodies the object's links and joints in its N nodes, and in its N (N -1)/2 edges respectively. Rather than directly denoising and then decoding a low-dimensional shape latent f i ∈ R D F for each link i into a 3D mesh [37, 36] , we propose a two-step approach that operates on a more manageable image latent g i ∈ R D G of the link. This image latent is derived by training a Vector-Quantized Variational Auto Encoder (VQVAE) [89] on various views of every rigid body in the dataset. In our method, the structure generator denoises a latent representation of an image for each individual component of the articulated asset. This denoised image, along with the categorical information extracted from the graph, facilitates the creation of human-interpretable priors for each link, namely a text description and a front view image. These prompts can then be utilized to condition a 3D shape generation model yielding consistent meshes. Additionally, the graph itself encodes highly relevant structural features that act as supplementary conditioning signals, further guiding the shape generation model.Each node x i represents a link of the articulated asset and ∀i, j ∈ [0, N -1] the edge e ij establishes a connection between node x i and node x j . The node feature vector x i is comprised of multiple components, specifically,x i = [o i , a i , b i , d i , r i , t i , g i ] ∈ R D V, and the edge feature vector e ij is formulated ase ij = c ij , j ij , p ij , l ij ∈ R D E, where D V and D E refer to the node and edge feature dimensions. To ensure a consistent graph size across various objects, N is set to the maximum count of parts anticipated, using o i ∈ [0, 1] as a binary indicator to denote part existence. Similarly, the edges denote symbolic existence and chirality through the indicator variable c ij ∈ [0, 1] 3 . Joint types are represented through j ij ∈ [0, 1] D J , a one-hot encoding that in our design has a dimensionality of three to signify the types -prismatic, revolute, or screw. Similarly,a i ∈ [0, 1] D A and b i ∈ [0, 1] D Bdenote the one-hot encoding vectors of asset (resp. body) categories. Similar to [37] , we use Plücker coordinates as joint parametrization as this provides a compact representation for joint types, such as revolute, prismatic, and screw joints. The Plücker joint parametrization, articulated as p ij ∈ R 6 , provides a harmonious portrayal of the different joint types under consideration, and joint limits are entailed by l ij ∈ R D L . Ultimately, the vectors d i ∈ R 3 , r i ∈ R 3 and t i ∈ R 3 depict the bounding box dimensions, orientation and position of each part. Note that r i and t i are defined for every part relative to its parent reference frame.Unlike previous approaches [37, 46] , we do not assume that the parts of the dataset are provided in a canonical orientation. Assuming that objects are in canonical orientation and not explicitly considering the orientation leads to degrading performance due to misaligned orientations of the dataset (see Figure 5 , as well as Appendix B-Figure 10 and Appendix B-Figure 11 ). Instead, we estimate this orientation for each body in the dataset by computing the corresponding Oriented Bounding Box (OBB), which is then used to position the body in a canonical pose. The process (see Appendix B) begins by generating a convex hull from the mesh. This step simplifies the complex geometry into a more manageable form, reducing dependency on the internal structure of the object. Next, we voxelize the convex hull to collect volumetric samples. We then execute a Principal Component Analysis (PCA) to determine the primary axes of the mesh providing an initial approximation of the object's orientation. Following the PCA, we perform gradient descent on the attitude quaternion of the OBB, using the PCA-based initial estimate as a starting point. The objective of this optimization is to minimize the volume of the OBB. This approach allows for a more flexible and accurate alignment of dataset bodies into canonical poses, thereby enhancing the overall quality and consistency of the articulated objects generated by our framework (see Figure 5 , Table 3 ).To maintain the interpretability of the denoised quantity as a Plücker vectorp ij = [u ij , v ij ],our approach introduces a novel parametrization allowing the diffusion process to be directly executed on the Plücker manifold P. In contrast, the approach proposed in [37] , denoises within R 6 , thereby requiring a projection operation p : R 6 → P at every denoising step. We propose an alternative formulationk ij = [m ij , n ij ] ∈ R 5 of the joint parameters where m ij = [θ ij , φ ij ] ∈ R 2 represent the spherical parametrization of the Plücker axis u ij . This formulation yields u ij = [sin(θ ij ) cos(φ ij ), sin(θ ij ) sin(φ ij ), cos(θ ij )] ∈ R 3 with ∥u ij ∥ = 1. The vector n ij ∈ R 3 is then defined such that v ij = n ij × u ij .This alternative parametrization guarantees the normalization of u ij and enforces by construction the orthogonality between u ij and v ij , which are key features of the Plücker manifold [30] . Consequently, our approach inherently satisfies the manifold's constraints, thereby streamlining the computational workflow by eliminating the necessity for iterative projections.The shape generator of MIDGArD aims to create the mesh for each component of an articulated object using the denoised object graph, which contains both semantic (object category) and visual (image latents) data. We leverage a multi-modal 3D generative model that is trained independently from the structure generator. Our modular setup is designed to accommodate any generative model, allowing for seamless integration of the latest advances in 3D generation. In this work, the SDFusion model [8] is used, conditioned on image, text as well as graph features input, and enhanced with a novel approach for bounding-box-constrained generation. The training pipeline is shown in Figure 2 .The core of SDFusion is a latent diffusion model, see [8] for details. Object meshes are transformed into truncated signed distance functions (TSDFs) and encoded using a pretrained and frozen 3D VQ-VAE [89] model. The diffused latent representations, along with image and text condition signals, are denoised through a 3D U-Net model. The resulting output latent is then decoded into a TSDF and converted back into a mesh using a marching cube algorithm [56] . It is worth emphasizing that we train and apply the diffusion model on the individual parts of each objects, querying the model multiple times at inference time to generate a full articulated object (see Figure 1 ). The model is trained on the PartNet Mobility dataset [97] with the same train-test split as the structure generator. We modified SDFusion's conditioning [8] to align it with the output of the structure generator. The model is conditioned through cross-attention on the following modalities, either independently or in combination.• Single view image: A pretrained ResNet-18 model [19] encodes images. During training, the model uses renderings of the part mesh from a frontal view. At inference, the model decodes the image from the node features of the articulation graph.• Text: We employ BERT [11] for text encoding (similar to [84, 8] ). The text is constructed using the object category and asset type from the PartNet dataset in the format "A <asset type> as part of a <object category>"; for instance, "A lid as part of a trash can".• Graph: The output from the structure generator is encoded and concatenated with imageand text embeddings to provide information about the part's role within the object, such as its expected size and its relation to other parts, i.e. joint types.This multi-modal conditioning approach ensures the generation of consistent, high-quality parts that can be seamlessly integrated to form complete articulated objects, enhancing the overall performance and applicability of our framework. Enforcing Bounding Constraints Generating object parts independently offers increased flexibility and quality but introduces challenges related to the size and orientation of the generated parts. Intuitively, the shape generator must ensure that parts conform to r i and d i in the articulation graph.To achieve the correct orientation, we generate links in a canonical pose and subsequently apply a post-processing rotation based on r i , the 3D orientation vector derived during structure generation.To train the model accordingly, by estimating and adjusting the pose of parts to a canonical orientation, as detailed in Appendix B.To ensure that generated parts match the sizes specified within the graph node features, we introduce a bounding-box constraint to the generative model (see Figure 2 ). Instead of generating the object directly, the diffusion model is trained on the difference relative to a predefined bounding box. To enable this schema for latent diffusion, we calculate the difference as 4 Experiments, Results and Discussionz ∆ = z o -z b ,Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Data of the results is provided at https://quentin-leboutet.github. io/MIDGArD/. Opensource code will be provided upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment setup: 
 Dataset All experiments were conducted using the PartNet Mobility dataset [97] , which contains a diverse set of articulated 3D objects with detailed geometric and kinematic annotations. Each mesh in the dataset underwent a two-step preprocessing routine. First, we enforced the manifold property as described in [26] . Second, we performed orientation estimation to establish a canonical pose for each object and accurately compute the corresponding bounding boxes (see Appendix B for details).Evaluation Metrics We adopt the evaluation framework from NAP [37] , which introduces the Instantiation Distance (ID) metric for comparing two articulated objects. ID measures the average Chamfer distance over sampled joint states, accounting for both geometric and kinematic differences.It is defined as:EQUATION)where Q is a set of M uniformly sampled joint poses and d is the minimum Chamfer distance over all possible canonicalizations of the mesh. We use ID in conjunction with standard metrics for unconditional generation, namely minimum matching distance (MMD), coverage (COV) and 1-nearest-neighbor accuracy (1-NNA).Implementation Details We trained the structure generator and the image VQ-VAE on an NVIDIA RTX 3090 GPU, while the shape generator was trained on an NVIDIA RTX 6000 GPU. Evaluation took place on a single NVIDIA RTX 3090 GPU. The image VQ-VAE, which encodes latent representations of objects in the shape generator, was trained on 256 × 256-pixel front renderings of the the PartNet Mobility object meshes. The denoising model used in the structure generator contains six graph attention blocks, with a latent embedding size of 512 and 32 attention heads. We set the maximum number of nodes in the graph to N = 8. Our training parameters closely follow those in NAP, with the key difference being the use of an implicit denoising diffusion pipeline [80] over 100 time steps, as opposed to a DDPM with 1,000 time steps. This modification significantly accelerates inference speed. Our shape generator is adapted from SDFusion [8] and trained on the PartNet Mobility dataset. We used the same hyperparameters as the multimodal model in SDFusion and utilized their pre-trained VQ-VAE checkpoint. We excluded 10 categories from training due to their objects containing numerous equally-shaped parts (e.g., keyboards with over 30 keys).Unconditional Articulated Asset Generation To the best of our knowledge, NAP is the only approach for generating articulated 3D objects without prior knowledge about their geometric structure. We first evaluate MIDGArD in an unconditional generation setting, comparing its performance to NAP. For a fair comparison, we retrained NAP on our preprocessed version of the PartNet Mobility dataset. The results are presented in Table 1 . MIDGArD outperforms NAP in terms of MMD and COV metrics, indicating better diversity and coverage of the generated samples. Specifically, we observe a 6.4% improvement in MMD and a 3.7% improvement in COV. The 1-NNA metric is comparable between both methods. We also perform an ablation study on the effect of the Plücker manifold parameterization. As shown in Table 1 using the Plücker manifold improves the MMD and COV metrics, suggesting enhanced consistency and diversity in the generated assets. We sampled 400 objects generated by each method and counted each joint type only once per object to minimize the impact of objects with multiple joints of the same type. The results in Table 2 show that NAP predominantly produces screw joints, which are rare in the training data. In contrast, MIDGArD generates objects with a joint type distribution closely matching that of the training data, thereby enhancing the plausibility of the generated assets. We computed the Chi-Square statistic to quantify the deviation from the expected joint type distribution. NAP's generated data yielded a high χ 2 value of 5618, indicating a significant difference from the training data distribution. In contrast, MIDGArD's generated data resulted in a low χ 2 value of 12.7, demonstrating close alignment with the expected distribution. Additionally, as shown in Figure 3 .C, the absence of categorical information in NAP leads to the generation of objects composed of mismatched parts that do not cohesively fit together. In our approach, while there may be occasional issues with geometry or kinematics, the object parts remain semantically consistent, preserving the integrity of the overall structure. Finally, as illustrated in Figure 3 .D, our conditional part-generation proves to be robust to images from different viewpoints, yielding enhanced flexibility and generalization capability of our method. Conditional Generation and Controllability One of the key advantages of MIDGArD over existing approaches is its enhanced controllability, enabling users to guide the generation process using human-understandable graph attributes such as articulated asset types and body types. We showcase this capability in a "(Image+Text)-To-Object" setup (see Figure 3 .A1), and a "Part-To-Motion" setup (see Figure 3 .A2), where the model is provided with part features only (i.e., no joint data) and outputs consistent articulated assets. Notably, MIDGArD can also be controlled using only assetlevel data; for instance, users can query for specific categories such as "storage furniture" or "fan" without needing to provide detailed information for each node (see Appendix C for additional qualitative results). Furthermore, the modular structure of our generation pipeline enables finegrained, part-level control. Users can specify the desired appearance or attributes of individual parts by adjusting the human-interpretable articulation solutions synthesized by the structure generator accordingly. The resulting images and text description will then be used as conditioning signals for the shape generation module. As illustrated in Figure 4 , varying the image inputs while keeping other attributes fixed results in generated parts that adapt accordingly, demonstrating the flexibility and responsiveness of our approach. This level of control allows for precise customization of generated assets, facilitating applications that require specific design features or aesthetic qualities (see Appendix D for additional qualitative results).Figure 4 : Image guidance of the shape generation process EQUATIONEQUATIONSecondly, we evaluate the generation quality after transforming the TSDF into a mesh, computing the Chamfer distance (CD) between generated and real objects based on 5000 sampled points. Table 3 compares our bb prior approach to the original SDFusion model trained on PartNet Mobility. For encoding the part dimensions, we use an MLP with three hidden layers (size 16, 64 and 256 respectively). For a comparison to a GAT-encoding, see Appendix E. Our first modification of preprocessing the dataset to rectify objects into canonical pose, dubbed "original + bb MLP + oriented dataset", already improves the generation results significantly, leading to lower OBB count , OBB sum and CD. The bb prior method, however, consistently ensures accurate proportions and sizes for the generated links, diminishing erroneous volume. Additionally, bb prior + bb MLP method secures the best Chamfer Distance at 0.072. An alternative technique involves resizing generated objects to suit the bounding box dimensions. Such postprocessing of the model outputs reduces the average CD to 0.0061, close to matching that of the bb prior method. Nonetheless, such resizing may lead to significant distortions in object proportions, thereby underscoring the superiority of the bb prior method.Analysis of Oriented Bounding Boxes As Table 3 shows, using oriented bounding boxes drastically improves the reconstruction performance. To quantify the change to the dataset, we measured the volume reduction of the bounding boxes after applying our PCA-based rectification approach. On average, the bounding box volume decreased by 17.4%, with one-third of the samples experiencing a volume decrease of more than 10% and 20% of the samples experiencing a volume decrease of more than 30%. Figure 5 illustrates failure cases where using axis-aligned bounding boxes leads to unrealistic part shapes, such as a thick cabinet door. 
simulatable assets:: 
 Offering a pipeline to create fully simulatable assets compatible with physics engines such as MuJoCo.Code and models are available at https://quentin-leboutet.github.io/MIDGArD.In this work, we introduced MIDGArD, a novel framework for generating 3D articulated objects. MIDGArD employs a modular approach, combining interpretable articulation graph generation with high-quality shape generation. By leveraging categorical parametrization, MIDGArD enhances the consistency and plausibility of articulated objects, producing high-quality meshes with accurate dimensions through a constrained shape generation mechanism. The human-interpretable representation of images and text within the articulation graph allows for part-level and multi-modal control over the generation process. The models produced by MIDGArD are fully simulatable, paving the way for applications in text-guided or image-guided content creation.MIDGArD addresses several issues identified in Neural 3D Articulation Prior (NAP) by improving joint conditioning with categorical embeddings, enforcing physical constraints via bounding-box constrained generation, and providing part-level control using latent image codes. Our results demonstrate MIDGArD's superior performance in terms of structural quality, consistency, and interpretability compared to existing methods. The framework's ability to generate diverse articulated objects based on partial graph inputs and shape conditioning underscores its potential for broad applications in digital content creation and robotics.Despite the advancements introduced by MIDGArD, there are several limitations and areas for future improvement: 1) We observe that the current metrics for articulated generation are still very limited and benchmarks as well as evaluation frameworks must be developed for this field. For instance, Instantiation Distance is still based on point clouds, which ignores cases of small parts of the object with unrealstic motion, such as a cart wheel moving up to a meter away from the cart. 2) Overcoming limitations of graph scalability posed by node number constraints remains an important challenge.3) Furthermore, it would be interesting to investigate the use of mixed integer noise methods such as MiDi [91] in the articulation graph generation pipeline to process the continuous and discrete variables. 4) Enhanced conditioning of the structure generation process using natural language or image data represents a highly promising avenue for future research. Foundational efforts in this area, such as those by Cai et al. [5] , have laid the groundwork for further exploration. 5) Integrating specialized templates, such as human body poses, could significantly enhance MIDGArD's modelling versatility. Lastly, 6) testing alternative 3D generation models and adapting MIDGArD to handle different geometric primitives could broaden its generalization capabilities.Figure 8 : Detail of the structure generation inference pipeline.A relevant property of such a diffusion process is that x t can be sampled from x 0 throughEQUATIONwhere α t := 1 -β t and ᾱt = t s=1 α s , eventually yieldingx t = √ ᾱt x 0 + √ 1 -ᾱt ε.A.2.2 Reverse Process.Initiating from a standard Gaussian distribution, x T ∼ N (0, I), a denoising model p θ parameterized by trainable weights θ, learns to approximate a series of Gaussian transitions p θ (x t-1 |x t ). These transitions incrementally denoise the signal such thatEQUATIONEQUATIONFollowing the approach from [21] , previous work on articulated asset generation [37] [46] defineµ θ = 1 √ αt x t -βt √ 1-ᾱt ε θ (x t , t) and Σ θ (x t , t) = σ 2 t I yielding the following Langevin dynamicsEQUATIONwhere ε θ (x t , t) is a learnable network approximating the per-step noise on x t .The variational lower bound is used to optimize the negative log-likelihood. Following the simplified training objective outlined in DDPM [21] , the training loss can be simplified to the following expressionEQUATION)B Data PreprocessingHigh-quality meshes are essential for applications such as generating Truncated Signed Distance Functions (TSDFs), which are in turn widely used within advanced 3D latent diffusion models such as SDFusion [8] . However, many meshes in popular datasets, including PartNet-Mobility [97] , contain inconsistencies and errors that render them unsuitable for such purposes. To address these issues, we incorporate a preprocessing routine that includes a manifoldization process [26] to repair and convert flawed meshes into watertight, manifold counterparts (see Figure 9 ). Unlike previous approaches [37, 46] , we do not assume that the dataset meshes are already provided in a canonical orientation. Our method hence involves an orientation estimation process for each individual mesh, performed through the calculation of its Oriented Bounding Box (OBB). This OBB is subsequently utilized to align each body into a standard pose. The orientation estimation begins by applying Principal Component Analysis (PCA) to the mesh data. To mitigate the influence of complex internal geometries, such as shelves within furniture models, we first compute the convex hull of the mesh. This step simplifies the geometry while preserving the overall shape. We then voxelize the convex hull and uniformly sample 10,000 volumetric points from it. Applying PCA to these sampled points provides an initial estimate of the mesh's principal axes and orientation. Building upon this initial estimate, we refine the attitude quaternion of the OBB through gradient descent optimization.The objective is to minimize the volume of the OBB, achieving a tighter fit around the mesh. During optimization, we iteratively adjust the orientation by recalculating the OBB volume along the oriented principal axes. To reliably navigate the solution space and avoid local minima, we incorporate a heuristic that explores potential orientations within a cone defined by an angle θ centered around the current best estimate. This approach allows for nuanced adjustments and aids in converging to a global minimum, thereby enhancing the precision of the orientation estimation. By combining PCA for initial alignment with heuristic-augmented gradient descent for volume minimization, our method ensures accurate and robust alignment of dataset meshes into their canonical poses (see Figure 10 ). This preprocessing step significantly enhances the overall quality and consistency of the articulated objects generated by our framework. MIDGArD incorporates categorical embeddings for asset types and object categories directly into the node features of the articulation graph. These embeddings enable the model to understand and generate objects that belong to a specified category, capturing the common structural and kinematic features associated with that category. The model is trained on a diverse dataset that includes various object categories with different articulation patterns. By learning statistical priors over these categories, MIDGArD can infer plausible structures and motions for new objects within the same category. The structure generator is designed to handle incomplete or partial inputs, allowing it to fill in missing details based on learned patterns. In the absence of detailed node-level information, the model can perform unconditional generation, producing coherent articulation graphs solely based on the provided asset category. The use of categorical embeddings and latent codes that correspond to human-understandable concepts (like "storage furniture") makes it easier for users to interact with and guide the generation process. A key advantage of MIDGArD's modular setup is the controllability of the generative process, not only on a graph level (by supplying partial graphs) but also on a part-level, by replacing the generated image or text information. In Figure 13 , we provide examples how modifying the input image to the shape generator (SDFusion) changes the object appearance. Here, we randomly choose images from PartNet Mobility, sampling among all parts with the same category. For example, we sample from from all meshes with the descriptions "A wheel as part of a cart" to generate the wheel in the fifth row of Figure 13 . For capturing the relationships between parts, we experiment with graph encodings as input to the conditional shape generation process. To generate the geometry for a specific part ϕ, we construct a simplified graph from the articulation graph that provides information about the size of the ϕ in comparison to other parts, as well as the joints between ϕ and other parts. Specifically, let G ϕ = (v ϕ , e ϕ ) be a graph with t i as the node feature vector for the i-th node and the Plücker coordinates p ij serving as edge features. We test two options to mark ϕ as the part to be generated:(1) a tree structure where v phi is constructed by reducing the graph to the parts adjacent to ϕ, and (2) a complete graph (v ϕ = v) where the node features are augmented with a binary indicator that equals 1 for the node representing ϕ. These structures are processed with a Graph Attention Network (GAT), which is trained concurrently with the diffusion model. As shown in Table 4 , the results do not match the ones of a simple MLP embedding, neither with nor without our bb prior method. We hypothesize that this is due to concurrently training the GAT with the diffusion model, possibly converging slower than an MLP. Furthermore, many objects in PartNet Mobility have few parts, leading to small graphs of less than 4 nodes. Future work could try to pretrain a GAT on graph-encoding to improve the conditional generation with object-level information. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: we rigorously outlined the dataset, training method, implementation detail in the Result section and will enable reproducibility of results through open-source code upon acceptance.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA]Justification: no experiments that produce error bars or similar.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: we produced all codes and are original owner of assets. For third-party code, we explicitely cite and mark the dependency to those and credit the authors accordingly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We provided detailed explanation in the paper, will have a documentation in the open-source code, and provide a detailed Supplementary Materials pdf with additional info.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
: 
 } } } } } } } 
 
 Appendix:  


Decision: This paper received divergent initial opinions with three reviewers in support of acceptance and one negative reviewer.  The main concerns expressed by the reviewers were: unclear comparisons or missing comparisons against the most relevant prior work (NAP and CAGE), questions about the fairness of the evaluation methodology, a somewhat contrived "single part in an image" conditioning setup, seemingly marginal quantitative improvements over the prior work baselines, and a lack of discussion about how the contributions of the paper relate to those of the most relevant prior work (NAP and CAGE).

The rebuttal responded to reviewer concerns with detailed clarifications and additional experiments.  The positive reviewers remained positive. The one initially negative reviewer raised their strong reject to a weak reject but remained unconvinced with the rebuttal clarifications on whether the comparisons and evaluation against prior work are complete and fair enough to substantiate the claimed contributions.  Due to the severity of these concerns, there was extensive discussion in the rebuttal and post-rebuttal period.  Taking into account the willingness of the authors to clarify and improve the paper, the AC and reviewers support acceptance, provided the following revisions are incorporated into the manuscript:
1. More precisely articulate the contributions (e.g., how image/text/graph control is specified by the users, and how the Plücker manifold enhances the representation).
1. Acknowledge aspects of the approach inspired by prior work and how their benefits align with those demonstrated in the prior work (e.g., conditioning on object categories to achieve semantically consistent parts and adding joint type as a categorical label to improve kinematic plausibility, as shown in CAGE).
1. Qualitatively demonstrate the diversity of outputs when using the same image/text input.
1. Be transparent about how the comparison with NAP is conducted and clarify why CAGE was not included as a comparison point.

Thus, the AC recommends acceptance and strongly encourages the authors to incorporate clarifications and revisions from the review cycle and especially targeting the above points, into the final manuscript.

Reviews: review: paper_summary: paper_summary: This work addresses the task of 3D asset generation for articulated objects.

This work aims to enhance the prior approach by achieving three main objectives: 1) increasing the interpretability and controllability of the generation process; 2) generating more natural joint motions and 3) reducing violations of physical constraints.

For the first goal, they propose a framework MIDGArD to decompose the generation into two sequential modules: 1) a structure generator that unconditionally generates a kinematic graph and attributes for each articulated part; 2) a shape generator to produce part SDF that allows conditioning on the graph from the first module, an image, and a text description for each part.

For the second goal, they introduce a representation for joint parameters that allows the diffusion process to operate on the Plucker manifold directly.

For the third goal, they replace the AABBs with OBBs to bound the part shape which is claimed to achieve better part alignment.
strength_weakness: strength_weakness: - The technical contribution is limited: 
	- The network architecture of the structure generator is the same as NAP [35]. The contribution for this part is only the representation alternation for joint parameters and bounding boxes. These are reasonable changes for slightly better distribution modeling, but the insight is not particularly significant.
	- The shape generator is also adapted from an existing work SDFusion [7], where the benefits of detailed geometry modeling with TSDF and the flexibility of multimodal conditioning are just inherited from the original work. The only modification is an additional conditioning from a graph. However, how this graph condition affects the part geometry output and whether it is necessary is unknown. Also, there is no strong/sufficient qualitative evidence to show a better geometry generation compared to NAP.
- The experiments and evaluation are insufficient and unclear:
	- The side-by-side qualitative comparison with prior work (e.g. NAP [35], CAGE [43]) is missing to convince the improvement in the claimed aspects.
	- Unfair comparison with NAP in the “reconstruction” setting in Table 1. According to the description in lines 280-282, NAP is only provided with graph topology and motion attributes for each node. In this case, there is no way for NAP to reconstruct the object with no shape information (which should have been given the latent code extracted from the ground truth part geometry). In contrast, MIDGArD is additionally provided with shape features extracted from the ground truth node images.
	- Lack of quantitative and qualitative results to support the argument of more “natural joint motion” and “less physical constraint violation” and how these improvements are correlated with the specific designs. Other ablation studies discussed in the manuscript are also generally hard to follow.
- The related work section does not contextualize this work very well.
questions: questions: - The argument in lines 47-48 “leveraging categorical embeddings for improving consistency across the graph” is unclear. What does consistency mean exactly? The ablation experiment to support this argument is missing.
- Would it be possible to compare with CAGE [43] on certain aspects of the generation? E.g., in terms of the physical plausibility of objects in both abstraction level (bounding boxes) and mesh output. 
- The argument in line 33 “both NAP [35] and CAGE [43] provide no control on the part level due to their opaque encoding as a latent” is a bit misleading. What specific control on part is missing from the prior work? Also, to correct the fact, the representation used in CAGE has no latent encoding for any attribute, while the representation used in NAP only makes the geometric feature encoded as latent and leaves others in an explicit form.
- How robust and generalizable the image control is?
limitations: limitations: - Under the formulation of this work, the image control from users is not particularly practical:
	- The image of a single articulated part is not natural to find easily.
	- The control is more of a post-editing. The user has to wait to see what object is generated and then find compatible part images to feed into the corresponding node in the graph. Otherwise, it won’t be matched with the articulation parameters and the part arrangement being generated.
paper_summary: paper_summary: This paper proposes several interesting improvements over existing articulated object modeling and highlights higher-quality part generation.

Specifically, an articulated object is parameterized to a graph where parts are nodes and joints are edges. This paper proposes a multi-model part VAE for generating higher quality and controllable shapes and improving physical awareness through a bounding box alignment. It also improves the kinematic structure generation by considering parameterize on the plucker manifold. With these technical contributions, the results show significant improvement over baselines and the supplemental repo provides simulatable mujoco files, showing the practical value of this work.

I recommend a clear acceptance of this paper.
strength_weakness: strength_weakness: - More conditioned generation results: while the part-level multimodel information is used in shape encoding, it would be better to show more conditioned generation results using this information.
- Open source the part text and bounding box alignment data: it's not clear whether the shape part encoding training data will be publicly released or not.
questions: questions: Please see the weakness
limitations: limitations: Adequately discussed in the paper.
paper_summary: paper_summary: This work tackles the problem of generating articulated 3D assets that are animatible. The authors mention that their generated shapes are directly compatible with existing physics simulation tools, i.e. MuJoCo. To this end, they first propose a structure generator, which conditionally or unconditionally generates an articulation graph encoding structural object information such as the kinematics attributes, i.e. the kinematic tree and joint types. Next, the multi-modal shape generator synthesizes a shape that follows the kinematic tree and size of the object while also allowing conditioning on multi-modal inputs such as text and images. Results demonstrate some improvement over the SOTA (NAP).

Overall, I like the setting that the authors propose as well as their technical contribution to solve it. There are a few minor concerns about the presentation in terms of writing, clarity, and qualitative visualizations (see comments above). However, I believe those are minor. Thus, I recommend acceptance.
strength_weakness: strength_weakness: - Writing:
    - I would limit the contribution bullets to the technical contribution of this work rather than focusing on results and open sourcing.
- Clarity
    - I would introduce proper notations for all steps discussed in section 3.3 as I feel it is hard to understand the setting just from the textual descriptions, e.g. what is input, what is output.
    - I have a similar concern regarding section 3.4
- Video
    - I would have expected to also see some video results of the generated and animated objects. While this is not strictly required, it would have made the exposition much more complete.
questions: questions: --
limitations: limitations: Limitations are sufficiently discussed in the main paper.
paper_summary: paper_summary: This paper presents a diffusion-based framework for generating articulated objects represented as part graphs. Different from prior work [35] that generates both the part shapes and graph structures simultaneously, this work introduces a two-stage strategy that first generates the structure with per-part conditions and then generates the part shapes. It also proposes a part latent encoding that enables multimodal conditions including text and images.
strength_weakness: strength_weakness: - As also briefly discussed in the paper, generating the parts independently is a bit unnatural to this problem, especially given the importance of inter-part motions/relations in articulated objects. But overall I am OK with this point given all the node conditions in the framework.
- Quantitatively, the improvements compared to [35] on unconditioned generation seem marginal.
- The description "simulation-ready" sounds overclaimed to me. When talking about "simulation", I feel most people will expect physical viability and accuracy. But from the demonstration on the website, it seems that the parts are not well connected. There is also no explicit physics enforcement in the framework.

Minor things:
- I believe it would be better to swap the two sentences "This image latent is derived by" (line 148) and "...the structure generator denoises a latent representation of an image..." (line 150), so the readers know where the "image latent" in line 148 comes from.
questions: questions: I am curious about some details:
- Sec. 3.4 says "During training, the model uses renderings of the part mesh from a frontal view" (line 217). How is the frontal view chosen? For inference-time image-conditioned generation, do the images have to be in frontal view, or can they be of any view? Are the images in Suppl. Fig. 5 of "frontal view"?
- I wonder how exactly are the bounding boxes represented and predicted. Because for articulated objects, the part poses are changing during part motions. [35] only uses the rest-state bounding boxes (together with the joint limits). In this work, is the rest-state bounding box viewed as having the identity pose matrix?
limitations: limitations: Limitations are well-discussed in the paper.

Comments: title: Keep my original positive score
comment: Thank you for acknowledging the additional effort and enhancements made. We would like to assure you that we will incorporate the experiments and improvements into our manuscript, including more qualitative results for image control as suggested. 

We would appreciate if you could also transfer the updated score to your original response (if possible).
Thank you very much for your kind words and positive rating. 
We are glad to hear that your concerns about the evaluation and baseline comparisons have been well addressed. 
We will integrate this feedback to improve the clarity and comprehensiveness of our work.
Thanks for the detailed responses. My concern about the evaluation and baseline comparisons is very well addressed. I will keep my positive rating. I would also encourage the authors to add these explanations (especially about the "frontal view") to the paper in their revisions.
Thank the authors for the further discussion and clarification. 

Your response did address some of my concerns, so I can raise my score to weak reject for all the additional effort made during the rebuttal. I believe this paper can be much improved to demonstrate its contribution more clearly and precisely by refining the manuscript and adding more experiment results.

However, I cannot base my final decision on the future revision of the manuscript, so I would still suggest the next round of submission.

Just a clarification of my suggestions:
- Regarding "qualitative results to demonstrate how different image inputs affect the final output in variations", I was suggesting showing multiple outputs by giving the same set of image inputs (with different input examples of course). From a user perspective of a generative model, it would be interesting to see diverse outputs in terms of bounding box layout and kinematics under a part shape constraint.
- Regarding the comparison with CAGE, I think both the retrieval and generation mode of MIDGArD in comparison with prior work can help to show the advantage in fine-grained control in several aspects (e.g., semantic consistency given object category, geometry/motion compatibility given different levels of part description, etc.) and also reveal the limitation of prior work. I would expect the generation mode can produce more faithful geometries compared with retrieval-based methods, which demonstrate the necessity of the shape generator module proposed in this work.
Thank you for your encouraging feedback.
Thank you for your detailed response and positive evaluation; your support for our work's suitability for NeurIPS is greatly appreciated.
We appreciate your detailed feedback and your recognition of our key contributions, which you have deemed "critical and well-supported." While we regret any oversight in our previous analysis, we would like to clarify that our review does indeed address most concerns you raised. It is possible that these points may not have been fully apparent, and we welcome this opportunity to elaborate further:

- **Controllability - Image control:** The qualitative impact of varying image inputs while keeping other attributes fixed was precisely illustrated in Fig. 3 of the main paper as well as Fig. 5 of the supplementary material, and Fig. D of the rebuttal. These examples demonstrate the modularity and flexibility of our approach in controlling part geometries.
We respectfully disagree with the impracticality claim of part image condition, considering that (1) images are easier to handle than 3D representations, (2) our experiments show the shape generator works even without input image condition (the results will be integrated into the revised version of the paper) and (3) modern segmentation methods can streamline the process (but falls outside the scope of this paper).

- **Controllability - Text control:** In addition to the text+image conditioning examples provided in our supplementary material, we have conducted further experiments where the shape generator is conditioned solely on textual descriptions. Our findings indicate that removing image conditioning does not degrade the quality of the generated objects. Variations in text label (e.g., changing ''furniture'' to ''oven'') leads to corresponding changes in geometry.

- **Controllability - Graph control:** We regret any confusion regarding graph conditioning. The term graph conditioning refers in the original manuscript to the possibility of using any feature of the graph object produced by the structure generator, as an additional condition mechanism to the shape generation process. During our experiments, we only used the node bounding box features, as stated in line 332-333 of the article. Combined with our bounding box prior, this design choice is a core innovation that significantly improves the physical plausibility of generated objects, as evidenced by the results in Table 2.

- **Representation:** Regarding denoising on the Plücker manifold: we only claim that it "enhances interpretability" (line 201 in the manuscript) and "eliminating the necessity for iterative projections" (line 212). These claims are supported *by definition*, since our method indeed eliminates the need for post-processing and makes the output directly interpretable as Plücker coordinates (see methods section). These benefits are inherent to the design and are thus supported by the methodology itself, requiring no additional empirical validation. As part of our ablation study, we evaluated the ID metrics—MMD, COV, and 1-NNA—comparing unconditional generation scenarios with and without Plücker manifold parameterization. The results are presented in the table below:

|  Metric \ Method | Ours no-manifold  | Ours + manifold  | NAP |
|-----|-----|-----|-----|
| 1-NNA | 0.6221 | **0.5831** | **0.5831** |
| MMD | 0.0270 | **0.0264** | 0.0282 |
| COV | 0.4779 | **0.4857** | 0.4675 |

These results suggest improvements in terms of coverage and MMD, supporting our claim that diffusion over Plucker manifold improves asset consistency. 

- **About the claim of ''more natural joint motion'' and ''less physical constraint violation'':** This was covered by experiment 3) and 4) in our general Author Rebuttal above, which provides *quantitative* evidence for these points. 

- **Comparison with CAGE:** We agree that a comparison with CAGE on a limited set of categories and with geometry retrieval could be feasible; however, our modular approach is specifically designed to enable fine-grained control and flexibility in part generation, which contrasts with CAGE’s retrieval-based method. Implementing a retrieval-based mode within our framework would necessitate significant modifications that would undermine one of our method’s key strengths—its ability to generate parts from scratch using multimodal inputs. We believe that such a comparison, would not fairly represent the advantages of our method in terms of generalizability and user control. For the sake of transparency, we will explain in the revision why we did not compare with CAGE directly.

While we are currently unable to include additional figures due to rebuttal constraints, we will incorporate the results into the revised manuscript. We hope this response addresses your concerns and clarifies the contributions of our work. We are committed to further improving our manuscript based on your valuable feedback and are confident that the additional evidence we plan to include will more clearly demonstrate the effectiveness and potential impact of our approach.

Thank you again for your constructive feedback.
I appreciate the effort and detailed response provided in the rebuttal. 

At a high level, I agree with the authors that modularity is an effective strategy for enabling flexible control over part geometry. Although the current approach—providing images of each part rendered separately—feels somewhat unnatural, this work is trying to address an important problem and has the potential to inspire further exploration in this increasingly significant field. I appreciate the insights and the effort that went into this paper.

However, I still have concerns that the experiments in the current state cannot well support the central claims and effectiveness of the proposed components.
- **About controllability**: I have concerns about the practicality and effectiveness of the proposed approach based on the limited results presented.
    - For image control: It would be more convincing to show qualitative results to demonstrate how different image inputs affect the final output in variations while keeping other attributes fixed. Plus, the requirement of per-part rendering may limit the method's generalizability to real-world scenarios. While the authors suggest that these images could be extracted from a single picture using image segmentation in the rebuttal, there is no supporting evidence for this claim, making the viability of this approach uncertain.                                         
    - For text control: The results provided do not sufficiently demonstrate cases where each part's geometry is edited using purely text-based inputs. The supplementary material only includes two examples of combined image and text control, with four additional examples in the rebuttal PDF. It remains unclear how the text input influences the process, and how diverse and compatible outputs can be generated with varying descriptions.
    - For graph control: there is no evidence presented on how graph-based control can be implemented or how effective and useful it might be.
- **About representation** used in the structure generator: this work introduces several modifications to the representation used in NAP. However, I consider only two of them (using OBB and image latent) to be critical and well-supported contributions. Denoising joint parameters on the Plucker manifold is an interesting design, but the work lacks ablation studies or evidence to demonstrate its specific advantages.
- **About the claim of “more natural joint motion” and “less physical constraint violation”**: Only three cases are presented for qualitative comparison with NAP—one in Fig. 3 of the main paper and the laptop and drawer examples in the rebuttal. This limited number of examples is insufficient to demonstrate improvement across a broader distribution, as it is unclear whether these cases were selectively chosen.
- **Comparison with CAGE**: I disagree with the authors' assertion that this comparison is impossible. I believe it is a crucial comparison that should be conducted, at least qualitatively. 
    - I understand that CAGE makes certain simplifications in its assumptions, which may work well for specific categories of objects. However, these categories represent only a subset of the objects considered in MIDGArD. It should be feasible to demonstrate certain aspects of the comparison for these categories.
    - On the geometry side, both NAP and CAGE can operate in a retrieval-based setting. My understanding is that it should be an alternative mode for MIDGArD, using an image as a retrieval proxy. In this context, it is important to compare both the retrieval and generation modes of MIDGArD with NAP and CAGE. The difference between generation and retrieval modes does not justify excluding such a comparison.

Overall, I believe this is a valuable work in terms of its motivation and potential impact. Once the experiments are fully completed to substantiate all the contributions the authors claim, I would be inclined to support the acceptance of this paper. However, in its current form, I would suggest resubmission in a future round.
Thank you for your detailed response. 

After reading the rebuttal and other reviews, I am still convinced that this work is suitable for Neurips. 

Therefore, I keep my original rating.
After reading the reviews and author response, the reviewer feel very good that the area of articulated object modeling is blooming. I believe this paper provides practically much more improvement over NAP. I keep my original score.