Title: URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: articulated objects

3D multimodal language models
Abstract: Constructing accurate digital twins of articulated objects is essential for robotic simulation training and embodied AI world model building, yet historically requires painstaking manual modeling or multi-stage pipelines. In this work, we propose \textbf{URDF-Anything}, an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM). URDF-Anything utilizes an autoregressive prediction framework based on point-cloud and text multimodal input to jointly optimize geometric segmentation and kinematic parameter prediction. It implements a specialized [SEG] token mechanism that interacts directly with point cloud features, enabling fine-grained part-level segmentation while maintaining consistency with the kinematic parameter predictions.
Experiments on both simulated and real-world datasets demonstrate that our method significantly outperforms existing approaches regarding geometric segmentation (mIoU 17\% improvement), kinematic parameter prediction (average error reduction of 29\%), and physical executability (surpassing baselines by 50\%). Notably, our method exhibits excellent generalization ability, performing well even on objects outside the training set. This work provides an efficient solution for constructing digital twins for robotic simulation, significantly enhancing the sim-to-real transfer capability.


Peer reviews:
comments: SUMMARY:
The paper presents URDF-Anything, an end-to-end pipeline that turns RGB observations (single- or multi-view) into functional URDF parameters of articulated objects. A 3D multimodal LLM (ShapeLLM-7B + LoRA fine-tuning) jointly (i) segments a dense point-cloud into links and (ii) predicts joint parameters. The key technical novelty is a dynamic [SEG] token that the model inserts while autoregressively generating the JSON-style URDF: each [SEG] token’s hidden state is used, via cross-attention, as a query to produce a per-point mask for the corresponding link.

STRENGTHS AND WEAKNESSES:
**Strengths**
- The paper is well-written.
- To my knowledge, the first work to leverage a 3D MLLM with explicit token-based cross-modal segmentation for simultaneous geometry + kinematics prediction; elegant integration of language generation and point-level supervision.
- Comprehensive pipeline—from point-cloud creation (DUSt3R / LGM) through [SEG] cross-attention to mesh fusion—explained clearly.
- Strong quantitative gains on both ID and OOD splits; executability metric moves beyond static accuracy; ablations on modality and [SEG] tokens are informative.
- Produces ready-to-simulate URDFs, a strong link to robotics/embodied AI workflows.

**Weaknesses**
- Results lack error bars or significance tests; variance is only given for one table.
- The pipeline still relies on an external point-to-mesh post-process; errors there are not evaluated.
- Training on ShapeLLM-7B with LoRA is resource-heavy; training time is not reported.
- Mass, inertia, friction and joint damping are omitted; executability counts loading & actuation but not quantitative dynamics accuracy.

QUESTIONS:
See above

LIMITATIONS:
It would be better to show some failure cases.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces URDF-Anything, a framework for reconstructing articulated 3D objects as URDF models directly from visual input, using a 3D Multimodal Large Language Model (MLLM). The authors first convert single-view or multi-view RGBs to point-cloud via off-the-shelf DUStr and LGM. Then, they train a 3D multimodal Large Language Model (3D MLLM) to take the point-cloud and some text prompt to perform part segmentation, link hierarchy and joint prediction.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The idea of using a 3D MLLM for articulation is novel and interesting.\n- Ablation study 4.4 provides valuable insight, highlighting the benefits of explicit 3D input over 2D image-based alternatives.\n- The experimental results show gains in segmentation, joint prediction accuracy, and simulation executability.\n\n**Weaknesses**\n- The related work section is imprecise and at times inaccurate. \n\t- \"Research on [...] is a related but distinct field.\" is not accurate and confusing. Many works under this section are directly relevant to this work. In fact, the authors themselves compare against these methods as baselines.\n\t- \"[29] focuses on generating articulated simulation scenes from images, which is related but distinct from accurate individual object\" is not accurate. URDFormer can also generate individual objects. In fact, the authors did compare against this baseline.\n- The point cloud generation pipeline is unnecessarily complicated. The authors use DUSt3R for multi-view input and LGM for single-view, but don’t explain why a unified approach (e.g., always using LGM or DUSt3R after synthesized views) isn’t employed. This suggests convenience-driven decisions rather than principled design.\n\t- For example, they could hallucinate single-view to multi-view via off-the-shelf model (ImageDream) then use DUSt3R. Alternatively, they can also scrape off DUSt3R and just use multi-view posed RGBs directly with LGM.\n- Some important aspects of the system’s training and inference remain ambiguous:\n\t- The \"Uni3D w/ text\" baseline is underspecified. What exact text input is used? The generic prompt \"Segment the input point cloud into parts and predict articulation parameters.\" shown in Fig. 2 seems insufficient for meaningful conditioning.\n\t- It is unclear how \"Uni3D w/o text\" and \"Uni3D w/ text\" perform segmentation. Do they involve a classifier head trained on predefined categories? If I understand correctly, in the Uni3D paper, they use contrastive-style loss to do segmentation. So during inference, they likely provided a list of semantic part names that need to be segmented.\n\t- A more intuitive and technical explanation is needed to justify why the [SEG] mechanism outperforms \"Uni3D w/ text\", especially since both involve feature-text fusion at different levels.\nEvaluation choices raise some concerns:\n\t- Line 208 defines a custom “joint success” criterion but omits reporting success rate as defined in prior work [1]. Why?\n\t- Table 3 (physical executability) omits Real2Code, even though it's included in other comparisons.\n\nQUESTIONS:\n- How is object category (e.g., \"faucet\") and number of parts determined at inference time? Can your model infer this from the point cloud, or does it rely on annotations (as in PartNet-Mobility)?\n- Please provide a breakdown of performance by object category (e.g., cabinet, laptop, faucet) and a failure case analysis. This would help assess robustness and generalization.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes URDF-Anything, a method that, given a single or multiple views of an object, is able of segmenting its parts and predicting the joint parameters of the parts that can be moved (e.g., the hinge of a door). The method is tested on an extensive benchmark and reports metrics on part segmentation and joint prediction as well.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The task is significant due to its application to real-world scenarios\n- The method is simple, and the paper organization explains it very clearly\n- The results clearly explain the efficacy of the method, particularly the ones on the physical executability.\n\nWeakness\n\n- The novelty of the paper is unclear, it would be helpful to briefly summarize the contributions at the end of the introduction section. As one contribution the authors state (at 44) “a novel dynamic [SEG] token mechanism”, which is the one used in LISA [1]. Moreover, in 3D this technique has already be used in [2,3], to name a few. The novelty in its use is not clear to me (see question 1).\n- The paper is lacking some explainations on the modality used during evaluation (see questions 3-4-5-6)\n\n[1] Lai, Xin, et al. \"Lisa: Reasoning segmentation via large language model.\" CVPR 2024.\n\n[2] Huang, Kuan-Chih, et al. \"Reason3d: Searching and reasoning 3d segmentation via large language model.\" 3DV 2025.\n\n[3] He, Shuting, et al. \"Segpoint: Segment any point cloud via large language model.\" ECCV 2024.\n\nQUESTIONS:\n1. [43-44]: \"In addition, we introduce a novel dynamic [SEG] token mechanism [...]\". From the paper content, I struggle to see the novelty in this mechanism, as it was introduced by LISA [1] and later adopted by various methods for similar tasks [2,3]. It should be discussed more in detail how the paper modify this token mechanism with respect to prior work.\n2. About the physical executability, is there an error anlysis about the sources of error? E.g., percentage of cases in which the URDF is not correctly formed, and percentage in which the format is correct, but the joints are wrong\n3. Tab1: \"count accuracy\" is not defined in the text\n4. Tab2: what is \"oracle\"?\n5. Tab2: Why are Articulate-Anything class split not reported?\n6. [473-484]: from this example, it seems that the network relies on prior knowledge about the object, i.e. the fact that it has N joints. Is this correct? If it is, this is an important assumption for the method, and it should be clearly stated in the paper.\n\n[1] Lai, Xin, et al. \"Lisa: Reasoning segmentation via large language model.\" CVPR 2024.\n[2] Huang, Kuan-Chih, et al. \"Reason3d: Searching and reasoning 3d segmentation via large language model.\" 3DV 2025.\n[3] He, Shuting, et al. \"Segpoint: Segment any point cloud via large language model.\" ECCV 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents URDF-Anything, a novel end-to-end framework that leverages 3D Multimodal Large Language Models (3D MLLMs) to automatically reconstruct articulated object digital twins in URDF format from visual observations (images or point clouds). The method introduces a specialized [SEG] token mechanism to jointly predict object segmentation and kinematic parameters. Experiments on the PartNet-Mobility dataset show substantial improvements in segmentation accuracy, kinematic prediction precision, and physical executability over prior baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The paper addresses an important and underexplored problem in robotic simulation: automatic articulated object modeling and URDF file generation.\n\n+ The framework is end-to-end, combining geometry understanding and kinematic reasoning.\n\n+ Clear reproducibility, and good experimental results compared to former works.\n\n\nWeaknesses:\n- While the use of the [SEG] token for reasoning-based segmentation is intuitive and well-motivated in the LISA framework, its extension in this paper raises certain concerns. In the context of this work, applying [SEG] to guide 3D point cloud segmentation is understandable, since the token can serve as a semantic anchor to align text-based part labels with geometric features. However, what is less clear—and somewhat black-box in nature—is how the model is able to simultaneously infer accurate joints using the same mechanism. The paper lacks a concrete explanation or ablation showing how the [SEG] token contributes to learning the kinematic structure, as opposed to just part segmentation. It almost appears as if the model “magically” learns to associate geometry with articulation semantics, without clearly defined inductive biases or intermediate supervision. A deeper analysis or visualization would greatly help demystify this process.\n\n- This paper lacks sufficient ablation studies to support its methodological choices. Most components are fixed without justification or comparison. In particular, the choice of Uni3D as the 3D backbone is taken for granted, but alternative backbones should have been evaluated for comparison. A more thorough experimental analysis, similar to what was done in the paper Articulate-Anything, would significantly strengthen the paper. That work serves as a good reference for the types of diagnostic or ablation studies that could be conducted. While I will not list them all here, following a similar level of evaluation would be beneficial for this paper.\n\n- My key concern is that this is a compositional work that primarily integrates existing ideas rather than introducing novel concepts. Each component of the paper—whether it's the use of 3D multimodal large language models (3D MLLMs), LoRA for fine-tuning, the incorporation of [SEG] tokens, point cloud generation, mesh reconstruction, or structured output formatting—is based on prior methods. While the system as a whole may function effectively, there is limited novelty in the individual technical contributions. The framework appears to repackage existing tools and techniques into a cohesive pipeline rather than pushing the boundaries of any specific subfield. Please let me know if the authors have any arguments regarding the novelty of this work.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an Multi-modal LLM based pipeline for estimating the URDF of articulated objects. The input is a single or multi-view RGB images from which pointcloud is extracted using off the shelf methods like Dust3r or LGM as well as text. The output of MMLM is a SEG token and kinematic parameters. Segmentation per point is decoded via SEG token using a dedicated deocder. Relevant experiments are conducted and the method is compared to various baselines like Real2Code and URDFformer.\n\nSTRENGTHS AND WEAKNESSES:\nIn my opinion, below are the strengths of this work:\n\n1. A neat integration and usage of MMLMs for articulated object segmentation and kinematic parameter prediction. \n\n2. The method works for arbitrary number of parts in an object which is a major benefit, whereas some of the other methods are restricted to one or two parts. \n\n3. Significant improvement in quantitative results when compared to relevant baselines. \n\n4. The paper is nicely written, easy to follow and diagrams complement the text well. \n\nBelow are the weakness of the method and my questions to authors:\n\n1. I am curious why the method doesn't discuss or compare to feedforward approaches like CARTO [1] or PARIS line of work [2,3] which require multi-view images of start and end state. I am wondering how does the proposed method compares to these approaches interms of speed, accuracy etc. If there is a trade-off for instance the output of autoregressive MMLMs + decoder network are slow compared to feed-forward category-level methods, that should be clearly stated etc. \n\n2. What is considered Out of Distribution instances and how are they determined? Is only the size varied b/w In-distribution and OOD?\n\n3. Is the method category-level or works for novel-classes as well?\n\n4. Lack of real-world evaluation is a limitation. PARIS [2] provides GT annotations which can be used to report real-world metrics, I wonder why did the authors not consider reporting those metrics or on a more challenging sim dataset with more categories [3]\n\n5. While the qualitative results are nice, It is hard to judge the shape reconstruction quality without a more concrete metric like Chamfer distance.\n\n[Minor]\n\n1. Can the method be easily extended to other articulated objects like robot arm etc.?\n\n2. Can the method be solely trained in simulation and transferred to real-world without additional training or finetuning?\n\n\n[1] CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects, CVPR 2023\n[2] PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects, ICCV 2023\n[3] SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splatting, ICCV 2025\n\nQUESTIONS:\nPlease see weakness section for questions. I am looking forward to seeing the author's response in the rebuttal.\n\nLIMITATIONS:\nMentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}