URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model
Authors: Zhe Li
Xiang Bai
Jieyu Zhang
Zhuangzhe Wu
Che Xu
Ying Li
Chengkai Hou
Shanghang Zhang
Venue: NeurIPS 2025 spotlight ()
Keywords: articulated objects
3D multimodal language models
Primary area: deep_learning
TL;DR: URDF-Anything is an end-to-end MLLM-based framework that jointly optimizes geometric segmentation and kinematic prediction for articulated object reconstruction.

Abstract: Constructing accurate digital twins of articulated objects is essential for robotic simulation training and embodied AI world model building, yet historically requires painstaking manual modeling or multi-stage pipelines. In this work, we propose \textbf{URDF-Anything}, an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM). URDF-Anything utilizes an autoregressive prediction framework based on point-cloud and text multimodal input to jointly optimize geometric segmentation and kinematic parameter prediction. It implements a specialized [SEG] token mechanism that interacts directly with point cloud features, enabling fine-grained part-level segmentation while maintaining consistency with the kinematic parameter predictions.
Experiments on both simulated and real-world datasets demonstrate that our method significantly outperforms existing approaches regarding geometric segmentation (mIoU 17\% improvement), kinematic parameter prediction (average error reduction of 29\%), and physical executability (surpassing baselines by 50\%). Notably, our method exhibits excellent generalization ability, performing well even on objects outside the training set. This work provides an efficient solution for constructing digital twins for robotic simulation, significantly enhancing the sim-to-real transfer capability.

URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model
Zhe Li Xiang Bai Jieyu Zhang", Zhuangzhe Wu', Che Xu', Ying Li', Chengkai Hou' Shanghang Zhang
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, PUniversity of Washington CCorresponding author.
*Equal contribution and co-first authors.
URDF- Anything
Segment input point cloud by object ports and predict the articulation parameters of the articulated objects.
Generating Functional URDF Digital Twins from Visual Observa- tions(single multi-view images). Our framework, utilizing 3D Multimodal Large Language Model and guided by instructions (c.g.. "Segment parts and predict parameters"), processes the point cloud to jointly infer geometric part segmentation and kinematic structure. The output is segmented 3D model with defined joints (represented here by different part colors), forming a functional URDF digital twin directly usable in physics simulators.
Figure UnDF-Any
Abstract
Constructing accurate digital twins of articulated objects is essential for robotic simulation training and embodied world model building, yet historically requires painstaking manual modeling or multi-stage pipelines. In this work, propose CLDF-Anything, an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM). URDF-Anything utilizes toregressive prediction framework based on point-cloud and text multimodal input jointly optimize geometric segmentation and kinematic parameter prediction. It implements a specialized [SEG] token mechanism that interacts directly with point cloud features, enabling fine-grained part-level segmentation while main- taining consistency with the kinematic parameter predictions. Experiments on both simulated and real-world datasets demonstrate that method significantly outperforms existing approaches regarding geometric segmentation (mloU 17% improvement). kinematic parameter prediction (average error reduction of 29%), and physical executability (surpassing basclines by 50%). Notably, our method ex- hibits excellent generalization ability, performing well even objects outside the training set. This work provides an efficient solution for constructing digital twins for robotic simulation, significantly enhancing the sim-to-real transfer capability.
Project page: https://lzvsdy..grr http://www....tttuuoo/o/////aa//
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Constructing high-fidelity digital twins is essential for accurately simulating real-world physical dynamics and complex interactions across numerous fields. These include core applications in robotic simulation and training [ 2 as well as expanding domains like autonomous driving 4 and interactive virtual/augmented reality environments 7 g]. However, populating these interactive virtual environments requires precise digital representations of their components, especially articulated objects (doors, drawers, scissors) with their complex internal structures and diverse degrees of freedom, which typically demands painstaking manual modeling, and extensive development time. Automating the reconstruction of these objects into fully functional, high-fidelity digital twins (for example, in URDF format) not only alleviates this laborious effort but also offers an efficient pathway to enrich virtual worlds and enable robust simulation-to- to-rell transfer in robotic training.
However, automatically reconstructing articulated objects from visual observations presents signifi- cant challenges. Unlike rigid objects, articulated objects consist of multiple parts connected by joints, introducing complexities related to inferring both the geometry of individual links and the kinematic parameters (type, origin, axis, limits) governing their motion. Prior efforts reconstruct articulated objects by composing multiple models into a sophisticated pipeline and either rely on a given mesh assets library involve separate part segmentation stage (0).
In this paper, we explore an end-to-end approach for generating functional URDF models directly from visual input; we propose URDF- (nythAny which leverages 3D Multimodal Large Language Models (MLLMs) to jointly interpret object geometry and semantic attributes, infer kinematic structure, and automatically produce high-fidelity URDF descriptions (Figure 2 iillustrates the overall pipeline). 3D MLLM is uniquely suited for this task, as it can natively handle multimodal input (visual features and text instructions), encodes powerful priors on 3D shapes from large-scale pretraining, and directly understands spatial relationships to output precise coordinates. These capabilities make it ideal for predicting detailed kinematic parameters and generalizing to unseen objects. In addition, we leverage a dynamic [SEG] token mechanism [TT] within the MLLM's autoregressive generation, which enables simultaneous prediction of the symbolic articulated structure (link names, joint parameters) and emission of explicit segmentation signals that guide geometric segmentation of object parts from point-cloud features via cross rross-attention. This tight coupling between symbolic output and geometric segmentation, combined with end-to-end training, ensures full consistency between predicted kinematics and reconstructed geometry, and enables an end-to-end approach for reconstructing articulated objects.
Experimental evaluation the PartNet-Mobilic dataset [12] including both in in-distribution and challenging outto--distriuutio objects, demonstrates the superior performance of Anything compared to existing methods. Quantitatively, for part-level geometric segmentation, achieve higher mIoU on OOD instances (0.62) compared to the best baseline (0.51), and a much higher Count Accuracy (0.97), surpassing the best baseline (0.84) by over 15 points. In joint parameter prediction, our method consistently achieves significantly lower errors across all object categories compared to baselines, outperforming them by considerable margin, particularly physical executability rate of our generated URDF models is substantially higher than baselines(50%% improvement), enabling more robust simulation for unseen objects. These comprehensive results highlight the effectiveness and strong generalization capability of our end-to-end MLLM framework for automated articulated object reconstruction.
OOD objects. Crucially, the
In summary, our main contributions are as follows:
Proposing the first end-to-end 3D MLLM framework for articulated object reconstruction, championing new paradigm from complete 3D input to joint prediction output.
Achieving deep coupling and joint prediction of kinematic parameters and geometric seg- mentation through an innovative application of the [SEG] token. Demonstrating the superiority of this new paradigm through extensive experiments.
2 Related Work
LLMs for 3D Tasks. Recent years have seen significant progress in applying Large Multimodal 14
Models (LMMs) to 3D understanding and interaction tasks [3 models leverage multimodal inputs (such point clouds images, combined with language) for 3D
16
18
19
20 21). These

---

spatial reasoning, perception, and structured output generation. Relevant advancements include the development of 3D MILM backbones like ShapeLLM [22]. capable of handling point cloud-language interaction and demonstrating strong 3D understanding. Other works have explored language- -ggdeed
3D segmentation [23], open-world 3D understanding [24], and tasks requiring precise 3D spatial understanding from visual input (25]. While these models demonstrate powerful capabilities in general 3D perception and generating text or segmentation masks, they do not typically address the complex, coupled task of jointly inferring detailed link geometry and precise kinematic parameters for articulated objects. Our work builds upon these advancements in 3D LMMs and their capabilities for spatial reasoning and multimodal generation, specifically applying them to the challenging task of articulated object reconstruction.
Articulated object modeling. The automated modeling of articulated objects for robotic manipulation
an active field of research, with several distinct methodological approaches. Physics-based
interactive methods [2 27 28 utilize interaction in simulation or the real world to refine or build models, demonstrating high accuracy with sufficient interaction but often requiring initial models, struggling with robust passive reconstruction (without interaction), or limited adaptation to novel geometries without extensive interaction data. Automation methods [TO, 9 leveraging visual-language models automate digital twin creation, using techniques such VLM-to-code generation, iterative refinement, mesh retrieval [0] or abstracting parts as OBBs with LLM parameter prediction [9]. However, these methods can be limited by constraints such reliance asset databases, brittleness in iterative processes, or loss of geometric detail and potential parameter inaccuracy from using simplified representations like OBBs. 29 is fundamentally limited by its reliance on a hard-coded system to assign kinematic parameters and retrieve meshes based on the network's discrete part classification output, method which compromises final geometric and kinematic fidelity. [30) is a specialized, small-scale feed-forward model with severe input and output restrictions. Other works [31 32
35) explore novel representations like 3D Gaussians for reconstruction or focus on detection of specific features [36]. addressing sub-problems rather than the complete, end-to-end geometry and kinematics inference pipeline needed for functional URDE Our work contributes to the field of automated articulated object modeling by proposing novel end-to-end framework that leverages the capabilities of 3D Large Multimodal Models for robust reconstruction from visual input. While recent works have explored VLMs or 2D-based LLMs for URDF generation [T0 9 none have leveraged raw 3D point clouds as the primary input to an MLLM for end-to-end URDF synthesis a key enabler of geometric precision in our framework.
33
34
3 Method
URDF-Anything is an end-to-end framework for reconstructing articulated objects from visual observations into URDF-formatedd digital twins. The pipeline consists of three main stages: (1) Input Representation, where we generate dense 3D point clouds from single or multi-view RGB images, in Sed3.2: ; (2) Multimodal Articulation Parsing, where 3D Multimodal Large Language Model (MLLM) jointly predicts part segmentation and kinematic parameters, in Sed3.3 ; and (3) Mesh Conversion, where segmented point clouds are converted into meshes for simulation, in SecB.5 The key innovation lies in the integration of geometric and semantic features through a dynamic [SE token mechanism, enabling precise part-level segmentation and joint parameter prediction a unified framework.
3.1 Task Definition
To ensure compatibility with standard 3D simulators (e-g., MuloCo, PyBullet), we represent artieu- lated objects as URDF (Unified Robot Description Format) models. A URDF structure defines an object as a hicrarchical tree composed of:
Links: Rigid components representing object parts (e.g., cabinet's door, drawer, or base). Each link contains geometric (mesh) and inertial properties.
Joints: Connections between links that specify kinematic constraints. Each joint includes:
Type: Prismatic, revolute, continuous, floating, planar or fixed.
Parent/Child: Links connected by the joint.
Origin: 3D position (x, y, 2) e 3 and orientation (r. P. v) 123 relative to the parent link.
3

---

point clouds
Paky
point
URDF File
Backbone
Chy vera
3D Decoder
3D MLLM
lora
Origin ayz (1.57. 0.00, 1.57)
spintt: Segment the cloud into prodict ponameters
Origin (-0.16. 067.-0.28) Zoint type (revolute)
Ois -SE5
Figure 2: Overview of the UnDF-Any Framework. The pipeline takes a 3D point cloud (from image) and structured language instruction input. The 3D MILLM(finetunne) with LoRA) autore- gressively generates symbolic output (kinematic parameters) and [SEG] tokens. The embeddings corresponding the generated [SEG] tokens then interact with the point cloud features via 3D Decoder to perform fine- ane-grained geometric segmentation of the point cloud into individual links. Finally, the jointly predicted kinematic parameters and the segmented geometry are integrated into functional URDF file, resulting in complete articulated 3D model ready for physics simulation.
Axis: Normalized 3D vector defining the motion direction (c.g., [0, 1, 0) for vertical sliding). Limit: Defines the range of motion for prismatic, revolute and planar joints.
Our objective is to automatically infer these URDF components-speccineee the type, origin, parent, child, and axis for each joint from visual input of the articulated object.
the mesh for each link,
Input Representation
The initial step of our pipeline is to transform the raw visual observation of articulated object into a 3D representation suitable for structural parsing. Real-world observations can vary significantly, sometimes providing multiple viewpoints of the object and other times offering only a single view. Accordingly, our point cloud acquisition strategy adapts to the input modality:
When multiple RGB images from different viewpoints are available, we leverage DUS:3R 37 to generate a dense 3D point cloud Porj. DUSt3R excels establishing dense 2D-to-3D correspondences and reconstructing accurate 3D geometry from multiple views.
Multi-view Input:
For the more challenging single-vies scenario, we utilize generative approach LGM[38]. We first employ pre-trained diffusion model to synthesize consistent multi-view images. Subsequently, we reconstruct the 3D geometry from these synthesized views, which can be represented as 3D Gaussian Splatting model or converted into a point cloud Pabj-
Single-view Input:
Regardless of the input type (single multi-view), the output of this stage is dense point cloud Pots E R xG representing the entire articulated object. is crucial to note that this initial point cloud a monolithic representation; captures the overall geometry but does inherently differentiate or segment the object into its individual links or sub-components. This wholc-object point cloud Pot serves the primary geometric input for the subsequent multimodal articulation parsing stage. More details about this section are reported in Appendix A.2
3.3 Articulation Parsing with 3D MILM
recent 3D MILM, as our backbone, which combines point cloud encoder(39) with large language model[40). ShapeLLM has demonstrated the ability perform 3D visual grounding and output structured 3D information, such 3D bounding- box coordinates. This pre-existing capability is particularly beneficial for our task of predicting joint parameters, which inherently involve 3D locations (origin, axis) and orientations (origin). This unified architecture offers two advantages: (1) Open-world generalization: The 3D MILM''s geometric understanding capability, enhanced by large-scale 3D-language pretraining, enables robust reasoning about both seen and
We adopt ShapeLLM 22

---

unseen object categories. (2) Structured output capability: The MLLM inherently supports generating coom-tomattedd outputs of arbitrary length and complexity, enabling flexible representation joint hierarchies and part relationships.
The input point cloud Parj is encoded by 3D encoder extract dense geometrie features Fee E R Mxdee, where M is the number of points (potentially downsampled) and dze is the feature dimension. Simultaneously. natural language instructions are provided to guide the parsing process. We employ structured instruction template that explicitly incorporates the geometric features and prompts the model for specific URDF information. The textual part of the instruction, Xtst, is processed by the LLM's standard word embedding layer to yield text embeddings Fist E Rxdde where L is the number of tokens and dræt is the embedding dimension. These multimodal features (FFc and Fixt) are integrated and processed by the LLM's layers. This core multimodal processing and output generation can be formally represented as:
Input:
Youtput = MLLM(Fpe> Feer)
Output: While 3D MLLMS excel processing multimodal inputs and generating structured tex- tual outputs, they are not inherently equipped to generate dense, per-point predictions required for geometric segmentation. Our task necessitates not only predicting the symbolic URDF structure and parameters but also precisely segmenting the corresponding point cloud geometry for each link. Inspired by LISA[[]], objective. The MLLM is trained to autoregressively generate a structured ISON output sequence that contains the predicted joint parameters alongside descriptions of the links. Crucially, each link description in the output is associated with a FSG token (e.g., "link_0" "base "base_cabinet|] F.G.", "link_1" 1": "drawer[S [[ This means the MILMM simultaneously predicts the symbolic representa- tion of the articulated structure and places markers (SEG) indicating the geometry corresponding each part needs be identified. More details reported in Appendix A.3
extend the vocabulary with special token [SEG] to achieve this joint
3.4 Geometric Segmentation from Spkeial-Token Mechanism
Geometric segmentation is performed for each object part indicated by [SEG] token in the MLLM's output sequence Youtput- For each generated [SEG] token, we leverage its final hidden state hagg- combined with the preceding part category token's 's state Reategory to form a fused token representation hrombined [heategory; hege) This representation is then used as the query
H Hquery MLP query MLPqqerrg anneegghhommbrr (beombined)
in cross-attention mechanism. Another point feature Spe is generated from 3D backbone Spc F (Por)) Then a mechanism attends over the projected point cloud features F% MLP, (Spo). effectively computing per-point scores
Umask CrossAttn(Q = -aatttmQ@-Quuaaaa K = F V F2e)
indicating the likelihood of each point belonging to the part. These scores Ymask are converted into a binary segmentation mask for the corresponding link via sigmoid and threshold. This process is repeated for every [SEG] token, yielding masks for all predicted parts.
3.5 Mesh Conversion and URDF File Generation
The final stage of our pipeline consolidates the reconstructed geometry and kinematics into standard URDF XML file, ready for direct use in physics simulators. Segmented point clouds for each link, obtained from the geometric segmentation process (Section are converted into 3D mesh representations (e.g., OBJ format) using point-to-mesh conversion method 41 42 Simultaneously. the MILM's s structured JSON output provides the complete kinematic structure, including joint types, parent/child relationships, origins, and axis. We parse this JSON and assemble the final URDF XML file, where each link references its generated mesh and joint elements are populated with the predicted parameters. The resulting URDF model can be directly imported into standard physics simulators A complete example of a generated URDF file for sample object
(e.g., MuloCo 43 Sapiens
44
instance is provided in Figure8
5

---

3.6 Model Training
The URDF-Anything model is trained end-to-end to jointly generate the structured URDF parameters and predict accurate part-level segmentation masks from the point cloud. The overall training objective L is weighted sum of the language modeling loss Lient and the segmentation loss Leegi
L Atezt Litext + Asei
Σ Ti,seg
where Atm and Aseg are hyperparameters balancing the contribution of each loss, and N is the num of object parts. The segmentation loss uses combination of binary cross-entropy (BCE) loss and Dice loss (DICE). For a given part associated with a [SEG] token, let Mgt E (0, 1 AM be the ground truth binary mask for that part. The segmentation loss for this part is:
Tseg Anee BCF(M, Mgt) + DICH (M, Mgs)
where Abce and Adice hyperparameters weighting the BCE and DICE components.
4 Experiments
Implementation Details: We employ ShapeLLM 22 as our 3D MLLM backbone, with LLap TB-general-v1.0 checkpoint the default settings. For the 3D backbone, We Uni3D 45 to extract dense geometric features. We adopt one NVIDIA 80G A800 GPU for training. We employ LoRA 46 for efficient fine-tuning, with the rank of LoRA set to 8 by default. We use AdamW optimizer 47 with the learning rate and weight decay set to 0.0003 and 0. respectively. We use the cosine learning rate scheduler, with the warm-up iteration ratio set 0.03. The batch size per device is set to 2, and the gradient accumulation step is set to 10. Our model was fine-tuned in 2.5 hours on single NVIDIA A800 (80GB) GPU.
We train and evaluate our framework on the Porbiit-Mobility dataset 12], large collection of 3D articulated objects with URDF annotations. The dataset is partitioned into In In-Distribution (ID) and nut-of-Distribur
Dataset:
(OOD) subsets based solely on object categories. Since our method utilizes image input, generated visual data by rendering multi-view and single-view RGB images from the dataset's s 3D models within a simulation environment. Ground truth kinematic and geometric information from the original URDF files processed and reorganized into a compact JSON format matching our model's 's output structure. The dataset is divided into standard training and testing sets. More details are reported in Appendix A.I
We compare against three prior methods: (DArtiiclate-AAnyhinng an actor-critic system for iterative refinement via mesh retrieval mechanism to generate code that be compiled into simulators. (2) Real2Code: an approach that represents object parts with oriented bounding boxes, and uses fine-tuned LLM to predict joint parameters code. (3)RRDFormer: pipeline that constructs simulation scenes complete with articulated structure directly from real-world images.
Baselines:
Evaluation Metrics: We evaluate the performance of our URDF-Anything framework on three key aspects: part-level geometric segmentation, kinematic parameter prediction accuracy, and the physical executability of the final reconstructed URDF.
Part Segmentation Accuracy:
For evaluating how accurately our method segments the point cloud into individual links, we use mloU. A higher mloU indicates better alignment between predicted and ground truth part geometry. We use Count Ace to measure the percentage of test samples where the number of predicted articulated parts (i.e., links with non-fixed joints) exactly matches the ground-truth count.
Joint Parameter Prediction Accuracy: Following Articulate-Anyin Anythinytthhiiii [10), evaluate the correctness of several critical components for each joint prediction compared to the ground truth binary metric indicating if the predicted joint type (e.g., revolute, prismatic) matches the ground truth. (2)Joint Axis Error: Quantifies the angular difference between the predicted and ground truth joint axes, normalized be within [0, m). (3)Joint Origin
URDF: (1)Jt Type Error:
Error: Measures the positional error of the joint origin.
Physical Executability: Beyond static accuracy metrics, the ultimate test of a reconstructed URDF is its functionality in a physics simulation. We evaluate the physical exccutability by

---

Table 1: Quantitative Results for Part-Level Link Segmentation. Comparison of our method ([SEG](Ours) against baselines (Uni3D w/o text, Uni3D w/ text) on In-Distribution (ID) and Out-of- Distribution (OOD) object instances. Performance is measured by mloU (segment accuracy) and
Count Accuracy (Count Acc), both where higher is better (t).
m
Count Ace f
Models
ALL
ID
0OD
ALL
ID
OOD
Uni3D w/o text
0.001
0.50
0.33 0.01
0.70
0.83
0.70
Uni3D w/ text
0.54
0.51 0.64
00002
0.04
0.91
0.02
URDF-Anything ([SEG)
0.63 (16.7%))
0.69 0.01
0.62 0.01 0.97 (15.4%1)
0.99
1.902
loading the generated URDF models into a standard 3D simulator (e.g., Sapiens [4]).. We then programmatically or manually attempt to actuate the joints. The physical executability metric is defined the percentage of generated URDFs that can be loaded and actuated correctly in the simulator without exhibiting non non-physical behavior (e.g., parts flying off, joints freezing, unexpected rotations/translations)) This metric provides crucial validation of the end-to-end reconstruction quality and its suitability for sim-to-real applications.
4.1 Link Segmentation Results
Table presents the quantitative results for part-level geometric segmentation. This evaluation assesses how accurately each method can delineate the individual links of an articulated object within its point cloud representation. We compare our proposed method against two baselines that represent different configurations of utilizing Uni3D features: (1) Uni3D w/o text, a supervised closed-set model that adds an MLP head to elassify points into predefined part categories using only geometric features; and (2) Uni3D w/ text, which follows Uni3D's original text-guided paradigm by aligning point features with a fixed list of part-name prompts (e.g., drawer", base") via feature propagation layer. Both baselines lack the dynamic, context-aware coupling between segmentation and kinematic structure enabled by our [SEG] token mechanism.
The poor performance Uni3D w/o text demonstrates that relying solely on geometric features is insufficient, highlighting the necessity of semantic guidance. Its weakness stems from paradigm mismatch- it discards the cross- modal semantic aligument Uni3D was pre-trained for. Meanwhile, method significantly outperforms Uni3D w/ text, showing that our [SEG] token mechanism provides a superior guidance strategy: unlike static text prompting, it dynamically and end-to-end couples segmentation with the autoregressive generation of kinematic structure, yielding more accurate and context-aware results.
Our method achieves state-of-the-art performance across all metrics and datasets. Quantitatively, URDF-Anythin ([S EG) significantly outperforms the baseline on average, achieving 0.63 mloU (a 16.7% improvement) and 0.97 Count Accuracy (a 15.4% improvement). This strong performance holds for both ID and OOD data, particularly on challenging OOD instances, demonstrating superior generalization capability compared to baselines. The significant improvement in mloU on OOD data demonstrates superior generalization capability in segmenting novel object geometries and structures not seen during training.
Table 2: Quantitative Comparison of Joint Parameter Prediction Accuracy. This table presents the average prediction errors for joint Type (fraction of incorrect), Axis (radians), and Origin (meters) across different methods. Results are shown for All, In-Distribution (ID), and Out-of-Distribution (OOD) object classes. Lower values indicate better performance (Q). "Oracle" denotes baselines evaluated with ground-truth part segmentation to isolate kinematic prediction performance, following
the protocol in [10].
Method
All Classes
ID Classes
OOD Classes
Type
Axis
Origin
Type
Axis I
Origio
Type 4
Axis
Origin
Real2Code Oracle
0.537=cm
1.006
0.294 204r7
0.410 2.16 sumn 0.344 2040
0.576 ome
0.937
0.272
URDFormer Oraclo
0.556 =
0.374
#03s
0.609 0.208
0.679
0.643
0.513+0340
Articulate- ArticulateAAnything
0.025 2000
045
20450
0,292
#0392
0.018
0. 143
0. 09195
0.026 00005
0,145 0.208 =04m
URDE-Aay thing (Ours)
0.164
0.007 2a00
0.009
0.136::0as
0.173

---

Cabinet
Desk
Drawer
Laptop
Image
URDFormer
Articulate
Anything
Ours
Figure 3: Qualitative Comparison of Articulated Object Reconstruction Results. The top row displays the input image for various articulated object instances (each column represents a different object). We can find that baseline methods frequently struggle in predicting incorrect object types, generating distorted geometry, or exhibiting significant errors link placement, leading misaligned or incorrect structures.
4.2 Joint Parameter Prediction Results
Table presents the quantitative results for joint parameter prediction accuracy. We compare our proposed framework, URDF-Anything (Ours), against several prior methods. As shown in Table 2 URDF-Anything demonstrates superior performance compared to baseline methods across all evaluated categories. Specifically, our method achieves notably lower Type, Axis, and Origin errors when evaluated on All Classes, ID Classes, and particularly OOD Classes. This indicates a significant improvement in accurately predicting kinematic parameters for articulated objects, including robust performance on object categories not during training (OOD instances).
This strong performance on OOD articulated objects highlights the effectiveness of our MLLM-based end-to-end approach. The MLLM's inherent capabilities understanding complex structures and generalizing from large-scale pretraining, combined with our method's 's joint reasoning of geome- try and kinematics, enables more accurate parameter prediction in challenging, unseen scenarios compared to prior methods that may struggle with complexity or lack strong generalization priors.
4.3 Physical Executability
Beyond evaluating static link segmentation and joint parameter accuracy, critical validation of our end-to-end reconstruction framework is the physical executability the generated URDF models in simulation environment. As defined in Section this metric assesses the percentage of generated

---

Table 3: Physical Executability Rate (% 1) across methods on ID and OOD subsets.
Method
ALL
ID Classes
OOD Classes
URDFormer Oracle
24%
34%
15%
Real2Code Oracle
41 %
23% 49%
Articulate-nyything
52%
61%
44%
URDF-Anythhng(OOrs))
78 % (50.0%))
86 %
71
Table Ablation Study on Input Modality for Joint Parameter Prediction.
Method Variant
Input Modality
Type +
Axis
Origin
OBB
Text
0.42
0.70
0.47 +0.22
Point Cloud only
Point Cloud
0.14
0.29
0.26 20.15
Owen2.5-VL-7B Qwen2.5-VL-7B +
Image Text
0.57
0.85
0.23 0.116
Image + Text
0.38
0.81 #0024
0.18 0.
Point Cloud + Text (Ours)
Point Cloud + Text
0.00
0.142
0.024
URDFs that can be successfully loaded and actuated a physics simulator without errors or non- physical behavior.
As shown in Table URDF-Anything achieves high physical executability rate, significantly surpassing baseline methods, particularly for OOD objects. This demonstrates the superior overall pipeline robustness our approach. Compared to prior methods like Real2Code (9, which rely on complex, sequential pipelines where errors in one step cascade and require manual intervention for refinement, or Articulate-Anything [10), which may depend on iterative refinement for parameter estimation, our framework utilizes a unified, end-to-end MLLM that jointly reasons about geometry and kinematics. This direct, integrated approach minimizes error propagation, allowing the model to leverage rich multimodal context for robust prediction of consistent geometric and kinematic structure in a single pass. Figure and provides qualitative results illustrating the visual quality of our reconstructed articulated objects, showcasing both the generated link segmentation and the resulting mesh geometry for various challenging instances.
4.4 Ablation Study
Impact of the geometric representation Modality. We conducted an ablation study on input modality, with results in Table First, found that even powerful, fine-tuned image MLLMs (e.g.. "Qwen2.5-VL-7B + ft') struggle to infer precise 3D kinematic parameters from 2D images, yielding significantly higher errors than any 3D-based approach. This highlights the necessity of explicit 3D geometry for this task.
Next, we evaluated different 3D representations. We observed that simplified geometry, such as Oriented Bounding Boxes ('OBB'), insufficient due to the loss of crucial geometric detail. While using detailed point cloud alone ("Point Cloud only') improves performance, ('Point Cloud Text'), ) which integrates language guidance into MLLM, achieves the best results by significant margin. This highlights the necessity of language and MLLM. Collectively, this study validates our core design choice: achieving high-fidelity reconstruction requires the combination of detailed 3D geometric input and effective language guidance within 3D MLLM framework.
full method
Importance of Joint Geometric and Kinematic Prediction. core hypothesis of our work is that jointly predicting an object's geometry and kinematics is superior to tackling these tasks independently. We posit that structured reasoning about kinematics provides crucial context for geometric understanding, and similarly, geometric features must ground the kinematic predictions. To validate this, we compare our full, jointly-trained model against two decoupled variants:
Kinematics-Only: This model is trained solely on the language modeling loss (Lrezt) to predict URDF parameters, without the [SEG] token or the associated segmentation loss.
Segmentation-Onle This model trained to generate only the link names and [SEG] tokens, optimizing exclusively for the segmentation loss (Leeg), without predicting the kinematic structure.

---

The results, presented in Table clearly demonstrate the performance degradation when the tasks are decoupled. The "Kinematics-nnly model, lacking the geometric regularization provided by the segmentation task, shows a decline in parameter accuracy. This quantitatively confirms that the segmentation task acts as a crucial "booster" for learning accurate kinematic structures. More strikingly, the "Segmentation-Onle This suggests that the structured reasoning required to predict the full kinematic tree forces the model to learn a more coherent and robust internal representation of the object's 's structure, which in turn benefits the geometric segmentation task.
model also shows a drop in both mloU and part count accuracy.
These results provide strong evidence for the mutual benefits of joint prediction. Geometry provides essential regularization for predicting accurate kinematics, while the task of inferring kinematics imposes structural prior that enhances geometric segmentation. This validates our end-to-end joint prediction paradigm as essential for achieving high-pes pegh-performacc reconstruction.
Table 5: Ablation Joint Geometric and Kinematic Prediction.
Kinematic
Segmentation
Model Variant
Loss
Type
Axis
Origin
mloU
Count Acc
Kinematics-Only Segmentation-Only
Lrext Leseg
0.009
0.138
0.175
0.61
0.89
URDF-Anything (Joint)
Loeg + Ltext
0.008
0.132
0.164
0.63
0.97
The results from these key ablations underscore a central finding of work: constructing successful framework for this task is technically non-trivial. The preceding experiments demonstrate that seemingly plausible, simpler approaches systematically fail. For instance, relying 2D) image inputs (Table 4 leads to geometrically imprecise results, while decoupling the geometric and kinematic prediction tasks (Ta results in error propagation and performance degradation in both domains. Our success therefore stems not from simple combination of modules, but from a series of deliberate, non-trivial design choices-from the foundational use of 3D point clouds to the end-to-end joint reasoning mechanism.
Summary of Design Choices.
5 Conclusion
In this paper, presented URDF-Anything, novel end-to-end framework that leverages the power 3D Multimodal Large Language Models (MLLMs) to reconstruct functional URDF digital twins of articulated objects directly from visual observations. By utilizing the MLLM's inherent capabilities and introducing a dynamic [SEG] token mechanism for joint geometric segmentation and kinematic parameter prediction, method overcomes limitations of prior decoupled simplified approaches. Experiments demonstrate that URDF-Anything achieves state-of offtthe-art performance across segmentation, parameter prediction, and physical executability metrics on the PartNet- Mobiiltyyyyy dataset, exhibiting superior generalization to complex and unseen objects. This work provides a robust and efficient solution for automated articulated object digital twin creation, significantly advancing capabilities for robotic simulation.
Limitations. While URDF- Anything demonstrates significant advancements, limitations include the inability to generate certain URDF properties (e.g.. mass, moment of inertia), partly due to training data and base model constraints. Additionally, the pipeline is not fully end-to-end, relying on an external point-to- to-mthh conversion module to generate link geometry. Precision of numerical parameters is also limited by the loken-based generation approach.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (62476011),
10

---

References
[1] Alessio Baratta, Antonio Cimino, Francesco Longo, and Letizia Nicoletti. Digital twin for human- robot collaboration enhancement in manufacturing systems: Literature review and direction for future developments. Computers & Industrial Engineering. 187:109764, 2024.
[2] Junfei Li and Simon X Yang. Digital twins embodied artificial intelligence: review and perspective. Intelligence & Robotics, 5(1) ):20-222,, 2025.
[3] S M Mostaq Hossain, Sohag Kumar Saha, Shampa Banik, and Trapa Banik. A new era of mobility: Exploring digital twin applications in autonomous vehicular systems, 2023.
[4] Yangyang Zhou, Dong Hu, Chao Huang, and Hailong Huang. 2024 8th CAA International Conference Vehicular Control and Intelligence (CVCI), pages 1-6, 2024.
Review of digital twins in intelligent driving.
[5] Orkun Kizilirmak, Emre Kaplan, and Çağr Güzza. Digital twin architecture for autonomous driving validation and verification. 2023 14th International Conference on Electrical and Electromics Engineering (ELECO), pages 1-6, 2023.
[6] Youngho Lee, Sejin Oh, Choonsung Shin, and Woontack Woo. Recent trends in ubiquitous virtual reality. In 2008 International Symposium on Ubiquitous Virtual Reality, pages 33-36. IEEE, 2008.
[7] Miao Wang, Xu Qu-Quan Lyu, Yi-Jun Li, and Fang-Lue Zhang. Vr content creation and exploration with deep learning: A survey. Computational Visual Media, 6:3-28, 2020.
[8] Dooyoung Kim. Taewook Ha, Jinseok Hong. Seonji Kim, Selin Choi, Heejeong Ko, and Woontack Woo. Meta-objects: Interactive and multisensory virtual objects learned from the real world for use in augmented reality, 2025.
[9] Zhao Mandi, Yijia Weng, Dominik Bauer, and Shuran Song. Real2code: Reconstruct articulated objects via code generation, 2024.
[10] Long Le, Jason Xie, William Liang, Hung-Ju Wang, Yue Yang, Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, and Eric Eaton. Arriculate-anything: Automatic modeling of articulated objects via a vision-language foundation model, 2025.
[11] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan. Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-9589, 2024.
[12] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding, 2018.
[13] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 24428826438, 2024.
[14] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14455-14465, 2024.
[15] Weipeng Deng. Jihan Yang. Runyu Ding, Jiahui Liu, Yijiang Li, Xiaojuan Qi, and Edith Ngai. Can 3d vision-language models truly understand natural language? arXiv preprint arX.v:2403. 14760, 2024.
[16] Zhangyang Qi, Ye Fang. Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for languae understanding and generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 26417-26422 2024.
[17] Yilun Chea, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.103: 10370, 2024.
[18] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empower- ing large language models to understand point clouds. In European Conference on Computer Vision, pages 131-147. Springer, 2025.
[19] Zhihao Yuan, Jinke Ren, Chun-Mei Feng. Hengshuang Zhao, Shuguang Cui, and Zhen Li. Visual programming for zero-shot open- vocabulcy 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20623 20633, 2024.
11

---

[20] Jiajun Deng. Tianyu He, Li Jiang. Tianyu Wang, Feras Dayoub, and Ian Reid. 3d-llava: Towards generalist 3d Imms with omni superpoint transformer. arXiv preprini arXiv:2501.01 01150010116,,, 2025.
[21] Senqiao Yang. Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-llm: Ilm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312. 14074. 2023.
[22] Zekun Qi, Runpei Dong. Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal object understanding for embodied interaction, 2024.
[23] Shuting He, Henghui Ding, Xudong Jiang, and Bihan Wen. Segpoint: Segment any point cloud via large language model, 2024.
[24] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng. Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointelip v2: Prompting clip and gpt for powerful 3d open-world learning. 2023.
[25] Fan Yang. Sicheng Zhao, Yanhao Zhang, Hui Chen, Haonan Lu, Jungong Han, and Guiguang Ding. Llmi3d: Milm-based 3d perception from single 2d image, 2025.
[26] Jun Lv,, Qiaojun Yu, Lin Shao, Wenhai Liu, Wenqiang Xu, and Cewu Lu. Sagei-system: Towards sample-efficient, generalizable, compositional, and incremental robot learning, 2022.
[27] Cheng- Chuu Hsu, Ben Abbatemattco, Zhenyu Jiang. Yuke Zhu, Roberto Martin-Martín, and Joydeep Biswas. Kinscene: Model-based mobile manipulation articulated scenes, 2024.
[28] Liqian Ma, Jiaojiao Meng, Shuntao Liu, Weihang Chen, Jing Xu, and Rui Chen. Sim2rcal?; Actively building explicit physics model for precise articulated object manipulation, 2023.
[29] Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo. Alex Fang, Karthikeya Vemuri, Alan Wu, Dieter Fox, and Abhishek Gupta. Urdformer: A pipeline for constructing articulated simulation environments from real-world images, 2024.
[30] Nick Heppert, Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Rares Andrei Ambrus, Jeannette Bohg, Abhinav Valada, and Thomas Kollar, Carto: Category and joint agnostic reconstruction of articulated objects. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 21201-21210. IEEE, June 2023.
[31] Yu Liu, Baoxiong Jia, Ruijic Lu, Junfeng Ni, Song-Chun Zhu, and Siyuan Huang. Artgs: Building interactable replicas of complex articulated objects via gaussian splatting, 2025.
[32] Jiayi Liu, Ali Mahdavi-Amiri, and Manolis Savva, Paris: Part-level reconstruction and motion analysis for articulated objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 352-363, 2023.
[33] Jiahui Lei, Yufu Wang, Georgios Pavlakos, Lingjie Liu, and Kostas Daniildis. Gart: Gaussian articulated template models, 2023.
[34] Shaojic Ma, Yawei Luo, Wei Yang, and Yi Yang. Mags: Reconstructing and simulating dynamic 3d objects with mesh-adsorhed gaussian splatting, 2024.
[35] Di Wu, Liu Liu, Zhou Linli, Anran Huang, Liangtu Song, Qiaojun Yu, Qi Wu, and Cewu Lu. Reartgs: Reconstructing and generating articulated objects via 3d gaussian splatting with geometric and motion constraints, 2025.
[36] Xiaohao Sun, Hanxiao Jiang, Manolis Savva, and Angel Xuan Chang. Opdmulti: Openable part detection for multiple objects, 2023.
[37] Shuzhe Wang. Vincent Leroy. Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy, 2024.
[38] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwci Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation, 2024.
[39] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge. Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining, 2023.
[40] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ana Lachaux. Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
12

---

[41] Н. Edelsbrunner, D. Kirkpatrick, and R. Seidel. On the shape of set of points in the plane. IEEE Transactions Information Theory. 29(4):511-59, 1983.
[42] F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva, and G. Taubin. The hall-pivoting algorithm for surface reconstruction. IEEE Transactions Visualization and Computer Graphics, 5(4):349-335 1999.
[43] Emanuel Todorov, Tom Crez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033, IEEE, 2012.
[44] Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stu- Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models. arXiv preprint arXiv:2408. 12569, 2024.
[45] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation scale. International Conference Learning Representations (ICLR), 2024.
[46] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contributions claimed correspond with experiment result in Sectior 4 and we provide detailed figures and tables.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This study focuses empirical/applied aspects rather than theoretical deriva-
tions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide experiment detail in Section4
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We provide supplemental material, including instruction to download and prepare data and how use our code and derive the result. We also include our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//uuui guwde//ddee/ee/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experiment details in section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The reported experimental results are not accompanied by error bars, confi- dence intervals, or statistical significance tests to quantify the variability of the performance metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include compute resources information in section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://////uapppppccccappaaa////
Answer: [Yes]
Justification: We went through the guidelines carefully and we are sure that our research conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper pose such risks as novel model.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper pose such risks as novel model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all relevant papers in out references.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed README file in supplementary material for ease of use.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't 't involve crowdsourcing and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We don't involve crowdsourcing and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
The proposed framework utilizes Large Language Model as and important component of its methodology. The role and implementation details of the LLM are described in Section 333 and Section 4
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy lattps://neurip:: http://arrps.cccooooooorrrrrr/// for what should or should not be described.
20

---

A Appendix
A.1 Dataset Preparation Details
Following prior work 9 10), define In-Distribution (ID) objects those from five categories: Laptop, Box, Refrigerator, StorageFurniture, and Table. All other 41 categories in the PartNet- Mobility dataset are held out as But-of-Distribuute (OOD) evaluate generalization to entirely unseen object classes.
The PartNet-Mobility dataset provides raw URDF files and mesh files. To prepare the data for our MLLM framework, we performed several processing steps to ensure consistency and extract necessary information.
Firstly, we performed a coordinate normalization and URDF structure regularization process. This involved establishing a consistent "base" link as the root reference frame for each object's kinematic tree. We then programmatically calculated the transformation (position zyz and orientation rpy) from this "base" link to each subsequent link's frame. The URDF structure was then modified: joints not already having 'base' as their parent were re-configured so that their child link connects directly to the "base" link. The '<origin>' tag of these re-parented joints was updated to the previously calculated transformation from "base" to the child link's frame, effectively defining the child link's pose relative to the "base'. Additionally, for links that originally referenced multiple mesh files within their "<visual>' and/or "<collision>' tags (c.g., multiple '<visual>' elements each with a mesh), these URDF entries were consolidated. Each link was modified reference single, representative mesh file for its visual geometry and similarly for its collision geometry. The original local transform (origin zyz and rpy) specified in the first-found visual collision mesh entry for that link preserved and applied to these new consolidated entries. This overall simplification aimed to streamline subsequent geometric processing. An example of original URDF file, which serves input processing, is shown in Figure 8 The result of this URDF processing, particularly the consolidated representation for links, is illustrated in Figure
Secondly, for the purpose of dataset splitting and ensuring manageable complexity during training, filtered the dataset based on the number of parts. Specifically, objects with fewer than articulated parts (links involved in joints) were selected to form the primary dataset used for training and evaluation splits.
Finally, since our framework requires visual input, we generated synthetic multi-view and multi- state RGB images from the processed 3D models using the SAPIENS simulator. For multi-view rendering, we employed two strategies: capturing viewpoints from an equator plane and capturing viewpoints distributed spherically (Figure 4 The spherical viewpoint distribution utilized minimum potential energy method ensure relatively uniform coverage around the object. These rendered images, along with the corresponding point clouds derived from the 3D models, served as the visual input for our model.
A.2 Point Cloud Generation Details
As detailed in Section 3.2 the initial step of our pipeline is the generation a dense 3D point cloud Poty from the input visual observations. This process adapts based whether multi-view or single- view images are available. The generated point cloud serves as the primary geometric input for the subsequent multimodal processing stages.
A.2.1 Multi-view Point Cloud Generation
When multiple RGB images of the articulated object from different viewpoints are available, we leverage the capabilities of the pre-traue pre-trained DUSt3R model [37]. DUSt3R is specifically designed for establishing dense 2D-10-3D correspondences across multiple images and reconstructing accurate geometry. We feed the set of multi-view images into DUSt3R, which outputs dense point cloud Poty representing the reconstructed 3D structure of the object. Example results are showed in Figure
6
21

---

1
2
The articulated object [Object Category] consists of [Number of Parts] parts. [Optional descriptive phrases about links or overall object, e.g., This object is kitchen faucet. There are two distinct handles or witches positioned on either side the base.] Predict all joint parameters in JSON format, including type, origin, axis, parent, and equator plane rendening
distributed sphenically rendering
Figure 4: SAPIENS Simulator Rendering Strategies
A.2.2 Single-view Point Cloud Generation
For the more challenging scenario where only a single RGB image of the object is provided, we employ generative approach utilizing LGM (Large Generative Model) (38]. This process involves two main steps. First, we use a pre-trained diffusion model to synthesize a consistent set of multi- view images that plausible renderings of the object from viewpoints around the original single image. Second, we reconstruct the 3D geometry from these synthesized multi-view images. This reconstruction can be initially represented 3D Gaussian Splatting model, which is then converted into the final dense point cloud representation Porj. Example results are shown in Figure 5
Regardless of whether the point cloud is generated from multi-view images via DUSt3R or from single view via LGM and diffussin-bbssed synthesis, the output of this stage is a dense point cloud Potj IR Nx6 where each point is represented by its 3D spatial coordinates and RGB color value (XYZRGB). It is important to reiterate that this generated point cloud Pot provides a holistic, monolithic representation of the entire object geometry; does not intrinsically contain part-level segmentation information.
A.3
3D MLLM's input and output Design Details
Input Prompt Design
The input to our 3D MLLM consists of processed 3D point cloud features (passed to the model via designated token, e.g., <point>) and text instruction. To enhance the robustness and generalization capability of the model, we utilize several different text instruction templates during training. These templates vary in the level of detail provided about the object and its parts (e.g., including object category, number of parts, detailed description of objects) but consistently specify the desired task: predict joint parameters and segmentation masks a structured format. portion of these templates include the "number of parts," primarily serving as a pedagogical tool or strong supervisory signal ease the fitting pressure and help the model learn the correlation between an object's 's visual form and its internal articulated structure more effectively.
A representative example of general input instruction template structure is:
22

---

child. Segment each link in JSON format.
An instance of a concrete input prompt generated from template for specific object Faucet in this case) used in our experiments is shown below. Note that the full input to the MLLM would include the point cloud features encoded prior to this text sequence. During Inference, providing the number of parts is entirely optional. For all quantitative evaluations in the paper and for fair comparison with other models, we uniformly used generie prompt that did not contain any information about the number of parts or joints (e.g., "Please segment the object and predict its joint parameters").
This articulated object Faucet consists 4 parts. Predict all joint parameters in JSON format, including type, origin, axis, parent, and child. Segment each link JSON format.
Output Format Design
The MILM is trained to autoregressively generate its output structured text in JSON format. This output design critical simultaneously specifies the predicted kinematic structure and provides explicit signals for geometric segmentation. The output JSON contains two primary keys:
"joints": This key maps to a list, where each element represents a predicted joint. Each joint object includes the essential URDF parameters: "id", "type" (e.g.. "revolute", "pris- matic", "fixed"), "parent" link name, "child" link name, "origin" (containing "xyz" position and "rpy" orientation), and "axis" vector.
"links": This key maps to object that provides information about the predicted links. Specifically, it maps each predicted link name (c.g.. "link 0", "link 1") to the predicted semantic category of the part followed immediately by the special [SEG] token (c.g.. "switch [SEG] "). The presence and position of the [SEG] token in the output sequence is the explicit signal generated by the MLLM that triggers and guides the geometric segmenta- tion process for the corresponding link in the input point cloud (as detailed in Section 3f the main paper).
A complete example of the MLLM's generated output in JSON format for the Faucet instance, corresponding to the input prompt shown above, provided below:
"joints":
iid": "joint_0",
"type": "revolute",
"parent": "base"
"child": "link_0",,
"origin":
"xyz": [-0.079, -0.48747, -0.0],
"rpy": [1.5708, -0.0, 1.5708]
},
"axis": [0.0, 1.0, 0.0],
"limit": {"lower": 0, "upper": 1.57}
},
{
"id": "joint_1" 1", "type": "revolute", "parent": "base",
"child": "link_1", "origin": {
"xyz": [-0.079, 0.49568, -0.0], "rpy": [1.5708, -0.0, 1.5708]
},
"axis": [0.0, -1.0, 0.0],
"limit": {"lower": 0, "upper": 1.57}
},
23

---

{
"id": "joint_2",
"type": "continuous", "parent": "base", "child": "link_2",
"origin":
"xyz": [-0.079, 0.00411, -0.0],
"rpy": [1.5708, -0.0, 1.5708]
},
"axis": [o.0, 1.0, 0.0]
},
{
"id": "joint_3" "type": "fixed",
"parent": "base".
"child": "link_3",
"origin":
"xyz": [0.0, 0.0, 0.0],
"rpy": [1.5708, 0.0, 1.5708]
},
"axis": [1.0, 0.o, 0.o]
}
],
"links": {
"link_0" 0 "switch [SEC]
"link_1" 1": "switch [SEG] ",
"link_2" "spout [SEG]
"link_3": "faucet_base" [SEG]
}
}
A.4 Shape Reconstruction Quality
To quantitatively evaluate the geometric fidelity of the final mesh outputs- -ncpuut-inccluing the point-to- mesh conversion stage- we compute the Chamfer Distance (CD) between generated meshes and the ground-trun meshes from PartNet-Mooillit. We compare against two strong baselines that also produce explicit mesh outputs. The results, shown in Table clearly demonstrate the superiority of our method in shape reconstruction quality.
Table 6: Comparison of Shape Reconstruction Quality (Chamfer Distance).
Method
ALL
ID
OOD
CARTO
1.24
0.88
1.27
PARIS
3.06
2.17
3.13
URDF-Anything (Ours)
1.39
0.40
1.51
A.5 Failure Case Analysis
We analyze failure cases from the physical executability evaluation (22% failure rate overall). As shown in Tabld7 the vast majority of failures (21%) stem from incorrect kinematic parameters-sut as misaligned joint origins or axes-leading to non-physical motion (e.g., parts colliding or detaching). Only 1% of failures are due to invalid JSON format, confirming our MILM reliably generates valid syntax. To provide a more detailed error analysis for physical executability, we have conducted deeper failure analysis, shown in Table g
24

---

Input Image
1
MVDream 4 Views Image
d
2oa
Point Cloud
Figure 5: LGM: Point Cloud Generation via Multi-view Synthesis
Input Image
Point Cloud
Figure 6: Multi-view Point Cloud Generation using DUSt3R.
A.6 Comparison with Alternative Methodologies
For completeness, this appendix provides a targeted comparison against alternative methods like CARTO 30 and PARIS 32). which address more constrained problems under different assumptions.
an optimization-based method requiring images of start/end states. Its per-instance optimization is slow at inference (>3min), making it impractical for many applications.
PARIS
CARTO 30 is fast feed-forward model, but it is highly specialized for single-joint objects and cannot perform part segmentation or generate a complete, executable URDE
Tables 9 and 10 quantify the trade-offs in speed, accuracy, and capability. While CARTO is faster, its simplicity results in high prediction errors and extremely. PARIS's optimization-be approach also struggles to find physically plausible solutions, leading to poor accuracy and executability (25%).
25

---

image
point cloud
segmentation
URDF
Figure 7: Step-by-Stee Reconstruction from Real-World Image to Functional URDE
Table 7:
Breakdown of Physical Executability Failures.
Failure Type
Percentage
Incorrect joint parameters
21%
JSON Format Error
1 1
Overall Failure
22%
In contrast, URDF-Anythin's MLLM-based design provides robust balance, achieving vastly superior accuracy and high executability rate (90%) with a practical feed-forward inference time [13s). This result validates our design choice for tackling complex, general-purport reconstruction.
A.7 Zero-Shot Sim-to-Real Generalization Evaluation
To assess the real-world applicability our framework, we conducted a zero-shot sim-to-rea evaluation. Our model, trained exclusively the simulated PartNet-Mobill dataset, tested directly on the real-world portion of the PARIS dataset [3]. This dataset provides real-world captures and annotations for two main categories: Fridge and Storage.
The quantitative results of this evaluation are presented Table
These results provide several key insights into the model's sim-to-real transfer capabilities. First, our method achieves reasonable performance on geometric tasks (segmentation mloU and mesh CD) and continuous parameter prediction (axis/origin), despite the significant domain gap between synthetic training data and real-world images, For the Fridge and Storage categories present in the dataset, the model correctly identified the joint type in all cases.
A.8 More Ablation
Impact Context Fusion Segmentation. We also analyzed the impact of our proposed context fusion mechanism for the [SEG] token. We compared our approach, which fuses the hidden states of the [SEG] token and its preceding category token, against baseline that uses only the generic hidden state of the [SEG] token (hweg). Table 12 shows that our context fusion design achieves a clear improvement segmentation mloU, confirming effectiveness in providing part-aware context for fine-grained segmentation.
26

---

<?xml version=
<robot name= "UNNAMND_NAMMMNN
<!- base link --I> <link name="base"A <link name="link_0"
<visual>
'geometry>
<mesh oitoramee//iikk..ooomnnmmmmmmaaa
</geometry>
</visual>
<collision>
sgeometry>
<mesh Tiieeeeeeeiiiiiioooooooeeeeeeeee
</geometry>
</collision>
</link>
d-- link 1 --b>
<link name="link_1" 1">
<visual>
<orígin 0.0000.000.00..0000 0,7 0.2
<geometry> <mesh iicerameeeiiikkLLommmmeememmmemi
</geometry>
</visual>
<collision>
<origin
0,22.....
<geometry>
<mesh fflerrmmeeliikk...mmmmmmmmmmoo
s/geometry>
</collision>
</link>
<-- joint --
<joint type="fixed" name="joint_00 00>>
<parent link="base//>
<child inn="linkk00"/>
<origin rpy="1.57 0.0 1.57" 0.0000..0..000000 <axis 0
</joint>
<l- <l--joint 1--b>
<joint type="revolute" nome="
<parent link="base"/>
<child link="link_1"/>
<origin -p. -00 xyz="-yz1 0.67 -0.22%>
<axis 1,000-1000000000'
<limit lower="-1.83" upper="-0.0" effort="O.0" velocity="0.0"/
</joint>
</robot>
Figure 8: Example Generated URDF File
27

---

Table Breakdown of Physical Executability Failures by Category.
Joint Error ( (%)
Category
Total (%)
JSON Format Error (%)
Type
Axis
Origin
Window 15.5
0.00
0.03
0.05
0.07
Chair Globe
22.2
0.01
0,06
0.06
0.09
14.8
0.00
0.03
0.05
0.07
Table 9: Comparison of Inference Speed and Methodology.
Method
Core Methodology
Avg. Inference Time
CARTO 30
Feed-forward -ncoder-Decoder
1s
PARIS 32
Per-instance Optimization
>3min
URDF-Anything (Ours)
MLLM Inference
13s
Table 10: Comparison of Reconstruction Accuracy and Physical Executability. Our method
delivers substantially lower prediction errors and much higher rate of generating functional URDFs.
Method
mloU
CD
Type Error
Axis Error
Origin Error
Executability
CARTO 30
1.24
0.12
PARIS 32
0.44
3.06
0.25
0.84
0.30
25%
CRDF-Anything(Our))
0.69
1.39
0.007
0.121
0.13
90%
Table 11: Zero-Shot Sim-to-Real Performance on the PARIS Real-World Dataset. The model was trained only on simulation data and tested on real-world images.
Category
mloU
CD
Type Error
Axis Error
Origin Error
Fridge 0.57
1.03
0.0
0.335
0.256
Storage
0.56
0.99
0.0
0.362
0.349
Table 12: Ablation: Context Fusion Mechanism for Segmentation.
Segmentation Mechanism
Query Feature
mloU
Generic only
useg hontegory h beeg
0.58
Context Fusion (Ours)
heombined
0.63
A.9 Analysis of the Joint Segmentation and Kinematics Learning Mechanism
A critical aspect of our framework is the tight coupling between geometric segmentation and kinematic parameter prediction, facilitated by the '[SEG]' token and end-to-end joint joint optimization. This section aims demystify this process, explaining it as form of "Geometric Regularization" and providing empirical validation.
Theoretical Explanation: Geometric Regularization via Joint Optimization The core principle behind this synergy is that the segmentation task (Izeg) provides powerful physical anchor for the kinematic prediction task (Ltewt). A model predicting only kinematics might hallucinate joint that is textually plausible but physically baseless. Our joint optimization forces the model to ground its abstract kinematic predictions in the concrete geometric entities of point cloud. This geometric grounding effectively constrains the parameter search space, guiding the model toward more physically consistent articulation structures. This is enabled by the shared representation in our end-to-end model, which forces the hidden states used for both tasks to be mutually informative.
28

---

Experimental Validation qualitative evidence.
To empirically validate this theory, we provide both quantitative and
Quantitative Ablation. We trained a variant of our model without the '[SEG] token and its associated segmentation loss (Isseg) As shown in Table "Kinematics-Only removes the geometric regularization from the segmentation task, leading to a consistent increase in prediction error across all kinematic parameters. This quantitatively demonstrates that joint segmentation crucial for achieving high-fidelity kinematic inference.
Qualitative Visualization. To provide intuitive insight into the "black-box" mechanism, visualized the self-attention maps from a key transformer layer while the model generated a kinematic parameter token (e.g.. 'axis"). As illustrated in Figure 9 the full model's attention is sharply focused on the physically relevant joint region (the hinge). In contrast, the model trained without segmentation exhibits diffuse, unfocused attention. This visualization provides direct evidence that joint optimization strategy forces the model to ground abstract kinematic concepts in concrete geometric features, demystifying the learning process.
(a)
(b)
Figure 9: Attention Visualization for Kinematic Token Generation. (a) Our full model, trained with joint segmentation, concentrates its attention (yellow areas) on the relevant joint region when predicting the joint 'axis". (b) The model trained without the segmentation loss shows diffuse attention, lacking precise geometric grounding.
29


Reviews:
[{"review_id": "UjJLYVEVDa", "reviewer": "Reviewer_xYGo", "summary": "The paper proposes an Multi-modal LLM based pipeline for estimating the URDF of articulated objects. The input is a single or multi-view RGB images from which pointcloud is extracted using off the shelf methods like Dust3r or LGM as well as text. The output of MMLM is a SEG token and kinematic parameters. Segmentation per point is decoded via SEG token using a dedicated deocder. Relevant experiments are conducted and the method is compared to various baselines like Real2Code and URDFformer.", "questions": "Please see weakness section for questions. I am looking forward to seeing the author's response in the rebuttal.", "limitations": "Mentioned in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "In my opinion, below are the strengths of this work:\n\n1. A neat integration and usage of MMLMs for articulated object segmentation and kinematic parameter prediction. \n\n2. The method works for arbitrary number of parts in an object which is a major benefit, whereas some of the other methods are restricted to one or two parts. \n\n3. Significant improvement in quantitative results when compared to relevant baselines. \n\n4. The paper is nicely written, easy to follow and diagrams complement the text well. \n\nBelow are the weakness of the method and my questions to authors:\n\n1. I am curious why the method doesn't discuss or compare to feedforward approaches like CARTO [1] or PARIS line of work [2,3] which require multi-view images of start and end state. I am wondering how does the proposed method compares to these approaches interms of speed, accuracy etc. If there is a trade-off for instance the output of autoregressive MMLMs + decoder network are slow compared to feed-forward category-level methods, that should be clearly stated etc. \n\n2. What is considered Out of Distribution instances and how are they determined? Is only the size varied b/w In-distribution and OOD?\n\n3. Is the method category-level or works for novel-classes as well?\n\n4. Lack of real-world evaluation is a limitation. PARIS [2] provides GT annotations which can be used to report real-world metrics, I wonder why did the authors not consider reporting those metrics or on a more challenging sim dataset with more categories [3]\n\n5. While the qualitative results are nice, It is hard to judge the shape reconstruction quality without a more concrete metric like Chamfer distance.\n\n[Minor]\n\n1. Can the method be easily extended to other articulated objects like robot arm etc.?\n\n2. Can the method be solely trained in simulation and transferred to real-world without additional training or finetuning?\n\n\n[1] CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects, CVPR 2023\n[2] PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects, ICCV 2023\n[3] SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splatting, ICCV 2025", "was_revised": true, "final_justification": "The rebuttal answered my questions and I am happy to increase the score from 4 to 5. The revised paper would benefit from all the discussions/new results. When reporting real-world results on the PARIS dataset i.e. W-4, it would be great to add other methods which have also reported results on the same dataset for a fair apples-to-apples comparison to those i.e. PARIS baseline among others. \n\nI think the author should consider releasing the model/code and is looks like it would be beneficial to the community.", "rebuttal": "We sincerely thank the reviewer for their positive and encouraging feedback. We are particularly grateful for your recognition of our method's technical strengths—including its neat integration of MLLMs and its ability to handle objects with an arbitrary number of parts—as well as our significant quantitative results and the clarity of the paper.\n\n## W1: Comparisons\n\nThank you for this insightful question, which allows us to clarify our work's positioning relative to other important lines of research. We did not focus our comparison on methods like CARTO or the PARIS line of work because they address a fundamentally different and more constrained problem setting, and possess significant limitations that make a direct comparison inequitable. To clarify:\n\n- **Different Technical Routes & Inherent Flaws**\nThe primary reason for the separate comparison is that these methods follow different technical routes with inherent flaws that our work is designed to overcome.\n\n    - PARIS[2] represents a completely different approach. It relies on slow, online per-instance optimization, requiring over 3 minutes at test time. This is in stark contrast to our fast, feed-forward inference model (~13s). As our new experimental results show, this optimization-based method also suffers from high errors and very low physical executability. [3]was posted on arXiv after the NeurIPS submission deadline.\n    - CARTO[1] is a specialized, small-scale feed-forward model with severe input and output restrictions. It is designed only for single-joint objects and cannot perform part segmentation or generate a complete, executable URDF with precise kinematic parameters (axis, origin). This inherent lack of capability and poor generalization to complex objects makes it unsuitable for the general-purpose reconstruction task our work addresses.\n\n- **Quantifying the Differences in Speed and Capability**\nTo illustrate these fundamental differences, we conducted new experiments. The tables below quantify the critical trade-offs and highlight the limitations of these alternative approaches.\n\nTable 1: Comparison of Inference Speed and Methodology\n| Method | Core Methodology | Average Inference Time |\n|:-:|:-:|:-:|\n| URDF-Anything (Ours) | Feed-forward MLLM Inference | 13s |\n| CARTO | Feed-forward Encoder-Decoder | 1s |\n| PARIS | Per-instance Optimization | >3min |\n\nTable 2: Comparison of Reconstruction Accuracy and Physical Executability\n| Method | mIoU | CD | Type Error | Axis Error | Origin Error | Physics Executability Rate (%) |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| URDF-Anything (Ours) | 0.69 | 1.39 | 0.007 | 0.121 | 0.130 | 90 |\n| CARTO | - | 1.24 | 0.12 | - | - | - |\n| PARIS | 0.44 | 3.06 | 0.25 | 0.84 | 0.30 | 25 |\n\nWhile our MLLM-based approach has a modest speed trade-off compared to a simple model like CARTO(as shown in Table 1), this is precisely what enables us to overcome these limitations, delivering vastly superior accuracy, full-featured URDF generation, and the ability to generalize to complex, novel objects(demonstrated in Table 2). We will add this detailed comparison and discussion to our final paper.\n\n## W2: The definition of \"Out-of-Distribution\" (OOD)\n\nThat is an excellent question for clarifying our experimental setup. In our study, an Out-of-Distribution (OOD) instance is defined as an object belonging to a **category that was entirely unseen** during the model's training phase. We determine the In-Distribution (ID) and OOD splits through a strict, category-based partitioning of the dataset to rigorously evaluate the model's true generalization capabilities.\nSpecifically, to ensure a fair and consistent comparison, we adopted the experimental setup used by prior works such as Articulate-Anything and Real2Code. The **In-Distribution (ID)** set comprises five core object categories: 'Laptop', 'Box', 'Refrigerator', 'StorageFurniture', and 'Table'. Our model is trained and validated only on instances from these categories. The **Out-of-Distribution (OOD)** set consists of all instances from the remaining 41 distinct object categories in the dataset, which are completely held out from training.\nTherefore, the distinction between our ID and OOD sets is not merely a variation in object size. It is a fundamental difference in the **object category** itself, which encompasses distinct geometries, functionalities, and articulation structures. This setup is designed to test our model's ability to generalize to entirely novel classes.\n\n## W3: Generalization Scope\n\nOur method is explicitly designed to generalize to **novel classes**, leveraging a 3D MLLM backbone with robust geometric priors to reason effectively about unseen objects.\nWe systematically validate this capability in **Section 4 (Experiments)**. As demonstrated in **Tables 1, 2, and 3 in our paper**, our method's superior performance on Out-of-Distribution (OOD) objects across all key metrics—including segmentation accuracy, kinematic parameter precision, and physical executability—confirms the powerful generalization capabilities of our framework.\n\n\n## W4: Lack of Real-World Evaluation\n\nThank you for this constructive suggestion regarding real-world evaluation.\nFirst, regarding the suggestion to use the dataset from SplArt [3], we would like to clarify that this work was posted on arXiv after the NeurIPS submission deadline, so we were unable to include it in our original comparisons.\nRegarding the PARIS [2] dataset, it consists of two parts: a real-world set and a simulated set. Their simulated data is also derived from PartNet-Mobility, which is the primary dataset used in our work. However, to directly address your valuable point about real-world evaluation, we have conducted **new experiments on the real-world portion of the PARIS dataset** during this rebuttal period. This dataset is limited to two main categories: Fridge and Storage.\nThe quantitative results of our method, presented in Table 3, are as follows:\n\nTable 3: Zero-Shot Sim-to-Real Performance on the PARIS Real-World Dataset\n| | mIoU | CD | Type Error | Axis Error | Origin Error |\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| **Fridge** | 0.57 | 1.03 | 0 | 0.335 | 0.256 |\n| **Storage** | 0.56 | 0.99 | 0 | 0.362 | 0.349 |\n\nThese results(Table 3) provide several key insights. First, our method achieves reasonable performance on geometric and continuous-valued tasks (segmentation, mesh reconstruction, and predicting joint axis/origin) in a zero-shot, sim-to-real setting. We attribute this to the challenging sim-to-real domain gap.\nThis experiment provides a valuable and realistic baseline for zero-shot sim-to-real transfer on this complex task. It highlights that while our framework transfers well for geometry, robustly classifying kinematic types in the wild without any real-world fine-tuning remains a challenging open problem. We will include this full real-world evaluation and discussion in the revision of our paper.\n\n## W5: Quantitative Shape Reconstruction Metric\n\nThank you for this excellent suggestion. We agree that a quantitative metric for shape reconstruction quality is crucial for a complete evaluation.\nTo address this, we have conducted a new experiment during the rebuttal period to measure the geometric fidelity of the final output meshes using the **Chamfer Distance (CD)**, as you recommended. We compared our method against strong baselines that also produce explicit mesh outputs.\nThe results, shown in Table 4, clearly demonstrate the superiority of our method in shape reconstruction quality:\n\nTable 4: Comparison of Shape Reconstruction Quality (Chamfer Distance)\n| Method | CD |\n| :-: | :-: |\n| URDF-Anything (Ours) | 1.39 |\n| CARTO | 1.24 |\n| PARIS | 3.06 |\n\n## Minor 1: Extensibility to other objects\n\nYes, in principle, our framework can be extended to other categories of articulated objects, such as robot arms. The core of our method—the autoregressive generation of a flexible, structured JSON output—is general-purpose and capable of describing complex kinematic chains.\nHowever, our current model is trained on the PartNet-Mobility dataset, which primarily features furniture and household items. To achieve optimal performance on a specialized domain like robot arms, which have distinct visual features and structural priors, we would recommend fine-tuning the model on a domain-specific corpus to adapt to these new patterns.\n\n## Minor 2: Sim-to-real transfer\n\nYes, our method is designed to be trained solely in simulation and then deployed directly in the real world without fine-tuning.\nThis is achieved through a decoupled architecture. Our pipeline uses a powerful, pre-trained perception front-end (e.g., DUSt3R, LGM) to convert any image (sim or real) into a unified point cloud representation. Our core 3D MLLM then operates exclusively on this geometric data. This design ensures that the sim-to-real domain gap (e.g., lighting, textures) is primarily handled by the perception module, while our core model is shielded from pixel-level discrepancies and focuses on the underlying geometry, which transfers well.\n\n[1] CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects, CVPR 2023 [2] PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects, ICCV 2023 [3] SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splatting, ICCV 2025"}, {"review_id": "lujqQJCGr0", "reviewer": "Reviewer_euKz", "summary": "This paper presents URDF-Anything, a novel end-to-end framework that leverages 3D Multimodal Large Language Models (3D MLLMs) to automatically reconstruct articulated object digital twins in URDF format from visual observations (images or point clouds). The method introduces a specialized [SEG] token mechanism to jointly predict object segmentation and kinematic parameters. Experiments on the PartNet-Mobility dataset show substantial improvements in segmentation accuracy, kinematic prediction precision, and physical executability over prior baselines.", "questions": "See weaknesses.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ The paper addresses an important and underexplored problem in robotic simulation: automatic articulated object modeling and URDF file generation.\n\n+ The framework is end-to-end, combining geometry understanding and kinematic reasoning.\n\n+ Clear reproducibility, and good experimental results compared to former works.\n\n\nWeaknesses:\n- While the use of the [SEG] token for reasoning-based segmentation is intuitive and well-motivated in the LISA framework, its extension in this paper raises certain concerns. In the context of this work, applying [SEG] to guide 3D point cloud segmentation is understandable, since the token can serve as a semantic anchor to align text-based part labels with geometric features. However, what is less clear—and somewhat black-box in nature—is how the model is able to simultaneously infer accurate joints using the same mechanism. The paper lacks a concrete explanation or ablation showing how the [SEG] token contributes to learning the kinematic structure, as opposed to just part segmentation. It almost appears as if the model “magically” learns to associate geometry with articulation semantics, without clearly defined inductive biases or intermediate supervision. A deeper analysis or visualization would greatly help demystify this process.\n\n- This paper lacks sufficient ablation studies to support its methodological choices. Most components are fixed without justification or comparison. In particular, the choice of Uni3D as the 3D backbone is taken for granted, but alternative backbones should have been evaluated for comparison. A more thorough experimental analysis, similar to what was done in the paper Articulate-Anything, would significantly strengthen the paper. That work serves as a good reference for the types of diagnostic or ablation studies that could be conducted. While I will not list them all here, following a similar level of evaluation would be beneficial for this paper.\n\n- My key concern is that this is a compositional work that primarily integrates existing ideas rather than introducing novel concepts. Each component of the paper—whether it's the use of 3D multimodal large language models (3D MLLMs), LoRA for fine-tuning, the incorporation of [SEG] tokens, point cloud generation, mesh reconstruction, or structured output formatting—is based on prior methods. While the system as a whole may function effectively, there is limited novelty in the individual technical contributions. The framework appears to repackage existing tools and techniques into a cohesive pipeline rather than pushing the boundaries of any specific subfield. Please let me know if the authors have any arguments regarding the novelty of this work.", "was_revised": true, "final_justification": "My concerns have been well addressed during the rebuttal. As a compositional work, my primary concern was the novelty. The authors have convincingly demonstrated that their framework is technically non-trivial during the rebuttal. The integration of components is the result of careful design, and, to the best of my knowledge, their framework is the first to introduce the point cloud modality for LLM-based URDF generation. While additional ablation studies could further strengthen the work, I believe the proposed framework represents a meaningful contribution to this less-explored field.\n\nI recommend accepting the paper, as the authors have committed to releasing the model and code, which will be highly beneficial for the community.", "rebuttal": "We sincerely thank the reviewer for their positive feedback. We are particularly encouraged by your recognition of the importance of the problem we are addressing, the integrated nature of our end-to-end framework, and our strong experimental results and clear reproducibility.\n\n## W1: Unclear Mechanism of the [SEG] Token for Kinematics\nThank you for this insightful and critical question. In our paper (lines 47-50), we described this as a \"tight coupling\" that \"ensures full consistency between predicted kinematics and reconstructed geometry\". We would like to take this opportunity to \"demystify\" this mechanism with both a theoretical explanation and new experimental validation.\n\n**1. Theoretical Explanation:  Geometric Regularization via Joint Optimization**\nOur model does not learn this association \"magically\"; the core principle is Geometric Regularization, an effect that emerges from our end-to-end joint optimization.\nEssentially, the segmentation task ($L_{seg}$) provides a powerful physical anchor for the kinematic prediction task ($L_{text}$). A model predicting only kinematics might hallucinate a joint that is textually plausible but physically baseless (e.g., a rotation axis outside the object). Our joint segmentation task forces the model to ground its abstract kinematic predictions (e.g., a rotation axis) in the concrete geometric entities of the point cloud. To accurately segment a door, the model must implicitly learn where its center of rotation and axis should be. This geometric grounding effectively constrains the parameter search space, leading to more physically consistent and accurate articulation structures.\n\nThis is enabled by the Shared Representation in our end-to-end model. Because both tasks share most network parameters and are optimized via a unified loss function ($L = λ_{text}*L_{text} + λ_{seg}*L_{seg}$), the model must learn an internal representation effective for both. This means the hidden states used to generate kinematic parameters must also contain information useful for the subsequent geometric segmentation, thus tightly coupling them in the latent space.\n\n**2. Experimental Validation: From Ablation to Visualization**\nTo empirically validate this theory, we will add two key experiments to our revised manuscript to demonstrate how the [SEG] token facilitates kinematic learning:\n- **Ablation Study: Quantifying the Contribution of Segmentation.**\n    - **Design:** We will train a baseline model where the [SEG] token and its associated segmentation loss ($L_{seg}$) are removed. This model will predict kinematic parameters using only the language modeling loss ($L_{text}$). We then compare its parameter accuracy against our full model.\n    - **Results:** As shown in Table 1,  without the geometric regularization from the segmentation task, this baseline has higher error in its kinematic predictions. This result quantitatively proves that the segmentation task is crucial for learning accurate kinematic structures.\n\nTable 1: Quantifying the Contribution of Segmentation.\n| Method | Type Error | Axis Error | Origin Error |\n| :-: | :-: | :-: | :-: |\n| URDF-Anything w/ seg | 0.008 | 0.132 | 0.164 |\n| URDF-Anything w/o seg | 0.009 | 0.138 | 0.175 |\n\n- **Attention Visualization: Opening the \"Black Box\".**\n    - **Design:** To visually inspect how the model grounds its kinematic predictions, we visualized the self-attention distribution from a specific layer of the two model versions in our Ablation Study: our full model (URDF-Anything w/ seg) and the baseline without the segmentation task (URDF-Anything w/o seg). This allowed us to observe which parts of the input point cloud the models focus on when generating a kinematic parameter token (e.g., 'axis').\n    - **Results:** The visualizations (which can't be shown here and we will add to the revision) show that when generating kinematic tokens, our full model's attention is highly concentrated on the physically relevant joint regions of the point cloud (e.g., hinges, sliders). In contrast, the attention of the model trained without the segmentation loss is far more diffuse and unfocused.\n    - **Significance:** This visualization provides direct, intuitive evidence at a \"neural level\" of how our joint training forces the model to associate abstract kinematic concepts with concrete geometric features. It demystifies the process, showing that the connection is not \"magic\" but a learned behavior enforced by the geometric regularization of the segmentation task, directly addressing your concern about the \"black-box\" nature of the mechanism.\n\nWe will incorporate these new experiments, including the visualizations, into our revised manuscript. Thank you again for pushing us to clarify this crucial aspect of our work.\n\n\n\n## W2: Insufficient Ablation Studies\n\nWe sincerely thank you for your valuable suggestion regarding the ablation studies.\n\nWe would like to take this opportunity to clarify the core focus of our current work. Our primary objective is to be the first to demonstrate and validate the feasibility and superiority of a novel, unified 3D MLLM framework for the task of articulated object reconstruction. Through this, we aim to help guide the field's evolution from traditional multi-stage or optimization-based paradigms toward a more efficient and versatile **end-to-end large model paradigm.** Based on this core objective, our experimental design (including the ablation study in Section 4.4) prioritized validating the most fundamental hypothesis of this new paradigm: that **multimodal inputs combining explicit 3D point clouds and text instructions** are crucial compared to relying solely on 2D images or pure geometric information. We believe that in the initial stages of establishing a completely new framework, proving the correctness of its core design philosophy is more critical than optimizing one of its replaceable components.\n\nHowever, we also agree that conducting more comprehensive diagnostic experiments would better demonstrate the superiority and robustness of our unified MLLM joint prediction architecture. Therefore, we have supplemented our work with a new preliminary experiment. Specifically, we replaced the original Uni3D backbone with a classic alternative, PointNet++, while keeping all other parts of the model (such as the 3D MLLM, the [SEG] token mechanism, and the training strategy) unchanged.\n\nTable 2: Ablation Study on 3D Backbone\n\n| Method | mIoU ALL ↑ | mIoU ID ↑ | mIoU OOD ↑ | Count Acc ALL ↑ | Count Acc ID ↑ | Count Acc OOD ↑ |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| **ours (uni3d)** | 0.63 | 0.69 | 0.62 | 0.97 | 0.99 | 0.96 |\n| **ours (pointnet++)** | 0.57 | 0.66 | 0.56 | 0.96 | 0.99 | 0.95 |\n\nOur preliminary results show the following:\n\n**Validation of Our Choice:** When using PointNet++ as the backbone, the model's overall performance sees a slight decrease compared to using Uni3D. This, to some extent, validates our initial choice of Uni3D as a powerful and suitable feature extractor.\n\n**Demonstration of Framework Robustness:** Despite the slight performance drop, this new PointNet++-based variant still significantly outperforms all baseline methods compared in our paper across all key metrics.\n\nThis strongly demonstrates that the success of our framework is not merely dependent on a single specific SOTA component; its core advantage stems from the effectiveness of our proposed unified MLLM joint prediction architecture. This result also reaffirms that there is ample room for future research to conduct more detailed component-level comparisons and optimizations (e.g., exploring more efficient backbones) upon our framework to further enhance performance.\n\n\n\n## W3: Limited Novelty / Compositional Work\n\nWe appreciate the opportunity to clarify our perspective on the novelty of URDF-Anything.\n\nWe respectfully argue that the novelty of our work lies not in inventing a single, isolated algorithm, but in successfully conceiving, implementing, and validating a **new end-to-end paradigm** for articulated object reconstruction. Within this new paradigm, we performed crucial, non-trivial adaptations of existing technologies to solve challenges unique to this domain.\n\nPreviously, the field relied on fragmented, multi-stage pipelines with simplified inputs, which led to error accumulation and physically inconsistent models. Our unified paradigm solves this by processing complete 3D point clouds to **jointly optimize geometry and kinematics in a single pass.** The success of this fundamental shift is demonstrated by a >50% improvement in physical executability and superior accuracy over piecemeal approaches (Table 2 in paper), validating its superior robustness.\n\nThis paradigm is not a simple composition; it required key technical innovations and has pushed the boundaries of the subfield:\n\n**Technical Novelty:** We didn't just reuse the [SEG] token; we fundamentally adapted it. Our context-fusion mechanism (merging category and [SEG] token information) provides the essential part-level semantic understanding required for this complex, single-object deconstruction task.\n\n**Pushing Boundaries:** Our most significant contribution is a leap in generalization. By leveraging a 3D MLLM, we shift the field from dataset-specific models to those that can generalize to entirely new object categories. This is proven by our strong performance on out-of-distribution (OOD) objects (Tables 1-3 in paper), a crucial advancement for the field.\n\nIn essence, our work provides the first robust \"existence proof\" for applying 3D MLLMs to this domain. We believe establishing this powerful and effective new baseline is a significant contribution that opens new avenues for future research."}, {"review_id": "7eoxu2MdWr", "reviewer": "Reviewer_KtPh", "summary": "This paper proposes URDF-Anything, a method that, given a single or multiple views of an object, is able of segmenting its parts and predicting the joint parameters of the parts that can be moved (e.g., the hinge of a door). The method is tested on an extensive benchmark and reports metrics on part segmentation and joint prediction as well.", "questions": "1. [43-44]: \"In addition, we introduce a novel dynamic [SEG] token mechanism [...]\". From the paper content, I struggle to see the novelty in this mechanism, as it was introduced by LISA [1] and later adopted by various methods for similar tasks [2,3]. It should be discussed more in detail how the paper modify this token mechanism with respect to prior work.\n2. About the physical executability, is there an error anlysis about the sources of error? E.g., percentage of cases in which the URDF is not correctly formed, and percentage in which the format is correct, but the joints are wrong\n3. Tab1: \"count accuracy\" is not defined in the text\n4. Tab2: what is \"oracle\"?\n5. Tab2: Why are Articulate-Anything class split not reported?\n6. [473-484]: from this example, it seems that the network relies on prior knowledge about the object, i.e. the fact that it has N joints. Is this correct? If it is, this is an important assumption for the method, and it should be clearly stated in the paper.\n\n[1] Lai, Xin, et al. \"Lisa: Reasoning segmentation via large language model.\" CVPR 2024.\n[2] Huang, Kuan-Chih, et al. \"Reason3d: Searching and reasoning 3d segmentation via large language model.\" 3DV 2025.\n[3] He, Shuting, et al. \"Segpoint: Segment any point cloud via large language model.\" ECCV 2024.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The task is significant due to its application to real-world scenarios\n- The method is simple, and the paper organization explains it very clearly\n- The results clearly explain the efficacy of the method, particularly the ones on the physical executability.\n\nWeakness\n\n- The novelty of the paper is unclear, it would be helpful to briefly summarize the contributions at the end of the introduction section. As one contribution the authors state (at 44) “a novel dynamic [SEG] token mechanism”, which is the one used in LISA [1]. Moreover, in 3D this technique has already be used in [2,3], to name a few. The novelty in its use is not clear to me (see question 1).\n- The paper is lacking some explainations on the modality used during evaluation (see questions 3-4-5-6)\n\n[1] Lai, Xin, et al. \"Lisa: Reasoning segmentation via large language model.\" CVPR 2024.\n\n[2] Huang, Kuan-Chih, et al. \"Reason3d: Searching and reasoning 3d segmentation via large language model.\" 3DV 2025.\n\n[3] He, Shuting, et al. \"Segpoint: Segment any point cloud via large language model.\" ECCV 2024.", "was_revised": true, "final_justification": "The authors clarified all the main points I raised in my review. After considering also the results provided to other reviewers requests (e.g., the additional ablation on the 3D network architecture, and the context fusion mechanism), I raise my recommendation to Borderline accept. I did not go for a full Accept as I still find the [SEG] token contribution a bit weak considered how it was previously used in the literature. Nonetheless, the paper is solid and the task and results are significant.", "rebuttal": "We sincerely thank the reviewer for their positive assessment. We thanks for your recognition of our task's real-world significance, the clarity of our method and paper organization, and the efficacy of our results, especially the physical executability metric.\n\n## W1 & Q1: Novelty & Re-use of the [SEG] Token \n\nThanks for your valuable feedback. \n\n## Primary contribution\n\nOur work's primary contribution is not a minor technical improvement, but rather the introduction of a fundamental **Paradigm Shift** to the field of articulated object reconstruction.\nPrior to our work, the prevailing paradigm in this field suffered from several inherent limitations. It typically relied on multi-stage, error-prone pipelines—for instance, using one model to segment parts and another to predict parameters. At the input stage, to simplify the problem, these methods either depended on representations with significant loss of geometric detail (e.g., OBBs) or were limited by 2D images, making it difficult to infer precise 3D structures. On the modeling front, they either employed specialized, small-scale models lacking world knowledge and generalization capabilities, or adopted methods requiring slow, per-instance online optimization (such as those based on Gaussian splatting), which are difficult to scale.\nOur work champions and implements **a new paradigm of end-to-end joint reasoning**. We are the first to demonstrate that a unified 3D Multimodal Large Language Model (3D MLLM) can directly take a **complete 3D point cloud** as input and, through **joint optimization**, produce a functional model that is both **geometrically precise and kinematically consistent** in a single, end-to-end pass. This new paradigm leverages the powerful prior knowledge and reasoning capabilities of large models to fundamentally address the problems of information loss, error accumulation, and poor generalization that plagued traditional methods. The most direct evidence of its effectiveness is that our model achieves a **greater than 50% improvement in Physical Executability**, directly validating the robustness and superiority of this new paradigm.\n\n## Innovative application of the [SEG] token\n\nThe innovation lies in being the first to fundamentally adapt and extend its application and mechanism for end-to-end articulated model reconstruction. This is primarily reflected in the following three core aspects, which also represent the essential differences from prior works like SegPoint and Reason3D:\n\n- **Operating Scale:** SegPoint and Reason3D both operate at the scene level. In contrast, our URDF-Anything focuses on fine-grained part-level analysis.\n- **Function of Segmentation:**  In our framework, segmentation plays another critical role--**A Booster:** Through end-to-end joint training, the precise segmentation task in turn provides strong geometric constraints for the prediction of kinematic parameters, thereby improving their accuracy. As shown in our ablation study (Table 1 below), removing the segmentation task leads to a significant increase in kinematic prediction error, validating its \"booster\" effect.\n- **Implementation Mechanism:** Both SegPoint and Reason3D directly use the hidden state of the [SEG] token ($h_{seg}$) to guide the segmentation decoder. We have made a key improvement upon this: we fuse the hidden state $h_{seg}$ with the hidden state of its immediately preceding \"category token\" (e.g., \"drawer\"), $h_{category}$, resulting in $h_{combined} = [h_{seg} ; h_{category}]$. This design injects strong, part-aware context into the segmentation process, which is crucial for distinguishing between multiple visually similar parts on the same object. We validated this through a dedicated ablation study (Table 2 below), which shows our context fusion design achieves superior segmentation accuracy.\n\n\nTo experimentally support the above claims, we conducted two key ablation studies:\n\nTable 1: Validating the \"Booster\" Effect of Segmentation on Kinematic Prediction\n| Method | Type Error ↓ | Axis Error ↓ | Origin Error ↓ |\n| :-: | :-: | :-: | :-: |\n| URDF-Anything w/ seg (Full Model) | 0.008 | 0.132 | 0.164 |\n| URDF-Anything w/o seg (No Seg) | 0.009 | 0.138 | 0.175 |\n\nTable 2: Validating the Superiority of the Context Fusion Mechanism\n| Segmentation Mechanism | mIoU ↑ |\n| :-: | :-: |\n| Context Fusion (Ours) | 0.63 |\n| Generic $h_{seg}$ only | 0.58 |\n\nFinally, we will revise the end of our introduction to clearly summarize our core contributions, including: (1) Proposing the first end-to-end 3D MLLM framework for articulated object reconstruction, championing a new paradigm from complete 3D input to joint prediction output; (2) Achieving deep coupling and joint prediction of kinematic parameters and geometric segmentation through an innovative application of the [SEG] token; (3) Demonstrating the superiority of this new paradigm through extensive experiments. And we add citations to these relevant works [1, 2, 3] in our revised paper.\n\n\n\n## W2: Explanation on Input Modality\n\nWe thank the reviewer for pointing out that the explanations regarding the input modalities for evaluation could be clearer. We agree and appreciate the opportunity to clarify this here and in revision.\n\nAs the reviewer noted, we have provided more details in our answers to the specific Q 3, 4, 5, and 6.\n\n## Q2: Error Analysis on Physical Executability\n\nThanks for this  question. As your suggestion, we have conducted a detailed analysis of the failure cases for our method. The results are as follows, break down the 22% overall failure rate observed in Table 3.\n\nTable 3: Breakdown of failure cases for URDF-Anything on the physical executability task.\n| Overall Failure| JSON Format Error | Incorrect Joint Parameters |\n|:-:|:-:|:-:|\n| 22% | 1% | 21% |\n\nThis analysis reveals that the vast majority of failures (**21%** out of 22%) are due to the prediction of incorrect or physically implausible joint parameters (e.g., origin, axis). In contrast, structural errors, such as generating a malformed JSON or an invalid URDF file, are extremely rare (**1%**).\n\nThis breakdown confirms that our MLLM-based framework is highly proficient at learning the correct syntax and structure for URDF generation. The primary remaining challenge lies in the precise quantitative prediction of kinematic parameters, not in generating a valid file format.\n\nDue to space constraints, we did not include this analysis in the initial submission. We will add this table and the corresponding discussion to the experimental section in the revision to provide a more comprehensive evaluation.\n\n## Q3: \"Count Accuracy\" Metric\n\nThank you for pointing out this omission. You are correct, we should have defined this metric in the text. **\"Count Accuracy\"** measures the percentage of test samples for which the number of predicted articulated parts (i.e., links with non-fixed joints) exactly matches the ground-truth number of parts.\nWe will add this definition to Section 4 (Evaluation Metrics) in the final version of the paper.\n\n## Q4: \"Oracle\" Term\n\nThank you for the question. \nThe \"Oracle\" setting in Table 2, a protocol from the Articulate-Anything paper, provides baselines with ground-truth part segmentations. This is to fairly evaluate their kinematic prediction module in isolation, by removing errors from their preceding segmentation step.\n\n\n## Q5: Baseline Results Split \n\nThank you for pointing out this omission.\nTo clarify our experimental setup, our data split is consistent across the baselines. Our In-Distribution (ID) split consists of the five categories shown in our supplementary materials: \"StorageFurniture\", \"Laptop\", \"Box\", \"Table\", and \"Refrigerator\". The remaining 41 object categories from the PartNet-Mobility dataset constitute our Out-of-Distribution (OOD) split.\n\n## Q6: Concern about Reliance on Prior Knowledge\n\nThank you for the question. The example you saw in the appendix is indeed one of the diverse templates we used during training, but it does not mean our method must rely on this prior knowledge at inference time. We apologize for any potential misunderstanding the appendix may have caused and wish to offer a detailed clarification here.\nOur approach is divided into two phases: training and inference.\n\n**During Training**: We designed highly diverse prompt templates to enhance the model's robustness. A portion of these templates did include the \"number of parts,\" primarily serving as a pedagogical tool or a strong supervisory signal to ease the fitting pressure and help the model learn the correlation between an object's visual form and its internal articulated structure more effectively. However, our training data also contained numerous generic templates that did not provide this information, which compelled the model to learn to infer the number of parts autonomously from visual and geometric features, rather than relying solely on the prompt.\n\n**During Inference (Testing)**: Providing the number of parts is entirely optional at inference time. It is crucial to emphasize that for all quantitative evaluations in the paper and for fair comparison with other models, we uniformly used a generic prompt that did not contain any information about the number of parts or joints (e.g., \"Please segment the object and predict its joint parameters\").\n\nWe will revise the appendix to feature a simpler object JSON as an example for better clarity."}, {"review_id": "FARDPsEARu", "reviewer": "Reviewer_qimd", "summary": "This paper introduces URDF-Anything, a framework for reconstructing articulated 3D objects as URDF models directly from visual input, using a 3D Multimodal Large Language Model (MLLM). The authors first convert single-view or multi-view RGBs to point-cloud via off-the-shelf DUStr and LGM. Then, they train a 3D multimodal Large Language Model (3D MLLM) to take the point-cloud and some text prompt to perform part segmentation, link hierarchy and joint prediction.", "questions": "- How is object category (e.g., \"faucet\") and number of parts determined at inference time? Can your model infer this from the point cloud, or does it rely on annotations (as in PartNet-Mobility)?\n- Please provide a breakdown of performance by object category (e.g., cabinet, laptop, faucet) and a failure case analysis. This would help assess robustness and generalization.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The idea of using a 3D MLLM for articulation is novel and interesting.\n- Ablation study 4.4 provides valuable insight, highlighting the benefits of explicit 3D input over 2D image-based alternatives.\n- The experimental results show gains in segmentation, joint prediction accuracy, and simulation executability.\n\n**Weaknesses**\n- The related work section is imprecise and at times inaccurate. \n\t- \"Research on [...] is a related but distinct field.\" is not accurate and confusing. Many works under this section are directly relevant to this work. In fact, the authors themselves compare against these methods as baselines.\n\t- \"[29] focuses on generating articulated simulation scenes from images, which is related but distinct from accurate individual object\" is not accurate. URDFormer can also generate individual objects. In fact, the authors did compare against this baseline.\n- The point cloud generation pipeline is unnecessarily complicated. The authors use DUSt3R for multi-view input and LGM for single-view, but don’t explain why a unified approach (e.g., always using LGM or DUSt3R after synthesized views) isn’t employed. This suggests convenience-driven decisions rather than principled design.\n\t- For example, they could hallucinate single-view to multi-view via off-the-shelf model (ImageDream) then use DUSt3R. Alternatively, they can also scrape off DUSt3R and just use multi-view posed RGBs directly with LGM.\n- Some important aspects of the system’s training and inference remain ambiguous:\n\t- The \"Uni3D w/ text\" baseline is underspecified. What exact text input is used? The generic prompt \"Segment the input point cloud into parts and predict articulation parameters.\" shown in Fig. 2 seems insufficient for meaningful conditioning.\n\t- It is unclear how \"Uni3D w/o text\" and \"Uni3D w/ text\" perform segmentation. Do they involve a classifier head trained on predefined categories? If I understand correctly, in the Uni3D paper, they use contrastive-style loss to do segmentation. So during inference, they likely provided a list of semantic part names that need to be segmented.\n\t- A more intuitive and technical explanation is needed to justify why the [SEG] mechanism outperforms \"Uni3D w/ text\", especially since both involve feature-text fusion at different levels.\nEvaluation choices raise some concerns:\n\t- Line 208 defines a custom “joint success” criterion but omits reporting success rate as defined in prior work [1]. Why?\n\t- Table 3 (physical executability) omits Real2Code, even though it's included in other comparisons.", "was_revised": true, "final_justification": "The authors have addressed all my concerns during the rebuttal", "rebuttal": "We sincerely thank the reviewer for their positive feedback. We are particularly encouraged by your recognition of the novelty of our core idea, the valuable insights from our ablation study, and the strong experimental gains across segmentation, joint prediction, and simulation executability.\n\n## W1: Related Work Section\n\nThank for pointing out the imprecisions in our Related Work section. We will make the following corrections:\n\nRewrite \"Research on [...] is a related but distinct field.\" to \"The automated modeling of articulated objects for robotic manipulation is an active field of research, with several distinct methodological approaches.\"\n\nRewrite \"[29] focuses on generating articulated simulation scenes from  images, which is related but distinct from accurate individual object\" to \"[29] is fundamentally limited by its reliance on a hard-coded system to assign kinematic parameters and retrieve meshes based on the network's discrete part classification output, a method which compromises final geometric and kinematic fidelity.\"\n\n## W2: Input Pipeline\n\nThanks for in-depth thinking on our input pipeline design. We fully agree that the design should be principled rather than based on convenience.\n\nOur core principle is to use the most suitable, highest-fidelity method available for each type and quality of input source. This ensures that the geometric information received by our core model is of the highest possible quality. The two input scenarios (real multi-view vs. single-view) are fundamentally different problems in nature and difficulty:\n\n**For multi-view input (using DUSt3R)**: When real, high-quality multi-view images are available, they contain the most reliable geometric information. In this case, the most direct and high-fidelity approach is to use a specialized multi-view 3D reconstruction algorithm. We chose DUSt3R because it is a classical method in this specific domain (reconstructing geometry from real multi-view images). Forcing the use of a generative method like LGM in this scenario would actually degrade the quality of the point cloud fed to our core model by introducing artifacts and uncertainties. It is unnecessary to \"downgrade\" real data to generated data before processing.\n\n**For single-view input (using LGM)**: When only a single-view image is available, the problem fundamentally changes from 'reconstruction' to 'conditional generation' or 'hallucination'. We must leverage a generative model to complete the missing 3D information. We chose LGM because it is a classical solution for this specific problem (generating 3D from a single view).\n\nRegarding your suggestions for a unified pipeline:\n\n**On using \"ImageDream + DUSt3R\"**: This insight actually points to the internal workings of models like LGM. Advanced single-view generative models such as LGM already integrate modules similar to MVDream/ImageDream for synthesizing multiple views from a single one. Therefore, using the highly integrated and optimized LGM is a more rational and efficient choice than manually assembling a similar pipeline.\n**On using \"LGM in the multi-view case as well\"**: As mentioned before, this would mean applying a tool designed for \"generation\" to a \"reconstruction\" problem, which is not the optimal choice.\n\n\n## W3\n\n### W3.1 & W3.2: Uni3D Baselines / Uni3D\n\nHere we clarify the implementation and purpose of our two key baselines:\n**‘Uni3D w/ text’** simulates text-guided segmentation. Following the Uni3D paper, we freeze the backbone and train a feature propagation layer to align local point features with text features from a provided list of part names (e.g., ['furniture body', 'laptop base']). At inference, each point is assigned the label of the text prompt with the highest feature similarity.\n\n**‘Uni3D w/o text’** is a supervised closed-set method designed to test the raw quality of the pre-trained geometric features. Inspired by classic methods like PointNet, we add an MLP head to the frozen Uni3D backbone and train only this head to classify points into a predefined set of all part categories from complete dataset(including both ID and OOD objects). This process requires no text input.\n\nThe purpose of these baselines is twofold:\nThe poor performance of ‘Uni3D w/o text’ demonstrates that relying on purely geometric features is insufficient and proves the necessity of semantic guidance for this task. Its weakness stems from a paradigm mismatch, as it discards the rich, cross-modal semantic alignment capability Uni3D was pre-trained for.\n\nBy outperforming ‘Uni3D w/ text’, we show that our [SEG] token mechanism is a superior guidance strategy. Unlike static text prompting, our method dynamically and end-to-end couples the segmentation signal with the autoregressive generation of the kinematic structure, leading to more accurate and context-aware results.\n\n### W3.3: Justification for [SEG]'s Superiority\n\nThe superiority of our [SEG] token mechanism over the baseline stems from two fundamental differences:\nFirst, our [SEG] token is a **dynamic and context-rich signal**, unlike a static word used in the ‘Uni3D w/ text’ baseline. It is generated by the LLM after the model has already processed both the **complete point cloud geometry** and the **preceding kinematic parameters**(e.g., parent-child links, joint types) within the autoregressive sequence. Therefore, the token’s embedding is inherently informed by both geometric and structural context, making its guidance for segmentation far more precise than a simple class label.\nSecond, our training paradigm **learns a joint probability distribution over both kinematic structure and geometry**. Through end-to-end optimization, the generation of a plausible structure and the successful segmentation of its corresponding geometry are tightly coupled. This is fundamentally different from the baseline's decoupled approach, where segmentation is merely a post-hoc classification task. In essence, our method transforms segmentation from simple matching into a context-aware grounding problem, leading to superior accuracy.\nTo prove the crucial role of our [SEG] token mechanism, we have conducted new ablation studies. We summarize the key experiments and findings below:\n1. **Segmentation-Only**: This model only outputs [SEG] tokens, trained solely for part segmentation and does not generate any kinematic structure. This experiment validates the necessity of \"context\" and \"joint training\".\n2. **Geometry-Blind**: In this model, the LLM is given ground-truth kinematic structure but has no access to the input point cloud when attempting to output segmentation commands. This experiment validates the necessity of \"geometric features\" for the [SEG] token.\n\nResults are as follows:\n\n| Model | Segmentation Accuracy (mIoU ↑) | Count Accuracy (Count Acc ↑) |\n| :-: | :-: | :-: |\n| Uni3D w/ text | 0.54 | 0.84 |\n| Our Full Model | 0.63 | 0.97 |\n| | | |\n| Ablation 1: Segmentation-Only | 0.61 | 0.96 |\n| Ablation 2: Geometry-Blind | ~0.1 (Near-random) | - |\n\nThe performance of the \"Segmentation-Only\" model is slightly lower than that of the full model, indicating that kinematic structural context and joint structure-geometry training are beneficial for achieving high-precision segmentation.\nThe \"Geometry-Blind\" model completely fails the segmentation task, decisively proving that the **LLM's direct perception of geometric features** is the cornerstone of the [SEG] mechanism's functionality.\n\n### W3.4: Omission of Standard \"Joint Success Rate\" Metric\n\nThanks for pointing this out. Due to the length constraints of rebuttal, please refer to Table 1 in our response to reviewer KtPh, for a complete breakdown of the joint success rate.\n\n### W3.5: Omission of Real2Code in Table 3 / Table \n\nThe reason for its initial absence is that the official LLM model checkpoint and training code for Real2Code have not been released. To provide this missing result, we have now completed the evaluation by reproducing the Real2Code pipeline ourselves. We have updated Table 3 with the physical executability result from this implementation.\n\n| Method | ALL | ID Classes (Simple) | ID Classes (Complex) | OOD Classes (Simple) | OOD Classes (Complex) |\n| :- | :-: | :-: | :-: | :-: | :-: |\n| Real2Code | 41% | 55% | 37% | 34% | 19% |\n\n## Q1: Determination of Object Properties at Inference\n\nThank you for the question. Our model infers both the object category and the number of parts directly from the input point cloud, without requiring annotations.\n**For the object category**: As shown in Section 3.3 and our Appendix, our model's output directly includes semantic part categories (e.g., \"link_0\": \"base_cabinet[SEG]\"). This demonstrates that the model implicitly understands the overall object category by deconstructing it into its constituent parts during the generation process. Separately, one could also directly query our ShapeLLM backbone for the object's category, as this is an inherent capability from its pre-training.\n**For the number of parts**: This is not an input but an outcome of the autoregressive process. The model generates the structure joint-by-joint and implicitly learns when to stop, determining the final number of parts by the length of the predicted joint list.\n\n## Q2: Performance Breakdown and Failure Analysis\n\nThank you for the question. To provide a more detailed error analysis for physical executability, we have conducted a deeper failure analysis. Due to the rebuttal's length constraints, we selected a representative subset of 3 Out-of-Distribution (OOD) categories (Window, chair, Globe) for this analysis. The results are showed below:\n\n| Category | Failure Rate (%) | Json format Error | Joint Type Error | Joint Axis Error | Joint Origin Error | Subjective Failure Rate (%) | \n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Window | 15.5 | 0 | 0.03 | 0.05 | 0.07 | 17.2 | \n| chair | 22.2 | 0.01 | 0.06 | 0.06 | 0.09 | 23.5 | \n| Globe | 14.8 | 0 | 0.03 | 0.05 | 0.07 | 18.0 |"}, {"review_id": "Weu3JhLxW5", "reviewer": "Reviewer_7j6V", "summary": "The paper presents URDF-Anything, an end-to-end pipeline that turns RGB observations (single- or multi-view) into functional URDF parameters of articulated objects. A 3D multimodal LLM (ShapeLLM-7B + LoRA fine-tuning) jointly (i) segments a dense point-cloud into links and (ii) predicts joint parameters. The key technical novelty is a dynamic [SEG] token that the model inserts while autoregressively generating the JSON-style URDF: each [SEG] token’s hidden state is used, via cross-attention, as a query to produce a per-point mask for the corresponding link.", "questions": "See above", "limitations": "It would be better to show some failure cases.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper is well-written.\n- To my knowledge, the first work to leverage a 3D MLLM with explicit token-based cross-modal segmentation for simultaneous geometry + kinematics prediction; elegant integration of language generation and point-level supervision.\n- Comprehensive pipeline—from point-cloud creation (DUSt3R / LGM) through [SEG] cross-attention to mesh fusion—explained clearly.\n- Strong quantitative gains on both ID and OOD splits; executability metric moves beyond static accuracy; ablations on modality and [SEG] tokens are informative.\n- Produces ready-to-simulate URDFs, a strong link to robotics/embodied AI workflows.\n\n**Weaknesses**\n- Results lack error bars or significance tests; variance is only given for one table.\n- The pipeline still relies on an external point-to-mesh post-process; errors there are not evaluated.\n- Training on ShapeLLM-7B with LoRA is resource-heavy; training time is not reported.\n- Mass, inertia, friction and joint damping are omitted; executability counts loading & actuation but not quantitative dynamics accuracy.", "was_revised": true, "final_justification": "revised score", "rebuttal": "We sincerely thank the reviewer for their encouraging and positive assessment. We are especially grateful for your acknowledgment of the paper's clarity, the novelty of our technical approach in using a 3D MLLM for joint prediction, and the elegance of the integration. Furthermore, we appreciate your recognition of our strong quantitative results, particularly on OOD data, and the practical value of our ready-to-simulate URDF outputs for the robotics community.\n\n## W1: Lack of Error Bars\n\nWe sincerely thank the reviewer for this constructive feedback. We agree that providing a thorough analysis of result variance is crucial for demonstrating the robustness of our method.\n\nWe have now calculated the standard deviations for our key metrics, which we are happy to provide for clarification.\n\nTable 1: Quantitative Results for Part-Level Link Segmentation.\n\n| Models | mIoU ALL ↑ | mIoU ID ↑ | mIoU OOD ↑ | Count Acc ALL ↑ | Count Acc ID ↑ | Count Acc OOD ↑ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| Uni3D w/o text | 0.36±0.01 | 0.50±0.02 | 0.33±0.01 | 0.73±0.02 | 0.83±0.03 | 0.70±0.01 |\n| Uni3D w/ text | 0.54±0.02 | 0.64±0.01 | 0.51±0.02 | 0.84±0.02 | 0.91±0.02 | 0.82±0.02 |\n| **URDF-Anything ([SEG])** | **0.63**(16.7%↑)±0.01 | **0.69**±0.01 | **0.62**±0.01 | **0.97**(15.4%↑)±0.02 | **0.99**±0.02 | **0.96**±0.02 |\n\nTable 3: Physical Executability Rate (%↑).\n\n| Method | ALL | ID Classes (Simple) | ID Classes (Complex) | OOD Classes (Simple) | OOD Classes (Complex) |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| URDFormer Oracle | 24%±8.24% | 40%±4.53% | 23%±5.30% | 27%±5.67% | 11%±11.62% |\n| Articulate-Anything | 52%±12.63% | 71%±6.24% | 42%±9.49% | 69%±11.70% | 35%±14.72% |\n| **URDF-Anything(Ours)** | **78%**(50.0%↑)±1.51%  | **90%** ±1.20% | **78%** ±0.87% | **83%** ±1.56% | **67%** ±1.58% |\n\nTable 4: Ablation Study on Input Modality for Joint Parameter Prediction.\n\n| Method Variant | Input Modality | Type ↓ | Axis ↓ | Origin ↓ |\n| :--- | :--- | :---: | :---: | :---: |\n| OBB | Text | 0.42±0.18 | 0.70±0.26 | 0.47±0.22 |\n| Point Cloud only | Point Cloud | 0.34±0.11 | 0.29±0.18 | 0.26±0.15 |\n| Qwen2.5-VL-7B | Image + Text | 0.57±0.14 | 0.85±0.31 | 0.23±0.16 |\n| Qwen2.5-VL-7B + ft | Image + Text | 0.38±0.09 | 0.81±0.24 | 0.18±0.10 |\n| **Point Cloud + Text (Ours)** | Point Cloud + Text | **0.008**±0.001 | **0.132**±0.486 | **0.164**±0.260 |\n\nWe will incorporate these updated tables with full variance reporting into the revision of our paper. Thank you again for your valuable suggestion, which has helped improve the rigor of our work.\n\n## W2: Post-processing Step\n\nThank you for highlighting this important evaluation aspect. To address your point, we have conducted a new experiment to quantitatively evaluate the final mesh output from our pipeline, including the point-to-mesh post-processing stage.\nWe measured the geometric fidelity using the standard **Chamfer Distance (CD) metric**. Results are as follows. We will add this evaluation to the revision.\n\n| Method | ALL | ID Classes | OOD Classes |\n| :- | :-: | :-: | :-: |\n| Ours | 1.39 | 0.40 | 1.51 |\n\n\n\n## W3: Unreported Training Resources\n\nThank you for pointing out this omission. Our model was fine-tuned in \n**2.5 hours on a single NVIDIA A800 (80GB) GPU**. We will add these details to the revision of the paper and make the code open source.\n\n\n## W4: Lack of Dynamic Parameters\n\nWe thank the reviewer for this insightful question, which allows us to clarify the well-defined scope of our work. Our primary focus is on a central challenge in robotic manipulation: reconstructing an object's **geometry and kinematic structure** from visual input.\n\nThis focus **is consistent with the established scope** of other state-of-the-art methods in this domain. Prominent works, including Articulate-Anything, Real2Code, and URDFormer, also concentrate exclusively on geometric and kinematic parameters. The estimation of dynamic properties (mass, inertia, friction, etc.) from vision alone is a distinct and highly challenging research problem, which the community typically treats separately.\n\nWe argue that providing a kinematically accurate model is, in itself, a highly valuable contribution. An accurate kinematic model alone is sufficient to **meet the requirements of a vast range of robotic applications**, including manipulation planning, grasping, and task simulation. In many real-world scenarios, a robot's own controllers (e.g., through impedance or admittance control) **can adapt to an object's unknown dynamics** during interaction, making the kinematic model the most critical component for planning and execution.\n\nTherefore, our contribution is a robust and complete solution to the core kinematic reconstruction problem. While incorporating dynamics is an exciting future direction, our work establishes the essential structural foundation upon which such dynamic models could eventually be built.\n\n## Limitations: Failure Case Analysis\n\nThank you for this valuable suggestion. We will add a dedicated section to the appendix in our revision to include and discuss visualizations of typical failure cases."}]

Author rebuttal:

Meta-review: This paper develops an end-to-end 3d Multimodal-LLM pipeline that jointly performs point-cloud segmentation and prediction of kinematic parameters to generate URDF models. Overall the paper and proposed approach is likely to be impactful with strong (practical!) empirical performance and generalization to out-of-distribution categories.  The rebuttal significantly improved the clarity of various aspects of the paper and added various improvements, including ablations, (unseen) attention visualizations, and a small zero-shot evaluation (the real-world portion of the PARIS dataset), resulting in several reviewers increasing their scores. The authors are expected to incorporate these aspects into their revised paper as well as address the various points raised in the reviews. Models and code should also be released, as promised.  There are a few very minor remaining concerns, as included in the reviews and discussion, but the contribution overall is solid.
Decision: Accept (spotlight)