Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model

## Abstract

Abstract Interactive 3D simulated objects are crucial in AR/VR, animations, and robotics, driving immersive experiences and advanced automation. However, creating these articulated objects requires extensive human effort and expertise, limiting their broader applications. To overcome this challenge, we present Articulate-Anything , a system that automates the articulation of diverse, complex objects from many input modalities, including text, images, and videos. Articulate-Anything leverages vision-language models (VLMs) to generate code that can be compiled into an interactable digital twin for use in standard 3D simulators. Our system exploits existing 3D asset datasets via a mesh retrieval mechanism, along with an actor-critic system that iteratively proposes, evaluates, and refines solutions for articulating the objects, self-correcting errors to achieve a robust outcome. Qualitative evaluations demonstrate Articulate-Anything ’s capability to articulate complex and even ambiguous object affordances by leveraging rich grounded inputs. In extensive quantitative experiments on the standard PartNet-Mobility dataset, Articulate-Anything substantially outperforms prior work, increasing the success rate from 8.7–12.2% to 75% and setting a new bar for state-of-the-art performance. We further showcase the utility of our generated assets by using them to train robotic policies for fine-grained manipulation tasks that go beyond basic pick and place.

## 1 Introduction

One of the most promising avenues in the quest for generally capable robots is the pursuit of scaling robot training in simulation for deployment in the real world. We have developed fast, accurate simulators that scale to millions of FPS and hundreds of GPUs (Xiang et al., ; Makoviychuk et al., ), enabling policy learning on a staggering scale. However, a critical bottleneck in this research direction persists: the immense human labor required to construct realistic, *interactable* environments for these agents to learn within. Despite the existence of large, open libraries of static object geometries — with the largest open dataset containing over 10 million objects (Deitke et al., ) — we have comparatively minuscule open libraries of *articulated* 3D objects (only around 2,300 objects (Xiang et al., )). This scarcity stems from the time-consuming, labor-intensive, and expertise-demanding nature of the manual annotation process.

To address this challenge, we present Articulate-Anything, a novel approach in automatic articulation that harnesses the power of leading foundation vision-language models (VLMs) to articulate a diverse range of objects of arbitrary complexity through iterative feedback (Fig. 1). Articulate-Anything represents a step function improvement in quality, accuracy (8.7-12.2% to 75%), and generalizability over prior art (Chen et al., ; Mandi et al., ), overcoming previous limitations that restricted success to only a narrow range of object categories and joint types. Unlike prior art, which has been limited by the impoverished input of bounding boxes or static images, Articulate-Anything affords the flexibility of consuming rich, grounded inputs from text, images, or even videos, enabling users to request exotic articulation descriptions or resolve articulation ambiguities. For example, the right column of Fig. 6 features a digital model of a window that could plausibly slide or tip to open; when Articulate-Anything is shown an in-the-wild video demonstration, it accurately produces the desired sliding motion.

To achieve this level of flexibility and accuracy, Articulate-Anything employs an actor-critic system with two core components: (1) a vision-language actor that synthesizes high-level Python code, which can be compiled into Unified Robot Description Format (URDF) files and (2) a vision-language critic that provides feedback on the rendered prediction compared against available ground-truth. The result is an agentic system that can automatically self-evaluate and iteratively improve the articulation of complex objects. Beyond robotics, the flexibility of Articulate-Anything’s inputs married with its high-quality outputs puts automatic generation of rich, high-quality, and diverse virtual environments within reach with broad-reaching applications to 3D/VR (Kim et al., ), human-computer interaction (Jiang et al., ), and animation (Yang et al., ).

Our key contributions include:

- 1.
Articulate-Anything: We present a vision-language actor-critic system that accurately articulates objects from diverse input modalities, including texts, images, and videos.
- 2.
Articulation as program synthesis: We develop a high-level Python API compilable into URDFs, enabling the VLM actor to generate compact, easily debuggable programs.
- 3.
Visually grounded inputs enable closed-loop iterative refinement:
We highlight the critical role of grounded visual inputs in articulating ambiguous and complex objects, leveraging such inputs in a closed-loop system that self-evaluates and self-improves its articulations.
- 4.
Extensive evaluation demonstrates superior performance: Prior works in automatic articulation are only evaluated on a handful of object categories. Quantitative analysis on the entire PartNet-Mobility dataset shows dramatic improvement over existing methods, increasing the success rate from 8.7–12.2% of prior work to 75%.

## 2 Related Work

3D Asset Datasets. Large open datasets of 3D objects, such as 3D Warehouse and Objaverse (Deitke et al., ), provide extensive collections of 3D models uploaded by independent artists. However, most objects in these repositories lack articulation. Datasets like ShapeNet (Chang et al., ) and PartNet (Mo et al., ) decompose objects into parts (links) but do not include kinematic joints. Some datasets, such as PartNet-Mobility (Xiang et al., ) and GAPartNet (Geng et al., ), do feature articulated objects with links and joints, but these annotations are manually created so their number of objects is modest. Our work aims to automate the creation of articulated 3D assets suitable for robotic tasks, reducing the reliance on manual annotation.

Articulated Object Modeling. Articulated object modeling is a significant area of research, encompassing tasks such as perception, reconstruction, and generation (Liu et al., ). In perception, works like (Zeng et al., ; Hu et al., ) focus on identifying and understanding articulated objects. Reconstruction efforts from single RGB images (Chen et al., ), RGBD images (Weng et al., ), multi-view images (Liu et al., ), or point clouds (Jiang et al., ) aim to rebuild articulated models. Generation methods, including those leveraging connectivity graphs (Liu et al., ) or neural approaches (Lei et al., ), focus on creating new articulated objects. Our goal is to develop an end-to-end pipeline that spans from perception to reconstruction and generation, utilizing intuitive and easily accessible inputs from humans, such as language or videos, instead of relying on more specialized modalities, such as point clouds or graphs.

## 3 Problem Formulation

To ensure compatibility with standard 3D simulators, we represent 3D models in the Unified Robot Description Format (URDF). In URDF, an object is structured in a hierarchical tree consisting of nodes (links) and edges (joints). The links represent parts of an object (e.g., drawers and doors of a kitchen island), and joints specify how each part moves. The two most common joints are prismatic, representing a translational movement (e.g., a sliding drawer), and revolute, representing a rotation (e.g., a pivoting door).

Given an input $x$ of text, image, or video depicting an object, we assume there exists a ground-truth URDF $u^{*}$ (e.g., one that a human could create). The goal of articulation is to automatically construct a URDF model $\hat{u}$ that comprises the same set of links as $u^{*}$ while minimizing the difference between URDF models via the following loss:

$$ $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{mesh}}+\mathcal{L}_{\text{link}} +\mathcal{L}_{\text{joint}}\enspace,$ (1) $$

where $\mathcal{L}_{\text{mesh}}=\sum_{i=1}^{N}\text{Chamfer}(\hat{P}_{i},P^{*}_{i})$
measures the discrepancy in visual appearances between the predicted and ground-truth links using point clouds $\hat{P}_{i}$ and $P^{*}_{i}$. $N$ is the number of links. $\mathcal{L}_{\text{link}}$ captures 3D pose differences between predicted and ground-truth links:

$$ $\mathcal{L}_{\text{link}}=\sum_{i=1}^{N}\underbrace{\|\mathbf{\hat{x}}_{i}- \mathbf{x^{*}}_{i}\|_{2}}_{\text{position error}}+\underbrace{2\arccos(|\hat{q }_{i}\cdot q^{*}_{i}|)}_{\text{orientation error}}\enspace\mbox{,}$ (2) $$

where $\mathbf{x}_{i}$ is a 3D coordinate and $q_{i}$ is a quaternion. $\mathcal{L}_{\text{joint}}$ quantifies the joint discrepancies.

$$ $\mathcal{L}_{\text{joint}}=\mathcal{L}_{\text{joint type}}+\mathcal{L}_{\text{ joint axis}}+\mathcal{L}_{\text{joint origin}}+\mathcal{L}_{\text{joint limit} }\enspace,$ (3) $$

where $\mathcal{L}_{\text{joint type}}$ is a cross entropy loss, $\mathcal{L}_{\text{joint axis}}$ is the angular difference between axes, $\mathcal{L}_{\text{joint origin}}$ is the 3D pose difference between joint origins, and $\mathcal{L}_{\text{joint limit}}$ measures the difference in range and direction of motion. In practice, jointly optimizing this objective becomes intractable. Consequently, we instead tackle each loss sequentially, decomposing the problem into three phases: (1) Mesh Retrieval, (2) Link Placement, and (3) Joint Prediction. Furthermore, access to $u^{*}$ is often limited, making traditional optimization methods for computing these losses impractical. To overcome this, we leverage vision-language model (VLM) agents to approximate the loss functions directly from visual inputs. Additionally, instead of refining solutions through a conventional optimizer, VLM agents are also used to propose solutions in the form of code and iteratively refine them based on feedback from a critic module. This process is detailed in the following sections.

## 4 Articulated Object Generation via Actor-Critic VLMs

In this section, we introduce Articulate-Anything, a system capable of generating articulated objects from various input modalities. Figure 2 presents an overview of our method, which includes three main components: (1) Mesh Retrieval (Sec. 4.1), which reconstructs the 3D structure for each object part by retrieving meshes from a 3D asset library, (2) Link Placement (Sec. 4.2), which spatially arranges the parts, and (3) Joint Prediction (Sec. 4.3), which determines the potential kinematic movements between parts. Additionally, we describe an optional step, targeted affordance extraction, in Sec. 4.4. Our system consists of many specialized VLM agents. Each agent is a Google’s Gemini Flash-1.5 visual language model (Team et al., ) prompted with no more than 20 in-context examples to perform different tasks. All system and task instruction prompts are provided in the source code.

Figure: Figure 2: Method Overview. Given a text, image or video input, Articulate-Anything operates in three stages: (1) Mesh Retrieval (Sec. 4.1) retrieves a mesh for each object part from a 3D asset library, (2) Link Placement (Sec. 4.2) places the parts together, and (3) Joint Prediction (Sec. 4.3) predicts the allowed kinematic movements between parts. Optionally, instead of generating all possible kinematic joints, we can target a specific joint from the input video (Sec. 4.4). The link placement and joint prediction systems consist of an actor and a critic, which are VLMs working together. The actor proposes solutions, and the critic examines those solutions and gives feedback.
Refer to caption: x2.png

Crucially, our system produces high-level Python code compilable into URDFs via a custom API we have developed. Directly generating URDFs is challenging due to (1) the verbosity of URDF/XML, which increases the likelihood of hallucination in long contexts, and more importantly, (2) the need for performing complex mathematics in-place in many cases. For instance, articulating a rotation along the global z-axis for a door requires forward kinematics calculations down the URDF tree to translate it to the relative frame. Even seemingly simple tasks, like positioning a toilet seat above the bowl, can be complicated due to non-trivial mesh geometries. However, this can easily be remedied by performing iterative collision checks in a physics engine.

Figure: Figure 3: Mesh retrieval. The top and bottom diagrams provide overviews for reconstructing visual (i.e., image or video) and text inputs, respectively. For visual input, we match the ground-truth object to a template object in the library using an efficient divide-and-conquer retrieval mechanism. For text input, we first prompt an LLM to predict the different object parts and their dimensions. Then, we retrieve a mesh for each part using precomputed CLIP embeddings and subsequently scale the meshes to specifications. More details in Sec. 4.1.
Refer to caption: x3.png

### 4.1 Mesh Retrieval

We employ two separate mechanisms for reconstructing meshes from visual (i.e., image or video) and text inputs. An overview of these processes is provided in Fig. 3.

Visual Inputs. For visual inputs, we perform mesh retrieval by matching the input to a 3D object from the PartNet-Mobility dataset (Mo et al., ). First, a VLM is instructed to detect an object of interest from the input. Since there might be multiple objects in the frame, when a video is provided, the VLM is instructed to take advantage of motion cues for more accurate identification. Then, to reduce the search space, we identify the top-$k$ most similar object categories in the library using CLIP similarity (Radford et al., ). Lastly, a VLM agent Hierarchical ObjectSelector
is tasked with selecting a candidate among several simulated objects based on their visual similarity to the ground truth. This visual similarity estimate is a proxy for the $\mathcal{L}_{\text{mesh}}$ loss described in Sec. 3. To manage the potentially large candidate pool, we adopt a divide-and-conquer tournament approach. Specifically, we recursively select the best candidate among a batch of max_num_images images until there is only one candidate left.

Text Inputs. For text prompts, we employ a multi-step process using large language model (LLM) agents. First, a TaskSpecifier densifies the potentially sparse prompt (please see Fig. 3 (b) for an example).
Then, a LayoutPlanner is instructed to describe each object part along with its dimensions. Finally, mesh retrieval is performed for each part by matching the object parts’ CLIP language embeddings to the mesh descriptions in the PartNet-Mobility dataset via cosine similarity. Since the original mesh descriptions from the dataset are coarse, we compute our own annotations by prompting a VLM to describe each mesh (e.g., material, shape, function) given its image. The retrieved meshes are finally rescaled to fit the specified dimensions.

### 4.2 Link Placement

The link placement system can handle text inputs containing the semantic names of each part, along with an optional input image. For videos, we extract the first frame as the image input. The system consists of an actor and a critic. The link actor is responsible for placing links together in the 3D space.
When processing visual inputs, a critic is also employed to verify the actor’s solution.
The critic is prompted to describe any visual dissimilarity between the input image and the predicted 3D model rendered in simulation, pinpoint the source of error in the actor’s code, and give a realism_rating between 0 and 10. This realism rating serves as a proxy for the $\mathcal{L}_{\text{link}}$ loss.
The actor is instructed to take the feedback into account and adjust its Python code accordingly. This process terminates when the rating exceeds a threshold of 5.
Figure 4 (Left) provides an illustrative example of this process.

Figure: Figure 4: Both link placement and joint prediction systems consist of an actor and a critic. The actor produces Python code, which is automatically compiled into URDFs and rendered in simulation. Source code, predicted, and input modalities (images for link and videos for joint) are given to the critic for evaluation. The synergy between the actor and critic enables self-correction of errors (red border) and successful articulation (green border).
Refer to caption: x4.png

### 4.3 Joint Prediction

Similar to link placement, the joint prediction system accepts inputs in the form of text containing the Python code for link placement, as well as an optional input image or video. The actor then extends the Python code to include kinematic joint prediction between parts. For video inputs, a critic is also utilized
to verify the actor’s solution. The critic is instructed to compare the input against a video of the predicted joint moving in simulation and examine the source code to identify either success or one of 4 failure cases. A realism_rating between 0 and 10, serving as a proxy for the $\mathcal{L}_{\text{joint}}$ loss, is also used to categorize the most egregious error (incorrect joint type) to the least (incorrect joint limit).
As before, the actor is instructed to rewrite its code, taking the critic’s feedback into account as long as the realism rating falls below a threshold of 5. See an example in Fig. 4 (Right).

### 4.4 Targeted Affordance Extraction

Optionally, instead of generating all possible kinematic joints, we can target a specific joint from the input video. This can be useful to reduce the number of input tokens to VLMs, thereby lowering the cost or making automatic debugging of predictions easier. To achieve this, we prompt a VLM agent targeted_affordance_extractor to determine which child link should be annotated with a kinematic joint given the input video and a segmented image of the simulated asset, where each part is distinctively colored.

## 5 Experiments

Datasets: We use the Partnet-Mobility dataset (Mo et al., ) which includes human annotations for $\sim$2.3K objects, $\sim$1.9K revolute joints, and $\sim$7.6K prismatic joints.

Tasks: Given a ground-truth articulated URDF from PartNet-Mobility, we mask out all link and joint information. The objective is to reconstruct the masked URDF. Solving this objective includes two primary tasks: (1) link placement—determine the spatial arrangement of object parts, and (2) joint prediction—infer the movement capabilities between parts.

Evaluation Metrics:
We assess performance using success rates for each task.

- 1.
Link placement: Success is determined by the pose difference between predicted and ground-truth links falling below a small threshold.
- 2.
Joint prediction: We evaluate differences in joint type, axis, origin, and limit. Success requires all criteria to be within a small threshold.

The position threshold is set to 50mm and the angular threshold to 0.25 radian ($\sim$ 14.3 degree). When evaluating Articulate-Anything, if an object’s link placement is incorrect, all of its joints are also counted as incorrect.
More mathematical details and some visualizations of the different joint errors are available in Appendix A.1.

Baselines: We compare against two prior state-of-the-art methods: URDFormer (Chen et al., ) and Real2Code (Mandi et al., ). Both methods were trained or fine-tuned on five object categories in the PartNet-Mobility dataset. We evaluate the performance of these five (in-distribution) and the remaining 41 (out-of-distribution) classes. URDFormer requires a bounding box for each object part in the input image, which can be obtained via a fine-tuned Grounding DINO (Liu et al., ) or an oracle using ground-truth boxes from a physics engine. Real2Code requires oriented object bounding boxes (OBBs) as input texts to query a LLM. On PartNet-Mobility, these were obtained using oracle RGB-D images and ground-truth segmentation masks from Blender. Implementation details of these baselines are provided in Appendix A.4.

Implementation Details for Articulate-Anything: Our system employs Google’s Gemini Flash-1.5 (Team et al., ) as the vision-language model (VLM). Physics computations are performed using PyBullet (Coumans & Bai, ), while rendering is done in Sapien (Xiang et al., ). Motion traces in videos are annotated using CoTracker (Karaev et al., ) for visualization. We use few-shot prompting with around 20 in-context examples.

### 5.1 How well does Articulate-Anything perform versus prior work?

Figure: Figure 5: Comparison against the baselines. Our approach significantly outperforms all baselines in the joint prediction task, representing a great leap forward in automatic articulation. We use few-shot prompting and make no distinction between in-distribution and out-of-distribution classes, so we only report results for all classes.
Refer to caption: x5.png

We benchmark Articulate-Anything against two prior works in automatic articulation. The key differences between Articulate-Anything and prior work include (1) our system operates on high-level Python abstraction instead of low-level inputs, (2) our system can handle more grounded instruction inputs, including videos, and (3) our actor-critic system allows iterative refinement on more difficult objects. In contrast, URDFormer directly predicts part position (discretized) coordinates, and Real2code operates on oriented bounding box coordinate inputs. Prior works are also limited to simplified inputs as they cannot handle videos. This also means that their articulation must be done in an open loop. Our innovations allow Articulate-Anything to achieve superior performance as demonstrated in Fig. 5. Tab. 1 reveals the raw joint prediction errors behind the success rate of Fig. 5, including errors of joint type, joint origin, and joint axis.

**Table 1: Comparison of average joint prediction errors across methods (lower is better). ‘Type’ shows the fraction of incorrect joint type predictions, ‘Axis’ error is measured in radians, and ‘Origin’ error in meters. Articulate-Anything substantially outperforms all previous works. Articulate-Anything uses few-shot prompting and makes no distinction between ID and OOD classes, so we only report results for All Classes.**
|  | All Classes | ID Classes | OOD Classes |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Type $\downarrow$ | Axis $\downarrow$ | Origin $\downarrow$ | Type $\downarrow$ | Axis $\downarrow$ | Origin $\downarrow$ | Type $\downarrow$ | Axis $\downarrow$ | Origin $\downarrow$ |
| Real2Code Oracle | 0.538 | 1.006 | 0.294 | 0.410 | 1.164 | 0.344 | 0.576 | 0.937 | 0.272 |
| URDFormer Oracle | 0.556 | 0.374 | 0.581 | 0.418 | 0.208 | 0.609 | 0.679 | 0.643 | 0.514 |
| URDFormer DINO | 0.460 | 0.261 | 0.547 | 0.288 | 0.133 | 0.582 | 0.722 | 0.758 | 0.438 |
| Articulate-Anything | 0.021 | 0.141 | 0.200 | - | - | - | - | - | - |

Figure: Figure 6: In-the-wild Reconstruction. We demonstrate Articulate-Anything’s performance input modalities compared to prior works URDFormer and Real2Code. Green and red borders denote correct and incorrect predictions with respect to the input videos. Our simulated videos are generated by varying joint limits and annotated with Cotracker. Casual inputs: Our video-based approach excels with casually captured inputs in cluttered environments while baselines require extensive manual curation (more details in Appendix A.4 Fig. 15 and 16). Ambiguous articulation: Interestingly, our video-based method can resolve ambiguities in static inputs (e.g., chair rotation vs. height adjustment, window sliding vs. tipping). Baseline limitations: URDFormer consistently predicts drawer-like structures and is sensitive to minor misalignments (e.g., slightly tilted drawers). Real2Code, which uses multi-view images for mesh reconstruction and text-oriented bounding boxes (OBBs) for joint prediction, achieves good global alignment from DUSt3R but produces low-quality 3D segmentation using a finetuned SAM, leading to joint prediction errors. Please visit our website for additional demos: [https://articulate-anything.github.io](https://articulate-anything.github.io).
Refer to caption: x6.png

Figure 6 compares Articulate-Anything against the baselines on in-the-wild object reconstruction. As before, we select both OOD and ID objects. Our video-based method performs the best while baselines fail on all tasks, susceptible to minor misalignments or segmentation inaccuracies. Notably, leveraging rich video input allows Articulate-Anything to resolve articulation ambiguities. For example, our non-video methods predict the chair leg sliding up and down (for adjusting height), and the window tipping to open. These predictions are actually reasonable given the input images and texts but turn out to be wrong when the ground-truth videos are revealed.

### 5.2 How do different input modalities affect articulation accuracy?

Figure: Figure 7: Input modality ablation. Performance on articulation tasks improves with more grounded modalities. Videos are only used for joint prediction. Input videos provided to the link placement system are automatically transformed into image inputs by extracting the first frames.
Refer to caption: x7.png

While LLMs trained on internet-scale data possess remarkable common sense abilities, we hypothesize that visual inputs are still crucial for accurate object articulation. For example, consider a simple faucet with a spout and a lever. Where should the lever be placed? In fact, there is no standard answer: there are faucets whose levers are on top and those whose levers are to the side. Joint prediction without videos is similarly difficult (e.g., see Fig. 6). Thus we contend that more grounded input modalities such as videos are crucial in resolving difficult or ambiguous articulation.

In this section, we compare the articulation success rates across three input modalities: text, image, and video. For the text modality, we provide only the semantic part names for link placement; during joint prediction, the model receives the Python code for link placement and is tasked with inferring the joints. With the image modality, we additionally supply a static image of the ground-truth object in addition to the textual information. For the video modality, a video showcasing the object’s motion is provided in addition to the text. Note that videos are only used for joint prediction. Input videos provided to the link placement system are automatically turned into images by extracting the first frames.

We conduct this ablation study on the Faucet and StorageFurniture object categories. We run an actor-only system for one iteration without the critic to isolate the effect of input modalities. Results in Fig. 7 demonstrate that richer input modalities consistently improve success rates, underscoring the importance of visual information in articulation tasks.

Figure: Figure 8: Breakdown of failure percentages in all classes. In Articulate-Anything, incorrect link placement leads to all predicted joints being marked incorrect. For baselines, 59.1% of URDFormer’s and 17.5% of Real2Code’s outputs are invalid, containing cyclic structures, repeated links, or syntax errors. Details are in Appendix A.1.
Refer to caption: x8.png

Figure: Figure 9: Iterative Improvement. Articulate-Anything employs a critic agent for self-evaluation and refinement over subsequent iterations. We gain around $5.8\%$ and $2.4\%$ improvement on link placement and joint prediction, respectively. While the ground-truth success (based on human annotations) is not available to the agent, our critic’s evaluation closely correlates with ground-truth. The confusion matrix is provided in Appendix Fig. 12.
Refer to caption: x9.png

### 5.3 Can articulate-anything evaluate and refine its own predictions?

We show quantitatively the iterative improvement capability of articulate-anything in Fig. 9. The critic success rate is computed as the percentage of predictions that are deemed successful by our VLM critic while the ground-truth success rate is computed from human annotations. Via iterative refinement, we gain around $5.8\%$ and $2.4\%$ improvement in accuracy for link placement and joint prediction, respectively. The critic evaluation shows a very high correlation with the ground truth, although it tends to overestimate success (please see Appendix Fig. 12 for the confusion matrix).

## 6 An Application in Robotics

A 3D model without articulation can only afford trivial interaction such as pick and place. In this section, we show that generated assets from Articulate-Anything can be leveraged to train reinforcement learning (RL) policies in simulation to perform four finer-grained manipulation tasks such as closing a drawer. We train these policies using PPO (Schulman et al., ) in Robosuite (Zhu et al., ; Todorov et al., ) on a Franka arm, with details given in Appendix A.3. Figure 13 qualitatively visualizes the trained policies.

Figure: Figure 10: Simulated Robotics: We train manipulation policies for four tasks in simulation via reinforcement learning using Articulate-Anything’s generated assets.
Refer to caption: x10.png

## 7 Conclusion and Limitation

In this paper, we have presented Articulate-Anything, a novel vision-language agentic system capable of accurately articulating diverse objects from various input modalities. By reimagining articulation as a program synthesis problem and leveraging the power of VLMs within a self-improving actor-critic framework, Articulate-Anything automates a process previously constrained by intensive human labor and expertise. We hope that our work contributes to bridging the gap between the digital and physical worlds, enabling 3D creators to focus on artistic vision rather than tedious tasks, and paving the way for richer simulated environments that can massively scale up robot learning and interaction. Our current approach relies on mesh retrieval to ensure high-quality 3D assets and focus on the problem of automatic articulation. Integrating Articulate-Anything with an upstream foundation mesh reconstruction model (e.g., SF3D (Boss et al., )) could allow users to generate more customized assets, thereby broadening the system’s generality.

## Appendix A Appendix

Figure: Figure 11: Joint prediction failure visualization. We visualize different types of joint failures, ranging from the most egregious, joint type, to the least, joint limit.
Refer to caption: x11.png

### A.1 Link and Joint Prediction Error Computation

#### A.1.1 Link Error

A link is represented by a position $\mathbf{x}\in\mathbb{R}^{3}$ and quaternion orientation $q\in\mathbb{H}$, where $\mathbb{H}$ is the space of unit quaternions. Given a predicted link state $(\mathbf{x}_{p},q_{p})$ and ground truth state $(\mathbf{x}_{g},q_{g})$, we define two error metrics:

Position Error: The Euclidean distance between predicted and ground truth positions:

$$ $e_{\text{pos}}=|\mathbf{x}_{p}-\mathbf{x}_{g}|_{2}$ (4) $$

Orientation Error: The geodesic distance on the unit sphere, computed as:

$$ $e_{\text{orient}}=2\arccos(|q_{p}\cdot q_{g}|)$ (5) $$

where $\cdot$ denotes the quaternion dot product. This metric represents the smallest rotation angle between the predicted and ground truth orientations.
The total link error is an average of these components.

#### A.1.2 Joint Error

Joint error is computed by comparing several components of the predicted joint state against the ground truth. Let $J_{p}$ and $J_{g}$ denote the predicted and ground truth joint states, respectively. The joint error components, from most to least egregious, are as follows:

- 1.
Joint Type Error: A binary measure indicating whether the predicted joint type matches the ground truth:
$e_{\text{type}}=\begin{cases}0&\text{if }\text{type}(J_{p})=\text{type}(J_{g})
\\
1&\text{otherwise}\end{cases}$
(6)
- 2.
Joint Axis Error: The angle between predicted and ground truth joint axes:
$e_{\text{axis}}=\min\bigg{(}\arccos\bigg{(}\frac{\mathbf{a}_{p}\cdot\mathbf{a}
_{g}}{|\mathbf{a}_{p}|_{2}|\mathbf{a}_{g}|_{2}}\bigg{)},\arccos\bigg{(}\frac{-
\mathbf{a}_{p}\cdot\mathbf{a}_{g}}{|\mathbf{a}_{p}|_{2}|\mathbf{a}_{g}|_{2}}
\bigg{)}\bigg{)}$
(7)
where $\mathbf{a}\in\mathbb{R}$ is the axis of rotation for revolute joints or of translation for prismatic joints, and $e_{\text{axis}}\in[0,\pi]$.
- 3.
Joint Origin Error:
For revolute joints, we compute the shortest distance between the joint axes using cross product:
$e_{\text{origin_pos}}=\frac{|\mathbf{p}\cdot(\mathbf{a}_{p}\times\mathbf{a}_{
g})|}{|\mathbf{a}_{p}\times\mathbf{a}_{g}|}$
(8)
where $\mathbf{p}=\mathbf{x}_{p}-\mathbf{x}_{g}$ is the difference between predicted and ground truth origins, and $\mathbf{a}_{p}$, $\mathbf{a}_{g}$ are the predicted and ground truth joint axes.
For prismatic joints, we use the Euclidean distance:
$e_{\text{origin_pos}}=|\mathbf{x}_{p}-\mathbf{x}_{g}|_{2}$
(9)
- 4.
Joint Limit Error: Comprises two sub-components:
(a)
Motion Range Difference:
$e_{\text{limit_range}}=|\mathbf{m}_{p}-\mathbf{m}_{g}|_{2}$
(10)
where $\mathbf{m}_{i}=\mathbf{a}_{i}(u_{i}-l_{i})$, with $u_{i}$ and $l_{i}$ being the upper and lower joint limits.
(b)
Motion Direction Difference:
$e_{\text{limit_dir}}=1-\frac{\mathbf{m}_{p}\cdot\mathbf{m}_{g}}{|\mathbf{m}_{
p}|_{2}|\mathbf{m}_{g}|_{2}}$
(11)
This metric ranges from 0 (identical direction) to 2 (opposite direction), with 1 indicating perpendicular directions.

A joint prediction succeeds when all of the joint component predictions are within a tolerance. Otherwise, a joint failure is attributed in a natural order where is the most egregious error – joint type – is detected first and the least – joint limit – last.

A visualization of different joint failure types is given in Fig. 11.

Figure: Figure 12: Critic-groundtruth confusion matrix. Our visual critic is highly correlated with the ground-truth. The largest disagreement comes from false positive case i.e., the critic falsely declares an incorrect articulation as correct. These cases include difficult-to-notice errors.
Refer to caption: x12.png

### A.2 Critic-groundtruth Agreement

Fig. 12 visualizes the confusion matrices between the ground-truth and our own critic predictions of success in two articulation tasks. Our visual critic is highly correlated with the groundtruth. The largest disagreement comes from false positive case i.e., the critic falsely declares an incorrect articulation as correct. These cases include difficult-to-notice errors.

### A.3 Robotic Training Details

We train a Franka arm to perform four robotic manipulation tasks in the Robosuite simulator using PPO and our generated assets.
The policy either takes a RGB-D or depth images along with object-state and robot proprioceptive states. Each policy class (RGB-D or depth-only) is trained over 3 random seeds per task for 2 million environment steps. Physical properties of the assets and RL rewards are manually designed. PPO implementation is provided in Stable-Baselines3 library Raffin et al. (). We use a small two-layer convolution network to extract the visual features and a two-layer feed-forward network to process non-visual inputs. We use the visual inputs captured by a camera attached to the end-effector (i.e., robot0-eye-in-hand in Robosuite).

Figure 13 shows the learning curves for these policies.

Figure: Figure 13: Simulated robotics learning curves with shaded standard deviations.
Refer to caption: x13.png

### A.4 Baselines

#### A.4.1 Casual Inputs

Our video-based approach excels with casually captured inputs in cluttered environments while baselines require extensive manual curation: we adjusted DINO bounding boxes for URDFormer and carefully curated foreground segmentation and other hyper-parameters for Real2code. On our project website, we show a diversity of object categories casually captured on an iPhone. The inputs have different view angles and are sometimes inadvertently titled. This kind of inputs presents great difficult for the baselines. For example, Fig. 16 shows the manual adjustment needed for URDFormer. Fig. 15 shows the manually curated foreground segmentation mask and other tuned hyper-parameters to clean up object-part masks and point clouds for real2code.

#### A.4.2 Reproducing Real2code

Real2code LLM Training
Real2code has not released their LLM model checkpoint nor training code. We therefore have done our best to reproduce their LLM model based on their paper description and other code. We obtain the LLM training dataset using their preprocessing code, and finetune the CodeLLama-7B-Instruct model Roziere et al. () using LoRA Hu et al. () with 4-bit quantization as described in their paper. The prompt for the LLM is given in Fig. 14.

Figure: Figure 14: Our instruction prompt for real2code reproduction. Neither training code nor model checkpoint were available from the original work’s Github.

Figure: Figure 15: Real2code manually curated inputs and intermediate outputs. We used about 3 to 7 input images per object from different views to obtain good global alignment from DUSt3R. Foreground segmentation masks were manually curated by querying the base SAM model. The segmented point-clouds and meshes were curated by tuning various segmentation hyper-parameters (e.g., which input images to use, number of query points, minimum point-cloud size et cetera) and surface reconstruction parameters (i.e., $\alpha$). Incorrectly segmented point clouds were manually removed and spurious points were removed using Open3D’s radius outlier removal and only keeping the largest connected component from DBSCAN clustering Ester et al. () per segmented mask. LLM outputs contain incorrect syntax and extra rambling which were parsed manually.
Refer to caption: extracted/5899507/Figures/real2code_render_outputs/input_img/toilet/1.jpg

Real2code In-the-wild reconstruction
We were not able to reproduce the shape completion model based on the released code. Thus, we instead use alpha surface reconstruction Edelsbrunner et al. () from Open3D Zhou et al. () to reconstruct meshes from point clouds, and manually tune the hyper-parameters. We found that this method produces on-par or higher quality meshes from their meshes that were publicly released.

We manually select query points for SAM to ensure near-perfect foreground segmentation masks, tune hyper-parameters for the 3D segmentation scheme (e.g., the input images, number of query points, minimum point-cloud size et cetera), clean up point-clouds, masks and tune mesh extraction. The kinematic-aware SAM model was finetuned using the code from the baseline’s authors.

We manually curate the foreground segmentation masks, remove incorrectly segmented point-clouds, tune hyper-parameters and manually parse their LLM outputs. Fig. 15 visualizes the inputs and intermediate outputs from real2code.

Figure: Figure 16: URDFormer manual correction. Comparison of bounding boxes from their fine-tuned DINO and our manual correction.
Refer to caption: extracted/5899507/Figures/urdformer_manual_correct/dino_lab_toilet.png

#### A.4.3 Evaluating URDFormer

URDFormer predicts one link per bounding box, with each non-root link associated with a joint. To evaluate URDFormer’s predictions against ground truth, we employ a link matching algorithm that aligns predicted links with ground truth links. Each URDFormer bounding box is matched with a ground-truth link by computing the overlap between the box and link’s segmentation mask given by the physics engine. For in-the-wild reconstruction, we manually corrected the object-part bounding boxes as shown in Fig. 16.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: In addition to the questions implied above, could I also ask if you experimented with weighting the losses in Eq 1? 

In section 5, on page 6, you also write that "Success requires all criteria to be within a small threshold... The position threshold is set to 50mm and the angular threshold to 0.25 radian (∼ 14.3 degree)." 

How did you choose these thresholds? Some defence of or rationale for these choices would be helpful, unless they are arbitrary?
soundness: 3
strengths: This paper does a lot of interesting things. Most of all, the idea of using LLMs to generate code is well applied here, and carried through evaluations that lead to at least proof of concept robot applications (though I believe still in simulation).

To the best of my knowledge, this is an *original* and *significant* contribution. For example, I don't know the prior work well, but the baselines sound reasonable.

The *quality* and *clarity* of the paper is decent as well, though could be improved (I have some suggestions below).

The multi-modality of the approach is also notable, and I appreciated the ablation of the different modalities (section 5.1).
summary: This paper presents a method for generating simulated 3D objects, with moving parts, that can be inferred with from multi-modal inputs (text, images, videos). This is interesting because is suggests a method of inferring models of objects, so that (robot) agents can be trained on richer and more representative simulations for things that exist in the real world. 

The method leverages LLMs to generate URDF descriptions (effectively code) which can then be iteratively refined based on feedback from a critic module. This is a neat use of LLMs that follows a trend in ML, but is well applied here.

Success is primarily evaluated by taking ground-truth URDF descriptions from an existing dataset (PartNet-Mobility), masking out information in them, and then comparing the reconstructed URDF with the ground-truth, using a threshold tolerance to define "success rates".
weaknesses: Three weaknesses in the paper stood out but could be improved in revision. 

The first is relatively minor, but there seemed to be missing references to other works that have used LLM agents in an actor-critic setup. 

One paper that came to mind was the Text2Reward paper (https://arxiv.org/abs/2309.11489v3). But more generally, it would help to emphasise that this general idea - which is very cleverly used here - is not part of the paper's novely.

The second weakness is a lack of statistics and confidence intervals in the results. 

For example, Table 1 shows average joint prediction errors and it is possible to see differences between these averages. But without statistical testing (e.g. F-test or t-tests) it is hard to appreciate how meaningful these differences are. Some measure of variance would be useful here too (e.g. +/- standard deviation, to appreciate the consistency of the different models). In general, plotting more than summary statistics (e.g. averages) would be much stronger (e.g. violin or raincloud plots).

Relatedly, the Figure 5 caption says that the approach "significantly outperforms all baselines". But as before, there are no statistical tests to back this up - just comparative %success rages without confidence intervals. I agree that the proposed model results appear better, but without a sense of the confidence intervals it's hard to be certain. In this case, Chi-square tests (if the samples are large) or Fisher's exact tests (for smaller sample sizes) might be appropriate.

The final weakness might be harder to address as it concerns fitting all of the main results into the main body of the paper. 

To me, it was most notable that the application for robotics section felt incomplete and even confusing. I was not able to properly evaluate what was done or whether it was good based on the short section, which is a shame because it is a very interesting culmination of what the paper builds to. The paper also frames this as a key result, but then doesn't quite get all the way there.

Of course I appreciate that there are page limits, and I see that there are additional details in an appendix. But it would be better to fit more of that into the main paper for completeness.

Similarly, the in the wild results seemed incomplete in the main text as well, though I see there are more details in the appendix.

Unfortunately I don't have any great suggestions on what you could cut to make room for these things. I suppose you might play with resizing some of the figures, cutting words where you can, and reformatting carefully.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What types of inputs does this work use when comparing to prior methods? Are these inputs—such as text, images, or videos—consistent with those used in previous works?

2. In terms of object retrieval, since this work aims to create digital twins from real-world observations, does the generated articulated object accurately resemble the real-world counterpart? Given that retrieval relies on CLIP scores based on text descriptions, the generated objects might be different from the original objects.

3. Could accuracy be reported across different object categories? Performance might vary significantly by category; for instance, simpler objects like windows and storage furniture may be easier to generate, whereas complex items like toilet seats or shopping carts could present more challenges.
soundness: 3
strengths: 1. Compared to previous works, this approach can begin with various inputs, including text, images, and videos.

2. By using iterative refinement with an actor-critic model, this method generates articulated assets with greater accuracy than prior approaches.

3. It utilizes vision-language models (VLMs) to perform the task without requiring network training or fine-tuning, eliminating the need for large-scale datasets, which are challenging to obtain for articulated objects.
summary: *Articulate-Anything* is a system that automates the articulation of complex objects from various input types, including text, images, and videos. It utilizes vision-language models (VLMs) to generate code that creates interactable digital twins compatible with 3D simulators. The system incorporates a mesh retrieval mechanism to leverage existing 3D assets and employs an actor-critic approach to iteratively propose, assess, and refine articulations, self-correcting to enhance accuracy.
weaknesses: 1. While VLMs hold the potential to go beyond existing dataset, this method retrieves parts from those datasets. Thus, it is still constrained by existing datasets.

2. The framework is mainly prompting VLMs to construct articulated objects, with limited technical contributions. And the iterative refinement phrase only raise limited accuracy.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. According to Fig.3, a template object is retrieved when visual input is provided. How do the follow-up link prediction and joint prediction steps work on template objects? I assume that the link and joint information of the template object are already provided in PartNet-Mobility dataset.

2. What are the 'dimensions' given by the Layout Planner based on? How to ensure that the dimensions of multiple object parts given by the Planner are not contradictory?

3. Using VLMs to directly output numerical values (dimensions, angles, etc.) seems not a good idea for embodied applications which have high demands on safety and reliability, as studies have shown that LLMs are not strong enough to process numerical operations.
soundness: 2
strengths: 1. The paper is well-motivated. Understanding the structure of articulated objects is important for vision and robotics tasks.
2. The idea of closed-loop iterative refinement is interesting and technically sound.
3. The approach is able to handle diverse input modalities, including texts, images, and videos.
4. It is a good idea to use structural representations like code to model articulation structures.
summary: This paper proposes Articulate-Anything, a closed-loop vision-language actor-critic system that is capable of converting real objects (provided via image, video, or text descriptions) into Python codes and further compiles them into URDFs. The proposed approach is evaluted against baselines of previous proposes and noticable improvements over success rates of link placement and joint prediction are achieved.
weaknesses: 1. It is not explained how the retrived part meshes do not conflict with each other. For example, in Fig.4(a), the code fragments do not show mechanisms preventing the 'door' having collisions with the drawer in the 'furniture_body'. In other words, how is the detailed position of the 'door'  (including its height) determined?

2. Some technical details are not clear. For example, the target affordance extraction part in Sec.4.4 and how link placement / joint prediction works for text / visual inputs, as the mesh retrieval results of text / visual inputs are different. A pseudo code can be help.

3. It seems that the method and experiments do not support well the task studied in this paper. According to Sec.3, the task is consistent with a formulation of reconstruction task. However, the proposed approach reterives similar meshes from a dataset rather than reconstruct the mesh in Sec.4, which cannot ensure the consistency between the retrieved mesh and the real object part and may result in large L_mesh. And the experiments just quantitatively evalutate the accuracy of link placement and joint prediction. Evaluations with important metrics like CD for reconstruction are not provided.

4. The robotic application is not solid enough. The test results like the success rate of different manipulation tasks are not provided, making it difficult to give a proper evalulation on the effectiveness of the generated assets. And additional real-world experiments will make this point stronger.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Could the authors provide examples illustrating how the performance of link placement and joint prediction evolves over the iterations?

2. Does the number of prompted examples affect the pipeline's performance? It would be helpful if the authors could provide an ablation study on the number of prompted items to determine whether performance varies based on this factor.
soundness: 2
strengths: 1. The proposed methods for automatically modeling articulated objects address an interesting and meaningful topic, with potential to greatly benefit fields such as robotics.

2. The techniques introduced are well-founded. By leveraging vision-language models (VLMs), which encode valuable information about object affordances, the system effectively determines common articulations for various objects.

3. The paper provides a detailed analysis of failure cases, as shown in Figure 8, highlighting how the proposed pipeline outperforms other methods and identifying areas for further improvement.

4. The paper is well-written and easy to follow, making the concepts accessible to readers.
summary: This paper presents a system designed to automate the articulation of complex 3D objects for applications in AR/VR, animations, and robotics, addressing the challenge of extensive human effort required in traditional methods. It utilizes vision-language models (VLMs) to convert inputs such as text, images, and videos into interactable digital twins for 3D simulators. The system incorporates a mesh retrieval mechanism from existing 3D datasets and actor-critic frameworks to iteratively refine object articulation, achieving robust results.
weaknesses: 1. The proposed methods have only been validated on limited types of articulated objects (i.e., prismatic and revolute). When the types become more complex, such as cylindrical or universal joints, the current pipeline may struggle to handle these cases.

2. The pipeline has only been tested using Google's Gemini VLM. It is recommended to conduct ablation studies to assess whether the inference performance varies with different VLM architectures, in order to evaluate the system's robustness.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 3
questions: - Is there any failure case analysis? For example, under what circumstances does the retrieval process select the wrong mesh? It would be helpful to show when retrieval might fail and how challenging those cases are.

- Concerns regarding Section 4.2:
  1) For instance, in Fig. 4a (left example), 'furniture body'  is place to the left relative to the 'door'. Since the body has thickness, why is the door placed on the left side of the front surface rather than the back? As the description only defines 2D spatial relationships, it's unclear how to position the link in 3D space.
  2) How many candidates does the actor's output generate? Can the relation fully cover all possibilities?

- Regarding Section 6's robotic experiment, where do the generated assets come from? Does Articulated Anything input video or images from RoboSuite to generate assets, and then train PPO with the generated assets, instead of directly using the assets in RoboSuite?
soundness: 4
strengths: The proposed method for generating digital twins of articulated objects is intriguing and highly effective, significantly enhancing performance during evaluation. The website is comprehensive, offering visualizations and related code.
summary: This work proposes a retrieval method for generating articulated objects in a simulator, taking into account both link and joint placement. It also demonstrates promising results in partner mobility
weaknesses: Please refer to 'Questions' section

Meta-review: This work introduces a novel approach to automate the articulation of complex objects from multi-modal inputs (text, images, videos). Four reviewers have expressed positive feedback, while one reviewer has raised some concerns. The author has done a good job during rebuttal. The Area Chair finds the work interesting and recommends it for publication at NeurIPS 2024. The reviewers have highlighted some valuable concerns that should be addressed in the final camera-ready version. The authors are encouraged to incorporate the necessary revisions in the final submission.

Decision: Accept (Poster)