Articulate-Anything:  Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model
Authors: Long Le
Jason Xie
William Liang
Hung-Ju Wang
Yue Yang
Yecheng Jason Ma
Kyle Vedder
Arjun Krishna
Dinesh Jayaraman
Eric Eaton
Venue: ICLR 2025 Poster (2025)
Keywords: Computer vision
Vision-Language Models
Robotics
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce Articulate-Anything, a state-of-the-art method for generating diverse and high-quality interactable digital twins from many inputs including text, images, and videos.

Abstract: Interactive 3D simulated objects are crucial in AR/VR, animations, and robotics, driving immersive experiences and advanced automation.
However, creating these articulated objects requires extensive human effort and expertise, limiting their broader applications. To overcome this challenge, we present Articulate-Anything, a system that automates the articulation of diverse, complex objects from many input modalities, including text, images, and videos. Articulate-Anything leverages vision-language models (VLMs) to generate code that can be compiled into an interactable digital twin for use in standard 3D simulators. Our system exploits existing 3D asset datasets via a mesh retrieval mechanism, along with an actor-critic system that iteratively proposes, evaluates, and refines solutions for articulating the objects, self-correcting errors to achieve a robust out- come. Qualitative evaluations demonstrate Articulate-Anything's capability to articulate complex and even ambiguous object affordances by leveraging rich grounded inputs. In extensive quantitative experiments on the standard PartNet-Mobility dataset, Articulate-Anything substantially outperforms prior work, increasing the success rate from 8.7-11.6\% to 75\% and setting a new bar for state-of-art performance.  We further showcase the utility of our generated assets by using them to train robotic policies for fine-grained manipulation tasks that go beyond basic pick and place.

Published as a conference paper at ICLR 2025
ARTICULATE-ANYTHING AUTOMATIC MODELING OF ARTICULATED OBJECTS VIA A VISION-LANGUAGE FOUNDATION MODEL
Long Le, Jason Xie, William Liang, Hung-Ju Wang, Yue Yang, Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, Eric Eaton University Pennsylvania
https://articulate- https///frtticlllat-aannhnnng git..u..u io/
ABSTRACT
Interactive 3D simulated objects are crucial in AR/VR, animations, and robotics, driving immersive experiences and advanced automation. However, creating these articulated objects requires extensive human effort and expertise, limiting their broader applications. To overcome this challenge, we present ARTICULATE- ANYTHING, a system that automates the articulation of diverse, complex objects from many input modalities, including text, images, and videos. ARTICULATE: A ANYTHING leverages vision-language models (VLMs) to generate code that can be compiled into an interactable digital twin for use in standard 3DD simulators. Our system exploits existing 3D asset datasets via a mesh retrieval mechanism, along with an actor-critic system that iteratively proposes, evaluates, and refines solutions for articulating the objects, self-correcting errors to achieve a robust out- come. Qualitative evaluations demonstrate ARTICULATE -ANTTHAN- capability to articulate complex and even ambiguous object affordances by leveraging rich grounded inputs. extensive quantitative experiments on the standard PartNet- Mobility dataset, SEHICULATE-ANYTHIEE increasing the success rate from 8.7-12.2% to 75% and setting a new bar for state-of-the-an the-art performance. We further showcase the utility of our system by generating 3D assets from in-the-wild wild video inputs, which are then used to train robotic policies for fine-grained manipulation tasks in simulation that go beyond basic pick and place. These policies are then transferred to a real robotic system.
substantially outperforms prior work,
In-the-Wild Objects
Sim2Real Robot Learning
Text
filing
Images
Articulate Anything
Articulation
Videos
Figure 1: Given text, images, videos showing an object's motion, ARTICULATE-ANYTIINE matically generates its 3D interactable digital twin, handling wide variety of objects and affordances. Among other applications, these articulated assets can be used to train robotic manipulation policies in Sem2Real. Full video demonstrations and source code are available on the website.
INTRODUCTION
One of the most promising avenues in the quest for generally capable robots the pursuit of scaling robot training in simulation for deployment in the real world. We have developed fast, accurate

---

Published as a conference paper at ICLR 2025
simulators that scale to millions of FPS and hundreds of GPUs (Xiang et al., 2020; Makoviychuk et al., 2021), enabling policy learning on a staggering scale. However, a critical bottleneck this research direction persists: the immense human labor required to construct realistic, interactable environments for these agents to learn within. Despite the existence of large, open libraries of static object geometries
with the largest open dataset containing over 10 million objects (Deitke et al., 2024) - we have comparatively minuscule open libraries of articulated 3D objects (only around 2,300 objects (Xiang et al., 2020)). This scarcity stems from the time-consuming, labor-intensive, and expertise- deeandie- nature of the manual annotation process.
To address this challenge. we present POTTCULATE-ANYTHIEE articulation that harnesses the power of leading foundation vision-language models (VLMs) to articulate a diverse range of objects of arbitrary complexity through iterative feedback (Fig. 1). ARTICULATE--ANTHHEE
a novel approach in automatic
represents step function improvement in quality, accuracy (8.7-12.2%% to 75%), and generalizability over prior art (Chen et al., 2024; Mandi et al., 2024), overcoming previous limitations that restricted success to only narrow range of object categories and joint types. Unlike prior art, which has been limited by the impoverished input of bounding boxes or static images, ARTICULATE-ANT -NNUTHHT- affords the flexibility of consuming rich, grounded inputs from text, images, or even videos, enabling users to request exotic articulation descriptions or resolve articulation ambiguities. For example, the right column of Fig. 7 features a digital model of a window that could plausibly slide or tip to open; when ARTICUL ATTTCULLAFF-NNT ANYTHING G is shown an in-the-wild video demonstration, it accurately produces the desired sliding motion.
To achieve this level of flexibility and accuracy, ARTICULATE-AN ANYTHING employs an actor-critic system with two core components: (1) vision-language actor that synthesizes high-level Python code, which can be compiled into Unified Robot Description Format (URDF) files and (2) a vision-language critic that provides feedback on the rendered prediction compared against available ground-truth The result is an agentic system that can automatically self-evaluate and iteratively improve the articulation of complex objects. Beyond robotics, the flexibility of ARTICUL ANYTHING's inputs married with its high-quality outputs puts automatic generation of rich, high-quality, and diverse virtual environments within reach with groad-reaching applications to 3D/VR (Kim et al., 2024), human-computer interaction (Jiang et al., 2023), and animation (Yang et al., 2022).
Our key contributions include:
1. ARTICULATE-ANYTIEA NYTHING:
We present a vision-language actor-critic system that accurately articulates objects from diverse input modalities, including texts, images, and videos.
2. Articulation as program synthesis: We develop a high-level Python API compilable into URDFs, enabling the VLM actor to generate compact, easily debuggable programs.
3. Visually grounded inputs enable closed-loop iterative refinement: We highlight the critical role of grounded visual inputs in articulating ambiguous and complex objects, leveraging such inputs in a closed-loop system that self-evaluates and self-improves its articulations.
4. Extensive evaluation demonstrates superior performance: Prior works in automatic articulation are only evaluated on handful of object categories. Quantitative analysis on the entire PartNet- Mobility dataset shows dramatic improvement over existing methods, increasing the success rate from 8.7-12.2% of prior work to 75%
2 RELATED WORK
3D Asset Datasets.
Large open datasets of 3D objects, such 3D Warehouse and Objaverse (Deitke et al., 2024), provide extensive collections of 3D models uploaded by independent artists However, most objects in these repositories lack articulation. Datasets like ShapeNet (Chang et al., 2015) and PartNet (Mo et al., 2018) decompose objects into parts (links) but do not include kinematic joints. Some datasets, such as ParbNit-Mobility (Xiang et al., 2020) and GAPartNet (Geng et al., 2022), do feature articulated objects with links and joints, but these annotations are manually created so their number of objects is modest. Our work aims to automate the creation of articulated 3D assets suitable for robotic tasks, reducing the reliance manual annotation.
Articulated Object Modeling, Articulated object modeling is a significant area of research, encom- passing tasks such as perception, reconstruction, and generation (Liu et al., 2024). In perception, works like (Zeng et al., 2021; Hu et al., 2017) focus on identifying and understanding articulated objects. Reconstruction efforts from single RGB images (Chen et al., 2024), RGBD images (Weng
2

---

Published as a conference paper at ICLR 2025
al., 2024), multi-view images (Liu et al., 2023a), point clouds (Jiang et al., 2022) aim rebuild articulated models. Generation methods, including those leveraging connectivity graphs (Liu et al., 2023b) or neural approaches (Lei et all, 2023), focus creating new articulated objects. Our goal to develop an end-to-end pipeline that spans from perception to reconstruction and generation, utilizing intuitive and easily accessible inputs from humans, such language videos, instead of relying on more specialized modalities, such as point clouds or graphs.
PROBLEM FORMULATION
To ensure compatibility with standard 3D simulators, represent 3D models in the Unified Robot Description Format (URDF). In URDF, an object is structured in a hierarchical tree consisting of nodes (links) and edges (joints). The links represent parts of object (e.g., drawers and doors of kitchen island), and joints specify how each part moves. The two most common joints are prismatic, representing a translational movement (e.g., sliding drawer), and revolute, representing rotation (e.g., pivoting door).
Given input x of text, image, video depicting an object, assume there exists ground-truth URDF u" (e.g., one that a human could create). The goal of articulation is to automatically construct URDF model that comprises the same set of links as u' while minimizing the difference between URDF models via the following loss:
Liaal -mesh + Llink + Lyoint
where Lmesh 1= Chamfer P,, P measures the discrepancy in visual appearances between the predicted and ground-truth -tuth links using point clouds P. and P N is the number of links, Link captures 3D pose differences between predicted and ground-truth links:
Lrikk
Σ
ill2 + arccos( (4 Q(I)
position erroe
orientation error
where is 3D coordinate and qi is quaternion. Gjain quantifies the joint discrepancies.
Jjui Cjoint type + Cjoint axis -oint origin + Cyoin limit
where Ljoin type is a cross entropy loss, Cjoint axis is the angular difference between axes, Cyoint origin the 3D pose difference between joint origins, and Cjaim limit measures the difference in range and direction of motion. In practice, jointly optimizing this objective becomes intractable. Consequently, we instead tackle each loss sequentially, decomposing the problem into three phases: (1) Mesh Retrieval, (2) Link Placement, and (3) Joint Prediction. Furthermore, access to u is often limited, making traditional optimization methods for computing these losses impractical. To overcome this, we leverage languane model (VLM) agents approximate the loss functions directly from visual inputs. Additionally, instead of refining solutions through a conventional optimizer, VLM agents are also used to propose solutions in the form of code and iteratively refine them based on feedback from a critic module. This process is detailed in the following sections.
4 ARTICULATED OBJECT GENERATION VIA ACTOR-CRITIC VLMS
system capable of generating articulated objects from various input modalities. Figure 2 presents an overview of our method, which includes three main components: (1) Mesh Retrieval (Sec. 4. 4)),, which reconstructs the 3D structure for each object part by retrieving meshes from a 3D asset library, (2) Link Placement (Sec. 4.2), which spatially arranges the parts, and (3) Joint Prediction (Sec. 4.3), which determines the potential kinematic movements between parts. Additionally, we describe an optional step, targeted affordance extraction, in Sec. 4.4. Our system consists of many specialized VLM agents. Each agent a Google's Gemini Flash-1.5 1.5 visual language model (Team et al., 2023) prompted with no more than 20 in-context examples to perform different tasks. All system and task instruction prompts are provided in the source code.
In this section, we introduce ARTICULATE-ANYTHIN
Crucially, our system produces high-level Python code compilable into URDFs via a custom API we have developed. Directly generating URDFs is challenging due to (1) the verbosity of URDF/XML, which increases the likelihood of hallucination in long contexts, and more importantly, (2) the need for performing complex mathematics in-place in many cases. For instance, articulating rotation
3

---

Published as a conference paper at ICLR 2025
Input Text, Image, Video
Mesh Retrieval
3.5 Targeted Affordance
Bouble-huun
glass panes. Link Placement
soleston
Actor
pred,
pred_robot zokot, add line(Linkes witloow)) zelative_tel "window", sxnndo pllceewnt="nneiie cleasancewO.e
"renlser, sattingg: "18"
"description": "The prodiction visually Adentical to the groundtzuth.
Critic
Joint Prediction
Actor
pxed_ wobot. wtt.:pootootc.eeeeeeeeeee "window,
Slace_solative_te
"st_dese": "window slides and down along axis",
dasc": "window zotates doxnaze along "Suggeation"s "use
pxed_robor www.pooooonac....o
www..naeeeepprrrcccc..m.m prisrariir jointt
"eake_ wwwppaatee_eennntteee..
Sanget_windin link, "windon lower_point, uposs_point
"renlisr, rrating"
Critic
Figure 2: Method Overview. Given text, image video input, ARTICULATE- ANYTHHNGN operates in three stages: (I) Mesh Retrieval (Sec. 4.1) retrieves a mesh for each object part from a 3D asset library, (2) Link Placement (See. 4.2) places the parts together, and (3) Joint Prediction (Sec. 4.3) predicts the allowed kinematic movements between parts. Optionally, instead of generating all possible kinematic joints, we can target specific joint from the input video (Sec. 4.4). The link placement and joint prediction systems consist of an actor and critic, which are VI.Ms working together. The actor proposes solutions, and the critic examines those solutions and gives feedback.
along the global z-axis for a door requires forward kinematics calculations down the URDF to translate it to the relative frame. Even seemingly simple tasks, like positioning toilet seat above the bowl, can be complicated due to non-trivial mesh geometries. However, this can easily be remedied by performing iterative collision checks in physics engine.
4.1 MESH RETRIEVAL
We employ two separate mechanisms for reconstructing meshes from visual (i.e., image video) and text inputs. An overview of these processes is provided in Fig. 3.
Visual Inputs. For visual inputs, we perform mesh retrieval by matching the input 3D object from the PartNet-Mobility dataset (Mo et al., 2018). First, a VLM instructed to detect an object of interest from the input. Since there might be multiple objects in the frame, when video is provided, the VLM is instructed to take advantage of motion cues for more accurate identification. Then, to reduce the search space, identify the top-k most similar object categories in the library using CLIP similarity (Radford et al., 2021). Lastly, a VLM agent Hierarchical ObjectSelector is tasked with selecting candidate among several simulated objects based on their visual similarity to the ground truth. This visual similarity estimate is proxy for the Lmesh loss described in Sec. 3. To manage the potentially large candidate pool, we adopt divide-and-comee and-cuu tournament approach. Specifically, recursively select the best candidate among batch of max_num-image images until there is only one candidate left.
Text Inputs. For text prompts, we employ a multi-step process using large language model (LLM) agents. First, a TaskSpecifies fier densifies the potentially sparse prompt (please see Fig. 3 (b) for an example). Then, LayoutPlanner is instructed to describe each object part along with its dimensions. Finally, mesh retrieval is performed for each part by matching the object parts' CLIP language embeddings the mesh descriptions in the PartNet-Mobility dataset via cosine similarity.
4

---

Published as a conference paper at ICLR 2025
Visual Input
VLM Description
Top-K
Candidate Selection
"object": computex monitos", "affordance":
Carwza
Display
conitor can be 11ttbe"
CLIP
Dataset Categories
[a]
Input Text
Layout Planner
Dzaner
Part Dims
"Eiting www.onngcceetttteeeeeeeeeeeeeeee bady", "deccription" "the main body fillng cabinet",
"nane" 1:
Task Specifler
"dinonstons": (9.46, 9.38,
A three droeex
1.
t:inng cabinet
on the botton and body top
Dataset Part Descriptions
(b)
Figure 3: Mesh retrieval. The top and bottom diagrams provide overviews for reconstructing visual (i.e., image video) and text inputs, respectively. For visual input, we match the ground-truti object to template object in the library using an efficient siivide-nddcoonee text input, we first prompt an LLLM to predict the different object parts and their dimensions. Then, we retrieve mesh for each part using precomputed CLIP embeddings and subsequently scale the meshes to specifications. More details Sec. 4.1.
retrieval mechanism. For
Since the original mesh descriptions from the dataset are coarse, we compute our own annotations by prompting VLM to describe each mesh (e.g., material, shape, function) given its image. The retrieved meshes are finally rescaled to fit the specified dimensions.
4.2 LINK PLACEMENT
The link placement system can handle inputs containing the semantic names of each part, along with an optional input image. For videos, we extract the first frame as the image input. The system consists of an actor and a critic. The link actor is responsible for placing links together in the 3D space. In the API, placement is done by aligning aligning the child and parent's link centers along an axis and perform collision checks to ensure tight placement without intersection. When processing visual imputs, critic is also employed verify the actor's solution. The critic is prompted to describe any visual dissimilarity between the input image and the predicted 3D model rendered simulation, pinpoint the source of error in the actor's code, and give realism www.iimmraiinnn rating between 0 and 10. This realism rating serves as a proxy for the Liink loss. The actor instructed to take the feedback into account and adjust its Python code accordingly. This process terminates when the rating exceeds threshold of 5. Figure 4 (Left) provides an illustrative example of this process.
4.3 JOINT PREDICTION
Similar to link placement, the joint prediction system accepts inputs in the form of text containing Python code for link placement, well an optional input image video. The actor then extends the Python code to include kinematic joint prediction between parts. For video inputs, critic is also utilized to verify the actor's solution. The critic is instructed to compare the input against video of the predicted joint moving in simulation and examine the source code to identify either success or one of 4 failure cases. A is also used to categorize the most egregious error (incorrect joint type) to the least (incorrect joint limit). As before, the actor is instructed rewrite its code, taking the critic's feedback into account as long as the realism rating falls below a threshold of 5. See an example in Fig. 4 (Right).
between 0 and 10, serving as a proxy for the Lyoim loss,
4.4 TARGETED AFFORDANCE EXTRACTION
Optionally, instead of generating all possible kinematic joints, we can target a specific joint from the input video. This be useful to reduce the number of input tokens VLMs, thereby lowering the cost or making automatic debugging of predictions easier. To achieve this, we prompt a VLM agent targeted affordance extractor to determine which child link should be annotated with
5

---

Published as a conference paper at ICLR 2025
Actor
row
rne metrt Ltrelloned
light latran
Fieet LatS (FU))
Foant-uddh Bettee
"cear";
hundiire bedy. paee soest alacs pelation
"aurrbaa beay"
pead, edlat nlea, pewslltt, "Euondhure beay".
pratlratet www.linnn.irrrrr..... "Bucstrure
Find, relatt www.lltteee Sodant "darrauue body".
olearance
"fume for boly
plasaraii Seant",
glonal artuet 10, wor.oaoiieeeeee plret, pwwpertteeeoooooooo.o pedd
saggiaaaeeeee pleors wwwp:rpsdrriittttt.co patrt,
start, petteepaeteeeeoo
"Amcetattoely "The seadd in net sery asde
"anertirtien pontiction gossditsutt.. "ovelion _ww.ingg: 1
Same nemeettett coestten" "Thy side doct Frant- Lett Jocton) to make the juint groedtreut
"terremment vigaenttinn contt "En segatiin Linct
Teecovmenn sedpostaer"
the caldnec",
Eron 0 dn codec
settingt:
won cxteend,
"ceallin pusting i
Critic
Cfitlc
[a] Link Placament
(b) Joint Prediction
Figure 4: Both link placement and joint prediction systems consist of an actor and critic. The actor produces Python code, which is automatically compiled into URDFs and rendered in simulation. Source code, predicted, and input modalities (images for link and videos for joint) are given to the critic for evaluation. The synergy between the actor and critic enables self-correction of errors (red border) and successful articulation (green border).
kinematic joint given the input video and segmented image of the simulated asset, where each part distinctively colored.
EXPERIMENTS
Datasets: We use the Partnet-Mobilig dataset (Mo et al., 2018) which includes human annotations for 2233 objects, 1..9K revolute joints, and ~7.6K prismatic joints.
Given a ground-truth articulated URDF from PartNet-Mobility, we mask out all link and joint information. The objective is to reconstruct the masked URDF. Solving this objective includes two primary tasks: (1) link placement determine the spatial arrangement of object parts, and (2) joint prediction-ing infer the movement capabilities between parts.
Tasks:
Evaluation Metrics: We assess performance using success rates for each task.
1. Link placement: Success is determined by the pose difference between predicted and ground -ruth links falling below a small threshold.
2. Joint prediction: We evaluate differences in joint type, axis, origin, and limit. Success requires all criteria be within small threshold.
The position threshold set to 50mm and the angular threshold to 0.25 radian (~ 14.3 degree). When evaluating ARTICULATE AATT-AATTIINGGGG if object's link placement is incorrect, all of its joints are also counted as incorrect. More mathematical details and some visualizations of the different joint errors are available in Appendix A.
We compare against two prior sfute-of-the-art methods: URDFormer (Chen et al., 2024) and Real2Code (Mandi al., 2024). Both methods were trained or fine-tuned on five object categories in the PartNet-Mobiligg dataset. We evaluate the performance of these five (in-distribution) and the remaining 41 (out-of-distribution classes. Implementation details are provided Appendix A.4.
Baselines:
Our system employs Google's Gemini F1 (Team et al., 2023) as the vision- model (VLM). Physics computations are performed using PyBullet (Coumans & Bai, 2),6), while rendering is done in Sapien (Xiang et al., 2020). Motion traces in videos are annotated using CoTracker (Karaev al., 2023) for visualization. We use sew- prompting with around 20 in-context examples.
Implementation Details for ARTICULATE--ANTHHINE
5.1 How WELL DOES ARTICULATE-ANE ANYTHING PERFORM VERSUS PRIOR WORK?
We benchmark ARTICULATE- ANYTHINN against two prior works in automatic articulation. In this ex- periment, ARTICULATE ATTTHATHI--AGGNG is fed input videos. The key differences between ARTICULATE-

---

Published as a conference paper at ICLR 2025
100
80-
75.0
Articulate Any
Text
Image
Video
URDFormer Oracle
88.6
U'DDFormer DINO
Shicess Rate (%)
60-
Real2Code Oracke
Success Rate (%)
77.7
40-
68.6
24,7
20.2
20-
8.7
13.5
8.2
1LI
48.9
54.4
1.3
classes OOD)
Unseen
(ID)
(OOD)
Link Plavement
Joint Prediction
Figure Comparison against the baselines. Our approach significantly outperforms all base- lines in the joint prediction task. We use few-shot prompting and make no distinction between ID and OOD classes, so we only report results for all classes. 95% confidence intervals are included.
Figure 6: Input modality ablation. Perfor- mance on articulation tasks improves with more grounded modalities. Videos are only used for joint prediction. Input videos provided to the link placement system are automatically transformed into image inputs by extracting the first frames.
ANYTHING and prior work include (1) our system operates on high-level Python abstraction instead of low-level inputs, (2) our system can handle more grounded instruction inputs, including videos, and (3) our actor-critic system allows iterative refinement on more difficult objects. In contrast, URDFormer directly predicts part position (discretized) coordinates, and Real2code operates on oriented bounding box coordinate inputs. Prior works are also limited to simplified inputs as they cannot handle videos. This also means that their articulation must done in open loop.
Our innovations allow ARTICULATE-ANYTHINE in Fig. 5. Appendix A.6, table reveals the raw joint prediction errors behind the success rate of Fig. 5, including errors of joint sype.joinn origin, and joint axis. The error distribution is visualized via violin plots in Fig. 20. Fig. breaks down the failure reasons for each method. The breakdowns of success rate of our method by object categories an ablation where our method is given the same impoverished input modality as the baselines in Appendix A.5. Mesh reconstruction comparison is given in Appendix A.7.
to achieve superior performance as demonstrated
provided in Fig. 21 and 22. We also provide
Figure compares ARTICULATE AATTE-ANYTHINGG against the baselines on in-the- wild object reconstruc- tion. As before, select both OOD and ID objects. Our video-based method performs the best while baselines fail on all tasks, susceptible to minor misalignments or segmentation inaccuracies. Notably, leveraging rich video input allows ARTICULATE- ANYTHING resolve articulation ambiguities. For example, our non-video methods predict the chair leg sliding up and down (for adjusting height), and the window tipping to open. These predictions are actually reasonable given the input images and texts but turn out to be wrong when the ground- -tunth videos are revealed.
5.2 HOW DO DIFFERENT INPUT MODALITIES AFFECT ARTICULATION ACCURACY?
While LLMs trained on internet-scale data possess remarkable common sense abilities, we hypothe- size that visual inputs are still crucial for accurate object articulation. For example, consider simple faucet with a spout and lever. Where should the lever be placed? In fact, there is standard answer: there are faucets whose levers are on top and those whose levers are to the side. Joint prediction without videos is similarly difficult (e.g.. see Fig. 7). Thus we contend that more grounded input modalities such as videos are crucial in resolving difficult or ambiguous articulation.
In this section, we compare the articulation success rates across three input modalities: text, image, and video. For the text modality, we provide only the semantic part names for link placement; during joint prediction, the model receives the Python code for link placement and tasked with inferring the joints. With the image modality, we additionally supply a static image of the ground- truth object addition to the textual information. For the video modality, a video showcasing the object's motion is provided in addition the text. Note that videos only used for joint prediction. Input videos provided to the link placement system are automatically turned into images by extracting the first frames.
7

---

Published as a conference paper at ICLR 2025
Public tollettocToo
Input Texts
window, unite Twn glaas
Office whenk
casinet
retracrable tanale
Input Images
Input Videos
Ours w/ Text
Ours
Image
Ours w/ Video
Ours w/ Video Generation
URDFormer
Jike creapid Inaged)
00D
OOD
1D
Real2Code (ua lucaa)
ODD
BOD
GOD OOD
Figure 7: In-the-wild Reconstruction. We demonstrate ARTICULATE- ANYTIIING' performance input modalities compared to prior works URDFormer and Real2Code, Green and red borders denote correct and incorrect predictions with respect to the input videos. Our simulated videos are generated by varying joint limits and annotated with Cotracker. Casual inputs: Our iideo-based approach excels with casually captured inputs in cluttered environments while baselines require extensive manual curation (more details Appendix A.4 Fig. 17 and 18). Ambiguous articulation: Interestingly, our video-based method can resolve ambiguities in static inputs (c.g., chair rotation height adjustment, window sliding vs. tipping). Bascline limitations: URDFormer consistently predicts drawer-like structures and is sensitive minor misalignments (e.g.. slightly tilted drawers). Real2Code, which uses multi-view images for mesh reconstruction and text-oriented bounding boxes (OBBs) for joint prediction, achieves good global alignment from DUSt3R but produces low-quality 3D segmentation using a finetuned SAM, leading to joint prediction errors. We also included some results using a mesh generation model instead of retrieval to allow for more customized asset creation with more details in Appendix A.8. Mesh reconstruction quality result is included in Tab. ?? in Appendix A.6. Please visit our website for additional demos: https://articull annntiiillaeteaattiingnggg gi thub. io/.
8

---

Published as a conference paper at ICLR 2025
Articulate Anything
URDioormrr
Real2Cade Oracle
joint limit
inceerect link piacrmett
5.7
output
175
joiat type
88
2
Join origin
type
3.7
75.0
ourpu
joint type
24A
Join
Figure 8: Breakdown of failure percentages in all classes. In ARTICULATE ANYTHIING incorrect link placement leads to all predicted joints being marked incorrect. For baselines, 59. % of URD- Former's and 17.5% of Real2Code's 's outputs are invalid, containing cyclie structures, repeated links, or syntax errors. Details are Appendix A.1.
Link Placement
Joint Prediction
100
80
Success Rate (%)
Ground Truth Critic
Ground Truth
77.3
77.6
90.1
Critic
84.0
85.5
86.0
75
74.8
75.00
80.2
72.6
70
70
Iteration
Iteration
Figure 9: Iterative Improvement. SETICULATE-ANYTHINE employs a critic agent for self- evaluation and refinement over subsequent iterations. We gain around 5.8% and 2.4% improvement on link placement and joint prediction, respectively. While the ground-truth success (based on human annotations) is available the agent, our critic's evaluation closely correlates with ground- truth. The confusion matrix is provided in Appendix Fig. 15.
Link Placement
Joint Prediction
100
Success Rate (%)
100
90.2
96.5
Suceess Rate (%)
77.7
75-
58.4
50
44.5
25-
17.8
4.6
10 0-
Number Examples
Number of Examples
Figure 10: In-context learning, examples, demonstrating in-context learning. The zero-shot performance example) included.
improves with the number of prompting
We conduct this ablation study the Faucet object category for link placement and StorageFurni ture for joint prediction. We run an actor-only system for one iteration without the critic isolate the effect of input modalities. Results in Fig. demonstrate that richer input modalities consistently improve success rates, underscoring the importance of visual information in articulation tasks.
5.3
CAN ARTICULATE-ANYTIINE EVALUATE AND REFINE ITS OWN PREDICTIONS?
We show quantitatively the iterative improvement capability of articulate-anythinn in Fig. 9. The critic success rate is computed the percentage of predictions that deemed successful VLM critic while the ground-truth success rate is computed from human annotations. Via iterative refinement, gain around 5.8% and 2.4% improvement in accuracy for link placement and joint prediction, respectively. The critic evaluation shows very high correlation with the ground truth, although tends to overestimate success (please see Appendix Fig, 15 for the confusion matrix).
5.4 How DO THE NUMBER OF IN-CONTEXT EXAMPLES AND THE CHOICE VISION-LANGUAGG MODELS AFFECT OUR PERFORMANCE?
Frontier VLMs are known to benefit from in-context examples. Furthermore, the performance of these models vary across architectures. In this section, investigate how these factors affect our system. We conduct this ablation study on the Stor StorageFurniture category using an actor-only system from input videos. Figure 10 shows
ability perform in-context

---

Published as a conference paper at ICLR 2025
learning, improving the success rate with increasing number of prompting examples. Figure 11 shows SEEIICLLAEE-ANTTIIE robustness to different choices of leading VLMs, maintaining high success rates throughout.
55.7
77.7
70.4
Laptc
Manvoo
Obisat
GPT-40
Minaee
Son percesing Time
Car Time
Join Time
Figure 11: Base VL. Our method is robust to the choice of VLMs, maintaining high success rates throughout.
Figure 12: Human annotation time. Our method average saves 40.58 minutes in hu- man manual labor per task.
AN APPLICATION IN ROBOTICS
A 3D model without articulation can only afford trivial interaction such as pick and place. In this section, we show that generated assets from ARTICULATE- ANYTHING can be leveraged to train reinforcement learning (RL) policies in simulation to perform four finer-grained manipulation tasks. train these policies using PPO (Schulman al., 2017) in Robosuite (Zhu al., 2020; Todorov et al., 2012), with details given Appendix A.3. These policies are then executed on real Franka arm. For comparison, we also use a human expert to manually annotate the assets via the RialTo (Torne al. (2024)) GUI, which had been explicitly designed facilitate the scanning and annotation process. Both policies trained via SETTCULAEE-ANYTIIE ' and human-ame aamann-nnotttte assets achieve 100% success rate in the real world. However, ARTICULATE- LAE--AATTIINGG would have saved average of 40.58 minutes in human labor per task as shown in Fig. 12.
Articulate real-world videos
RL training in simulation
Transfer to real
Figure 13: Robotic Application: ARTICULATE- CELULLATE-ANTTHHNNN ANYTHING can automatically generate assets given in- the- input videos. We then train manipulation policies for four tasks in simulation via RL. The policies are deployed back on real Franka Panda arm, successfully completing all tasks. Please visit our website for the video demonstration:
thub. io/
7 CONCLUSION AND LIMITATION
In this paper, we have presented AETICULATE-ANYTIIOE capable of accurately articulating diverse objects from various input modalities. By reimagining artic- ulation program synthesis problem and leveraging the power of VLMs within improviig actor-critic framework, ARTICULATE- AEETHINYTHHGG automates process previously constrained by intensive human labor and expertise. We hope that our work contributes to bridging the gap between the digital and physical worlds, enabling 3D creators to focus on artistic vision rather than tedious tasks, and paving the way for richer simulated environments that can massively scale up robot learning and interaction. Our current approach relies mesh retrieval to ensure high-quality 3D assets and focus on the problem automatic articulation. Integrating ARTICULATE- AN AN YTHING with an upstream foundation mesh reconstruction model could allow users generate assets, thereby broadening the system's 's generality. Some preliminary results on this direction is included in Appendix A.8.
novel vision-language agentic system
customized
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research partially supported by the Army Research Office under MURI award I9121000-00 the DARPA Triage Challenge under award PRR0011242030 DARPA TIA- MAT HR001 12490421, Institute for Translational Medicine and Therapeutics and National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number ULITR001878, the National Science Foundation under NSF CAREER 2239301, NSF 2331783, and the University of Pennsylvania ASSET center. Any opinions, findings, and conclusion or recommen- dations expressed in this material are those of the authors and do not necessarily reflect the view of DARPA, the Army, the NIH, or the US government.
REFERENCES
Mark Boss, Zixuan Huang, Aaryaman Vasishta, and Varun Jampani. Sf3d: Stable fast 3d mesh recon- struction with uv-unwrapping and illumination disentanglement. arXiv preprint 0tt::4408.005533 2024.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository, 2015.
Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo, Alex Fang, Karthikeya Vemuri, Alan Wu, Dieter Fox, and Abhishek Gupta. Urdformer: A pipeline for constructing articulated simulation environments from real-world images. arXiv preprint arXiv:2405.11656 2024.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning, 2016.
Deemos. Rodin text-to-3D gen-1 (0525) v0.5, 2024. URL https://hyperhhman deemos. com/ rodin.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
Herbert Edelsbrunner, David Kirkpatrick, and Raimund Seidel. On the shape of set of points in the plane. IEEE Transactions on information theory, 29(4)5551-559) 1983.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226-231, 1996.
Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross-category domain-generalizzblee object perception and manipulation via generalizable and actionable parts. arXiv preprint arXiv:22/1.0527777 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 10X1::2106.06655.. 2021.
Ruizhen Hu, Wenchao Li, Oliver van Kaick, Ariel Shamir, Hao Zhang, and Hui Huang. Learning to predict part mobility from a single static snapshot. ACM Transactions on Graphics (Proc. SIGGRAPHI Asia), 36(6))227:1-13, 2017.
Nan Jiang, Tengyu Liu, Zhexuan Cao, Jieming Cui, Yixin Chen, He Wang. Yixin Zhu, and Siyuan Huang. boll-body articulated human-object interaction. In ICCV, 2023.
Zhenyu Jiang, Chheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: is better to track together. arXiv preprint arXiv:2307. 377.07635, 2023.
Dooyoung Kim, Taewook Ha, Jinseok Hong, Seonji Kim, Selin Choi, Heejeong Ko, and Woontack Woo. Meta-object: Interactive and multisensory virtual object learned from the real world for the post-metaverse. arXiv preprint arXiv:2404 17179, 2024.
11

---

Published as a conference paper at ICLR 2025
Jiahui Lei, Congyue Deng, Bokui Shen, Leonidas Guibas, and Kostas Daniilidis, NAP: Neural 3d articulated object prior. In seirth- Conference on Neural Information Processing Systems, 2023. URL https::/oppereviiii wttp:///ooooomiddtt...yyooo
Jiay Liu, Ali Mahdavi-Amiri, and Manolis Savva. PARIS: Part-level reconstruction and motion analysis for articulated objects. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2023a.
Jiayi Liu, Hou In Ivan Tam, Ali Mahdavi-Amiri, and Manolis Savva. CAGE: Controllable Articulation GEneration. 2023b.
Jiayi Liu, Manolis Savva, and Ali Mahdavi-Amiri. Survey on modeling articulated objects, 2024.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023c.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo. Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint a08 10470, 2021.
Zhao Mandi, Yijia Weng, Dominík Bauer, and Shuran Song. Real2code: Reconstruct articulated objects via code generation. arXiv preprint arXiv:2406.0844 0844444 2024.
Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto. Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 2021.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.000 1707.06347, 2017.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean -ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:23122. 1805, 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEF/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real-to-sim-too-rall approach for robust manipulation. Arxiv, 2024.
Yijia Weng, Bowen Wen, Jonathan Tremblay, Valts Blukis, Dieter Fox, Leonidas Guibas, and Stan Birchfield. Neural implicit representation for building digital twins unknown articulated objects. In CVPR, 2024.
12

---

Published as a conference paper at ICLR 2025
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang. Yifu Yuan, He Wang, Li Yi. Angel X. Chang. Leonidas Guibas, and Hao Su. Sapien: A simulated part-based interactive environment, 2020.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint 4rXiv:2404.07191, 2024.
Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo.
Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2863-2873, 2022.
Vicky Zeng, Tabitha Edith Lee, Jacky Liang, and Oliver Kroemer. Visual identification of articulated object parts, 2021.
Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: modular simulation framework and benchmark for robot learning. arXiv preprint wrXXv:2009.122900 2020.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
Groundtruth
Prediction
Joint type
Joint axis
Joint origin
Joint limit
Joint prediction failure visualization. We visualize different types of joint failures, ranging from the most egregious, joint type, to the least, joint limit.
Figure 14:
A.1 LINK AND JOINT PREDICTION ERROR COMPUTATION
A. LINK ERROR
link is represented by a position 3 and quaternion orientation g H., where HH the space of unit quaternions. Given a predicted link state (xp: 9p) and ground truth state (xax qg), we define two error metrics:
Position Error: The Euclidean distance between predicted and ground truth positions:
Cpos
Xxp
Xg
(4)
Orientation Error: The geodesic distance on the unit sphere, computed
Carient arccos([gq. qgl)
(5)
where denotes the quaternion dot product. This metric represents the smallest rotation angle between the predicted and ground truth orientations. The total link error is average of these components.
14

---

Published as a conference paper at ICLR 2025
Code Joint failure attribution.
</>
deff compute... compute__join_.dde jeint_diff_scomttttecc joint_diff, angular_com cutoof, euclidean_ cutoff, angular_lini cutoff, limit_cutaff,
...
return outcone (i.e., 'success" error foilure reason)
if joint_diff[" dili_diff("joiiiii joint_type"] I=
joint type is not predicted correctly
elif "joint_type"
aigular_cuttofi
return "joint_axis"
elif joint_diff[" joint_siff"jjiint.riignn joint_rriin"
return "Joint_or origin"
elif joint_diff[' joint_ Linit_cutaff; return "joint_linit"" "success"
A.1.2 JOINT ERROR
Joint error is computed by comparing several components of the predicted joint state against the ground truth. Let Jp and Jg denote the predicted and ground truth joint states, respectively. The joint error components, from most to least egregious, are as follows:
Joint Type Error: A binary measure indicating whether the predicted joint type matches the ground truth:
{o
if typp(e)))) = type(Jg)
Ctype
(6)
otherwise
Joint Axis Error: The angle between predicted and ground truth joint axes:
ap ag
(aa a, ag
Caxis = min
arccos
arccos
(7)
(app2lagg2
app2 ag
where a E IR is the axis of rotation for revolute joints of translation for prismatic joints, and Caxis E [0, T].
3. Joint Origin Error: For revolute joints, compute the shortest distance between the joint using cross product:
|p (ap ag)
Corigin_pos
aap x ag.
where p Xy is the difference between predicted and ground truth origins, and ap. ag are the predicted and ground truth joint axes. For prismatic joints, we use the Euclidean distance:
Coriiin pos
= Xp X
4. Joint Limit Error: Comprises two sub-components:
(a) Motion Range Difference:
runge m myl2
(10)
where a - 4.), with and (ę being the upper and lower joint limits.
(b) Motion Direction Difference:
mp m.
Elimit dir
(11)
1 mgle
This metric ranges from 0 (identical direction) to 2 (opposite direction), with indicating perpendicular directions.
joint prediction succeeds when all of the joint component predictions are within a tolerance. Otherwise, joint failure is attributed in natural order in Code I where is the most egregious emror joint type - is detected first and the least joint limit last.
visualization of different joint failure types is given in Fig. 14.
15

---

Published as a conference paper at ICLR 2025
Link Pla Placement
Joint Prediction
Pos
Palse Nog
True Pos
Falure Ground Success
Succos
84.42%
1.59%
73.06%
2.39%
False PPs
True Neg
False Pos
True Neg
Falure
5.68%
4.57%
Predicted
Figure 15: Critic-groundtruth confusion matrix. Our visual critic is highly correlated with the ground-truth. The largest disagreement comes from false positive case i.c. the critic falsely declares an incorrect articulation correct. These cases include difficult-o-nnotice errors.
A.2 CRITIC-GROUNTTRUTH AGREEMENT
Fig. 15 visualizes the confusion matrices between the ground-truth and our own critic predictions of success two articulation tasks. Our visual critic highly correlated with the groundtruth. The largest disagreement comes from false positive case i.c. the critic falsely declares an incorrect articulation as correct. These cases include difficult-to-notice errors.
A.3 ROBOTIC TRAINING DETAILS
We train Franka
perform four robotic manipulation tasks in the Robosuite simulator using PPO and our generated assets. The policy outputs joint and gripper positions. We train policies over random seeds per task for 2 million environment steps using PPO in Stable-Buselines library Raffin et al. (2021). We randomize physics (friction, damping, frictionloss ect), objects' scales and poses to obtain robust policies.
A.4 BASELINES
URDFormer requires bounding box for each object part in the input image, which can be obtained a fine-tuned Grounding DINO (Liu et al., 2023c) or an oracle using ground-truth boxes from physics engine. Real2Code requires oriented object bounding boxes (OBBs) as input texts to query a LLM. On PartNet-Mooillity these were obtained using oracle RGB-D images and ground-truth segmentation masks from Blender.
A.4.1 CASUAL INPUTS
Our video-based approach excels with casually captured inputs in cluttered environments while baselines require extensive manual curation: we adjusted DINO bounding boxes for URDFormer and carefully curated foreground segmentation and other hyper-parameters for Real2code. On our project website, we show a diversity of object categories casually captured on an iPhone. The inputs have different view angles and are sometimes inadvertently titled. This kind of inputs presents great difficult for the baselines. For example, Fig. 18 shows the manual adjustment needed for URDFormer. Fig. 17 shows the manually curated foreground segmentation mask and other tuned parameesss to clean up object-part masks and point clouds for real2code.
A.4.2 REPRODUCING REAL22CDE
Real2code LLM Training Real2code has not released their LLM model checkpoint training code. We therefore have done our best to reproduce their LLM model based on their paper description and other code. We obtain the LLM training dataset using their preprocessing code, and finetune
16

---

Published as a conference paper at ICLR 2025

 You are an AI assistant trained to understand 3D scenes and object relationships. Given the following Oriented Bounding Box (QBB) information, your task is to generate list of child joints that describes the articulations be tween object parts.
OBB Information:
Generate list of child joints. Each joint should be described by dictionary with the following keys:
box: The ID of the child bounding box
type: The joint type ("hinge' for revolute joints, 'slide' for prismatic joints)
idx: The rotation axis index (0 for x-axis, 1 for y-axis, for z-axis) example [1, -1]
edge: Edge coordinates on OBB,
sign: Direction of the joint (+1 or -1)
IMPORTANT: Your response must contain ONLY the child joints list, exactly shown below, with additional text before or after:
child_joints =
(box OBB ID], type=[joint typel, idx=[rotation axis index], edge=[edge coordinates], sign=(direction),, additional joints as needed
Generate chind_joints list: 

Figure 16:
Our instruction prompt for realZcode reproduction. Neither training code nor model checkpoint were available from the original work's Github.
the CodeLLama- 7B-raa--B-I-nucu model Roziere et al. (2023) using LoRA Hu al. (2021) with 4-bit quantization described in their paper. The prompt for the LLM is given in Fig. 16.
Real2code In-the wild reconstruction We were not able reproduce the shape completion model based on the released code. Thus, we instead use alpha surface reconstruction Edelsbrunner et al. (1983) from Open3D Zhou et al. (2018) reconstruct meshes from point clouds, and manually the hyper-parameters. We found that this method produces on-par or higher quality meshes from their meshes that were publicly released.
We manually select query points for SAM to ensure near-perfect foreground segmentation masks, tune hyper- parameeersssss for the 3D segmentation scheme (e.g., the input images, number of query points, minimum point-cloud size et cetera), clean up point-clouds, masks and tune mesh extraction. The kinematic-aware SAM model was finetuned using the code from the baseline's authors.
We manually curate the foreground segmentation masks, remove incorrectly segmented point-clouds, and manually parse their LLM outputs. Fig. 17 visualizes the inputs and intermediate outputs from real2code.
tune
A.4.3 EVALUATING URDFORMER
URDFormer predicts one link per bounding box. with each non-root link associated with a joint. To evaluate URDFormer's predictions against ground truth, we employ link matching algorithm that aligns predicted links with ground truth links. Each URDFormer bounding box is matched with ground-trute -trth link by computing the overlap between the box and link's segmentation mask given by the physics engine. For in-the-wild reconstruction, manually corrected the object-part bounding boxes as shown Fig. 18.
17

---

Published as a conference paper at ICLR 2025
Input Image (one of many multi-view images)
Foreground Seg Mask (one of many)
SAM Output (one of many)
Segmented Point Clouds
Segmented Meshes
Figure 17: RealZcode manually curated inputs and intermediate outputs. We used about to 7 input images per object from different views obtain good global alignment from DUSt3R. Foreground segmentation masks were manually curated by querying the base SAM model. The segmented point-clouds and meshes were curated by tuning various segmentation hyper-parametto (e.g., which input images to use, number query points, minimum point-cloud size et cetera) and surface reconstruction parameters (i.e., Incorrectly segmented point clouds manually removed and spurious points were removed using Open3D's radius outlier removal and only keeping the largest connected component from DBSCAN clustering Ester al. (1996) per segmented mask. LLM outputs contain incorrect syntax and extra rambling which were parsed manually.
18

---

Published as a conference paper at ICLR 2025
DINO
Manual
Figure 18: URDFormer manual correction. Comparison of bounding boxes from their fine-tuned DINO and our manual correction.
19

---

Published as a conference paper at ICLR 2025
Text based Methods
Image based Methods
60
60
Articulate Anyding (Text)
Articulate Anything (Image)
48.5
Success Rate (3)
Real2Code Oracle
URDFormer Oracke
40.3
40
40
URDFormer DINO
24.7
20.2
13.5
Figure 19: Comparable Inputs. We compare BETHCULATE-ANYTHIEE with two baselines, Real2Code and UDRFormer using the same input modalities. The ablation done on the cor- responding classes of the baselines.
Table Comparison of average joint prediction errors across methods (lower better). 'Type' shows the fraction of incorrect joint type predictions, 'Axis' error is measured in radians, and substantially outperforms all previous works. uses few-shot prompting and makes no distinction between ID and OOD classes, so we only report results for All Classes. 95% confidence interval is included for classification emror (joint type) and standard deviation for continuous errors (axis and origin).
'Origin' eгror in meters. ARTICULATE-ANY
SEHICULATE-ANTHHIE
All Classes
ID Classes
0OD Classes
Method
Type
Axis
Origin |
Type !
Origin
Type
Origin
Real2Code Oracle
0.537 omm4 1,006 zarzs
0.294 2uar 0.410 1.164 uue
0.344 Loan 0.576 come 0.937 purs 0.272 1171
Oracle
0.581 a.418
#0307
URDFormer DINO
0,4
0,261 20.8
0.547 0.288=ace
0.133 0.582 0.722 =007
0.758 20 0.438
ARTICULATC-AYTTING
a.02 0.141
0.200=0.30
Joint Origin Error
Joint Axis Error
Ereee (inetees)
Erver (rediaa))
Articulate Anything
Real2Code
URDEormer
URDioormer DINO
Articualato Anything
Real2Code Oraela
URDFocmer
URDiorme
Dracke
Oracle
Omce
DINO
Figure 20: Violin plots of the continuous errors. Joint type is binary error and is not applicable for this plot. PETTCULATE-ANYTHIE has the smallest average origin and axis error as presented Tab. 1. For joint axis, we notice two distinct modes the baselines, indicating the errors for in-distribution and OOD data.
A.5 COMPARABLE INPUT MODALITIES
Fig. 19 compares ARTICULATE-ANYTING against the baselines using the same input modalities. We achieve higher accuracy than previous work using the same impoverished modalities by leveraging foundation models' common sense and high-level planning capability.
A.6 ADDITIONAL STATISTICS
Figures 21 and 22 provide breakdown of the success rate by each object category for the link placement and joint prediction tasks respectively. We use the camera pose for all tasks. Small object parts or exotic irregular movement tend to pose greater difficulty ARTICULATE-ANYYHING Further optimizations such zooming in with camera might improve performance. Table 1 provides the raw average errors for our method and prior work with statistical confidences included. Figure 20 further visualizes the distribution of errors via violin plots.
20

---

Published as a conference paper at ICLR 2025
Keyboard
1.00
Box
1.00
Window
0.98
StongeFuniiture
0.98
Laptop
0,98
Refrigerator
0/98
Fan
0298
0.97
Knife
0.95
Switch
0.94
Oven
0.93
Mecrowave
KitchenPot
0.92
Door
Bottle
0.91
Fances
0.90
0.90
TrashCan
0.90
0.90
Remote
0.88
0.86
Table
0.86
0.86
Globn
0.88
Lamp
9.84
Phone
0.83
Eyeglasses
Dispenser
0.82
MachngMachime
0.82
Chair
0.81
Flerss
Display
0,78
CaffesMachine
0.78
0.77
0.86
Suitcase
0.71
Toaster
0.68
0.65
Pen
0.62
Backet
0.58
Chair
0.51
Lighter
Stapler
0.39
Success Rate
Figure 21: PORTICULATE-AYTHHEE link placement success rate by object categories.
21

---

Published as a conference paper at ICLR 2025
Romete
Keyboard
0.95
Microwave
Taptop
0.92
FtorageFumituur
0.88
Table
0.80
Phome
0.80
Window
0.79
KitcheniPot
0.79
Dishwasher
0.73
0.68
MashingMachine
0.67
CocheeMachinee
Toaster
0.53
Mouse
0.52
Switch
0.50
TrashCan
Refrigerator
0:41
Chair
0.41
USB
0.38
Kettle
0.35
0.35
Oven
0.34
Dispenser
0.31
Totlet
0.29
0.22
Lighter
0.21
0.20
Knife
0.13
Safe
0.08
Average
0.75
Display
0.06
Lamp
Backet
0.05
Stapler
0.04
Eyeglasses
0.01
PoldingChair
0.00
Clock
Globe
0.00
Pliers
0.00
Scissars
0.00
Success Rate
Figure 22: ARTICULATE-ANVTHING joint prediction success rate by object categories.
22

---

Published as a conference paper at ICLR 2025
A.7 MESH RECONSTRUCTION
Table Mesh reconstruction quality. Chamfer distance is included (lower better) for different models for in-the-wild results. Best results are bolded, second best are underlined. ARTICULATE- ANYTHING integrated with a mesh generation far better than any other baselines. Even with retrieval, our system still better than all prior works, including Real2Code which uses DUSt3R to do explicit 3D scene reconstruction.
Toilet
Cabinet
Suitcase
Chair
Avg.
Ours (Generation)
0.0637
0.0740
0.0735
0.0698
0.0703
Ours (Retrieved)
0.1133
0.1215
0.0781
0.1210
0.1
Real2Code
0.1192
0.1397
0.0877
0.0987
0.10
URDFormer
0.4191
0.2164
0.1531
0.1278
0.2291
Table Comparison of Chamfer distances across different methods. Lower values indicate better mesh reconstruction quality. Means and standard deviations are included.
Method
Chamfer distance
Articulate-Anynee Articulate-nyything (retrieval)
0.1007 0.062
Real2Code (Oracle)
0.229 + 0.166
URDFormer (Oracle)
0.429 + 0.267
URDFormer (DINO)
0.437 + 0.217
Table includes the mesh reconstruction results for in-the-wild objects in Fig. 7. The ground- truth meshes are captured on a Lidar-equipped iPhone. Window capture glasses well. ARTICULATE AITT-ANNYHING integrated with a mesh generation is far better than any other baselines. Even with retrieval, our system still better than all prior works, including Real2Code which uses DUSt3R to do explicit 3D scene reconstruction. Table 3 compares the reconstruction quality on the PartNet-Mobility dataset. Ground-truth RGBD images were used by Real2Code. When evaluating an object, remove that object from the candidate pool, preventing ARTICULATE-ANTTIING from retrieving the exact same object.
included as Lidar does not
A.8 GENERATING NEW ASSETS
Cabinet
Chair
Suitcase
Toilet
Window
Figure 23: A ARTICULLATE-NNTHINGG with mesh generation. ARTICULLATE-ANYTHINGG ANYTHING can be integrated with mesh generation model to produce high-quality articulated objects. Please see our website for video demonstrations:
gi thub. io/.
Currently, ARTICULATE- ANYTHING employs mesh retrieval mechanism utilize existing 3D datasets. Open repositories such as Objaverse (Deitke et al., 2024), which contain over 10 million static objects, offer rich source of assets that our system can bring to life through articulation. However, generate
more customized assets, promising future direction is to leverage large-scale mesh generation models.
In this section, present preliminary results toward this goal. Given visual inputs to ARTICULATE- ANYTHING such as videos or images we first extract single image of the target object. A mesh generation model is then used to produce 3D model of the object. 24 compares the mesh quality produced by three models: Rodin (Deemos, 2024), Instant Mesh (Xu et al., 2024), and (Boss al., 2024).
23

---

Published as a conference paper at ICLR 2025
We select Rodin for its high-quality output. Subsequently, et al., 2024) to obtain a 3D segmentation of the object into parts. Unconditional segmentation is unreliable because object parts may be under- over-segmented, depending the task. To address this, we condition the segmentation on the video inputs by instructing a VLM to identify the moving part, which serves the segmentation target. We then render the 3D model from multiple camera views. For each view, we apply Grounded SAM, which first obtains bounding box for the object part and then runs SAM obtain fine-graide greinge segmentation mask. Figure 25 visualizes the 2D segmentation masks for each view.
Grounded Segment-Anytnn (Ren
Source image
Rodin
Instant Mesh
Stable-Fast. Fast-3D
(a) front
(a) front
(a) front
(b) side
(b) side
(b) side
Figure 24: Mesh generation. Comparison of 3D mesh generation from single source image using different models. Each model's s output is shown from the front and side views.
Figure 25: Grounded SAM. A 2D segmentation mask is obtained for each view of object using Grounded SAM. For each view, the input RGB image is shown on the left, and the resulting mask overlaid in red is shown the right.
The 2D masks are lifted into 3D via projective geometry. For each view, the camera parameters and depth maps to project the 3D points into pixels. Then, the 2D segmentation masks are queried to obtain segmentation labels for the 3D points. We merge these segmented 3D points across all
24

---

Published as a conference paper at ICLR 2025
views to obtain a complete 3D segmentation of the object into its semantic parts. The potentially sparse segmented point clouds are used to segment the dense meshes via nearest neighbor matching. MAACCLLAEEAANTHIIE
With the 3D object now properly segmented, we can apply resulting 3D objects not only possess high material quality that matches the ground truth but also exhibit realistic movement when articulated in simulation. Please refer to our website for the video demonstrations.
before. The
25


Reviews:
[{"review_id": "YgofRilvka", "reviewer": "Reviewer_SpUi", "summary": "This paper presents a method for generating simulated 3D objects, with moving parts, that can be inferred with from multi-modal inputs (text, images, videos). This is interesting because is suggests a method of inferring models of objects, so that (robot) agents can be trained on richer and more representative simulations for things that exist in the real world. \n\nThe method leverages LLMs to generate URDF descriptions (effectively code) which can then be iteratively refined based on feedback from a critic module. This is a neat use of LLMs that follows a trend in ML, but is well applied here.\n\nSuccess is primarily evaluated by taking ground-truth URDF descriptions from an existing dataset (PartNet-Mobility), masking out information in them, and then comparing the reconstructed URDF with the ground-truth, using a threshold tolerance to define \"success rates\".", "questions": "In addition to the questions implied above, could I also ask if you experimented with weighting the losses in Eq 1? \n\nIn section 5, on page 6, you also write that \"Success requires all criteria to be within a small threshold... The position threshold is set to 50mm and the angular threshold to 0.25 radian (∼ 14.3 degree).\" \n\nHow did you choose these thresholds? Some defence of or rationale for these choices would be helpful, unless they are arbitrary?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper does a lot of interesting things. Most of all, the idea of using LLMs to generate code is well applied here, and carried through evaluations that lead to at least proof of concept robot applications (though I believe still in simulation).\n\nTo the best of my knowledge, this is an *original* and *significant* contribution. For example, I don't know the prior work well, but the baselines sound reasonable.\n\nThe *quality* and *clarity* of the paper is decent as well, though could be improved (I have some suggestions below).\n\nThe multi-modality of the approach is also notable, and I appreciated the ablation of the different modalities (section 5.1).", "weaknesses": "Three weaknesses in the paper stood out but could be improved in revision. \n\nThe first is relatively minor, but there seemed to be missing references to other works that have used LLM agents in an actor-critic setup. \n\nOne paper that came to mind was the Text2Reward paper (https://arxiv.org/abs/2309.11489v3). But more generally, it would help to emphasise that this general idea - which is very cleverly used here - is not part of the paper's novely.\n\nThe second weakness is a lack of statistics and confidence intervals in the results. \n\nFor example, Table 1 shows average joint prediction errors and it is possible to see differences between these averages. But without statistical testing (e.g. F-test or t-tests) it is hard to appreciate how meaningful these differences are. Some measure of variance would be useful here too (e.g. +/- standard deviation, to appreciate the consistency of the different models). In general, plotting more than summary statistics (e.g. averages) would be much stronger (e.g. violin or raincloud plots).\n\nRelatedly, the Figure 5 caption says that the approach \"significantly outperforms all baselines\". But as before, there are no statistical tests to back this up - just comparative %success rages without confidence intervals. I agree that the proposed model results appear better, but without a sense of the confidence intervals it's hard to be certain. In this case, Chi-square tests (if the samples are large) or Fisher's exact tests (for smaller sample sizes) might be appropriate.\n\nThe final weakness might be harder to address as it concerns fitting all of the main results into the main body of the paper. \n\nTo me, it was most notable that the application for robotics section felt incomplete and even confusing. I was not able to properly evaluate what was done or whether it was good based on the short section, which is a shame because it is a very interesting culmination of what the paper builds to. The paper also frames this as a key result, but then doesn't quite get all the way there.\n\nOf course I appreciate that there are page limits, and I see that there are additional details in an appendix. But it would be better to fit more of that into the main paper for completeness.\n\nSimilarly, the in the wild results seemed incomplete in the main text as well, though I see there are more details in the appendix.\n\nUnfortunately I don't have any great suggestions on what you could cut to make room for these things. I suppose you might play with resizing some of the figures, cutting words where you can, and reformatting carefully.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NHh1JTtVhB", "reviewer": "Reviewer_bBwZ", "summary": "*Articulate-Anything* is a system that automates the articulation of complex objects from various input types, including text, images, and videos. It utilizes vision-language models (VLMs) to generate code that creates interactable digital twins compatible with 3D simulators. The system incorporates a mesh retrieval mechanism to leverage existing 3D assets and employs an actor-critic approach to iteratively propose, assess, and refine articulations, self-correcting to enhance accuracy.", "questions": "1. What types of inputs does this work use when comparing to prior methods? Are these inputs—such as text, images, or videos—consistent with those used in previous works?\n\n2. In terms of object retrieval, since this work aims to create digital twins from real-world observations, does the generated articulated object accurately resemble the real-world counterpart? Given that retrieval relies on CLIP scores based on text descriptions, the generated objects might be different from the original objects.\n\n3. Could accuracy be reported across different object categories? Performance might vary significantly by category; for instance, simpler objects like windows and storage furniture may be easier to generate, whereas complex items like toilet seats or shopping carts could present more challenges.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Compared to previous works, this approach can begin with various inputs, including text, images, and videos.\n\n2. By using iterative refinement with an actor-critic model, this method generates articulated assets with greater accuracy than prior approaches.\n\n3. It utilizes vision-language models (VLMs) to perform the task without requiring network training or fine-tuning, eliminating the need for large-scale datasets, which are challenging to obtain for articulated objects.", "weaknesses": "1. While VLMs hold the potential to go beyond existing dataset, this method retrieves parts from those datasets. Thus, it is still constrained by existing datasets.\n\n2. The framework is mainly prompting VLMs to construct articulated objects, with limited technical contributions. And the iterative refinement phrase only raise limited accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ms4RTBrURm", "reviewer": "Reviewer_yrhN", "summary": "This paper proposes Articulate-Anything, a closed-loop vision-language actor-critic system that is capable of converting real objects (provided via image, video, or text descriptions) into Python codes and further compiles them into URDFs. The proposed approach is evaluted against baselines of previous proposes and noticable improvements over success rates of link placement and joint prediction are achieved.", "questions": "1. According to Fig.3, a template object is retrieved when visual input is provided. How do the follow-up link prediction and joint prediction steps work on template objects? I assume that the link and joint information of the template object are already provided in PartNet-Mobility dataset.\n\n2. What are the 'dimensions' given by the Layout Planner based on? How to ensure that the dimensions of multiple object parts given by the Planner are not contradictory?\n\n3. Using VLMs to directly output numerical values (dimensions, angles, etc.) seems not a good idea for embodied applications which have high demands on safety and reliability, as studies have shown that LLMs are not strong enough to process numerical operations.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated. Understanding the structure of articulated objects is important for vision and robotics tasks.\n2. The idea of closed-loop iterative refinement is interesting and technically sound.\n3. The approach is able to handle diverse input modalities, including texts, images, and videos.\n4. It is a good idea to use structural representations like code to model articulation structures.", "weaknesses": "1. It is not explained how the retrived part meshes do not conflict with each other. For example, in Fig.4(a), the code fragments do not show mechanisms preventing the 'door' having collisions with the drawer in the 'furniture_body'. In other words, how is the detailed position of the 'door'  (including its height) determined?\n\n2. Some technical details are not clear. For example, the target affordance extraction part in Sec.4.4 and how link placement / joint prediction works for text / visual inputs, as the mesh retrieval results of text / visual inputs are different. A pseudo code can be help.\n\n3. It seems that the method and experiments do not support well the task studied in this paper. According to Sec.3, the task is consistent with a formulation of reconstruction task. However, the proposed approach reterives similar meshes from a dataset rather than reconstruct the mesh in Sec.4, which cannot ensure the consistency between the retrieved mesh and the real object part and may result in large L_mesh. And the experiments just quantitatively evalutate the accuracy of link placement and joint prediction. Evaluations with important metrics like CD for reconstruction are not provided.\n\n4. The robotic application is not solid enough. The test results like the success rate of different manipulation tasks are not provided, making it difficult to give a proper evalulation on the effectiveness of the generated assets. And additional real-world experiments will make this point stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PhSfizq3Xh", "reviewer": "Reviewer_VbEW", "summary": "This paper presents a system designed to automate the articulation of complex 3D objects for applications in AR/VR, animations, and robotics, addressing the challenge of extensive human effort required in traditional methods. It utilizes vision-language models (VLMs) to convert inputs such as text, images, and videos into interactable digital twins for 3D simulators. The system incorporates a mesh retrieval mechanism from existing 3D datasets and actor-critic frameworks to iteratively refine object articulation, achieving robust results.", "questions": "1. Could the authors provide examples illustrating how the performance of link placement and joint prediction evolves over the iterations?\n\n2. Does the number of prompted examples affect the pipeline's performance? It would be helpful if the authors could provide an ablation study on the number of prompted items to determine whether performance varies based on this factor.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed methods for automatically modeling articulated objects address an interesting and meaningful topic, with potential to greatly benefit fields such as robotics.\n\n2. The techniques introduced are well-founded. By leveraging vision-language models (VLMs), which encode valuable information about object affordances, the system effectively determines common articulations for various objects.\n\n3. The paper provides a detailed analysis of failure cases, as shown in Figure 8, highlighting how the proposed pipeline outperforms other methods and identifying areas for further improvement.\n\n4. The paper is well-written and easy to follow, making the concepts accessible to readers.", "weaknesses": "1. The proposed methods have only been validated on limited types of articulated objects (i.e., prismatic and revolute). When the types become more complex, such as cylindrical or universal joints, the current pipeline may struggle to handle these cases.\n\n2. The pipeline has only been tested using Google's Gemini VLM. It is recommended to conduct ablation studies to assess whether the inference performance varies with different VLM architectures, in order to evaluate the system's robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JtK2E8f4tf", "reviewer": "Reviewer_Bhiq", "summary": "This work proposes a retrieval method for generating articulated objects in a simulator, taking into account both link and joint placement. It also demonstrates promising results in partner mobility", "questions": "- Is there any failure case analysis? For example, under what circumstances does the retrieval process select the wrong mesh? It would be helpful to show when retrieval might fail and how challenging those cases are.\n\n- Concerns regarding Section 4.2:\n  1) For instance, in Fig. 4a (left example), 'furniture body'  is place to the left relative to the 'door'. Since the body has thickness, why is the door placed on the left side of the front surface rather than the back? As the description only defines 2D spatial relationships, it's unclear how to position the link in 3D space.\n  2) How many candidates does the actor's output generate? Can the relation fully cover all possibilities?\n\n- Regarding Section 6's robotic experiment, where do the generated assets come from? Does Articulated Anything input video or images from RoboSuite to generate assets, and then train PPO with the generated assets, instead of directly using the assets in RoboSuite?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The proposed method for generating digital twins of articulated objects is intriguing and highly effective, significantly enhancing performance during evaluation. The website is comprehensive, offering visualizations and related code.", "weaknesses": "Please refer to 'Questions' section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)