Real2Code: Reconstruct Articulated Objects via Code Generation
Authors: Zhao Mandi
Yijia Weng
Dominik Bauer
Shuran Song
Venue: ICLR 2025 Poster (2025)
Keywords: articulated objects
code generation LLMs
foundation models
Primary area: foundation or frontier models, including LLMs

Abstract: We present Real2Code, a novel approach to reconstructing articulated objects via code generation. Given visual observations of an object, we first reconstruct its part geometry using image segmentation and shape completion. We represent these object parts with oriented bounding boxes, from which a fine-tuned large language model (LLM) predicts joint articulation as code. By leveraging pre-trained vision and language models, our approach scales elegantly with the number of articulated parts, and generalizes from synthetic training data to real world objects in unstructured environments. Experimental results demonstrate that Real2Code significantly outperforms the previous state-of-the-art in terms of reconstruction accuracy, and is the first approach to extrapolate beyond objects' structural complexity in the training set, as we show for objects with up to 10 articulated parts. When incorporated with a stereo reconstruction model, Real2Code moreover generalizes to real-world objects, given only a handful of multi-view RGB images and without the need for depth or camera information.

Published as a conference paper at ICLR 2025
REAL2CODE: RECONSTRUCT ARTICULATED OBJECTS VIA CODE GENERATION
Zhao Mandi', Yijia Weng', Dominik Bauer", Shuran Song' 1 Stanford University Columbia University
ABSTRACT
We present Real2Code, novel approach to reconstructing articulated objects via code generation. Given visual observations an object, we first reconstruct its part geometry using image segmentation and shape completion. We represent these object parts with oriented bounding boxes, from which a fine-tuned large language model (LLM) predicts joint articulation as code. By leveraging pre-trained vision and language models, our approach scales elegantly with the number of articulated parts, and generalizes from synthetic training data to real world objects in unstruc- tured enviromments. Experimental results demonstrate that Real2Code significantly outperforms the previous sfate-of-the fi-thee-rt art in terms of reconstruction accuracy, and is the first approach to extrapolate beyond objects' structural complexity in the train- ing set, as we show for objects with up to 10 articulated parts. When incorporated with stereo reconstruction model, Real2Code moreover generalizes real-world objects, given only a handful of multi-view RGB images and without the need for depth camera information.
INTRODUCTION
The ability reconstruct real-world objects in simulation (real-to-sim) promises various downstream applications: automating asset creation for building VR/AR experiences, enabling embodied agents to verify their interaction in simulation before execution the real world (Lim et al., [2022] Wang et al.) 2023a: Torne et al., (2024). or building large-scale simulation environments that support data-driven policy learning (Katara et al., 2023). We are particularly interested in articulated objects, for both their ubiquity in household and industrial settings and the unique challenges they pose in contrast to single-body rigid objects. To reconstruct articulated objects. prior learning -based methods typically train supervised (Jiang et al. 2022b) or test-time-optmmzeet (Liu et al.) 2023a) models on synthetic objects with simple articulation structures (i.e., one or two moving parts per object). This results in limited generalization ability to objects with more complex visual appearances and kinematics. In addition, prior work only provides object part reconstructions of limited quality: the extracted meshes often incomplete and the predicted articulation parameters require manual cleanup before being usable for simulation.
We propose Real2Code, novel approach to address the above limitations. We represent object articulation with code programs, and use language modeling to predict these code programs from visual observations. This formulation scales elegantly with objects' structural complexity: to process an articulated object with multiple joints, prior methods would require either changing the output dimension of their articulation prediction model, or run multiple inferences on pairs of before- and after- interaction observations to predict one joint at time. In contrast, the next-token prediction formulation in language modeling allows generating arbitrary-length outputs, i.e., the model archi- tecture needs no adjustment to handle varying number of object joints. Whereas prior work on shape programs (Tian et al. 2019) needs to define task-specific code syntax, we represent objects with simulation code in Python, which takes advantage of recent progress in large language models (LLMs) that are pre-trained with code generation capabilities.
Although capable at code generation, LLMs pre-trained on text are not equipped at predicting accurate numerical values from spatial geometry information, which is required in our task in order
Project Website: https://rea11 12cdea.aaccceeeoo thub. 1

---

Published as a conference paper at ICLR 2025
Real2Code
In: Unstructured RGBs
Out: Interactable Digital Twins
We propose novel method for reconstructing articulated objects via code generation leveraging pre-trained large language models (LL.Ms). Real2Code takes visual observation input and performs both part-level geometry reconstruction and joint prediction. When evaluated extensive set of real and synthetic objects with varying levels of kinematic complexity (up to 10 parts), Real2Code demonstrates high reconstruction accuracy and generalizes to real world objects from a handful of pose-free RGB images.
Figure 1:
to obtain articulated joint parameters. To address this, propose to use oriented bounding boxes (OBBs) as an abstraction layer that summarizes the raw sensory observation to the LLM a concise yet precise manner. Given partial observations of object, we first perform part-level segmentation and reconstruction via a combination of 2D segmentation and 3D shape completion model; next, OBBs extracted from the reconstructed object parts, and serve as input to the LLM. The LLM then predicts joints as a classification problem by selecting the closest OBB rotation axis and box edges.
In unstructured real world environments, another challenge is the lack of accurate depth and camera information. To address this, we incorporate a pre-trained dense stereo reconstruction model, namely DUSt3R (Wang et al. 2023b), into our pipeline: we show the dense 2D-to-3D point-map prediction from DUS(3R can be combined with our fine-tuned SAM model achieve view-consistent 3D segmentation. As result, Real2Code can then reconstruct real world objects from only handful of pose-free RGB images.
For more systematic evaluation, we validate the performance of Real2Code on the well-established PartNet-Mobility dataset (Mo et al., 2019), using an extensive test set unseen objects that contain various numbers of articulated parts. Compared to the prior state of the art, Real2Code significantly improves both 3D reconstruction and joint prediction accuracy. Real2Code the only approach to reliably reconstruct objects with more than three articulated parts, whereas prior methods fail completely on such objects. Fig. highlights our results on both synthetic multi-par objects (first column), where we show Real2Code reconstructs both synthetic objects with up 10 parts (first column) and real-world objects (second column) using in-the-wild RGB images.
In summary, contributions threefold:
We present Real2Code, a novel approach to reconstructing articulated objects from a handful of unstructured RGB images. We formulate joint prediction code generation problem and adapt pre-trained large language models to specialize in this task.
2. We address part reconstruction via kinematic-aware new-consistent image segmentation and learned 3D shape completion model, which leads to high-quality mesh extraction that generalizes to multi-part real-world objects.
3. Empirical results demonstrate that Real2Code significantly outperforms the prior state of the art both articulation estimation and part reconstruction. To the best of our knowledge, Real2Code is the first method to accurately predict objects with more than three parts, and generalizes beyond the training dataset with at most parts objects with up to 10 parts.
RELATED WORK
LLMss for Visual Tasks. Pre-trained LLMs have been used for visual reasoning and grounding tasks (Zeng et al. 2022; Hsu et al.] 2023b). LLMs' onde-generation capability has also been
exploited for generating programs that solve visual tasks (Gupta & Kembhavi 2022: Surís et al. 2023. Subramanian et al., 2023). These works use zero-shot pre-trained LLMs such as GPTs (Brown al. 2020 OpenAl 2023) and require prompt engineering, such providing in-context examples,
2

---

Published as a conference paper at ICLR 2025
to guide the model to generate desirable outputs; in contrast, we directly fine-tune the weights of sode-generation model to specialize in our articulation prediction task without prompt tuning.
Shape Programs. Code-like programs have been studied in computer vision compact rep- resentation for 2D and 3D shapes. A main challenge for learning code programs is the lack of supervision, and prior work has explored using learned differentiable code executor (Tian al. 2019), pseudo-labeling (Jones et al.] 2022). differentiable rendering (Liang) (022). imitation learning on code sequences (Willis et al.) 2021). or reinforcement learning (Tulsiani al. (016). More recent work has explored constructing large-scale datasets of shapes (Ganin et al., (021) or scene layouts (Avetisyan al. 2024) and train supervised LLM-like models generate code outputs. In contrast to ours, these proor works focus on either individual object shapes or scene-level room layouts, but do not estimate joint articulations. austomarily-desigeee language syntax (Tian et al., 2019 Jones et al. [022] Avetisyan et al. (2024) or Computer-Aided Design (CAD) code (Willis et al. [022] Ganin et al. 2021). we represent object articulation with Python code that 1) closely matches the pre-training distribution of code-generation LLMs, which allows fine-tuning with limited data, and 2) can be directly executed by a physics simulator (Todorov et al. 2012), which makes the reconstruction more usable for simulation and requires less manual cleanup.
addition, instead of the task-speciffe code programs, such as
Articulation Model Estimation. Prior work has investigated estimating pose and joint properties of articulated objects without full reconstruction. A common setup is to assume physical interactions on an object to infer its articulation information: classical sampling-base based algorithms (Huang et al. [014] Katz et al. (013) are proposed to estimate joint parameters based on sensory inputs from an object's different configuration states; other learning-basee methods train end-to-end models to predict part- level segmentation, kinematic structure, object part poses, or articulated joint parameters (Hu et al.
2017: Yi et al., 2018; Wang et al.] 2019 Michel et al., 2015; Li et al. 2020 Zeng et al. [222 I Huang et 22.2 seng et al. 2022, Rabdull-Rsshin et al. [202]] Jiang et al. 2022a; Liu et al. 2023b). [Buchanan et al. (2023); : Heppert et al. (2022); Sun et al. (2023) propose specialized neural
network architectures to improve the estimation performance. Other works focus on learning to
propose the most informative physical interactions on an object to help robot manipulation (Mo)
al. 2021). or to better isolate and segment articulated parts and joints (Gadre et al. (02I). These articulation estimation tasks provide useful metrics for 3D shape reasoning (Wang et (209), and
Liu al. [2022); An al. (2023); Geng et al. (203bb a) show that the predicted object pose and joint information are useful for robotic tasks. However, prior work typically handles objects with simple structure (i.e., one or two moving parts) and does not address full object reconstruction. In contrast, our method handles objects with more than ten moving parts, and performs shape reconstruction via extraction of part meshes.
Articulated Object Reconstruction. Most closely related to ours are methods that reconstruct both the geometry and joints of articulated objects. A popular approach is train end-to-end models
on synthetic data to jointly segment articulated parts and predict joint parameters, assuming either
observations from interactions (Jiang et al. 20226; Hsu et al.] 2023a| Nie et al.
2023
Mu et al.
2021) or single-stage (Heppert et al. 2023 Irshad et al.] 2022; Kawana et al. 2022 Wei et al.
2022) observations. Another approach uses per-object optimization (Liu et al. 2023a b) without
training. Based on observations of the object in two or more different states, it optimizes for joint parameters to match observed motion correspondences and optionally performs 3D reconstruction using learned neural fields. Most existing methods assume single joint and do not scale well with an increasing number of joints: for example, to handle an object with N joints, methods like Ditto (Jiang et al. 2022b) would need to move the N joints one by one, record the observations before and after each interaction, and run N inferences on each observation pair. PARIS (Liu et al., 2023a) would need to optimize N neural fields and joint parameters, which may lead to much more complex optimization landscape. The approach presented by Liu et al. (023b) handles multiple joints but requires a complete sequence of point-cloud observations and is not able to reconstruct 3D shapes.
3 METHOD
We address the problem of reconstructing multi-part articulated objects from visual observations. An articulated object is composed of a set of rigid-body parts that are connected via joints. We assume that joint types are either prismatic or revolute: a prismatic joint is parameterized by joint axis u" E R and a translation offset d: a revolute joint is parameterized by a position p" e R, a rotation axis n² R33 and rotation angle 0. For object with N moving parts, assume each to be
3

---

Published as a conference paper at ICLR 2025
Pose-free RGBs
Segmented Part Point Clouds
Part
Oriented
Code output
Meshes
Bounding Boxes
DUSt3R + Fine-tuned SAM
Part-level Shape Completion Model
Fine-tuned
CodeLlama
Figure 2: Overview of Real2Code pipeline. Given unstructured multi-view RGB images, we leverage the pre-trained DUS13R model [Wang et al.] 2023b) to obtaín dense 2D-to-3D pointmaps, and use a fine-tuned 2D segmentation model(Kirillov et al. 2023) to perform part-level segmentation and project segmented 3D point clouds. We train a shape-completion model to take partial point cloud input and predict a dense occupancy field, which is used for part-level mesh extraction. We fine-tune large language model (LLM) (Rozière et al. 2023) that takes mesh information in the form of oriented bounding boxes, and outputs full code descriptions of the object that can directly be executed in simulation.
connected with its parent via exactly one 1-DoF joint. Therefore, the transformation between each part's frame and its parent's frame is uniquely determined by the joint parameters. Furthermore, for hinge joints, we focus on objects what have joint position lie closely with one of its oriented bounding box (OBB) edges- we remark that this true for many common household objects with cuboid shapes, such as doors, boxes, laptops, etc.
To obtain visual inputs, we assume an object is manipulated such that each articulated joint is at non-zero state, i.e., d > 0 or 0 > 0, when we capture multi-view RGB (and optionally depth) images. Our system outputs set of 3D meshes- each reconstruction of the object's parts and list of joint types and parameters represented as code. The outputs can then be used to create the object's digital twin in simulation for downstream applications.
Fig. 2 provides an overview of our method. Real2Code consists of two main steps: reconstruction of object parts' geometry (described in Sec. 3.I] and joint estimation via LLM code generation (described in Sec. 3.2 Between the two steps, the oriented bounding boxes (OBBs) of the object parts serve as an abstraction layer, enabling the LLM to reason about 3D spatial information and predict accurate joint parameters.
3.1 PART RECONSTRUCTION
To reconstruct an object's part-level shapes, we propose a 2D-to-3D approach that is antegory-agnostic and handles objects with arbitrary number of parts. First, we fine-tune SAM model that generates 2D segmentations from RGB images, and project them to partially-observed 3D point clouds. Next, train shape completion model that takes 3D point cloud input and extracts watertight meshes.
3.1.1 KINEMATICS-
PART SEGMENTATION
We leverage the pre-trained 2D segmentation model SAM (Kirillov et al., 202.3) to segment object parts based on their kinematic structure. This design is motivated by the need to generalize real world data, and 2) scalability to the number of object parts. In contrast to prior works that train 3D segmentation models using limited amount of synthetic data (Jiang et al. 2022b Mo et al. 2019 Xiang et al. 2020), SAM (Kirillov et al. 2023) pre-trained a much larger dataset. Therefore, SAM generalizer better to in-the-wild real world images, with a strong prior to identify moving object parts without the need for multi-step interactions.
However, because SAM (Kirillov et al. 2023) is originally designed for iterative user prompting. its zero-shot predictions do not always match the articulation structure, e.g., segmenting unnecessary de- tails on an object part. To address this, we fine-tune the pre-trained model using BartNer-Mobilgg (Mo) al., 2019: Xiang et al. 2020) data: the model's heavy- image encoder is kept frozen, update the lightweight prompt-decoder layer to take an image and one sampled 2D point prompt as input, and predict the correct corresponding mask. See appendix A.3 for more details.
4

---

Published as a conference paper at ICLR 2025
3.1.2 TEST-TIME PROMPTING FOR VIEW-CONSISTENT SEGMENTATION.
The point-based segmentation de- scribed above scales easily with the number of the object parts. However, this formulation also inherently lacks view consistency, as SAM is unaware of the correspondences across differ- ent camera views. To address this, we introduce test-time prompting pro- cedure to project predicted 2D masks into view-consistent 3D segmenta- tion. We discuss two different input settings based on the availability of depth and camera data: 1) Multi-view RGB-D and Camera Input: we first coarsely sample 2D points on each RGB image and run the SAM model to obtain the background masks. This allows us to segment the foreground object in the different views and sample 3D points uniformly on the object point cloud. Next, we project each such 3D point back onto each image, and obtain view-consistent 2D points for SAM prompting. Further, we rank the model's predicted masks based on the confidence and stability scores proposed by Kirillov et al. (2023), and filter them using non-maximum suppression (NMS) to produce the final 3D segmentation. 2) Multi-view Unstruc- tured RGB Input. To handle real world settings which often lack high-quality depth and camera information, we adopt a multi-view stereo reconstruction model to achieve part segmentation. We use the recently proposed DUSt3R [Wang et al. 2023b) model, which is pre-trained to predict dense 3D point-maps from RGB input images. We then sample 2D points from one RGB image and find each point's corresponding point in every other RGB images via nearest-neighhor. More details described in appendix A.4 This overall procedure of projecting between 3D to 2D prompting is similar SA3D (Cen al. 2023), which samples on a NeRF(Mildenhall et al. 2020) field and uses inverse rendering to effectively prompt SAM in 3D.
SAM
Fart Partt Chout
Vanvell
SAMI Jo
Somppled in Pernsoo Coyect Suriace
Figure 3: View-consistent segmentation. Illustration of method test-time prompting the fine-tuned SAM model. first sample 3D points from the foreground object point clouds, project each point onto 2D RCB images captured from different camera views, which are used prompt the model to generate view-consistent segmentations.
3.1.3 PART-LEVEL SHAPE COMPLETION.
Due to frequent self-occlusion, e.g. the inside of a drawer is often not visible, RGB-D input does not provide full observation of each object part, and subsequently a segmented point cloud does not recover complete part shape. This motivates learning shape completion model to obtain watertight meshes. Because part-segmentation already handled in the previous step, we here tackle shape completion the object part level. We build on top of Convolutional Occupancy Nets (Peng et al., 2020): the model architecture consists of PointNet++(QU et al., (017)) point-cloud encoder, followed by 3D Unet (Özgün Çięek al. 2016) encoder and linear MLP decoder that predicts logits for occupancy. We use the ground-trun part meshes from PartNet-Mobility (Mo et al.) 2019) to generate a dataset of partial point cloud inputs and occupancy labels. We normalize the occupancy grid using partial OBBs extracted from the input point cloud to avoid under-fitting the smaller-sized meshes. Marching Cubes (Lorensen & Cline 1987) is used extract the completed part meshes from predicted occupancy. See appendix A.3 for more details.
3.2 ARTICULATION PREDICTION VIA CODE GENERATION
Given a set of segmented object parts, we next predict their articulation structure using LLM-based code generation. This approach yields several advantages: first, code offers compact representation for joints, and when combined with L.MM's ability to predict arbitrary-length outputs, it scales elegantly with the complexity of object kinematic structure; second, pre-trained L.LMs are equipped with strong priors for both common-sense objects and for generating syntactically correct code, making them easily adaptable to our task; lastly, the LLM-generated code can be directly executed in simulation, removing the need for manual cleanup of predicted joint parameters as seen in prior work (Jiagg et al. 2022b). The following sub-sections first introduce our formulation of predicting joint parameters from oriented bounding boxes, then discuss our LLM tne-tuning procedure.
5

---

Published as a conference paper at ICLR 2025
atoresef
9if contect: i 255 25, 06.-0.120) 111 6.6 6.0.88888.0000 2.d1, 6.0.01 www. "oxcesss" (E 2,24,,22,,000000 8.891,,
1:f contert i 1e 13 (y6.191, W00. (..6,88.
(w.8 "oxcennes 29,1.23,,3..
mode1 esor Hooote conert obdeet"
model. asset adel heshh" nomee "Stnk_.", ttte="tinn_a... 8o0013
aodel. ascat adot traesh momee "Stnk_
Extract OBBs
aody_root moned www.ttoooyy..d.ooo...o. (baayy..
nament cost!)
CodeLlama
Root_geoe
enild_fsint diettbor=], 12ked, Expe- hingo" jedgo=(s, signnw),
Select OBB axis & edge
meshw""lisk
ior, doosiigh, jolet_typpl. Helper Function
003_edte Antectiree arrayicto.rdet) ito
point.pos abbl center") (181) lotbuadge sthil 'eccents') cos_ards onnd oond's'1Es, MoLaxis__oo)
Sofnt_axts
stray (ost_aasss aavss dthusten.
Link aody pacent body. and "acay"y nameer "Link, (wesh, (c)")
Link acdy. cost "sece". typen weah?. nanpet "Link, (nean 1a): gece", nesher "irk (waah ie)")
Lirk_aody 2d01 "oint"" 12m0-f" jolitt Snesht_ Ineshh pos-foirt_ass, axis-jidnt,axss, type-3oint.tyee, conge-Es, 11)
soant chald. Jeants:
Compute Joints
ads_bosy, www.ayy.no_.fotiioooooooooo
thavestter_dd), leln jelntt"asee") jelnetreten)), sotes["eyae"))
Figure 4: Articulation Prediction as Code. We fine-tune a Codellama (Rozière al. 2023 model that takes in oriented bounding boxes (OBBs) for segmented parts as input, and generates joint predictions via selecting OBB rotation axes and edges (model generation is highlighted in green). helper function is used to compute the absolute joint axis and position that assembles the object parts in simulation
3.2.1 ORIENTED BOUNDING Box AS INPUT ABSTRACTION.
Articulation prediction requires numerical precision at joint parameters (i.e., position and rotation) and reasoning from raw sensory input, but an LLM pre-trained on text is not adept at these challenges. We address this by representing the sensory input (object point clouds) boxes (OBBs), each representing a segmented and completed object part. Compared to alternative object representations such as 3D point clouds 2D images, OBBs strike balance between compactness (i.e., do not require an extra feature encoder) and preciseness (i.e., provide numerical 3D pose information). Further, OBBs provide a reference for joint information. Recall that the pose of an object part is determined by its 1-DoF joint at a non-zero state we can hence recover joint parameters from the observed displacement of object parts. Given an OBB of a part connected to its parent, the joint axis will be parallel to one of the three axes of the OBB's 's rotation matrix regardless of its joint type. We observe many common articulated objects consist of cuboid-like parts (e.g. doors or laptops). hence the position of their corresponding revolute joints will lie closely to, if not overlap with, one of the OBB edges. Combining these observations, we re-formulate the joint axis prediction problem by selecting OBB rotation axis the joint axis and, for revolute joints, choosing OBB edge parallel to the axis as the joint position. See Fig. 4 for an illustration.
set of oriented bounding
3.2.2 FINE-TUNING A CODE GENERATION LLM.
We now have an input formulation that effectively converts a regression task (i.c., predicting contin- uous values) to casier classification task (i.c., selecting axes and edges) for LLMs. We use the 7B-CodeLlama (Rozière et al., 2023) model for its ppen-source-availabbile
and built-in priors for code generation. We construct fine-tuning dataset using PartNet (Mo al., 2019) objects (the same assets used to generate our segmentation and shape completion data), and convert the native URDF files into MJCF code TTunyasuvunakool et al. 2220). which in the more compact Python syntax, 2) can be executed in MuloCo [Todorov et al., 2012)) physics simulation, and 3) has each object's s joints assigned with respect the corresponding part's OBB information. The LLM takes list of part-OBB information (i.e., center, rotation, and half-lengths) as input, and outputs joint predictions as list, where each line contains indices into the axes and edges of an OBB. More details can be found inappendix A.3
EXPERIMENTS
We evaluate Real2Code and compare to baseline methods to validate the effectiveness of our approach. Sec. 4.2 describes experiments on our kinematics-awar 2D image segmentation and 3D shape completion models. See. evaluates our fine-tuned code-generation model on articulation prediction.

---

Published as a conference paper at ICLR 2025
Category Number of Parts
Laptop 2
Box
Fridge 2-3
Furniture 2-4
Furniture
2
5-15
Metric
whole
part
whole
part
whole
part
whole
part
whole
part
Real2Code+ggeeg
0.57
2.33
1.54
7.65
0.51
2.04
1.46
13.3
5.84
16.8
Ditto
2.34
2.04
1.73
82.82
2.80
462.25
2.25
1105.86
2.21
4608.08
PARIS
84.29
206.31
15.35
158.73
20.63
1297.27
6.02
544.64
11.44
816.86
Real2Codle-SegOnly
1.74
7.19
11.46
10.52
0.90
23.44
17.43
206.49
N/A 19.70
N/A
Real2Code (Ours)
0.44
3.02
1.31
5.94
0.60
1.28
3.47
65.79
118.58
Table 1: We evaluate surface reconstruction quality by measuring Chamfer-Distance (CD) between predicted and ground-truth meshes. Results are reported separately for each object category, where we take average CD of objects' entire surface reconstruction ('whole' column) and of all part wholes ('part' column). Objects from Gtorage-Furnitur and Table are reported under Furniture and divided based on the number of parts.
Sec. 4.5 contains ablation studies that provide additional insights into our method. Sec. 4.6 shows qualitative results of our pipeline on real world objects.
4.1 EXPERIMENT SETUP
Datasets. We use assets from five categories in ParlNet-Mooility (Mo et al. 2019) dataset: Laptop, Box, Refrigerator, Storage-Furnineg Furniture and Table. The same split of 467 train and 35 test objects are used to construct our image segmentation, shape completion, and code datasets. We use Blender (Com) munity 2018 Denninger et al.) 2023) to render RGB-D and segmentation masks. The RGB-D images and masks are then used to generate part-level point clouds as partial observations. For code data, we extract OBBs from part meshes and process each object's 's raw URDF file into Python MJCF ('unyasuvunakool et al., 2020), where the joint rotation and position relative to the OBB of the child part that this joint connects to the parent part. Refer to appendix A.2 for more details.
Baselines. We compare Real2Code the following baseline methods:
PARIS (Liu et al. 2023a) is the prior state-of-the-art for articulated object reconstruction. takes multi-view RGB observations of a two-part articulated object at two different joint states, then optimizes NeRF-based reconstructions and joint parameters based on motion cues from the two observed states. We render our test objects at two random joint states, report the average performance across 5 random initialization seeds. We modify their method to optimize for more than two parts at once. However, we observe that their design of optimizing one neural field for each part runs out of memory when the number of joints exceeds 4.
Ditto (Jiang al. 2022b) is an end-to-end learned model that takes pair of before- and after-interaction point cloud inputs and predicts implicit part shapes and joint parameters. Notably, Ditto assumes only one object part is moved at time, which requires step-by-step interactions and observations, making evaluation less efficient. For an object with N joints, move part at time, render the corresponding N pairs of point cloud observations, and run their pre-trained model N times to obtain the final results.
GPT-4 OOpenAl 2023) is representative of recent state-of-the of-the-art art LLMs with strong reasoning and code-geme sedeegeneration capability. We use it reference for shoo LLM performance without fine-tuning. We prompt it with the same code header used in our LLM fine-tuning dataset, plus additional instructions on how format the output, which fine-tuned LLM does not need.
task
4.2
PART SEGMENTATION AND RECONSTRUCTION EXPERIMENTS
4.2.1 3D PART-LEVEL SHAPE COMPLETION.
Following the prompting procedure described in Sec. 3.1 we first run our fine-tuned SAM on images from the test set of unseen objects and obtain segmented part point clouds. We observe that, because we rank and filter the mask predictions (i.e., prioritize high predicted confidence score and stability score), the low-quality masks have less impact the final segmented point-cloud after the projection step. Next, we use the segmented part point clouds as input to evaluate our learned shape completion model: following Mu et al. (2021) y; Jiang et al. (2022b); Liu et al. (2023a), we uniformly sample 10, 000 points on the extracted mesh surface, and report the average Chamfer Distance between the extracted and ground-truth part meshes in Tab. 1 Because the predictions semantics-agnostic
7

---

Published as a conference paper at ICLR 2025
Parts (15)
Parts (9)
4-5
Parts (6)
6-15 Parts (7)
rot
post
type1|
rot
pos4
typet|
rot.
pos.
typetl
rot..
pos..
typet
Real2Code+guss
0.0
0.07
0.93
0.0
0.04
1.00
0.0
0.04
1.00
11.6
0.03
0.94
Ditto
40.04
4.04
0.57
35.57
2.47
0.70
49.77
3.20
0.43
63.
4.16
0.30
PARIS
48.44
2.67
0.51
32.35
3.63
0.84
55.97
2.14
0.43
N/A
N/A
N/A
GPT4
57.3
0.26
0.73
10.0
0.08
0.61
45.0
0.21
0.51
30.0
0.05
0.71
Real2Code (Ours)
7.5
0.08
0.80
0.0
0.04
0.89
0.63
0.07
0.97
30.2
0.05
0.89
Table 2: Joint prediction results from Real2Code and baseline methods, grouped by the number of moving parts in each object. We remark that Real2Code consistently outperforms baseline methods across objects with different kinematic structures; on objects with more moving parts, Real2Code predicts joints accurately whereas baseline methods fail.
(i.e., the model does not know if segmented part is a drawer or a door), we generate permutations of the set of predicted meshes and take the permutation that results in lowest error; the same logic is used for joint prediction results.
Overall, Real2Code achieves the best reconstruction quality and elegantly scales to a larger num- ber of parts (column 'Real2Code (Ours)"). We remark on the performance difference between Real2Code and baselines: the joint optimization of all parts in PARIS (Liu et al.. (2023a) suffers from a complex loss landscape and produces unsatisfactory reconstructions, especially when the number of parts increases. Ditto (Jiang al. 2022b) performs well on training categories (i.e., Laptop) but does not generalize well to unseen categories. In contrast, ours obtain better results because we factorize the problem into segmentation and shape completion, aggregate 2D segmentation from fine-tuned SAM and perform part-level shape completion.
To validate the need for our shape completion model, we observe that Due the partial observation and noise in the segmentation masks, simply extracting meshes from the part-level point clouds also results in subpar reconstruction results (column (Real2Code-SegOut mesh extractions are too noisy to match with GT mesh). 2) If we use ground-truth segmentation, the mesh extraction from the aggregated point clouds are better than using SAM segmentation, but are still incomplete (column 'Real2Code-gtee -tgee")
where 'N/A' indicates the
4.2.2 KINEMATICS- 2D IMAGE SEGMENTATION.
To demonstrate the effectiveness of SAM fine-tuning, we evaluate the fine-tuned model on unseen object images by uniformly sampling a grid of 32 x 32 query points and compare the predicted segmentation with ground-truth masks. We use NMS filtering the model's predicted confidence score to take the top-K masks that fill the image to more than 95% total pixels. We observe a significant improvement over zero-shot SAM: object parts are segmented much more closely following their kinematics structure, obtaining a 92% mean loU score on the final used masks and 84% match rate ground-truth masks.
the predicted masks, then sort with
4.3 ARTICULATION PREDICTION EXPERIMENTS
After completing pert-level reconstruction on test objects, extract OBBs for each object part and compose a text-prompt for our fine-tuned Codel.lama model. We parse the model's code generation and append it with code header lines (e.g. import packages) such that the post-processed code can be directly executed to produce object simulation. We then evaluate the accuracy of articulation prediction by measuring the error of joint joint axis, and (for revolute joints only) joint position predictions. As shown in Tab. 2 Real2Code outperform all baseline methods by a large margin. The where
effectiveness of our OBB abstraction is further accentuated by column 'Real2Code+gtBl we feed oracle OBB to the code generation module and achieve highly accurate predictions even on unseen objects with a large number of parts.
4.4 QUALITATIVE RESULTS
For qualitative results, we show objects with range of varying kinematic complexities, from two-part laptop ten-part multi-drawer table. We visualize the final reconstructed objects from ours and baselines methods in Fig. Whereas all methods handle the simpler laptop articulation, baseline methods struggle as the number of object part increases while Real2Code performs much more accurate reconstruction. See our submission website for more visualizations: pictpps//siit
goog]e e ctm/com///// vi ew/ real 2code- subm osioon
8

---

Published as a conference paper at ICLR 2025
Parts (15)
Parts (9)
4-5 Parts (6)
6-15
Parts (6)
Inp.
Out
rot.
pos4
typet
rot.
pos4
typet
rot.
pos..
type?
rot..
pos.
typet
OBB
Abs.
7.5
0.06
0.92
0.0
0.03
1.0
0.0
0.6
0.83
0.0
0.7
0.73
OBB
Rot.
0.0
0.18
0.73
0.3
0.23
1.00
0.9
0.19
0.83
5.9
0.06
0.59
+RGB
Rel.
0.0
0.06
0.80
5.0
0.03
1.0
0.0
0.03
0.89
0.0
0.02
0.67
OBB
Rel.
0.0
0.07
0.93
0.0
0.04
1.0
0.0
0.04
1.00
11.6
0.03
0.94
Table 3: Joint prediction results from ablation experiments on Real2Code. Using a regression formulation, the LLM is still able to output reasonable values for two or three part objects, but generates much less accurate joint positions when the number of articulated parts increase. Additional RGB image input yields no clear improvements, which suggests the OBB input alone can provide sufficient information.
4.5 ABLATION STUDIES
To further validate formulation of using OBB reference for articulation prediction, experiment with alternative input and output representation for ablation:
Regression on Joint Parameters. We fine-tune two more Cama models to take the same input but outputs continuous numerical values for joint parameters: the first model directly predicts 3 values for each joint axis and 3 for every joint position (ecc|4.3]roo row 'OBB Abs.": "): the second model predicts joint axis the same way Real2Code, but predicts joint position relative position the OBB's 's center (Sec. 4.5 column 'OBB Rel.)).
Provide L.L. with Visual Inputs. We fine-tune model with hoth RGB and OBB inputs. We adopt the OpenFlamingo (Alayrac et al. 2022] Awadalla et al.] 2023) approach for interleaving image embeddings with the CodeLlama model weights, and uses the same praited ViT (Dosovitskiy el al. 2020) weights for image
encoder.
Results from the ablation experiments are reported in Tab. We make the following remarks: first, regression formulation predicts less accurate joint positions. Both predicting absolute joint positions (column 'OBB Abs.') and relative position from OBB center (column 'OBB Rot.') yield a higher error. In contrary, the rotation error is still on a reasonable scale: we found this is due to the model learns to copy the correct axis column from the OBB rotation matrices contained in the input prompt. Second, RGB input does not yield significant improvement. Comparing row '+RGB Rel.' and 'OBB Rel.', we see the OBB input provides sufficient information for articulation prediction task.
4.6 EXPERIMENTS ON REAL WORLD OBJECTS
To validate the generalization ability of Real2Code, we gather set of in-the- wild articulated objects and collect multi- view RGB data as inputs. We run Real2Code with DUS13R (Wang et al.] 2023b) to achieve reconstruction from multi-view pose-free RGB images. Due the lack of ground-trutte -ruthss, we show quali- tative results in Fig. that Real2Code gen- eralizes well to real objects and pro- duces good quality reconstructions from RGB-only inputs. However, although the learned DUSt3R (Wang et al. (2023b) model performs well on overall shape and exterior surface areas of the objects, it pre- dicts less accurate point maps areas in- side the drawers, (likely due to the lack of similar data in their training dataset). As a result, the segmented part point clouds display noises (second row in Fig. 7i and leads to lower quality mesh extractions. See appendix A.4 for more details on our
Prompt with OBB Input
stevene sicronstdd 1n au 10), 10 RURT)) Aulii 1.--5-100 "wwtwrss (e 29.1.22,.4 a.1))
inerodeeest oedece))
and leatly
12lee" (iar
cody. mocal unr.deeyyyyy anc redd", rame- ra0e-129001
Different Joint Output Types
Additional Input
Joints
Absolute Pos. + Asis
-100:008-1, 1900 'annge" poo--ee. 2, 6.0, 16), axts- Li 9,1,9,,,111111
Relative Pos.
Fict:/0oped, 1dee1, type- "annge vegge- (0 0..1 sigens)
OBB Axis
/Set//ox-t Sdeed type- "atnne" Ledgo- 11, -12.smm3))))
DBB Edgo + Axis
Figure 5: Qualitative comparison of the code output format in ablation experiments. Each formulation occupics one line. In 'Absolute Pos. + Axis', LLM outputs continuous position and axis values; in 'Relative Pos. + OBB Axis', LLM outputs one index into the OBB's rotation axis, and a 2D joint position relative to the selected axis; Real2Code uses 'OBB Edge + Axis", where LLM outputs index to rotation axes in an OBB, and two values to indicate the OBB edge. Bottom right of the figure shows one example of additional RGB image input to the I.L.M.
evaluation setup.
5 CONCLUSION
We present Real2Code, novel method for reconstructing articulated objects that leverages the capability pre- trained vision and large language models. We empirically show that Real2Code achieves a new state-of-ther th- in both geometry reconstruction and articulation prediction. We hope Real2Code unlocks new opportunitics in robotics and mixed reality applications.

---

Published as a conference paper at ICLR 2025
N/A
PARIS
Ditto
Real2Code w/o Shape Completion
Real2Code (ours)
GT
Figure 6: Qualitative results that compare Real2Code to baseline methods. We show test on objects with varying kinematic complexities, from two-part laptop to ten-part multi-drawer tablc. Whereas all methods handle the simpler laptop articulation, baseline methods struggle the number of object parts increases, and Real2Code performs reconstruction much more accurately. PARIS out of memory and fails ten-part table object ('N/A').
the
Example
RGB input
Segmented Part Point Cloud
Reconstructed
Object
Figure 7: We evaluate Real2Code on real world objects using RGB data. For each object, we use 10 pose-free RGB images captured in-the-wild and run Real2Code with DUS(3R DWSg et al, (0236). We show one example RGB input (1st row), segmented point clouds (2nd row) and full reconstruction (3rd row) for each object.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
This work was supported in part by the Toyota Research Institute, NSF Award #2143601, Sloan Fellowship. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied. of the sponsors. The authors would like to thank the following friends and colleagues for help during various stages of the project: Zhenjia Xu, Samir Gadre, Cheng Chi, Hanxiao Jiang, Bingjic Tang, and members of REALab: Zeyi Liu, Xiaomeng Xu, Chuer Pan, Huy Ha, Yihuai Gao, Mengda Xu, Austin Patel for valuable foedback and discussion on the paper manuscript. We also thank administrators of Stanford EE department, namely Kenny Green, Steve B. Cousins and Mary K. McMahon for their support during real world data collection.
REFERENCES
3D Scanner App. 3D Scanner App.
2024. Accessed: 2024-03-13.
Hameed Abdul-Rashid, Miles Freeman, Ben Abbatematteo, George Konidaris, and Daniel Ritchie. Learning infer kinematic hierarchies for novel object instances. 2022 International Conference on Robotics and Automation (ICRA), pp. 8461-8467, IEEE, 2022.
Captitatiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong. Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022.
Boshi An, Yiran Geng, Kai Chen, Xiaoqi Li, Qi Dou, and Hao Dong. Rgbmanip: Monocular image-based robotic manipulation through active object pose estimation. ArXiv, abs/2310.03478, 2023. URL httpa //api. semant coamanticschol.o icscholar. ott/CoorpssD::23377220
Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Y Yang. Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Camphell Orme, Jakob Engel, Edward Miller, Richard Newcombe, and Vasileios Balntas. Scenescript: Reconstructing scenes with an autoregressive structured language model, 2024.
Anas Awadalla, Irena Gao, Josh Gardner. Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Korbllihh, Pang Wei Koh, Gabriel Ilarroo, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Russell Buchanan, Adrian Röfer. João Moura, Abhinav Valada, and Sethu Vijaykkumar, Online estimation of articulated objects with factor graphs using vision and proprioceptive sensing. ArXiv, hts/2309..16433 2023. UtR.httss:://at semantt cscho ar. hwtp/org//oruuu ID: 2:3 34377
Jiazhong Cen, Zanwei Zhou, Jiemin Fang. Chen Yang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs, 2023.
Blender Online Community, Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. BtLphttp://wee http://wwwwwwbb.e...eeeee bwwnder....ee.......... org
Maximilian Denninger, Dominik Winkelbauer, Martin Sundermeyer, Wout Boerdijk, Markus Knauer, Klaus H. Strobl, Matthias Humt, and Rudolph Triebel. Blenderproc2: A procedural pipeline for photorealistic rendering. Journal of Open Source Software, 8(82):4001, 2023, doi: 10.21 (0221105/jos.049000 URL |RL
//doi.org// 2ttp:/1...111////11 Joss. 04901
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov. Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image worth 16x16 words: Transformers for image recognition scale, 2020.
11

---

Published as a conference paper at ICLR 2025
Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche. Towaki Takikawa, Jiehan Wang, Charles Loop. Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozantsev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian, Kaolin: A pytorch library for accelerating 3d deep learning research, comm Http///ICIAAaaeeor..... kaolin in, 2022.
Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Act the part: Learning interaction strategics for articulated object part discovery, 2021.
Yaroslav Ganin, Sergey Bartunov, Yujia Li, Ethan Keller, and Stefano Saliceti. Computer-aide aiddd design as language, 2021.
Haoran Geng, Ziming Li, Yiran Geng, Jiayi Chen, Hao Dong, and He Wang. Partmanip: Learning tross-category generalizable part manipulation policy from point cloud observations, 2023a.
Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross- category dommaingenerralzzalle object perception and manipulation via generalizable and actionable parts, 2023b.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training, 2022.
Nick Heppert, Toki Migimatsu, Brent Yi, Claire Chen, and Jeannette Bohg. Category-independen articulated object tracking with factor graphs. In 2022 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS). IEEE, October 2022. doi: 10.1 10..100/im044711222222299222 URL http:/ /x
org/10. 1109 2RA.99/RRS4411222 2022. 9982029
Nick Heppert, Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Rares Ambrus, Jeannette Bohg, Abhinav Valada, and Thomas Kollar. Carto: Category and joint agnostic reconstruction articulated objects. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21201-21210, 2023.
WtRL:ttps::///
www.etetttcciie...e... www.stttcccooooooooooooooo org
Cheng-Chus Chun Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in house: Building articulation models of indoor scenes through interactive perception. 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 3933-3939, 2023a.
/ api semant iirenticsshhi ccscholar www.orr///orrrree CCrpporrrppiiiCi INTE:::55555555
Joy Hsu, Jiayuan Mao, and Jiajun Wu. Ns3d: Neuro-symbolic grounding of 3d objects and relations, 2023b.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Ruizhen Hu, Wenchao Li, Oliver Van Kaick, Ariel Shamir, Hao Zhang, and Hui Huang. Learning predict part mobility from a single static snapshot. ACM Transactions on Graphics (TOG), 36(6):1-13, 2017.
Jiahui Huang, He Wang, Tolga Birdal, Minhyuk Sung, Federica Arrigoni, Shi-Min Hu, and Leonidas Guibas. Multibodysyne: Multi-body segmentation and motion estimation via sean synchronization. Proceedings of the Computer Vision and Pattern Recognition (CVPR), 2021. URL
06605
Xiaoxia Huang, Ian D. Walker, and Stan Birchfield. Occlusion-aware multi-view reconstruction of ar- ticulated objects for manipulation. Robotics Auton. Syst., 62:497-508 505, 2014. URL https::/appi ssmanticschhlar wttpg///ooousi:iiioo 11:31777
Muhammad Zubair Irshad, Thomas Kollar, Michael Laskey, Kevin Stone, and Zsolt Kira. Centersnap: Single- shot multi-object 3d shape reconstruction and categorical 6d pose and size estimation, 2022. https
// 1/arxiv. abs/ 2203. 01929
Hanxiao Jiang, Yongsen Mao, Manolis Savva, and Angel X Chang. Opd: Single-view 3d openable part detection. In Computer Vision-ECC ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX. pp. 426. Springer. 2022a.
Zhenyu Jiang, Cheng-Chre Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction, 2022b.
R. Kenny Jones, Homer Walke, and Daniel Ritchie. Plad: Learning to infer shape programs with pscudo-labels and approximate distributions, 2022.
Pushkal Katara, Zhou Xian, and Katerina Fragkiadaki. Gen2sim: Scaling up robot learning in simulation with generative models, 2023.
12

---

Published as a conference paper at ICLR 2025
Dov Katz, Moslem Kazemi, J Andrew Bagnell, and Anthony Stentz. Interactive segmentation, tracking, and kinematic modeling of unknown articulated objects. 2013 IEEE International Conference on Robotics and Automation, pp. 5003-500 50100000. IEEE, 2013.
Yuki Kawana, Yusuke Mukuta, and Tatsuya Harada. Unsupervised pose-aware part decomposition for man-made articulated objects. In European Conference on Computer Vision, pp. 558-575. Springer, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023.
Xiaolong Li. He Wang. Li Yi, Leonidas Guibas, A. Lynn Abbott, and Shuran Song. Category-level articulated object pose estimation, 2020.
Yi-Chuan Liang, Learning infer 3d shape programs with differcentiable renderer. ArXiv, abs/2206.12675, 2022. URL UtRthttss://aptt otw.aaaaa.cccooo....oooooooooooa
Vincent Lim, Huang Huang, Lawrence Yunliang Chen, Jonathan Wang, Jeffrey Ichnowski, Daniel Seita, Michael Laskey, and Ken Goldberg. Planar robot casting with real2sim2real self-supervised learning, 2022.
Tsung-V Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, 2018.
Jiayi Liu, Ali Mahdavi-Amiri, and Manolis Savva. Paris: Part-level reconstruction and motion analysis for articulated objects, 2023a.
Liu Liu, Wengiang Xu, Haoyuan Fu, Sucheng Qian, Yang Han, and Cewu Lu. Akb-48: A real-world articulated object knowledge base, 2022.
Shaowei Liu, Saurabh Gupta, and Shenlong Wang. Building rearticulable models for arbitrary 3d objects from 4d point clouds. In Proceedings the Computer Vision and Pattern Recognition (CVPR). 2023b. URL
https://arzx www.aawwww....rraaeaaeaeeooeee/e a2tg///bb///3/....///// 30979
William Lorensen and Harvey Cline. Marching cubes: high resolution surface construction algorithm. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '87, pp. 163-169, New York, NY, USA, 1987. Association for Computing Machinery. ISBN 0897912276. doi: 10.1 14/1144474411344222 https:::: /doi wwwp//o.g/11.... 1115/...11133333
37422
Frank Michel, Alexander Krull, Eric Brachmann, Michael Ying Yang, Stefan Gumhold, and Carsten Rother. Pose estimation of kinematic chain instances via object coordinate regression. In British Machine Vision Conference, 2015. URL [attattt///pppp semant lesemanniscccooaa... wwt.:/ar...rr/oooo......... 1221100451
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020.
Kaichun Mo, Shilin Zhu, Angel X. Chang. Li Yi, Subarna Tripathi, Lconidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hicrarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Kaichun Mo, Leonidas Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels actions for articulated 3d objects, 2021.
Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representation, 2021.
Neil Nie, Samir Yitzhak Gadre, Kiana Elsaani and Shuran Song. Structure from action: Learning interactions articulated object 3d structure discovery, 2023.
OpenAl. Gpt-4 technical report, 2023.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision (ECCV), 2020.
Polycam. Polycam LiDAR & 3D Scanner for iPhone & Android. ht:ps://poly. cam/ 2024. Accessed: 2024-03-07.
C. Qi. L. Yi. Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Neural Information Processing Systems, 2017. URL. https://api semant cacholar
www.cc/aa..ogg//oooooeeeeeeeeeee wtt/Corpps 1745976
13

---

Published as a conference paper at ICLR 2025
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2023.
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang. Arsha Nagrami, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. Modular visual question answering via code generation, 2023.
Carole H. Sudre, Wenqi Li. Tom Vercauteren, Sebastien Ourselin, and M. Jorge Cardoso. Generalised Dice Overlap a Deep Leurning Loss Function for Highly Unbulaneed Segmentations, Pp. 240-248. Springer doi: 10. 11110009788--333--77588888888888 31 67588 9_28
International Publishing, 2017. ISBN 97833196755888
URL http:
didx.doi.org/o
1007
Xiaohao Sun, Hanxiao Jiang, Manolis Savva, and Angel X. Chang. Opdmulti: Openable part detection for multiple objects. ArXiv, abs/2303.14087, 2023. URL https: semant icschotccccccccac org/
(D:
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning, 2023.
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 2.1...6611...
Marcel Torne. Anthony Simeonov, Zechu Li. April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real--o-sim-tt--eal approach for robust manipulation. 2024.
Wei-Cheng Tseng, Hung-Ju Liao, Yen-Chen Lin, and Min Sun. Cla-nerf: Category- articulated neural radiance field. 2022 International Conference on Robotics and Automation (ICRA), pp. 8454 8460, 2022.
URL URRLhttps: //ppi. semant wescccccchlaaaa org/ Corpus ID; 237397845
Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. 2017 JEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 1466-1474, 2016. URL https: //ps: semant i ioamanttcacholaan wwwpcccolar..oo/oorroeoeeeeo ID
2380406
Saran Tunyasuvunakool, Alistair Muldal, Yolam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, November 2020. ISSN 2665 2665-9638. doi: 10.1016/j.simaa.2222 100022. URL
http:// org/ 10. 1ttp://11.11111..........mm 2020. 100022
Luobin Wang. Runlin Guo, Quan Vuong, Yuzhe Qin, Hao Su, and Hearik Christensen. A real2sim2real method for robust object grasping with neural surface reconstruction, 2023a.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud, Dust3r: Geometric 3d vision made casy, 2023b.
Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes, 2019.
Fangyin Wei, Rohan Chabra, Lingni Ma, Christoph Lassner, Michael Zollhöfer, Szymon Rusinkiewicz, Chris Sweeney. Richard Newcombe, and Mira Slavcheva. supe-super neural articulated shape and appearance models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 15816-15826, 2022.
Yijia Weng, Bowen Wen, Jonathan Tremblay. Valts Blukis, Dieter Fox, Leonidas Guibas, and Stan Birchfield. Neural implicit representation building digital twins of unknown articulated objects, 2024. URL Itttpa
// www.iii.ogggaaaa.....aaoa abs/2404 2404, 01440
Karl D. D. Willis, Yewen Pu, Jieliang Luo, Hang Chu, Tao Du, Joseph G. Lambourne, Amando and Wojeiech Matusik. Fusion 360 gallery: A dataset and environment for programmatic cad construction from human design sequences, 2021.
14

---

Published as a conference paper at ICLR 2025
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang. Yi, Angel X. Chang, Leonidas Guibas, and Hao Su. SAPIEN: simulated part-based interactive environment, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Yi, Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas. Deep part induction from articulated object pairs. ACM Transactions on Graphics, 37(6):1-15, 2018. URL httpss//aaxxx org/abs/ / 1809, 07417
Andy Zeng. Maria Attarian, Brian Ichter. Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Composing zero-shot multimodal reasoning with language, 2022.
Socratic models:
Vicky Zeng. Tabitha Edith Lee, Jacky Liang, and Oliver Kroemer. Visual identification of articulated object parts. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 2443-2450. IEEE 2021.
Özgün Çişek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: Learning dense volumetric segmentation from sparse annotation, 2016.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DISCUSSIONS & LIMITATIONS
In this section, we discuss a few key limitations that point to interesting directions for future work:
Real2Code currently handles one single object at a time. To achieve scene-level reconstruction, i.c. multiple objects cach with multiple articulated parts, additional processing is required on top of the current pipeline. For example, given a sequence of multi-view image inputs of a multi-object scene, we first object detection model to single-out each detected object, then use the original Real2Code pipeline to handle object-level reconstruction.
2. Test-time computational efficiency. Due to the iterative prompting method described grid-sampling of prompt points on each single RGB image and 2) prompting on all camera views for each object part, our test-time compute for SAM forward pass scales linearly with the number of input camera views: if an object has M camera views, K object parts, and uses Nz grid for initial prompt points sampling (we use N 16), then the SAM Kirillov et al. (2023) is prompted with N + ( 1) single 21>-point and RGB image pairs. Notably, M the main compute bottleneck because we can cache the image embedding from SAM Kirillov et al. (2023) and only call the light-weight prompt decoder for additional prompt points. Additionally, the inference compute for LLM code generation is dependent on the number of object parts. and roughly scales linearly with the generation token length. Overall, our system is slower inference time when compared with end-lo-end methods such as CARTC[eeppert et al. (2023) and Ditto Diang et al. (2022b), but is more scalable to more complex articulation structures because it handles arbitrary numbers of object parts.
3. Cascading dependency. Because Real2Code composed of multiple modules, failure cases happen when the errors from cach module propagate and lead to sub-optimal final object reconstructions. We found that the articulation prediction accuracy is sensitive to failures in the first 2D image segmentation module, i.e., OBBs from wrong segmentations directly obstruct the LLM reasoning of ohject structures. To increase robustness, we can improve the system by providing human corrective feedback as proposed in Kirillov et al. (202)). i.e., user provides additional points and prompt the model to adjust its mask predictions. Then only feed the input with satisfactory OBB extractions to LLM for code generation.
4. Objects with hinge joints that do not overlap with OBB edge. To handle new object categories. we remark that 1) the geometry reconstruction part of Real2Code (both part segmentation shape completion) can handle complex geometry shapes (c.g. scissors, faucet handles) when given the training data for fine-tuning the part segmentation model and shape completion model. 2) However, because we select OBB edge as rotation center, our method can handle sliding joints (e.g. a sliding oven rack) but will be inaccurate for hinge joints where the joint is not overlapping with any OBB edge (e.g. scissors). To address this, one could add another fine-tuning head to further adjust the LLM outputs (which selects one OBB edge) by predicting an offset value improve the joint position accuracy.
A.2 DATASET PREPARATION DETAILS
Base: PartNet-Mobiligg Object Assets. We use the same set of 467 training and 35 testing objects from 4 categories in PartNet-Mobilltt (Mo et 2019).. The raw dataset contains a rich collection of object meshes. textures, and URDF files that contain articulation information. We further process the data as follows:
RGB-D Image Rendering We render each object individually using Blender (Community) 2018 Denninger ct al. (202)) for 5 loops. For cach rendering loop. the object is centered at the scene origin and the rendering
camera poses are randomly sampled; we render 12 RGB-D D images and all the segmentation masks corresponding to the all the object parts. During rendering, we also randomly sample joint states in the object such that all its doors or drawers are partially open partially open to remove ambiguity and provide more observation view into object insides.
make the assumption that all the parts our train and test objects are
Mesh Pre-processing, The original BurNeet-Mobllig assets contain highly fine-grained meshes, ine.,one drawer part is comprised of more than ten panel or bar- sha-sh meshes. To prepare data for part-level shape completion, we group these fine-grained meshes such that meshes from the same object part are merged into one single mesh. Mesh textures are ignored during grouping, resulting in grouped texture-less part-level meshes. The RGB-D images and masks are then used to generate pert-level point clouds as partial observations. We use Kaolin (Fuji Tsang et al. 2022) to sample label occupancy values from object part meshes.
unde-generation LLMs we first use the rendered RGB-D images and segmentation masks to obtain ground-truth part-level point-clouds, which are used to extract oriented-bounding boxes (OBBs) for each part. Next, we take the object URDF files and generate shorter
Code-Generation Data. To prepare data for fine-tuning
16

---

Published as a conference paper at ICLR 2025
copy with our grouped part meshes. Because the URDE/XML syntax contain long unnecessary details, we manually translate them into Python-like MJCF (Tunyasuvunakool et al. 2020) code, which are lot more compact and familiar to the pre-trained LLMs. Finally, for each the rendering loops per object, we re-w re-write the object code again to replace the absolute joint information with the relative position and rotation of cach joint with respect to the extracted OBBs. We further augment the data by randomly rotating the OBBs along the z-axis, times per object. This results in 168 5 5 11700 training samples for LLM fine-tuning.
A.3 MODEL. TRAINING DETAILS
SAM Fine-tuning. The fine-tuning data consists of 28, 020 RGB images, and cach image corresponds to set of binary segmentation masks, per each object part plus background mask. We fine-tune only the decoder layers of pre-trained SAM (Kirillov al. (023) this custom dataset while keeping the rest of the model weights frozen. Each fine-tuning batch contains RGB images; every RGB image the batch, we sample 16 prompt points uniformly across each image's s ground-truth masks, i.e.,only sample points from the positive mask area. Hence cach training batch of size B 24 contains 24 images and 24 x 16 pairs of prompt point and ground-truth truth masks. Following the original paper (Kirillov al. 2023), update the model with weighted average Focal Loss (Lin al. 2018), Dice Loss (Sudre 2017 and MSE IoU prediction loss.
image
Zero-shot SAM
Fine-tuned SAM
Fine Fine-tuning Dataset
Figure Kinematics-aware SAM Fine-tuning. Given an RGB input image, the pre-trained zero-shot SAMKirillov et al. (2023) produces unnecessarily detailed segmentation masks (column Zero-shot SAM'. We construct dataset of objects' RGB images and kinematics- aliignd ground -ouuhhttruth masks (column 'Fine-tuning Dataset'). The model is fine-tuned to take one image and sampled 2D query point and predict the corresponding part mask. We compare the output of the model after fine-tuning the same image (column FFine-tuned SAM').
Training Shape Completion Model. We use 6, 260 pairs of partial point clouds and size 962 occupancy grids and train PointNet++ (Qi et al. (017) based occupancy prediction model from scratch. For training batch of size B, sample point clouds of size 2048, and sample grids. Notably, because object parts are different scales, we normalize occupancy grid using partial OBBs extracted from the input point cloud to avoid under-fitting the smaller-sized meshes. When sampling training query points, we found sampling 25% occupied works the best for balancing between occupied areas and empty space, and we add random shifting step the occupied grids improve model accuracy the near-surface areas. test time, we query on 963 grid and use Marching Cubes (Lorensen Cline 1987 to extract completed part meshes.
12, 000 query points the label occupancy
Fine-tuning Code Generation LLM. We use the pre-trained Codellama (Rozière et al, 2023) model on our code dataset, which contains code samples generated from PartNet (Mo et 20199 objects as described above. We use LoRA (Hu et al. 202T). low-rank weight fine-tuning technique, fine-tune the model with the next-token prediction loss. training efficiency. we compress the training sequences by removing unnecessary empty character spaces and overhead code lines (such as package import statements). The resulting training set contains under 800 tokens sequence for objects with up to 7 parts (i.c., 6 articulated joints). Despite the short trinning data, we found the model be able to extrapolate unseen test set objects with up 15 parts when the ground-truth segmentation is provided.
A.4 DETAILS ON REAL WORLD EVALUATIONS
Data Collection. We collect data from a set of common furniture objects, including cabinets, laptops, night stands, dressers, ranging from to 3 moving parts. Each object is scanned using LiDARRequipped iPhone camera and 3dScanner App (3D Scanner App 2024 to capture series of RGB images from the front 180 view. We then select 10 RGB images per object, and crop and resize them into 512 x 512 images used by SAM (Krillov et al. 2023) and DUSt3R (Wang et al. 2023b).
Part Segmentation from Unstructured RGB images. Fig. 9 visualizes the DUS(3R model output on an example object in: notably, the model predicts dense point-maps on the object's 's surface area that can be globally-aligned into object point cloud; but the 3D points less accurate the partially occluded areas,
17

---

Published as a conference paper at ICLR 2025
Globally Aligned Dense Point-maps from DUSt3R
Vewwconsistent Prompt Points
Figure 9: 3D part segmentation from Pose-free RGB images. Illustration of how DUSt3R (Wang et al. 2023b is used achieve 3D part segmentation from unstructured RGB images. For each object, we take around 10 pose-free RGB images as input to the pre-trained DUSt3R (Wang et al.) 2023b) model, which outputs set of globally-alignee pixel on each image is matched to a point in 3D. This correspondence enables cross-view pixel matching via finding nearest-neighbor in 3D space. We can therefore sample view-consistent 2D points for prompting our fine-tuned SAM model, and the resulting segmented masks are grouped into 3D part segmentation.
2D-to-3D dense point-maps, ivery 2D
Jewelry Box
Gameboy
Figure 10: We demonstrate that Real2Code can be used for labeling and animating real world objects. We evaluate Real2Code on scanned real objects from Polycam (Polycam) (0024) and export the resulting mesh and joints in MuJoCo (Todorov et al. 2012). Blue arrows indicate the simulated joint axis and position; mesh corresponding to the moving part is colored in green.
such as inside of the drawer. This is likely due to these areas are less common in the model's pre-training dataset. Also notice that, because we sample each 2D point from one RGB image first and uses nearest neighbor in the predicted 3D point-map to find its matching 2D point in another image, it might find wrong match if the point is occluded and not visible in the other image. We address this by manually setting distance threshold, and decidc match cannot be found if its 3D point's distance to the nearest neighbor is above set threshold.
A.5 ADDITIONAL RESULTS ON ANIMATING SCANNED REAL WORLD OBJECTS
In addition object reconstruction from RGB images, we show Real2Code can also be used animate scanned ohjects. We use real world scanned object meshes uploaded by users of the Polycam (Polycam] 2024 App, and use our Blender rendering pipeline to render RGB-D images. We evaluate our image segmentation, shape completion, and code generation models on these images, and demonstrate only the qualitative results due the lack of ground-trutt trutt data. We execute the final model output code show the objects can be simulated MuloCo (Todorov al., (0122). See Fig. 10 for visualizations. These real world objects feature complex visual appcarance that falls outside our SAM fine-tum mun-itning distribution, but Real2Code is still able to successfully segment parts and predict reasonable joint positions and rotations.
A.6 ADDITIONAL BASELINE RESULTS
18

---

Published as a conference paper at ICLR 2025
Type 12043
Art. Parts
CD (whole)
CD (part)
rot.
pos
Refriggrator
1.01
0.594
0.19
0.05
Refrigerator
12059
0.84
0.504
0.24
0.014
Refrigerator
12066
0.78
120.32
8.86
2.07
StorageFurniture
45694
0.41
0.06
StorageFurniture
44853
1.71
101.09
37.84
0.0
Table 4: Additional experimental results that evaluates DigitalTwinArt Weng et al. (2024) (a concur-
rent work) on objects with or more moving parts. We evaluate each object on individual seeds and report the average performance across runs. We show that, although this method reported better performance than our compared baseline method PARISLiu et al.] (2033)) it still struggles with objects with more than I moving parts and under-performs Real2Code.
19


Reviews:
[{"review_id": "0kWfHPKY49", "reviewer": "Reviewer_S2Gc", "summary": "The paper reconstructs articulated objects from visual observations. The approach utilizes a modular pipeline which first reconstructs part level geometry from segmentation and then uses a codegen LLM to combine the individual parts into an articulate assembled model to be executed in simulation. The paper compares to relevant recent baselines and demonstrate strong improvement. The approach also scales well to increasing number of joints due to its modular approach.", "questions": "Please see the weakness section above for clarification questions. I look forward to seeing them in the rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In my opinion, below are the strengths of the paper:\n\n1. The paper scales well to increased number of joints. This has been a major limitation of preceding works and this work address it nicely with a modular approach i.e, part level reconstruction and code-gen integration for the subsequent steps. \n\n2. Strong quantitative improvement numbers compared to recent state-of-the-art baselines, especially for increasing number of parts. \n\n3. The presentation of the paper in nice and paper writing is easy to follow.", "weaknesses": "I have some question to the authors. In my opinion, below are the paper's weaknesses:\n\n1. Why does the PARIS baseline struggle a lot? even for 2-part case? Did the authors try to tune their method? Based on the PARIS results' from the paper, it looks like it should reasonably work well for a simpler 2 part setting?\n\n2. Despite good qualitative results, why are the resutls only shown on simpler objects like cupboards and laptop? Does the method work for varied articulated objects like scissors, stapler etc? Is this an inherent limitation of their method they only work for a subset of articulated objects for which they ahve a prior? If yes, that should be clearly stated as other baselines seems to work for more complicated articulated objects as well?\n\n3. What is the timing result of the method? Some of the baselines mentioned i.e. CARTO, follow-up from CenterSnap [1] are very fast and don't require manual SAM prompting i.e. single-shot. This is not discussed very well in the related works. \n\n4. I didn't find rigorous details on pretraining datasets for shape completion as well as datasets used for finetuning code llama. Those should be helpful to include. Also do authors plan to open-source their code? It looks like that will be helpful as well for the community to build up on?\n\n[1] Irshad et al. CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4FhwCFO3TT", "reviewer": "Reviewer_9fCy", "summary": "This paper introduces Real2Code, a method for reconstructing articulated objects from multi-view images through code generation. The method first reconstructs part geometry using image segmentation and shape completion. Then it predicts joint information as code generation from fine-tuned LLM which takes an object part as oriented bounding boxes. Experiments show that this method outperforms previous method in generating parts with over three parts and can generalize to real object reconstruction by training only on synthetic data.", "questions": "1. Tab. 3 and its corresponding text has some typos: row 2 has \"Rot\" in out column, but it is referred to as \"Rel\" in the text (if I understand it correctly).\n\n2. In Tab. 3, the first row has 0 error for \"rot\" on 3, 4-5, 6-15 parts. Then why the rot error suddenly becomes very big for 2 parts?\n\n3. How do you determine the parent vs. child node / the canonical pose, especially for real-world objects? For example, the two parts of a laptop have very similar geometries/OBB. If a laptop is placed upside down, would this method also instead treats the keyboard part is child and the screen part as parent?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This method formulates joint prediction as a code generation problem, which is different from prior work. The biggest advantage of such a formulation is the ability to scale well with different numbers of parts (prior work works mostly for objects with <=3 parts).\n\n2. The overall pipeline is novel -- it leverages a few different modules including Vision models for part segmentation and completion as well as LLM for code generation. This way, the problem is decomposed into a few smaller steps which is shown solvable with previous methods.\n\n3. The text and figures are overall well-written and easy to follow.\n\n4. Experiments have been conducted to validate each proposed components. Results seem to achieve state of the art, especially on objects with many parts.", "weaknesses": "1. In Sec. 4.2.1, it mentions that \"we generate permutations of the set of predicted meshes and take the permutation that results in lowest error; the same logic is used for joint prediction results\". I was wondering why this is needed to evaluate this method. Is it because the proposed method is not very stable? How much more time would this cost for the inference of this method?\n\n2. The link to more visualizations included in Sec. 4.4 does not contain any result visualizations -- it seems it only has a method overview figure and an abstract.\n\n3. The content in Tab. 1 is a bit confusing:\n(1) what is ``Real2Code+gtSeg``, the paper does not seem to mention / analyze this row anywhere in the text.\n\n(2) If I understand ``Real2Code+gtSeg`` the same way as ``Real2Code+gtBB`` in Tab. 2, it should be an upper bound of ``Real2Code (Ours)``, if so, why does ``Real2Code+gtSeg`` perform worse than ``Real2Code (Ours)`` in a few columns like Whole & Parts for Box, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u6qU4oAwYb", "reviewer": "Reviewer_GDby", "summary": "The paper focuses on the task of articulated objects reconstruction given only a few images of an object. The pipeline proposed first reconstructs parts from images and then leverages LLM to predict the joint parameters, which generalizes to objects with multiple joints. The method is evaluated on five categories in the PartNet-Mobility dataset and outperforms previous methods.", "questions": "The results presented in the paper are interesting, but I believe that additional evaluations would strengthen the significance and impact of the work.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a new pipeline Real2Code to reconstruct articulated shapes from images. It shows promising results on multiple categories with different joint types in the PartNet-Mobility dataset.\n\n2. I find its generalization ability to multiple-joint shapes particularly interesting, which could potentially enable many real-world robot manipulation tasks.\n\n3. The paper is overall easy to read.", "weaknesses": "1. The proposed pipeline consists of multiple components and, as a result, rather fragile from what I understand, since a failure in any component in the middle can cause the entire pipeline to break down. For example, if the part bounding boxes parameters (segmentation or shape completion) are inaccurate, the joint prediction part will carry these errors. Since the whole procedure is open-loop, I wonder if the method still produces reasonable shapes assuming initial bounding box predictions are inaccurate?\n\n2. The method is only evaluated on five categories, and these categories (Box, Refrigerator, Storage-Furniture and Table) are all quite similar in topology, similar for the real-world examples. So I think it would be helpful to see results on more diverse shapes. In addition, is CodeLlama trained on all categories together? How does CodeLlama handle scale differences of different objects / categories? Or all shapes normalized so the input to the LLM is kind of already normalized?\n\n3. Which component is the bottleneck of the pipeline? Is it the part segmentation or joint prediction of CodeLlama? Ablation studies on this are essential to better evaluate the approach.\n\n4. To better support the claim of being able to extrapolate beyond objects’ structural complexity in the training set, I think it would be important to provide more results. For example, does the trained model generalize to other categories?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G20bprSf2x", "reviewer": "Reviewer_1whT", "summary": "This paper formulates joint prediction as a code-generation problem and adapts LLM to this task, which makes it scale elegantly to process an articulated object with multiple joints. It also introduces a part reconstruction pipeline leveraging 2D part segmentation and part-level shape completion.", "questions": "Why were only these five object categories chosen from PartNet-Mobility for evaluation? The current formulation relies on assumptions that appear somewhat unsubstantiated. Is this why Real2Code is hard to evaluate in more diverse categories?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Formulating joint prediction as a code-generation problem provides an elegant way to handle varying numbers of object joints.\n- Part-level shape completion makes sense since part structures are much simpler than structures of whole objects. Table 1 demonstrates the effectiveness of the proposed shape completion model.", "weaknesses": "The selection of object categories for evaluation is limited. \n- For part-level shape completion, it would be more compelling to include categories with a greater diversity of part shapes rather than focusing primarily on cuboid-like forms. For instance, objects such as globes and lamps in PartNet Mobility exhibit a variety of shapes, including spherical and cylindrical forms, which provide a more comprehensive basis for evaluation. Additionally, the assumption that 'many common articulated objects consist of cuboid-like parts' is not fully substantiated when considering the full range of object categories in PartNet-Mobility. \n- In articulation prediction, the formulation assumes that 'the position of corresponding revolute joints will lie closely to, if not overlap with, one of the OBB edges'. However, this assumption seems not to be solid enough either. Take “folding chairs” in PartNet-Mobility for example, the revolute joints of many instances lie not close enough to OBB edges (quadrisection point or even trisection point). Do these assumptions restrict the range of categories suitable for evaluation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QiFMlKEDwy", "reviewer": "Reviewer_mGLU", "summary": "This paper presents Real2Code, a novel approach for reconstructing articulated objects from visual observations. \nMain Contributions:\n1. A new method (Real2Code) that reconstructs articulated objects by generating code, using fine-tuned LLMs specialized for this task.\n2. A part reconstruction pipeline that combines:\n    - Kinematic-aware view-consistent part segmentation model.\n    - 3D shape completion model.\n    - Fine-tuned LLMs to predict joint articulation.\n3. Significant performance improvements over previous methods:\n    - First approach to accurately handle objects with more than three parts\n    - Generalizes beyond training data (trained on up to 7 parts, works on up to 10 parts)\n    - Works with just a few RGB images, without requiring depth or camera information", "questions": "1. Why is the method from [1] not included in the baseline comparisons, given that it demonstrates better performance than PARIS?\n2. How is the cross-category generalization ability of Real2Code, especially on real-world data?\n3. How long do the training and inference of the entire pipeline take respectively？\n4. How to extract oriented bounding box of each part?\n\n[1] Yijia Weng, Bowen Wen, etc. Neural implicit representation for building digital twins of unknown articulated objects.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and easy to follow.\n2. The proposed pipeline innovatively formulates the articulation reconstruction as code generation, which naturally combine current powerful foundation models (SAM, LLM) for articulated object reconstruction.\n3. Real2Code demonstrates significant performance improvements over previous methods. It accurately handle objects with more than three parts and only requires RGB images without requiring depth or camera information.\n4. This paper provides details for the training of the whole pipeline, including data preparation and training of key components (SAM, completion model, and CodeLlama), which demonstrates good reproducibility and technical soundness.", "weaknesses": "1. Real2Code demonstrates good performance on trained categories (Laptop, Box, Refrigerator, Storage-Furniture, and Table). However, the performance of unseen categories is not explored. I don't expect the model to generalize well to all other categories, but I do expect some experiments to show whether there is still a problem with category generalization. \n2. There is no discussion of when the model will fail, especially if some of the components of the model fail.  For example, fine-tuned SAM might not segment parts accurately, or the LLM might output an incorrect result under certain circumstances.\n\n\nThese weaknesses don't necessarily diminish the paper's contribution but addressing them would strengthen the work and increase its impact. Many could be addressed through additional experiments and analysis rather than fundamental changes to the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)