A3VLM: Actionable Articulation-Aware Vision Language Model
Authors: Siyuan Huang
Haonan Chang
Yuhan Liu
Yimeng Zhu
Hao Dong
Abdeslam Boularias
Peng Gao
Hongsheng Li
Venue: CoRL 2024 (2024)
Keywords: LLM
VLM
Manipulation
Articulation
TL;DR: We introudce the A3VLM, focusing on the articulation structure and action affordances of objects.

Abstract: Vision Language Models (VLMs) for robotics have received significant attention in recent years. As a VLM can understand robot observations and perform complex visual reasoning, it is regarded as a potential universal solution for general robotics challenges such as manipulation and navigation. However, previous robotics VLMs such as RT-1, RT-2, and ManipLLM have focused on directly learning robot actions. Such approaches require collecting a significant amount of robot interaction data, which is extremely costly in the real world. Thus, we propose A3VLM, an object-centric, actionable, articulation-aware vision language model. A3VLM focuses on the articulation structure and action affordances of objects. Its representation is robot-agnostic and can be translated into robot actions using simple action primitives. Extensive experiments in both simulation benchmarks and real-world settings demonstrate the effectiveness and stability of A3VLM.

A3VLM: Actionable Articulation-Aware Vision Language Model
Siyuan Huang"
Haonan Chang"
Yuhan Liu Rutgers University
Yimeng Zhu Yuandao Al
Shanghai Jiao Tong University Shanghai AI Laboratory
Rutgers University
Hao Dong Peking University
Abdeslam Boularias' Rutgers University
Peng Gaot
Shanghai Al Laboratory
abdeslam. www..aaaaaaarruurrrrsr....
gaopengåpjlab org.
Hongsheng Li CUHK MMLab
Shanghai AI Laboratory CPII under InnoHK
Abstract: Vision Language Models (VLMs) have received significant attention in recent years in the robotics community. VLMs are shown to be able per- form complex visual reasoning and scene understanding tasks, which makes them regarded potential universal solution for general robotics problems such manipulation and navigation. However, previous VLMs for robotics such as RT- RT-2 [2]. and ManipLLMM [3] have focused on directly learning robot-centric actions. Such approaches require collecting a significant amount of robot interac- tion data, which extremely costly the real world. Thus, we propose A3VLM, an object-centric, actionable, articulation-aware vision language model. A3VLM focuses the articulation structure and action affordances of objects. Its repre- sentation is robot-agnostic and can be translated into robot actions using simple action primitives. Extensive experiments in both simulation benchmarks and real- world settings demonstrate the effectiveness and stability of A3VLM. We release our code and other materials at https//gtthbbboommmmahaamnnn///
Keywords: LLM, VLM, Manipulation, Articulation
Wer: Mhtt stould do est paper?
Type provide the F/B 60 171
16.72 25, 8-
952 852,
0...8.3..2...... e3, a7 0,,0 8222),10
80ox &AAel
Figure 1: Sequential inference with prompts. To the first question, A3VLM identifies the correspond- ing action type and bounding box for the movable part of the scissors. Then, the second answer reveals the articulation structure the scissor leg, A manipulation action can be performed based on the answer.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
The use of Large Language Models (LLMs) [4] for robotics applications has recently attracted sig- nificant attention. Combined with robot control APIs [5, 6]. open-vocabulary scene graphs [7], or motion planners [8], LLMs have demonstrated an extraordinary ability understand users' com- mands, reason about the environment, and select the correct action from skill pools. In these works, LLMs usually rely external vision tools such open-vocabulary detector to convert scene information into text. This straightforward combination, however, prevents LLMs from acquiring further scene understanding and certain important details about the environment. Subsequently, re- searchers have begun to directly train Vision Language Models (VLMs) for manipulation and navi- gation tasks. Compared to methods that combine LLMs with external vision tools, native VLMs can capture detailed visual data and perform complex visual reasoning, making it possible to be an all- in-one solution for solving general manipulation tasks. However, the training of VLMs is extremely data-hungry, and the collection of image-tex pair data for robotics VLMs can be significant prob- lem. Furthermore, the output of VLMSs is purely text-based, which is fundamentally different from the standard robot action representation as trajectories. Thus, robot VLMs need to define a tailored and precise action representation for a robotic system.
There have been several attempts to address the data gathering and the action representation prob- lems. RT-1 [1] and RT-2 [2] directly represent the robot's end-effector pose using a discretized 6D pose and collect a large amount of image-action pair data using teleoperation. ManipLLM [3] sim- plifies this action representation by replacing the parallel gripper with suction gripper, requiring only the computation of a contact point and the gripper's 's direction. Interaction data is then collected in simulation. Such large amounts of robot interaction data are expensive to collect in the real world.
To address this issue, we shift focus from directly learning actions to learning an object-centeric representation that is independent of the robot's configuration and that can casily be translated into low-level manipulation actions. We propose the Actionable ArticulationAAwar VLM (A3VLM), with representation that describes the object's articulation structure and action affordance simulta- neously. Compared to previous robot-centric action representations [2, 3], A3VLMM's representation is object-centric, which makes it possible to learn actionable models of objects without collecting expensive robot interaction data, and the same learned object models can be used by various robots.
Given single RGB image of an unknown object and language task description, A3VLM locates an actionable part of the object and provides necessary articulation information for a manipulation action. Results on the PartNet-Mobility simulation benchmark show that our proposed A3VLM out- performs previous related models by a large margin. Extensive real-world experiments also demon- strate A3VLM's excellent robustness and potential for real-world robot manipulation applications.
2 Related Work
Manipulation of Articulated Objects. An articulated object refers to object composed of multi- ple rigid parts connected by movable joints, such as drawer or dishwasher. Articulated object ma- nipulation is an important topic in robotics. A common practice for articulated object manipulation is to determine its articulation structure first and then manipulate it with predefined action primi- tives. Some works [9, 10, 11] use past-based pose estimation methods to locate different articulation structures. Other methods, such as FlowBot3D [12], predict the per-point articulation movement of every point on the object's 3D point cloud. A subsequent technique, FlowBot++ [13], predicts for each point an articulation parameter instead of movement. Although effective, these methods only perform articulation estimation. GaPartNet [14] shares similar insights with our work by combining articulation and affordance. Object links in GaPartNet are classified into nine different articulation prototypes, each with a different canonical pose and affordance. Link poses and affordances are detected from a point cloud. Inspired by GaPartNet, A3VLM also predicts the articulation of each part of the object, as well as its corresponding action prototype. A3VLM simplifies GaPartNer's nine prototypes to only two types, prismatic and revolute. Lastly, most existing articulated object detection and manipulation methods [12, 13, 14] are based 3D point clouds. 3D plont data in real-world environments can be noisy and inaccurate due to reflections and transparency, as

---

illustrated in Fig. 6. With the support of a strong VLM backbone, A3VLM is able to predict 3D articulation structures directly from a single RGB image, without any depth data.
LLMs/VLMs for Manipulation. Existing LLMs/VLMs for manipulation can be divided into three main categories. The first type, such Code-as-Policies [15], Instruct2Act [6], SayCan [16], and others, uses LLMs/VLMs to generate high-level semantic action plans in pure text code. The system then relies on external low-level action models or hand-crafted APIS to execute these plans. This approach heavily depends on the implementation of the low-level skills and APIs. and it is primarily limited to simple tasks such as pick-and-place. In contrast, methods like RT-1 [2], RT- 2 [2], and ManipLLM [3] aim to generate robot actions directly, which enables them handle more complex manipulation tasks, such as opening drawers and closing doors. However, action-based LLMs/VLMMs typically require substantial amount of robot-environment interaction data, which can be costly to collect in the real world. Additionally, since this data is often collected from specific types of robots, it cannot be directly used for different robots. The third main category involves using LLMs/VLMs to generate intermediate representations, such as cost maps (VoxPoser [17]), action constraints (MOKA [18)), or affordances (ManipVQA [19]), which are then translated into robot actions using simple action primitives or controls. Our A3VLM falls into this third category. Unlike previous methods, A3VLM focuses on the articulation structure of objects, which enables complex manipulation actions. To the best of our knowledge, A3VLM is the first VLM capable of accurately and consistently locating and understanding the articulation structure for robot manipulation.
3 Method
3.1 Proposed Articulation Representation
Unlike the robot-centric action representation in RT-1 [1], RT-2 [2] and Ma [3], A3VLM uses object-centric representation that focuses on the articulations and affordances of the movable parts within an object. Compared to a single articulation detection pipeline such as FlowBot++ [13], A3VLM can predict the affor- dance of each part of the object and locate the appropriate part manipulate based the desired task. GapartNet [14] has similar representation. However, GapartNet uses nine different types of ar- ticulation prototypes, whereas we unify articulation structures into two basic types: prismatic articulation and revolute articulation.
Axis
J
S: (Revolute, Handle, Rotate)
DHMIL
Actionable parts, affordances, and articulation structures A3VLM are represented triad: (Bounding box B, Axis A, Se- mantie label S). Bounding box B locates actionable part of inter- est in the given image. Axis A represents the articulation structure of the part. Semantic label S refers to the articulation type (pris- matic or revolute), the link name and the action type. Examples of this representation are shown in Fig. 2.
Axis A
BBox J
S: (Prismatic, Cap, Slide)
Figure 2: Articulation Repre- sentations in A3VLM. Top: Rev- olute handle (rotate). Bottom: Prismatic cap (slide).
In practice, the 3D bounding box represented by its eight ver- tices { V/. :=):=...... Here, (I2, ys) is the 2D projected posi- tion of vertex i in the given image's plane, and z; is normalized to the range (0,1) using the maximum and minimum depth values. Axis A is represented using two edge points {(as, 3t, 7)) in the 3D space, and it normalized using the same method used for the bounding box B3
3.2 Instruction Dataset Construction
Capabiliiies
Tooks
Examples laok Templetes
Noma
Panial Uadesstand
Detection
theie houndigg User: Plase di the bonding box of ezaion this sements deacrlee lid
A7913: Them own manipelaaii
manipullbe
daeir ID
and IlBix
Pantal Oitject Undentaad
LiCk
A3911: Bilas 8
17NK
Anicalation Undersud..
ReG-Jain
User: Tease growinc the jom a 19pe and 30 an incel is otgeat part Bils 8 Link Name
ABVLI, Joint type 5 and in Asts
18K
Astion Groundin
REC
type oaggeed obyeot's
doarthed
notsed bounding bos epposentations by following inaiuation: Open dumags.
Table 1: Overview of the instraction-followingg dataset, including task templates, associated capabilitiss, exam- ples, and sample counts. From top to bottom, 4 different tasks presented: Detection, REC-Link, REG-Joint and REC-Action.

---

Coffee Machine
Faucet
Door
Laplop
Reftigerator
Flash Disk
Cabinet
Figure 3: Annotations used for training A3VLM the PartNet-Mobility [20] dataset. The white box illus- trates the Bounding box B for articulation structure, while the black axis represents the Axis A. This figure shows annotations from different categories.
As training a VLM requires colossal resources in terms of data and computation, we do not train A3VLM from scratch, Instead, we fine-tune an established VLM. To fine-tune a VLM, we need to build an instruction-following dataset, where the input an image and text prompt, and the answer should be structured text. In this section, will discuss how to construct this instruction-follost dataset. As we mentioned in Sec. 3.1, we use triad (B, A, S) to describe the location, articulation structure, and action affordance of movable part. In practice, we do not ask the VLM to generate everything in one inference step but separate the tasks into four different types of sub-tasks. This separation allows the VLM to focus on one concept during each inference.
Raw A3 Annotation Generation. The first step in generating an instruction instruction-follooinng dataset is to create object-level raw annotations. We use MoobiietMobility [20], which provides more than 2,000 different articulated objects across 46 categories in URDF format. First, we render these objects into RGB images using PyRender', incorporating random camera positions, lighting, and joint values to generate 40 different images for each object. For each image, we use ControlNet [21] to generate augmented images for data augmentation (see Sec. 3.3 for more details on data augmentation).
Within each image, we provide an annotation (B,A,S) for each visible and movable link. We categorize all links into prismatic and revolute types. For revolute links, axis A the rotation axis provided in the URDF. For prismatic links, the axis direction, ensuring that axis A passes through the 3D center of the link. After determining axis A, we project the link points along A and compute minimal 2D bounding box for the projected shape. We use the longer edge of this bounding box as the x-axis, the shorter edge as the y-axis, and axis A as the z-axis for the hounding box B. The center of the bounding box B is the 3D) center the link. The width, height, and length of the bounding box B are computed based on the distance between the furthest points of the link and the center. Semantic information S stores the articulation type, name, and affordable action the link. Fig. shows multiple annotation examples from different categories in the MarbiittMo dataset.
the prismatic direction provided in the URDF as
Noticeably, the affordable actions of the links in the PartNet- Mobility dataset are not provided. We therefore select the actions from a robotic skill library as defined in [22]. To ensure that the selected skills are compatible with the corresponding links of the objects, we use GPT-4 for skill selection. The prompt used during this process can be found in Appendix C.
Sub-tasks Construction. To fit into the established VLM training pipeline, follow the widely used VLM task templates: Referring Expression Comprehension (REC) and Referring Expression Generation (REG). In nutshell, REC requires the VLM provide bounding box according to text description, while REG asks the VLM to provide a description of region within an image referred to by bounding box. The definitions of REG and REC can be found in [23].

---

Following these definitions, we construct four different sub-tasks: (1) Detection, (2) REC-Link, (3) REG-Joint, and (4) REC-Action. Each sub-task consists of an image, a text question, and a text answer. Examples of each sub-task question can be found in Tab. 1. The detection task requires the VLM to locate all manipulable parts within an image by outputting a bounding box for each part. Since this task generates more than one bounding box, it does not follow the traditional definition of REC tasks. The REC-Link task involves locating part/link based on description. The REG-Joint task asks the VLM to provide the joint axis A and the joint type S for part specified by bounding box B. REC-Action requires the VI.M to locate movable part by providing bounding box B and corresponding action type S according to an action task, such as "Open the storage."
3.3 Data Augmentation Strategy A limitation of the original PartNet-Mobility dataset is the absence of texture details. As depicted in Figure 3, most objects are rendered in plain gray, which does not reflect real-world conditions. To address the simulation-to-reality (Sim2Real) gap, we employed ControlNet [21] to generate more realistic images, using depth maps as the primary control signal due to their ability to convey both geometric and semantic information. For objects with minimal depth variance, we utilized semantic segmentation as the control signal. To enhance the diversity of the generated images using Stable Diffusion, we employed ChatGPT to generate broader range of detailed descriptions, enriching the contextual input necessary for producing varied visual outputs. Specific prompts used with ChatGPT and examples of the augmented data are detailed in Appendix G.
3.4 Model and Training
LLaMA
thirsty!
Vesal
Open
Action
Projection
Revolute
CLIP
Dinov2
Q-Former
Action primitives
Resized
Sub-putches
Figure 4: The A3VLM architecture. The input image is resized and divided into sub-patches before being processed by visual encoders extract local and global visual features. These features then concatenated and aligned with language tokens through projection layers. The fire emoji indicates that the projection layer and LLaMA are fine-tuned, while the snow emoji signifies that the visual encoders remain frozen.
Model Architecture. A3VLM is developed based on the SPHINX-X [24] with LLaMA2 serving as the language backbone. We select this model as is uniquely tailored to focus on the partial or regional details target objects, necessitating fine- ane-ggrined visual analysis. Our architecture is shown in Fig. 4. Following the SPHINX "any resolution" approach [25, 24], the input image first partitioned into sub-images and then visual encoders are applied to extract visual features. More- over, given the necessity for both global and local visual grounding ability in manipulation tasks, integrate the visual encoder from CLIP [26]. DINOv2 [27] to extract local semantic features, and Q-Former [28] for the global visual features summarization. Then, local and global features are channel-wise concatenated. The spatial alignment between visual tokens and language tokens achieved with projection layers. Bounding box values are normalized to the range (0, 1) and expressed with precision up to two decimal places.
Fine-tuning Strategy. As discussed Section 3.2, our training paradigm follows the conventional Visual Question Answering (VQA) framework and encapsulates all information about articulations within natural language framework. As a result, the training objective only employs the cross- entropy loss, which is departure from previous works [3, 29]. To bridge the visual disparity between specialized dataset and generic natural imagery, employ two-stage fine-tuning
5

---

strategy. Initially, the visual projection layers are fine-tuned using straightforward image caption tasks, utilizing a basic template such as "This [OBJ]" to generate naive captions. Then, we fine-tune the visual projection layers and LLM simultaneously on the instruction-following dataset.
3.5 Action Primitives
M
Axis
Axis d
pileties
Amspees
Slide
Rotate
Scroll
Figure 5: Action primitives for sliding, rotating, and scrolling. Prismatic joints use slide actions, revolute joints use rotate actions, and scroll actions are for semantically labeled targets like control knobs.
As previously mentioned, A3VLM utilizes an object-centric representation. To translate this into robot movement, need to define specific action primitives. A3VLM is designed for with various types of robots; therefore, it is not optimized for any particular type of end-effector, such as parallel or suction gripper. An independent generic grasp-poss proposer is required to generate a list of grasp-pose candidates. During manipulation, we utilize the triad A,, generated by A3VLM along with the grasp pose candidates.
Rotate, Slide, and Scroll. For given link, if its cor- responding joint type is prismatic, select the slide action; if it is revolute, choose the rotate action, unless the target link is semantically labeled as bottle cap or scroll button, in which case we opt for the scroll action. If the action selected is "scroll", we ensure that the grasp pose overlaps with the rotation axis A. Otherwise, we randomly select grasp pose within the bounding box B to serve as the contact point C. We then generate a trajectory using C and A for each action type, as illustrated Fig These trajectories constitute our generated actions.
We define three types action primitives:
Experiment
4.1 Implementation Details
We fine-tuned the A3VLM model using the SPHINX framework [24] on eight NVIDIA A100 (80 GB) GPUs The fine-tuning was completed in three epochs, which took approximately 24 hours. The visual encoders were kept frozen throughout the fine-tuning phase to maintain the integrity of the pre-trained features. We utilized the SPHINX-IK model, sourced directly from the official repository, as pre-trained base. Training
conducted with batch size of 4 and learning rate set to 2 x 10- 5 To effectively manage the computational load and enhance the training dynamics, we employed a gradient accumulation strategy with factor of 4.
4.2 Qualitative Evaluation
To evaluate the action capabilities of A3VLM, we modified upon the settings of ManipLLM [3]. This benchmark utilizes Sapien [20] as the simulator and PartNet-Mobility [30] objects as the target objects. Object's joint values will initialized to the middle value. We use a flying Franka Panda Robot's gripper with suction ability as the end-effector.
Evaluation Metric. The goal of this benchmark is to evaluate the model's ability to interact with the environment. In each task, we will load an articulated object from PartNet-Mobility dataset. Then, will drive the end-effector interact with the articulated object. The base of the object is fixed, so the robot needs locate movable part and understand the correct direction to drive the part. Re- garding the definition of successful task, we follow the definition in ManipLLM,, where we measure the movement of a part in the articulated objects as d. If, after manipulation, the part's movement exceeds a threshold σ, we define it as a success. We use the same threshold as ManipLLM, where a 0.012.
MManiLLLM uses two thresholds = 0.01 and = 0.1. But their main evaluation is performed under 0.01.

---

Baselines. We compare the robot manipulation success rates across 20 training and 10 testing cat- egories, evaluating our method against five baselines: (1). WhereAttt [31]: This method processes point-cloud input to score potential contact points, selecting the highest scorer. It predicts 100 orientations for the end-effector and chooses the top-scoring one, modified here to use a suction gripper. (2). UMPNet [32]: This approach manipulates at the predicted contact point, aligning the end-effector's orientation perpendicularly to the object's surface.(3) (3)oolo)) [12]: It identifies motion direction in point clouds, selecting the point with the greatest flow magnitude as the interac- tion point, with the flow direction determining the end-effector's orientation. (4). Implicitt)) [33]: This framework uses the Transporter network to identify keypoints on 3D articulated objects, guid- ing the end-effector's ' pose for future tasks. (M).anipLLLM [3]: The state-of-the method on this benchmark, employs a vision-language model to predict the contact point and forward direction of suction gripper using an RGBD image and language prompt.
Action Primitives Details. For each object, we first detect a list of action parts with corresponding bounding boxes B, axes A. joint types, and link names S. We select random action part from the list and use its bounding box B and axis A to generate two robot trajectories. For example, for handle of faucet, we will generate trajectories to rotate it clockwise and counter-clockwise, We will execute these trajectories in two attempts. The task is regarded as successful if it succeed in either try.
Result. Table. 2 shows the performance of A3VLM and five other baselines. From the table, it is evident that A3VLM outperforms all baselines by a large margin for most object categories. We at- tribute this improvement to two factors. The first factor is the accurate grounding of actionable parts and articulation structure, and the second the introduction of action primitives. Action primitives enable A3VLM perform various actions on different articulated objects. Further discussion on the benchmark results can be found in Appendix.
Method
Where2Aet
0.26
0.36
0.19
0.27
0.23
0.11
0.15
0.47
0.14
0.24
0.13
0.12
0.56
0.68
0.07
0.60
UMPNet
0.46
0.43
0.15
0.28
0.54
0.32
0.28
0.56
0.41
0.40
0.10
0.23
0.18
0.54
0.20
0.42
FkoBBo3D
0.67
0.55
0.20
0.32
0.27
0.31
0.61
0.68
0.15
0,28
0.36
0.18
0:21
0.70
0.18
0.26
ImplicsAD
0.35
0.28
0.58
0.57
0.34
0.39
ManipLLM
0.68
0.64
0.36
0.77
0.43
0.62
0.65
0.61
0.65
0.52
0.53
0.40
0.54
0.71
0.60
0.64
0.49
0.35
Mehhod
umg
JC
AVG
F
2
2
AVG
Waere2Ast
0.13
0.18
0.13
0.40
0.26
0.18
0.35
0.38
0.28
0.05
0.21
0.17
0.20
0.15
0.15
0.21
UMPNet
0.22
0.33
0.26
0.64
0.35
0.12
0.20
0.35
0:42
0.29
0.20
0.26
0.25
0.25
0.15
0.28
HlowBor3D
0.17
0.29
0.42
0.37
0.25
0.42
0.51
0.13
ImplicatDD
0.27
0.65
0.20
0.33
0.46
0.45
0.17
0.30
0.53
0.15
0.69
0.41
0.31
0.30
0.31
0.41
Ma
0.41
0.75
0:14
0.57
0.59
0.38
0.22
0.81
0.86
0.38
0.85
0:42
0.83
0.26
0.38
0.54
Ouns
0.62
0.50
0.90
0.73
0.91
0.88
0.76
0.74
0.86
0.79
0.67
0.96
0.50
0.62
0.72
0.76
Table 2: Comparisons of our method against baseline methods on PartNet-Mobillity. The first 20
categories are training categories and the following 10 categories are testing categories.
4.3 Real-World Application
Real-World Inference Test. To test the stability and accuracy of A3VLM in the real world, we perform an inference test for A3VLM on many objects across different categories. turns out A3VLM is able correctly detect the movable parts of the objects and accurately recognize the articulation structure of those parts. As shown in Fig. 6, A3VLM consistently generates correct inferences 20 different real-world objects. It is worth-noting that A3VLM is able to correctly perform inference on objects with reflecting or transparent surface (e.g. microwave oven, pot and coke bottle), which are very challenging for point-cloud -ased methods due to their inaccurate depth. Robot Manipulation. To test the effectiveness of the manipulation ability of A3VLM, we selected five differ- ent objects from the aforementioned twenty tested categories. We used Kuka robot equipped with a RealSense D415 depth camera and a Robotiq three-finger gripper. We followed the action primitive mentioned in Sec. 3.5. Five trials with dif- ferent initial positions are performed for each object. Manip-
Table 3: Manipulation success rates the real-world objects.
Object
Success
5/5
5/5
5/5
5/5
4/5
7

---

Figure 6: A3VLM predictions on various real-world objects using a single RGB image. A3VLLM able perform correct inference challenging real world objects, even for those with transparent reflecting surfaces.
ulation is considered successful if the manipulated part has successfully moved more than 40% of its joint limit. Success rate results can be found in Tab. 3, and experiment start and end states are shown in Fig. 7. As shown in Tab. and Fig. 7, A3VLM successfully manipulated the articulated objects in the real world with simple action primitives. Further implementation details on the robot experiments can be found in Appendix D.
Start
End
Pot
Dispenser
Microwave Oven
Bottle
Box
Figure 7: Real-worl world manipulation experiments using a Kuka robot with a Robotiq hand.
5
Conclusions
We present the Actionable Articulation-Aware Vision Language Model (A3VLM), an object-centric robot VLM designed understand the articulation and action affordances of articulated objects. Unlike previous action-centric robot VLMMs, A3VLM does not require any robot interaction data and can be adapted various robot configurations. Although it trained solely on simulated data, A3VLM demonstrates significant manipulation capabilities and inference stability across both sim- ulation benchmarks and real-world robot experiments, establishing area. We provide further discussions on the limitations in the Appendix. We believe that A3VLM represents a promising direction for future research in robot VLMs.
state of the art in this
Acknowledgments
This project funded in part by the National Key R&D Program of China Project 2022ZD0161 100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Tech- nology Commission (ITC)'s InnoHK, by Smart Traffic Fund PSRI/76/211//PR by RGC General Research Fund Project 14204021. Hongsheng Li is PI of CPII under the InnoHK. Moreover, this work is partially supported by NSF awards 1846043 and 2132972.

---

References
[1] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068/ 2022.
[2] A. Brohan, N. Brown, Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Dricss, A. Dubey, Finn, al. Rt-2: aasiionllnguage-acce models transfer web knowledge to robotic control. arXiv preprini arXiv:2307.158/8 2023.
[3] X. Li, M. Zhang. Y. Geng, H. Geng, Y. Long, Y. Shen, R. Zhang, J. Liu, and H. Dong. Ma- niplim: Embodied multimodal large language model for object-centric robotic manipulation. arXiv preprint arXiv:2312.16217 2023.
[4] Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[5] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500. IEEE, 2023.
[6] Huang, Z. Jiang, H. Dong, Qiao, Gao, and H. Li. Instruct2act: Mapping multi-modality instructions robotic actions with large language model. arXiv preprint urXiv:2305.11171 2023.
[7] H. Chang, K. Boyalakuntla, S. Lu, S. Cai, E. Jing, S. Keskar, S. Geng, A. Abbas, L. Zhou, K. Bekris, et al. Context-awaad entity grounding with open- vocabbocyyy 3d scene graphs. arXiv preprint arXiv:2309. 15940, 2023.
[8] H. Chang, K. Gao, K. Boyalakuntla, Lee, B. Huang, H. U. Kumar, Yu, and A. Boularias. Lgmets: Language-guided monte-carlo tree search for executable semantic object rearrange- ment. arXiv preprint (tti::2309//822/1 2023.
[9] R. Hu, W. Li, O. Van Kaick, A. Shamir, H. Zhang, and H. Huang. Learning to predict part mobility from single static snapshot. ACM Trans. Graph., 36(6), nov 2017. ISSN 0730-0301. doi: 10.1 15550333300003111
URL https: / //doi. ooceooo..ogg////eooeooo 111/1/00114//3110000000 011.5880000.3381....
[10] X. Li, H. Wang, Yi, L. Guibas, A. L. Abbott, and Song. Category-level articulated object pose estimation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020.
/1l11/////224200020000000 10. 1109/11////// cv000/00vp2222000. 2020 00376.
URL http://////xx.o http:/xxxxooooooooooooooooooo
[11] X. Wang, B. Zhou, Y. Shi, X. Chen, Q. Zhao, and K. Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2019. doi:10. 110:000109///yp22119.00000 URL http://ás doi orgo/r///// 10. 1109/CVPR. 20199 2019.00900.
[12] B. Eisner, H. Zhang, and D. Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint urXiv:2205.04388 2022.
[13] H. Zhang, B. Eisner, and D. Held. Flowbot++: Learning generalized articulated objects ma- nipulation via articulation projection. arXiv preprint arXiv:2306. /2893, 2023.
[14] H. Geng, H. Xu, Zhao, C. Xu, Yi, S. Huang, and H. Wang- Gapartnet: Cross-category domain--enerraizablee object perception and manipulation via generalizable and actionable parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 708 10791,, 2023.

---

[15] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:220900755,, 2022.
[16] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
[17] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.0597, 2023.
[18] F. Liu, K. Fang, P. Abbeel, and S. Levine. Moka: Open-vocabulary robotic manipulation through mark- visual prompting, 2024.
[19] S. Huang, I. Ponomarenko, Z. Jiang, X. Li, X. Hu, P. Gao, H. Li, and H. Dong. Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large lan- guage models. arXiv preprint arXiv: /289, 2024.
[20] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11097-11101 11107, 2020.
[21] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[22] H. Bharadhwaj. J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar. Roboagent: Gener- alization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv preprint 0.XXi:23009019188 2023.
[23] K. Chen, Z. Zhang, W. Zeng. R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15195 2023.
[24] Gao, R. Zhang, Liu, L. Qiu, S. Huang, W. Lin, S. Zhao, Geng, Z. Lin, P. Jin, et al. Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprini 240:::240..05335 2024.
[25] Z. Lin, C. Liu, R. Zhang, P. Gao, L. Qiu, H. Xiao, H. Qiu, C. Lin, W. Shao, K. Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-mode large language models. arXiv preprint arXiv:2311.00 135.0533 2023.
[26] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 3748-8763. PMLR, 2021.
[27] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafranice, V. Khalidov, Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.07719,, 2023.
[28] J.BB. Alayrac, J. Donahue, Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Mil- lican, M. Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-233
2022.
[29] S. Qian, W. Chen, M. Bai, X. Zhou, Z. Tu, and L. Li. Affordancellm: Grounding affordance from vision language models. arXiv preprint ar arXiv:2401.06341, 2024.
[30] K. Mo, Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, and H. Su. Partnet: large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pages 909-918, 2019.
10

---

[3]] K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani. Where2act: From pixels to actions for articulated 3d objects. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6813-6823, 2021.
[32] Z. Xu, Z. He, and S. Song. Universal manipulation policy network for articulated objects. IEEE robotics and automation letters, ((2):2447-244 2022.
[33] C. Zhong, Y. Zheng, Y. Zheng, H. Zhao, L. Yi, X. Mu, L. Wang, P. Li, G. Zhou, C. Yang. al. 3d implicit transporter for temporally consistent keypoint discovery. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3889,,880, 2023.
[34] A. Kirillov, E. Mintun, Ravi, Mao, Rolland, Gustafson, Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, al. Segment anything. arXiv preprint 0tti::234400244333 2023.
[35] M. Gualtieri, A. ten Pas, K. Saenko, and R. Platt. High precision grasp pose detection in dense clutter, 2016.
[36] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Craspnct-lbillion A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11444-1144 1143,, 2020.
[37] X. Yu, L. Tang, Y. Rao, T. Huang, Zhou, and Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 19313-19322, 2022.
[38] Qi, R. Dong, Zhang, H. Geng, Han, Z. Ge, L.. Yi, and K. Ma. Shapellm: Universal 3d object understanding for embodied interaction. arXiv preprint arXiv:240..17766, 2024.
[39] R. Xu, X. Wang. Wang, Y. Chen, J. Pang, and D. Lin, Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308. 16911, 2023.
[40] T. Luo, C. Rockwell, H. Lee, and J. Johnson. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems, 36, 2024.

---

A Limitation and Future Direction
Although A3VLM shows promising results in both simulation benchmarks and real-world experi- ments, several limitations remain. First, A3VLM currently operates an open-loop manner, with no feedback process integrated into the current setup. Exploring how to incopporate feedback into this object-centric pipeline is valuable direction for future research. Secondly, the value in A3VLM's output based on normalized depth, which requires the user to provide a depth range for the scene to convert it back to real depth. Combining A3VLM with image-to-depti models could address this limitation. Lastly, A3VLM currently focuses on generating single-step manipulation for articulated objects. However, this object-centric representation can be extended to tasks such as pick and place, tool usage, and more. Therefore, expanding A3VLM's representation to encompass a broader range of manipulation tasks is another promising direction.
B Further Ablations
To study the effectiveness of the proposed methods, we conducted the following experiments:
One-Step Reasoning: Instead of the multiple-round QA in the default A3VLM, for- mulated the articulated object manipulation tasks in one-step format, outputting the 3D bounding box and attached axis representation in single step.
Lower Image Resolution (224 pixels): We trained model using images with resolution of 224x224 pixels. This is done by discarding the sub-patches techniques used in our default setting.
As shown in Table 4, one-step reasoning resulted in a slight performance degradation compared to the default setting. We believe the chain-of-thought (CoT) approach reduces task reasoning dif- ficulty. Additionally, results with lower image resolution align with our previous findings with LLaVA, highlighting that higher resolution aids in capturing fine-grained structures and improves generalization ability. Larger image resolutions (beyond 448 pixels) are not supported by the chosen MLLM due to limitations in the original visual foundation models, which generally accept images with 224 pixels.
Table 4: Ablation Studies.
CoT Inference
Higher Img Res
Train-Category
Test-Category
All-Avg
0.87
0.74
0.85
0.88
0.50
0.82
0.91
0.76
0.88
C Prompt Engineering for action affordance generation
D Real-world Experiments Details
Since A3VLM is trained on purely simulated data, necessary to address the discrepancies be- tween simulation and real-world data. In simulations, the image resolution is 960x960, the focal length is 1000 pixels, and the rendered object images lack backgrounds. To align real-world im- ages with those of the simulation, we first tuned the real-world camera's 's intrinsics to match those of the simulation using a homography transformation. Then, we used the Segmenn-Anytthng-Mooee (SAM) [34] to segment out the background.
worth noting that the output z-value of the 3D bounding box is not an absolute value but normalized one. To denormalize it, need to know the minimal and maximal depths of the scene. For this purpose, use the depth images from the depth camera. Although real-world depth images
12

---

Description:
Develop syatematic approach for generating grounding tasks involving object links, vhere each task involves a limited nunber of stepe and utilizes predefined action prinitives fron robot skill 1ibrary.
Robot Skill Library::
Actions available include:
"elide_opea", siide"" "flap_open", close" "cap", "uncap", "pick", "place", "slide_in", "alide_out", "uipe", "press", "rotate", "StatuaComplate".
Requiremente and Constrainte:
1. Tasks and actions nust be tailored based an the current status of the 1ink joint types: prisnatic, revolute, static, etc. 3. All actions nust be sourced exclusively from the provided skill library.
may
The actions should corresponding actions
sequence length, order, and sementice. Create tasks involving both single and multiple links where applicable. add conponents explicitly specified.
Exemples:
extampess...
svilible links information
Instruction:
Now please generate the tasks and actions for the OBJECT_CLASS'S 1ink part with the Links LINK_INFO. generated
previous folloving BISTORY GENERATION, please nake sure tasks and actions are different from the previous anes.
Please ONLY generate the tasks and actions in the valid juon fornat.
Listing I: An example prompt for guiding ChatGPT to generate the action grounding task for specific object.
are full of noise and inaccuracies, we can still use the minimal and maximal depth values, as the relative error is small.
Regarding the robot experiments, the robot setup is illustrated in Fig. 8. We translate A3VLM's output into actions using the primitives described in Section 3.5. For generating grasp pose can- didates, we can employ tools like the Grasp Pose Generator (GPG) [35], GraspNet [36], or define them manually. Since this is not the primary focus of our work, we have simplified the process by manually providing a list of grasp poses.
Discussion on Input Modality
A3VLM takes RGB images as input. Compared with other methods that rely on depth images or point clouds, RGB suffers less noise real-world experiments. However, as A3VLM learning a 3D bounding box, a natural concern is whether pure RGB input is less accurate compared with depth input. To explore this problem, we trained a new version of A3VLM, i.c. A3VLM-depth, using depth images input. To adapt the depth images for the A3VLM pipeline, first normalized the depth values to the range [0. I] and then converted them to RGB values.
We evaluated the performance of A3VLM and A3VLM-depth on the PanNNetMobility simulation benchmark as described in Section 4.2. The performance is shown Tab. 5. From the results, we can see that A3VLM and A3VLM-depth perform similarly on the training categories. However, in the testing categories, A3VLM shows significant improvement, indicating that the pure RGB input is actually better for generalization. This ablation study confirms our initial hypothesis of using pure RGB as the input modality.
F Exploration on More Input Modalities
In addition the modalities discussed in Section E, also explored expanding experiments to include RGB-D and point cloud modalities.
13

---

Bbka
Figure 8: Robot setup: we use a Kuka robot with a three-fingers Robotiq hand. A realsense d415 camera placed the side capture RGBD images. The depth camera the table is not used for this experiment.
Method
0.30
0.86
0.97
A3VLM-depp depch
0.92
0.89
0.93
1.00
0.35
0.56
1.00
0.63
0.72
0.87
1,00
0.67
0.50
0.95
0.29
0.36
ASVLM
0.79
0.96
0.72
AJVLM-depch
Table 5: Comparisons of A3VLM against A3VLM-depth. The first categories training categories following 10 categories are testing categories. A3VLM with RGB as the input modality has better performance and generalization ability.
RGB-D Modality
For the RGB-D experiments, RGB and depth images were processed separately using distinct visual encoders. The outputs, termed visual tokens, were appended sequentially-wiit special tokens img > and < depth > indicating their respective modalitics then fed into the Large Language Model (LLM). However, the LLM struggled with instruction following, likely due two main challenges:
<
Domain Gap:
The visual foundation models, originally pre-trained only on natural RGB images, fail to reliably extract features from depth images that lack visual textures.
The combination of tokens from both modalities resulted in excessively long input sequences, which the LLM could not process effectively due to its limitations in handling long sequences.
Token Length:
14

---

F.2
Point Cloud Modality
For point cloud inputs, we utilized PointBert [37] and RECON [38] as the point encoders, following practices established in SL [38] and PointLLM [39]. The point features were aligned with language features using the Cap3D [40] dataset during fine-tumin tine-tuning of the projection layers. Despite successful training where the Modified Large Language Model (MLLM) produced high-quality cap- tions, the LLM failed to predict partial object bounding box coordinates. This difficulty is attributed to:
Lack of Visual Texture: Point clouds inherently contain fewer visual texture features, com- plicating the task of partial-level object detection.
Model and Data Limitations: The point cloud models, having fewer parameters and smaller training dataset compared to visual foundation models, exhibit weaker performance capabilities.
G Data Augmentation Examples
In this section, we displayed more data augmentation examples in Figure 9. The first row and third rows display the raw images rendered directly from the PartNet Mobility, while the second row and the last row display the generated ones. And used the prompts in the List. 2 to guide the ChatGPT generate more diverse texture descriptions for the target object.
Bole and Task Description: You good assistant, exilled providing accurate proupte for stable diffusion.
stable diffusion to dras catagory,
prompts with diffarent styles.
targat object daily-use
slready have the descriptions like nrevioue.descripe please avoid repeats.
Only give ne the novly generated prompts in a list, and nothing else.
Listing 2: An example prompt for guiding ChatGPT to generate diverse texture description for specific object.
15

---

Figure 9: Comparison between raw images with the Stable Diffusion generated ones.
16


Reviews:
[{"review_id": "rl9FKSngpp", "reviewer": "Reviewer_tJSN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "It proposes a fine-tuned VLM to predict articulation parameters from an image and demonstrates that it can be performant combined with action primitives.", "summary_of_recommendation": "Recommend weak accept. It was easy to read, it proposes a straight-forward approach with good result from a fair comparison. It can be useful in practice if we can combine different tasks in fine-tuning.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "* Depth representation\n  * Is the normalized depth best? I wonder if you have tried another representation for 3D bounding-box or depth representation.\n  * Are the depth range values fixed during the test and the training? Are they the same?\n  * I wonder if this is sensitive to depth boundaries.\n* RGBD model\n  * I wonder if you tried multi-modal models that also encodes and takes depth images in training.\n  * Also, combined with SAM, a bounding-box prediction with an RGBD image might be more efficiently done without assuming the depth range.\n* I guess it would be good to have some comparison to robot-centric approaches.\n* It would be good to have some ablation on accuracy vs. success rate. (e.g. is an axis error affecting more than the bounding-box error? how tolerant motion primitives are w.r.t axis error or bounding-box error?)\n* The model only predicts the bounding-box and axis but not the angle or progression. Was the experiment done in closed-loop control? How fast it is? How the action is done without knowing it is done? Is there any reason not training the progression or angle?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tWXmTn6OkU", "reviewer": "Reviewer_FBJx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper provides an unified structure to predict the 3D bounding box of the target object and controll the end effector to manipulate. The fully data collection, training process and experiments demonstrates the effectiveness of the work.", "summary_of_recommendation": "This paper provides an unified structure regardless of the robotic structure to predict the 3D bounding box of the target object and controll the end effector for mainpulation. The authors design a novel chain-of-thought to build the framework by first predict the action type and bounding box for the movable part of the mentioned object and then predict the articulation structure of the object. The model performance in the simulator surpasses all the baseline methods by a large margin, and the realworld experiments are promising. Thus i recommand accept this paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "See weakness", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rh4riFaopk", "reviewer": "Reviewer_W221", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors in this work propose a vision language model (VLM), A3VLM, that is object-centric and articulation-aware. Prior work has focused on directly learning robot-centric data for interacting with objects, requiring them to collect a significant amount of robot interaction data that is costly in the real world. A3VLM, instead, focuses on the articulation structure and action affordances of objects, which helps it to be robot-agnostic. This approach is validated through experiments in both simulated benchmarks and real-world scenarios, demonstrating the effectiveness of A3VLM for a variety of object interaction tasks.", "summary_of_recommendation": "The authors present a promising method for fine-tuning an open-source VLM model to interact with complex household objects, focusing on object-centricity and articulation awareness. While the approach demonstrates clear merit, I have reservations about the evaluation strategy used to evaluate the proposed approach. If the authors can provide additional experimental analysis to address my concerns, I would be willing to accept the paper for publication.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Q1. Could the evaluation metric be refined to better account for potential damage caused by excessive force or torque applied orthogonal to the articulation axis, a critical consideration for real-world articulated object interaction? Given the limitations of the current metric, to what extent are the results presented in Table 2 truly representative of real-world performance?\n\nQ2. How is the use of an arbitrarily small gripper model in certain simulation experiments (e.g., pliers, suitcase, laptop, toaster) justified?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper is well motivated, timely, tackles a relevant problem for the robotics community, contains sufficient novelty, is well written, and sufficient experiments are provided.

The requests of the reviewers during the rebuttal phase have sufficiently been addressed.

I recommend acceptance.
Decision: Accept