A3VLM: Actionable Articulation-Aware Vision Language Model

Authors: siyuan  huang
haonan  chang
yuhan  liu
yimeng  zhu
yuandao  ai
abdeslam  boularias
peng  gao
hongsheng  li
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 19:41:51

Abstract: Vision Language Models (VLMs) for robotics have received significant attention in recent years. As a VLM can understand robot observations and perform complex visual reasoning, it is regarded as a potential universal solution for general robotics challenges such as manipulation and navigation. However, previous robotics VLMs such as RT-1, RT-2, and ManipLLM have focused on directly learning robot actions. Such approaches require collecting a significant amount of robot interaction data, which is extremely costly in the real world. Thus, we propose A3VLM, an object-centric, actionable, articulation-aware vision language model. A3VLM focuses on the articulation structure and action affordances of objects. Its representation is robot-agnostic and can be translated into robot actions using simple action primitives. Extensive experiments in both simulation benchmarks and real-world settings demonstrate the effectiveness and stability of A3VLM.

Main paper: 
 introduction: 
 The use of Large Language Models (LLMs) [4] for robotics applications has recently attracted significant attention. Combined with robot control APIs [5, 6] , open-vocabulary scene graphs [7] , or motion planners [8] , LLMs have demonstrated an extraordinary ability to understand users' commands, reason about the environment, and select the correct action from skill pools. In these works, LLMs usually rely on external vision tools such as an open-vocabulary detector to convert scene information into text. This straightforward combination, however, prevents LLMs from acquiring further scene understanding and certain important details about the environment. Subsequently, researchers have begun to directly train Vision Language Models (VLMs) for manipulation and navigation tasks. Compared to methods that combine LLMs with external vision tools, native VLMs can capture detailed visual data and perform complex visual reasoning, making it possible to be an allin-one solution for solving general manipulation tasks. However, the training of VLMs is extremely data-hungry, and the collection of image-text pair data for robotics VLMs can be a significant problem. Furthermore, the output of VLMs is purely text-based, which is fundamentally different from the standard robot action representation as trajectories. Thus, robot VLMs need to define a tailored and precise action representation for a robotic system.There have been several attempts to address the data gathering and the action representation problems. RT-1 [1] and RT-2 [2] directly represent the robot's end-effector pose using a discretized 6D pose and collect a large amount of image-action pair data using teleoperation. ManipLLM [3] simplifies this action representation by replacing the parallel gripper with a suction gripper, requiring only the computation of a contact point and the gripper's direction. Interaction data is then collected in simulation. Such large amounts of robot interaction data are expensive to collect in the real world.To address this issue, we shift our focus from directly learning actions to learning an object-centeric representation that is independent of the robot's configuration and that can easily be translated into low-level manipulation actions. We propose the Actionable Articulation-Aware VLM (A3VLM), with a representation that describes the object's articulation structure and action affordance simultaneously. Compared to previous robot-centric action representations [2, 3] , A3VLM's representation is object-centric, which makes it possible to learn actionable models of objects without collecting expensive robot interaction data, and the same learned object models can be used by various robots.Given a single RGB image of an unknown object and a language task description, A3VLM locates an actionable part of the object and provides necessary articulation information for a manipulation action. Results on the PartNet-Mobility simulation benchmark show that our proposed A3VLM outperforms previous related models by a large margin. Extensive real-world experiments also demonstrate A3VLM's excellent robustness and potential for real-world robot manipulation applications. 
related work: 
 Manipulation of Articulated Objects. An articulated object refers to an object composed of multiple rigid parts connected by movable joints, such as a drawer or dishwasher. Articulated object manipulation is an important topic in robotics. A common practice for articulated object manipulation is to determine its articulation structure first and then manipulate it with predefined action primitives. Some works [9, 10, 11] use part-based pose estimation methods to locate different articulation structures. Other methods, such as FlowBot3D [12] , predict the per-point articulation movement of every point on the object's 3D point cloud. A subsequent technique, FlowBot++ [13] , predicts for each point an articulation parameter instead of a movement. Although effective, these methods only perform articulation estimation. GaPartNet [14] shares similar insights with our work by combining articulation and affordance. Object links in GaPartNet are classified into nine different articulation prototypes, each with a different canonical pose and affordance. Link poses and affordances are detected from a point cloud. Inspired by GaPartNet, A3VLM also predicts the articulation of each part of the object, as well as its corresponding action prototype. A3VLM simplifies GaPartNet's nine prototypes to only two types, prismatic and revolute. Lastly, most existing articulated object detection and manipulation methods [12, 13, 14] are based on 3D point clouds. 3D point-cloud data in real-world environments can be noisy and inaccurate due to reflections and transparency, as illustrated in Fig. 6 . With the support of a strong VLM backbone, A3VLM is able to predict 3D articulation structures directly from a single RGB image, without any depth data.LLMs/VLMs for Manipulation. Existing LLMs/VLMs for manipulation can be divided into three main categories. The first type, such as Code-as-Policies [15] , Instruct2Act [6] , SayCan [16] , and others, uses LLMs/VLMs to generate high-level semantic action plans in pure text or code. The system then relies on external low-level action models or hand-crafted APIs to execute these plans. This approach heavily depends on the implementation of the low-level skills and APIs, and it is primarily limited to simple tasks such as pick-and-place. In contrast, methods like RT-1 [2] , RT-2 [2] , and ManipLLM [3] aim to generate robot actions directly, which enables them to handle more complex manipulation tasks, such as opening drawers and closing doors. However, action-based LLMs/VLMs typically require a substantial amount of robot-environment interaction data, which can be costly to collect in the real world. Additionally, since this data is often collected from specific types of robots, it cannot be directly used for different robots. The third main category involves using LLMs/VLMs to generate intermediate representations, such as cost maps (VoxPoser [17] ), action constraints (MOKA [18] ), or affordances (ManipVQA [19] ), which are then translated into robot actions using simple action primitives or controls. Our A3VLM falls into this third category. Unlike previous methods, A3VLM focuses on the articulation structure of objects, which enables complex manipulation actions. To the best of our knowledge, A3VLM is the first VLM capable of accurately and consistently locating and understanding the articulation structure for robot manipulation. Unlike the robot-centric action representation in RT-1 [1] , RT-2 [2] and ManipLLM [3] , A3VLM uses an object-centric representation that focuses on the articulations and affordances of the movable parts within an object. Compared to a single articulation detection pipeline such as FlowBot++ [13] , A3VLM can predict the affordance of each part of the object and locate the appropriate part to manipulate based on the desired task. GapartNet [14] has a similar representation. However, GapartNet uses nine different types of articulation prototypes, whereas we unify articulation structures into two basic types: prismatic articulation and revolute articulation.Actionable parts, affordances, and articulation structures in A3VLM are represented as a triad: (Bounding box B, Axis A, Semantic label S). Bounding box B locates an actionable part of interest in the given image. Axis A represents the articulation structure of the part. Semantic label S refers to the articulation type (prismatic or revolute), the link name and the action type. Examples of this representation are shown in Fig. 2 .In practice, the 3D bounding box B is represented by its eight vertices {(x i , y i , z i )} i=1,...,8 . Here, (x i , y i ) is the 2D projected position of vertex i in the given image's plane, and z i is normalized to the range (0,1) using the maximum and minimum depth values. Axis A is represented using two edge points {(α i , β i , γ i )} i=1,2 in the 3D space, and it is normalized using the same method used for the bounding box B. As training a VLM requires colossal resources in terms of data and computation, we do not train A3VLM from scratch. Instead, we fine-tune an established VLM. To fine-tune a VLM, we need to build an instruction-following dataset, where the input is an image and a text prompt, and the answer should be structured text. In this section, we will discuss how to construct this instruction-following dataset. As we mentioned in Sec. 3.1, we use a triad (B, A, S) to describe the location, articulation structure, and action affordance of a movable part. In practice, we do not ask the VLM to generate everything in one inference step but separate the tasks into four different types of sub-tasks. This separation allows the VLM to focus on one concept during each inference. 
instruction-following dataset construction: 
 Raw A3 Annotation Generation. The first step in generating an instruction-following dataset is to create object-level raw annotations. We use PartNet-Mobility [20] , which provides more than 2,000 different articulated objects across 46 categories in URDF format. First, we render these objects into RGB images using PyRender1 , incorporating random camera positions, lighting, and joint values to generate 40 different images for each object. For each image, we use ControlNet [21] to generate augmented images for data augmentation (see Sec. 3.3 for more details on data augmentation).Within each image, we provide an annotation (B, A, S) for each visible and movable link. We categorize all links into prismatic and revolute types. For revolute links, axis A is the rotation axis provided in the URDF. For prismatic links, we use the prismatic direction provided in the URDF as the axis direction, ensuring that axis A passes through the 3D center of the link. After determining axis A, we project the link points along A and compute a minimal 2D bounding box for the projected shape. We use the longer edge of this bounding box as the x-axis, the shorter edge as the y-axis, and axis A as the z-axis for the bounding box B. The center of the bounding box B is the 3D center of the link. The width, height, and length of the bounding box B are computed based on the distance between the furthest points of the link and the center. Semantic information S stores the articulation type, name, and affordable action of the link. Fig. 3 shows multiple annotation examples from different categories in the PartNet-Mobility dataset.Noticeably, the affordable actions of the links in the PartNet-Mobility dataset are not provided. We therefore select the actions from a robotic skill library as defined in [22] . To ensure that the selected skills are compatible with the corresponding links of the objects, we use GPT-4 for skill selection.The prompt used during this process can be found in Appendix C. Sub-tasks Construction. To fit into the established VLM training pipeline, we follow the widely used VLM task templates: Referring Expression Comprehension (REC) and Referring Expression Generation (REG). In a nutshell, REC requires the VLM to provide a bounding box according to a text description, while REG asks the VLM to provide a description of a region within an image referred to by a bounding box. The definitions of REG and REC can be found in [23] . A limitation of the original PartNet-Mobility dataset is the absence of texture details. As depicted in Figure 3 , most objects are rendered in plain gray, which does not reflect real-world conditions. To address the simulation-to-reality (Sim2Real) gap, we employed ControlNet [21] to generate more realistic images, using depth maps as the primary control signal due to their ability to convey both geometric and semantic information. For objects with minimal depth variance, we utilized semantic segmentation as the control signal. To enhance the diversity of the generated images using Stable Diffusion, we employed ChatGPT to generate a broader range of detailed descriptions, enriching the contextual input necessary for producing varied visual outputs. Specific prompts used with ChatGPT and examples of the augmented data are detailed in Appendix G. Model Architecture. A3VLM is developed based on the SPHINX-X [24] with LLaMA2 serving as the language backbone. We select this model as it is uniquely tailored to focus on the partial or regional details of target objects, necessitating fine-grained visual analysis. Our architecture is shown in Fig. 4 . Following the SPHINX "any resolution" approach [25, 24] , the input image is first partitioned into sub-images and then visual encoders are applied to extract visual features. Moreover, given the necessity for both global and local visual grounding ability in manipulation tasks, we integrate the visual encoder from CLIP [26] , DINOv2 [27] to extract local semantic features, and Q-Former [28] for the global visual features summarization. Then, local and global features are channel-wise concatenated. The spatial alignment between visual tokens and language tokens is achieved with projection layers. Bounding box values are normalized to the range (0, 1) and expressed with precision up to two decimal places.Fine-tuning Strategy. As discussed in Section 3.2, our training paradigm follows the conventional Visual Question Answering (VQA) framework and encapsulates all information about articulations within a natural language framework. As a result, the training objective only employs the crossentropy loss, which is a departure from previous works [3, 29] . To bridge the visual disparity between our specialized dataset and generic natural imagery, we employ a two-stage fine-tuning strategy. Initially, the visual projection layers are fine-tuned using straightforward image caption tasks, utilizing a basic template such as "This is a [OBJ]" to generate naive captions. Then, we fine-tune the visual projection layers and LLM simultaneously on the instruction-following dataset. As previously mentioned, A3VLM utilizes an object-centric representation. To translate this into a robot movement, we need to define specific action primitives. A3VLM is designed for use with various types of robots; therefore, it is not optimized for any particular type of end-effector, such as a parallel or a suction gripper. An independent generic grasp-pose proposer is required to generate a list of grasp-pose candidates. During manipulation, we utilize the triad (B, A, S) generated by A3VLM along with the grasp pose candidates.We define three types of action primitives: Rotate, Slide, and Scroll. For a given link, if its corresponding joint type is prismatic, we select the slide action; if it is revolute, we choose the rotate action, unless the target link is semantically labeled as a bottle cap or scroll button, in which case we opt for the scroll action. If the action selected is "scroll", we ensure that the grasp pose overlaps with the rotation axis A. Otherwise, we randomly select a grasp pose within the bounding box B to serve as the contact point C. We then generate a trajectory using C and A for each action type, as illustrated in Fig 5 . These trajectories constitute our generated actions. 
implementation details: 
 We fine-tuned the A3VLM model using the SPHINX framework [24] on eight NVIDIA A100 (80 GB) GPUs. The fine-tuning was completed in three epochs, which took approximately 24 hours. The visual encoders were kept frozen throughout the fine-tuning phase to maintain the integrity of the pre-trained features. We utilized the SPHINX-1K model, sourced directly from the official repository, as our pre-trained base. Training was conducted with a batch size of 4 and a learning rate set to 2 × 10 -5 . To effectively manage the computational load and enhance the training dynamics, we employed a gradient accumulation strategy with a factor of 4.To evaluate the action capabilities of A3VLM, we modified upon the settings of ManipLLM [3] . This benchmark utilizes Sapien [20] as the simulator and PartNet-Mobility [30] objects as the target objects. Object's joint values will be initialized to the middle value. We use a flying Franka Panda Robot's gripper with suction ability as the end-effector. Evaluation Metric. The goal of this benchmark is to evaluate the model's ability to interact with the environment. In each task, we will load an articulated object from PartNet-Mobility dataset. Then, we will drive the end-effector to interact with the articulated object. The base of the object is fixed, so the robot needs to locate a movable part and understand the correct direction to drive the part. Regarding the definition of successful task, we follow the definition in ManipLLM, where we measure the movement of a part in the articulated objects as d. If, after manipulation, the part's movement d exceeds a threshold σ, we define it as a success. We use the same threshold as ManipLLM, where σ = 0.012 .Baselines. We compare the robot manipulation success rates across 20 training and 10 testing categories, evaluating our method against five baselines: (1).Where2Act [31] : This method processes point-cloud input to score potential contact points, selecting the highest scorer. It predicts 100 orientations for the end-effector and chooses the top-scoring one, modified here to use a suction gripper. (2) .UMPNet [32] : This approach manipulates at the predicted contact point, aligning the end-effector's orientation perpendicularly to the object's surface.(3).Flowbot3D [12] : It identifies motion direction in point clouds, selecting the point with the greatest flow magnitude as the interaction point, with the flow direction determining the end-effector's orientation. (4).Implicit3D [33] : This framework uses the Transporter network to identify keypoints on 3D articulated objects, guiding the end-effector's pose for future tasks. (5) .ManipLLM [3] : The state-of-the-art method on this benchmark, it employs a vision-language model to predict the contact point and forward direction of a suction gripper using an RGBD image and language prompt. Action Primitives Details. For each object, we first detect a list of action parts with corresponding bounding boxes B, axes A, joint types, and link names S. We select a random action part from the list and use its bounding box B and axis A to generate two robot trajectories. For example, for a handle of a faucet, we will generate trajectories to rotate it clockwise and counter-clockwise. We will execute these trajectories in two attempts. The task is regarded as successful if it succeed in either try.Result. Real-World Inference Test. To test the stability and accuracy of A3VLM in the real world, we perform an inference test for A3VLM on many objects across different categories. It turns out A3VLM is able to correctly detect the movable parts of the objects and accurately recognize the articulation structure of those parts. As shown in Fig. 6 , A3VLM consistently generates correct inferences on 20 different real-world objects. It is worth-noting that A3VLM is able to correctly perform inference on objects with reflecting or transparent surface (e.g. microwave oven, pot and coke bottle), which are very challenging for point-cloud-based methods due to their inaccurate depth. Object Success 5/5 5/5 5/5 5/5 4/5Real-World Robot Manipulation. To test the effectiveness of the manipulation ability of A3VLM, we selected five different objects from the aforementioned twenty tested categories. We used a Kuka robot equipped with a RealSense D415 depth camera and a Robotiq three-finger gripper. We followed the action primitive mentioned in Sec. 3.5. Five trials with different initial positions are performed for each object. Manip-Figure 6 : A3VLM predictions on various real-world objects using a single RGB image. A3VLM is able to perform correct inference on challenging real world objects, even for those with transparent or reflecting surfaces.ulation is considered successful if the manipulated part has successfully moved more than 40% of its joint limit. Success rate results can be found in Tab.  
conclusions: 
 We present the Actionable Articulation-Aware Vision Language Model (A3VLM), an object-centric robot VLM designed to understand the articulation and action affordances of articulated objects. Unlike previous action-centric robot VLMs, A3VLM does not require any robot interaction data and can be adapted to various robot configurations. Although it is trained solely on simulated data, A3VLM demonstrates significant manipulation capabilities and inference stability across both simulation benchmarks and real-world robot experiments, establishing a new state of the art in this area. We provide further discussions on the limitations in the Appendix. We believe that A3VLM represents a promising direction for future research in robot VLMs. 
 
 Appendix: Although A3VLM shows promising results in both simulation benchmarks and real-world experiments, several limitations remain. First, A3VLM currently operates in an open-loop manner, with no feedback process integrated into the current setup. Exploring how to incorporate feedback into this object-centric pipeline is a valuable direction for future research. Secondly, the z value in A3VLM's output is based on a normalized depth, which requires the user to provide a depth range for the scene to convert it back to real depth. Combining A3VLM with image-to-depth models could address this limitation. Lastly, A3VLM currently focuses on generating single-step manipulation for articulated objects. However, this object-centric representation can be extended to tasks such as pick and place, tool usage, and more. Therefore, expanding A3VLM's representation to encompass a broader range of manipulation tasks is another promising direction.To study the effectiveness of the proposed methods, we conducted the following experiments:• One-Step Reasoning: Instead of the multiple-round QA in the default A3VLM, we formulated the articulated object manipulation tasks in a one-step format, outputting the 3D bounding box and attached axis representation in a single step.• Lower Image Resolution (224 pixels): We trained a model using images with a resolution of 224x224 pixels. This is done by discarding the sub-patches techniques used in our default setting.As shown in Table 4 , one-step reasoning resulted in a slight performance degradation compared to the default setting. We believe the chain-of-thought (CoT) approach reduces task reasoning difficulty. Additionally, results with lower image resolution align with our previous findings with LLaVA, highlighting that higher resolution aids in capturing fine-grained structures and improves generalization ability. Larger image resolutions (beyond 448 pixels) are not supported by the chosen MLLM due to limitations in the original visual foundation models, which generally accept images with 224 pixels. Since A3VLM is trained on purely simulated data, it is necessary to address the discrepancies between simulation and real-world data. In simulations, the image resolution is 960x960, the focal length is 1000 pixels, and the rendered object images lack backgrounds. To align real-world images with those of the simulation, we first tuned the real-world camera's intrinsics to match those of the simulation using a homography transformation. Then, we used the Segment-Anything-Model (SAM) [34] to segment out the background.It is worth noting that the output z-value of the 3D bounding box is not an absolute value but a normalized one. To denormalize it, we need to know the minimal and maximal depths of the scene.For this purpose, we use the depth images from the depth camera. Although real-world depth imagesRole and Task Description: Develop a systematic approach for generating grounding tasks involving object links, where each task involves a limited number of steps and utilizes predefined action primitives from a robot skill library.Robot Skill Library:: Actions available include: "slide_open", "slide_close", "flap_open", "flap_close", "cap", "uncap", "pick", "place", "slide_in", → "slide_out", "wipe", "press", "rotate", "StatusComplete".Requirements and Constraints:1. Tasks and actions must be tailored based on the current status of the link . Regarding the robot experiments, the robot setup is illustrated in Fig. 8 . We translate A3VLM's output into actions using the primitives described in Section 3.5. For generating grasp pose candidates, we can employ tools like the Grasp Pose Generator (GPG) [35] , GraspNet [36] , or define them manually. Since this is not the primary focus of our work, we have simplified the process by manually providing a list of grasp poses.A3VLM takes RGB images as input. Compared with other methods that rely on depth images or point clouds, RGB suffers less noise in real-world experiments. However, as A3VLM is learning a 3D bounding box, a natural concern is whether pure RGB input is less accurate compared with depth input. To explore this problem, we trained a new version of A3VLM, i.e. A3VLM-depth, using depth images as input. To adapt the depth images for the A3VLM pipeline, we first normalized the depth values to the range [0, 1] and then converted them to RGB values.We evaluated the performance of A3VLM and A3VLM-depth on the PartNet-Mobility simulation benchmark as described in Section 4.2. The performance is shown in Tab. 5. From the results, we can see that A3VLM and A3VLM-depth perform similarly on the training categories. However, in the testing categories, A3VLM shows a significant improvement, indicating that the pure RGB input is actually better for generalization. This ablation study confirms our initial hypothesis of using pure RGB as the input modality.In addition to the modalities discussed in Section E, we also explored expanding our experiments to include RGB-D and point cloud modalities.For point cloud inputs, we utilized PointBert [37] and RECON [38] as the point encoders, following practices established in ShapeLLM [38] and PointLLM [39] . The point features were aligned with language features using the Cap3D [40] dataset during fine-tuning of the projection layers. Despite successful training where the Modified Large Language Model (MLLM) produced high-quality captions, the LLM failed to predict partial object bounding box coordinates. This difficulty is attributed to:• Lack of Visual Texture: Point clouds inherently contain fewer visual texture features, complicating the task of partial-level object detection.• Model and Data Limitations: The point cloud models, having fewer parameters and a smaller training dataset compared to visual foundation models, exhibit weaker performance capabilities.In this section, we displayed more data augmentation examples in Figure 9 . The first row and third rows display the raw images rendered directly from the PartNet Mobility, while the second row and the last row display the generated ones. And we used the prompts in the List. 2 to guide the ChatGPT to generate more diverse texture descriptions for the target object.Role and Task Description: You are a good assistant, skilled in providing accurate prompts for stable diffusion.I want to use stable diffusion to draw a category, please give me ten prompts with different styles.Note that the target object is the daily-use item.I already have the descriptions like previous description, please avoid repeats.Only give me the newly generated prompts in a list, and nothing else.¦ Listing 2: An example prompt for guiding ChatGPT to generate diverse texture description for a specific object.https://github.com/mmatl/pyrenderManipLLM uses two thresholds σ = 0.01 and σ = 0.1. But their main evaluation is performed under σ = 0.01. 


Decision: The paper is well motivated, timely, tackles a relevant problem for the robotics community, contains sufficient novelty, is well written, and sufficient experiments are provided.

The requests of the reviewers during the rebuttal phase have sufficiently been addressed.

I recommend acceptance.

Reviews: review: title: title: Though it tackles a specific types of motions/articulation, it provides a practical method that leverages VLM effectively only from simulation but achieves good performance and that has a potential to be generalized to various robots.
paper_summary: paper_summary: It proposes a fine-tuned VLM to predict articulation parameters from an image and demonstrates that it can be performant combined with action primitives.
main_review: main_review: ### Strengths
* Straight forward approach with strong performance on the benchmark.
* Does not require the real-world data collection.
### Weaknesses
* It follows the very same setup as ManipLLM did. If it is generalizable to different robots, it would be better to apply on another robot and compare the performance. It is hard to understand the contribution of the choice of the primitives.
* Limited to the specific actions. If it was effective to fine-tune VLM with carefully simulated data to annotate bounding boxes, it can be also applied to pick and place tasks which can make the model more general and comparable to various models.
* Normalized depth representation requires a depth bound. It is not sure if it can be inference-time generalizable to different depth ranges. This may make the approach less practical to be applied to tasks that are not bounded in a table-top setup. (It uses a depth image to determine the scene bound for depth normalization; it makes the method tightly coupled with the scene and potentially not generalizable to different scenes.)
* Not well explained about why depth or RGBD models did not work well or worse. Leveraging depth can boost the accuracy in 3D in general.
questions: questions: * Depth representation
  * Is the normalized depth best? I wonder if you have tried another representation for 3D bounding-box or depth representation.
  * Are the depth range values fixed during the test and the training? Are they the same?
  * I wonder if this is sensitive to depth boundaries.
* RGBD model
  * I wonder if you tried multi-modal models that also encodes and takes depth images in training.
  * Also, combined with SAM, a bounding-box prediction with an RGBD image might be more efficiently done without assuming the depth range.
* I guess it would be good to have some comparison to robot-centric approaches.
* It would be good to have some ablation on accuracy vs. success rate. (e.g. is an axis error affecting more than the bounding-box error? how tolerant motion primitives are w.r.t axis error or bounding-box error?)
* The model only predicts the bounding-box and axis but not the angle or progression. Was the experiment done in closed-loop control? How fast it is? How the action is done without knowing it is done? Is there any reason not training the progression or angle?
limitations: limitations: 1
review_summary: review_summary: Recommend weak accept. It was easy to read, it proposes a straight-forward approach with good result from a fair comparison. It can be useful in practice if we can combine different tasks in fine-tuning.
title: title: The strength and weakness of A3VLM
paper_summary: paper_summary: This paper provides an unified structure to predict the 3D bounding box of the target object and controll the end effector to manipulate. The fully data collection, training process and experiments demonstrates the effectiveness of the work.
main_review: main_review: **Strengths**:
This paper provides an unified structure regardless of the robotic structure to predict the 3D bounding box of the target object and controll the end effector for mainpulation. 

The authors design a novel chain-of-thought to build the framework by first predict the action type and bounding box for the movable part of the mentioned object and then predict the articulation structure of the object.

The analysis could be utilized to controll the robot for manipulation. 

The data collection process is reasonable and the constructed dataset is valuable for the community.

The model performance in the simulator surpasses all the baseline methods by a large margin, and the realworld experiments are promising.
All the data and code are open-sourced.

**Weakness**:

The limitations of the paper is not well addressed.

The comparison baselines could involve some other vision-language models like llava, gpt4-o to generate the required coordination information in the same process.

More ablation studies like chaning the chain-of-thought or directly predict all the required spatial information, improve the resolution of the image should be added.
questions: questions: See weakness
limitations: limitations: 2
review_summary: review_summary: This paper provides an unified structure regardless of the robotic structure to predict the 3D bounding box of the target object and controll the end effector for mainpulation. The authors design a novel chain-of-thought to build the framework by first predict the action type and bounding box for the movable part of the mentioned object and then predict the articulation structure of the object. The model performance in the simulator surpasses all the baseline methods by a large margin, and the realworld experiments are promising. Thus i recommand accept this paper.
title: title: A3VLM manuscript review
paper_summary: paper_summary: The authors in this work propose a vision language model (VLM), A3VLM, that is object-centric and articulation-aware. Prior work has focused on directly learning robot-centric data for interacting with objects, requiring them to collect a significant amount of robot interaction data that is costly in the real world. A3VLM, instead, focuses on the articulation structure and action affordances of objects, which helps it to be robot-agnostic. This approach is validated through experiments in both simulated benchmarks and real-world scenarios, demonstrating the effectiveness of A3VLM for a variety of object interaction tasks.
main_review: main_review: The authors present a promising method for fine-tuning an open-source VLM model to interact with complex household objects, focusing on object-centricity and articulation awareness. While the approach demonstrates clear merit, the evaluation strategy could be strengthened to more rigorously demonstrate the model's effectiveness and maximize the impact of this work. Specific suggestions for improvement are detailed below:

**Strengths**:
- Robot-Agnostic Generalization: The object-centric representation learning approach employed by A3VLM facilitates knowledge transfer across diverse robotic platforms, requiring minimal platform-specific data and enhancing the model's potential for real-world deployment.
- Real-World Applicability: The explicit modeling of object articulation within A3VLM provides a robust framework for understanding and interacting with complex, articulated objects commonly encountered in real-world scenarios.
- The manuscript clearly and concisely communicates the main research contributions, methodology, and experimental results.

**Weaknesses**
- Evaluation Metric Limitations: The evaluation metric employed in the study, specifically the 1cm part movement success criterion, does not adequately reflect real-world scenarios. In practical applications, this criterion may not be indicative of meaningful object manipulation. Furthermore, the metric fails to account for potential damage caused by excessive force or torque applied orthogonal to the articulation axis, a critical consideration when interacting with articulated objects. Consequently, the results presented in Table 2 may not be fully representative of real-world performance.
- Simulation Experiment Concerns: The use of an arbitrarily small gripper model in certain simulation experiments (e.g., pliers, suitcase, laptop, toaster) raises concerns regarding the generalizability of the findings to real-world scenarios. Physical limitations and potential grasp-related issues in real-world manipulation may significantly impact performance, underscoring the need for caution in interpreting the simulation results.
- Real-World Experiment Limitations: The real-world experiments do not adequately demonstrate the potential consequences of inaccurate articulation parameter estimation. For instance, the pot lid pick-up experiment does not involve a meaningful articulation, limiting its relevance for assessing the VLM's ability to learn effective parameters. The inclusion of grasping and subsequent manipulation in real-world experiments would provide a more comprehensive evaluation of the VLM's capabilities.

**Minor updates**:
- Please add detailed captions in all figures and tables, that briefly describe the salient points of the plots and figures, explaining the axes, legends, and main takeaways.
- In Figure 4, please explain in the caption what different symbols mean, e.g., the fire emoji.
- Please prefer using the term `end-effector` instead of `manipulator` in the text. Usually, in the robotics literature, the term `manipulator` refers to the complete robot arm, and the term `end-effector` refers to the gripper attached at the end of the robot arm.
- Show the speed-up factor in the included videos for the experiments.
- Define `sufficient distance` in line 255
- Prefer citing thoroughly in the text, e.g., cite PartNet-Mobility in line 120, PyRender in line 122.

**Potential Future Extensions**:
Jain et. al. [1] proposed a category-independent articulation model parameterization to represent common articulation types. Using the proposed screw parametrization as the semantic label $\mathcal{S}$ should simplify the formulation while maintaining its generality.

**References**:
- [1] A. Jain, R. Lioutikov, C. Chuck and S. Niekum, "ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory," 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi'an, China, 2021, pp. 13670-13677, doi: 10.1109/ICRA48506.2021.9561132.
questions: questions: Q1. Could the evaluation metric be refined to better account for potential damage caused by excessive force or torque applied orthogonal to the articulation axis, a critical consideration for real-world articulated object interaction? Given the limitations of the current metric, to what extent are the results presented in Table 2 truly representative of real-world performance?

Q2. How is the use of an arbitrarily small gripper model in certain simulation experiments (e.g., pliers, suitcase, laptop, toaster) justified?
limitations: limitations: 2
review_summary: review_summary: The authors present a promising method for fine-tuning an open-source VLM model to interact with complex household objects, focusing on object-centricity and articulation awareness. While the approach demonstrates clear merit, I have reservations about the evaluation strategy used to evaluate the proposed approach. If the authors can provide additional experimental analysis to address my concerns, I would be willing to accept the paper for publication.