Autonomous Interactive Correction MLLM for Robust Robotic Manipulation

Authors: chuyan  xiong
chengyu  shen
xiaoqi  li
kaichen  zhou
jeremy  liu
ruiping  wang
hao  dong
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-27 17:35:36

Abstract: The ability to reflect on and correct failures is crucial for robotic systems to interact stably with real-life objects. Observing the generalization and reasoning capabilities of Multimodal Large Language Models (MLLMs), previous approaches have aimed to utilize these models to enhance robotic systems accordingly. However, these methods typically focus on high-level planning corrections using an additional MLLM, with limited utilization of failed samples to correct low-level contact poses which is particularly prone to occur during articulated object manipulation. To address this gap, we propose an Autonomous Interactive Correction (AIC) MLLM, which makes use of previous low-level interaction experiences to correct SE(3) pose predictions for articulated object. Specifically, AIC MLLM is initially fine-tuned to acquire both pose prediction and feedback prompt comprehension abilities. We design two types of prompt instructions for interactions with objects: 1) visual masks to highlight unmovable parts for position correction, and 2) textual descriptions to indicate potential directions for rotation correction. During inference, a Feedback Information Extraction module is introduced to recognize the failure cause, allowing AIC MLLM to adaptively correct the pose prediction using the corresponding prompts. To further enhance manipulation stability, we devise a Test Time Adaptation strategy that enables AIC MLLM to better adapt to the current scene configuration. Finally, extensive experiments are conducted in both simulated and real-world environments to evaluate the proposed method. The results demonstrate that our AIC MLLM can efficiently correct failure samples by leveraging interaction experience prompts.

Main paper: 
 introduction: 
 Developing a versatile robot has always been a core goal in embodied artificial intelligence. Despite various generalization strategies proposed [1, 2, 3, 4] , failures remain inevitable due to the complexity of real-world environments. Therefore, beyond directly enhancing the robot's generalization ability, it is also crucial to enable the robot to reflect on and correct its failure actions [5, 6, 7] .In recent years, the powerful common-sense generalization and reasoning abilities of Multimodal Large Language Models (MLLMs) [8, 9, 10, 11, 12, 13] have attracted the attention of robotic researchers. These capabilities make MLLMs and LLMs naturally suited for understanding, analyzing, explaining failures, and even making corrections based on them. REFLECT [6] innovatively summarizes hierarchical sensor information and inputs it into an LLM in textual form to obtain explanations for errors. Additionally, other works [14, 15, 16, 7, 17, 18, 19, 20] are also exploring the ways and capabilities of LLMs in handling errors in the field of robotics.Figure 1 : Correction process of AIC MLLM. Given a failed interaction, we first extract feedback information regarding the object's geometry, then enable the model to reflect and correct both position and rotation estimation, thereby generating a more accurate SE(3) pose which will be executed by robot. We use a right-hand coordinate system to show the end effector's rotation pose. In the rightmost image, the yellow line represents the x-axis, the red line represents the y-axis, and the red line represents the z-axis.While these works can reflect and correct high-level tasks, such as task planning, they still struggle to correct low-level SE(3) pose actions and still fail to complete the atomic tasks decomposed by task planning. For example, even if a robot recognizes that it failed to open a microwave to place food inside, it may still encounter difficulty in selecting the correct contact pose, ultimately resulting in task failure. In such cases, previous methods cannot automatically recover from these errors [6] . To address this gap, we focus on low-level pose correction for atomic tasks. Among these tasks, articulated objects require more precise interaction poses due to their multiple interconnected parts that can move relative to each other, leading to higher degrees of freedom and increased complexity in manipulation. As a result, tasks involving articulated objects are more prone to errors and urgently need effective correction mechanisms. Therefore, we aim to develop the correction of articulated object manipulation. Given that learning from failed scenarios is an inherent capability of humans, we aim to enhance the robot's ability to understand geometry by utilizing failed attempts, and thus improve its generation of low-level actions.Leveraging this insight, we propose an Autonomous Interactive Correction (AIC) MLLM, which utilizes previous low-level interaction experiences to correct manipulation SE(3) poses. First, we fine-tune a pre-trained MLLM to enable it not only to predict manipulation poses but also to understand the instructions provided by the feedback information extraction system and reflect on them for correction. To realize this goal, we carefully design two types of prompt instructions to assist the MLLM in failure correction, including position and rotation prompts. Specifically, we create visual masks to highlight unmovable parts for position correction and use textual descriptions to indicate potential manipulation directions for rotation correction. During inference, the model initially performs pose prediction. If an error occurs, the Feedback Information Extraction module is introduced to detect the cause of failure, enabling the AIC MLLM to adaptively correct the end-effector pose prediction using the corresponding prompts. To further enhance manipulation stability, we devise a Test Time Adaptation strategy for AIC MLLM. This allows our model to iteratively update itself during inference and adapt to the current scene configuration. With our proposed AIC MLLM, the robot can efficiently improve its manipulation stability through interactive attempts.The main contributions of this paper can be summarized as follows: 1) We introduce AIC MLLM, a framework that utilizes MLLM for correcting SE(3) pose predictions by learning from low-level interaction failures. 2) We design visual and textual prompts to guide position and rotation corrections, and incorporate a feedback information extraction module to adaptively correct pose predictions based on identified failure causes. 3) We implement a test-time adaptation module to enhance manipulation stability. Extensive experiments demonstrate the efficacy of AIC MLLM in both simulated and real-world environments. 3. The contact point is at (𝑥, 𝑦), the gripper up direction is [𝑥 u , 𝑦 u , z u ].1. Yes.2. Yes, no, yes…….no.4. The gripper up direction is [𝑥 u , 𝑦 u , z u ].Figure 2 : Training of AIC MLLM. We gradually enable the model to predict poses and comprehend both visual and textual feedback prompts including object parts and axis information.gories: human-guided correction and self-correction. the former approaches [17, 7, 15] like YAY robot [17] and DROC [7] utilize human language correction to adjust robot behavior. Conversely, self-reflective methods [18, 14, 6, 16, 19, 20] such as inner monologue [14] and REFLECT [6] enable robots to learn from errors and correct autonomously. We focus on the latter type, aiming to develop a system capable of automatic self-reflection and correction at low-level pose stage. Due to space limitations, we provide additional related work in Appendix A. 
task formulation: 
 We define the manipulation task as follows: the policy model π predicts a manipulation contact pose a(a pos , a rot ) based on the current scene image I and the instruction L, which includes the required action primitive for the task. After contacting the object, we use the axis direction of end-effector as the pose trajectory after contacting (for pull primitive we use red line y-axis in Fig. 1 ) . The model π is allowed N opportunities to correct a, and during the testing phase, the model can update itself using only the current sample.During training (3.3) , we enable the model to predict poses, and comprehend given feedback prompts, such as visual masks highlighting unmovable part or text describing axis information, and correct the pose prediction accordingly. During testing, if interaction fails, we use an Feedback Information Extraction (FIE) module to obtain error information related to the geometry articulated object (3.4.1) . This error information is then fed back into the trained model, allowing it to reflect on the failure's cause and predict a new SE(3) pose for interacting with the object, aiming to achieve successful manipulation (3.4.2). To enable the model to better adapt to the current testing configuration, we devise a Test Time Adaptation (TTA) strategy. This strategy allows the model to learn from the ongoing sample and improve its ability to handle upcoming samples (3.4.3).In this section, we introduce how we train our model capable of both executing manipulation tasks and correcting failure interaction based on feedback. To achieve this goal, we first enable our model to output the manipulation pose which contains both position and rotation, enabling interaction with the object. Next, to enable our model to correct failure interaction based on feedback, we have constructed two prompt interfaces: a visual mask prompt and linguistic prompts. These interfaces convey part positional and rotational information of the object, extracted from failure samples, to the model. This enables the model to generate new predictions based on this prior knowledge.Based on our observations, if the model predicts points on unmovable parts, no method can successfully manipulate these parts. Therefore, to prevent the model from making invalid interactions on unmovable parts, we summarize these positional error information by segmenting out unmovable part, and enable our model to comprehend it. Specifically, we denote a part of the articulated object as unmovable by applying a red-colored mask. We then ensure that our model comprehends the red mask on image and avoid from predicting on unmovable parts through progressively training with the following three VQA tasks:(1) Mask Image Classification To ensure our model recognizes unmovable parts labeled with a red mask, we first train it to detect whether a mask is present in the image. We add a red mask to the unmovable parts in the simulator and prompt the model with the question, "Is there a red mask in the image?" The ground truth answers are "Yes" or "No," supervised using cross-entropy loss L H .(2) Mask Position Reasoning After the model learns what a mask is, we need it to identify its location. We randomly select N pixel points (x 1 , y 1 )...(x N , y N ) and ask the model to determine whether each point is on the mask. As shown in the second prompt on Fig. 2 , we format the text prompt with corresponding answer of "Yes" and "No" based on the ground truth mask, supervised using cross-entropy loss L P .(3) Correct Based On Mask In the third prompt of Fig. 2 , once the model comprehends the mask and its location, we prompt it to predict a new pose based on the constraint of unmovable part. Given that MLLM are good at handling discretized data, we discretize the direction vectors into 100 bins, with each bin spanning 0.02. This task is supervised under cross-entropy loss L C .Position-related information only affects the correction of the contact point position, while pose correction also requires adjusting the gripper direction. For articulated objects, the most crucial information for determining a valid gripper direction is the type and direction of the joint. There are primarily two types of joints: prismatic and revolute. We observe that often, even when the part of the articulated object fails to reach the expected state, it still exhibits some degree of movement. For instance, when a robotic arm tries to open a door but fails to fully complete it, external forces may still cause the door to move to some extent. Therefore, we utilize the movement in this interaction to capture information about the type of joint and the axis direction, enabling the model to reflect on and correct its actions accordingly. Specifically, as shown in the last prompt in Fig 2 , we incorporate the joint type information and direction information into the text prompt, modeling it as a VQA task. The model is required to provide new direction predictions based on joint information, the current contact point, and the current normal direction, which can also provide some hints for contact direction prediction. This task is supervised under cross-entropy loss L R . Once the training data is obtained from the simulator, we inject noise to simulate test conditions and improve robustness. See Appendix B.1 for details.We jointly train the four tasks under the objectiveL = L H + L P + L C + L R .We introduce the mechanism our framework employs during test testing phase. Specifically, given an object image, the model predicts the 2D contact point and rotation. The contact point is subsequently transformed into 3D coordinates using a depth map, enabling initial interaction with the object. If the interaction fails, we use an FIE module(3.4.1) to extract feedback information from previous failure attempt. This feedback is integrated into visual and linguistic prompts fed into the trained model, allowing it to reflect and correct, and generate new action predictions(3.4.2). Additionally, after inference on each test sample, the model undergoes parameter updates in the TTA module (3.4.3) to better generalize to configuration. The complete prompts in Fig. 3 During training, essential information like joint types is obtained from the simulation environment. However, during testing, these data need to be extracted automatically through specific mechanisms. This section will describe the module designed to accomplish this task.Position-related Information Extraction During interaction, our goal is to determine if the predicted contact point lies on unmovable part. If so, our aim is to pinpoint the unmovable part's location and prevent from any subsequent operations on it. Specifically, we first employ the off-the-shelf VIP-LLaVA [21] to confirm whether it is unmovable. This model allows users to mark images with visual prompts, i.e., a dot on the image, and interact with the model via language queries. Therefore, we draw the predicted contact point as a red dot on the object. Then, as shown in Fig. 3 , we input this image with the textual prompt to VIP-LLaVA, obtaining its response to determine if the part is movable. If contact point is on unmovable part, we then use SAM [22] to perform hierarchical segmentation with the red dot as the prompt, selecting the smallest mask to represent the immovable part. Note that multiple failed interactions with an object may occur, so the unmovable parts information generated from each interaction will accumulate as multiple red masks on the object.Our goal is to obtain joint type and axis direction based on the history interaction attempt. When interacting with the object, e.g, when pulling a door, we can determine the trajectory of the end effector. If the trajectory is a straight line, we classify it as a prismatic axis. Conversely, if the trajectory is of an arc shape, we classify it as a revolute axis. To determine the axis direction, for prismatic axis, we consider the axis direction to be the same as its direction of movement. For a revolute axis, even if the interaction fails, as long as the end effector moves, we can determine the axis direction from the end effector's trajectory. Specifically, if there is only one failed interaction, we select the start position p 1 s , end position p 1 e , and midpoint p 1 m of the trajectory, all belonging to R 3 , forming two directional vectors---→ p 1 s p 1 m and ---→ p 1 m p 1 e .The cross product of these two vectors yields the axis direction:- → A = ---→ p 1 s p 1 m × ---→ p 1 m p 1 e .If there are multiple n failed interactions with movement, we use the end effector's movement vectors from these interactions to perform cross products, providing a more accurate axis direction:- → A = --→ p 1 s p 1 e × ... --→ p n s p n e .After extracting feedback information from the failed experiences, the trained model should reflect and correct itself. Since failures may stem from either invalid position or direction, we first identify the cause before making corrections. Specifically, a failure interaction can result in either movement or no movement. In cases of the former, it indicates that the contact point is on a movable part and can cause movement. Therefore, we consider this a valid contact point and focus on rotation correction. For the latter cause, we utilize the normal direction to interact with the object, which usually results in slight movements if the position is movable. If even the normal direction fails, we conclude that position correction is necessary.Position Correction After determining position needs to be corrected, we employ chain-of-thought [23] to guide the model to understand feedback, i.e., visual mask prompts, and generate new contact point to prevent from predicting on unmovable part. As shown in Fig 3 , the first two steps are consistent with the training phase steps. In the third step, we prompt the model to determine whether the previously predicted points are on unmovable parts, compelling it to reflect on the reason behind the previous failures. Next, in the fourth step, we request the model to output a new SE(3) pose based on the reflection. Finally, in the fifth step, we ask the model to evaluate whether the newly predicted pose will result in a successful interaction, reinforcing its ability to reflect on the outcome.If rotation correction is necessary, as shown in Fig. 3 , we adopt the same direction correction process as in the training phase. We directly perform direction correction using the error information obtained from FIE with text prompts guidance. During this process, the contact point remains fixed and the model will output a new gripper direction.To enable continuous evolution of the model and allow it to rapidly adapt to the current configuration, we propose the Test Time Adaptation (TTA) strategy. During TTA, the model updates itself after each sample inference to learn from the sample it has just processed. The updated model then infers on the next sample, updates again, and this process of model updating continues. Specifically, we use model π t-1 at timestamp t -1 to inference on test sample at timestamp t. The model can only utilize the attempts made on the sample at timestamp t to update itself to π t , thus enhancing its performance when dealing with subsequent samples under the same testing configuration.The input-ground truth pairs used to update the model are as follows: To start with, since we can obtain the ground truth for the first two steps in Fig. 2 -whether there is a mask and where the mask is, we update the model with these position-related VQA pairs. This enables the model to recognize which parts of the object are unmovable, thereby avoiding predictions on those parts. In addition, we also want the model to improve its manipulation ability by learning from successful correction experiences. Therefore, for poses that result in successful manipulation after correction, we use the fourth step of Fig. 2 to update the model, with the corrected pose serving as the ground truth.We use cross-entropy as the supervision signal to update the model. Additionally, to prevent the model from forgetting the knowledge acquired during the training phase, we gradually decrease the learning rate as the number of test samples increases. 
experiment setting: 
 Implementation Details. We use SAPIEN [27] and the PartNet-Mobility [28] dataset to set up the experiment environment. We employ a Franka Panda on-the-fly suction gripper to execute the end-effector actions. We randomly sample about 12K successful manipulation samples across 20 categories to build dataset D o . After that, we augment D o by supplementing each example with the necessary training information, such as mask, joint direction, joint type, etc., thus obtaining D aug .For testing, we sample about 1K successful manipulation samples across 30 categories to ensure the objects can be manipulated. Follow the work before [26] , we only evaluate the pulling action primitives. We finetuned LLaMA-Adapter [29] on an 80G A800 GPU for 10 epochs. a difference of more than 0.01 units between the initial and final object poses or 0.5 relative to the total motion range of the articulated part, and we also require the dot product of the predicted gripper direction and the actual movement direction of the object to be greater than 0.3.Baseline & Ablation Setting. 7) Ours: using four different types of data to train the MLLM to understand the prompt and applying AIC MLLM framework. For the settings from ( 4) to (7) , we perform four corrections.AIC MLLM can correct itself based on erroneous experiences, thereby achieving improved performance. Tab. 1 summarizes all the experimental results. From the results, it is evident that our model shows significant improvement over the base model in both train and test categories. Moreover, As the number of corrections increases, the success rate progressively improves (Fig. 4 ). This indicates that our model can correct itself based on failed experiences and continuously gather information and reflect on it to make further corrections as these experiences accumulate.AIC MLLM is more generalizable. Our model achieved a 0.75 success rate on unseen test categories, significantly outperforming the best baseline (Tab. 1). This demonstrates the strong generalization capability of our framework, indicating that the information extracted pertains to the universal characteristics of articulated objects.Training the model to understand the prompt is important. In Tab. 1, by comparing Ours-w/o pretrain and Ours-w/o TTA, we can see that training the model to understand interface prompts is crucial. This allows our model to improve from 56% to 80% on the train categories and from 44% to 76% on the test categories. TTA is useful for AIC MLLM to adapt to the test domain. From the table 1, we can observe that comparing ours-w/o TTA with ours, the addition of the TTA module resulted in higher or unchanged success rates in 6 unseen test categories and 15 seen train categories. This indicates that incorporating TTA allows the model to adapt to a broader range of object categories. A slight amount of forgetting occurred in a few categories, which may be due to the small number of samples in those categories. However, the minimal difference in average success rates between the two shows that our TTA strategy effectively mitigates forgetting.After validating the effectiveness of our method in simulation, we conduct experiments in the real world. The detailed experimental setup and results can be found in Appendix F. 
conclusion: 
 We introduce AIC MLLM, a framework leveraging MLLM to correct SE(3) position predictions through learning from low-level interaction failures. We design visual and textual prompts for guiding position and rotation corrections, along with the integration of a feedback information extraction module to adaptively correct pose predictions based on identified failure causes. We implement the test-time adaptation module to improve manipulation stability.Comprehensive experiments showcase the effectiveness of AIC MLLM across simulated and real-world environments. 
limitation & future work: 
 In this work, we primarily focus on low-level contact pose correction for cross-category articulated object manipulation. However, several potential limitations can be considered and addressed in future work. Firstly, while our current approach focuses on pose-level corrections, existing frameworks like REFLECT [6] handle only high-level corrections. Future work could integrate these frameworks to achieve long-term corrections at both the pose and high levels. Secondly, our task involves correcting action primitive poses rather than operating in a closed-loop manner. Leveraging more temporal data and designing alternative prompts could offer promising solutions for improvement. Additionally, our framework utilizes the off-the-shelf foundation model VIP-LLaVA and SAM, which is not specifically designed for object manipulation property reasoning. Finetuning this model on object-centric manipulation data could enhance the overall performance of our framework. 
 
 Appendix: In this section, we will provide a detailed introduction to how we perform joint type classification.After the end effector (EE) reaches the predicted pose and adheres to the object, it will move along the predicted gripper direction for a certain distance. We will record the pose trajectory of EE for 20 frames during this process. We denote the i-th pose in the trajectory as W P E i . We perform the operation of subtracting the previous pose from the current pose for all poses in the sequence which can be denoted as -→ v = W P E i -W P E i-1 . We gather the -→ v to form a vector list. Then we calculate the angle between adjacent vectors. Next, we set a threshold. If all the angles are less than this threshold, then the joint will be considered a prismatic joint; otherwise, it will be considered a revolute joint.Our model architecture follows LLaMA-Adapter [29] . Given an RGB imageI ∈ R H×W ×3 , we employ pre-trained CLIP [32] as the visual encoder to extract visual features. Text prompts P are transformed into text features using the tokenizer from the pre-trained 7B LLaMA [49] . After aligning the visual and text feature representations via a multi-modal projection module consisting of 32 transformer blocks, LLaMA acts as the decoder to output appropriate answers. The model adopts a set of learnable adaptation prompts (LoRa [50] ) and inserts these prompts into the multimodal alignment module and LLaMA for quick finetuning to other tasks. Since we used the LLaMA Adapter for pretraining, we adopt the same network structure. This is because its alignment layer, already trained on a large amount of real-world data, can perfectly align the feature of visual encoder CLIP with LLaMA, providing strong generalization for downstream tasks such as robotics. During finetuning, we use the pretrained model of LLaMA-Adapter, and only finetune the injected LoRa to adapt to the tasks presented in Figure 2 . This approach ensures that the model retains most of its general reasoning abilities for real-world applications while gaining additional robotics-related correction capabilities.In this section, we will include additional implementation details that were not mentioned in the main paper. During data collection for the dataset D o , we randomly sample the contact points and the gripper directions, then determine if the interaction is successful. If it is, we save the data; otherwise, we delete it. We sample both pulling and pushing action primitives. During the training phase, for the VQA task 'Mask Position Reasoning,' we mentioned that we randomly select N points.In implementation, N is set to 20. During testing, it is important to note that all MLLM temperatures are set to 0. In the tta training process, we initialize the learning rate (lr) to 5e-8 and set the weight decay to 2e-3. Every 300 iterations, we reduce the lr by 70%. The FIE acquires environmental information, including joint type, joint axis and the normal direction of previously predicted contact point. The MLLM uses this information, along with the contact point, to infer whether to correct its response by suggesting a new gripper direction.You're trying to pull an object part, the joint's type is REVOLUTE, the joint axis direction is [-22, -1, 44], Your contact point is (79, 170), the norm direction of the point is [-47, 9, 12] . Please tell me your gripper direction.AIC-MLLM: The gripper direction is [-47, 13, -13] .You're trying to pull an object part, the joint's type is REVOLUTE, the joint axis direction is [10, 6, -48] , Your contact point is (170, 159), the norm direction of the point is [-29, 0, 40] .Please tell me your gripper direction.AIC-MLLM: The gripper direction is [-28, 35, -20 ].We present some examples of corrections from our simulation experiments. First, for position correction in Figure 5 , we used red dots to represent the points predicted by the model and showed the effects before and after correction. Then, we demonstrated the model's ground truth movable mask. Next is the rotation correction shown in Figure 6 , where we used a coordinate system to represent the direction of the end effector. The rightmost image in Figure 1 shows how this coordinate system is defined, with the red axis indicating the direction of the end effector's pulling motion. The first object is a display from PartNet-Mobility that requires rotating the screen, a revolute joint case. The second and third objects are doors that need to be opened, also involving revolute joints. To validate our end-to-end method beyond simulations, we conducted real-world experiments using a Franka Emika robotic arm equipped with an Intel RealSense D415 sensor without finetuning in real-world.Experimental Setup. The setup included a Franka Emika Panda robotic arm, known for its precision and versatility. An Intel RealSense D415 sensor was used to capture RGB-D data, providing 3D information necessary for the robotic arm's operations. We closed the traditional gripper and applied double-sided tape to its head as a suction gripper. We tested 8 different articulated objects. Specifically, we chose 5 revolute objects: Toilet, Trashcan, Microwave, Cabinet and Faucet, and we chose 3 prismatic objects: Pumpkin-like Mug, Pot and Drawer. Each object undergoes five trials with random camera distances and view angles. Each trial allows up to five attempts. If the method succeeds at least once within these five attempts, it is considered successful; otherwise, it is deemed a failure. For this statistical experiment, we visualize the model's predictions and control the robot accordingly. And we do not perform TTA module. For each object, we provided a detailed description of each task involving that object. For the toilet, the robot needs to lift the toilet lid. For the trashcan, it needs to open the trashcan. When working with the microwave, the robot is required to open the microwave door. For the pumpkin-like mug, the task involves removing its lid. For the cabinet, the robot needs to open its door, and for the pot, it must lift the lid. Additionally, the robot is required to adjust the direction of the faucet and pull the drawer.Initialization. The robotic arm and the D415 sensor were calibrated to ensure accurate spatial data capture and precise movements.Image Processing. The D415 sensor captures high-resolution RGB-D images, which are processed to meet the requirements of our pipeline. Specifically, our method necessitates images sized at 336x336 pixels. To preserve information during resizing, we apply both cropping and padding techniques. This involves cropping or adding white borders around the captured images to resize them to 336x336 pixels. The resulting image, whether cropped or padded, is then used for further analysis in our end-to-end pipeline.Task Execution. Using our end-to-end method, the robotic arm was directly controlled to perform tasks based on the RGB-D data input. The tasks involved the arm autonomously approaching and manipulating objects within the workspace.Demonstration. The robotic arm demonstrated its ability to perform these tasks seamlessly, showcasing the practical applicability and robustness of our end-to-end method in handling real-world scenarios. In the video, the robot was tasked with correctly pulling the cabinet door open. As demonstrated, the robot failed on its first attempt. However, after applying our framework for correction, it successfully opened the cabinet door.Experimental Result. Table 3 presents the real-world performance results. Our framework demonstrates an improvement in the average success rate from 37.5% to 65%, reflecting a trend similar to that observed in the simulation experiments. This consistency indicates that our framework effectively transfers from simulation to real-world applications.  


Decision: Update: The paper presents a novel method that improves spatial understanding of foundation models for correcting robot policy executions. The authors responded well in the rebuttals to reviewer feedback. I am recommending an accept for this submission.

Original: The reviewers agree that the problem being studied by this paper is important and timely. However, they are concerned aabout the generalizabiliy of the proposed method and the paper's findings, as well as the lack of clarity on how the method can be best integrated for downstream applications. I ask the authors to carefully address all of the reviewers' concerns during the rebuttal period.

Reviews: review: title: title: Autonomous Interactive Correction MLLM for Robust Robotic Manipulation: Recommend to accept with significant revision
paper_summary: paper_summary: With the stated goal of enabling robots to learn from mistakes, the authors propose a method they call “Autonomous Interactive Correction” (AIC), which takes as inputs image and end-effector positions from a set of failed object-interactions, and leverages a specially fine-tuned multi-modal foundation model to extracts relevant information about the failures to enable a better pose prediction for the next attempt.
main_review: main_review: The topic of self-improvement is very important to robot learning, and the method presented here is novel and interesting. The claims of the paper could be more clearly articulated and supported. Revision could strengthen the paper by better clarifying the scope of what the authors have (and haven’t!) investigated, articulating limitations of the current work and next steps to identify the limits of the method (assuming the method has not been pushed to its limits yet), and discussing experiments, including hardware experiments, with technical/scientific rigor. More specifically:

*Unclear task scope*. Regarding scope of tasks, the manuscript doesn’t have a clear message and seems to conflict with itself.
* The paper opens with emphasis on generalization. 
* Method includes prismatic and revolute joints, claiming “For articulated objects, there are mainly two types of joints: prismatic and revolute.” 
* Image in figure2 is a pot, where the main interaction I see would be to remove the lid–neither prismatic nor revolute. 
* In evaluation, the manuscript states, “we only evaluate the pulling action primitives”. 

A consistent story would include (a) what task categories would be covered in a hypothetical ideal case, (b) what is covered by this work, and (c) what motivated this choice of coverage.

*Missing limitations and next steps*. The method seems to have promise, but it’s not clear from the manuscript how well the authors have actually developed and tested it so far. It’s also not clear how, if at all, it has been integrated into a policy. A discussion of limitations and next steps would help the reader understand what the authors have actually investigated, and where the authors see potential from their work so far.

*Integration*: I don’t see an example of how the method is integrated into a policy, e.g. that will behave one way in success cases, and then in case of failure gracefully leverage the presented method to improve performance. However, perhaps the authors have only built the method to distill a set of failed experiences into a useful improvement, but not yet integrated it into a policy? Does the method as demonstrated so far require a human-selected series of failed episodes, or has the method been tested with a robot in a “self-play” mode where, starting from the first episode, a policy captures and categorizes successes and failures and uses them according to the method? *How is the intended task communicated to the robot in the first place? Or is the goal of the method only to discover possible affordances of articulated objects?*

*Masking*: The method trains on a red mask added to objects. How does this impact the model’s performance working with (identifying, segmenting, analyzing) objects that are already red?

**Clarification nits:**

~Line 200: “In the third step, we prompt the model to determine whether the previously predicted points are on unmovable parts, compelling it to reflect on the reason behind the previous failures…” What fraction of failures are actually due to incorrect points–are there other common failure modes?  If so, how does this method respond in such cases?

Line 159-161: not sure I understand what the authors are trying to communicate–consider a rewrite with clarity in mind.


*End-effector pose labels in figures*: Fig4: clarify the end-effector pose labels in annotation? The rightmost image shows left-dir pointing to the right, up-dir pointing into or away from the page, and I don’t see for-dir annotated in any of the images. 

This notation is used again in the Appendix fig2, and there it is not clear what the intended success would look like for each task (e.g. the first object looks like a TV–not sure how it’s supposed to articulate; last object looks like a suitcase on wheels–is the goal to extend a handle, or push/pull the suitcase in a particular direction?), or how to map the “end effector pose” color labels to progress. The labels appear to set a coordinate space, but not to set a direction of intended movement, however the paper claims to improve the movement trajectory, not just the identification of a coordinate space.

*Polish*: Please check for spelling mistakes, there are at least a few typos per page, including some incorrect words.

*Originality*: I’m between “Good” and “Very Good”; better understanding the scope of the claims could clarify.
questions: questions: Method sounds interesting/promising, but some claims are not clearly articulated or supported.

1) I have trouble understanding the extent of the claims. Clearly articulating scope, stating what is *and isn’t* included in the contributions, would add clarity. E.g.

a) How deployable is the method now? Has it been tested in isolation, with failed episodes selected by a human and provided to the method? Has it been tested integrated into a policy, where the model will behave one way in success cases, and then in case of failure gracefully leverage the presented method to improve performance?

b) What tasks have the authors considered? Tested? Which tasks are out of scope here?

c) Statistical analysis would add credibility to experimental results.

2) Critically missing: Experiments–almost no information is provided. There is reference to a video, which contains an image of a moving robot. Need clarity on what experiments have been run, with statistical analysis.

3) Critically missing: limitations, next steps.
limitations: limitations: 1
review_summary: review_summary: I recommend accepting the paper with significant revision. The topic of self-improvement is very important to robot learning, and the method presented here is novel and interesting. Revision could strengthen the paper by better clarifying the scope of what the authors have (and haven’t!) investigated, articulating limitations of the current work and next steps to identify the limits of the method (assuming the method has not been pushed to its limits yet), and discussing experiments, including hardware experiments, with technical/scientific rigor.
title: title: Autonomous Interactive Correction MLLM for Robust Robotic Manipulation
paper_summary: paper_summary: The authors propose fine tuning an off-the-shelf multimodal language model to make use of low-level interaction experiences and correct pose predictions. The model can be prompted by either a visual mask highlighting parts for position correction or language instructions describing the desired correction. The training curriculum is carefully designed to shape the models understanding of the desired prompts. Additionally, the authors propose a test time adaption strategy that updates the model from online experience.  Experiments are performed in simulation with SAPIEN and PartNet-Mobility. Finally, the failure correction approach is demonstrated in the real world using a Franka robot arm to open a cupboard door.
main_review: main_review: Strengths:
- The paper is well written and easy to follow. + the figures are clear and helpful.
- The method of correcting failures by prompting a model with masks and language is clever
- The training curriculum designed for progressively shaping the MLLM’s understanding of the desired prompts is interesting
- Experimental results indicate that this is a promising direction
- The proposed approach is demonstrated on a real-world robot

 Weaknesses:
- The approach seems heavily tailored toward certain types of failures (grasping immovable parts of articulated objects). I would have liked to see more discussion about the limitations and how the framework might be extended for other types of failures.
- While the training curriculum is well described, there are very little details or discussion about the underlying model being trained.
- There is very little details or discussion about the real-world robot experiments. It’s not clear what tasks were demonstrated in the real-world and the supplementary video only shows one task (opening a cupboard door) and only shows one failed execution / recovery. I would have liked to see much more details about the tasks performed and results of the real-world experiments because I am left with questions about how well the promising performance from the simulated benchmarks transfers to the real-world.
- The feedback information extraction is limited by the performance of some off-the-shelf models (e.g. VIP-LLaVA and SAM)

Minor suggestions:
- Typo on line 103: Postional -> Positional
- On line 149 it’s not clear to me what ‘process’ in this sentence is supposed to mean: “We introduce the process of our framework during the testing phase.”
- On line 258, missing space between Results &
questions: questions: - While the training curriculum is clearly described, the actual model and architecture are not immediately clear to me. At the beginning of your method section you start by saying “we enable the model to predict poses”, and throughout the method section you very frequently reference the model, but as far as I can tell you do not define anything about the model. In the experiment phase you do specify fine tuning LLaMA-Adapter, but I feel like the architecture of the model could be better described in the context of your method.
- In section 3.3.1 you describe your method of extracting joint type and axis based on interaction attempts… e.g. if the motion trajectory is an arc it is classified as revolute. But how are the initial trajectories generated? From what is described in the overview (section 3.1) the model is predicting poses, and per the description of the testing phase (3.3) the model is outputting a 2D contact point (lifted into 3D by the depth map) and rotation. But it’s not clear to me how is the post contact motion trajectory generated? 
    - It might be worth directly describing the problem formulation up front to make it really clear the domain, action space, etc that your framework is operating in.
- Does the fifth step of position correction help? You ask the model to evaluate whether the newly predicted pose will result in a successful interaction — but is that evaluation used for anything? Did you perform experiments without that step?
- I appreciate the included details about LLaMA-Adapter fine tuning. I am curious how some of the design choices were made. E.g. decision of visual encoder, number of transformer layers for alignment bridge, etc. Did you experiment with different options or did you have some a priori evidence that informed these decisions?
- What tasks were performed in the real-world with the Franka robot? How does performance in the real-world compare to the performance you report on the simulated benchmarks?
- How might the framework be extended for other types of failures and tasks beyond grasping and pulling parts of articulated objects?
limitations: limitations: 1
review_summary: review_summary: I feel that the paper is missing some details that I expect to be addressed during the rebuttal period. But the overall approach seems interesting and worth sharing with the robotics learning community.
title: title: Weak Reject (Concerns about Scope, Limited Hardware Experiments)
paper_summary: paper_summary: This work proposes a method for failure detection and recovery using the few-shot reasoning capabilities of foundation models. The proposed system is evaluated on simulated benchmarks along with a real-world demonstration on the Franka.
main_review: main_review: Strengths: 
- The paper is well-written, straightforward, and the provided figures and diagrams are great. The provided supplementary video is also quite helpful for understanding the framework.
- It is good to see some initial demonstrations on a real-world setting
- The related work is pretty thorough and baselines are well-motivated. I appreciate that the authors compared against several ablations of their method to understand the different design choices.

Weaknesses:
- Where is the limitations section?
- The proposed idea of iteratively masking out regions in which a manipulation action failed, before re-prompting a VLM to identify new interaction point candidates seems like it is a bit overkill. I think the main challenge with detecting failures/successes in manipulation is not really bottlenecked so much by perception/object identification -- recent works like RoboPoint (which should be cited btw), along with several other works that use VLMs for planning have shown that with the right combination of open-vocabulary object detectors can generally localize the kinds of semantic parts this paper looks at
- For example, in the real-world demonstration, why is the initial point of the cabinet handle so poorly predicted (it is on the joint)? This seems like current SOTA open-vocabulary object detectors should really be able to localize this with a lot more accuracy from the beginning -- is this because LLaMa just really struggles with keypoint prediction?
- Much of the prompting seems very task-specific, and relying privileged information like knowing whether a cabinet is revolute/prismatic, etc. It is unclear how much prompt tuning will be required to use this approach on other tasks with their own specifics. 
- Still, I think that this approach is reasonable, but the bigger issue is that it seems to view failure recovery as just re-predicting a new pose for the robot to go to. I think this may apply to some simple grasping/cabinet opening related tasks, but is generally not applicable to any kind of more precise/fine-grained task that requires closed loop control. Asking a VLM to reason about 6DoF actions is also a really tough ask, and I wonder if a more principled approach could be more useful (i.e. detecting grasp candidates either analytically or from a grasp generator like 6DoF grasp or AnyGrasp or ContactGraspNet, etc. and having a VLM choose from these seems like a better posed problem).
questions: questions: - Have you tried other methods of keypoint detection beyond just prompting LLaMa? Can you ablate a few in the rebuttal (maybe RoboPoint, GroundedSAM / RAM, etc.)? 
- How do you see this approach scaling to more complex tasks where failures are not at the pose-level, but may be either even more precise low-level failures, or higher-level task-planning failures?
- How sensitive is this approach to the prompts? Did you have to do a lot of prompt engineering, and is it possible to do more ablations on how much this matters? 
- In practice, how will you retrieve information like the joint type of an articulated object? This seems like a big assumption, and so it would be useful to either show how this works on a different task, or show how you can sense these things possibly from vision, etc.
limitations: limitations: 1
review_summary: review_summary: I appreciate the authors efforts to present their approach clearly and to demonstrate it on both simulated and real scenarios; the main concern is the amount of task-specific assumptions in this approach, and how it will really scale to a variety of real-world tasks.