AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation

## Abstract

Abstract Robotic manipulation in open-world settings requires not only task execution but also the ability to detect and learn from failures. While recent advances in vision-language models (VLMs) and large language models (LLMs) have improved robots’ spatial reasoning and problem-solving abilities, they still struggle with failure recognition, limiting their real-world applicability. We introduce Aha , an open-source VLM designed to detect and reason about failures in robotic manipulation using natural language. By framing failure detection as a free-form reasoning task, Aha identifies failures and provides detailed, adaptable explanations across different robots, tasks, and environments. We fine-tuned Aha using FailGen , a scalable framework that generates the first large-scale dataset of robotic failure trajectories, the Aha dataset. FailGen achieves this by procedurally perturbing successful demonstrations from simulation. Despite being trained solely on the Aha dataset, Aha generalizes effectively to real-world failure datasets, robotic systems, and unseen tasks. It surpasses the second-best model (GPT-4o in-context learning) by 10.3% and exceeds the average performance of six compared models—including five state-of-the-art VLMs—by 35.3% across multiple metrics and datasets. We integrate Aha into three manipulation frameworks that utilize LLMs/VLMs for reinforcement learning, task and motion planning, and zero-shot trajectory generation. Aha ’s failure feedback enhances these policies’ performances by refining dense reward functions, optimizing task planning, and improving sub-task verification, boosting task success rates by an average of 21.4% across all three tasks compared to GPT-4 models.

## 1 Introduction

In recent years, foundation models have made remarkable progress across various domains, demonstrating their ability to handle open-world tasks[, , , ]. These models, including large language models (LLMs) and vision-language models (VLMs), have shown proficiency in interpreting and executing human language instructions[], producing accurate predictions and achieving strong task performance. However, despite these advancements, key challenges remain—particularly with hallucinations, where models generate responses that deviate from truth. Unlike humans, who can intuitively detect and adjust for such errors, these models often lack the mechanisms for recognizing their own mistakes[, , ].

Learning from failure is a fundamental aspect of human intelligence. Whether it’s a child learning to skate or perfecting a swing, the ability reason over failures is essential for improvement[, , ]. The concept of improvement through failures is widely applied in training foundation models and is exemplified by techniques such as Reinforcement Learning with Human Feedback (RLHF)[, ], where human oversight and feedback steers models toward desired outcomes. This feedback loop plays a critical role in aligning generative models with real-world objectives. However, a crucial question persists: How can we enable these models to autonomously detect and reason about their own failures, particularly in robotics, where interactions and environments are stochastic and unpredictable?

This need is particularly pressing in robotics, where foundation models such as VLMs and LLMs are increasingly used to address open-world tasks. Recent advancements have enabled these models to tackle spatial reasoning, object recognition, and multimodal problem-solving—skills vital for robotic manipulation[, , , , ]. VLMs and LLMs are already being integrated to automate reward generation for reinforcement learning[, ], develop task plans for motion planning[], and even generate zero-shot robot trajectories[, , , ]. While these models excel at task execution, they often face challenges in detecting and reasoning over failures—skills that are crucial for navigating dynamic and complex environments. For example, if a robot drops an object mid-task, a human observer would immediately recognize the error and take corrective action. How can we empower robots with similar capabilities, allowing them not only to perform tasks but also to detect and learn from their mistakes?

To learn from their mistakes, robots must first detect and understand why they failed. We introduce Aha, an open-source vision-language model (VLM) that uses natural language to detect and reason about failures in robotic manipulation. Unlike prior work that treats failure reasoning as a binary detection problem, we frame it as a free-form reasoning task, offering deeper insights into failure mode reasoning. Our model not only identifies failures but also generates detailed explanations. This approach enables Aha to adapt to various robots, camera viewpoints, tasks, and environments in both simulated and real-world scenarios. It can also be integrated into downstream robotic applications leveraging VLMs and LLMs. We make the following three major contributions:

1. We introduce FailGen, a data generation pipeline for the procedural generation of failure demonstration data for robotic manipulation tasks across simulators.
To instruction-tune Aha, we developed FailGen, the first automated data generation pipeline that procedurally creates the Aha dataset—a large-scale collection of robotic manipulation failures with over 49K+ image-query pairs across 79 diverse simulated tasks. Despite being fine-tuned only on the Aha dataset, Aha demonstrates strong generalization to real-world failure datasets, different robotic systems, and unseen tasks, as evaluated on three separate datasets not included in the fine-tuning. FailGen is also flexible data generation pipeline integrates seamlessly with various simulators, enabling scalable procedural generation of failure demonstrations.

2. We demonstrate that Aha excels in failure reasoning, generalizing across different embodiments, unseen environments, and novel tasks, outperforming both open-source and proprietary VLMs.
Upon fine-tuning Aha, we benchmarked it against six state-of-the-art VLMs, both open-source and proprietary, evaluating performance across four metrics on three diverse evaluation datasets, each featuring different embodiments, tasks, and environments out-of-distribution from the training data. Aha outperformed GPT-4o model by more than 20.0% on average across datasets and metrics, and by over 43.0% compared to LLaVA-v1.5-13B [], the base model from which Aha is derived. This demonstrates Aha’s exceptional ability to detect and reason about failures in robotic manipulation across embodiment and domains.

3. We show that Aha enhances downstream robotic applications by providing failure reasoning feedback. We demonstrate that Aha can be seamlessly integrated into robotic applications that utilize VLMs and LLMs. By providing failure feedback, Aha improves reward functions through Eureka reflection, enhances task and motion planning, and verifies sub-task success in zero-shot robotic manipulation. Across three downstream tasks, our approach achieved an average success rate 21.4% higher than GPT-4 models, highlighting Aha’s effectiveness in delivering accurate natural language failure feedback to improve task performance through error correction.

Figure: Figure 1: Aha is a Vision-Language Model designed to detect and reason about failures in robotic manipulation. As an instruction-tuned VLM, it can enhance task performance in robotic applications that utilize VLMs for reward generation, task planning, or sub-task verification. By incorporating Aha into the reasoning pipeline, these applications can achieve accelerated and improved performance.
Refer to caption: extracted/5889089/image/fig1.png

## 2 Related Work

Aha enables language reasoning for failure detection in robotic manipulation, enhancing downstream robotics applications. To provide context, we review progress in: 1) failure detection in robotic manipulation, 2) data generation in robotics, and 3) foundation models for robotic manipulation.

Failure Detection in Robotic Manipulation. Failure detection and reasoning have long been studied in the Human-Robot Interaction (HRI) community [, ] and in works leveraging Task and Motion Planning (TAMP) []. With the recent widespread adoption of LLMs and VLMs in robot manipulation systems—either for generating reward functions or synthesizing robot trajectories [, ] in a zero-shot manner—the importance of detecting task failures has regained prominence [, , , ]. Most modern approaches focus on using off-the-shelf VLMs or LLMs as success detectors [, , , ], and some employ instruction-tuning of VLMs to detect failures []. However, these methods are often limited to binary success detection and does not provide language explanations for why failures occur. Our framework introduces failure reasoning in a new formulation, generating language-based explanations of failures to aid robotics systems that leverage VLMs and LLMs in downstream tasks.

Data Generation in Robotics
There have been many methods in robotic manipulation that automate data generation of task demonstrations at scale [, ], whether for training behavior cloning policies, instruction-tuning VLMs [], or curating benchmarks for evaluating robotic policies in simulation [, ]. A well-known example is MimicGen [], which automates task demonstration generation via trajectory adaptation by leveraging known object poses. Additionally, works like RoboPoint use simulation to generate general-purpose representations for robotic applications, specifically for fine-tuning VLMs. Similarly, systems like The Colosseum [] automate data generation for curating benchmarks in robotic manipulation. Our approach aligns closely with RoboPoint, as we also leverage simulation to generate data for instruction-tuning VLMs. However, unlike RoboPoint, we focus on synthesizing robotic actions in simulation rather than generating representations like bounding boxes or points.

Foundation Models for Robotic Manipulation.
In recent years, leveraging foundation models for robotic manipulation has gained significant attention due to the effectiveness of LLMs/VLMs in interpreting open-world semantics and their ability to generalize across tasks [, , , ]. Two main approaches have emerged: the first uses VLMs and LLMs in a promptable manner, where visual prompts guide low-level action generation based on visual inputs [, , ]. The second focuses on instruction-tuning VLMs for domain-specific tasks []. For example, RoboPoint [] is tuned for spatial affordance prediction, and Octopi [] for physical reasoning using tactile images. These models generalize beyond their training data and integrate seamlessly into manipulation pipelines. Our approach follows this second path, developing a scalable method for generating instruction-tuning data in simulation and fine-tuning VLMs specialized in detecting and reasoning about robotic manipulation failures, with applications that extend beyond manipulation tasks to other robotic domains.

## 3 The Aha Dataset

We leveraged FailGen to procedurally generate the Aha dataset from RLBench tasks [] and used it for the instruction-tuning of Aha. In this section, we begin by categorizing common failure modes in robotics manipulation and defining a taxonomy of failures in Section 3.1. Next, we explain how this taxonomy is used with FailGen to automate the data generation for the Aha dataset in simulation in Section 3.2.

**Table 1: Aha datasets for instruction-tuning. We combined the Aha dataset, our large-scale robotic manipulation failure dataset, with VQA and object detection data. By incorporating this diverse data mix into the fine-tuning process, Aha is able to reason about failures in robotic manipulation across different domains, embodiments, and tasks.**
| Source | The Aha dataset (Train) | VQA [] | LVIS [] |
| --- | --- | --- | --- |
|  |  |  |  |
| Quantity | 49K | 665K | 100K |
| Query | For the given sub-tasks, first determine it has succeed by choosing from ["yes", "no"] and then explain the reason why the current sub-tasks has failed. | What is the cat doing in the image? | Find all instances of drawer. |
| Answer | No, The robot gripper rotated with an incorrect roll angle | The cat is sticking its head into a vase or container, possibly drinking water or investigating the interior of the item. | [(0.41, 0.68, 0.03, 0.05), (0.42, 0.73, 0.04, 0.08), …] |

### 3.1 Failure Modes in Robotic Manipulation

To curate an instruction-tuning dataset of failure trajectories for robotic manipulation tasks, we began by systematically identifying prevalent failure modes. Our approach involved a review of existing datasets, including DROID [] and Open-X Embodiment [], as well as an analysis of policy rollouts from behavior cloning models. We examined failures occurring in both teleoperated and autonomous policies. Building upon prior works, such as REFLECT [], we formalized a taxonomy encompassing seven distinct failure modes commonly observed in robotic manipulation: incomplete grasp, inadequate grip retention, misaligned keyframe, incorrect rotation, missing rotation, wrong action sequence, and wrong target object.

Incomplete Grasp (No_Grasp) Failure: No_Grasp is an object-centric failure that occurs when the gripper reaches the desired grasp pose but fails to close before proceeding to the next keyframe.

Inadequate Grip Retention (Slip) Failure: Slip is an object-centric failure that happens after the object has been successfully grasped. As the gripper moves the object to the next task-specific keyframe, the grip loosens, causing the object to slip from the gripper.

Misaligned keyframe (Translation) Failure: This action-centric failure occurs when the gripper moves toward a task keyframe, but a translation offset along the X, Y, or Z axis causes the task to fail.

Incorrect Rotation (Rotation) Failure: Rotation is an action-centric failure that occurs when the gripper reaches the desired translation pose for the sub-task keyframe, but there is an offset in roll, yaw, or pitch, leading to task failure.

Missing Rotation (No_Rotation) Failure: No_Rotation is an action-centric failure that happens when the gripper reaches the desired translation pose but fails to achieve the necessary rotation (roll, yaw, or pitch) for the sub-task, resulting in task failure.

Wrong Action Sequence (Wrong_action) Failure: Wrong_action is an action-centric failure that occurs when the robot executes actions out of order, performing an action keyframe before the correct one. For example, in the task put_cube_in_drawer, the robot moves the cube toward the drawer before opening it, leading to task failure.

Wrong Target Object (Wrong_object) Failure: Wrong_object is an object-centric failure that occurs when the robot acts on the wrong target object, not matching the language instruction. For example, in the task pick_the_red_cup, the gripper picks up the green cup, causing failure.

Figure: Figure 2: Overview of Aha Pipeline. (Top) The data generation for Aha is accomplished by taking a normal task trajectory in simulation and procedurally perturbing all keyframes using our taxonomy of failure modes. Through FailGen, we systematically alter keyframes to synthesize failure demonstrations conditioned on the original tasks. Simultaneously, we generate corresponding query and answer prompts for each task and failure mode, which are used for instruction-tuning. (Bottom) The instruction-tuning pipeline follows the same fine-tuning procedure as LLaVA-v1.5 [], where we fine-tune only the LLM base model—in this case, LLaMA-2-13B and the projection linear layers, while freezing the image encoder and tokenizer.
Refer to caption: extracted/5889089/image/method_2.png

### 3.2 Implementation of the Aha dataset

The Aha dataset is generated with RLBench [], utilizing its keyframe-based formulation to dynamically induce failure modes during task execution. RLBench natively provides keyframes for task demonstrations, which enables flexibility in both object manipulation (handling tasks with varying objects) and the sequence of actions (altering the execution order of keyframes). Building on this foundation, we leverage FailGen, our custom environment wrapper to wrap around RLBench that allows for task-specific trajectory modifications through keyframes perturbations, object substitutions, and reordering of keyframe sequences. FailGen systematically generates failure trajectories aligned with the taxonomy defined in Section 3.1, yielding a curated dataset of 49k failure-question pairs.

To generate the Aha dataset, we systematically sweep through all keyframes in each RLBench task, considering all potential configurations of the seven failure modes that could result in overall task failure. By leveraging the success condition checker in the simulation, we procedurally generate YAML-based configuration files by sweeping through each failure mode across all keyframes. These files provide details on potential failure modes, parameters (such as distance, task sequence, gripper retention strength, etc.), and corresponding keyframes that FailGen should perturb to induce failure. Additionally, we incorporate language templates to describe what the robot is doing between consecutive keyframes. Using these descriptions along with the failure modes, we can systematically curate question-answer pairs for each corresponding failure mode.

For specific failure modes, No_Grasp is implemented by omitting gripper open/close commands at the relevant keyframes, effectively disabling gripper control. Slip introduces a timed release of the gripper shortly after activation. Translation and Rotation perturb the position and orientation of a keyframe, respectively, while No_Rotation constrains the keyframe’s rotational axis. Wrong_Action reorders keyframe activations to simulate incorrect sequencing, and Wrong_Object reassigns the keyframes intended for one object to another, maintaining the relative pose to mimic improper object manipulation. Using this pipeline, we also successfully generated a failure dataset from ManiSkill [] and adapted RoboFail [] for the evaluation of Aha. This further demonstrates the generalizability and versatility of FailGen in generating failure cases across different simulation environments.

## 4 Method

This section outlines the failure reasoning problem formulation (Sec.4.1) used to fine-tune and evaluate Aha. Next, we discuss the curated data mix used for co-finetuning Aha (Sec.4.2). Finally, we detail the instruction fine-tuning pipeline and the model architecture selection for Aha (Sec.4.3).

### 4.1 Failure Reasoning Formulation

Unlike previous works [, , ] that primarily focus on detecting task success as binary classification problem, we approach failure reasoning by first predicting a binary success condition ("Yes" or "No") of the given sub-task based on a language specification and an input image prompt. If the answer is "No", the VLM is expected to generate a concise, free-form natural language explanation detailing why the task is perceived as a failure. To formulate failure reasoning, we prompt the VLMs to analyze the trajectory failures at the current sub-task and provide reasoning for *why* or *what* led to the failure. We define manipulation task trajectories as a series of sub-tasks $\{S_{0},S_{1},S_{2},\dots,S_{t}\}$, where each sub-task is represented by two consecutive keyframes. For example, in a task like "stacking cubes", a sub-task could represent a primitive action, such as ’picking up the cube’.
For the input formulation in VLMs for instruction fine-tuning and evaluation, we required a query prompt with an input image for prompting the VLMs. The query prompt was generated using a template corresponding to the current sub-task the robot is performing. To capture the temporal relationships within the action sequence, the input image was constructed by selecting a single frame that represents the robot’s trajectory up to the current sub-task and concatenating it with frames from other viewpoints in the rollout sequence, as shown in Table 3.

This input frame is built by concatenating all keyframes up to the current sub-task in temporal order, from left to right, with any remaining keyframes replaced by white image patches. To mitigate occlusions, we also included all the available camera viewpoints, concatenating them alongside the temporal sequence, and provide a detailed task description in the prompt, as illustrated in Table 3 (left image). The image data is structured as a matrix $\mathbf{I}$, where each row corresponds to a different camera viewpoint $\{V_{0},V_{1},\dots,V_{n}\}$ and each column captures the temporal sequence of keyframes $\{S_{0},S_{1},S_{2},\dots,S_{t}\}$.
This formulation for curating images serves as a general approach for formatting all datasets used for fine-tuning and evaluation. This structured input enables consistent handling of data across different tasks and viewpoints.
Overall, our failure reasoning problem is to prompt VLM with sub-task discription and keyframe trajectory image to predict the success condition and language description of failure reason for each sub-task, as shown in Table 3.

### 4.2 Synthetic Data for Instruction-tuning

To facilitate the instruction-tuning of Aha, we needed to systematically generate failure demonstration data. To achieve this, we developed FailGen, an environment wrapper that can be easily applied to any robot manipulation simulator. FailGen systematically perturbs successful robot trajectories for manipulation tasks, transforming them into failure trajectories with various modes of failure as depicted in Figure 2 (Top image). Using FailGen, we curated the Aha dataset (Train) dataset by alternating across 79 different tasks in the RLBench simulator, resulting in 49k failure image-text pairs. Furthermore, following proper instruction-tuning protocols for VLMs [] and building on prior works [, ], co-finetuning is crucial to the success of instruction fine-tuning of VLMs. Therefore, in addition to the Aha dataset, we co-finetuned Aha with general visual question-answering (VQA) datasets sourced from internet data, which helps models retain pre-trained knowledge. Specifically, we included the VQA dataset [], containing 665k conversation pairs, and the LVIS dataset [], which comprises 100k instances with predicted bounding box centers and dimensions, as summarized in Table 3.

### 4.3 Instruction Fine-tuning

We followed the instruction-tuning pipeline outlined by []. As depicted in Fig. 2, our model architecture includes an image encoder, a linear projector, a language tokenizer, and a transformer-based language model. The image encoder processes images into tokens, projected by a 2-layer linear projector into the same space as the language tokens. These multimodal tokens are then concatenated and passed through the language transformer. All components are initialized with pre-trained weights. During fine-tuning, only the projector and transformer weights are updated, while the vision encoder and tokenizer remain frozen. The model operates autoregressively, predicting response tokens and a special token marking the boundary between instruction and response.

## 5 Experimental Results

In this section, we evaluate Aha’s detection and reasoning performance against six state-of-the-art VLMs, including both open-source and proprietary models, some utilizing in-context learning. The evaluation spans three diverse datasets, covering out-of-domain tasks, various simulation environments, and cross-embodiment scenarios. We then assess Aha’s ability to retain general world knowledge after fine-tuning on domain-specific data. Finally, we explore its potential to improve downstream robotic manipulation tasks.

**Table 2: Quantitative Evaluation on Failure Detection and Reasoning. Aha-13B was evaluated and benchmarked against three open and three proprietary VLMs and one visual prompting baseline across three evaluation datasets. Aha-13B outperformed all other VLMs on every evaluation dataset and nearly every evaluation metric, with the exception of the Aha (Test) dataset, where GPT-4o exceeded by less than 3%.**
| Models | Evaluation Datasets | Evaluation Metrics |  |  |  |
| --- | --- | --- | --- | --- | --- |
| $\text{ROUGE}_{L}$ $\uparrow$ | Cosine Similarity $\uparrow$ | Binary Success(%) $\uparrow$ | LLM Fuzzy Match $\uparrow$ |  |  |
| LLaVA-v1.5-13B [] | Aha dataset (Test set) | 0.061 | 0.208 | 0.080 | 0.648 |
| ManiSkill-Fail | 0.000 | 0.208 | 0.022 | 0.270 |  |
| RoboFail [] | 0.000 | 0.203 | 0.000 | 0.404 |  |
| LLaVA-NeXT-34B [] | Aha dataset (Test set) | 0.013 | 0.231 | 0.017 | 0.626 |
| ManiSkill-Fail | 0.001 | 0.195 | 0.007 | 0.277 |  |
| RoboFail [] | 0.018 | 0.188 | 0.017 | 0.351 |  |
| Qwen-VL [] | Aha dataset (Test set) | 0.000 | 0.161 | 0.000 | 0.426 |
| ManiSkill-Fail | 0.037 | 0.301 | 0.116 | 0.034 |  |
| RoboFail [] | 0.000 | 0.159 | 0.000 | 0.050 |  |
| Gemini-1.5 Flash [] | Aha dataset (Test set) | 0.120 | 0.231 | 0.371 | 0.566 |
| ManiSkill-Fail | 0.003 | 0.121 | 0.014 | 0.032 |  |
| RoboFail [] | 0.000 | 0.042 | 0.000 | 0.393 |  |
| GPT-4o | Aha dataset (Test set) | 0.251 | 0.308 | 0.500 | 0.784 |
| ManiSkill-Fail | 0.142 | 0.335 | 0.688 | 0.453 |  |
| RoboFail [] | 0.114 | 0.318 | 0.554 | 0.438 |  |
| GPT-4o-ICL (5-shot) | Aha dataset (Test set) | 0.226 | 0.380 | 0.611 | 0.776 |
| ManiSkill-Fail | 0.341 | 0.429 | 0.971 | 0.630 |  |
| RoboFail [] | 0.236 | 0.429 | 0.571 | 0.418 |  |
| AHA-7B | Aha dataset (Test set) | 0.434 | 0.574 | 0.691 | 0.695 |
| ManiSkill-Fail | 0.609 | 0.680 | 1.000 | 0.532 |  |
| RoboFail [] | 0.204 | 0.394 | 0.625 | 0.439 |  |
| AHA-13B (Ours) | Aha dataset (Test set) | 0.446 | 0.583 | 0.702 | 0.768 |
| ManiSkill-Fail | 0.600 | 0.681 | 1.000 | 0.633 |  |
| RoboFail [] | 0.280 | 0.471 | 0.643 | 0.465 |  |

**Table 3: Quantitative Evaluation on Standard VQA Benchmarks. Aha-13B performs on par with LLaVA-13B [], the VLM from which Aha adapts its fine-tuning strategy.**
|  | MMBench [] | ScienceQA [] | TextVQA [] | POPE [] | VizWiz[] |
| --- | --- | --- | --- | --- | --- |
| LLaVA-13B (LLama-2) [] | 67.70 | 73.21 | 67.40 | 88.00 | 53.01 |
| AHA-13B (LLama-2) | 65.20 | 71.94 | 65.20 | 85.74 | 53.45 |

### 5.1 Experimental Setup

To quantitatively evaluate Aha’s detection and reasoning capabilities for failures in robotic manipulation, we curated two datasets and adapted an existing failure dataset for benchmarking. To ensure a fair comparison of free-form language reasoning, we also employed four different evaluation metrics to measure semantic similarity between sentences.

Benchmarks. We curated three datasets to evaluate Aha’s reasoning and failure detection capabilities, benchmarking against other state-of-the-art VLMs. The first dataset, Aha dataset (Test), includes 11k image-question pairs from 10 RLBench tasks, generated similarly to the fine-tuning data via FailGen (Section 3.2) but without overlapping with the tasks from the finetuning dataset. It evaluates Aha’s ability to generalize to novel, out-of-domain tasks. The second dataset, ManiSkill-Fail, comprises 130 image-question pairs across four tasks in ManiSkill [], generated using Failgen wrapper on Maniskill simulator. This dataset assesses Aha’s performance in a different simulator and under changing viewpoints. Lastly, we adapted a failure benchmark from the RoboFail dataset [], which features real-world robot failures in seven UR5 robot tasks. This allows for evaluation across simulation and real-world trajectories and across different embodiments.

Figure: Figure 3: (Left) Scaling law with the Aha dataset. Scaling of effect of model performance with varying domain specific fine-tuning data. (Right) Downstream Robotic Application Performance. Aha-13B outperforms GPT-4o in reasoning about failures within these robotic applications, leading to improved performance of the downstream tasks.
Refer to caption: extracted/5889089/image/scale3.png

Evaluation Metrics. To fairly evaluate success detection and free language reasoning across all datasets and baselines, we employ four metrics. First, the ROUGE-L score measures the quality of generated text by focusing on the longest common subsequence between candidate and reference texts. Second, we use Cosine Similarity to assess similarity between texts or embeddings, avoiding the "curse of dimensionality". Third, LLM Fuzzy Matching utilizes an external language model—specifically, Anthropic’s unseen model, claude-3-sonnet—to evaluate semantic similarity in a teacher-student prompting format. Lastly, we calculate a Binary success rate by comparing the model’s predictions directly against the ground truth for success detection.

### 5.2 Quantitative Experimental Results

We contextualize the performance of Aha by conducting a systematic evaluation of failure reasoning and detection across these three datasets, general VQA datasets, and performed ablation studies.

Aha generalizes across embodiments, unseen environments, and novel tasks. To ensure fairness and eliminate bias in the detection and reasoning capabilities of Aha, we evaluated it on three different datasets that were never seen during fine-tuning, each designed to test a specific form of generalization. First, on the Aha dataset (test) dataset, Aha demonstrated its ability to generalize reasoning across tasks and new behaviors within the same domain, outperforming the second-best performing VLM, GPT-4o ICL, by an average margin of 12.6% difference across all evaluation metrics. Second, we assessed Aha-13B on a dataset generated by the Failgen wrapper in a different simulation domain, ManiSkill, showing that our model outperforms GPT-4o-ICL by an average of 13.4% difference across all metrics as depicted in Table 5. Lastly, to demonstrate generalization to real-world robots and different embodiments, we evaluated Aha-13B on RoboFail  [], where it outperforms GPT-4o-ICL by 4.9% difference.

Aha retains common sense knowledge. We evaluated Aha-13B’s performance on various VQA benchmarks and present the results in Table 3
. Aha-13B performs comparably to LLaVA-v1.5-13B (LLama-2) [] , with only a 1.5% margin difference as depicted in Table 3. Notably, LLaVA-v1.5-13B is a VLM trained on the same pre-trained weights as Aha-13B but fine-tuned on VQA data. This indicates that Aha-13B is capable of functioning as a general purpose VLM, in addition to excelling at failure reasoning.

Aha’s performance scales with data size.
We evaluated Aha’s performance using a range of Aha data for instruction fine-tuning, spanning [3k, 6k, 12k, 34k, 48k, 60k], and co-trained individual checkpoints corresponding to these data sizes as shown in Figure 3 (Left). The model was then assessed on the ManiSkill-Fail dataset across four evaluation metrics. An average quadratic fit gradient of 0.0022 across all four metrics demonstrates a scaling effect with fine-tuning on our procedurally generated data pipeline. This suggests that further scaling of the generated data may lead to improved model performance.

### 5.3 Downstream Robotics Tasks

Figure: Figure 4: Downstream Robotic Application. We demonstrated that Aha can be integrated into existing LLM/VLM-assisted robotic applications to provide failure reasoning and feedback, helping to accelerate and improve task success rates in these systems.
Refer to caption: extracted/5889089/image/dowstrask.png

We demonstrate that Aha’s failure detection and reasoning capabilities are useful across a wide spectrum of downstream robotics applications. This includes automatic reward generation for reinforcement learning applications [], automatic task plan generation for task and motion planning applications [], and as an improved verification step for automatic data generation systems []. Videos and detailed improved reward function, task plan, example videos from each applications and etc can be found on the project page: [aha-vlm.github.io](https://aha-vlm.github.io/).

Aha enables efficient reward synthesis for reinforcement learning. To evaluate this downstream task, we adapted Eureka’s [] implementation to the ManiSkill simulator, which offers more state-based manipulation tasks. We strictly followed the Eureka reward function generation and reflection pipeline, modifying it by incorporating perception failure feedback via either Aha-13B or GPT-4o (acting as a baseline) to enhance the original LLM reflection mechanism. Instead of only including a textual summary of reward quality based on policy training statistics for automated reward editing, we further incorporated explanations of policy failures based on evaluation rollouts. We evaluated our approach on five reinforcement learning tasks from ManiSkill, ranging from tabletop to mobile manipulation. To systematically assess the reasoning capabilities of different VLMs under budget constraints, we sampled one reward function initially and allowed for iterations over two sessions of GPT API calls. Each policy was trained using PPO over task-specific training steps and evaluated across 1,000 test steps. During policy rollouts, we employed either Aha-13B or GPT-4o for reward reflection to improve the reward function. Comparing the evaluated policy success rates using different failure feedback VLMs, we observed that Aha-13B provided intuitive, human-level failure reasoning
that aided in modifying and improving generated dense reward functions. This resulted in success across all five tasks within the budget constraints, and our approach outperformed GPT4o by a significant margin of 22.34% in task success rate shown in Figure 3 (Right).

Aha refines task-plan generation for TAMP.
To demonstrate Aha’s utility within a planning system, we incorporated our approach into PRoC3S [].
The PRoC3S system solves tasks specified in natural language by prompting an LLM for a Language-Model Program (LMP) that generates plans, and then testing a large number of these plans within a simulator before executing valid plans on a robot.
If no valid plan can be found within a certain number of samples (100 in our experiments), the LLM is re-prompted for a new LMP given failure information provided by the environment.
Importantly, as is typical of TAMP methods, the original approach checks for a finite set of failures (inverse kinematics, collisions, etc.) from the environment, and returns any sampled plan that does not fail in any of these ways.
We incorporated a VLM into this pipeline in two ways: (1) we prompt the VLM with visualizations of failed plan executions within the simulator, ask it to return an explanation for the failure, and feed this back to PRoC3S’ LLM during the LMP feedback stage, (2) after PRoC3S returns a valid plan, we provide a visualization of this to the VLM and ask it to return whether this plan truly achieves the natural language goal, with replanning triggered if not.
We compared GPT-4o and Aha-13B as the VLM-based failure reasoning modules within this implementation of PRoC3S across three tasks (shown in Figure 4). Each task was evaluated over 10 trials, with a maximum of 100 sampling steps and three feedback cycles provided by either GPT-4o or Aha-13B. The success rate for each task was recorded. As shown in Figure Figure 3 (Right), utilizing Aha-13B for failure reasoning significantly improved the task success rate and outperforming GPT-4o by a substantial margin of 36.7%.

Aha improves task verification for zero-shot robot data generation. To demonstrate Aha’s utility in zero-shot robot demonstration generation, we integrated our approach into the Manipulate-Anything framework. This open-ended system employs various Vision-Language Models (VLMs) to generate diverse robot trajectories and perform a wide range of manipulation tasks without being constrained by predefined actions or scenarios. A critical component of Manipulate-Anything is its sub-task verification module, which analyzes past and current frames to decide whether a sub-task has been achieved before proceeding or re-iterating over the previous sub-task. We replaced the original VLM (GPT-4V) in the sub-task verification module with Aha-13B and evaluated performance across four RLBench tasks (Figure 4), conducting 25 episodes for each task. Our results show that substituting the sub-task verification module’s VLM with Aha improved reasoning accuracy and overall task success by an average of 5%.

## 6 Conclusion

Limitations.
Aha currently outputs language reasoning that is closely aligned with the failure scenarios in the fine-tuning data. However, there is an opportunity to output more open-ended failures, to cover those arising from modes outside of the ones included in the failure taxonomy.
Additionally, while FailGen systematically curates failure data from simulations, distilling large pretrained policies to perform diverse tasks in simulation and sampling failure modes would allow us to generate more open-ended failure examples, potentially enhancing the instruction-tuned performance of Aha.

Conclusion. We introduce Aha, an open-source vision-language model that significantly enhances robots’ ability to detect and reason about manipulation task failures using natural language. By framing failure detection as a free-form reasoning task, Aha not only identifies failures but also provides detailed explanations adaptable to various robots, tasks, and environments. Leveraging FailGen and the curated Aha dataset, we trained Aha on a diverse set of robotic failure trajectories. Our evaluations show that Aha outperforms existing models across multiple metrics and datasets. When integrated into manipulation frameworks, its natural language feedback greatly improves error recovery and policy performance compared to GPT-4 models. These results demonstrate Aha’s effectiveness in enhancing task performance through accurate error detection and correction.

## 7 Acknowledgement

Jiafei Duan is supported by the Agency for Science, Technology and Research (A*STAR) National Science Fellowship. Wilbert Pumacay is supported by grant 234-2015-FONDECYT from Cienciactiva of the National Council for Science, Technology and Technological Innovation (CONCYTEC-PERU).

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Side questions (major ones in the Weaknesses section above)

- In subsection 3.1., why separate the rotation failure into incorrect and missing ones? Should these two belong to the same category?  

- An ablation study on how many camera viewpoints would be insightful for the real-robot applicability of the proposed method.
soundness: 3
strengths: - The paper tackles an important problem that is highly beneficial and safety-critical when learning-based robotic algorithms are deployed in the real world;

- The developed procedural pipeline of simulation data generation is useful for researchers in the community who are willing to contribute to this direction;

- It is inspiring to see that not only improved performance is shown on the standalone task, but to see the strong benefits brought by such improvement on different related and popular downstream tasks.

- Though failure detection is an important prerequisite for the reasoning part, I am still glad to see the efforts in investigating the failure reasoning and especially its usage in the downstream tasks.

- The presentation is generally well-structured and easy to follow.

- Though benchmarking experiments are conducted in the simulation, their results are comprehensive and informative.
summary: This work targets the task of open-world failure detection and reasoning under the scenario of robotic manipulation, which is critical and valuable for diverse downstream tasks. Specifically, the authors propose a procedural pipeline for generating failure demonstration data for robotic manipulation in simulation, upon which they fine-tune a vision-language model (VLM) for failure detection and reasoning. Moreover, they demonstrate improved performance brought by the fine-tuned VLM in three downstream tasks, namely reward function design, task-plan refinement, and sub-task verification. Comprehensive comparison with the other six VLMs also shows the superior performance of the proposed technique on this task.
weaknesses: Presentation:
- It would be clearer if subsection 4.1. can be shortened a bit and described with a more concise mathematical formulation. 

- To reinforce the contribution, I would suggest adding a subsection in the method section to explain how improved failure detection and reasoning performance can help these downstream tasks. In the current draft, putting this part in the experiment section is less apparent and inspiring for potential readers in these fields. 

- Closely related to the point above, it seems that the usage of the proposed idea in downstream task 2 (refine task plans in TAMP) and task 3 (improve task verification) is essentially similar, i.e., detect and reason for the failure of the sub-tasks. It might be more lucid for the readers to group these two tasks together along with the reward design task and make a taxonomy for different high-level manners of how failure detection and reasoning can help. 

- in sub-section 4.1., "Unlike previous works that primarily focus on detecting task success as binary classification problem, we approach failure reasoning by first predicting a binary success condition ("Yes" or "No") of the given sub-task based on a language specification and an input image prompt." seems less straightforward to understand. Logically, I suspect the author wants to express that they have an additional reasoning part after the detection step. 


- Fig.1 and Fig.2 are hard to read. It would be better to reorganize the sub-figures and increase the sizes of the images and texts.

Methodology:

- The first point I would like to raise is the lack of discussion of hallucination in LLM/VLMs. In the introduction, the authors motivate the failure recognition with the shortcoming of hallucinations in LLMs/VLMs. However, the proposed fine-tuned VLM will still suffer from this issue. It would be important to discuss how to mitigate this issue in detecting failures. In this regard, there is a missing citation [1] that is closely related to this work, which made an initial attempt in this direction based on the uncertainty derived from the VLM. I would suggest the authors add a discussion on this aspect. It would also be meaningful to see the influence of fine-tuning with the proposed data set on the performance of uncertainty estimation. Will there be a trade-off between the performance of the prediction accuracy and uncertainty estimation?  

- The second point is related to the design of the technique. In [1], it has been observed that using the "chain-of-thoughts" (CoT) analysis in the prompt template can improve the performance of failure detection. Therefore, related to the reasoning part of this work, will the reasoning part be able to help improve the performance failure detection? Will the performance be influenced by the different prompting templates, e.g., with and w/o CoT?

Experiments:

- Due to missing real-world experiments, therefore, it is hard to evaluate the performance loss during sim-to-real transfer and the actual practicality of the proposed technique. 
 

[1] "Zheng, Zhi, et al. "Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners." arXiv preprint arXiv:2406.00430 (2024)."

---

confidence: 3
contribution: 4
initial_rating: 5
presentation: 3
questions: See the weaknesses part. The idea is interesting and I would like to raise my score if the authors can show some rigorous results to clarify the points above, especially 1 and 2. I also encourage discussion on negative results to help readers see the limitation and potential of the current version.
soundness: 2
strengths: 1. The idea of using failure detection VLM to improve VLM-based robotic application is interesting.
2. The effort in developing a failure dataset is valuable towards building robust large models that can understand the physical world.
summary: This paper introduces AHA, a new VLM model to identify and describe failure modes in robot manipulation. The authors introduce a pipeline called FailGen to generate failure demonstrations and use the data to train AHA. In the experiments, AHA demonstrates improved performance in failure reasoning. The authors also show its effectiveness in downstream application including VLM-based reward design, planning, and subtask verification.
weaknesses: 1. [Soundness] It remains unclear to me whether incorporating failure descriptions is a reliable approach to improve VLM in reward design, planning, and subtask verification. For example, in the VLM-based reward design, although VLM can provide some penalty reward terms based on AHA input to correct behavior, there is no guarantee the new reward can make robot perform better than the previous one (e.g. the penalty can be too large and hinder skill learning). In this case, I would like to see if the proposed method can show some iterative reward tuning behavior based on failure feedback and the procedure can converge (i.e., tuning the strength of new terms or number of terms to correct behavior). 

2. [Clarity] The paper can be much clearer and convincing if the authors can show more extensive comparisons between AHA and GPT4 to understand what leads to the improvements. For example, is the improvement due to GPT4 not being able to generate accurate failure description? Or, is it simply because the generated failure text is too lengthy for the downstream VLM model to understand? I encourage more explicit discussion on what a good failure description looks like (for downstream task VLMs). I also suggest comparing to human oracles with different writing styles.

3. [Result: Generalization] The paper is focusing on simple pick-and-place style tasks in clean backgrounds. How about more challenging and dexterous tasks? I would like to see the results on more challenging task such as guiding a robot hand to perform grasping and reorientations where the failure modes can be more fine-grained and complex. Besides, explicitly discussing generalization to diverse backgrounds / objects is also necessary. The results on these harder setups can help readers to understand the opportunities and challenges of AHA in application.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1.What is the difference between rotation and no rotation?
2.What are the details of your baseline settings? I couldn't find them in your paper.
3.Are you using the REFLECT framework for your baseline? If not, I believe REFLECT is highly relevant to your work, so I would like to see a comparison of its performance with your model.
4.How do you plan to extend your failure types? As of now, I still find them somewhat rough.

My main concern is the sufficiency of the experiments and the work still seems somewhat preliminary. However, if you address my concerns, I will be inclined to raise my score.
soundness: 2
strengths: 1. The motivation is meaningful, and I agree that addressing failures in robotics is important.
2. It can be utilized in downstream tasks.
3. The website has many great demos.
summary: This work proposes a VLM for detecting and reasoning about failures in robotic manipulation, which can be applied in various downstream tasks.
weaknesses: I think the types of failure modes and the reasoning responses are somewhat rough. And more details and experiments need to be conducted. For further details, please refer to the "Questions" section.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - View Angle Concerns: The camera view angles during training are fixed according to the RLBench settings. This raises concerns about the view angles during testing. Specifically: 1) How many views does the model require during testing? 2) Should the view angles during testing match the exact poses used in the RLBench training setting?

- Missing Reference to Figure 1: Figure 1 is not referenced or discussed anywhere in the paper, which needs to be addressed for clarity and completeness.
soundness: 3
strengths: Failure trajectory generation can be extended to other scenarios (simulatores), encompassing a comprehensive set of failure categories. Failure reasoning plays a critical role in downstream policy learning, which is essential for advancing the entire community
summary: The paper presents a Vision-Language Model (VLM) for robotic failure reasoning, introducing a failure dataset and fine-tuning the VLM to achieve this goal. The approach demonstrates promising performance.
weaknesses: Baseline Comparisons:
 - Clarity of Baseline Implementation: The implementation details of the baselines are unclear. What type of input data are the baselines using? Is it a single image, a sequence of images, images from multiple views, or video? This distinction significantly affects baseline performance. For instance, Gemini excels at reasoning for robotic manipulation when using video inputs, but its performance would likely degrade if multiple-view images were used instead.

 - Lack of Comparison with Robotic-Specific Methods: Why not compare your approach with other robotic-oriented reasoning methods mentioned in the 'Introduction' or 'Related Work' sections? The current experiments only compare against general Vision-Language Models (VLMs), which are not specifically designed for robotic tasks.

I would consider raising my score if these issues are addressed or clarified. Otherwise, it's difficult to properly assess how well the model performs.

Meta-review: The paper introduces AHA, a vision-language model for detecting and reasoning about failures in robotic manipulation tasks. The method demonstrates promising performance in analyzing manipulation failures and improving downstream robotic applications, receiving generally positive reviews across all reviewers (scores 5-8). Reviewers consistently praised the paper's meaningful motivation of addressing failure reasoning in robotics, its novel contribution of a failure dataset generation framework, and the demonstrated utility in multiple downstream applications including reward design, planning, and verification tasks.

Initial concerns centered on three main areas: the lack of rigorous evaluation and comparison with robotics-specific baselines like REFLECT, unclear implementation details about input modalities and baseline configurations, and questions about generalization beyond simple pick-and-place scenarios. The authors provided detailed responses, implementing new comparisons with REFLECT that showed AHA's superior performance, clarifying their standardized input protocol across baselines, and conducting additional ablation studies on viewpoint variations. They also demonstrated comparable performance to human oracle evaluations in some cases.

While two reviewers were satisfied with these responses and maintained positive scores, one reviewer remained concerned about the fundamental limitations of using failure descriptions for improving downstream tasks, noting that even with human-generated descriptions the success rates remained below 80%. However, the authors clarified that AHA serves to enhance existing frameworks rather than replace them entirely, showing consistent improvements over baselines across different applications.

Decision: Accept (Poster)