AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
Authors: Jiafei Duan
Wilbert Pumacay
Nishanth Kumar
Yi Ru Wang
Shulin Tian
Wentao Yuan
Ranjay Krishna
Dieter Fox
Ajay Mandlekar
Yijie Guo
Venue: ICLR 2025 Poster (2025)
Keywords: Robotic Manipulation; Data Generation; Vision-Language-Model; Failure Reasoning; Failure Detection
Primary area: applications to robotics, autonomy, planning
TL;DR: A vision-language model for detecting and reasoning about failures in robotic manipulation, which can be used to improve many downstream robotic applications.

Abstract: Robotic manipulation in open-world settings requires not only task execution but also the ability to detect and learn from failures. While recent advances in vision-language models (VLMs) and large language models (LLMs) have improved robots' spatial reasoning and problem-solving abilities, they still struggle with failure recognition, limiting their real-world applicability. We introduce AHA, an open-source VLM designed to detect and reason about failures in robotic manipulation using natural language. By framing failure detection as a free-form reasoning task, AHA identifies failures and provides detailed, adaptable explanations across different robots, tasks, and environments. We fine-tuned AHA using FailGen, a scalable framework that generates the first large-scale dataset of robotic failure trajectories, the AHA dataset. FailGen achieves this by procedurally perturbing successful demonstrations from simulation. Despite being trained solely on the AHA dataset, AHA generalizes effectively to real-world failure datasets, robotic systems, and unseen tasks. It surpasses the second-best model (GPT-4o in-context learning) by 10.3% and exceeds the average performance of six compared models including five state-of-the-art VLMs by 35.3% across multiple metrics and datasets. We integrate AHA into three manipulation frameworks that utilize LLMs/VLMs for reinforcement learning, task and motion planning, and zero-shot trajectory generation. AHA’s failure feedback enhances these policies' performances by refining dense reward functions, optimizing task planning, and improving sub-task verification, boosting task success rates by an average of 21.4% across all three tasks compared to GPT-4 models. Project page: https://aha-vlm.github.io

Published as a conference paper at ICLR 2025
АНA: A VISION-LANGUAGE-MODDE FOR DETECT- ING AND REASONING OVER FAILURES IN ROBOTIC MA-
NIPULATION
Jiafei Duan Wilbert Pumacay
Nishanth Kumar
Yi Ru Wang 1.2
Shulin Tian Ajay Mandlekar" Yijie Guo
Wentao Yuan 1,
Ranjay Krishna Dieter Fox 12
IVIIDI, University of Washington, Universidad Católica San Pablo, "MIT
Nanyang Technological University, Allen Institute for Artificial Intelligence
ABSTRACT
Robotic manipulation in open-world settings requires not only task execution but also the ability to detect and learn from failures. While recent advances in vision- language models (VLMs) and large language models (LLMs) have improved robots' spatial reasoning and problem-solving abilities, they still struggle with failure recognition, limiting their real-world applicability. We introduce AHA, an open- source VLM designed to detect and reason about failures in robotic manipulation using natural language. By framing failure detection as free-form reasoning task, AlIA identifies failures and provides detailed, adaptable explanations across different robots, tasks, and environments. We fine-tuned AнA using F Faill faiGee Gen,, scalable framework that generates the first large-scale dataset of robotic failure trajectories, the AHA dataset. Fai Gen achieves this by procedurally perturbing successful demonstrations from simulation. Despite being trained solely the
AhIA dataset, AIIA generalizes effectively to real-world failure datasets, robotic systems, and unseen tasks. It surpasses the second-best model (GPT-4o in-context learning) by 10.3% and exceeds the average performance of six compared models including five state-of-the-ar VLMs by 35.3% across multiple metries and datasets. We integrate AHA into three manipulation frameworks that utilize LLMs/VLMS for reinforcement learning, task and motion planning, and zero-shot trajectory generation. AHA 's failure feedback enhances these policies' performances by refining dense reward functions, optimizing task planning, and improving sub-task verification, boosting task success rates by average of 2 2.44% across all three tasks compared to GPT-4 models. Project page: chaa-ylm.itthu....
INTRODUCTION
In recent years, foundation models have made remarkable progress across various domains, demon- strating their ability handle open-world tasks (Driess et al., 2023 Alayrac et al.) 2022 Achiam jet al.; 2023 ZZhng et al., 2023). These models, including large language models (LLMs) and vision- laiiual models (VLMs), have shown proficiency interpreting and executing human lan- guage instructions (Ouyang et al., 2022), producing accurate predictions and achieving strong task performance. However, despite these advancements, key challenges remain remain-pparticulary with hal- lucinations, where models generate responses that deviate from truth. Unlike humans, who can intuitively detect and adjust for such errors, these models often lack the mechanisms for recognizing their own mistakes (Lin et al. 2021 Chen al. 2021
Heyman 2008).
Learning from failure a fundamental aspect of human intelligence. Whether it's a child learning to skate or perfecting swing, the ability reason over failures is essential for improvement (Young) 2009, Gopnik| 20200 Heyman| 2008). The concept of improvement through failures is widely applied training loundation models and is exemplified by techniques such as Reinforcement Learning with
"Equal advising

---

Published as a conference paper at ICLR 2025
Sub-task Verification
droced alyn
MA
gnap ano
suppoe groep
Manipulate-Aupthhin
VLM Reward Function Generation
VLM Task-plan Generation
YLM Gouraidd Breard Dunccie actlos: far's Nintes Sult Bo Foward
Intar reard
Falurr delection and reawming
reassiing
raber gripper maved posiies along
grtpper
posiions with direction whre stakking.
Eureha
Task and Motion Planning
Jagrovl
Iegeoved Tamkplass Rovaiiua
Atpet stsiect acouu ather miteer far pam
Figure I: AHA is Vision-Language Model designed to detect and reason about failures in robotic manipulation. As an instruction-tuned VLM, it can enhance task performance in robotic applications that utilize VLMs for reward generation, task planning, or sub-task verification. By incorporating AHA into the reasoning pipeline, these applications can achieve accelerated and improved performance.
Human Feedback (RIHF) (Ouyang al. 2022 Christiano et al.) 2017), where human oversight and feedback steers models toward desired outcomes. This feedback loop plays critical role in aligning generative models with real-world objectives. However, crucial question persists: How can we enable these models to autonomously detect and reason about their own failures, particularly in robotics, where interactions and environments are stochastic and unpredictable?
This need is particularly pressing in robotics, where foundation models such as VLMs and LLMs are increasingly used to address open- open-world world tasks. Recent advancements have enabled these models tackle spatial reasoning, object recognition, and multimodal problem-solving- skills vital for robotic manipulation (Reid et al. 2024
Yuan et al., 20224] Chen et al. 2024 Wang 2023b). VLMs and LLMs are already being integrated automate reward generation for reinforcement learning (Ma et al. 2023 2024), develop task plans for motion planning (Curtis et al. 2024). and even generate zero-shot robot trajectories (Huang et 2023 2024a; Duan al. 2024 Huang et al. 2024b). While these models excel at task execution, they often face challenges in detecting and reasoning over failures-ss skills that are crucial for navigating dynamic and complex environments. For example, if robot drops an object mid-task, a human observer would immediately recognize the error and take corrective action. How can we empower robots with similar capabilities, allowing them not only to perform tasks but also to detect and learn from their mistakes?
To learn from their mistakes, robots must first detect and understand why they failed. We introduce AIIA, an open-source VLM that uses natural language to detect and reason about failures in robotic manipulation. Unlike prior work that treats failure reasoning as a binary detection problem, we frame it free-form reasoning task, offering deeper insights into failure mode reasoning, Our model not only identifies failures but also generates detailed explanations. This approach enables AhA to adapt various robots, camera viewpoints, tasks, and environments in both simulated and real- worldwrldd
2

---

Published as a conference paper at ICLR 2025
Takan Them WIG
Data Generation
Fallr (501s WFF
Falker Mode
Greendanck
keih for deeere the moinas
Task: Troest the square block Into the red
Sharp taregh Sondato
Palice
soreadl and Dascil de
Vierdder Willee Medio
dieeii aloie
Generating Fallure Caulii
Curating Reudering Contig
Laheling
Instruction Tuning
Chappt Impioss
Large Language Model (Llama 2-13B)
Ripoeee pippe.
Image tilees
Larguge
Longage:
Irogr
Image
Projection
Teleetber
40
Staraned
Pfuc mutio
Figure Overview of AнΛ Pipeline. (Top) The data generation for AHA is accomplished by taking normal task trajectory in simulation and procedurally perturbing all keyframes using laxonomy of failure modes. Through Fai Fein we systematically alter keyframes to synthesize failure demonstrations conditioned on the original tasks. Simultaneously, we generate corresponding query and answer prompts for each task and failure mode, which are used for instruction-tunie (Bottom) The instruction-tune truninnin-ntinggggggg pipeline follows the same fine-tuning procedure as LLa Liu ctt al. (202a)), where we fine-tune only the LLM base model-in this case, LLaMA-2-13RB and the projection linear layers, while freezing the image encoder and tokenizer.
scenarios. It can also be integrated into downstream robotic applications leveraging VLMs and LLMs, shown in Figure ! We make the following three major contributions:
We introduce FailGen, data generation pipeline for the procedural generation of failure demonstration data for robotic manipulation tasks across simulators. To instruction-tune AHA, we developed
Gen, the first automated data generation pipeline that procedurally creates the AHA dataset large-scale collection of robotic manipulation failures with over 49K+ image-query pairs across 79 diverse simulated tasks. Despite being fine-tuned only on the Анha dataset, АНАa demonstrates strong generalization to real-world failure datasets, different robotic systems, and unseen tasks, as evaluated on three separate datasets not included the fine-tuning. Fai Gen is also flexible data generation pipeline integrates seamlessly with various simulators, enabling scalable procedural generation of failure demonstrations.
2. We demonstrate that ÁhA excels in failure reasoning, generalizing across different embodi- ments, unseen environments, and novel tasks, outperforming both open-source and proprietary VLMs. Upon fine-tuning АНл,, we benchmarked against six state-of-the-ar VLMs, both open- source and proprietary, evaluating performance across four metrics on three diverse evaluation datasets, each featuring different embodiments, tasks, and environments out-of-distribution from the training data. AHA outperformed GPT-4o model by more than 20.0% on average across datasets and metrics, and by over 43.0 ,0%% compared from which AHA is derived. This demonstrates AHA' 's exceptional ability to detect and reason about failures in robotic manipulation across embodiment and domains.
(Liu al., 2023a), the base model
3. We show that AHA enhances downstream robotic applications by providing failure reasoning feedback. We demonstrate that AlIA can be seamlessly integrated into robotic applications that utilize VLMs and LLMs. By providing failure feedback. AllA improves reward functions through Eureka reflection, enhances task and motion planning. and verifies sub-task success in zero-shot robotic manipulation. Across three downstream tasks, approach achieved average success rate 1.4% higher than GPT-4 models, highlighting AHA'S 's effectiveness delivering accurate natural language failure feedback to improve task performance through error correction.
3

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
AhA enables language reasoning for failure detection in robotic manipulation, enhancing downstream robotics applications. To provide context. review progress in: 1) failure detection in robotic manipulation. 2) data generation in robotics, and 3) foundation models for robotic manipulation.
Failure Detection in Robotic Manipulation. Failure detection and reasoning have long been studied in the Sluman-Robor Interaction (HRI) community (Ye et al.) 2019 Khanna et al. 2023 and in works leveraging Task and Motion Planning (TAMP) (Garett et al. 2020). With the recent widespread adoption of LLMS and VLMS in robot manipulation systems- either for generating reward functions
or synthesizing robot trajectories (Ma al., 2023 2024) in zero-shot manner-the the importance of detecting task failures has regained prominence (Huang et al. [023 Duan et al. [0024] [Sreeta et al.
2024| Ha al. 2023 ?). Most modern approaches focus on using off-the-shelf VLMs or LLMs as
success detectors (Ma et al., 2022 Ha et al., (2023; Wang et al. 2023a Duan et al. 2024 Dai et al.
cerruction-tuuinng of VLMs detect failures (Du et al. [023)) Furthermrt Zheng et al. (2024). However, these methods are often limited to binary success detection and does not provide language explanations for why failures occur. Our framework introduces failure reasoning in new formulation, generating language- baased explanations of failures to aid robotics systems that leverage VL.Ms and LLMs in downstream tasks. Additionally, we investigated whether AHA suffers from hallucinations by analyzing the prediction probabilities of sentence tokens. We found that AHA exhibits fewer hallucinations compared to other VLMs (see supplementary material).
2024), and some employ
hallucinations often occur in LLMs and VLMs. Methods that leverage these models for failure detection can mitigate this issue by detecting uncertainty in VLMs, as demonstrated in this work
Data Generation in Robotics There have been many methods in robotic manipulation that automate data generation of task demonstrations at scale (Mandlekar et al. 2023 Hoque et al. (024), whether
for training behavior cloning policies, instruction-tuning VLMs (Yuan et al.) 2024), or curating benchmarks for evaluating robotic policies in simulation (Xie et al.) 2024 Pumacay et al. (022)). A
well-known example MimicGen (Mandlekar al., 2023). which automates task demonstration generation via trajectory adaptation by leveraging known object poses. Additionally, works like RoboPoint use simulation to generate general purposs representations for robotic applications. specifically for fine-tuning VLMs. Similarly, systems like The Colosseum Pumacay et al. (2024) automate data generation for curating benchmarks in robotic manipulation. Our approach aligns closely with RoboPoint, as we also leverage simulation to generate data for instruction-tun tuning VLMs. However, unlike RoboPoint, we focus on synthesizing robotic actions in simulation rather than generating representations like bounding boxes points.
Foundation Models for Robotic Manipulation. In recent years, leveraging foundation models for robotic manipulation has gained significant attention due to the effectiveness of LLMs//VLMs in interpreting open-world semantics and their ability to generalize across tasks (Duan et al. 2022 Hu al. 2023 Firoozi et al. 2023 Urain et al. 2024 Two main approaches have emerged: the first uses VLMs and LLMs in a promptable manner, where visual prompts guide low-level action generation based on visual inputs (Liu et al. 2024a| Huang et al.) 2024a b). The second focuses on instruction-tuniin
VLMs for domain-specific tasks (Li et al. 2024). For example, RoboPoint (Yuan et al. (024) tuned for spatial affordance prediction, and Octopi (Yu al., (024) for physical reasoning using tactile images. These models generalize beyond their training data and integrate seamlessly into manipulation pipelines. Our approach follows this second path, developing a scalable method for generating instruction-tur -unning data in simulation and fine-tuning VLMs specialized in detecting and reasoning about robotic manipulation failures, with applications that extend beyond manipulation tasks to other robotic domains.
THE AHA DATASET
We leveraged Fa 1Gen to procedurally generate the AllA dataset from RLBench tasks (James) et al. 2020) and used it for the cntrucctiontuning of AHA. In this section, we begin by categorizing common failure modes in robotics manipulation and defining a taxonomy of failures in Section 3.1 Next, we explain how this taxonomy is used with FailGen to automate the data generation for the AHA dataset in simulation in Section 3.2
4

---

Published as a conference paper at ICLR 2025
Table 1: Ahа datasets for instruction-tuningg We combined the AHA dataset, our large-scale robotic manipulation failure dataset, with VQA and object detection data. By incorporating this diverse data mix into the fine-tuning process, AНА is able to reason about failures in robotic manipulation across different domains, embodiments, and tasks.
The (Train)
VQA (Lin
2023a
IVIS
Gupta et
Quantity
49K
665K
100K
Query
For the given suih-tasks, first determine it has succeed by choosing from l'yes", explain reason why the current sub-tasks has failed.
What is the doing the image?
Find all instances of drawer.
"no") ]
Answer
No, The rohot gripper rntated with incorrect rull angle
The is sticking head into or container, possibly drinking water or investigating the interior
[[0.4, 0.68, 0.03, 0.05), (0.42, 0.73, 0.04, 0.08).
3.1 FAILURE MODES IN ROBOTIC MANIPULATION
To curate an instruction-tuning dataset of failure trajectories for robotic manipulation tasks, we began by systematically identifying prevalent failure modes. Our approach involved review of existing datasets, including DROID (Khazatsky et al. 2024) and Open-X Embodiment (Padalkar et al. 2023), as well as analysis of policy rollouts from behavior cloning models. We examined failures occurring in both teleoperated and autonomous policies. Building upon prior works, such as REFLECT (Liu et al. 2023d), we formalized laxonomy encompassing seven distinct failure modes commonly observed in robotic manipulation: incomplete grasp, inadequate grip retention, misaligned keyframe, incorrect rotation, missing rotation, wrong action sequence, and wrong target object.
Incomplete Grasp (No_Grasp) Failure: No_Grasp is an object-centric failure that occurs when the gripper reaches the desired grasp pose but fails close before proceeding to the next keyframe.
Inadequate Grip Retention (Slip) Failure:
Slip is an object-centric failure that happens after the object has been successfully grasped. As the gripper moves the object to the next task-specific keyframe, the grip loosens, causing the object to slip from the gripper.
This action-centric failure occurs when the gripper moves toward task keyframe, but a translation offset along the X, Y, or Z axis causes the task to fail with respect to a fixed reference coordinate system.
Misaligned keyframe (Translation) Failure:
Incorrect Rotation (Rotation) Failure:
Rotation occurs when the gripper reaches the correct position but rotates to an incorrect angle in roll, pitch, or yaw relative to a fixed reference point. Although it attempts the rotation, the misalignment due to rotation results in task failure.
Missing Rotation (No (oo_Rotaiion) Failure: No_Rotati in occurs when the gripper reaches the correct position but fails to perform the necessary rotation in roll, pitch, yaw. The absence of any rotation when it is required leads to misalignment and ultimately causes the task to fail.
Wrong Action Sequence (Wrong action) Failure:
Wrong_act action is an action-centric failure that occurs when the robot executes actions out of order, performing an action keyframe before the correct one. For example, in the task put__ubt cube_ in drawer, the robot moves the cube toward the drawer before opening it, leading to task failure.
Wrong Target Object (Wrong _bjjctt Failure: Wrong_ ob ject is object-centric failure that occurs when the robot acts on the wrong target object, not matching the language instruction. For example, in the task i ck tthe_red_ccu the gripper picks up the green cup, causing failure.
5

---

systematically Published as a conference paper at ICLR 2025
3.2 IMPLEMENTATION OF THE AHA DATASET
The AНA dataset is generated with RLBench James et al. (2020), utilizing its keyframe-based formulation to dynamically induce failure modes during task execution. RLBench natively provides keyframes for task demonstrations, which enables flexibility in object manipulation (handling tasks with varying objects) and the sequence of actions (altering the execution order of keyframes). Building leverage Fai Gen, our custom environment wrapper around RLBench that
on this foundation, allows for task-specific trajectory modifications through keyframes perturbations, object substitutions, and reordering of keyframe sequences. Fa Gen systematically generates failure trajectories aligned with the taxonomy defined in Section 3.1 yielding a curated dataset of 49k failure-question pairs.
To generate the AHA dataset, systematically sweep through all keyframes in each RLBench task, considering all potential configurations of the seven failure modes that could result in overall task failure. By leveraging the success condition checker in the simulation, we procedurally generate YAML-based configuration files by sweeping through each failure mode across all keyframes. These files provide details on potential failure modes, parameters (such as distance, task sequence, gripper retention strength, etc.), and corresponding keyframes that Fai Fail Gen should perturb to induce failure. Additionally, we incorporate language templates to describe what the robot is doing between consecutive keyframes. Using these descriptions along with the failure modes, curate question-anse answer pairs for each corresponding failure mode.
For specific failure modes, No Grasp is implemented by omitting gripper open/close commands at the relevant keyframes, effectively disabling gripper control. Slip introduces a timed re- lease of the gripper shortly after activation. Translation and Rotation perturb the position and orientation of keyframe, respectively, while Rotatioonnooooo. constrains the keyframe's 's rota- tional axis. Wrong_ ion reorders keyframe activations to simulate incorrect sequencing, and grong__bjjcct reassigns the keyframes intended for one object to another, maintaining the relative pose mimic improper object manipulation. Using this pipeline, we also successfully generated a failure dataset from ManiSkill (Tao et al. (024) and adapted RoboFail (Liu et al. (023d) for the evaluation of AHA. This further demonstrates the generalizability and versatility of Fa Gen in generating failure cases across different simulation environments.
4 METHOD
This section outlines the failure reasoning problem formulation (Sec 4.1 used fine-tune and evaluate AHA. Next, we discuss the curated data mix used for co-finetuning AHA (Sec Finally, we detail the instruction fine-tuning pipeline and the model architecture selection for AHIA (See 4.3
4.1 FAILURE REASONING FORMULATION
We extend prior work (Skreta et al. 2024 Duan et al. 2024 by introducing two-step framework for robot failure analysis that combines sub-task success detection and failure reasoning. Sub-task success is evaluated as binary classification problem (Yes/No), while failure reasoning is performed using vision-language models (VLMs) to generate natural language explanations for the causes of failure. This approach allows for both precise failure detection and interpretability in robot manipulation tasks. Manipulation tasks are represented as trajectories consisting of sequence of sub-tasks (S,, Si...., Sr where each sub-task Se is defined by two consecutive keyframes (Ke, Ke+1). Each sub-task corresponds to an atomic manipulation action, such as "grasping a cube" stacking task. For each sub-task, the input the VLM includes query prompt and a structured image representation. The query prompt is generated using template specific to the sub-task and describes the task context and success condition.
The image input is represented as matrix I E B where rows correspond to camera viewpoints { (V,, Vi..., Vn-1) and columns correspond to temporal keyframes (Ko, Ki, Ki.... ia.....,y Kr} To capture the spatiotemporal progression of the task, frames are arranged in temporal order, and missing keyframes are replaced with white patches. We include several camera viewpoints to mitigate occlusions and ensure comprehensive spatial context. This combined representation enables the VLM to reason over the robot's 's trajectory and diagnose failure causes effectively, as demonstrated in Table 1

---

Published as a conference paper at ICLR 2025
4.2 SYNTHETIC DATA FOR INSTRUCTION-TUNISS
To facilitate the instruction-tuning of AHA, we needed to systematically generate failure demonstration data. To achieve this, we developed Fa Gen, an environment wrapper that can be easily applied to any robot manipulation simulator. Fai Gen systematically perturbs successful robot trajectories for manipulation tasks, transforming them into failure trajectories with various modes of failure as depicted in Figure 2 (Top image). Using Fa FailGen, curated the AHA dataset (Train) dataset by alternating across 79 different tasks in the RLBench simulator, resulting in 49k failure image-texi pairs. Furthermore, following proper instruction-tuning protocols for VLMs (Liu al.
2023a) and
building on prior works (Brohan et al. 2023 Yuan et al. 2024), co-finetuning is crucial to the success of instruction fine-tuning of VLMs. Therefore, in addition to the АНA dataset, we co-finetuned (VQA) datasets sourced from internet data, which helps models retain pre-trained knowledge. Specifically, we included the VQA dataset (Liu al. [2023a), containing 665k conversation pairs, and the LVIS dataset (Gupta et al.) 2019), which comprises 1O0k instances with predicted bounding box centers and dimensions, as summarized in Table
AHA with general visual question
4.3 INSTRUCTION FINE-TUNING
We followed the instruction-tuning pipeline outlined by (Liu et al. 2023b). As depicted in Fig. 2 our model architecture includes an image encoder, a linear projector, a language tokenizer, and waanformer-basee language model. The image encoder processes images into tokens, projected by 2-layer linear projector into the same space as the language tokens. These multimodal tokens are then concatenated and passed through the language transformer. All components initialized with pre-trained weights. During fine-tuning, only the projector and transformer weights are updated, while the vision encoder and tokenizer remain frozen. The model operates autoregressively, predicting response tokens and a special token marking the boundary between instruction and response.
4.4 IMPACT ON DOWNSTREAM TASKS
АнА integrates failure reasoning address limitations in downstream robotics methods, improving reward synthesis, decision Decision-making, and feedback efficiency. In reinforcement learning (RL), AHA refines reward synthesis by analyzing rollouts to provide failure explanations, enabling iterative ad- justments to dense reward functions and improving sample efficiency, as demonstrated in approaches such as Eureka (Ma et al. 2023). In task and motion planning (TAMP) systems like PRoC3S (Curtis) al. 2024), AHA enhances feedback loops by interpreting visualizations of failed plans, generating failure explanations, and informing language-modee based plan refinement. This process improves robustness in long-horizon tasks by addressing semantic errors overlooked by finite failure checks.
In open-ended frameworks like Manipulate-Anythhng (Duan et al., 2024), AlIA improves subtask verification by analyzing sequential frames for task progression errors, reducing failure propagation in zero-shot data generation. These integrations enable systematic reasoning improvements across RL, TAMP, and data generation, directly enhancing task success and robustness.
5 EXPERIMENTAL RESULTS
In this section, we evaluate АНa'S detection and reasoning performance against six state-of-the-art VLMs, including both open-source and proprietary models, some utilizing in in-context learning. The evaluation spans three diverse datasets, covering out-of-domain tasks, various simulation environ- ments, and cross-embodiment scenarios. We then assess AHA's 's ability to retain general world knowledge after fine-tuning on domain-specifie data. Finally, we explore its potential to improve downstream robotic manipulation applications.
5.1 EXPERIMENTAL SETUP
To quantitatively evaluate AHA'S 's detection and reasoning capabilities for failures in robotic manipu- lation, curated two failure datasets and adapted an existing failure dataset for benchmarking. To ensure a fair comparison free-form language reasoning, we also employed four different evaluation metrics to measure semantic similarity between sentences.
7

---

Published as a conference paper at ICLR 2025
Table Quantitative Evaluation on Failure Detection and Reasoning. AHA-13B was evaluated and benchmarked against three open and three proprietary VLMs and one visual prompting baseline across three evaluation datasets. AHA-13B outperformed all other VLMs on every evaluation dataset and nearly every evaluation metrie, with the exception of the AIIA (Test) dataset, where GPT-4o exceeded by less than 3%.
AlIa detisert (Tind sed)
ManiSkill-Fal
REFLECT
Models
BOUTEL11
DoeSin
3e5.o5%)
Der Maah1
Coilint
EeSac:5:1
Ruay Marhh
BOUCEL-
Daflecrt11
Fler-aac
0.361
0.208
0.080
0.645
0.030
0.256
0.022
0.271
10,000
0.201
0,000
0.404
0.013
0:231
0.017
0.635
0.011
0.195
0.000
0.277
0.01%
0.388
0.017
0:351
QecariL
0.000
0:425
0.30
0.014
0.000
0.159
0.000
0.251
0.308
0.500
0.784
0112
0.335
0.683
0.453
0.114
0.318
0.554
0.45
GPT (oaen
0.226
0.580
0.611
0.726
0.341
0.429
0.911
0.610
0.236
0.499
0.571
0-18
0:278
01375
07091
02255
80009
01000
11000
05522
02201
0.394
02525
0.081
0.633
Table 3: Quantitative Evaluation on Standard VQA Benchmarks. AНА-13B performs on par with LLaVA-13B Liu et al. (2023a), the VLM from which AНА adapts its fine-tuning strategy.
MMBBench 2013c
ScienceOA 2022
TexVVQA Singh 2019
POPE
VirWiz
1aVA-11R (m--))) er al. 2023a
73.21
$3.01
AHA-13B (CLama-33
65,20
71.94
65
85.74
$3.45
Benchmarks. We curated three datasets to evaluate AHA's reasoning and failure detection capabilities, benchmarking against other state-of-the-art VL.Ms. The first dataset, AHA dataset (Test), includes 11k image-question pairs from 10 RI Bench tasks, generated similarly to the fine-tuning data via FailGen (Section 3.2) but without overlapping with the finetuning dataset. It evaluates AHA'S ability to generalize to novel, out-of-domain tasks. The second dataset, ManiSkill-Fail, comprises 130 image-question pairs across four tasks ManiSkill (Tao al. 2024), generated using Fai gen wrapper on the ManiSkill simulator. This dataset assesses AHA'S 's performance in different simulator and under changing viewpoints. Lastly, we adapted failure benchmark from the RoboFail dataset (Liu et al., 2023d), which features real-world robot failures in seven URS robot tasks, allowing for evaluation across simulation, real-world trajectories, and different embodiments.
STAMP
Eureta
Scaling the Artk datoeet
Figure 3: (Left) Scaling law with the AHA dataset. Scaling of effect of model performance with varying domain specific fine-tuning data. (Right) Downstream Robotic Application Performance. AHA-13B outperforms GPT-4o in reasoning about failures within these robotic applications, leading to improved performance of the downstream tasks.
Evaluation Metrics. To fairly evaluate success detection and language reasoning across all datasets and baselines, we employ four metrics. First, the ROUGE-L score measures the quality of generated text by focusing on the longest common subsequence between candidate and reference texts. Second, we use Cosine Similarity to assess similarity between texts or embeddings, avoiding the "curse of dimensionality". Third, LLM Fuzzy Matching utilizes external language model--secifficall,, Anthropic's 's unseen model, c claude- wowssoonetttte evaluate semantic similarity in teacher- student prompting format (Zhou al. 2023). Lastly. calculate Binary success rate by comparing the model's 's predictions directly against the ground truth for success detection.
5.2 QUANTITATIVE EXPERIMENTAL. RESULTS
We contextualize the performance of AHA by conducting a systematic evaluation of failure reasoning and detection across these three datasets, general VQA datasets, and performed ablation studies.
8

---

Published as a conference paper at ICLR 2025
VLM Reward Generation
Hax the baa
"Ope drowe"
"Pic up the roveae
"Puh the surger
VLM Task-plan Generation
VLM Sub-tasks Verification
tanona
yollin and
"Place bonana
"Brck vellp
"Pac cake"
concabae
Figure 4: Downstream Robotic Application. We demonstrated that AIIA can be integrated into existing LLM/VLM-assiste robotic applications provide failure reasoning and feedback, helping to accelerate and improve task success rates in these systems.
AнА generalizes across embodiments, unseen enviromments, and novel tasks. To ensure fairness and eliminate bias in the detection and reasoning capabilities of AHA, we evaluated it on three different datasets that were never seen during fine-tuning, each designed to test a specific form of generalization. First, the AHA dataset (test) dataset, AHA demonstrated its ability to generalize reasoning across tasks and new behaviors within the same domain, outperforming the second- best performing VLM, GPT-4o ICL, by an average margin of 12.6% difference across all evaluation metrics. Second, we assessed AHA-13B on a dataset generated by the Fai Failgen wrapper a different simulation domain, ManiSkill, showing that model outperforms GPT--4-ICL by an average of 13.4% difference across all metrics as depicted in Table Lastly, to demonstrate evaluated AtAА-1З
generalization to real-world robots and different embodiments, RoboFail (Liu et al. 2023d), where outperforms GPT-4o-IC -CL- by 4.9% difference.
AHA retains common sense knowledge. We evaluated AНА- 13B's performance on various VQA benchmarks and present the results in Table AHA-13B performs comparably to LLaVA-v1.5. 13B (LLama-22 (Liu al. 2023a) with only a 1.5% margin difference as depicted in Table is VLM trained on the same pre-trained weights as AHA-13B but fine-tuned on VQA data. This indicates that AНA-13B capable of functioning as general purpose VLM, in addition to excelling at failure reasoning.
3. Notably, LLaVA-v1.5-12
AHA'S performance scales with data size. We evaluated Aha's performance using a range of AHA data for instruction fine-tuning, spanning [3k, 6k, 12k, 34k, 48k, 60k), and co-trained individual checkpoints corresponding to these data sizes as shown in Figure 3 (Left). The model was then assessed on the ManiSkill-Fail dataset across four evaluation metrics. An average quadratic fit of 0.0022 across all four metrics demonstrates a scaling effect with fine-tuning on our procedurally generated data pipeline. This suggests that further scaling can improved model performance.
5.3 DOWNSTREAM ROBOTICS TASKS
We demonstrate that AlIA's failure detection and reasoning capabilities useful across wide spectrum of downstream robotics applications. This includes automatic reward generation for reinforcement learning applications (Ma et al. |l.||2222222 (023). automatic task plan generation for task and motion planning applications (Curtis et al. 2024), and as an improved verification step for automatic data generation systems (Duan et 2024).
AHA enables efficient reward synthesis for reinforcement learning. To evaluate this downstream task, we adapted Eureka's (M et al. 2023) implementation to the ManiSkill simulator, which offers more state-based manipulation tasks. We strictly followed the Eureka reward function generation and reflection pipeline, modifying it by incorporating perception failure feedback via either AHA- 13B or GPT-4o (acting baseline) enhance the original LLM reflection mechanism. Instead of only including a textual summary of reward quality based on policy training statistics for automated reward editing, further incorporated explanations of policy failures based evaluation rollouts. We

---

Published as a conference paper at ICLR 2025
evaluated our approach on five reinforcement learning tasks from ManiSkill, ranging from tabletop to mobile manipulation. To systematically assess the reasoning capabilities of different VLMs under budget constraints, we sampled one reward function initially and allowed for iterations over two sessions of GPT API calls. Fach policy was trained using PPO over task-speeciic training steps and evaluated across 1,000) test steps. During policy rollouts, we employed either AHA-13B or GPT--4o for reward reflection to improve the reward function. Comparing the evaluated policy success rates using different failure feedback VLMs, we observed that AIIA-13B provided intuitive, human-level failure reasoning that aided in modifying and improving generated dense reward functions. This resulted in success across all five tasks within the budget constraints, and our approach outperformed GPT-4o by a significant margin of 22.34% in task success rate shown in Figure 3 (Right).
AHA refines task-plan generation for TAMP. To demonstrate AHA'S utility within a planning system, we incorporated our approach into PRoC3S (Curtis et al. (024). The PRoC3S system solves tasks specified in natural language by prompting an LLM for a Language-Modd Program (LMP) that generates plans, and then testing a large number of these plans within a simulator before executing valid plans on a robot. If no valid plan can be found within a certain number of samples (100 in our experiments), the LLM is re- prompted for a new LMP given failure information provided by the environment. Importantly, as is typical of TAMP methods, the original approach checks for a finite set of failures (inverse kinematics, collisions, etc.) from the environment, and returns any sampled plan that does not fail in any of these ways. We incorporated a VI.M into this pipeline in two ways: (I) we prompt the VLM with visualizations of failed plan executions within the simulator, ask it to return an explanation for the failure, and feed this back to PRoC3S LLM during the LMP feedback stage, (2) after PRoC3S returns valid plan, we provide a visualization of this to the VLM and ask it to return whether this plan truly achieves the natural language goal, with replanning triggered not. We compared GPT-4o and AHA- 1 13B as the VLM- based failure reasoning modules within this implementation of PRoC3S across three tasks (shown in Figure 4. Each task was evaluated over 10 trials, with a maximum of 100 sampling steps and three feedback cycles provided by either GPT-40 or AHA- 13B. The success rate for each task recorded. As shown in Figure Figure (Right), utilizing AHB-13B for failure reasoning significantly improved the task success rate and outperforming GPT-4o by a substantial margin of 36.7%.
AHA improves task verification for zero-shot robot data generation. To demonstrate AHA'S utility in zero-shot robot demonstration generation, we integrated our approach into the Manipulate-Anythi.e
framework. This onen-ended system employs various Vision-Language Models (VLMs) to generate diverse robot trajectories and perform wide range of manipula- tion tasks without being constrained by predefined actions or scenarios. A critical component of Aaaanipulate-Ayttiii
its sub-task verification module, which analyzes past and current frames to decide whether sub-task has been achieved before proceeding or re-iterating over the previous sub-task. We replaced the original VLM (GPT-4V) in the sub-task verification module with AHA- and evaluated performance across four RLBench tasks (Figure 4 conducting 25 episodes for each task. Our results show that substituting the sub-task verification module's VLM with АНA improved reasoning accuracy and overall task success by an average of 5%.
6 CONCLUSION
Limitations. AHA currently outputs language reasoning that is closely aligned with the failure scenarios in the fine-tuning data. However, there is an opportunity to output more open-ended failures, to cover those arising from modes outside of the ones included in the failure taxonomy. Additionally, while FailGen systematically curates failure data from simulations, distilling large pretrained policies to perform diverse tasks in simulation and sampling failure modes would allow us to generate more open-ended failure examples, potentially enhancing the instruction-tur tuned performance of AHA.
Conclusion. We introduce AHA, an open-source Iision-language model that significantly enhances robots' ability to detect and reason about manipulation task failures using natural language. By framing failure detection as a free-form reasoning task, AHA not only identifies failures but also provides detailed explanations adaptable to various robots, tasks, and environments. Leveraging FailGen and the curated AILA dataset, we trained AIIA on a diverse set of robotic failure trajectories. Our evaluations show that AHA outperforms existing models across multiple metrics and datasets. When integrated into manipulation frameworks, its natural language feedback greatly improves error recovery and policy performance compared to GPT-4 models. These results demonstrate AHA's effectiveness in enhancing task performance through accurate error detection and correction.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project done as part of Jiafei Duan's NVIDIA internship project. Jiafei Duan is supported by the Agency for Science, Technology and Research (A*STAR) National Science Fellowship. Wilbert Pumacay is supported by grant 234-2015-FONDECYS from Cienciactiva of the National Council for Science, Technology and Technological Innovation (CONCYTEC-PER)) I would also like to extend my gratitude to all members of the NVIDIA Seattle Roboties Lab for their invaluable suggestions and feedback.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint wwXiv:2303.08774 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning, Advances in neural information processing systems, 35:2371 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-actinn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818 2023.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialvIm: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint arXiv:2401.12168, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint rrXiv:2107.03374 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane I Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Aidan Curtis, Nishanth Kumar, Jing Cao, Tomás Lozano-Pérez, and I Leslie Pack Kaelbling. Trust the proc3s: Solving long-horizon robotics problems with 1lms and constraint satisfaction, 2024. URL
wwwpr//////////// htttp:://ariv.. org. abs 2406. 05572
Yinpei Dai, Jayjun Lee, Nima Fazeli, and Joyce Chai. Racer: Rich language-guider failure recovery policies for imitation learning. arXiv preprint M.XXv:2409.14677 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzuan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378. 2023.
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. lision-la models as success detectors. arXiv preprint arXiv:2303.07280, 2023.
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230-244 244,, 2022.
Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. manipulate-anything Automating roal-world robots using vision- languaee models. arXiv preprint arXiv:2406. 18915, 2024.
11

---

Published as a conference paper at ICLR 2025
Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, et al. Foundation models in robotics: Applica- tions, challenges, and the future. arXiv preprint arXiv:2312.07843, 2023.
Caelan Reed Garrett, Tomás Lozano-Pérees Pérez, and Leslie Pack Kaelbling. PddIstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the international conference on automated planning and scheduling, volume 30, pp. 440-448, 2020.
Alison Gopnik. Childhood as a solution to explore-exploit tensions. Philosophical Transactions of the Royal Society B, 375(1 (00000800))000000000 2020.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18995-19012, 2022.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot
skill acquisition. In Conference on Robot Learning, pp. 3766-3777. PMLR, 2023.
Gail D Heyman. Children's 's critical thinking when learning from others. Current directions in psychological science, 17(5):344-347,22) 2008.
Ryan Hoque, Ajay Mandlekar, Caelan Garrett, Ken Goldberg, and Dieter Fox. Intervengen: Inter- ventional data generation for robust and data-efficient robot imitation learning. arXiv preprint arXiv:2405.01477 2024.
Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Zhibo Zhao, et al. Toward general-purposs robots via foundation models: A survey and meta-analysis. arXiv preprint arXiv:2312.08788 2023.
Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.08248, 2024a.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973. 2023.
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal rea- soning of relational keypoint constraints for robotic manipulation. arXiv preprint arXhv:2409.01652. 2024b.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Parag Khanna, Elmira Yadollahi, Mårten Björkman, Iolanda Leite, and Christian Smith. User study exploring the role of explanation of failures by robots in human robot collaboration tasks. arXiv preprint arXiv:2303. 16010, 2023.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna. Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXivv2403.129.
2024.
12

---

Published as a conference paper at ICLR 2025
Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai. Yong Jae Lee, et al. Llara: Supercharging robot learning data for vision-language policy. arXiv preprint arXiv:2406.2000
2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprini arXiv:2305. 10355, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human
falsehoods. arXiv preprint arXiv:2109.07955 2021.
Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through
visual prompting. arXiv preprint arXiv:2403.03/74, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao. Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et Mmbench: your multi-modal model an all-around player? 2023c.
arXiv preprint arXiv:
Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306. 2023d.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun
Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference Neural Information Processing Systems (NeurIPS), 2022.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implice pre- vre-training. arXiv preprint arXhv:2210.0000 2022.
Yecheng Jason Ma. William Liang, Guanzhi Wang. De-An Huang, Osbert Bastani, Dinesh Jayaraman. Yuke Zhu, Linxi Fan, and Anima Anandkumar. Fureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 1...2231...991.... 12931, 2023.
Yecheng Jason Ma, William Liang, Hung-Ju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. Dreureka: Language model guided sim-to-real transfer. arXiv preprint arXiv:2406.01967 2024.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:23 /7596, 2023.
OpenAl. Hello gpt-4o, May 2024. URL
wwt.iic.ccmiiiiiaaaa.aaoee.eeoam http:///ww...cmmm//////////e///e 1o -pp-ppp--440 40
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.0886. 2023.
Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The colosseum: benchmark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.08 191, 2024.
13

---

Published as a conference paper at ICLR 2025
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, lean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings the the IEEE Conference on Computer Vision and Pattern Recognition, pp. 131-63326 2019.
Marta Skreta, Zihan Zhou, Jia Lin Yuan, Kourosh Darvish, Alán Aspuru-Guzik, and Animesh Garg. Replan: Robotic replanning with perception and language models. arXiv preprint 1rX1::2401.04157, 2024.
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnay Gurha, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai, 2024. URL https
//arxiv. org/ 2b10 2410. 00425
Julen Urain, Ajay Mandlekar, Yilun Du, Mahi Shafiullah, Danfei Xu, Katerina Fragkiadaki, Georgia Chalvatzaki, and Jan Peters. Deep generative models in robotics: A survey on learning from multimodal demonstrations. arXiv preprint WrXiv:2408.04360 2024.
Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang. Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint arXiv:2310.0136/ 2023a.
Yi Ru Wang, Jiafei Duan, Dieter Fox, and Siddhartha Srinivasa. Newton: Are large language models capable of physical reasoning? arXiv preprint arXiv:2310.07018 2023b.
Annic Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 3153-3160. IEEE, 2024.
Sean Ye, Glen Neville, Mariah Schrum, Matthew Gombolay, Sonia Chernova, and Ayanna Howard. Human trust after robot mistakes: Study of the effects of different forms of robot communication. In 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 1-7. IEEE, 2019.
Peyton Young. Learning by trial and error. Games and economic behavior, 65(2):626-643) 2009.
Samson Yu, Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. Octopi: Object property reasoning with large tactile- models. arXiv preprint arXiv:2405.02794, 2024.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A languae model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.1093 10721, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 3836-3847, 2023.
Zhi Zheng, Qian Feng, Hang Li, Alois Knoll, and Jianxiang Feng. Evaluating uncertainty-bbsed failure detection for closed-loop Ilm planners, 2024. URL htlL/tppps///xxi..crr/////////// 00430
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint 23X.v:2307.38888 2023.
14

---

Published as a conference paper at ICLR 2025
7 APPENDIX
7.1 OVERVIEW
The Appendix contains the following content.
Failure Taxonomy (Appendix (72): more thorough definition and figure to discussions about the different failure modes.
FailGen Data Generation Pipeline (Appendix 7.3 more discussion of FailGen imple- mentation with example configurations files.
Анa Datasets (Appendix 7.4) more details on the instruction-tuning dataset and evaluation datasets.
Additional Experimental Results (Appendix 7.5 more details and experiments with
instruction finetuning.
Downstream Robotic Application: VLM Reward Generation (Appendix 7.6 more
policy rollouts, generated reward function examples, and prompts.
Downstream Robotic Application: VLM Task-plan Generation(Apppedixx more policy rollouts, generated task-plan examples, and prompts.
Downstream Robotic Application: VLM Sub-task Verification(Appendii 7.8 more
policy rollouts.
7.2 FAILURE TAXONOMY
We conducted an in-depth study of recent real-world, diverse datasets (such as Open-X X (Padalkar ct al., 2023), DROID (Khazatsky et al. (024), and EGO4D (Grauman et al. 2022)) and the policies trained using these datasets. Through this analysis, we identified several common modes of failure, which can be categorized into seven types: incomplete grasp, inadequate grip retention, misaligned keyframe, incorrect rotation, missing rotation, wrong action sequence, and wrong target object.
Incomplete Grasp (No alo_Graspp Failure: No_Gra No_Graap is an object-centric failure that occurs when the gripper reaches the desired grasp pose but fails close before proceeding the next keyframe.
Slip is an object-centric failure that occurs after the object has been successfully grasped. As the gripper moves the object toward the next task-specific keyframe, the grip weakens, causing the object to slip from the gripper. For generating the AhA dataset for training and evaluation, configured 5-timestep activation for the S1 ip failure mode, triggering the object to drop from the gripper.
Inadequate Grip Retention (Slip) Failure:
Misaligned keyframe (Translations Failure: This action-centric failure occurs when the gripper moves toward task keyframe, but a translation offset along the X, Y, or Z axis causes the task to fail. For the AнА training and evaluation dataset, we introduced translation offset of [-0.5, 0.5] meters. In the ManiSkill-F -Fiil dataset, we applied translation noise of [O, 0.1] meters along either the X, Y, or Z. axis from the original waypoint. The translation coordinate system is depicted Figure 7 (Left)
Rotati on is an action failure that occurs when the gripper reaches the desired translation pose for the sub-task keyframe, but there is an offset roll, yaw, pitch, leading to task failure. For the AlIA dataset, set rotation offset of [-3.14, 3. 3.14] in radians along roll, yaw, or pitch. The rotation coordinate system is depicted in Figure (Right).
Incorrect Rotation (Rotation) Failure:
Missing Rotation (No Rotation) Failure: No_Rotat ion is action-centric failure that happens when the gripper reaches the desired translation pose but fails to achieve the necessary rotation (roll, yaw, pitch) for the sub-task, resulting in task failure.
Wrong Action Sequence [Wrong_action) Failure:
Wrong_ace acttg_cctiii ion an action-centric failure that occurs when the robot executes actions out of order, performing action keyframe before the correct one. For example, in the task qut__uue____ iiniinnddree drawer, the robot moves the cube toward the drawer before opening it, leading to task failure.
Wrong Target Object (Wrong (Wron__object) Failure: Wzong_ _oog___b________ tep_oojjecc ject is an object-centric failure that occurs when the robot acts the wrong target object, not matching the language instruction. For
15

---

Published as a conference paper at ICLR 2025
Pitch !
X
Roll
Yaw
Figure 5: Failure mode reference coordinate systems. (Left) Translation coordinate system, and (Right) rotation coordinate system.
example, in the task perform a sweep through all manipulable objects, swapping them with the target object in the scene.
li__re__cvl cup, the gripper picks up the green cup, causing failure. We
7.3 FAILGEN DATA GENERATION PIPELINE
We developed Fai Gen, environment wrapper that be easily integrated into any simulator. leverages pre-defined or hand-crafted robot demonstrations for imitation learning, where each trajectory is represented task, with each sub-task linked to a predefined set of language descriptions. FailGen allows for modifications environment parameters, such gripper end-effector translation, rotation, and open/close state. By altering these parameters, we systematically generate failures at every waypoint. However, for the 79 tasks collected from RLBench, do not initially know which sub-task will fail due to specific failure modes. To address this, we perform systematic sweep, using RI RIBeench' 's built- success conditions explore all possible combinations. This generates a configuration of failures for each task, which we then use procedurally generate all failure training data. Additionally, manually annotate each sub-task with natural language instructions describing the task, and pair this with failure mode explanations serve as language input for instruction-tuning Example of the configuration files are depicted at Figure
waypoint-basee -based system. Two consecutive waypoints form sub-
7.4
AHA DATASET
Using FailGer Gein curated datasets from RLBench (James 2020). The first is the training dataset, AlIA dataset (train), which is used for instruction-tuning AIIA alongside the co-train dataset. The second the testing dataset, AIIA dataset (test), used for evaluation. AIIA dataset (train) contains approximately 49k image- query pairs of failures derived from 79 tasks, while AHA dataset (test) consists of around 11k querg pairs from 10 hold-out tasks.
7.5 ADDITIONAL EXPERIMENTAL RESULTS
We conducted additional experiments better understand and visualize AIIA's predictions. We trained two versions of the AHA model with 13B parameters, using different language models for fine-tuning: Llama-2- 13B and Vicuna-1.5-115 The results showed less than 2.5% performance difference between the two models, indicating that our fine-tuning data effective regardless the base language model. These results presented in Table Additionally, visualized the output
16

---

Published as a conference paper at ICLR 2025
whore
the deeis
sane_attc
1 save_path:
sappciati (+,
2 obs_mades rgb
faclarc, plaoocresooppo
3 render_mode: sensors
Falle
shader: default
Types lon_
5
sim_backend: auto
Falis weypetadis [1-2,10
6
image_size: [256, 256]
7
stages: [0, 3]
Types retation,,
8
failures:
Patte
woypetnd's
type: grasp
10
enabled: false
Types wrong, pequaace
11
stages: [2]
voypetntat 12,31
Palie
12
type: trans_]
13
enabled: false
task,
14
stages: [e, 1,
cargetas Iaaill
wayparess]
15
noise: 0.1
"grass onto the clack Inei",
16
type: trans_y
"pick clack lnne"
17
enabled: false
Cask,
18
stages: [0, 1, 3]
Cargetia [ati]
procesonst beyostntt, woppart221 *rstate the leak",
19
noise: 0.1
20
type: trans_z
keal?
21
enabled: false
22
stages: [0, 1, 3]
Cargetia [ail]]
23
noise: 0.1
procesess (oepelant,, wayslrer)
24
*lat "valease grioper"
Figure 6: (Left) Example of config file one task for Maniskill-Fal. (Right) Example of config file for AHA task
direent Jee0 drrbutte
EPV
Figure 7: Data distribution of AНA dataset for both train and test.
predictions from various baselines compared to our model and evaluated performance across all three datasets, with the results shown in Figure 7
AHA model performance uncertainty estimation. To evaluate the relationship between uncertainty estimation and model performance, we conducted additional experiments across various evaluation datasets. Specifically, we compared the sentence token prediction probabilities of AHA-13B with those of LLaVA v3. AHA-13B exhibited significantly higher average prediction probabilities, reflecting its superior accuracy. These findings underscore the positive impact of fine-tuning with the AHA failure dataset model confidence and performance depicted in Table 5
17

---

Published as a conference paper at ICLR 2025
No_ No_grasp and Rotation x. Row 2: Rotation_ and Rotation_z. Row 3: slip and Translation and Translation Row Translation_2 and Wrong_object.
Figure 8: Examples of differrnt failure modes. Row 1:
Row 4:
Effects Viewpoints on Evaluation. We evaluated the reasoning capabilities of the AHA model on the ManiSkill-Failun dataset across three different viewpoint configurations (one, two, and three viewpoints). Interestingly, we observed a slight performance advantage when using single-viewpoint images. We attribute this to the resolution limitations of the LLaVA- visual encoder (256x256), where kingle-viewpoint inputs offer clearer and more focused visual information for failure reasoning, summarized in Table
Model: AHA-13B (Viewpoints)
Binary Success
ROUGE-L
LLM Fuzzy Match
Cosine Similarity
One viewpoint Two viewpoints
1.000
0.673
0.587
0.712
1,000
0.615
0.587
0.671
Three viewpoints
1,000
0.600
0.633
0.681
Table 4: Performance comparison across different numbers of viewpoints for AHA-13B
18

---

Published as a conference paper at ICLR 2025
Dataset
A1IA-131 (Output Probubilities Cosine Similarity) L.L.VA-13.....55 (Output Probabilities Cosine Similarity)
AIIA Dataset (Test)
0.670/0.50 00.58
0.066/ 0.208
Maniskill Fail
0.457 0.681
0.024 0,2008
RoboFail
0.292 /
2.000/0.200
Table 5: Performance against model prediction sentence probabilities likelihood evaluated across datasets.
Table 6: Ablation on Different Base LLMss for Fine- Fine-Tuning. We fine-tuned AНА-13 13B using both LLaMA- 2-13B and Vicuna- 1.5- 13B base LLM models. The quantitative results show that the average performance difference between two models are effective regardless of the base model selection.
than 2.5%, indicating our failure formulation and AHA dataset
Alla dhtnet Floo
Mandkili-ar
BetaPal
Modds ARDL
ROCUR:
On Sini
FRog NEAET
ROCOEC
On Six
Paoss Naxk
HOUGEE
CosSen
Briswet9)
Fancy Macs
0/30
6X81
B633
8643 6661
nax
ARDL 1S UNform 5)
158
6.911
62W9
045
0.5%
0661
1/00
0.30
7.6 VLM REWARD GENERATION
In this section, present reward functions generated by GPT-4o and AHA for comparison, shown in Figure [ Additionally, we demonstrate RL policy rollouts improved through AНA 's failure feedback across all five tasks along with all the final dense reward function modified by AHA shown in Figure 10 and II For all tasks, except put sphere _n holder (trained with PPO for 10M steps), PPO was trained for 25M steps prior reflection and evaluation.
Simulation task Details We describe each of the 4 tasks in detail, along with their Maniskill variations and success condition.
7.6.1 PICXUP YCB
Filename: pick_single_yee
Task:
Pick up the single YCB object and lift it up to target height.
Success Metric: The object position within goal thresh (default 0.025) euclidean distance of the goal position.
7.6.2 PUSH
Filename: push_T.py
Task: Push the into the T shaped area.
Success Metrie: The 3D T block covers at least 90
7.6.3 PLACE SPHERE
Filename: place_sphere__l. wtw.peeeeev.py..
Task: Pick up the sphere and place it into the bin.
Success Metric: the sphere is top of the bin. That is, the sphere's xy-distance to the bin goes near and its z-distance to the bin goes near the sphere radius the bottom bin block's side length the object is static. That is, its linear and angular velocities are bounded with small value the gripper is not grasping the object.
7.6.4 STACK CUBE
Filename: stack_cube_vl_
Task: Pick up the red cube and stack it onto the green cube.
Success Metric: the red cube on top of the green cube (to within half the cube size), the red cube is static, the red cube is not being grasped by the robot (robot must let go of the cube).
19

---

Published as a conference paper at ICLR 2025
Table Examples of detection and reasoning over failures by various VLMs. We sampled one evaluation data point from each of the three evaluation datasets and compared the reasoning predic- tions generated by different state-of-the-ane VLMs alongside our AHA- 13B. Our model consistently provides the most semantically similar reasoning the ground-truth answers for the given queries.
Evaluation Datasets
AHA dataset (Test)
Maniskill Fail
RohoFail
(Liu al. 2023d
Question
At the current suh-task, the robot picking up the hockey süick. First, determine whether it sucoeded by choosing ["yes", "no"]. If not, explain why lailed?
current sub-task, the mbot is picking up red cube. determine whether has suc- choosing l"yes", "no"]. ]. If not, explain why it Failed?
At the current sub-tack, the rohox placing the apple into bowl. First, determine whether succeeded by choosing from ["yes", "no"]. If not, explain why failed!
LLaVA 1.5-13B
The image displays a time-lapse sequence of rohotic arm grah-
Based on the image, it depicts different sub-task of rohot grasp-
For the pick-up task, i would the robot arm has succeeded in picking.
bing hockey stick.
ing block.
GPT4o
No, the robot arm fails to hold the backey securely.
Yes, the robot arm has success- fully picked up the block.
No, the rebot gripper failed to pick up in the
Gomini 1.5 Flash
Yos.
Yea, Yes, Yos.
Frame 1, no the gripper is not
grasping bowl and move
AнA-13B (Ours)
No, the robot slip the object out of its gripper.
No, the robot gripper fails close the gripper.
No, this not the right action se- queace for
Ground-trath Answer
No, the rohor slip the object out of ibs gripper.
Nouthe robot gripper fails to close.
No, this not the right action se-
quence for task.
7.6.5 OPEN DRANER
Filename: awee__cabinetddraweeee
Task:
Pull open the drawer.
Success Metric: The drawer is open at least 90% of the way, and the angular/linear velocities of the drawer link are small.
EXT benc
Loneeg prape
movicc bark tant, 20k 2111
Figure 9: (Left) Example of improved dense reward function using GPT-4o for reflection. (Right) Example of improved dense reward function using AhA for reflection
20

---

Published as a conference paper at ICLR 2025
Timstaps
Figure 10: RL policy roll-outs via improved with AнА. Row I: pickup_ YCB. Row 2: push T. Row 4: stack_cube. Row 5: open_drawer.
Row 3: Pl Place_sphere
7.7 VLM TASK-PLAN GENERATION
In this section, present the policy rollouts improved by AHA in Figure 12 along with the modified task plans in Figure 13
Simulation task Details We describe each of the tasks in detail, along with their PyBullet variations and success condition.
7.7.1 PUT BANANA CENTRE
Filename: currs_rave__cc_pice
Task:
Pick up the banana and place it onto the centre of the table.
Success Metric:
The success condition on the final location of the banana with respect to the table
area.
7.7.2 STACK BANANA
Filename: ours_yck hars__ybbbananaa www.aaaaanaaaaaa sarrra_spaa_sstaae
Task:
Pick up the banana and place it onto the spam can.
Success Metric: The position of the banana should be the spam can, and rest stably.
21

---

Published as a conference paper at ICLR 2025
'Push to shaped area'
"Stack the red cube onto cube*
'Put sphere into sphere holder*
oigirn and caae 1ar campharnnn maane Eide
Hoaeh grggen
L pepeec tas
up Sear, Bait)
up
sat pinc
2011
Lif You soaing
Jeuit
Hist cRIeA W0st
Figure 11: Examples of modified reward function via AHA
7.7.3 STACKS CUBES
Filename: ours_ravess raw___vve__aaaae wwwavee__ww_sttak..__om
Task:
Pick up the green cube and place into the green bowl, and then take the yellow cube and stack it on top of the green.
Success Metric: When the yellow cube stably stack on top of the green in the green bowl.
7.8 VLM SUB-TASK VERIFICATION
(Duan et al. 2024) as the main policy framework, integrating it with AHA. AHA functions as a sub-task verifier VLM, playing a crucial role in ensuring task success when using Saanipulaae-Anythinn Examples of the roll-outs are shown in Figure [4]
In this section, we leverage
Simulation task Details We describe each of the 4 tasks in detail, along with their RLBench variations and success condition.
7.8.1 PUT BLCCK
Filename: put_block.py
Task:
Pick up the green block and place it on the red mat.
Success Metric: The success condition on the red mat detects the target green block.
7.8.2 PICXUP CUP
Filename: pickup_cup.py
Task: Pick up the red cup.
Success Metric: Lift up the red cup above the pre-defined location.
22

---

Published as a conference paper at ICLR 2025
Figure 12: TAMP policy roll-outs via improved with AhA. Row I: put_banama_centtre
Row stack banana. Row 3: stack_cubes
7.8.3 SORT MUSTARD
Filename: port_mustard.p.
Task: Pick up the yellow mustard bottle, and place into the red container.
Success Metric: The yellow mustard bottle inside red container.
7.8.4 PICK LIFT
Filename: pick_and and lift.py
Task: Pick up the red cube.
Success Metric: The red eube is lifted up.
7.9 LIMITATIONS AND OPPORTUNITIES
While AHA quite capable, it without limitations. the following sections, we discuss some of these limitations and potential risks for real-world deployment.
Does not cover all failure eases. While AHA is instruction-ttund address the seven most common failure modes in robotic tabletop manipulation, and provides fine ime-grained failure identification down axis-level translation or orientation offsets of the gripper, is inevitable that certain edge cases remain uncovered. These edge cases fall outside the distribution of failure modes considered in our approach and may occur in real-world scenarios. on distilling policies already trained with action-based "artifacts" to generate naturally occurring failures. These could then labeled using human annotations or hybrid approach combining AHA and -o---ed labeling system. Such efforts could enhance the robustness and applicability of failure detection across a broader range of scenarios.
address this limitation, future work could focus
Improving policies directly with AHA Currently, have demonstrated AHA's effectiveness in enhancing existing robotics systems that utilize off-the-shelf VLM&/LLMs to automate various pipeline components. AHA has shown significant impact in improving code generation, task-plan
23

---

Published as a conference paper at ICLR 2025
"Stack banana onto spam can"
"Stack two cubes into a bowl"
Lamee 1972 gren Viek (m))) ADRO)
offec"
vlark yo vleemed Viek eruuing cemeerty
"Move banana to centre"
coaen orte Dng LEENIST use WRApeT US PRttt0T
AAI A.R dllnnyy tor weth ofter
siyt pestic wills coet ofter
Figure 13: Examples of modified task- via AHA
generation, and sub-task success detection. However, AHA is not yet capable of directly influencing
low-level trained policies through failure language reasoning. A potential next step would involve training low-level policies with Lunguage-condition
demonstrations covering a diverse range of corrective actions tied various fallure modes. This would enable low-level policies to interpret failure reasoning in a counterfactual manner and generate corrective actions directly.
24

---

Published as a conference paper at ICLR 2025
Figure 14: Examples of zero-shot data generator trajectories with AНA as sub-tasks verifier.
Row 1: pickup_cube, pickup_cup. Row 2: put_block, sort_mustard
25


Reviews:
[{"review_id": "BOZH6XcQLw", "reviewer": "Reviewer_zG7S", "summary": "This work targets the task of open-world failure detection and reasoning under the scenario of robotic manipulation, which is critical and valuable for diverse downstream tasks. Specifically, the authors propose a procedural pipeline for generating failure demonstration data for robotic manipulation in simulation, upon which they fine-tune a vision-language model (VLM) for failure detection and reasoning. Moreover, they demonstrate improved performance brought by the fine-tuned VLM in three downstream tasks, namely reward function design, task-plan refinement, and sub-task verification. Comprehensive comparison with the other six VLMs also shows the superior performance of the proposed technique on this task.", "questions": "Side questions (major ones in the Weaknesses section above)\n\n- In subsection 3.1., why separate the rotation failure into incorrect and missing ones? Should these two belong to the same category?  \n\n- An ablation study on how many camera viewpoints would be insightful for the real-robot applicability of the proposed method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an important problem that is highly beneficial and safety-critical when learning-based robotic algorithms are deployed in the real world;\n\n- The developed procedural pipeline of simulation data generation is useful for researchers in the community who are willing to contribute to this direction;\n\n- It is inspiring to see that not only improved performance is shown on the standalone task, but to see the strong benefits brought by such improvement on different related and popular downstream tasks.\n\n- Though failure detection is an important prerequisite for the reasoning part, I am still glad to see the efforts in investigating the failure reasoning and especially its usage in the downstream tasks.\n\n- The presentation is generally well-structured and easy to follow.\n\n- Though benchmarking experiments are conducted in the simulation, their results are comprehensive and informative.", "weaknesses": "Presentation:\n- It would be clearer if subsection 4.1. can be shortened a bit and described with a more concise mathematical formulation. \n\n- To reinforce the contribution, I would suggest adding a subsection in the method section to explain how improved failure detection and reasoning performance can help these downstream tasks. In the current draft, putting this part in the experiment section is less apparent and inspiring for potential readers in these fields. \n\n- Closely related to the point above, it seems that the usage of the proposed idea in downstream task 2 (refine task plans in TAMP) and task 3 (improve task verification) is essentially similar, i.e., detect and reason for the failure of the sub-tasks. It might be more lucid for the readers to group these two tasks together along with the reward design task and make a taxonomy for different high-level manners of how failure detection and reasoning can help. \n\n- in sub-section 4.1., \"Unlike previous works that primarily focus on detecting task success as binary classification problem, we approach failure reasoning by first predicting a binary success condition (\"Yes\" or \"No\") of the given sub-task based on a language specification and an input image prompt.\" seems less straightforward to understand. Logically, I suspect the author wants to express that they have an additional reasoning part after the detection step. \n\n\n- Fig.1 and Fig.2 are hard to read. It would be better to reorganize the sub-figures and increase the sizes of the images and texts.\n\nMethodology:\n\n- The first point I would like to raise is the lack of discussion of hallucination in LLM/VLMs. In the introduction, the authors motivate the failure recognition with the shortcoming of hallucinations in LLMs/VLMs. However, the proposed fine-tuned VLM will still suffer from this issue. It would be important to discuss how to mitigate this issue in detecting failures. In this regard, there is a missing citation [1] that is closely related to this work, which made an initial attempt in this direction based on the uncertainty derived from the VLM. I would suggest the authors add a discussion on this aspect. It would also be meaningful to see the influence of fine-tuning with the proposed data set on the performance of uncertainty estimation. Will there be a trade-off between the performance of the prediction accuracy and uncertainty estimation?  \n\n- The second point is related to the design of the technique. In [1], it has been observed that using the \"chain-of-thoughts\" (CoT) analysis in the prompt template can improve the performance of failure detection. Therefore, related to the reasoning part of this work, will the reasoning part be able to help improve the performance failure detection? Will the performance be influenced by the different prompting templates, e.g., with and w/o CoT?\n\nExperiments:\n\n- Due to missing real-world experiments, therefore, it is hard to evaluate the performance loss during sim-to-real transfer and the actual practicality of the proposed technique. \n \n\n[1] \"Zheng, Zhi, et al. \"Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners.\" arXiv preprint arXiv:2406.00430 (2024).\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dalptAoRZA", "reviewer": "Reviewer_Enxm", "summary": "This paper introduces AHA, a new VLM model to identify and describe failure modes in robot manipulation. The authors introduce a pipeline called FailGen to generate failure demonstrations and use the data to train AHA. In the experiments, AHA demonstrates improved performance in failure reasoning. The authors also show its effectiveness in downstream application including VLM-based reward design, planning, and subtask verification.", "questions": "See the weaknesses part. The idea is interesting and I would like to raise my score if the authors can show some rigorous results to clarify the points above, especially 1 and 2. I also encourage discussion on negative results to help readers see the limitation and potential of the current version.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. The idea of using failure detection VLM to improve VLM-based robotic application is interesting.\n2. The effort in developing a failure dataset is valuable towards building robust large models that can understand the physical world.", "weaknesses": "1. [Soundness] It remains unclear to me whether incorporating failure descriptions is a reliable approach to improve VLM in reward design, planning, and subtask verification. For example, in the VLM-based reward design, although VLM can provide some penalty reward terms based on AHA input to correct behavior, there is no guarantee the new reward can make robot perform better than the previous one (e.g. the penalty can be too large and hinder skill learning). In this case, I would like to see if the proposed method can show some iterative reward tuning behavior based on failure feedback and the procedure can converge (i.e., tuning the strength of new terms or number of terms to correct behavior). \n\n2. [Clarity] The paper can be much clearer and convincing if the authors can show more extensive comparisons between AHA and GPT4 to understand what leads to the improvements. For example, is the improvement due to GPT4 not being able to generate accurate failure description? Or, is it simply because the generated failure text is too lengthy for the downstream VLM model to understand? I encourage more explicit discussion on what a good failure description looks like (for downstream task VLMs). I also suggest comparing to human oracles with different writing styles.\n\n3. [Result: Generalization] The paper is focusing on simple pick-and-place style tasks in clean backgrounds. How about more challenging and dexterous tasks? I would like to see the results on more challenging task such as guiding a robot hand to perform grasping and reorientations where the failure modes can be more fine-grained and complex. Besides, explicitly discussing generalization to diverse backgrounds / objects is also necessary. The results on these harder setups can help readers to understand the opportunities and challenges of AHA in application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vYO2rYggf1", "reviewer": "Reviewer_FdiA", "summary": "This work proposes a VLM for detecting and reasoning about failures in robotic manipulation, which can be applied in various downstream tasks.", "questions": "1.What is the difference between rotation and no rotation?\n2.What are the details of your baseline settings? I couldn't find them in your paper.\n3.Are you using the REFLECT framework for your baseline? If not, I believe REFLECT is highly relevant to your work, so I would like to see a comparison of its performance with your model.\n4.How do you plan to extend your failure types? As of now, I still find them somewhat rough.\n\nMy main concern is the sufficiency of the experiments and the work still seems somewhat preliminary. However, if you address my concerns, I will be inclined to raise my score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is meaningful, and I agree that addressing failures in robotics is important.\n2. It can be utilized in downstream tasks.\n3. The website has many great demos.", "weaknesses": "I think the types of failure modes and the reasoning responses are somewhat rough. And more details and experiments need to be conducted. For further details, please refer to the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SLm4xMe9Eo", "reviewer": "Reviewer_qHQf", "summary": "The paper presents a Vision-Language Model (VLM) for robotic failure reasoning, introducing a failure dataset and fine-tuning the VLM to achieve this goal. The approach demonstrates promising performance.", "questions": "- View Angle Concerns: The camera view angles during training are fixed according to the RLBench settings. This raises concerns about the view angles during testing. Specifically: 1) How many views does the model require during testing? 2) Should the view angles during testing match the exact poses used in the RLBench training setting?\n\n- Missing Reference to Figure 1: Figure 1 is not referenced or discussed anywhere in the paper, which needs to be addressed for clarity and completeness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Failure trajectory generation can be extended to other scenarios (simulatores), encompassing a comprehensive set of failure categories. Failure reasoning plays a critical role in downstream policy learning, which is essential for advancing the entire community", "weaknesses": "Baseline Comparisons:\n - Clarity of Baseline Implementation: The implementation details of the baselines are unclear. What type of input data are the baselines using? Is it a single image, a sequence of images, images from multiple views, or video? This distinction significantly affects baseline performance. For instance, Gemini excels at reasoning for robotic manipulation when using video inputs, but its performance would likely degrade if multiple-view images were used instead.\n\n - Lack of Comparison with Robotic-Specific Methods: Why not compare your approach with other robotic-oriented reasoning methods mentioned in the 'Introduction' or 'Related Work' sections? The current experiments only compare against general Vision-Language Models (VLMs), which are not specifically designed for robotic tasks.\n\nI would consider raising my score if these issues are addressed or clarified. Otherwise, it's difficult to properly assess how well the model performs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)